[RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

Hey everyone,

We're having problems with large systems hitting a BUG in
xen_memory_setup, due to extra e820 entries created in the
XENMEM_machine_memory_map callback.  The change in the patch gets things
working, but Boris and I wanted to get opinions on whether or not this
is the appropriate/entire solution, which is why I've sent it as an RFC
for now.

Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
which is a detail worth discussig.  He proposed possibly adding
CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
value than E820MAX, since the Xen e820 table isn't bound by the
zero-page memory limitations.

I do *slightly* question the use of E820_X_MAX here, only from a
cosmetic prospective, as I believe this macro is intended to describe
the maximum size of the extended e820 table, which, AFAIK, is not used
by the Xen HV.  That being said, there isn't exactly a "more
appropriate" macro/variable to use, so this may not really be an issue.

Any input on the patch, or the questions I've raised above is greatly
appreciated!

- Alex

Alex Thorlton (1):
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
1.8.5.6

[PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On systems with sufficiently large e820 tables, and several IOAPICs, it
is possible for the XENMEM_machine_memory_map callback (and its
counterpart, XENMEM_memory_map) to attempt to return an e820 table with
more than 128 entries.  This callback adds entries to the BIOS-provided
e820 table to account for IOAPIC registers, which, on sufficiently large
systems, can result in an e820 table that is too large to copy back into
xen_e820_map.

This change simply increases the size of xen_e820_map to E820_X_MAX to
ensure that there is enough room to store the entire e820 map returned
from this callback.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index f8960fc..5e1ecc7 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -41,7 +41,7 @@
 unsigned long xen_released_pages;
 
 /* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820MAX] __initdata;
+static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
 static u32 xen_e820_map_entries __initdata;
 
 /*
@@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
 	max_pfn = min(max_pfn, xen_start_info->nr_pages);
 	mem_end = PFN_PHYS(max_pfn);
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = E820_X_MAX;
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	op = xen_initial_domain() ?
@@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
 	int i;
 	int rc;
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = E820_X_MAX;
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Ingo Molnar]

* Alex Thorlton <athorlton@sgi.com> wrote:

> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.

This means:

 #ifdef CONFIG_EFI
 #include <linux/numa.h>
 #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
 #else   /* ! CONFIG_EFI */
 #define E820_X_MAX E820MAX
 #endif

It's a bit weird to key it off of CONFIG_EFI - why isn't it unconditional?

But I have no objections to the principle if it fixes a crash.

Thanks,

	Ingo

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On Tue, Nov 15, 2016 at 07:30:35AM +0100, Ingo Molnar wrote:
> 
> * Alex Thorlton <athorlton@sgi.com> wrote:
> 
> > On systems with sufficiently large e820 tables, and several IOAPICs, it
> > is possible for the XENMEM_machine_memory_map callback (and its
> > counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> > more than 128 entries.  This callback adds entries to the BIOS-provided
> > e820 table to account for IOAPIC registers, which, on sufficiently large
> > systems, can result in an e820 table that is too large to copy back into
> > xen_e820_map.
> > 
> > This change simply increases the size of xen_e820_map to E820_X_MAX to
> > ensure that there is enough room to store the entire e820 map returned
> > from this callback.
> 
> This means:
> 
>  #ifdef CONFIG_EFI
>  #include <linux/numa.h>
>  #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
>  #else   /* ! CONFIG_EFI */
>  #define E820_X_MAX E820MAX
>  #endif
> 
> It's a bit weird to key it off of CONFIG_EFI - why isn't it unconditional?
> 
> But I have no objections to the principle if it fixes a crash.

I originally thought that Juergen and Jan's alternative solution
eliminated the need for my change, but it appears that I was incorrect,
at least if we want to get the kernel booting on the older HV
without having to modify it.

I'll remove the ifdef here to make this unconditional.

Thanks, Ingo!

- Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 01:11, Alex Thorlton wrote:
> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: Russ Anderson <rja@sgi.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: x86@kernel.org
> Cc: xen-devel@lists.xenproject.org
> ---
>  arch/x86/xen/setup.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index f8960fc..5e1ecc7 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -41,7 +41,7 @@
>  unsigned long xen_released_pages;
>  
>  /* E820 map used during setting up memory. */
> -static struct e820entry xen_e820_map[E820MAX] __initdata;
> +static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
>  static u32 xen_e820_map_entries __initdata;
>  
>  /*
> @@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
>  	max_pfn = min(max_pfn, xen_start_info->nr_pages);
>  	mem_end = PFN_PHYS(max_pfn);
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = E820_X_MAX;

Please use ARRAY_SIZE(xen_e820_map) here and ...

>  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
>  
>  	op = xen_initial_domain() ?
> @@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
>  	int i;
>  	int rc;
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = E820_X_MAX;

... here.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On Wed, Nov 16, 2016 at 07:09:01AM +0100, Juergen Gross wrote:
> On 15/11/16 01:11, Alex Thorlton wrote:
> > On systems with sufficiently large e820 tables, and several IOAPICs, it
> > is possible for the XENMEM_machine_memory_map callback (and its
> > counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> > more than 128 entries.  This callback adds entries to the BIOS-provided
> > e820 table to account for IOAPIC registers, which, on sufficiently large
> > systems, can result in an e820 table that is too large to copy back into
> > xen_e820_map.
> > 
> > This change simply increases the size of xen_e820_map to E820_X_MAX to
> > ensure that there is enough room to store the entire e820 map returned
> > from this callback.
> > 
> > Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> > Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> > Cc: Russ Anderson <rja@sgi.com>
> > Cc: David Vrabel <david.vrabel@citrix.com>
> > Cc: Juergen Gross <jgross@suse.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: "H. Peter Anvin" <hpa@zytor.com>
> > Cc: x86@kernel.org
> > Cc: xen-devel@lists.xenproject.org
> > ---
> >  arch/x86/xen/setup.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> > index f8960fc..5e1ecc7 100644
> > --- a/arch/x86/xen/setup.c
> > +++ b/arch/x86/xen/setup.c
> > @@ -41,7 +41,7 @@
> >  unsigned long xen_released_pages;
> >  
> >  /* E820 map used during setting up memory. */
> > -static struct e820entry xen_e820_map[E820MAX] __initdata;
> > +static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
> >  static u32 xen_e820_map_entries __initdata;
> >  
> >  /*
> > @@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
> >  	max_pfn = min(max_pfn, xen_start_info->nr_pages);
> >  	mem_end = PFN_PHYS(max_pfn);
> >  
> > -	memmap.nr_entries = E820MAX;
> > +	memmap.nr_entries = E820_X_MAX;
> 
> Please use ARRAY_SIZE(xen_e820_map) here and ...
> 
> >  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
> >  
> >  	op = xen_initial_domain() ?
> > @@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
> >  	int i;
> >  	int rc;
> >  
> > -	memmap.nr_entries = E820MAX;
> > +	memmap.nr_entries = E820_X_MAX;
> 
> ... here.

Got it.  Thanks, Juergen!

- Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 01:11, Alex Thorlton wrote:
> Hey everyone,
> 
> We're having problems with large systems hitting a BUG in
> xen_memory_setup, due to extra e820 entries created in the
> XENMEM_machine_memory_map callback.  The change in the patch gets things
> working, but Boris and I wanted to get opinions on whether or not this
> is the appropriate/entire solution, which is why I've sent it as an RFC
> for now.
> 
> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
> which is a detail worth discussig.  He proposed possibly adding
> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
> value than E820MAX, since the Xen e820 table isn't bound by the
> zero-page memory limitations.
> 
> I do *slightly* question the use of E820_X_MAX here, only from a
> cosmetic prospective, as I believe this macro is intended to describe
> the maximum size of the extended e820 table, which, AFAIK, is not used
> by the Xen HV.  That being said, there isn't exactly a "more
> appropriate" macro/variable to use, so this may not really be an issue.
> 
> Any input on the patch, or the questions I've raised above is greatly
> appreciated!

While I think extending the e820 table is the right thing to do I'm
questioning the assumptions here.

Looking briefly through the Xen hypervisor sources I think it isn't
yet ready for such large machines: the hypervisor's e820 map seems to
be still limited to 128 e820 entries. Jan, did I overlook an EFI
specific path extending this limitation?

In case I'm right the Xen hypervisor should be prepared for a larger
e820 map, but this won't help alone as there would still be additional
entries for the IOAPICs created.

So I think we need something like:

#define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)

and use this for sizing xen_e820_map[].


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
> On 15/11/16 01:11, Alex Thorlton wrote:
>> Hey everyone,
>> 
>> We're having problems with large systems hitting a BUG in
>> xen_memory_setup, due to extra e820 entries created in the
>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>> working, but Boris and I wanted to get opinions on whether or not this
>> is the appropriate/entire solution, which is why I've sent it as an RFC
>> for now.
>> 
>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>> which is a detail worth discussig.  He proposed possibly adding
>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>> value than E820MAX, since the Xen e820 table isn't bound by the
>> zero-page memory limitations.
>> 
>> I do *slightly* question the use of E820_X_MAX here, only from a
>> cosmetic prospective, as I believe this macro is intended to describe
>> the maximum size of the extended e820 table, which, AFAIK, is not used
>> by the Xen HV.  That being said, there isn't exactly a "more
>> appropriate" macro/variable to use, so this may not really be an issue.
>> 
>> Any input on the patch, or the questions I've raised above is greatly
>> appreciated!
> 
> While I think extending the e820 table is the right thing to do I'm
> questioning the assumptions here.
> 
> Looking briefly through the Xen hypervisor sources I think it isn't
> yet ready for such large machines: the hypervisor's e820 map seems to
> be still limited to 128 e820 entries. Jan, did I overlook an EFI
> specific path extending this limitation?

No, you didn't. I do question the correlation with "large machines"
here though: The issue isn't with large machines afaict, but with
ones having very many entries (i.e. heavily fragmented).

> In case I'm right the Xen hypervisor should be prepared for a larger
> e820 map, but this won't help alone as there would still be additional
> entries for the IOAPICs created.
> 
> So I think we need something like:
> 
> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
> 
> and use this for sizing xen_e820_map[].

I would say that if any change gets done here, there shouldn't be
any static upper limit at all. That could even be viewed as in line
with recent e820.c changes moving to dynamic allocations. In
particular I don't see why MAX_IO_APICS would need adding in
here, but not other (current and future) factors determining the
(pseudo) E820 map Xen presents to Dom0.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
> On 15/11/16 01:11, Alex Thorlton wrote:
>> Hey everyone,
>> 
>> We're having problems with large systems hitting a BUG in
>> xen_memory_setup, due to extra e820 entries created in the
>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>> working, but Boris and I wanted to get opinions on whether or not this
>> is the appropriate/entire solution, which is why I've sent it as an RFC
>> for now.
>> 
>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>> which is a detail worth discussig.  He proposed possibly adding
>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>> value than E820MAX, since the Xen e820 table isn't bound by the
>> zero-page memory limitations.
>> 
>> I do *slightly* question the use of E820_X_MAX here, only from a
>> cosmetic prospective, as I believe this macro is intended to describe
>> the maximum size of the extended e820 table, which, AFAIK, is not used
>> by the Xen HV.  That being said, there isn't exactly a "more
>> appropriate" macro/variable to use, so this may not really be an issue.
>> 
>> Any input on the patch, or the questions I've raised above is greatly
>> appreciated!
> 
> While I think extending the e820 table is the right thing to do I'm
> questioning the assumptions here.
> 
> Looking briefly through the Xen hypervisor sources I think it isn't
> yet ready for such large machines: the hypervisor's e820 map seems to
> be still limited to 128 e820 entries. Jan, did I overlook an EFI
> specific path extending this limitation?

No, you didn't. I do question the correlation with "large machines"
here though: The issue isn't with large machines afaict, but with
ones having very many entries (i.e. heavily fragmented).

> In case I'm right the Xen hypervisor should be prepared for a larger
> e820 map, but this won't help alone as there would still be additional
> entries for the IOAPICs created.
> 
> So I think we need something like:
> 
> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
> 
> and use this for sizing xen_e820_map[].

I would say that if any change gets done here, there shouldn't be
any static upper limit at all. That could even be viewed as in line
with recent e820.c changes moving to dynamic allocations. In
particular I don't see why MAX_IO_APICS would need adding in
here, but not other (current and future) factors determining the
(pseudo) E820 map Xen presents to Dom0.

Jan

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 08:15, Jan Beulich wrote:
>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>> On 15/11/16 01:11, Alex Thorlton wrote:
>>> Hey everyone,
>>>
>>> We're having problems with large systems hitting a BUG in
>>> xen_memory_setup, due to extra e820 entries created in the
>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>> working, but Boris and I wanted to get opinions on whether or not this
>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>> for now.
>>>
>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>> which is a detail worth discussig.  He proposed possibly adding
>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>> zero-page memory limitations.
>>>
>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>> cosmetic prospective, as I believe this macro is intended to describe
>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>> by the Xen HV.  That being said, there isn't exactly a "more
>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>
>>> Any input on the patch, or the questions I've raised above is greatly
>>> appreciated!
>>
>> While I think extending the e820 table is the right thing to do I'm
>> questioning the assumptions here.
>>
>> Looking briefly through the Xen hypervisor sources I think it isn't
>> yet ready for such large machines: the hypervisor's e820 map seems to
>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>> specific path extending this limitation?
> 
> No, you didn't. I do question the correlation with "large machines"
> here though: The issue isn't with large machines afaict, but with
> ones having very many entries (i.e. heavily fragmented).

Fact is: non-EFI machines are limited to 128 entries.

One reason for fragmentation is NUMA - which tends to be present and
especially adding many entries only with lots of nodes - on large
machines.

So while you are of course right that the problem isn't the size of
the machine, but the memory fragmentation, the chances to show up
are much higher on large machines.

So I'd rephrase:

Looking briefly through the Xen hypervisor sources I think it isn't yet
ready for machines with many e820 entries due to memory fragmentation
to be found e.g. on very large NUMA machines with lots of nodes: ...

>> In case I'm right the Xen hypervisor should be prepared for a larger
>> e820 map, but this won't help alone as there would still be additional
>> entries for the IOAPICs created.
>>
>> So I think we need something like:
>>
>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>
>> and use this for sizing xen_e820_map[].
> 
> I would say that if any change gets done here, there shouldn't be
> any static upper limit at all. That could even be viewed as in line
> with recent e820.c changes moving to dynamic allocations. In
> particular I don't see why MAX_IO_APICS would need adding in
> here, but not other (current and future) factors determining the
> (pseudo) E820 map Xen presents to Dom0.

The hypervisor interface of XENMEM_machine_memory_map requires to
specify the size of the e820 map where the hypervisor can supply
entries. The alternative would be try and error: start with a small
e820 map and in case of error increasing the size until success. I
don't like this approach. Especially as dynamic memory allocations
are not possible at this stage (using RESERVE_BRK() isn't any better
than a static __initdata array IMO).

Which other current factors do you see determining the number of
e820 entries presented to Dom0?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 08:15, Jan Beulich wrote:
>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>> On 15/11/16 01:11, Alex Thorlton wrote:
>>> Hey everyone,
>>>
>>> We're having problems with large systems hitting a BUG in
>>> xen_memory_setup, due to extra e820 entries created in the
>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>> working, but Boris and I wanted to get opinions on whether or not this
>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>> for now.
>>>
>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>> which is a detail worth discussig.  He proposed possibly adding
>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>> zero-page memory limitations.
>>>
>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>> cosmetic prospective, as I believe this macro is intended to describe
>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>> by the Xen HV.  That being said, there isn't exactly a "more
>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>
>>> Any input on the patch, or the questions I've raised above is greatly
>>> appreciated!
>>
>> While I think extending the e820 table is the right thing to do I'm
>> questioning the assumptions here.
>>
>> Looking briefly through the Xen hypervisor sources I think it isn't
>> yet ready for such large machines: the hypervisor's e820 map seems to
>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>> specific path extending this limitation?
> 
> No, you didn't. I do question the correlation with "large machines"
> here though: The issue isn't with large machines afaict, but with
> ones having very many entries (i.e. heavily fragmented).

Fact is: non-EFI machines are limited to 128 entries.

One reason for fragmentation is NUMA - which tends to be present and
especially adding many entries only with lots of nodes - on large
machines.

So while you are of course right that the problem isn't the size of
the machine, but the memory fragmentation, the chances to show up
are much higher on large machines.

So I'd rephrase:

Looking briefly through the Xen hypervisor sources I think it isn't yet
ready for machines with many e820 entries due to memory fragmentation
to be found e.g. on very large NUMA machines with lots of nodes: ...

>> In case I'm right the Xen hypervisor should be prepared for a larger
>> e820 map, but this won't help alone as there would still be additional
>> entries for the IOAPICs created.
>>
>> So I think we need something like:
>>
>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>
>> and use this for sizing xen_e820_map[].
> 
> I would say that if any change gets done here, there shouldn't be
> any static upper limit at all. That could even be viewed as in line
> with recent e820.c changes moving to dynamic allocations. In
> particular I don't see why MAX_IO_APICS would need adding in
> here, but not other (current and future) factors determining the
> (pseudo) E820 map Xen presents to Dom0.

The hypervisor interface of XENMEM_machine_memory_map requires to
specify the size of the e820 map where the hypervisor can supply
entries. The alternative would be try and error: start with a small
e820 map and in case of error increasing the size until success. I
don't like this approach. Especially as dynamic memory allocations
are not possible at this stage (using RESERVE_BRK() isn't any better
than a static __initdata array IMO).

Which other current factors do you see determining the number of
e820 entries presented to Dom0?


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
> On 15/11/16 08:15, Jan Beulich wrote:
>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>> On 15/11/16 01:11, Alex Thorlton wrote:
>>>> Hey everyone,
>>>>
>>>> We're having problems with large systems hitting a BUG in
>>>> xen_memory_setup, due to extra e820 entries created in the
>>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>>> working, but Boris and I wanted to get opinions on whether or not this
>>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>>> for now.
>>>>
>>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>>> which is a detail worth discussig.  He proposed possibly adding
>>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>>> zero-page memory limitations.
>>>>
>>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>>> cosmetic prospective, as I believe this macro is intended to describe
>>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>>> by the Xen HV.  That being said, there isn't exactly a "more
>>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>>
>>>> Any input on the patch, or the questions I've raised above is greatly
>>>> appreciated!
>>>
>>> While I think extending the e820 table is the right thing to do I'm
>>> questioning the assumptions here.
>>>
>>> Looking briefly through the Xen hypervisor sources I think it isn't
>>> yet ready for such large machines: the hypervisor's e820 map seems to
>>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>>> specific path extending this limitation?
>> 
>> No, you didn't. I do question the correlation with "large machines"
>> here though: The issue isn't with large machines afaict, but with
>> ones having very many entries (i.e. heavily fragmented).
> 
> Fact is: non-EFI machines are limited to 128 entries.
> 
> One reason for fragmentation is NUMA - which tends to be present and
> especially adding many entries only with lots of nodes - on large
> machines.

Yes. However, at present Xen only supports up to 64 nodes, and
with one entry per node (assuming there are holes between nodes,
but contiguous memory within them) that's still unlikely to overflow
the table. Yet I'm not meaning to discard the fact that it's been
known for a long time that since Linux needed a larger table, Xen
would need to follow suit sooner or later.

>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>> e820 map, but this won't help alone as there would still be additional
>>> entries for the IOAPICs created.
>>>
>>> So I think we need something like:
>>>
>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>
>>> and use this for sizing xen_e820_map[].
>> 
>> I would say that if any change gets done here, there shouldn't be
>> any static upper limit at all. That could even be viewed as in line
>> with recent e820.c changes moving to dynamic allocations. In
>> particular I don't see why MAX_IO_APICS would need adding in
>> here, but not other (current and future) factors determining the
>> (pseudo) E820 map Xen presents to Dom0.
> 
> The hypervisor interface of XENMEM_machine_memory_map requires to
> specify the size of the e820 map where the hypervisor can supply
> entries. The alternative would be try and error: start with a small
> e820 map and in case of error increasing the size until success. I
> don't like this approach. Especially as dynamic memory allocations
> are not possible at this stage (using RESERVE_BRK() isn't any better
> than a static __initdata array IMO).

Well, I think as a first step we should extend
XENMEM_{,machine_}memory_map to the model used elsewhere
where passing in a NULL handle (and perhaps count being zero) is
a request for the number of needed entries. Granted this doesn't
help with Linux'es way of consuming the output, but it at least
allows for dynamic sizing. And Linux would then be free to prepare
a static array or have a RESERVE_BRK() as it sees fit.

> Which other current factors do you see determining the number of
> e820 entries presented to Dom0?

Just look at the implementation: The dependency is on iomem_caps,
and hence any code removing pages from there could affect the
number of entries. Which means the number of IOMMUs also has an
effect here. And further things could get added at any time, as we
happen to find them.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
> On 15/11/16 08:15, Jan Beulich wrote:
>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>> On 15/11/16 01:11, Alex Thorlton wrote:
>>>> Hey everyone,
>>>>
>>>> We're having problems with large systems hitting a BUG in
>>>> xen_memory_setup, due to extra e820 entries created in the
>>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>>> working, but Boris and I wanted to get opinions on whether or not this
>>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>>> for now.
>>>>
>>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>>> which is a detail worth discussig.  He proposed possibly adding
>>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>>> zero-page memory limitations.
>>>>
>>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>>> cosmetic prospective, as I believe this macro is intended to describe
>>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>>> by the Xen HV.  That being said, there isn't exactly a "more
>>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>>
>>>> Any input on the patch, or the questions I've raised above is greatly
>>>> appreciated!
>>>
>>> While I think extending the e820 table is the right thing to do I'm
>>> questioning the assumptions here.
>>>
>>> Looking briefly through the Xen hypervisor sources I think it isn't
>>> yet ready for such large machines: the hypervisor's e820 map seems to
>>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>>> specific path extending this limitation?
>> 
>> No, you didn't. I do question the correlation with "large machines"
>> here though: The issue isn't with large machines afaict, but with
>> ones having very many entries (i.e. heavily fragmented).
> 
> Fact is: non-EFI machines are limited to 128 entries.
> 
> One reason for fragmentation is NUMA - which tends to be present and
> especially adding many entries only with lots of nodes - on large
> machines.

Yes. However, at present Xen only supports up to 64 nodes, and
with one entry per node (assuming there are holes between nodes,
but contiguous memory within them) that's still unlikely to overflow
the table. Yet I'm not meaning to discard the fact that it's been
known for a long time that since Linux needed a larger table, Xen
would need to follow suit sooner or later.

>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>> e820 map, but this won't help alone as there would still be additional
>>> entries for the IOAPICs created.
>>>
>>> So I think we need something like:
>>>
>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>
>>> and use this for sizing xen_e820_map[].
>> 
>> I would say that if any change gets done here, there shouldn't be
>> any static upper limit at all. That could even be viewed as in line
>> with recent e820.c changes moving to dynamic allocations. In
>> particular I don't see why MAX_IO_APICS would need adding in
>> here, but not other (current and future) factors determining the
>> (pseudo) E820 map Xen presents to Dom0.
> 
> The hypervisor interface of XENMEM_machine_memory_map requires to
> specify the size of the e820 map where the hypervisor can supply
> entries. The alternative would be try and error: start with a small
> e820 map and in case of error increasing the size until success. I
> don't like this approach. Especially as dynamic memory allocations
> are not possible at this stage (using RESERVE_BRK() isn't any better
> than a static __initdata array IMO).

Well, I think as a first step we should extend
XENMEM_{,machine_}memory_map to the model used elsewhere
where passing in a NULL handle (and perhaps count being zero) is
a request for the number of needed entries. Granted this doesn't
help with Linux'es way of consuming the output, but it at least
allows for dynamic sizing. And Linux would then be free to prepare
a static array or have a RESERVE_BRK() as it sees fit.

> Which other current factors do you see determining the number of
> e820 entries presented to Dom0?

Just look at the implementation: The dependency is on iomem_caps,
and hence any code removing pages from there could affect the
number of entries. Which means the number of IOMMUs also has an
effect here. And further things could get added at any time, as we
happen to find them.

Jan

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 09:01, Jan Beulich wrote:
>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>> On 15/11/16 01:11, Alex Thorlton wrote:
>>>>> Hey everyone,
>>>>>
>>>>> We're having problems with large systems hitting a BUG in
>>>>> xen_memory_setup, due to extra e820 entries created in the
>>>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>>>> working, but Boris and I wanted to get opinions on whether or not this
>>>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>>>> for now.
>>>>>
>>>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>>>> which is a detail worth discussig.  He proposed possibly adding
>>>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>>>> zero-page memory limitations.
>>>>>
>>>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>>>> cosmetic prospective, as I believe this macro is intended to describe
>>>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>>>> by the Xen HV.  That being said, there isn't exactly a "more
>>>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>>>
>>>>> Any input on the patch, or the questions I've raised above is greatly
>>>>> appreciated!
>>>>
>>>> While I think extending the e820 table is the right thing to do I'm
>>>> questioning the assumptions here.
>>>>
>>>> Looking briefly through the Xen hypervisor sources I think it isn't
>>>> yet ready for such large machines: the hypervisor's e820 map seems to
>>>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>>>> specific path extending this limitation?
>>>
>>> No, you didn't. I do question the correlation with "large machines"
>>> here though: The issue isn't with large machines afaict, but with
>>> ones having very many entries (i.e. heavily fragmented).
>>
>> Fact is: non-EFI machines are limited to 128 entries.
>>
>> One reason for fragmentation is NUMA - which tends to be present and
>> especially adding many entries only with lots of nodes - on large
>> machines.
> 
> Yes. However, at present Xen only supports up to 64 nodes, and
> with one entry per node (assuming there are holes between nodes,
> but contiguous memory within them) that's still unlikely to overflow
> the table. Yet I'm not meaning to discard the fact that it's been
> known for a long time that since Linux needed a larger table, Xen
> would need to follow suit sooner or later.
> 
>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>> e820 map, but this won't help alone as there would still be additional
>>>> entries for the IOAPICs created.
>>>>
>>>> So I think we need something like:
>>>>
>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>
>>>> and use this for sizing xen_e820_map[].
>>>
>>> I would say that if any change gets done here, there shouldn't be
>>> any static upper limit at all. That could even be viewed as in line
>>> with recent e820.c changes moving to dynamic allocations. In
>>> particular I don't see why MAX_IO_APICS would need adding in
>>> here, but not other (current and future) factors determining the
>>> (pseudo) E820 map Xen presents to Dom0.
>>
>> The hypervisor interface of XENMEM_machine_memory_map requires to
>> specify the size of the e820 map where the hypervisor can supply
>> entries. The alternative would be try and error: start with a small
>> e820 map and in case of error increasing the size until success. I
>> don't like this approach. Especially as dynamic memory allocations
>> are not possible at this stage (using RESERVE_BRK() isn't any better
>> than a static __initdata array IMO).
> 
> Well, I think as a first step we should extend
> XENMEM_{,machine_}memory_map to the model used elsewhere
> where passing in a NULL handle (and perhaps count being zero) is
> a request for the number of needed entries. Granted this doesn't
> help with Linux'es way of consuming the output, but it at least
> allows for dynamic sizing. And Linux would then be free to prepare
> a static array or have a RESERVE_BRK() as it sees fit.

This still needs the definition of an upper limit, as RESERVE_BRK()
is a compile time feature.

For a fully dynamical solution we'd need a way to get a partial
E820 map from the hypervisor (e.g. first 128 entries) in order to
be able to setup at least some memory and later get the rest of
the memory map using some dynamically allocated memory.

>> Which other current factors do you see determining the number of
>> e820 entries presented to Dom0?
> 
> Just look at the implementation: The dependency is on iomem_caps,
> and hence any code removing pages from there could affect the
> number of entries. Which means the number of IOMMUs also has an
> effect here. And further things could get added at any time, as we
> happen to find them.

Its easier to see with that information. :-)


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
> On 15/11/16 09:01, Jan Beulich wrote:
>>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>>> e820 map, but this won't help alone as there would still be additional
>>>>> entries for the IOAPICs created.
>>>>>
>>>>> So I think we need something like:
>>>>>
>>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>>
>>>>> and use this for sizing xen_e820_map[].
>>>>
>>>> I would say that if any change gets done here, there shouldn't be
>>>> any static upper limit at all. That could even be viewed as in line
>>>> with recent e820.c changes moving to dynamic allocations. In
>>>> particular I don't see why MAX_IO_APICS would need adding in
>>>> here, but not other (current and future) factors determining the
>>>> (pseudo) E820 map Xen presents to Dom0.
>>>
>>> The hypervisor interface of XENMEM_machine_memory_map requires to
>>> specify the size of the e820 map where the hypervisor can supply
>>> entries. The alternative would be try and error: start with a small
>>> e820 map and in case of error increasing the size until success. I
>>> don't like this approach. Especially as dynamic memory allocations
>>> are not possible at this stage (using RESERVE_BRK() isn't any better
>>> than a static __initdata array IMO).
>> 
>> Well, I think as a first step we should extend
>> XENMEM_{,machine_}memory_map to the model used elsewhere
>> where passing in a NULL handle (and perhaps count being zero) is
>> a request for the number of needed entries. Granted this doesn't
>> help with Linux'es way of consuming the output, but it at least
>> allows for dynamic sizing. And Linux would then be free to prepare
>> a static array or have a RESERVE_BRK() as it sees fit.
> 
> This still needs the definition of an upper limit, as RESERVE_BRK()
> is a compile time feature.

That's why I did say "as it sees fit".

> For a fully dynamical solution we'd need a way to get a partial
> E820 map from the hypervisor (e.g. first 128 entries) in order to
> be able to setup at least some memory and later get the rest of
> the memory map using some dynamically allocated memory.

And we could of course also make the hypercall allow for that (e.g.
by defining the semantics of a specific error code, so far not used
by it, to avoid mis-interpretation of output on older hypervisors),
or introduce a new clone of the existing one(s).

Jan

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
> On 15/11/16 09:01, Jan Beulich wrote:
>>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>>> e820 map, but this won't help alone as there would still be additional
>>>>> entries for the IOAPICs created.
>>>>>
>>>>> So I think we need something like:
>>>>>
>>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>>
>>>>> and use this for sizing xen_e820_map[].
>>>>
>>>> I would say that if any change gets done here, there shouldn't be
>>>> any static upper limit at all. That could even be viewed as in line
>>>> with recent e820.c changes moving to dynamic allocations. In
>>>> particular I don't see why MAX_IO_APICS would need adding in
>>>> here, but not other (current and future) factors determining the
>>>> (pseudo) E820 map Xen presents to Dom0.
>>>
>>> The hypervisor interface of XENMEM_machine_memory_map requires to
>>> specify the size of the e820 map where the hypervisor can supply
>>> entries. The alternative would be try and error: start with a small
>>> e820 map and in case of error increasing the size until success. I
>>> don't like this approach. Especially as dynamic memory allocations
>>> are not possible at this stage (using RESERVE_BRK() isn't any better
>>> than a static __initdata array IMO).
>> 
>> Well, I think as a first step we should extend
>> XENMEM_{,machine_}memory_map to the model used elsewhere
>> where passing in a NULL handle (and perhaps count being zero) is
>> a request for the number of needed entries. Granted this doesn't
>> help with Linux'es way of consuming the output, but it at least
>> allows for dynamic sizing. And Linux would then be free to prepare
>> a static array or have a RESERVE_BRK() as it sees fit.
> 
> This still needs the definition of an upper limit, as RESERVE_BRK()
> is a compile time feature.

That's why I did say "as it sees fit".

> For a fully dynamical solution we'd need a way to get a partial
> E820 map from the hypervisor (e.g. first 128 entries) in order to
> be able to setup at least some memory and later get the rest of
> the memory map using some dynamically allocated memory.

And we could of course also make the hypercall allow for that (e.g.
by defining the semantics of a specific error code, so far not used
by it, to avoid mis-interpretation of output on older hypervisors),
or introduce a new clone of the existing one(s).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 10:45, Jan Beulich wrote:
>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>> On 15/11/16 09:01, Jan Beulich wrote:
>>>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>>>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>>>> e820 map, but this won't help alone as there would still be additional
>>>>>> entries for the IOAPICs created.
>>>>>>
>>>>>> So I think we need something like:
>>>>>>
>>>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>>>
>>>>>> and use this for sizing xen_e820_map[].
>>>>>
>>>>> I would say that if any change gets done here, there shouldn't be
>>>>> any static upper limit at all. That could even be viewed as in line
>>>>> with recent e820.c changes moving to dynamic allocations. In
>>>>> particular I don't see why MAX_IO_APICS would need adding in
>>>>> here, but not other (current and future) factors determining the
>>>>> (pseudo) E820 map Xen presents to Dom0.
>>>>
>>>> The hypervisor interface of XENMEM_machine_memory_map requires to
>>>> specify the size of the e820 map where the hypervisor can supply
>>>> entries. The alternative would be try and error: start with a small
>>>> e820 map and in case of error increasing the size until success. I
>>>> don't like this approach. Especially as dynamic memory allocations
>>>> are not possible at this stage (using RESERVE_BRK() isn't any better
>>>> than a static __initdata array IMO).
>>>
>>> Well, I think as a first step we should extend
>>> XENMEM_{,machine_}memory_map to the model used elsewhere
>>> where passing in a NULL handle (and perhaps count being zero) is
>>> a request for the number of needed entries. Granted this doesn't
>>> help with Linux'es way of consuming the output, but it at least
>>> allows for dynamic sizing. And Linux would then be free to prepare
>>> a static array or have a RESERVE_BRK() as it sees fit.
>>
>> This still needs the definition of an upper limit, as RESERVE_BRK()
>> is a compile time feature.
> 
> That's why I did say "as it sees fit".
> 
>> For a fully dynamical solution we'd need a way to get a partial
>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>> be able to setup at least some memory and later get the rest of
>> the memory map using some dynamically allocated memory.
> 
> And we could of course also make the hypercall allow for that (e.g.
> by defining the semantics of a specific error code, so far not used
> by it, to avoid mis-interpretation of output on older hypervisors),
> or introduce a new clone of the existing one(s).

I'd go with the new error code. What about E2BIG or ENOSPC?

I think the hypervisor should fill in the number of entries required
in this case.

In case nobody objects I can post patches for this purpose (both Xen
and Linux).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 10:45, Jan Beulich wrote:
>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>> On 15/11/16 09:01, Jan Beulich wrote:
>>>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>>>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>>>> e820 map, but this won't help alone as there would still be additional
>>>>>> entries for the IOAPICs created.
>>>>>>
>>>>>> So I think we need something like:
>>>>>>
>>>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>>>
>>>>>> and use this for sizing xen_e820_map[].
>>>>>
>>>>> I would say that if any change gets done here, there shouldn't be
>>>>> any static upper limit at all. That could even be viewed as in line
>>>>> with recent e820.c changes moving to dynamic allocations. In
>>>>> particular I don't see why MAX_IO_APICS would need adding in
>>>>> here, but not other (current and future) factors determining the
>>>>> (pseudo) E820 map Xen presents to Dom0.
>>>>
>>>> The hypervisor interface of XENMEM_machine_memory_map requires to
>>>> specify the size of the e820 map where the hypervisor can supply
>>>> entries. The alternative would be try and error: start with a small
>>>> e820 map and in case of error increasing the size until success. I
>>>> don't like this approach. Especially as dynamic memory allocations
>>>> are not possible at this stage (using RESERVE_BRK() isn't any better
>>>> than a static __initdata array IMO).
>>>
>>> Well, I think as a first step we should extend
>>> XENMEM_{,machine_}memory_map to the model used elsewhere
>>> where passing in a NULL handle (and perhaps count being zero) is
>>> a request for the number of needed entries. Granted this doesn't
>>> help with Linux'es way of consuming the output, but it at least
>>> allows for dynamic sizing. And Linux would then be free to prepare
>>> a static array or have a RESERVE_BRK() as it sees fit.
>>
>> This still needs the definition of an upper limit, as RESERVE_BRK()
>> is a compile time feature.
> 
> That's why I did say "as it sees fit".
> 
>> For a fully dynamical solution we'd need a way to get a partial
>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>> be able to setup at least some memory and later get the rest of
>> the memory map using some dynamically allocated memory.
> 
> And we could of course also make the hypercall allow for that (e.g.
> by defining the semantics of a specific error code, so far not used
> by it, to avoid mis-interpretation of output on older hypervisors),
> or introduce a new clone of the existing one(s).

I'd go with the new error code. What about E2BIG or ENOSPC?

I think the hypervisor should fill in the number of entries required
in this case.

In case nobody objects I can post patches for this purpose (both Xen
and Linux).


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
> On 15/11/16 10:45, Jan Beulich wrote:
>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>> For a fully dynamical solution we'd need a way to get a partial
>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>> be able to setup at least some memory and later get the rest of
>>> the memory map using some dynamically allocated memory.
>> 
>> And we could of course also make the hypercall allow for that (e.g.
>> by defining the semantics of a specific error code, so far not used
>> by it, to avoid mis-interpretation of output on older hypervisors),
>> or introduce a new clone of the existing one(s).
> 
> I'd go with the new error code. What about E2BIG or ENOSPC?

Either seems fine.

> I think the hypervisor should fill in the number of entries required
> in this case.

And you'd then mean the caller to imply that the passed in (and now
overwritten) count to describe how many entries got filled? That's
not that nice an interface. I'd rather return the number of entries
filled, and require the sizing variant (NULL handle) to be used to
obtain the number of entries. After all if a caller _wants_ to handle
a partial map, it doesn't really care how many further entries there
are.

Jan

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
> On 15/11/16 10:45, Jan Beulich wrote:
>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>> For a fully dynamical solution we'd need a way to get a partial
>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>> be able to setup at least some memory and later get the rest of
>>> the memory map using some dynamically allocated memory.
>> 
>> And we could of course also make the hypercall allow for that (e.g.
>> by defining the semantics of a specific error code, so far not used
>> by it, to avoid mis-interpretation of output on older hypervisors),
>> or introduce a new clone of the existing one(s).
> 
> I'd go with the new error code. What about E2BIG or ENOSPC?

Either seems fine.

> I think the hypervisor should fill in the number of entries required
> in this case.

And you'd then mean the caller to imply that the passed in (and now
overwritten) count to describe how many entries got filled? That's
not that nice an interface. I'd rather return the number of entries
filled, and require the sizing variant (NULL handle) to be used to
obtain the number of entries. After all if a caller _wants_ to handle
a partial map, it doesn't really care how many further entries there
are.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 11:44, Jan Beulich wrote:
>>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
>> On 15/11/16 10:45, Jan Beulich wrote:
>>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>>> For a fully dynamical solution we'd need a way to get a partial
>>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>>> be able to setup at least some memory and later get the rest of
>>>> the memory map using some dynamically allocated memory.
>>>
>>> And we could of course also make the hypercall allow for that (e.g.
>>> by defining the semantics of a specific error code, so far not used
>>> by it, to avoid mis-interpretation of output on older hypervisors),
>>> or introduce a new clone of the existing one(s).
>>
>> I'd go with the new error code. What about E2BIG or ENOSPC?
> 
> Either seems fine.
> 
>> I think the hypervisor should fill in the number of entries required
>> in this case.
> 
> And you'd then mean the caller to imply that the passed in (and now
> overwritten) count to describe how many entries got filled? That's
> not that nice an interface. I'd rather return the number of entries
> filled, and require the sizing variant (NULL handle) to be used to
> obtain the number of entries. After all if a caller _wants_ to handle
> a partial map, it doesn't really care how many further entries there
> are.

I just wanted to avoid two interface changes. IMO the caller knows the
buffer size he supplied and it is rather clear that e.g. E2BIG means
it has been filled completely.

And the assumption the caller doesn't care for the further entries is
not necessarily true: he might not care at the moment of the call, but
he will eventually care for them later when he is able to allocate an
appropriate sized buffer.

OTOH I'm not feeling strong in this case and you are the hypervisor
maintainer. Either solution is fine for me.


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 12:07, <JGross@suse.com> wrote:
> On 15/11/16 11:44, Jan Beulich wrote:
>>>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
>>> On 15/11/16 10:45, Jan Beulich wrote:
>>>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>>>> For a fully dynamical solution we'd need a way to get a partial
>>>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>>>> be able to setup at least some memory and later get the rest of
>>>>> the memory map using some dynamically allocated memory.
>>>>
>>>> And we could of course also make the hypercall allow for that (e.g.
>>>> by defining the semantics of a specific error code, so far not used
>>>> by it, to avoid mis-interpretation of output on older hypervisors),
>>>> or introduce a new clone of the existing one(s).
>>>
>>> I'd go with the new error code. What about E2BIG or ENOSPC?
>> 
>> Either seems fine.
>> 
>>> I think the hypervisor should fill in the number of entries required
>>> in this case.
>> 
>> And you'd then mean the caller to imply that the passed in (and now
>> overwritten) count to describe how many entries got filled? That's
>> not that nice an interface. I'd rather return the number of entries
>> filled, and require the sizing variant (NULL handle) to be used to
>> obtain the number of entries. After all if a caller _wants_ to handle
>> a partial map, it doesn't really care how many further entries there
>> are.
> 
> I just wanted to avoid two interface changes.

Just make one at a time.

Jan

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Jan Beulich]

>>> On 15.11.16 at 12:07, <JGross@suse.com> wrote:
> On 15/11/16 11:44, Jan Beulich wrote:
>>>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
>>> On 15/11/16 10:45, Jan Beulich wrote:
>>>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>>>> For a fully dynamical solution we'd need a way to get a partial
>>>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>>>> be able to setup at least some memory and later get the rest of
>>>>> the memory map using some dynamically allocated memory.
>>>>
>>>> And we could of course also make the hypercall allow for that (e.g.
>>>> by defining the semantics of a specific error code, so far not used
>>>> by it, to avoid mis-interpretation of output on older hypervisors),
>>>> or introduce a new clone of the existing one(s).
>>>
>>> I'd go with the new error code. What about E2BIG or ENOSPC?
>> 
>> Either seems fine.
>> 
>>> I think the hypervisor should fill in the number of entries required
>>> in this case.
>> 
>> And you'd then mean the caller to imply that the passed in (and now
>> overwritten) count to describe how many entries got filled? That's
>> not that nice an interface. I'd rather return the number of entries
>> filled, and require the sizing variant (NULL handle) to be used to
>> obtain the number of entries. After all if a caller _wants_ to handle
>> a partial map, it doesn't really care how many further entries there
>> are.
> 
> I just wanted to avoid two interface changes.

Just make one at a time.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 11:44, Jan Beulich wrote:
>>>> On 15.11.16 at 10:55, <JGross@suse.com> wrote:
>> On 15/11/16 10:45, Jan Beulich wrote:
>>>>>> On 15.11.16 at 09:42, <JGross@suse.com> wrote:
>>>> For a fully dynamical solution we'd need a way to get a partial
>>>> E820 map from the hypervisor (e.g. first 128 entries) in order to
>>>> be able to setup at least some memory and later get the rest of
>>>> the memory map using some dynamically allocated memory.
>>>
>>> And we could of course also make the hypercall allow for that (e.g.
>>> by defining the semantics of a specific error code, so far not used
>>> by it, to avoid mis-interpretation of output on older hypervisors),
>>> or introduce a new clone of the existing one(s).
>>
>> I'd go with the new error code. What about E2BIG or ENOSPC?
> 
> Either seems fine.
> 
>> I think the hypervisor should fill in the number of entries required
>> in this case.
> 
> And you'd then mean the caller to imply that the passed in (and now
> overwritten) count to describe how many entries got filled? That's
> not that nice an interface. I'd rather return the number of entries
> filled, and require the sizing variant (NULL handle) to be used to
> obtain the number of entries. After all if a caller _wants_ to handle
> a partial map, it doesn't really care how many further entries there
> are.

I just wanted to avoid two interface changes. IMO the caller knows the
buffer size he supplied and it is rather clear that e.g. E2BIG means
it has been filled completely.

And the assumption the caller doesn't care for the further entries is
not necessarily true: he might not care at the moment of the call, but
he will eventually care for them later when he is able to allocate an
appropriate sized buffer.

OTOH I'm not feeling strong in this case and you are the hypervisor
maintainer. Either solution is fine for me.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On Tue, Nov 15, 2016 at 10:55:49AM +0100, Juergen Gross wrote:
> I'd go with the new error code. What about E2BIG or ENOSPC?
> 
> I think the hypervisor should fill in the number of entries required
> in this case.
> 
> In case nobody objects I can post patches for this purpose (both Xen
> and Linux).

This sounds like a good solution to me.  I think it's definitely more
appropriate than simply bumping up the size of xen_e820_map, especially
considering the fact that it's theoretically possible for the e820 map
generated by the hypercall to grow too large, even on a non-EFI machine,
where my change would have no effect.

Thanks for your input!

- Alex

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 16:22, Alex Thorlton wrote:
> On Tue, Nov 15, 2016 at 10:55:49AM +0100, Juergen Gross wrote:
>> I'd go with the new error code. What about E2BIG or ENOSPC?
>>
>> I think the hypervisor should fill in the number of entries required
>> in this case.
>>
>> In case nobody objects I can post patches for this purpose (both Xen
>> and Linux).
> 
> This sounds like a good solution to me.  I think it's definitely more
> appropriate than simply bumping up the size of xen_e820_map, especially
> considering the fact that it's theoretically possible for the e820 map
> generated by the hypercall to grow too large, even on a non-EFI machine,
> where my change would have no effect.

Well, it won't help with the current hypervisor, so bumping up the size
of xen_e820_map will still be a good idea. I think using E820_X_MAX is
okay since in the end xen_e820_map will be transferred into a struct
e820map which can't hold more than E820_X_MAX entries (additional
entries are ignored here, so this won't let the boot fail).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 16:22, Alex Thorlton wrote:
> On Tue, Nov 15, 2016 at 10:55:49AM +0100, Juergen Gross wrote:
>> I'd go with the new error code. What about E2BIG or ENOSPC?
>>
>> I think the hypervisor should fill in the number of entries required
>> in this case.
>>
>> In case nobody objects I can post patches for this purpose (both Xen
>> and Linux).
> 
> This sounds like a good solution to me.  I think it's definitely more
> appropriate than simply bumping up the size of xen_e820_map, especially
> considering the fact that it's theoretically possible for the e820 map
> generated by the hypercall to grow too large, even on a non-EFI machine,
> where my change would have no effect.

Well, it won't help with the current hypervisor, so bumping up the size
of xen_e820_map will still be a good idea. I think using E820_X_MAX is
okay since in the end xen_e820_map will be transferred into a struct
e820map which can't hold more than E820_X_MAX entries (additional
entries are ignored here, so this won't let the boot fail).


Juergen

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On Tue, Nov 15, 2016 at 10:55:49AM +0100, Juergen Gross wrote:
> I'd go with the new error code. What about E2BIG or ENOSPC?
> 
> I think the hypervisor should fill in the number of entries required
> in this case.
> 
> In case nobody objects I can post patches for this purpose (both Xen
> and Linux).

This sounds like a good solution to me.  I think it's definitely more
appropriate than simply bumping up the size of xen_e820_map, especially
considering the fact that it's theoretically possible for the e820 map
generated by the hypercall to grow too large, even on a non-EFI machine,
where my change would have no effect.

Thanks for your input!

- Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 09:01, Jan Beulich wrote:
>>>> On 15.11.16 at 08:36, <JGross@suse.com> wrote:
>> On 15/11/16 08:15, Jan Beulich wrote:
>>>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>>>> On 15/11/16 01:11, Alex Thorlton wrote:
>>>>> Hey everyone,
>>>>>
>>>>> We're having problems with large systems hitting a BUG in
>>>>> xen_memory_setup, due to extra e820 entries created in the
>>>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>>>> working, but Boris and I wanted to get opinions on whether or not this
>>>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>>>> for now.
>>>>>
>>>>> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
>>>>> which is a detail worth discussig.  He proposed possibly adding
>>>>> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
>>>>> value than E820MAX, since the Xen e820 table isn't bound by the
>>>>> zero-page memory limitations.
>>>>>
>>>>> I do *slightly* question the use of E820_X_MAX here, only from a
>>>>> cosmetic prospective, as I believe this macro is intended to describe
>>>>> the maximum size of the extended e820 table, which, AFAIK, is not used
>>>>> by the Xen HV.  That being said, there isn't exactly a "more
>>>>> appropriate" macro/variable to use, so this may not really be an issue.
>>>>>
>>>>> Any input on the patch, or the questions I've raised above is greatly
>>>>> appreciated!
>>>>
>>>> While I think extending the e820 table is the right thing to do I'm
>>>> questioning the assumptions here.
>>>>
>>>> Looking briefly through the Xen hypervisor sources I think it isn't
>>>> yet ready for such large machines: the hypervisor's e820 map seems to
>>>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>>>> specific path extending this limitation?
>>>
>>> No, you didn't. I do question the correlation with "large machines"
>>> here though: The issue isn't with large machines afaict, but with
>>> ones having very many entries (i.e. heavily fragmented).
>>
>> Fact is: non-EFI machines are limited to 128 entries.
>>
>> One reason for fragmentation is NUMA - which tends to be present and
>> especially adding many entries only with lots of nodes - on large
>> machines.
> 
> Yes. However, at present Xen only supports up to 64 nodes, and
> with one entry per node (assuming there are holes between nodes,
> but contiguous memory within them) that's still unlikely to overflow
> the table. Yet I'm not meaning to discard the fact that it's been
> known for a long time that since Linux needed a larger table, Xen
> would need to follow suit sooner or later.
> 
>>>> In case I'm right the Xen hypervisor should be prepared for a larger
>>>> e820 map, but this won't help alone as there would still be additional
>>>> entries for the IOAPICs created.
>>>>
>>>> So I think we need something like:
>>>>
>>>> #define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)
>>>>
>>>> and use this for sizing xen_e820_map[].
>>>
>>> I would say that if any change gets done here, there shouldn't be
>>> any static upper limit at all. That could even be viewed as in line
>>> with recent e820.c changes moving to dynamic allocations. In
>>> particular I don't see why MAX_IO_APICS would need adding in
>>> here, but not other (current and future) factors determining the
>>> (pseudo) E820 map Xen presents to Dom0.
>>
>> The hypervisor interface of XENMEM_machine_memory_map requires to
>> specify the size of the e820 map where the hypervisor can supply
>> entries. The alternative would be try and error: start with a small
>> e820 map and in case of error increasing the size until success. I
>> don't like this approach. Especially as dynamic memory allocations
>> are not possible at this stage (using RESERVE_BRK() isn't any better
>> than a static __initdata array IMO).
> 
> Well, I think as a first step we should extend
> XENMEM_{,machine_}memory_map to the model used elsewhere
> where passing in a NULL handle (and perhaps count being zero) is
> a request for the number of needed entries. Granted this doesn't
> help with Linux'es way of consuming the output, but it at least
> allows for dynamic sizing. And Linux would then be free to prepare
> a static array or have a RESERVE_BRK() as it sees fit.

This still needs the definition of an upper limit, as RESERVE_BRK()
is a compile time feature.

For a fully dynamical solution we'd need a way to get a partial
E820 map from the hypervisor (e.g. first 128 entries) in order to
be able to setup at least some memory and later get the rest of
the memory map using some dynamically allocated memory.

>> Which other current factors do you see determining the number of
>> e820 entries presented to Dom0?
> 
> Just look at the implementation: The dependency is on iomem_caps,
> and hence any code removing pages from there could affect the
> number of entries. Which means the number of IOMMUs also has an
> effect here. And further things could get added at any time, as we
> happen to find them.

Its easier to see with that information. :-)


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 08:15, Jan Beulich wrote:
>>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
>> On 15/11/16 01:11, Alex Thorlton wrote:
>>> Hey everyone,
>>>
>>> We're having problems with large systems hitting a BUG in
>>> xen_memory_setup, due to extra e820 entries created in the
>>> XENMEM_machine_memory_map callback.  The change in the patch gets things
>>> working, but Boris and I wanted to get opinions on whether or not this
>>> is the appropriate/entire solution, which is why I've sent it as an RFC
>>> for now.

>> While I think extending the e820 table is the right thing to do I'm
>> questioning the assumptions here.
>>
>> Looking briefly through the Xen hypervisor sources I think it isn't
>> yet ready for such large machines: the hypervisor's e820 map seems to
>> be still limited to 128 e820 entries. Jan, did I overlook an EFI
>> specific path extending this limitation?
> 
> No, you didn't. I do question the correlation with "large machines"
> here though: The issue isn't with large machines afaict, but with
> ones having very many entries (i.e. heavily fragmented).

Alex, I would appreciate if you could send me the E820 map printed
at a bare metal Linux boot. I suspect it is already larger than
128 entries and the hypervisor is just cutting it off at the end.


Juergen

Removed all but Xen guys from CC list

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On Wed, Nov 16, 2016 at 05:42:11PM +0100, Juergen Gross wrote:
> On 15/11/16 08:15, Jan Beulich wrote:
> >>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
> >> On 15/11/16 01:11, Alex Thorlton wrote:
> >>> Hey everyone,
> >>>
> >>> We're having problems with large systems hitting a BUG in
> >>> xen_memory_setup, due to extra e820 entries created in the
> >>> XENMEM_machine_memory_map callback.  The change in the patch gets things
> >>> working, but Boris and I wanted to get opinions on whether or not this
> >>> is the appropriate/entire solution, which is why I've sent it as an RFC
> >>> for now.
> 
> >> While I think extending the e820 table is the right thing to do I'm
> >> questioning the assumptions here.
> >>
> >> Looking briefly through the Xen hypervisor sources I think it isn't
> >> yet ready for such large machines: the hypervisor's e820 map seems to
> >> be still limited to 128 e820 entries. Jan, did I overlook an EFI
> >> specific path extending this limitation?
> > 
> > No, you didn't. I do question the correlation with "large machines"
> > here though: The issue isn't with large machines afaict, but with
> > ones having very many entries (i.e. heavily fragmented).
> 
> Alex, I would appreciate if you could send me the E820 map printed
> at a bare metal Linux boot. I suspect it is already larger than
> 128 entries and the hypervisor is just cutting it off at the end.

No problem!  I'll get this to you today.

- Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

On Wed, Nov 16, 2016 at 10:53:36AM -0600, Alex Thorlton wrote:
> On Wed, Nov 16, 2016 at 05:42:11PM +0100, Juergen Gross wrote:
> > On 15/11/16 08:15, Jan Beulich wrote:
> > >>>> On 15.11.16 at 07:33, <JGross@suse.com> wrote:
> > >> On 15/11/16 01:11, Alex Thorlton wrote:
> > >>> Hey everyone,
> > >>>
> > >>> We're having problems with large systems hitting a BUG in
> > >>> xen_memory_setup, due to extra e820 entries created in the
> > >>> XENMEM_machine_memory_map callback.  The change in the patch gets things
> > >>> working, but Boris and I wanted to get opinions on whether or not this
> > >>> is the appropriate/entire solution, which is why I've sent it as an RFC
> > >>> for now.
> > 
> > >> While I think extending the e820 table is the right thing to do I'm
> > >> questioning the assumptions here.
> > >>
> > >> Looking briefly through the Xen hypervisor sources I think it isn't
> > >> yet ready for such large machines: the hypervisor's e820 map seems to
> > >> be still limited to 128 e820 entries. Jan, did I overlook an EFI
> > >> specific path extending this limitation?
> > > 
> > > No, you didn't. I do question the correlation with "large machines"
> > > here though: The issue isn't with large machines afaict, but with
> > > ones having very many entries (i.e. heavily fragmented).
> > 
> > Alex, I would appreciate if you could send me the E820 map printed
> > at a bare metal Linux boot. I suspect it is already larger than
> > 128 entries and the hypervisor is just cutting it off at the end.
> 
> No problem!  I'll get this to you today.

I've attached a boot log from a 4.9-rc5 kernel.  Sorry this took me a
little longer than I expected!

AFAICT, the kernel running bare metal reports the same 115 e820
entries as the HV was reporting on previous boots.  I believe that our
failure was due to 14 more entries being added during
XENMEM_machine_memory_map.  I have some debug output that shows this, if
you'd like to see it.

- Alex

[-- Attachment #2: mfg.32skt.baremetal.community --]
[-- Type: text/plain, Size: 382675 bytes --]

Script started on Thu Nov 17 17:14:43 2016
    0.310430 (    0.310430)| athorlton@stormcage:~/notes/11-16/ovm-logs$ fg\b\btmux attach-session -d\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bfg^[[K\b\b^[[K\a\a\a\afg\b\btmux attach-session -d\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bssh root@137.38.153.224\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bcd /home-local/\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bvim mfg.32skt.e820kern.2\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[9Pcd /home-local/\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bssh root@137.38.153.224\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1Ptmux attach-session -d\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bssh root@137.38.153.224\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bssh root@137.38.153.224\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1Ptmux attach-session -d\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bssh root@137.38.153.224
   11.718763 (   11.408333)| Warning: Permanently added '137.38.153.224' (ECDSA) to the list of known hosts.
   11.851892 (    0.133129)| root@137.38.153.224's password: 
   13.606361 (    1.754469)| 
   13.638750 (    0.032389)| SGI UV300 RMC, Rev. 1.1.87   [Bootloader  1.1.4]
   13.638792 (    0.000042)| Copyright 2012-2016, SGI.   All rights reserved.
   13.638827 (    0.000035)| 
   14.057541 (    0.418714)| uv300-or-rmc RMC:r001i21c> uvcon
   14.803484 (    0.745943)| uvcon: tty mode disabled, use 'CTRL-C' to exit
   14.803714 (    0.000230)| uvcon: attempting connection to localhost...
   14.818205 (    0.014491)| uvcon: connection to SMN/CMC (localhost) established.
   14.818246 (    0.000041)| uvcon: requesting baseio console access at r001i01b...
   14.818281 (    0.000035)| uvcon: console access denied - console already has an owner, retrying spy mode
   14.818330 (    0.000049)| uvcon: requesting baseio console access (spy) at r001i01b...
   14.823590 (    0.005260)| uvcon: console access established (SPY)
   14.834717 (    0.011127)| uvcon: CMC <--> BASEIO connection active
   14.834750 (    0.000033)| ************************************************
   14.834782 (    0.000032)| *******  START OF CACHED CONSOLE OUTPUT  *******
   14.834813 (    0.000031)| ************************************************
   14.843814 (    0.009001)| ************************************************
   14.843998 (    0.000184)| *******   END OF CACHED CONSOLE OUTPUT   *******
   14.849129 (    0.005131)| ************************************************
   14.849162 (    0.000033)| uvcon: escape codes:                              
   14.849193 (    0.000031)| uvcon:    ctrl-] s    steal console               
   14.849225 (    0.000032)| uvcon:    ctrl-] r    release console             
   14.849257 (    0.000032)| uvcon:    ctrl-] b    send break                  
   14.849289 (    0.000032)| uvcon:    ctrl-] c    connection status           
   14.849321 (    0.000032)| uvcon:    ctrl-] q    quit (if active console)    
   14.849353 (    0.000032)| uvcon:    ctrl-] ?|h  this help                   
  654.747952 (  639.898599)| ^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[0m^[[35m^[[40m^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[0m^[[37m^[[40m
  672.681832 (   17.933880)| BIOS Revision: SGI UV 300 series BIOS
  672.689803 (    0.007971)| Total Memory: 2031616 MB
  672.698083 (    0.008280)| Memory Mode: Independent
  672.713040 (    0.014957)| Microcode Revision: 0B00001D
  673.312500 (    0.599460)| ^[[1m^[[33m^[[40m^[[06;01HPress ESC key to skip memory test^[[01;01H1% of the system memory tested OK^[[01;01H2% of the system memory tested OK^[[01;01H3% of the system memory tested OK^[[01;01H4% of the system memory tested OK^[[01;01H5% of the system memory tested OK^[[01;01H6% of the system memory tested OK^[[01;01H7% of the system memory tested OK^[[01;01H8% of the system memory tested OK^[[01;01H9% of the system memory tested OK^[[01;01H10% of the system memory tested OK^[[01;01H11% of the system memory tested OK^[[01;01H12% of the system memory tested OK^[[01;01H13% of the system memory tested OK^[[01;01H14% of the system memory tested OK^[[01;01H15% of the system memory tested OK^[[01;01H16% of the system memory tested OK^[[01;01H17% of the system memory tested OK^[[01;01H18% of the system memory tested OK^[[01;01H19% of the system memory tested OK^[[01;01H20% of the system memory tested OK^[[01;01H21% of the system memory tested OK^[[01;01H22% of the system memory tested OK^[[01;01H23% of the system memory tested OK^[[01;01H24% of the system memory tested OK^[[01;01H25% of the system memory tested OK^[[01;01H26% of the system memory tested OK^[[01;01H27% of the system memory tested OK^[[01;01H28% of the system memory tested OK^[[01;01H29% of the system memory tested OK^[[01;01H30% of the system memory tested OK^[[01;01H31% of the system memory tested OK^[[01;01H32% of the system memory tested OK^[[01;01H33% of the system memory tested OK^[[01;01H34% of the system memory tested OK^[[01;01H35% of the system memory tested OK^[[01;01H36% of the system memory tested OK^[[01;01H37% of the system memory tested OK^[[01;01H38% of the system memory tested OK^[[01;01H39% of the system memory tested OK^[[01;01H40% of the system memory tested OK^[[01;01H41% of the system memory tested OK^[[01;01H42% of the system memory tested OK^[[01;01H43% of the system memory tested OK^[[01;01H44% of the system memory tested OK^[[01;01H45% of the system memory tested OK^[[01;01H46% of the system memory tested OK^[[01;01H47% of the system memory tested OK^[[01;01H48% of the system memory tested OK^[[01;01H49% of the system memory tested OK^[[01;01H50% of the system memory tested OK^[[01;01H51% of the system memory tested OK^[[01;01H52% of the system memory tested OK^[[01;01H53% of the system memory tested OK^[[01;01H54% of the system memory tested OK^[[01;01H55% of the system memory tested OK^[[01;01H56% of the system memory tested OK^[[01;01H57% of the system memory tested OK^[[01;01H58% of the system memory tested OK^[[01;01H59% of the system memory tested OK^[[01;01H60% of the system memory tested OK^[[01;01H61% of the system memory tested OK^[[01;01H62% of the system memory tested OK^[[01;01H63% of the system memory tested OK^[[01;01H64% of the system memory tested OK^[[01;01H65% of the system memory tested OK^[[01;01H66% of the system memory tested OK^[[01;01H67% of the system memory tested OK^[[01;01H68% of the system memory tested OK^[[01;01H69% of the system memory tested OK^[[01;01H70% of the system memory tested OK^[[01;01H71% of the system memory tested OK^[[01;01H72% of the system memory tested OK^[[01;01H73% of the system memory tested OK^[[01;01H74% of the system memory tested OK^[[01;01H75% of the system memory tested OK^[[01;01H76% of the system memory tested OK^[[01;01H77% of the system memory tested OK^[[01;01H78% of the system memory tested OK^[[01;01H79% of the system memory tested OK^[[01;01H80% of the system memory tested OK^[[01;01H81% of the system memory tested OK^[[01;01H82% of the system memory tested OK^[[01;01H83% of the system memory tested OK^[[01;01H84% of the system memory tested OK^[[01;01H85% of the system memory tested OK^[[01;01H86% of the system memory tested OK^[[01;01H87% of the system memory tested OK^[[01;01H88% of the system memory tested OK^[[01;01H89% of the system memory tested OK^[[01;01H90% of the system memory tested OK^[[01;01H91% of the system memory tested OK^[[01;01H92% of the system memory tested OK^[[01;01H93% of the system memory tested OK^[[01;01H94% of the system memory tested OK^[[01;01H95% of the system memory tested OK^[[01;01H96% of the system memory tested OK^[[01;01H97% of the system memory tested OK^[[01;01H98% of the system memory tested OK^[[01;01H99% of the system memory tested OK^[[01;01H100% of the system memory tested OK^[[01;01H17,041,892,966,400 bytes of system memory tested OK
  692.476137 (   19.163637)| ^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H  ============================================================================
  692.498379 (    0.022242)|                              SGI UV 300 series BIOS
  692.506976 (    0.008597)|   ============================================================================
  692.518447 (    0.011471)| 
  692.518455 (    0.000008)| SGI BIOS 5.2.5 built in rel-5.2.5-2016-08-09_13.43.57 by biosdev on 2016-08-09 at 18:44:29 UTC
  692.538526 (    0.020071)| 
  692.538537 (    0.000011)| 
  692.538541 (    0.000004)| 
  692.538545 (    0.000004)|      Press [Enter] to directly boot.
  692.542938 (    0.004393)|      Press [F2]    to enter setup and select boot options.
  692.546884 (    0.003946)|      Press [F7]    to show boot menu options.
  692.550949 (    0.004065)| 
  692.550953 (    0.000004)|      Copyright (c) 2010-2015, SGI. All rights reserved.
  692.558418 (    0.007465)|      Copyright (c) 2006-2015, Intel Corporation.
  769.474934 (   76.916516)| ^[[0m^[[37m^[[40m^[[1m^[[33m^[[40mEFI Shell version 2.40 [1.0]
  769.907609 (    0.432675)| ^[[0m^[[37m^[[40mCurrent running mode ^[[1m^[[37m^[[40m1.1.2^[[0m^[[37m^[[40m
  770.321985 (    0.414376)| ^[[1m^[[33m^[[40mDevice mapping table^[[0m^[[37m^[[40m
  770.333661 (    0.011676)|   ^[[1m^[[33m^[[40mfs0  ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40mhd61b32768a1 blk0 ^[[0m^[[37m^[[40m
  770.356870 (    0.023209)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)/HD(1,GPT,8227E210-2088-4E3C-860A-88D9702DBEBA,0x800,0xFA800)
  770.407320 (    0.050450)|   ^[[1m^[[33m^[[40mfs1  ^[[0m^[[37m^[[40m :Removable CDRom - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40mcd41a0c0b0b blk1 ^[[0m^[[37m^[[40m
  770.498455 (    0.091135)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1A,0x0)/USB(0x0,0x0)/USB(0x2,0x0)/USB(0x1,0x0)/CDROM(0x1,0x285,0x18E50C)
  770.498529 (    0.000074)|   ^[[1m^[[33m^[[40mblk0 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40mhd61b32768a1 fs0 ^[[0m^[[37m^[[40m
  770.498603 (    0.000074)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)/HD(1,GPT,8227E210-2088-4E3C-860A-88D9702DBEBA,0x800,0xFA800)
  770.578434 (    0.079831)|   ^[[1m^[[33m^[[40mblk1 ^[[0m^[[37m^[[40m :Removable CDRom - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40mcd41a0c0b0b fs1 ^[[0m^[[37m^[[40m
  770.578513 (    0.000079)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1A,0x0)/USB(0x0,0x0)/USB(0x2,0x0)/USB(0x1,0x0)/CDROM(0x1,0x285,0x18E50C)
  770.600071 (    0.021558)|   ^[[1m^[[33m^[[40mblk2 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.608914 (    0.008843)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)/HD(2,GPT,DC1A3C25-D613-4D63-9B30-96B5560A2576,0xFB000,0x20001800)
  770.653747 (    0.044833)|   ^[[1m^[[33m^[[40mblk3 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.677630 (    0.023883)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)/HD(3,GPT,B50D255B-17CB-4840-80D0-DB3125C9872D,0x200FC800,0xFFE800)
  770.726447 (    0.048817)|   ^[[1m^[[33m^[[40mblk4 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.759350 (    0.032903)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)/HD(4,GPT,1C400025-6E69-4ACC-A296-40F2C2075D6A,0x210FB000,0x3C171800)
  770.807761 (    0.048411)|   ^[[1m^[[33m^[[40mblk5 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.833673 (    0.025912)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)/HD(1,GPT,550C269F-5175-4C78-A612-B8E361C52F91,0x1E8800,0x104C5000)
  770.883458 (    0.049785)|   ^[[1m^[[33m^[[40mblk6 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.899804 (    0.016346)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)/HD(2,GPT,4477F90E-3C85-4660-924F-1CF730A30A13,0x800,0x178000)
  770.951016 (    0.051212)|   ^[[1m^[[33m^[[40mblk7 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  770.969176 (    0.018160)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)/HD(3,GPT,FB829883-5A6A-4410-BB39-1B0C286E29F2,0x106AD800,0x407000)
  771.018496 (    0.049320)|   ^[[1m^[[33m^[[40mblk8 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.047880 (    0.029384)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)/HD(4,GPT,12035CE6-D5B2-4F84-8E6E-ED9340F7885C,0x10AB4800,0x5001000)
  771.091759 (    0.043879)|   ^[[1m^[[33m^[[40mblk9 ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.150394 (    0.058635)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)/HD(5,GPT,E280DEEE-E465-4F3D-B000-E5CAFCFE82A3,0x15AB5800,0x477B7000)
  771.167246 (    0.016852)|   ^[[1m^[[33m^[[40mblkA ^[[0m^[[37m^[[40m :HardDisk - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.203050 (    0.035804)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x4,0x8000,0x0)/HD(1,GPT,1D91FB2D-4126-443A-9B94-83E7337B0006,0x800,0x74705DB0)
  771.252508 (    0.049458)|   ^[[1m^[[33m^[[40mblkB ^[[0m^[[37m^[[40m :BlockDevice - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.273602 (    0.021094)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0x8000,0x0)
  771.292694 (    0.019092)|   ^[[1m^[[33m^[[40mblkC ^[[0m^[[37m^[[40m :BlockDevice - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.314382 (    0.021688)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0x8000,0x0)
  771.321009 (    0.006627)|   ^[[1m^[[33m^[[40mblkD ^[[0m^[[37m^[[40m :BlockDevice - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.353818 (    0.032809)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x4,0x8000,0x0)
  771.374178 (    0.020360)|   ^[[1m^[[33m^[[40mblkE ^[[0m^[[37m^[[40m :Removable CDRom - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.403841 (    0.029663)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1A,0x0)/USB(0x0,0x0)/USB(0x2,0x0)/USB(0x1,0x0)/CDROM(0x0,0x150C,0x4)
  771.445270 (    0.041429)|   ^[[1m^[[33m^[[40mblkF ^[[0m^[[37m^[[40m :Removable BlockDevice - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.481717 (    0.036447)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1A,0x0)/USB(0x0,0x0)/USB(0x2,0x0)/USB(0x1,0x0)
  771.504703 (    0.022986)|   ^[[1m^[[33m^[[40mblk10^[[0m^[[37m^[[40m :Removable BlockDevice - ^[[1m^[[37m^[[40mAlias^[[0m^[[37m^[[40m ^[[1m^[[33m^[[40m(null)^[[0m^[[37m^[[40m
  771.537425 (    0.032722)|   ^[[1m^[[33m^[[40m     ^[[0m^[[37m^[[40m  PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x0,0x0)
  771.561328 (    0.023903)| 
  771.569300 (    0.007972)| Press ^[[1m^[[37m^[[40mESC^[[0m^[[37m^[[40m in 5 seconds to skip ^[[1m^[[33m^[[40mstartup.nsh^[[0m^[[37m^[[40m, any other key to continue. Press ^[[1m^[[37m^[[40mESC^[[0m^[[37m^[[40m in 4 seconds to skip ^[[1m^[[33m^[[40mstartup.nsh^[[0m^[[37m^[[40m, any other key to continue. Press ^[[1m^[[37m^[[40mESC^[[0m^[[37m^[[40m in 3 seconds to skip ^[[1m^[[33m^[[40mstartup.nsh^[[0m^[[37m^[[40m, any other key to continue. Press ^[[1m^[[37m^[[40mESC^[[0m^[[37m^[[40m in 2 seconds to skip ^[[1m^[[33m^[[40mstartup.nsh^[[0m^[[37m^[[40m, any other key to continue. Press ^[[1m^[[37m^[[40mESC^[[0m^[[37m^[[40m in 1 seconds to skip ^[[1m^[[33m^[[40mstartup.nsh^[[0m^[[37m^[[40m, any other key to continue. 
  777.892579 (    6.323279)| ^[[25;01H^[[1m^[[33m^[[40mShell> fs0:^[[25;12H^[[0m^[[37m^[[40m
  949.113935 (  171.221356)| 
  949.125966 (    0.012031)| ^[[25;01H^[[1m^[[33m^[[40mfs0:\> e^[[25;08HEFI\boot\^[[25;16H ^[[25;16H^[[25;15H ^[[25;15H^[[25;14H ^[[25;14H^[[25;13H ^[[25;13H^[[25;12H ^[[25;12Hboot^[[25;12Hsles\shim.efi^[[25;17HMokManager.efi^[[25;17Hgrub.efi      ^[[25;25H^[[25;17Hboot.csv^[[25;17Hgrub.cfg^[[25;17Hgrubx64.efi^[[25;17Hshim.efi   ^[[25;25H^[[25;17HMokManager.efi^[[25;17Hgrub.efi      ^[[25;25H^[[25;17Hboot.csv^[[25;17Hgrub.cfg^[[25;17Hgrubx64.efi^[[25;28H^[[0m^[[37m^[[40m
  961.095921 (   11.969955)| ^[[0m^[[30m^[[47mWelcome to GRUB!
  961.117963 (    0.022042)| 
  961.126005 (    0.008042)| ^[[0m^[[37m^[[40m^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[02;27HGNU GRUB  version 2.02~beta2
  961.166910 (    0.040905)| 
  961.166923 (    0.000013)| ^[[04;02H/----------------------------------------------------------------------------\^[[05;02H|^[[05;79H|^[[06;02H|^[[06;79H|^[[07;02H|^[[07;79H|^[[08;02H|^[[08;79H|^[[09;02H|^[[09;79H|^[[10;02H|^[[10;79H|^[[11;02H|^[[11;79H|^[[12;02H|^[[12;79H|^[[13;02H|^[[13;79H|^[[14;02H|^[[14;79H|^[[15;02H|^[[15;79H|^[[16;02H|^[[16;79H|^[[17;02H|^[[17;79H|^[[18;02H\----------------------------------------------------------------------------/^[[19;02H^[[20;02H     Use the ^ and v keys to select which entry is highlighted.          
  961.222409 (    0.055486)|       Press enter to boot the selected OS, `e' to edit the commands       
  961.234609 (    0.012200)|       before booting or `c' for a command-line.                           ^[[05;80H ^[[0m^[[30m^[[47m^[[05;03H*SLES 12-SP2                                                                ^[[0m^[[37m^[[40m^[[05;78H^[[06;03H Advanced options for SLES 12-SP2                                           ^[[06;78H^[[07;03H                                                                            ^[[07;78H^[[08;03H                                                                            ^[[08;78H^[[09;03H                                                                            ^[[09;78H^[[10;03H                                                                            ^[[10;78H^[[11;03H                                                                            ^[[11;78H^[[12;03H                                                                            ^[[12;78H^[[13;03H                                                                            ^[[13;78H^[[14;03H                                                                            ^[[14;78H^[[15;03H                                                                            ^[[15;78H^[[16;03H                                                                            ^[[16;78H^[[17;03H                                                                            ^[[17;78H^[[17;80H ^[[05;78H^[[23;01H   The highlighted entry will be executed automatically in 8s.                 ^[[05;78H^[[23;01H                                                                               ^[[24;01H                                                                               ^[[05;78H^[[05;03H SLES 12-SP2                                                                ^[[05;78H^[[0m^[[30m^[[47m^[[06;03H*Advanced options for SLES 12-SP2                                           ^[[0m^[[37m^[[40m^[[06;78H^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[02;27HGNU GRUB  version 2.02~beta2
  963.317571 (    2.082962)| 
  963.317596 (    0.000025)| ^[[04;02H/----------------------------------------------------------------------------\^[[05;02H|^[[05;79H|^[[06;02H|^[[06;79H|^[[07;02H|^[[07;79H|^[[08;02H|^[[08;79H|^[[09;02H|^[[09;79H|^[[10;02H|^[[10;79H|^[[11;02H|^[[11;79H|^[[12;02H|^[[12;79H|^[[13;02H|^[[13;79H|^[[14;02H|^[[14;79H|^[[15;02H|^[[15;79H|^[[16;02H|^[[16;79H|^[[17;02H\----------------------------------------------------------------------------/^[[18;02H^[[19;02H     Use the ^ and v keys to select which entry is highlighted.          
  963.373522 (    0.055926)|       Press enter to boot the selected OS, `e' to edit the commands       
  963.386322 (    0.012800)|       before booting or `c' for a command-line. ESC to return previous    
  963.394448 (    0.008126)|       menu.                                                               ^[[05;80H ^[[0m^[[30m^[[47m^[[05;03H*SLES 12-SP2, with Linux 4.9.0-rc5-clean+                                   ^[[0m^[[37m^[[40m^[[05;78H^[[06;03H SLES 12-SP2, with Linux 4.9.0-rc5-clean+ (recovery mode)                   ^[[06;78H^[[07;03H SLES 12-SP2, with Linux 4.4.21-69-default                                  ^[[07;78H^[[08;03H SLES 12-SP2, with Linux 4.4.21-69-default (recovery mode)                  ^[[08;78H^[[09;03H                                                                            ^[[09;78H^[[10;03H                                                                            ^[[10;78H^[[11;03H                                                                            ^[[11;78H^[[12;03H                                                                            ^[[12;78H^[[13;03H                                                                            ^[[13;78H^[[14;03H                                                                            ^[[14;78H^[[15;03H                                                                            ^[[15;78H^[[16;03H                                                                            ^[[16;78H^[[16;80H ^[[05;78H^[[23;01H                                                                               ^[[24;01H                                                                               ^[[05;78H^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40mLoading Linux 4.9.0-rc5-clean+ ...
  966.061486 (    2.667038)| Loading initial ramdisk ...
  966.120293 (    0.058807)| [    0.000000] Linux version 4.9.0-rc5-clean+ (athorlton@stormcage) (gcc version 4.8.5 (SUSE Linux) ) #104 SMP Tue Nov 15 10:58:05 CST 2016
  966.416369 (    0.296076)| [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-rc5-clean+ root=UUID=5916572d-2a57-447a-bf73-63cdf13310dd ro nowatchdog nortsched udev.children-max=512 add_efi_memmap intel_idle.max_cstate=1 crashkernel=450M nobau rcupdate.rcu_cpu_stall_suppress=1 processor.max_cstate=1 video=efifb:off cpuidle_sysfs_switch numa_balancing=disable relax_domain_level=2 pcie_aspm=on earlyprintk=ttyS0,115200 skew_tick=1 log_buf_len=8M pci=hpiosize=0,hpmemsize=0,nobar cgroup_disable=memory kgdboc=ttyS0,115200n8 console=ttyS0,115200 resume=/dev/sda3 splash=silent showopts
  966.468794 (    0.052425)| [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
  966.480323 (    0.011529)| [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
  966.484338 (    0.004015)| [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
  966.492340 (    0.008002)| [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
  966.500269 (    0.007929)| [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
  966.508249 (    0.007980)| [    0.000000] x86/fpu: Using 'eager' FPU context switches.
  966.516454 (    0.008205)| [    0.000000] e820: BIOS-provided physical RAM map:
  966.520053 (    0.003599)| [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000051fff] usable
  966.528432 (    0.008379)| [    0.000000] BIOS-e820: [mem 0x0000000000052000-0x000000000005afff] reserved
  966.536502 (    0.008070)| [    0.000000] BIOS-e820: [mem 0x000000000005b000-0x000000000009ffff] usable
  966.544632 (    0.008130)| [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000061ea1fff] usable
  966.552525 (    0.007893)| [    0.000000] BIOS-e820: [mem 0x0000000061ea2000-0x00000000620c9fff] ACPI data
  966.560369 (    0.007844)| [    0.000000] BIOS-e820: [mem 0x00000000620ca000-0x00000000622e4fff] usable
  966.568452 (    0.008083)| [    0.000000] BIOS-e820: [mem 0x00000000622e5000-0x0000000062734fff] ACPI data
  966.576513 (    0.008061)| [    0.000000] BIOS-e820: [mem 0x0000000062735000-0x0000000072829fff] usable
  966.580915 (    0.004402)| [    0.000000] BIOS-e820: [mem 0x000000007282a000-0x000000007282afff] reserved
  966.588372 (    0.007457)| [    0.000000] BIOS-e820: [mem 0x000000007282b000-0x0000000072837fff] usable
  966.596357 (    0.007985)| [    0.000000] BIOS-e820: [mem 0x0000000072838000-0x0000000072a77fff] reserved
  966.604343 (    0.007986)| [    0.000000] BIOS-e820: [mem 0x0000000072a78000-0x00000000751fefff] usable
  966.611893 (    0.007550)| [    0.000000] BIOS-e820: [mem 0x00000000751ff000-0x0000000075efefff] reserved
  966.620054 (    0.008161)| [    0.000000] BIOS-e820: [mem 0x0000000075eff000-0x0000000076efefff] ACPI NVS
  966.628132 (    0.008078)| [    0.000000] BIOS-e820: [mem 0x0000000076eff000-0x0000000076ffefff] ACPI data
  966.636239 (    0.008107)| [    0.000000] BIOS-e820: [mem 0x0000000076fff000-0x000000007effffff] usable
  966.644558 (    0.008319)| [    0.000000] BIOS-e820: [mem 0x0000000080000000-0x000000008fffffff] reserved
  966.652408 (    0.007850)| [    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
  966.660274 (    0.007866)| [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000007bffffffff] usable
  966.668560 (    0.008286)| [    0.000000] BIOS-e820: [mem 0x0000008000000000-0x000000807fffffff] usable
  966.675950 (    0.007390)| [    0.000000] BIOS-e820: [mem 0x0000010000000000-0x0000017bfeffffff] usable
  966.680270 (    0.004320)| [    0.000000] BIOS-e820: [mem 0x0000020000000000-0x0000027bfeffffff] usable
  966.689150 (    0.008880)| [    0.000000] BIOS-e820: [mem 0x0000030000000000-0x0000037bfeffffff] usable
  966.696599 (    0.007449)| [    0.000000] BIOS-e820: [mem 0x0000040000000000-0x0000047bfeffffff] usable
  966.704397 (    0.007798)| [    0.000000] BIOS-e820: [mem 0x0000050000000000-0x0000057bfeffffff] usable
  966.712558 (    0.008161)| [    0.000000] BIOS-e820: [mem 0x0000060000000000-0x0000067bfeffffff] usable
  966.720096 (    0.007538)| [    0.000000] BIOS-e820: [mem 0x0000070000000000-0x0000077bfeffffff] usable
  966.728518 (    0.008422)| [    0.000000] BIOS-e820: [mem 0x0000080000000000-0x0000087bfeffffff] usable
  966.736427 (    0.007909)| [    0.000000] BIOS-e820: [mem 0x0000090000000000-0x0000097bfeffffff] usable
  966.744611 (    0.008184)| [    0.000000] BIOS-e820: [mem 0x00000a0000000000-0x00000a7bfeffffff] usable
  966.748296 (    0.003685)| [    0.000000] BIOS-e820: [mem 0x00000b0000000000-0x00000b7bfeffffff] usable
  966.758209 (    0.009913)| [    0.000000] BIOS-e820: [mem 0x00000c0000000000-0x00000c7bfeffffff] usable
  966.764027 (    0.005818)| [    0.000000] BIOS-e820: [mem 0x00000d0000000000-0x00000d7bfeffffff] usable
  966.772080 (    0.008053)| [    0.000000] BIOS-e820: [mem 0x00000e0000000000-0x00000e7bfeffffff] usable
  966.780327 (    0.008247)| [    0.000000] BIOS-e820: [mem 0x00000f0000000000-0x00000f7bfeffffff] usable
  966.788158 (    0.007831)| [    0.000000] BIOS-e820: [mem 0x0000100000000000-0x0000107bfeffffff] usable
  966.796430 (    0.008272)| [    0.000000] BIOS-e820: [mem 0x0000110000000000-0x0000117bfeffffff] usable
  966.804357 (    0.007927)| [    0.000000] BIOS-e820: [mem 0x0000120000000000-0x0000127bfeffffff] usable
  966.812401 (    0.008044)| [    0.000000] BIOS-e820: [mem 0x0000130000000000-0x0000137bfeffffff] usable
  966.816003 (    0.003602)| [    0.000000] BIOS-e820: [mem 0x0000140000000000-0x0000147bfeffffff] usable
  966.824227 (    0.008224)| [    0.000000] BIOS-e820: [mem 0x0000150000000000-0x0000157bfeffffff] usable
  966.833505 (    0.009278)| [    0.000000] BIOS-e820: [mem 0x0000160000000000-0x0000167bfeffffff] usable
  966.840618 (    0.007113)| [    0.000000] BIOS-e820: [mem 0x0000170000000000-0x0000177bfeffffff] usable
  966.847900 (    0.007282)| [    0.000000] BIOS-e820: [mem 0x0000180000000000-0x0000187bfeffffff] usable
  966.856673 (    0.008773)| [    0.000000] BIOS-e820: [mem 0x0000190000000000-0x0000197bfeffffff] usable
  966.864321 (    0.007648)| [    0.000000] BIOS-e820: [mem 0x00001a0000000000-0x00001a7bfeffffff] usable
  966.872161 (    0.007840)| [    0.000000] BIOS-e820: [mem 0x00001b0000000000-0x00001b7bfeffffff] usable
  966.880347 (    0.008186)| [    0.000000] BIOS-e820: [mem 0x00001c0000000000-0x00001c7bfeffffff] usable
  966.883868 (    0.003521)| [    0.000000] BIOS-e820: [mem 0x00001d0000000000-0x00001d7bfeffffff] usable
  966.892255 (    0.008387)| [    0.000000] BIOS-e820: [mem 0x00001e0000000000-0x00001e7bfeffffff] usable
  966.900857 (    0.008602)| [    0.000000] BIOS-e820: [mem 0x00001f0000000000-0x00001f7bfeffffff] usable
  966.908518 (    0.007661)| [    0.000000] BIOS-e820: [mem 0x00002ffc00000000-0x00002ffc03dfffff] reserved
  966.916116 (    0.007598)| [    0.000000] BIOS-e820: [mem 0x00002ffc03f00000-0x00002ffc07dfffff] reserved
  966.923894 (    0.007778)| [    0.000000] BIOS-e820: [mem 0x00002ffc07f00000-0x00002ffc0bdfffff] reserved
  966.932217 (    0.008323)| [    0.000000] BIOS-e820: [mem 0x00002ffc0bf00000-0x00002ffc0bffffff] reserved
  966.940287 (    0.008070)| [    0.000000] BIOS-e820: [mem 0x00002ffc0ff00000-0x00002ffc13dfffff] reserved
  966.948198 (    0.007911)| [    0.000000] BIOS-e820: [mem 0x00002ffc13f00000-0x00002ffc17dfffff] reserved
  966.955863 (    0.007665)| [    0.000000] BIOS-e820: [mem 0x00002ffc17f00000-0x00002ffc1bdfffff] reserved
  966.963822 (    0.007959)| [    0.000000] BIOS-e820: [mem 0x00002ffc1bf00000-0x00002ffc1bffffff] reserved
  966.972074 (    0.008252)| [    0.000000] BIOS-e820: [mem 0x00002ffc1ff00000-0x00002ffc23dfffff] reserved
  966.980456 (    0.008382)| [    0.000000] BIOS-e820: [mem 0x00002ffc23f00000-0x00002ffc27dfffff] reserved
  966.988522 (    0.008066)| [    0.000000] BIOS-e820: [mem 0x00002ffc27f00000-0x00002ffc2bdfffff] reserved
  966.996087 (    0.007565)| [    0.000000] BIOS-e820: [mem 0x00002ffc2bf00000-0x00002ffc2bffffff] reserved
  967.000090 (    0.004003)| [    0.000000] BIOS-e820: [mem 0x00002ffc2ff00000-0x00002ffc33dfffff] reserved
  967.008274 (    0.008184)| [    0.000000] BIOS-e820: [mem 0x00002ffc33f00000-0x00002ffc37dfffff] reserved
  967.015939 (    0.007665)| [    0.000000] BIOS-e820: [mem 0x00002ffc37f00000-0x00002ffc3bdfffff] reserved
  967.023882 (    0.007943)| [    0.000000] BIOS-e820: [mem 0x00002ffc3bf00000-0x00002ffc3bffffff] reserved
  967.032305 (    0.008423)| [    0.000000] BIOS-e820: [mem 0x00002ffc3ff00000-0x00002ffc3fffffff] reserved
  967.040108 (    0.007803)| [    0.000000] BIOS-e820: [mem 0x00002ffc43f00000-0x00002ffc43ffffff] reserved
  967.048216 (    0.008108)| [    0.000000] BIOS-e820: [mem 0x00002ffc47f00000-0x00002ffc47ffffff] reserved
  967.056061 (    0.007845)| [    0.000000] BIOS-e820: [mem 0x00002ffc4bf00000-0x00002ffc4bffffff] reserved
  967.064138 (    0.008077)| [    0.000000] BIOS-e820: [mem 0x00002ffc4ff00000-0x00002ffc4fffffff] reserved
  967.071839 (    0.007701)| [    0.000000] BIOS-e820: [mem 0x00002ffc53f00000-0x00002ffc53ffffff] reserved
  967.080170 (    0.008331)| [    0.000000] BIOS-e820: [mem 0x00002ffc57f00000-0x00002ffc57ffffff] reserved
  967.088397 (    0.008227)| [    0.000000] BIOS-e820: [mem 0x00002ffc5bf00000-0x00002ffc5bffffff] reserved
  967.096794 (    0.008397)| [    0.000000] BIOS-e820: [mem 0x00002ffc5ff00000-0x00002ffc5fffffff] reserved
  967.104760 (    0.007966)| [    0.000000] BIOS-e820: [mem 0x00002ffc63f00000-0x00002ffc63ffffff] reserved
  967.111858 (    0.007098)| [    0.000000] BIOS-e820: [mem 0x00002ffc67f00000-0x00002ffc67ffffff] reserved
  967.119864 (    0.008006)| [    0.000000] BIOS-e820: [mem 0x00002ffc6bf00000-0x00002ffc6bffffff] reserved
  967.124024 (    0.004160)| [    0.000000] BIOS-e820: [mem 0x00002ffc6ff00000-0x00002ffc6fffffff] reserved
  967.132175 (    0.008151)| [    0.000000] BIOS-e820: [mem 0x00002ffc73f00000-0x00002ffc73ffffff] reserved
  967.140165 (    0.007990)| [    0.000000] BIOS-e820: [mem 0x00002ffc77f00000-0x00002ffc77ffffff] reserved
  967.148323 (    0.008158)| [    0.000000] BIOS-e820: [mem 0x00002ffc7bf00000-0x00002ffc7bffffff] reserved
  967.157670 (    0.009347)| [    0.000000] BIOS-e820: [mem 0x00002ffc7ff00000-0x00002ffc7fffffff] reserved
  967.163832 (    0.006162)| [    0.000000] BIOS-e820: [mem 0x00002ffd00000000-0x00002ffd02ffffff] reserved
  967.171844 (    0.008012)| [    0.000000] BIOS-e820: [mem 0x00002ffd04000000-0x00002ffd06ffffff] reserved
  967.179979 (    0.008135)| [    0.000000] BIOS-e820: [mem 0x00002ffd08000000-0x00002ffd0affffff] reserved
  967.188368 (    0.008389)| [    0.000000] BIOS-e820: [mem 0x00002ffd0c000000-0x00002ffd0effffff] reserved
  967.196048 (    0.007680)| [    0.000000] BIOS-e820: [mem 0x00002ffd10000000-0x00002ffd12ffffff] reserved
  967.204511 (    0.008463)| [    0.000000] BIOS-e820: [mem 0x00002ffd14000000-0x00002ffd16ffffff] reserved
  967.211865 (    0.007354)| [    0.000000] BIOS-e820: [mem 0x00002ffd18000000-0x00002ffd1affffff] reserved
  967.220793 (    0.008928)| [    0.000000] BIOS-e820: [mem 0x00002ffd1c000000-0x00002ffd1effffff] reserved
  967.229455 (    0.008662)| [    0.000000] BIOS-e820: [mem 0x00002ffd20000000-0x00002ffd22ffffff] reserved
  967.237348 (    0.007893)| [    0.000000] BIOS-e820: [mem 0x00002ffd24000000-0x00002ffd26ffffff] reserved
  967.246190 (    0.008842)| [    0.000000] BIOS-e820: [mem 0x00002ffd28000000-0x00002ffd2affffff] reserved
  967.248598 (    0.002408)| [    0.000000] BIOS-e820: [mem 0x00002ffd2c000000-0x00002ffd2effffff] reserved
  967.256394 (    0.007796)| [    0.000000] BIOS-e820: [mem 0x00002ffd30000000-0x00002ffd32ffffff] reserved
  967.264671 (    0.008277)| [    0.000000] BIOS-e820: [mem 0x00002ffd34000000-0x00002ffd36ffffff] reserved
  967.271907 (    0.007236)| [    0.000000] BIOS-e820: [mem 0x00002ffd38000000-0x00002ffd3affffff] reserved
  967.280139 (    0.008232)| [    0.000000] BIOS-e820: [mem 0x00002ffd3c000000-0x00002ffd3effffff] reserved
  967.288191 (    0.008052)| [    0.000000] BIOS-e820: [mem 0x00002ffd40000000-0x00002ffd42ffffff] reserved
  967.297098 (    0.008907)| [    0.000000] BIOS-e820: [mem 0x00002ffd44000000-0x00002ffd46ffffff] reserved
  967.305083 (    0.007985)| [    0.000000] BIOS-e820: [mem 0x00002ffd48000000-0x00002ffd4affffff] reserved
  967.313166 (    0.008083)| [    0.000000] BIOS-e820: [mem 0x00002ffd4c000000-0x00002ffd4effffff] reserved
  967.320288 (    0.007122)| [    0.000000] BIOS-e820: [mem 0x00002ffd50000000-0x00002ffd52ffffff] reserved
  967.328252 (    0.007964)| [    0.000000] BIOS-e820: [mem 0x00002ffd54000000-0x00002ffd56ffffff] reserved
  967.336265 (    0.008013)| [    0.000000] BIOS-e820: [mem 0x00002ffd58000000-0x00002ffd5affffff] reserved
  967.344181 (    0.007916)| [    0.000000] BIOS-e820: [mem 0x00002ffd5c000000-0x00002ffd5effffff] reserved
  967.351720 (    0.007539)| [    0.000000] BIOS-e820: [mem 0x00002ffd60000000-0x00002ffd62ffffff] reserved
  967.359750 (    0.008030)| [    0.000000] BIOS-e820: [mem 0x00002ffd64000000-0x00002ffd66ffffff] reserved
  967.368580 (    0.008830)| [    0.000000] BIOS-e820: [mem 0x00002ffd68000000-0x00002ffd6affffff] reserved
  967.377416 (    0.008836)| [    0.000000] BIOS-e820: [mem 0x00002ffd6c000000-0x00002ffd6effffff] reserved
  967.384417 (    0.007001)| [    0.000000] BIOS-e820: [mem 0x00002ffd70000000-0x00002ffd72ffffff] reserved
  967.387731 (    0.003314)| [    0.000000] BIOS-e820: [mem 0x00002ffd74000000-0x00002ffd76ffffff] reserved
  967.395803 (    0.008072)| [    0.000000] BIOS-e820: [mem 0x00002ffd78000000-0x00002ffd7affffff] reserved
  967.403892 (    0.008089)| [    0.000000] BIOS-e820: [mem 0x00002ffd7c000000-0x00002ffd7effffff] reserved
  967.412167 (    0.008275)| [    0.000000] bootconsole [earlyser0] enabled
  967.416199 (    0.004032)| [    0.000000] NX (Execute Disable) protection: active
  967.423883 (    0.007684)| [    0.000000] efi: EFI v2.40 by EDK II
  967.427862 (    0.003979)| [    0.000000]                              BIOS ID: 0 BIOS ID:\x02
  967.432067 (    0.004205)| [    0.000000] efi:  SMBIOS=0x753f0000  ACPI=0x76ffe000  ACPI 2.0=0x76ffe014  UVsystab=0x753e8000 
  967.444148 (    0.012081)| [    0.000000] SMBIOS 3.0 present.
  967.447974 (    0.003826)| [    0.000000] e820: last_pfn = 0x1f7bff000 max_arch_pfn = 0x400000000
  967.459657 (    0.011683)| [    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
  967.467945 (    0.008288)| [    0.000000] x2apic: enabled by BIOS, switching to x2apic ops
  967.476389 (    0.008444)| [    0.000000] e820: last_pfn = 0x7f000 max_arch_pfn = 0x400000000
  967.484089 (    0.007700)| [    0.000000] Scanning 1 areas for low memory corruption
  967.488350 (    0.004261)| [    0.000000] Using GB pages for direct mapping
  967.499716 (    0.011366)| [    0.000000] log_buf_len: 8388608 bytes
  967.503712 (    0.003996)| [    0.000000] early log buf free: 249548(95%)
  967.507687 (    0.003975)| [    0.000000] RAMDISK: [mem 0x3eee4000-0x3fffafff]
  967.516051 (    0.008364)| [    0.000000] ACPI: Early table checksum verification disabled
  967.520325 (    0.004274)| [    0.000000] ACPI: RSDP 0x0000000076FFE014 000024 (v02 INTEL )
  967.528613 (    0.008288)| [    0.000000] ACPI: XSDT 0x0000000076FFD0E8 000074 (v01 INTEL  TIANO    00000000      01000013)
  967.536151 (    0.007538)| [    0.000000] ACPI: FACP 0x0000000076FFC000 0000F4 (v04 INTEL  TIANO    00000000 MSFT 01000013)
  967.547824 (    0.011673)| [    0.000000] ACPI: DSDT 0x0000000076FF3000 007857 (v02 INTEL  TIANO    00000003 MSFT 01000013)
  967.555660 (    0.007836)| [    0.000000] ACPI: FACS 0x0000000076EFA000 000040
  967.560175 (    0.004515)| [    0.000000] ACPI: UEFI 0x0000000076EFC000 00005C (v01 INTEL  RstSataV 00000000      00000000)
  967.572066 (    0.011891)| [    0.000000] ACPI: HPET 0x0000000076FFB000 000038 (v01 INTEL  TIANO    00000001 MSFT 01000013)
  967.579986 (    0.007920)| [    0.000000] ACPI: SLIT 0x0000000076FF0000 00042C (v01 SGI3X  UVX      00000002 MSFT 00000001)
  967.588250 (    0.008264)| [    0.000000] ACPI: APIC 0x0000000076FE2000 0060DC (v03 SGI3X  UVX      00000002 MSFT 00000001)
  967.599853 (    0.011603)| [    0.000000] ACPI: SRAT 0x0000000076FCE000 009558 (v03 SGI3X  UVX      00000002 MSFT 00000001)
  967.607770 (    0.007917)| [    0.000000] ACPI: SSDT 0x00000000622E5000 22739E (v02 SGI3X  UVX      00000002 MSFT 01000013)
  967.615965 (    0.008195)| [    0.000000] ACPI: MCFG 0x0000000076FCC000 0002EC (v01 SGI3X  UVX      00000002 MSFT 00000001)
  967.628273 (    0.012308)| [    0.000000] ACPI: SPCR 0x0000000076FCB000 000050 (v01                 00000000      00000000)
  967.635918 (    0.007645)| [    0.000000] ACPI: DMAR 0x0000000076FCA000 0004D0 (v01 INTEL  TIANO    00000001 MSFT 01000013)
  967.647777 (    0.011859)| [    0.000000] UV: rev:1 part#:9578 nodeid:0000 n_skt:6 pnmsk:3f pn:0
  967.652024 (    0.004247)| [    0.000000] UV: apicid_shift:0 apicid_mask:0x3f
  967.659934 (    0.007910)| [    0.000000] UV: socketid_shift:6 pnode_mask:0x3f
  967.663914 (    0.003980)| [    0.000000] UV: OEM IDs SGI3X/UVX, System/HUB Types 2/6, uv_apic 0
  967.668470 (    0.004556)| [    0.000000] Setting APIC routing to cluster x2apic.
  967.676153 (    0.007683)| [    0.000000] SRAT: PXM 0 -> APIC 0x0000 -> Node 0
  967.680362 (    0.004209)| [    0.000000] SRAT: PXM 0 -> APIC 0x0002 -> Node 0
  967.684117 (    0.003755)| [    0.000000] SRAT: PXM 0 -> APIC 0x0004 -> Node 0
  967.691793 (    0.007676)| [    0.000000] SRAT: PXM 0 -> APIC 0x0006 -> Node 0
  967.695654 (    0.003861)| [    0.000000] SRAT: PXM 0 -> APIC 0x0008 -> Node 0
  967.699686 (    0.004032)| [    0.000000] SRAT: PXM 0 -> APIC 0x000a -> Node 0
  967.707726 (    0.008040)| [    0.000000] SRAT: PXM 0 -> APIC 0x0010 -> Node 0
  967.712710 (    0.004984)| [    0.000000] SRAT: PXM 0 -> APIC 0x0012 -> Node 0
  967.715821 (    0.003111)| [    0.000000] SRAT: PXM 0 -> APIC 0x0014 -> Node 0
  967.720171 (    0.004350)| [    0.000000] SRAT: PXM 0 -> APIC 0x0016 -> Node 0
  967.728522 (    0.008351)| [    0.000000] SRAT: PXM 0 -> APIC 0x0018 -> Node 0
  967.732647 (    0.004125)| [    0.000000] SRAT: PXM 0 -> APIC 0x001a -> Node 0
  967.735867 (    0.003220)| [    0.000000] SRAT: PXM 0 -> APIC 0x0020 -> Node 0
  967.743914 (    0.008047)| [    0.000000] SRAT: PXM 0 -> APIC 0x0022 -> Node 0
  967.747906 (    0.003992)| [    0.000000] SRAT: PXM 0 -> APIC 0x0024 -> Node 0
  967.751731 (    0.003825)| [    0.000000] SRAT: PXM 0 -> APIC 0x0026 -> Node 0
  967.755663 (    0.003932)| [    0.000000] SRAT: PXM 0 -> APIC 0x0028 -> Node 0
  967.763868 (    0.008205)| [    0.000000] SRAT: PXM 0 -> APIC 0x002a -> Node 0
  967.767904 (    0.004036)| [    0.000000] SRAT: PXM 0 -> APIC 0x0030 -> Node 0
  967.771928 (    0.004024)| [    0.000000] SRAT: PXM 0 -> APIC 0x0032 -> Node 0
  967.779792 (    0.007864)| [    0.000000] SRAT: PXM 0 -> APIC 0x0034 -> Node 0
  967.784818 (    0.005026)| [    0.000000] SRAT: PXM 0 -> APIC 0x0036 -> Node 0
  967.788096 (    0.003278)| [    0.000000] SRAT: PXM 0 -> APIC 0x0038 -> Node 0
  967.792183 (    0.004087)| [    0.000000] SRAT: PXM 0 -> APIC 0x003a -> Node 0
  967.799620 (    0.007437)| [    0.000000] SRAT: PXM 1 -> APIC 0x0040 -> Node 1
  967.803873 (    0.004253)| [    0.000000] SRAT: PXM 1 -> APIC 0x0042 -> Node 1
  967.807982 (    0.004109)| [    0.000000] SRAT: PXM 1 -> APIC 0x0044 -> Node 1
  967.815840 (    0.007858)| [    0.000000] SRAT: PXM 1 -> APIC 0x0046 -> Node 1
  967.819922 (    0.004082)| [    0.000000] SRAT: PXM 1 -> APIC 0x0048 -> Node 1
  967.824218 (    0.004296)| [    0.000000] SRAT: PXM 1 -> APIC 0x004a -> Node 1
  967.829106 (    0.004888)| [    0.000000] SRAT: PXM 1 -> APIC 0x0050 -> Node 1
  967.835788 (    0.006682)| [    0.000000] SRAT: PXM 1 -> APIC 0x0052 -> Node 1
  967.842187 (    0.006399)| [    0.000000] SRAT: PXM 1 -> APIC 0x0054 -> Node 1
  967.844578 (    0.002391)| [    0.000000] SRAT: PXM 1 -> APIC 0x0056 -> Node 1
  967.851564 (    0.006986)| [    0.000000] SRAT: PXM 1 -> APIC 0x0058 -> Node 1
  967.855835 (    0.004271)| [    0.000000] SRAT: PXM 1 -> APIC 0x005a -> Node 1
  967.860800 (    0.004965)| [    0.000000] SRAT: PXM 1 -> APIC 0x0060 -> Node 1
  967.864562 (    0.003762)| [    0.000000] SRAT: PXM 1 -> APIC 0x0062 -> Node 1
  967.873249 (    0.008687)| [    0.000000] SRAT: PXM 1 -> APIC 0x0064 -> Node 1
  967.876628 (    0.003379)| [    0.000000] SRAT: PXM 1 -> APIC 0x0066 -> Node 1
  967.880507 (    0.003879)| [    0.000000] SRAT: PXM 1 -> APIC 0x0068 -> Node 1
  967.888034 (    0.007527)| [    0.000000] SRAT: PXM 1 -> APIC 0x006a -> Node 1
  967.891323 (    0.003289)| [    0.000000] SRAT: PXM 1 -> APIC 0x0070 -> Node 1
  967.895507 (    0.004184)| [    0.000000] SRAT: PXM 1 -> APIC 0x0072 -> Node 1
  967.899978 (    0.004471)| [    0.000000] SRAT: PXM 1 -> APIC 0x0074 -> Node 1
  967.907766 (    0.007788)| [    0.000000] SRAT: PXM 1 -> APIC 0x0076 -> Node 1
  967.912160 (    0.004394)| [    0.000000] SRAT: PXM 1 -> APIC 0x0078 -> Node 1
  967.916922 (    0.004762)| [    0.000000] SRAT: PXM 1 -> APIC 0x007a -> Node 1
  967.924559 (    0.007637)| [    0.000000] SRAT: PXM 2 -> APIC 0x0080 -> Node 2
  967.927988 (    0.003429)| [    0.000000] SRAT: PXM 2 -> APIC 0x0082 -> Node 2
  967.931836 (    0.003848)| [    0.000000] SRAT: PXM 2 -> APIC 0x0084 -> Node 2
  967.935913 (    0.004077)| [    0.000000] SRAT: PXM 2 -> APIC 0x0086 -> Node 2
  967.943984 (    0.008071)| [    0.000000] SRAT: PXM 2 -> APIC 0x0088 -> Node 2
  967.948068 (    0.004084)| [    0.000000] SRAT: PXM 2 -> APIC 0x008a -> Node 2
  967.951827 (    0.003759)| [    0.000000] SRAT: PXM 2 -> APIC 0x0090 -> Node 2
  967.960037 (    0.008210)| [    0.000000] SRAT: PXM 2 -> APIC 0x0092 -> Node 2
  967.963818 (    0.003781)| [    0.000000] SRAT: PXM 2 -> APIC 0x0094 -> Node 2
  967.967733 (    0.003915)| [    0.000000] SRAT: PXM 2 -> APIC 0x0096 -> Node 2
  967.971814 (    0.004081)| [    0.000000] SRAT: PXM 2 -> APIC 0x0098 -> Node 2
  967.979998 (    0.008184)| [    0.000000] SRAT: PXM 2 -> APIC 0x009a -> Node 2
  967.984101 (    0.004103)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a0 -> Node 2
  967.987943 (    0.003842)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a2 -> Node 2
  967.995481 (    0.007538)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a4 -> Node 2
  967.999833 (    0.004352)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a6 -> Node 2
  968.003582 (    0.003749)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a8 -> Node 2
  968.007659 (    0.004077)| [    0.000000] SRAT: PXM 2 -> APIC 0x00aa -> Node 2
  968.016175 (    0.008516)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b0 -> Node 2
  968.019813 (    0.003638)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b2 -> Node 2
  968.023845 (    0.004032)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b4 -> Node 2
  968.032103 (    0.008258)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b6 -> Node 2
  968.035900 (    0.003797)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b8 -> Node 2
  968.039787 (    0.003887)| [    0.000000] SRAT: PXM 2 -> APIC 0x00ba -> Node 2
  968.043631 (    0.003844)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c0 -> Node 3
  968.051582 (    0.007951)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c2 -> Node 3
  968.055726 (    0.004144)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c4 -> Node 3
  968.059563 (    0.003837)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c6 -> Node 3
  968.067913 (    0.008350)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c8 -> Node 3
  968.071802 (    0.003889)| [    0.000000] SRAT: PXM 3 -> APIC 0x00ca -> Node 3
  968.076265 (    0.004463)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d0 -> Node 3
  968.079932 (    0.003667)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d2 -> Node 3
  968.089091 (    0.009159)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d4 -> Node 3
  968.091975 (    0.002884)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d6 -> Node 3
  968.096082 (    0.004107)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d8 -> Node 3
  968.103652 (    0.007570)| [    0.000000] SRAT: PXM 3 -> APIC 0x00da -> Node 3
  968.107495 (    0.003843)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e0 -> Node 3
  968.111708 (    0.004213)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e2 -> Node 3
  968.115736 (    0.004028)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e4 -> Node 3
  968.123767 (    0.008031)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e6 -> Node 3
  968.127943 (    0.004176)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e8 -> Node 3
  968.132016 (    0.004073)| [    0.000000] SRAT: PXM 3 -> APIC 0x00ea -> Node 3
  968.141249 (    0.009233)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f0 -> Node 3
  968.143599 (    0.002350)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f2 -> Node 3
  968.147605 (    0.004006)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f4 -> Node 3
  968.151594 (    0.003989)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f6 -> Node 3
  968.159567 (    0.007973)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f8 -> Node 3
  968.163893 (    0.004326)| [    0.000000] SRAT: PXM 3 -> APIC 0x00fa -> Node 3
  968.167753 (    0.003860)| [    0.000000] SRAT: PXM 4 -> APIC 0x0100 -> Node 4
  968.176231 (    0.008478)| [    0.000000] SRAT: PXM 4 -> APIC 0x0102 -> Node 4
  968.180142 (    0.003911)| [    0.000000] SRAT: PXM 4 -> APIC 0x0104 -> Node 4
  968.184553 (    0.004411)| [    0.000000] SRAT: PXM 4 -> APIC 0x0106 -> Node 4
  968.188259 (    0.003706)| [    0.000000] SRAT: PXM 4 -> APIC 0x0108 -> Node 4
  968.195392 (    0.007133)| [    0.000000] SRAT: PXM 4 -> APIC 0x010a -> Node 4
  968.199474 (    0.004082)| [    0.000000] SRAT: PXM 4 -> APIC 0x0110 -> Node 4
  968.203503 (    0.004029)| [    0.000000] SRAT: PXM 4 -> APIC 0x0112 -> Node 4
  968.211556 (    0.008053)| [    0.000000] SRAT: PXM 4 -> APIC 0x0114 -> Node 4
  968.216753 (    0.005197)| [    0.000000] SRAT: PXM 4 -> APIC 0x0116 -> Node 4
  968.220509 (    0.003756)| [    0.000000] SRAT: PXM 4 -> APIC 0x0118 -> Node 4
  968.225681 (    0.005172)| [    0.000000] SRAT: PXM 4 -> APIC 0x011a -> Node 4
  968.233182 (    0.007501)| [    0.000000] SRAT: PXM 4 -> APIC 0x0120 -> Node 4
  968.236093 (    0.002911)| [    0.000000] SRAT: PXM 4 -> APIC 0x0122 -> Node 4
  968.240266 (    0.004173)| [    0.000000] SRAT: PXM 4 -> APIC 0x0124 -> Node 4
  968.244203 (    0.003937)| [    0.000000] SRAT: PXM 4 -> APIC 0x0126 -> Node 4
  968.253684 (    0.009481)| [    0.000000] SRAT: PXM 4 -> APIC 0x0128 -> Node 4
  968.257531 (    0.003847)| [    0.000000] SRAT: PXM 4 -> APIC 0x012a -> Node 4
  968.260952 (    0.003421)| [    0.000000] SRAT: PXM 4 -> APIC 0x0130 -> Node 4
  968.268323 (    0.007371)| [    0.000000] SRAT: PXM 4 -> APIC 0x0132 -> Node 4
  968.271609 (    0.003286)| [    0.000000] SRAT: PXM 4 -> APIC 0x0134 -> Node 4
  968.275792 (    0.004183)| [    0.000000] SRAT: PXM 4 -> APIC 0x0136 -> Node 4
  968.284089 (    0.008297)| [    0.000000] SRAT: PXM 4 -> APIC 0x0138 -> Node 4
  968.288094 (    0.004005)| [    0.000000] SRAT: PXM 4 -> APIC 0x013a -> Node 4
  968.293143 (    0.005049)| [    0.000000] SRAT: PXM 5 -> APIC 0x0140 -> Node 5
  968.296276 (    0.003133)| [    0.000000] SRAT: PXM 5 -> APIC 0x0142 -> Node 5
  968.304374 (    0.008098)| [    0.000000] SRAT: PXM 5 -> APIC 0x0144 -> Node 5
  968.308412 (    0.004038)| [    0.000000] SRAT: PXM 5 -> APIC 0x0146 -> Node 5
  968.312001 (    0.003589)| [    0.000000] SRAT: PXM 5 -> APIC 0x0148 -> Node 5
  968.319525 (    0.007524)| [    0.000000] SRAT: PXM 5 -> APIC 0x014a -> Node 5
  968.323803 (    0.004278)| [    0.000000] SRAT: PXM 5 -> APIC 0x0150 -> Node 5
  968.328764 (    0.004961)| [    0.000000] SRAT: PXM 5 -> APIC 0x0152 -> Node 5
  968.332677 (    0.003913)| [    0.000000] SRAT: PXM 5 -> APIC 0x0154 -> Node 5
  968.342040 (    0.009363)| [    0.000000] SRAT: PXM 5 -> APIC 0x0156 -> Node 5
  968.342174 (    0.000134)| [    0.000000] SRAT: PXM 5 -> APIC 0x0158 -> Node 5
  968.348139 (    0.005965)| [    0.000000] SRAT: PXM 5 -> APIC 0x015a -> Node 5
  968.355734 (    0.007595)| [    0.000000] SRAT: PXM 5 -> APIC 0x0160 -> Node 5
  968.359968 (    0.004234)| [    0.000000] SRAT: PXM 5 -> APIC 0x0162 -> Node 5
  968.363731 (    0.003763)| [    0.000000] SRAT: PXM 5 -> APIC 0x0164 -> Node 5
  968.367929 (    0.004198)| [    0.000000] SRAT: PXM 5 -> APIC 0x0166 -> Node 5
  968.377451 (    0.009522)| [    0.000000] SRAT: PXM 5 -> APIC 0x0168 -> Node 5
  968.379824 (    0.002373)| [    0.000000] SRAT: PXM 5 -> APIC 0x016a -> Node 5
  968.383644 (    0.003820)| [    0.000000] SRAT: PXM 5 -> APIC 0x0170 -> Node 5
  968.391518 (    0.007874)| [    0.000000] SRAT: PXM 5 -> APIC 0x0172 -> Node 5
  968.396328 (    0.004810)| [    0.000000] SRAT: PXM 5 -> APIC 0x0174 -> Node 5
  968.400037 (    0.003709)| [    0.000000] SRAT: PXM 5 -> APIC 0x0176 -> Node 5
  968.403807 (    0.003770)| [    0.000000] SRAT: PXM 5 -> APIC 0x0178 -> Node 5
  968.412345 (    0.008538)| [    0.000000] SRAT: PXM 5 -> APIC 0x017a -> Node 5
  968.416485 (    0.004140)| [    0.000000] SRAT: PXM 6 -> APIC 0x0180 -> Node 6
  968.419900 (    0.003415)| [    0.000000] SRAT: PXM 6 -> APIC 0x0182 -> Node 6
  968.427488 (    0.007588)| [    0.000000] SRAT: PXM 6 -> APIC 0x0184 -> Node 6
  968.431520 (    0.004032)| [    0.000000] SRAT: PXM 6 -> APIC 0x0186 -> Node 6
  968.435585 (    0.004065)| [    0.000000] SRAT: PXM 6 -> APIC 0x0188 -> Node 6
  968.439703 (    0.004118)| [    0.000000] SRAT: PXM 6 -> APIC 0x018a -> Node 6
  968.447972 (    0.008269)| [    0.000000] SRAT: PXM 6 -> APIC 0x0190 -> Node 6
  968.452412 (    0.004440)| [    0.000000] SRAT: PXM 6 -> APIC 0x0192 -> Node 6
  968.456134 (    0.003722)| [    0.000000] SRAT: PXM 6 -> APIC 0x0194 -> Node 6
  968.463633 (    0.007499)| [    0.000000] SRAT: PXM 6 -> APIC 0x0196 -> Node 6
  968.469005 (    0.005372)| [    0.000000] SRAT: PXM 6 -> APIC 0x0198 -> Node 6
  968.472218 (    0.003213)| [    0.000000] SRAT: PXM 6 -> APIC 0x019a -> Node 6
  968.475915 (    0.003697)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a0 -> Node 6
  968.483479 (    0.007564)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a2 -> Node 6
  968.488235 (    0.004756)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a4 -> Node 6
  968.491448 (    0.003213)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a6 -> Node 6
  968.496006 (    0.004558)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a8 -> Node 6
  968.503684 (    0.007678)| [    0.000000] SRAT: PXM 6 -> APIC 0x01aa -> Node 6
  968.507729 (    0.004045)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b0 -> Node 6
  968.512765 (    0.005036)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b2 -> Node 6
  968.519569 (    0.006804)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b4 -> Node 6
  968.525030 (    0.005461)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b6 -> Node 6
  968.527338 (    0.002308)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b8 -> Node 6
  968.535733 (    0.008395)| [    0.000000] SRAT: PXM 6 -> APIC 0x01ba -> Node 6
  968.539931 (    0.004198)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c0 -> Node 7
  968.543518 (    0.003587)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c2 -> Node 7
  968.547817 (    0.004299)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c4 -> Node 7
  968.555706 (    0.007889)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c6 -> Node 7
  968.559551 (    0.003845)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c8 -> Node 7
  968.563888 (    0.004337)| [    0.000000] SRAT: PXM 7 -> APIC 0x01ca -> Node 7
  968.572079 (    0.008191)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d0 -> Node 7
  968.576706 (    0.004627)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d2 -> Node 7
  968.579642 (    0.002936)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d4 -> Node 7
  968.583839 (    0.004197)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d6 -> Node 7
  968.592053 (    0.008214)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d8 -> Node 7
  968.595657 (    0.003604)| [    0.000000] SRAT: PXM 7 -> APIC 0x01da -> Node 7
  968.599636 (    0.003979)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e0 -> Node 7
  968.609159 (    0.009523)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e2 -> Node 7
  968.613991 (    0.004832)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e4 -> Node 7
  968.616544 (    0.002553)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e6 -> Node 7
  968.620456 (    0.003912)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e8 -> Node 7
  968.627296 (    0.006840)| [    0.000000] SRAT: PXM 7 -> APIC 0x01ea -> Node 7
  968.631634 (    0.004338)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f0 -> Node 7
  968.635429 (    0.003795)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f2 -> Node 7
  968.644699 (    0.009270)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f4 -> Node 7
  968.648259 (    0.003560)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f6 -> Node 7
  968.651615 (    0.003356)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f8 -> Node 7
  968.656053 (    0.004438)| [    0.000000] SRAT: PXM 7 -> APIC 0x01fa -> Node 7
  968.663544 (    0.007491)| [    0.000000] SRAT: PXM 8 -> APIC 0x0200 -> Node 8
  968.668295 (    0.004751)| [    0.000000] SRAT: PXM 8 -> APIC 0x0202 -> Node 8
  968.671588 (    0.003293)| [    0.000000] SRAT: PXM 8 -> APIC 0x0204 -> Node 8
  968.680100 (    0.008512)| [    0.000000] SRAT: PXM 8 -> APIC 0x0206 -> Node 8
  968.683921 (    0.003821)| [    0.000000] SRAT: PXM 8 -> APIC 0x0208 -> Node 8
  968.687747 (    0.003826)| [    0.000000] SRAT: PXM 8 -> APIC 0x020a -> Node 8
  968.691839 (    0.004092)| [    0.000000] SRAT: PXM 8 -> APIC 0x0210 -> Node 8
  968.699589 (    0.007750)| [    0.000000] SRAT: PXM 8 -> APIC 0x0212 -> Node 8
  968.704122 (    0.004533)| [    0.000000] SRAT: PXM 8 -> APIC 0x0214 -> Node 8
  968.707862 (    0.003740)| [    0.000000] SRAT: PXM 8 -> APIC 0x0216 -> Node 8
  968.716854 (    0.008992)| [    0.000000] SRAT: PXM 8 -> APIC 0x0218 -> Node 8
  968.720331 (    0.003477)| [    0.000000] SRAT: PXM 8 -> APIC 0x021a -> Node 8
  968.724753 (    0.004422)| [    0.000000] SRAT: PXM 8 -> APIC 0x0220 -> Node 8
  968.727383 (    0.002630)| [    0.000000] SRAT: PXM 8 -> APIC 0x0222 -> Node 8
  968.735957 (    0.008574)| [    0.000000] SRAT: PXM 8 -> APIC 0x0224 -> Node 8
  968.739299 (    0.003342)| [    0.000000] SRAT: PXM 8 -> APIC 0x0226 -> Node 8
  968.743847 (    0.004548)| [    0.000000] SRAT: PXM 8 -> APIC 0x0228 -> Node 8
  968.751611 (    0.007764)| [    0.000000] SRAT: PXM 8 -> APIC 0x022a -> Node 8
  968.755463 (    0.003852)| [    0.000000] SRAT: PXM 8 -> APIC 0x0230 -> Node 8
  968.759639 (    0.004176)| [    0.000000] SRAT: PXM 8 -> APIC 0x0232 -> Node 8
  968.764170 (    0.004531)| [    0.000000] SRAT: PXM 8 -> APIC 0x0234 -> Node 8
  968.772762 (    0.008592)| [    0.000000] SRAT: PXM 8 -> APIC 0x0236 -> Node 8
  968.775398 (    0.002636)| [    0.000000] SRAT: PXM 8 -> APIC 0x0238 -> Node 8
  968.779713 (    0.004315)| [    0.000000] SRAT: PXM 8 -> APIC 0x023a -> Node 8
  968.787560 (    0.007847)| [    0.000000] SRAT: PXM 9 -> APIC 0x0240 -> Node 9
  968.791353 (    0.003793)| [    0.000000] SRAT: PXM 9 -> APIC 0x0242 -> Node 9
  968.795208 (    0.003855)| [    0.000000] SRAT: PXM 9 -> APIC 0x0244 -> Node 9
  968.799319 (    0.004111)| [    0.000000] SRAT: PXM 9 -> APIC 0x0246 -> Node 9
  968.807455 (    0.008136)| [    0.000000] SRAT: PXM 9 -> APIC 0x0248 -> Node 9
  968.811543 (    0.004088)| [    0.000000] SRAT: PXM 9 -> APIC 0x024a -> Node 9
  968.815972 (    0.004429)| [    0.000000] SRAT: PXM 9 -> APIC 0x0250 -> Node 9
  968.823722 (    0.007750)| [    0.000000] SRAT: PXM 9 -> APIC 0x0252 -> Node 9
  968.827445 (    0.003723)| [    0.000000] SRAT: PXM 9 -> APIC 0x0254 -> Node 9
  968.831629 (    0.004184)| [    0.000000] SRAT: PXM 9 -> APIC 0x0256 -> Node 9
  968.835287 (    0.003658)| [    0.000000] SRAT: PXM 9 -> APIC 0x0258 -> Node 9
  968.843555 (    0.008268)| [    0.000000] SRAT: PXM 9 -> APIC 0x025a -> Node 9
  968.847383 (    0.003828)| [    0.000000] SRAT: PXM 9 -> APIC 0x0260 -> Node 9
  968.851346 (    0.003963)| [    0.000000] SRAT: PXM 9 -> APIC 0x0262 -> Node 9
  968.859426 (    0.008080)| [    0.000000] SRAT: PXM 9 -> APIC 0x0264 -> Node 9
  968.863276 (    0.003850)| [    0.000000] SRAT: PXM 9 -> APIC 0x0266 -> Node 9
  968.867608 (    0.004332)| [    0.000000] SRAT: PXM 9 -> APIC 0x0268 -> Node 9
  968.871508 (    0.003900)| [    0.000000] SRAT: PXM 9 -> APIC 0x026a -> Node 9
  968.879336 (    0.007828)| [    0.000000] SRAT: PXM 9 -> APIC 0x0270 -> Node 9
  968.882824 (    0.003488)| [    0.000000] SRAT: PXM 9 -> APIC 0x0272 -> Node 9
  968.887342 (    0.004518)| [    0.000000] SRAT: PXM 9 -> APIC 0x0274 -> Node 9
  968.896525 (    0.009183)| [    0.000000] SRAT: PXM 9 -> APIC 0x0276 -> Node 9
  968.903493 (    0.006968)| [    0.000000] SRAT: PXM 9 -> APIC 0x0278 -> Node 9
  968.904189 (    0.000696)| [    0.000000] SRAT: PXM 9 -> APIC 0x027a -> Node 9
  968.907565 (    0.003376)| [    0.000000] SRAT: PXM 10 -> APIC 0x0280 -> Node 10
  968.915178 (    0.007613)| [    0.000000] SRAT: PXM 10 -> APIC 0x0282 -> Node 10
  968.919528 (    0.004350)| [    0.000000] SRAT: PXM 10 -> APIC 0x0284 -> Node 10
  968.923995 (    0.004467)| [    0.000000] SRAT: PXM 10 -> APIC 0x0286 -> Node 10
  968.931216 (    0.007221)| [    0.000000] SRAT: PXM 10 -> APIC 0x0288 -> Node 10
  968.937150 (    0.005934)| [    0.000000] SRAT: PXM 10 -> APIC 0x028a -> Node 10
  968.940215 (    0.003065)| [    0.000000] SRAT: PXM 10 -> APIC 0x0290 -> Node 10
  968.947827 (    0.007612)| [    0.000000] SRAT: PXM 10 -> APIC 0x0292 -> Node 10
  968.952377 (    0.004550)| [    0.000000] SRAT: PXM 10 -> APIC 0x0294 -> Node 10
  968.956033 (    0.003656)| [    0.000000] SRAT: PXM 10 -> APIC 0x0296 -> Node 10
  968.963478 (    0.007445)| [    0.000000] SRAT: PXM 10 -> APIC 0x0298 -> Node 10
  968.967337 (    0.003859)| [    0.000000] SRAT: PXM 10 -> APIC 0x029a -> Node 10
  968.971322 (    0.003985)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a0 -> Node 10
  968.979185 (    0.007863)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a2 -> Node 10
  968.984165 (    0.004980)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a4 -> Node 10
  968.988417 (    0.004252)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a6 -> Node 10
  968.996149 (    0.007732)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a8 -> Node 10
  968.999360 (    0.003211)| [    0.000000] SRAT: PXM 10 -> APIC 0x02aa -> Node 10
  969.004641 (    0.005281)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b0 -> Node 10
  969.011835 (    0.007194)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b2 -> Node 10
  969.016457 (    0.004622)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b4 -> Node 10
  969.019146 (    0.002689)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b6 -> Node 10
  969.027152 (    0.008006)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b8 -> Node 10
  969.031218 (    0.004066)| [    0.000000] SRAT: PXM 10 -> APIC 0x02ba -> Node 10
  969.035294 (    0.004076)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c0 -> Node 11
  969.043920 (    0.008626)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c2 -> Node 11
  969.048007 (    0.004087)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c4 -> Node 11
  969.051164 (    0.003157)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c6 -> Node 11
  969.059155 (    0.007991)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c8 -> Node 11
  969.063263 (    0.004108)| [    0.000000] SRAT: PXM 11 -> APIC 0x02ca -> Node 11
  969.068064 (    0.004801)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d0 -> Node 11
  969.075624 (    0.007560)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d2 -> Node 11
  969.079322 (    0.003698)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d4 -> Node 11
  969.083445 (    0.004123)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d6 -> Node 11
  969.091285 (    0.007840)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d8 -> Node 11
  969.095125 (    0.003840)| [    0.000000] SRAT: PXM 11 -> APIC 0x02da -> Node 11
  969.099355 (    0.004230)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e0 -> Node 11
  969.107047 (    0.007692)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e2 -> Node 11
  969.111269 (    0.004222)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e4 -> Node 11
  969.115101 (    0.003832)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e6 -> Node 11
  969.124922 (    0.009821)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e8 -> Node 11
  969.128031 (    0.003109)| [    0.000000] SRAT: PXM 11 -> APIC 0x02ea -> Node 11
  969.132399 (    0.004368)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f0 -> Node 11
  969.139031 (    0.006632)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f2 -> Node 11
  969.143235 (    0.004204)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f4 -> Node 11
  969.147089 (    0.003854)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f6 -> Node 11
  969.155549 (    0.008460)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f8 -> Node 11
  969.159516 (    0.003967)| [    0.000000] SRAT: PXM 11 -> APIC 0x02fa -> Node 11
  969.163347 (    0.003831)| [    0.000000] SRAT: PXM 12 -> APIC 0x0300 -> Node 12
  969.171707 (    0.008360)| [    0.000000] SRAT: PXM 12 -> APIC 0x0302 -> Node 12
  969.175165 (    0.003458)| [    0.000000] SRAT: PXM 12 -> APIC 0x0304 -> Node 12
  969.179194 (    0.004029)| [    0.000000] SRAT: PXM 12 -> APIC 0x0306 -> Node 12
  969.186964 (    0.007770)| [    0.000000] SRAT: PXM 12 -> APIC 0x0308 -> Node 12
  969.191591 (    0.004627)| [    0.000000] SRAT: PXM 12 -> APIC 0x030a -> Node 12
  969.195408 (    0.003817)| [    0.000000] SRAT: PXM 12 -> APIC 0x0310 -> Node 12
  969.204007 (    0.008599)| [    0.000000] SRAT: PXM 12 -> APIC 0x0312 -> Node 12
  969.207971 (    0.003964)| [    0.000000] SRAT: PXM 12 -> APIC 0x0314 -> Node 12
  969.210981 (    0.003010)| [    0.000000] SRAT: PXM 12 -> APIC 0x0316 -> Node 12
  969.219690 (    0.008709)| [    0.000000] SRAT: PXM 12 -> APIC 0x0318 -> Node 12
  969.224425 (    0.004735)| [    0.000000] SRAT: PXM 12 -> APIC 0x031a -> Node 12
  969.228083 (    0.003658)| [    0.000000] SRAT: PXM 12 -> APIC 0x0320 -> Node 12
  969.235775 (    0.007692)| [    0.000000] SRAT: PXM 12 -> APIC 0x0322 -> Node 12
  969.239910 (    0.004135)| [    0.000000] SRAT: PXM 12 -> APIC 0x0324 -> Node 12
  969.243443 (    0.003533)| [    0.000000] SRAT: PXM 12 -> APIC 0x0326 -> Node 12
  969.253089 (    0.009646)| [    0.000000] SRAT: PXM 12 -> APIC 0x0328 -> Node 12
  969.255461 (    0.002372)| [    0.000000] SRAT: PXM 12 -> APIC 0x032a -> Node 12
  969.259855 (    0.004394)| [    0.000000] SRAT: PXM 12 -> APIC 0x0330 -> Node 12
  969.271034 (    0.011179)| [    0.000000] SRAT: PXM 12 -> APIC 0x0332 -> Node 12
  969.273288 (    0.002254)| [    0.000000] SRAT: PXM 12 -> APIC 0x0334 -> Node 12
  969.275499 (    0.002211)| [    0.000000] SRAT: PXM 12 -> APIC 0x0336 -> Node 12
  969.283274 (    0.007775)| [    0.000000] SRAT: PXM 12 -> APIC 0x0338 -> Node 12
  969.287497 (    0.004223)| [    0.000000] SRAT: PXM 12 -> APIC 0x033a -> Node 12
  969.292277 (    0.004780)| [    0.000000] SRAT: PXM 13 -> APIC 0x0340 -> Node 13
  969.300051 (    0.007774)| [    0.000000] SRAT: PXM 13 -> APIC 0x0342 -> Node 13
  969.303851 (    0.003800)| [    0.000000] SRAT: PXM 13 -> APIC 0x0344 -> Node 13
  969.307736 (    0.003885)| [    0.000000] SRAT: PXM 13 -> APIC 0x0346 -> Node 13
  969.315319 (    0.007583)| [    0.000000] SRAT: PXM 13 -> APIC 0x0348 -> Node 13
  969.319396 (    0.004077)| [    0.000000] SRAT: PXM 13 -> APIC 0x034a -> Node 13
  969.323608 (    0.004212)| [    0.000000] SRAT: PXM 13 -> APIC 0x0350 -> Node 13
  969.332173 (    0.008565)| [    0.000000] SRAT: PXM 13 -> APIC 0x0352 -> Node 13
  969.336483 (    0.004310)| [    0.000000] SRAT: PXM 13 -> APIC 0x0354 -> Node 13
  969.340019 (    0.003536)| [    0.000000] SRAT: PXM 13 -> APIC 0x0356 -> Node 13
  969.347590 (    0.007571)| [    0.000000] SRAT: PXM 13 -> APIC 0x0358 -> Node 13
  969.351313 (    0.003723)| [    0.000000] SRAT: PXM 13 -> APIC 0x035a -> Node 13
  969.355040 (    0.003727)| [    0.000000] SRAT: PXM 13 -> APIC 0x0360 -> Node 13
  969.362961 (    0.007921)| [    0.000000] SRAT: PXM 13 -> APIC 0x0362 -> Node 13
  969.367331 (    0.004370)| [    0.000000] SRAT: PXM 13 -> APIC 0x0364 -> Node 13
  969.371640 (    0.004309)| [    0.000000] SRAT: PXM 13 -> APIC 0x0366 -> Node 13
  969.377021 (    0.005381)| [    0.000000] SRAT: PXM 13 -> APIC 0x0368 -> Node 13
  969.383286 (    0.006265)| [    0.000000] SRAT: PXM 13 -> APIC 0x036a -> Node 13
  969.389968 (    0.006682)| [    0.000000] SRAT: PXM 13 -> APIC 0x0370 -> Node 13
  969.394882 (    0.004914)| [    0.000000] SRAT: PXM 13 -> APIC 0x0372 -> Node 13
  969.399320 (    0.004438)| [    0.000000] SRAT: PXM 13 -> APIC 0x0374 -> Node 13
  969.403441 (    0.004121)| [    0.000000] SRAT: PXM 13 -> APIC 0x0376 -> Node 13
  969.411097 (    0.007656)| [    0.000000] SRAT: PXM 13 -> APIC 0x0378 -> Node 13
  969.415282 (    0.004185)| [    0.000000] SRAT: PXM 13 -> APIC 0x037a -> Node 13
  969.419257 (    0.003975)| [    0.000000] SRAT: PXM 14 -> APIC 0x0380 -> Node 14
  969.427383 (    0.008126)| [    0.000000] SRAT: PXM 14 -> APIC 0x0382 -> Node 14
  969.430963 (    0.003580)| [    0.000000] SRAT: PXM 14 -> APIC 0x0384 -> Node 14
  969.435041 (    0.004078)| [    0.000000] SRAT: PXM 14 -> APIC 0x0386 -> Node 14
  969.443396 (    0.008355)| [    0.000000] SRAT: PXM 14 -> APIC 0x0388 -> Node 14
  969.447501 (    0.004105)| [    0.000000] SRAT: PXM 14 -> APIC 0x038a -> Node 14
  969.451309 (    0.003808)| [    0.000000] SRAT: PXM 14 -> APIC 0x0390 -> Node 14
  969.459047 (    0.007738)| [    0.000000] SRAT: PXM 14 -> APIC 0x0392 -> Node 14
  969.463056 (    0.004009)| [    0.000000] SRAT: PXM 14 -> APIC 0x0394 -> Node 14
  969.466981 (    0.003925)| [    0.000000] SRAT: PXM 14 -> APIC 0x0396 -> Node 14
  969.475727 (    0.008746)| [    0.000000] SRAT: PXM 14 -> APIC 0x0398 -> Node 14
  969.479744 (    0.004017)| [    0.000000] SRAT: PXM 14 -> APIC 0x039a -> Node 14
  969.483233 (    0.003489)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a0 -> Node 14
  969.491102 (    0.007869)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a2 -> Node 14
  969.495350 (    0.004248)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a4 -> Node 14
  969.499382 (    0.004032)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a6 -> Node 14
  969.506911 (    0.007529)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a8 -> Node 14
  969.511154 (    0.004243)| [    0.000000] SRAT: PXM 14 -> APIC 0x03aa -> Node 14
  969.515413 (    0.004259)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b0 -> Node 14
  969.522974 (    0.007561)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b2 -> Node 14
  969.527366 (    0.004392)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b4 -> Node 14
  969.531184 (    0.003818)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b6 -> Node 14
  969.539638 (    0.008454)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b8 -> Node 14
  969.543093 (    0.003455)| [    0.000000] SRAT: PXM 14 -> APIC 0x03ba -> Node 14
  969.547096 (    0.004003)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c0 -> Node 15
  969.555281 (    0.008185)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c2 -> Node 15
  969.559409 (    0.004128)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c4 -> Node 15
  969.563324 (    0.003915)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c6 -> Node 15
  969.571378 (    0.008054)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c8 -> Node 15
  969.575132 (    0.003754)| [    0.000000] SRAT: PXM 15 -> APIC 0x03ca -> Node 15
  969.579182 (    0.004050)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d0 -> Node 15
  969.586892 (    0.007710)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d2 -> Node 15
  969.590902 (    0.004010)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d4 -> Node 15
  969.594942 (    0.004040)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d6 -> Node 15
  969.603895 (    0.008953)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d8 -> Node 15
  969.607554 (    0.003659)| [    0.000000] SRAT: PXM 15 -> APIC 0x03da -> Node 15
  969.611799 (    0.004245)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e0 -> Node 15
  969.618843 (    0.007044)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e2 -> Node 15
  969.623094 (    0.004251)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e4 -> Node 15
  969.626952 (    0.003858)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e6 -> Node 15
  969.635065 (    0.008113)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e8 -> Node 15
  969.639105 (    0.004040)| [    0.000000] SRAT: PXM 15 -> APIC 0x03ea -> Node 15
  969.643063 (    0.003958)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f0 -> Node 15
  969.653217 (    0.010154)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f2 -> Node 15
  969.655542 (    0.002325)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f4 -> Node 15
  969.658908 (    0.003366)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f6 -> Node 15
  969.668205 (    0.009297)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f8 -> Node 15
  969.685992 (    0.017787)| [    0.000000] SRAT: PXM 15 -> APIC 0x03fa -> Node 15
  969.686112 (    0.000120)| [    0.000000] SRAT: PXM 16 -> APIC 0x0400 -> Node 16
  969.686232 (    0.000120)| [    0.000000] SRAT: PXM 16 -> APIC 0x0402 -> Node 16
  969.687534 (    0.001302)| [    0.000000] SRAT: PXM 16 -> APIC 0x0404 -> Node 16
  969.695177 (    0.007643)| [    0.000000] SRAT: PXM 16 -> APIC 0x0406 -> Node 16
  969.698773 (    0.003596)| [    0.000000] SRAT: PXM 16 -> APIC 0x0408 -> Node 16
  969.702884 (    0.004111)| [    0.000000] SRAT: PXM 16 -> APIC 0x040a -> Node 16
  969.708336 (    0.005452)| [    0.000000] SRAT: PXM 16 -> APIC 0x0410 -> Node 16
  969.715121 (    0.006785)| [    0.000000] SRAT: PXM 16 -> APIC 0x0412 -> Node 16
  969.721682 (    0.006561)| [    0.000000] SRAT: PXM 16 -> APIC 0x0414 -> Node 16
  969.723093 (    0.001411)| [    0.000000] SRAT: PXM 16 -> APIC 0x0416 -> Node 16
  969.731198 (    0.008105)| [    0.000000] SRAT: PXM 16 -> APIC 0x0418 -> Node 16
  969.735214 (    0.004016)| [    0.000000] SRAT: PXM 16 -> APIC 0x041a -> Node 16
  969.739108 (    0.003894)| [    0.000000] SRAT: PXM 16 -> APIC 0x0420 -> Node 16
  969.748271 (    0.009163)| [    0.000000] SRAT: PXM 16 -> APIC 0x0422 -> Node 16
  969.751368 (    0.003097)| [    0.000000] SRAT: PXM 16 -> APIC 0x0424 -> Node 16
  969.756487 (    0.005119)| [    0.000000] SRAT: PXM 16 -> APIC 0x0426 -> Node 16
  969.762766 (    0.006279)| [    0.000000] SRAT: PXM 16 -> APIC 0x0428 -> Node 16
  969.766881 (    0.004115)| [    0.000000] SRAT: PXM 16 -> APIC 0x042a -> Node 16
  969.770973 (    0.004092)| [    0.000000] SRAT: PXM 16 -> APIC 0x0430 -> Node 16
  969.779013 (    0.008040)| [    0.000000] SRAT: PXM 16 -> APIC 0x0432 -> Node 16
  969.783386 (    0.004373)| [    0.000000] SRAT: PXM 16 -> APIC 0x0434 -> Node 16
  969.787567 (    0.004181)| [    0.000000] SRAT: PXM 16 -> APIC 0x0436 -> Node 16
  969.794879 (    0.007312)| [    0.000000] SRAT: PXM 16 -> APIC 0x0438 -> Node 16
  969.798927 (    0.004048)| [    0.000000] SRAT: PXM 16 -> APIC 0x043a -> Node 16
  969.803134 (    0.004207)| [    0.000000] SRAT: PXM 17 -> APIC 0x0440 -> Node 17
  969.811466 (    0.008332)| [    0.000000] SRAT: PXM 17 -> APIC 0x0442 -> Node 17
  969.814994 (    0.003528)| [    0.000000] SRAT: PXM 17 -> APIC 0x0444 -> Node 17
  969.819024 (    0.004030)| [    0.000000] SRAT: PXM 17 -> APIC 0x0446 -> Node 17
  969.827400 (    0.008376)| [    0.000000] SRAT: PXM 17 -> APIC 0x0448 -> Node 17
  969.831616 (    0.004216)| [    0.000000] SRAT: PXM 17 -> APIC 0x044a -> Node 17
  969.835335 (    0.003719)| [    0.000000] SRAT: PXM 17 -> APIC 0x0450 -> Node 17
  969.843585 (    0.008250)| [    0.000000] SRAT: PXM 17 -> APIC 0x0452 -> Node 17
  969.848222 (    0.004637)| [    0.000000] SRAT: PXM 17 -> APIC 0x0454 -> Node 17
  969.851727 (    0.003505)| [    0.000000] SRAT: PXM 17 -> APIC 0x0456 -> Node 17
  969.859441 (    0.007714)| [    0.000000] SRAT: PXM 17 -> APIC 0x0458 -> Node 17
  969.864392 (    0.004951)| [    0.000000] SRAT: PXM 17 -> APIC 0x045a -> Node 17
  969.867559 (    0.003167)| [    0.000000] SRAT: PXM 17 -> APIC 0x0460 -> Node 17
  969.875299 (    0.007740)| [    0.000000] SRAT: PXM 17 -> APIC 0x0462 -> Node 17
  969.879099 (    0.003800)| [    0.000000] SRAT: PXM 17 -> APIC 0x0464 -> Node 17
  969.884436 (    0.005337)| [    0.000000] SRAT: PXM 17 -> APIC 0x0466 -> Node 17
  969.891483 (    0.007047)| [    0.000000] SRAT: PXM 17 -> APIC 0x0468 -> Node 17
  969.894882 (    0.003399)| [    0.000000] SRAT: PXM 17 -> APIC 0x046a -> Node 17
  969.898853 (    0.003971)| [    0.000000] SRAT: PXM 17 -> APIC 0x0470 -> Node 17
  969.907760 (    0.008907)| [    0.000000] SRAT: PXM 17 -> APIC 0x0472 -> Node 17
  969.911668 (    0.003908)| [    0.000000] SRAT: PXM 17 -> APIC 0x0474 -> Node 17
  969.915358 (    0.003690)| [    0.000000] SRAT: PXM 17 -> APIC 0x0476 -> Node 17
  969.923910 (    0.008552)| [    0.000000] SRAT: PXM 17 -> APIC 0x0478 -> Node 17
  969.927300 (    0.003390)| [    0.000000] SRAT: PXM 17 -> APIC 0x047a -> Node 17
  969.935282 (    0.007982)| [    0.000000] SRAT: PXM 18 -> APIC 0x0480 -> Node 18
  969.938929 (    0.003647)| [    0.000000] SRAT: PXM 18 -> APIC 0x0482 -> Node 18
  969.943282 (    0.004353)| [    0.000000] SRAT: PXM 18 -> APIC 0x0484 -> Node 18
  969.947407 (    0.004125)| [    0.000000] SRAT: PXM 18 -> APIC 0x0486 -> Node 18
  969.954984 (    0.007577)| [    0.000000] SRAT: PXM 18 -> APIC 0x0488 -> Node 18
  969.959311 (    0.004327)| [    0.000000] SRAT: PXM 18 -> APIC 0x048a -> Node 18
  969.963448 (    0.004137)| [    0.000000] SRAT: PXM 18 -> APIC 0x0490 -> Node 18
  969.970668 (    0.007220)| [    0.000000] SRAT: PXM 18 -> APIC 0x0492 -> Node 18
  969.974984 (    0.004316)| [    0.000000] SRAT: PXM 18 -> APIC 0x0494 -> Node 18
  969.982911 (    0.007927)| [    0.000000] SRAT: PXM 18 -> APIC 0x0496 -> Node 18
  969.988451 (    0.005540)| [    0.000000] SRAT: PXM 18 -> APIC 0x0498 -> Node 18
  969.992133 (    0.003682)| [    0.000000] SRAT: PXM 18 -> APIC 0x049a -> Node 18
  969.999339 (    0.007206)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a0 -> Node 18
  970.002775 (    0.003436)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a2 -> Node 18
  970.006798 (    0.004023)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a4 -> Node 18
  970.010904 (    0.004106)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a6 -> Node 18
  970.019083 (    0.008179)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a8 -> Node 18
  970.023317 (    0.004234)| [    0.000000] SRAT: PXM 18 -> APIC 0x04aa -> Node 18
  970.029636 (    0.006319)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b0 -> Node 18
  970.035742 (    0.006106)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b2 -> Node 18
  970.039020 (    0.003278)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b4 -> Node 18
  970.047005 (    0.007985)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b6 -> Node 18
  970.050889 (    0.003884)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b8 -> Node 18
  970.054764 (    0.003875)| [    0.000000] SRAT: PXM 18 -> APIC 0x04ba -> Node 18
  970.063052 (    0.008288)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c0 -> Node 19
  970.066880 (    0.003828)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c2 -> Node 19
  970.071009 (    0.004129)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c4 -> Node 19
  970.078693 (    0.007684)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c6 -> Node 19
  970.082885 (    0.004192)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c8 -> Node 19
  970.086245 (    0.003360)| [    0.000000] SRAT: PXM 19 -> APIC 0x04ca -> Node 19
  970.094950 (    0.008705)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d0 -> Node 19
  970.098905 (    0.003955)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d2 -> Node 19
  970.103065 (    0.004160)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d4 -> Node 19
  970.111223 (    0.008158)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d6 -> Node 19
  970.114965 (    0.003742)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d8 -> Node 19
  970.119133 (    0.004168)| [    0.000000] SRAT: PXM 19 -> APIC 0x04da -> Node 19
  970.126850 (    0.007717)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e0 -> Node 19
  970.130979 (    0.004129)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e2 -> Node 19
  970.134863 (    0.003884)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e4 -> Node 19
  970.142996 (    0.008133)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e6 -> Node 19
  970.146751 (    0.003755)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e8 -> Node 19
  970.150906 (    0.004155)| [    0.000000] SRAT: PXM 19 -> APIC 0x04ea -> Node 19
  970.158644 (    0.007738)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f0 -> Node 19
  970.162821 (    0.004177)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f2 -> Node 19
  970.166713 (    0.003892)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f4 -> Node 19
  970.174746 (    0.008033)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f6 -> Node 19
  970.179303 (    0.004557)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f8 -> Node 19
  970.182716 (    0.003413)| [    0.000000] SRAT: PXM 19 -> APIC 0x04fa -> Node 19
  970.190690 (    0.007974)| [    0.000000] SRAT: PXM 20 -> APIC 0x0500 -> Node 20
  970.194795 (    0.004105)| [    0.000000] SRAT: PXM 20 -> APIC 0x0502 -> Node 20
  970.198924 (    0.004129)| [    0.000000] SRAT: PXM 20 -> APIC 0x0504 -> Node 20
  970.206671 (    0.007747)| [    0.000000] SRAT: PXM 20 -> APIC 0x0506 -> Node 20
  970.210662 (    0.003991)| [    0.000000] SRAT: PXM 20 -> APIC 0x0508 -> Node 20
  970.215019 (    0.004357)| [    0.000000] SRAT: PXM 20 -> APIC 0x050a -> Node 20
  970.223756 (    0.008737)| [    0.000000] SRAT: PXM 20 -> APIC 0x0510 -> Node 20
  970.227074 (    0.003318)| [    0.000000] SRAT: PXM 20 -> APIC 0x0512 -> Node 20
  970.231042 (    0.003968)| [    0.000000] SRAT: PXM 20 -> APIC 0x0514 -> Node 20
  970.236474 (    0.005432)| [    0.000000] SRAT: PXM 20 -> APIC 0x0516 -> Node 20
  970.242931 (    0.006457)| [    0.000000] SRAT: PXM 20 -> APIC 0x0518 -> Node 20
  970.247303 (    0.004372)| [    0.000000] SRAT: PXM 20 -> APIC 0x051a -> Node 20
  970.254972 (    0.007669)| [    0.000000] SRAT: PXM 20 -> APIC 0x0520 -> Node 20
  970.258998 (    0.004026)| [    0.000000] SRAT: PXM 20 -> APIC 0x0522 -> Node 20
  970.263172 (    0.004174)| [    0.000000] SRAT: PXM 20 -> APIC 0x0524 -> Node 20
  970.267022 (    0.003850)| [    0.000000] SRAT: PXM 20 -> APIC 0x0526 -> Node 20
  970.274906 (    0.007884)| [    0.000000] SRAT: PXM 20 -> APIC 0x0528 -> Node 20
  970.278852 (    0.003946)| [    0.000000] SRAT: PXM 20 -> APIC 0x052a -> Node 20
  970.283159 (    0.004307)| [    0.000000] SRAT: PXM 20 -> APIC 0x0530 -> Node 20
  970.291287 (    0.008128)| [    0.000000] SRAT: PXM 20 -> APIC 0x0532 -> Node 20
  970.294932 (    0.003645)| [    0.000000] SRAT: PXM 20 -> APIC 0x0534 -> Node 20
  970.303052 (    0.008120)| [    0.000000] SRAT: PXM 20 -> APIC 0x0536 -> Node 20
  970.306923 (    0.003871)| [    0.000000] SRAT: PXM 20 -> APIC 0x0538 -> Node 20
  970.310798 (    0.003875)| [    0.000000] SRAT: PXM 20 -> APIC 0x053a -> Node 20
  970.319196 (    0.008398)| [    0.000000] SRAT: PXM 21 -> APIC 0x0540 -> Node 21
  970.323031 (    0.003835)| [    0.000000] SRAT: PXM 21 -> APIC 0x0542 -> Node 21
  970.327293 (    0.004262)| [    0.000000] SRAT: PXM 21 -> APIC 0x0544 -> Node 21
  970.334816 (    0.007523)| [    0.000000] SRAT: PXM 21 -> APIC 0x0546 -> Node 21
  970.338840 (    0.004024)| [    0.000000] SRAT: PXM 21 -> APIC 0x0548 -> Node 21
  970.343152 (    0.004312)| [    0.000000] SRAT: PXM 21 -> APIC 0x054a -> Node 21
  970.350764 (    0.007612)| [    0.000000] SRAT: PXM 21 -> APIC 0x0550 -> Node 21
  970.354999 (    0.004235)| [    0.000000] SRAT: PXM 21 -> APIC 0x0552 -> Node 21
  970.358935 (    0.003936)| [    0.000000] SRAT: PXM 21 -> APIC 0x0554 -> Node 21
  970.366907 (    0.007972)| [    0.000000] SRAT: PXM 21 -> APIC 0x0556 -> Node 21
  970.371454 (    0.004547)| [    0.000000] SRAT: PXM 21 -> APIC 0x0558 -> Node 21
  970.376873 (    0.005419)| [    0.000000] SRAT: PXM 21 -> APIC 0x055a -> Node 21
  970.380021 (    0.003148)| [    0.000000] SRAT: PXM 21 -> APIC 0x0560 -> Node 21
  970.387706 (    0.007685)| [    0.000000] SRAT: PXM 21 -> APIC 0x0562 -> Node 21
  970.391189 (    0.003483)| [    0.000000] SRAT: PXM 21 -> APIC 0x0564 -> Node 21
  970.398122 (    0.006933)| [    0.000000] SRAT: PXM 21 -> APIC 0x0566 -> Node 21
  970.403650 (    0.005528)| [    0.000000] SRAT: PXM 21 -> APIC 0x0568 -> Node 21
  970.407867 (    0.004217)| [    0.000000] SRAT: PXM 21 -> APIC 0x056a -> Node 21
  970.411648 (    0.003781)| [    0.000000] SRAT: PXM 21 -> APIC 0x0570 -> Node 21
  970.419365 (    0.007717)| [    0.000000] SRAT: PXM 21 -> APIC 0x0572 -> Node 21
  970.423489 (    0.004124)| [    0.000000] SRAT: PXM 21 -> APIC 0x0574 -> Node 21
  970.427503 (    0.004014)| [    0.000000] SRAT: PXM 21 -> APIC 0x0576 -> Node 21
  970.435630 (    0.008127)| [    0.000000] SRAT: PXM 21 -> APIC 0x0578 -> Node 21
  970.439344 (    0.003714)| [    0.000000] SRAT: PXM 21 -> APIC 0x057a -> Node 21
  970.443349 (    0.004005)| [    0.000000] SRAT: PXM 22 -> APIC 0x0580 -> Node 22
  970.451005 (    0.007656)| [    0.000000] SRAT: PXM 22 -> APIC 0x0582 -> Node 22
  970.455549 (    0.004544)| [    0.000000] SRAT: PXM 22 -> APIC 0x0584 -> Node 22
  970.462932 (    0.007383)| [    0.000000] SRAT: PXM 22 -> APIC 0x0586 -> Node 22
  970.466843 (    0.003911)| [    0.000000] SRAT: PXM 22 -> APIC 0x0588 -> Node 22
  970.471023 (    0.004180)| [    0.000000] SRAT: PXM 22 -> APIC 0x058a -> Node 22
  970.478612 (    0.007589)| [    0.000000] SRAT: PXM 22 -> APIC 0x0590 -> Node 22
  970.482784 (    0.004172)| [    0.000000] SRAT: PXM 22 -> APIC 0x0592 -> Node 22
  970.486976 (    0.004192)| [    0.000000] SRAT: PXM 22 -> APIC 0x0594 -> Node 22
  970.494711 (    0.007735)| [    0.000000] SRAT: PXM 22 -> APIC 0x0596 -> Node 22
  970.498739 (    0.004028)| [    0.000000] SRAT: PXM 22 -> APIC 0x0598 -> Node 22
  970.502879 (    0.004140)| [    0.000000] SRAT: PXM 22 -> APIC 0x059a -> Node 22
  970.507054 (    0.004175)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a0 -> Node 22
  970.514949 (    0.007895)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a2 -> Node 22
  970.518946 (    0.003997)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a4 -> Node 22
  970.526851 (    0.007905)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a6 -> Node 22
  970.530890 (    0.004039)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a8 -> Node 22
  970.535280 (    0.004390)| [    0.000000] SRAT: PXM 22 -> APIC 0x05aa -> Node 22
  970.542839 (    0.007559)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b0 -> Node 22
  970.546767 (    0.003928)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b2 -> Node 22
  970.550783 (    0.004016)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b4 -> Node 22
  970.558834 (    0.008051)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b6 -> Node 22
  970.563181 (    0.004347)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b8 -> Node 22
  970.567010 (    0.003829)| [    0.000000] SRAT: PXM 22 -> APIC 0x05ba -> Node 22
  970.578300 (    0.011290)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c0 -> Node 23
  970.578419 (    0.000119)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c2 -> Node 23
  970.583197 (    0.004778)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c4 -> Node 23
  970.590769 (    0.007572)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c6 -> Node 23
  970.594914 (    0.004145)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c8 -> Node 23
  970.598947 (    0.004033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05ca -> Node 23
  970.606784 (    0.007837)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d0 -> Node 23
  970.610954 (    0.004170)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d2 -> Node 23
  970.614973 (    0.004019)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d4 -> Node 23
  970.623060 (    0.008087)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d6 -> Node 23
  970.626961 (    0.003901)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d8 -> Node 23
  970.630988 (    0.004027)| [    0.000000] SRAT: PXM 23 -> APIC 0x05da -> Node 23
  970.635012 (    0.004024)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e0 -> Node 23
  970.642778 (    0.007766)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e2 -> Node 23
  970.646764 (    0.003986)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e4 -> Node 23
  970.654816 (    0.008052)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e6 -> Node 23
  970.658993 (    0.004177)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e8 -> Node 23
  970.662973 (    0.003980)| [    0.000000] SRAT: PXM 23 -> APIC 0x05ea -> Node 23
  970.667133 (    0.004160)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f0 -> Node 23
  970.674731 (    0.007598)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f2 -> Node 23
  970.678895 (    0.004164)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f4 -> Node 23
  970.686981 (    0.008086)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f6 -> Node 23
  970.690939 (    0.003958)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f8 -> Node 23
  970.695297 (    0.004358)| [    0.000000] SRAT: PXM 23 -> APIC 0x05fa -> Node 23
  970.703107 (    0.007810)| [    0.000000] SRAT: PXM 24 -> APIC 0x0600 -> Node 24
  970.706905 (    0.003798)| [    0.000000] SRAT: PXM 24 -> APIC 0x0602 -> Node 24
  970.710763 (    0.003858)| [    0.000000] SRAT: PXM 24 -> APIC 0x0604 -> Node 24
  970.715080 (    0.004317)| [    0.000000] SRAT: PXM 24 -> APIC 0x0606 -> Node 24
  970.723126 (    0.008046)| [    0.000000] SRAT: PXM 24 -> APIC 0x0608 -> Node 24
  970.727279 (    0.004153)| [    0.000000] SRAT: PXM 24 -> APIC 0x060a -> Node 24
  970.732392 (    0.005113)| [    0.000000] SRAT: PXM 24 -> APIC 0x0610 -> Node 24
  970.739039 (    0.006647)| [    0.000000] SRAT: PXM 24 -> APIC 0x0612 -> Node 24
  970.742865 (    0.003826)| [    0.000000] SRAT: PXM 24 -> APIC 0x0614 -> Node 24
  970.750899 (    0.008034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0616 -> Node 24
  970.754809 (    0.003910)| [    0.000000] SRAT: PXM 24 -> APIC 0x0618 -> Node 24
  970.758863 (    0.004054)| [    0.000000] SRAT: PXM 24 -> APIC 0x061a -> Node 24
  970.767416 (    0.008553)| [    0.000000] SRAT: PXM 24 -> APIC 0x0620 -> Node 24
  970.770873 (    0.003457)| [    0.000000] SRAT: PXM 24 -> APIC 0x0622 -> Node 24
  970.775593 (    0.004720)| [    0.000000] SRAT: PXM 24 -> APIC 0x0624 -> Node 24
  970.783271 (    0.007678)| [    0.000000] SRAT: PXM 24 -> APIC 0x0626 -> Node 24
  970.786836 (    0.003565)| [    0.000000] SRAT: PXM 24 -> APIC 0x0628 -> Node 24
  970.790984 (    0.004148)| [    0.000000] SRAT: PXM 24 -> APIC 0x062a -> Node 24
  970.798854 (    0.007870)| [    0.000000] SRAT: PXM 24 -> APIC 0x0630 -> Node 24
  970.802884 (    0.004030)| [    0.000000] SRAT: PXM 24 -> APIC 0x0632 -> Node 24
  970.807472 (    0.004588)| [    0.000000] SRAT: PXM 24 -> APIC 0x0634 -> Node 24
  970.815091 (    0.007619)| [    0.000000] SRAT: PXM 24 -> APIC 0x0636 -> Node 24
  970.819173 (    0.004082)| [    0.000000] SRAT: PXM 24 -> APIC 0x0638 -> Node 24
  970.822685 (    0.003512)| [    0.000000] SRAT: PXM 24 -> APIC 0x063a -> Node 24
  970.830761 (    0.008076)| [    0.000000] SRAT: PXM 25 -> APIC 0x0640 -> Node 25
  970.834627 (    0.003866)| [    0.000000] SRAT: PXM 25 -> APIC 0x0642 -> Node 25
  970.838765 (    0.004138)| [    0.000000] SRAT: PXM 25 -> APIC 0x0644 -> Node 25
  970.846759 (    0.007994)| [    0.000000] SRAT: PXM 25 -> APIC 0x0646 -> Node 25
  970.850907 (    0.004148)| [    0.000000] SRAT: PXM 25 -> APIC 0x0648 -> Node 25
  970.855208 (    0.004301)| [    0.000000] SRAT: PXM 25 -> APIC 0x064a -> Node 25
  970.866394 (    0.011186)| [    0.000000] SRAT: PXM 25 -> APIC 0x0650 -> Node 25
  970.870774 (    0.004380)| [    0.000000] SRAT: PXM 25 -> APIC 0x0652 -> Node 25
  970.874568 (    0.003794)| [    0.000000] SRAT: PXM 25 -> APIC 0x0654 -> Node 25
  970.878633 (    0.004065)| [    0.000000] SRAT: PXM 25 -> APIC 0x0656 -> Node 25
  970.882856 (    0.004223)| [    0.000000] SRAT: PXM 25 -> APIC 0x0658 -> Node 25
  970.888063 (    0.005207)| [    0.000000] SRAT: PXM 25 -> APIC 0x065a -> Node 25
  970.891699 (    0.003636)| [    0.000000] SRAT: PXM 25 -> APIC 0x0660 -> Node 25
  970.898607 (    0.006908)| [    0.000000] SRAT: PXM 25 -> APIC 0x0662 -> Node 25
  970.903721 (    0.005114)| [    0.000000] SRAT: PXM 25 -> APIC 0x0664 -> Node 25
  970.911455 (    0.007734)| [    0.000000] SRAT: PXM 25 -> APIC 0x0666 -> Node 25
  970.914891 (    0.003436)| [    0.000000] SRAT: PXM 25 -> APIC 0x0668 -> Node 25
  970.919611 (    0.004720)| [    0.000000] SRAT: PXM 25 -> APIC 0x066a -> Node 25
  970.927258 (    0.007647)| [    0.000000] SRAT: PXM 25 -> APIC 0x0670 -> Node 25
  970.931479 (    0.004221)| [    0.000000] SRAT: PXM 25 -> APIC 0x0672 -> Node 25
  970.936269 (    0.004790)| [    0.000000] SRAT: PXM 25 -> APIC 0x0674 -> Node 25
  970.942856 (    0.006587)| [    0.000000] SRAT: PXM 25 -> APIC 0x0676 -> Node 25
  970.946595 (    0.003739)| [    0.000000] SRAT: PXM 25 -> APIC 0x0678 -> Node 25
  970.950691 (    0.004096)| [    0.000000] SRAT: PXM 25 -> APIC 0x067a -> Node 25
  970.959631 (    0.008940)| [    0.000000] SRAT: PXM 26 -> APIC 0x0680 -> Node 26
  970.963746 (    0.004115)| [    0.000000] SRAT: PXM 26 -> APIC 0x0682 -> Node 26
  970.967314 (    0.003568)| [    0.000000] SRAT: PXM 26 -> APIC 0x0684 -> Node 26
  970.971355 (    0.004041)| [    0.000000] SRAT: PXM 26 -> APIC 0x0686 -> Node 26
  970.980347 (    0.008992)| [    0.000000] SRAT: PXM 26 -> APIC 0x0688 -> Node 26
  970.983303 (    0.002956)| [    0.000000] SRAT: PXM 26 -> APIC 0x068a -> Node 26
  970.992098 (    0.008795)| [    0.000000] SRAT: PXM 26 -> APIC 0x0690 -> Node 26
  970.994736 (    0.002638)| [    0.000000] SRAT: PXM 26 -> APIC 0x0692 -> Node 26
  970.999437 (    0.004701)| [    0.000000] SRAT: PXM 26 -> APIC 0x0694 -> Node 26
  971.008595 (    0.009158)| [    0.000000] SRAT: PXM 26 -> APIC 0x0696 -> Node 26
  971.010868 (    0.002273)| [    0.000000] SRAT: PXM 26 -> APIC 0x0698 -> Node 26
  971.016720 (    0.005852)| [    0.000000] SRAT: PXM 26 -> APIC 0x069a -> Node 26
  971.023261 (    0.006541)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a0 -> Node 26
  971.026872 (    0.003611)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a2 -> Node 26
  971.030897 (    0.004025)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a4 -> Node 26
  971.039466 (    0.008569)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a6 -> Node 26
  971.042702 (    0.003236)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a8 -> Node 26
  971.048233 (    0.005531)| [    0.000000] SRAT: PXM 26 -> APIC 0x06aa -> Node 26
  971.055459 (    0.007226)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b0 -> Node 26
  971.058721 (    0.003262)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b2 -> Node 26
  971.063027 (    0.004306)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b4 -> Node 26
  971.071205 (    0.008178)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b6 -> Node 26
  971.075126 (    0.003921)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b8 -> Node 26
  971.080118 (    0.004992)| [    0.000000] SRAT: PXM 26 -> APIC 0x06ba -> Node 26
  971.087581 (    0.007463)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c0 -> Node 27
  971.090768 (    0.003187)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c2 -> Node 27
  971.095731 (    0.004963)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c4 -> Node 27
  971.100481 (    0.004750)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c6 -> Node 27
  971.106663 (    0.006182)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c8 -> Node 27
  971.110982 (    0.004319)| [    0.000000] SRAT: PXM 27 -> APIC 0x06ca -> Node 27
  971.115740 (    0.004758)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d0 -> Node 27
  971.123535 (    0.007795)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d2 -> Node 27
  971.127511 (    0.003976)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d4 -> Node 27
  971.132151 (    0.004640)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d6 -> Node 27
  971.139764 (    0.007613)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d8 -> Node 27
  971.142638 (    0.002874)| [    0.000000] SRAT: PXM 27 -> APIC 0x06da -> Node 27
  971.147739 (    0.005101)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e0 -> Node 27
  971.155422 (    0.007683)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e2 -> Node 27
  971.195785 (    0.040363)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e4 -> Node 27
  971.195905 (    0.000120)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e6 -> Node 27
  971.196022 (    0.000117)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e8 -> Node 27
  971.196139 (    0.000117)| [    0.000000] SRAT: PXM 27 -> APIC 0x06ea -> Node 27
  971.196255 (    0.000116)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f0 -> Node 27
  971.196371 (    0.000116)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f2 -> Node 27
  971.196488 (    0.000117)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f4 -> Node 27
  971.198674 (    0.002186)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f6 -> Node 27
  971.202745 (    0.004071)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f8 -> Node 27
  971.206889 (    0.004144)| [    0.000000] SRAT: PXM 27 -> APIC 0x06fa -> Node 27
  971.215442 (    0.008553)| [    0.000000] SRAT: PXM 28 -> APIC 0x0700 -> Node 28
  971.219136 (    0.003694)| [    0.000000] SRAT: PXM 28 -> APIC 0x0702 -> Node 28
  971.223076 (    0.003940)| [    0.000000] SRAT: PXM 28 -> APIC 0x0704 -> Node 28
  971.230613 (    0.007537)| [    0.000000] SRAT: PXM 28 -> APIC 0x0706 -> Node 28
  971.234811 (    0.004198)| [    0.000000] SRAT: PXM 28 -> APIC 0x0708 -> Node 28
  971.238781 (    0.003970)| [    0.000000] SRAT: PXM 28 -> APIC 0x070a -> Node 28
  971.247723 (    0.008942)| [    0.000000] SRAT: PXM 28 -> APIC 0x0710 -> Node 28
  971.250940 (    0.003217)| [    0.000000] SRAT: PXM 28 -> APIC 0x0712 -> Node 28
  971.255077 (    0.004137)| [    0.000000] SRAT: PXM 28 -> APIC 0x0714 -> Node 28
  971.263115 (    0.008038)| [    0.000000] SRAT: PXM 28 -> APIC 0x0716 -> Node 28
  971.270921 (    0.007806)| [    0.000000] SRAT: PXM 28 -> APIC 0x0718 -> Node 28
  971.271748 (    0.000827)| [    0.000000] SRAT: PXM 28 -> APIC 0x071a -> Node 28
  971.278923 (    0.007175)| [    0.000000] SRAT: PXM 28 -> APIC 0x0720 -> Node 28
  971.282803 (    0.003880)| [    0.000000] SRAT: PXM 28 -> APIC 0x0722 -> Node 28
  971.287121 (    0.004318)| [    0.000000] SRAT: PXM 28 -> APIC 0x0724 -> Node 28
  971.294980 (    0.007859)| [    0.000000] SRAT: PXM 28 -> APIC 0x0726 -> Node 28
  971.299116 (    0.004136)| [    0.000000] SRAT: PXM 28 -> APIC 0x0728 -> Node 28
  971.302933 (    0.003817)| [    0.000000] SRAT: PXM 28 -> APIC 0x072a -> Node 28
  971.311131 (    0.008198)| [    0.000000] SRAT: PXM 28 -> APIC 0x0730 -> Node 28
  971.315382 (    0.004251)| [    0.000000] SRAT: PXM 28 -> APIC 0x0732 -> Node 28
  971.318748 (    0.003366)| [    0.000000] SRAT: PXM 28 -> APIC 0x0734 -> Node 28
  971.327764 (    0.009016)| [    0.000000] SRAT: PXM 28 -> APIC 0x0736 -> Node 28
  971.330887 (    0.003123)| [    0.000000] SRAT: PXM 28 -> APIC 0x0738 -> Node 28
  971.335252 (    0.004365)| [    0.000000] SRAT: PXM 28 -> APIC 0x073a -> Node 28
  971.342666 (    0.007414)| [    0.000000] SRAT: PXM 29 -> APIC 0x0740 -> Node 29
  971.346924 (    0.004258)| [    0.000000] SRAT: PXM 29 -> APIC 0x0742 -> Node 29
  971.350662 (    0.003738)| [    0.000000] SRAT: PXM 29 -> APIC 0x0744 -> Node 29
  971.358774 (    0.008112)| [    0.000000] SRAT: PXM 29 -> APIC 0x0746 -> Node 29
  971.362919 (    0.004145)| [    0.000000] SRAT: PXM 29 -> APIC 0x0748 -> Node 29
  971.367410 (    0.004491)| [    0.000000] SRAT: PXM 29 -> APIC 0x074a -> Node 29
  971.376844 (    0.009434)| [    0.000000] SRAT: PXM 29 -> APIC 0x0750 -> Node 29
  971.378774 (    0.001930)| [    0.000000] SRAT: PXM 29 -> APIC 0x0752 -> Node 29
  971.386784 (    0.008010)| [    0.000000] SRAT: PXM 29 -> APIC 0x0754 -> Node 29
  971.390995 (    0.004211)| [    0.000000] SRAT: PXM 29 -> APIC 0x0756 -> Node 29
  971.394999 (    0.004004)| [    0.000000] SRAT: PXM 29 -> APIC 0x0758 -> Node 29
  971.399099 (    0.004100)| [    0.000000] SRAT: PXM 29 -> APIC 0x075a -> Node 29
  971.407890 (    0.008791)| [    0.000000] SRAT: PXM 29 -> APIC 0x0760 -> Node 29
  971.411234 (    0.003344)| [    0.000000] SRAT: PXM 29 -> APIC 0x0762 -> Node 29
  971.415019 (    0.003785)| [    0.000000] SRAT: PXM 29 -> APIC 0x0764 -> Node 29
  971.422845 (    0.007826)| [    0.000000] SRAT: PXM 29 -> APIC 0x0766 -> Node 29
  971.427587 (    0.004742)| [    0.000000] SRAT: PXM 29 -> APIC 0x0768 -> Node 29
  971.431313 (    0.003726)| [    0.000000] SRAT: PXM 29 -> APIC 0x076a -> Node 29
  971.438969 (    0.007656)| [    0.000000] SRAT: PXM 29 -> APIC 0x0770 -> Node 29
  971.443510 (    0.004541)| [    0.000000] SRAT: PXM 29 -> APIC 0x0772 -> Node 29
  971.446770 (    0.003260)| [    0.000000] SRAT: PXM 29 -> APIC 0x0774 -> Node 29
  971.454813 (    0.008043)| [    0.000000] SRAT: PXM 29 -> APIC 0x0776 -> Node 29
  971.458676 (    0.003863)| [    0.000000] SRAT: PXM 29 -> APIC 0x0778 -> Node 29
  971.462810 (    0.004134)| [    0.000000] SRAT: PXM 29 -> APIC 0x077a -> Node 29
  971.471336 (    0.008526)| [    0.000000] SRAT: PXM 30 -> APIC 0x0780 -> Node 30
  971.474582 (    0.003246)| [    0.000000] SRAT: PXM 30 -> APIC 0x0782 -> Node 30
  971.478655 (    0.004073)| [    0.000000] SRAT: PXM 30 -> APIC 0x0784 -> Node 30
  971.486637 (    0.007982)| [    0.000000] SRAT: PXM 30 -> APIC 0x0786 -> Node 30
  971.490938 (    0.004301)| [    0.000000] SRAT: PXM 30 -> APIC 0x0788 -> Node 30
  971.494667 (    0.003729)| [    0.000000] SRAT: PXM 30 -> APIC 0x078a -> Node 30
  971.502678 (    0.008011)| [    0.000000] SRAT: PXM 30 -> APIC 0x0790 -> Node 30
  971.507546 (    0.004868)| [    0.000000] SRAT: PXM 30 -> APIC 0x0792 -> Node 30
  971.510667 (    0.003121)| [    0.000000] SRAT: PXM 30 -> APIC 0x0794 -> Node 30
  971.519145 (    0.008478)| [    0.000000] SRAT: PXM 30 -> APIC 0x0796 -> Node 30
  971.523032 (    0.003887)| [    0.000000] SRAT: PXM 30 -> APIC 0x0798 -> Node 30
  971.527535 (    0.004503)| [    0.000000] SRAT: PXM 30 -> APIC 0x079a -> Node 30
  971.535326 (    0.007791)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a0 -> Node 30
  971.539156 (    0.003830)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a2 -> Node 30
  971.543513 (    0.004357)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a4 -> Node 30
  971.551378 (    0.007865)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a6 -> Node 30
  971.555325 (    0.003947)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a8 -> Node 30
  971.560332 (    0.005007)| [    0.000000] SRAT: PXM 30 -> APIC 0x07aa -> Node 30
  971.566876 (    0.006544)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b0 -> Node 30
  971.570899 (    0.004023)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b2 -> Node 30
  971.574722 (    0.003823)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b4 -> Node 30
  971.583248 (    0.008526)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b6 -> Node 30
  971.586786 (    0.003538)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b8 -> Node 30
  971.590787 (    0.004001)| [    0.000000] SRAT: PXM 30 -> APIC 0x07ba -> Node 30
  971.599126 (    0.008339)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c0 -> Node 31
  971.602727 (    0.003601)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c2 -> Node 31
  971.608010 (    0.005283)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c4 -> Node 31
  971.614684 (    0.006674)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c6 -> Node 31
  971.618761 (    0.004077)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c8 -> Node 31
  971.622687 (    0.003926)| [    0.000000] SRAT: PXM 31 -> APIC 0x07ca -> Node 31
  971.630858 (    0.008171)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d0 -> Node 31
  971.634915 (    0.004057)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d2 -> Node 31
  971.638688 (    0.003773)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d4 -> Node 31
  971.647152 (    0.008464)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d6 -> Node 31
  971.651315 (    0.004163)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d8 -> Node 31
  971.655116 (    0.003801)| [    0.000000] SRAT: PXM 31 -> APIC 0x07da -> Node 31
  971.662268 (    0.007152)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e0 -> Node 31
  971.666773 (    0.004505)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e2 -> Node 31
  971.670673 (    0.003900)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e4 -> Node 31
  971.678595 (    0.007922)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e6 -> Node 31
  971.682787 (    0.004192)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e8 -> Node 31
  971.686607 (    0.003820)| [    0.000000] SRAT: PXM 31 -> APIC 0x07ea -> Node 31
  971.694534 (    0.007927)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f0 -> Node 31
  971.698959 (    0.004425)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f2 -> Node 31
  971.702637 (    0.003678)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f4 -> Node 31
  971.710894 (    0.008257)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f6 -> Node 31
  971.714675 (    0.003781)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f8 -> Node 31
  971.718765 (    0.004090)| [    0.000000] SRAT: PXM 31 -> APIC 0x07fa -> Node 31
  971.726622 (    0.007857)| [    0.000000] SRAT: PXM 0 -> APIC 0x0001 -> Node 0
  971.730837 (    0.004215)| [    0.000000] SRAT: PXM 0 -> APIC 0x0003 -> Node 0
  971.734614 (    0.003777)| [    0.000000] SRAT: PXM 0 -> APIC 0x0005 -> Node 0
  971.738661 (    0.004047)| [    0.000000] SRAT: PXM 0 -> APIC 0x0007 -> Node 0
  971.746716 (    0.008055)| [    0.000000] SRAT: PXM 0 -> APIC 0x0009 -> Node 0
  971.750819 (    0.004103)| [    0.000000] SRAT: PXM 0 -> APIC 0x000b -> Node 0
  971.755086 (    0.004267)| [    0.000000] SRAT: PXM 0 -> APIC 0x0011 -> Node 0
  971.762531 (    0.007445)| [    0.000000] SRAT: PXM 0 -> APIC 0x0013 -> Node 0
  971.766693 (    0.004162)| [    0.000000] SRAT: PXM 0 -> APIC 0x0015 -> Node 0
  971.770752 (    0.004059)| [    0.000000] SRAT: PXM 0 -> APIC 0x0017 -> Node 0
  971.774799 (    0.004047)| [    0.000000] SRAT: PXM 0 -> APIC 0x0019 -> Node 0
  971.782976 (    0.008177)| [    0.000000] SRAT: PXM 0 -> APIC 0x001b -> Node 0
  971.786607 (    0.003631)| [    0.000000] SRAT: PXM 0 -> APIC 0x0021 -> Node 0
  971.790635 (    0.004028)| [    0.000000] SRAT: PXM 0 -> APIC 0x0023 -> Node 0
  971.798660 (    0.008025)| [    0.000000] SRAT: PXM 0 -> APIC 0x0025 -> Node 0
  971.802673 (    0.004013)| [    0.000000] SRAT: PXM 0 -> APIC 0x0027 -> Node 0
  971.806848 (    0.004175)| [    0.000000] SRAT: PXM 0 -> APIC 0x0029 -> Node 0
  971.810619 (    0.003771)| [    0.000000] SRAT: PXM 0 -> APIC 0x002b -> Node 0
  971.818545 (    0.007926)| [    0.000000] SRAT: PXM 0 -> APIC 0x0031 -> Node 0
  971.822689 (    0.004144)| [    0.000000] SRAT: PXM 0 -> APIC 0x0033 -> Node 0
  971.826846 (    0.004157)| [    0.000000] SRAT: PXM 0 -> APIC 0x0035 -> Node 0
  971.834862 (    0.008016)| [    0.000000] SRAT: PXM 0 -> APIC 0x0037 -> Node 0
  971.838710 (    0.003848)| [    0.000000] SRAT: PXM 0 -> APIC 0x0039 -> Node 0
  971.842942 (    0.004232)| [    0.000000] SRAT: PXM 0 -> APIC 0x003b -> Node 0
  971.846685 (    0.003743)| [    0.000000] SRAT: PXM 1 -> APIC 0x0041 -> Node 1
  971.854615 (    0.007930)| [    0.000000] SRAT: PXM 1 -> APIC 0x0043 -> Node 1
  971.859225 (    0.004610)| [    0.000000] SRAT: PXM 1 -> APIC 0x0045 -> Node 1
  971.863057 (    0.003832)| [    0.000000] SRAT: PXM 1 -> APIC 0x0047 -> Node 1
  971.870762 (    0.007705)| [    0.000000] SRAT: PXM 1 -> APIC 0x0049 -> Node 1
  971.874847 (    0.004085)| [    0.000000] SRAT: PXM 1 -> APIC 0x004b -> Node 1
  971.878793 (    0.003946)| [    0.000000] SRAT: PXM 1 -> APIC 0x0051 -> Node 1
  971.886810 (    0.008017)| [    0.000000] SRAT: PXM 1 -> APIC 0x0053 -> Node 1
  971.890587 (    0.003777)| [    0.000000] SRAT: PXM 1 -> APIC 0x0055 -> Node 1
  971.895115 (    0.004528)| [    0.000000] SRAT: PXM 1 -> APIC 0x0057 -> Node 1
  971.898959 (    0.003844)| [    0.000000] SRAT: PXM 1 -> APIC 0x0059 -> Node 1
  971.907264 (    0.008305)| [    0.000000] SRAT: PXM 1 -> APIC 0x005b -> Node 1
  971.911213 (    0.003949)| [    0.000000] SRAT: PXM 1 -> APIC 0x0061 -> Node 1
  971.914766 (    0.003553)| [    0.000000] SRAT: PXM 1 -> APIC 0x0063 -> Node 1
  971.918932 (    0.004166)| [    0.000000] SRAT: PXM 1 -> APIC 0x0065 -> Node 1
  971.926613 (    0.007681)| [    0.000000] SRAT: PXM 1 -> APIC 0x0067 -> Node 1
  971.931997 (    0.005384)| [    0.000000] SRAT: PXM 1 -> APIC 0x0069 -> Node 1
  971.935101 (    0.003104)| [    0.000000] SRAT: PXM 1 -> APIC 0x006b -> Node 1
  971.942977 (    0.007876)| [    0.000000] SRAT: PXM 1 -> APIC 0x0071 -> Node 1
  971.946880 (    0.003903)| [    0.000000] SRAT: PXM 1 -> APIC 0x0073 -> Node 1
  971.951169 (    0.004289)| [    0.000000] SRAT: PXM 1 -> APIC 0x0075 -> Node 1
  971.954605 (    0.003436)| [    0.000000] SRAT: PXM 1 -> APIC 0x0077 -> Node 1
  971.962653 (    0.008048)| [    0.000000] SRAT: PXM 1 -> APIC 0x0079 -> Node 1
  971.966714 (    0.004061)| [    0.000000] SRAT: PXM 1 -> APIC 0x007b -> Node 1
  971.971777 (    0.005063)| [    0.000000] SRAT: PXM 2 -> APIC 0x0081 -> Node 2
  971.978645 (    0.006868)| [    0.000000] SRAT: PXM 2 -> APIC 0x0083 -> Node 2
  971.982974 (    0.004329)| [    0.000000] SRAT: PXM 2 -> APIC 0x0085 -> Node 2
  971.987682 (    0.004708)| [    0.000000] SRAT: PXM 2 -> APIC 0x0087 -> Node 2
  971.991508 (    0.003826)| [    0.000000] SRAT: PXM 2 -> APIC 0x0089 -> Node 2
  971.998652 (    0.007144)| [    0.000000] SRAT: PXM 2 -> APIC 0x008b -> Node 2
  972.002621 (    0.003969)| [    0.000000] SRAT: PXM 2 -> APIC 0x0091 -> Node 2
  972.006846 (    0.004225)| [    0.000000] SRAT: PXM 2 -> APIC 0x0093 -> Node 2
  972.014885 (    0.008039)| [    0.000000] SRAT: PXM 2 -> APIC 0x0095 -> Node 2
  972.018689 (    0.003804)| [    0.000000] SRAT: PXM 2 -> APIC 0x0097 -> Node 2
  972.022797 (    0.004108)| [    0.000000] SRAT: PXM 2 -> APIC 0x0099 -> Node 2
  972.026992 (    0.004195)| [    0.000000] SRAT: PXM 2 -> APIC 0x009b -> Node 2
  972.034957 (    0.007965)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a1 -> Node 2
  972.038629 (    0.003672)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a3 -> Node 2
  972.042671 (    0.004042)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a5 -> Node 2
  972.050563 (    0.007892)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a7 -> Node 2
  972.054717 (    0.004154)| [    0.000000] SRAT: PXM 2 -> APIC 0x00a9 -> Node 2
  972.058836 (    0.004119)| [    0.000000] SRAT: PXM 2 -> APIC 0x00ab -> Node 2
  972.062719 (    0.003883)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b1 -> Node 2
  972.070889 (    0.008170)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b3 -> Node 2
  972.074614 (    0.003725)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b5 -> Node 2
  972.078991 (    0.004377)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b7 -> Node 2
  972.082933 (    0.003942)| [    0.000000] SRAT: PXM 2 -> APIC 0x00b9 -> Node 2
  972.090736 (    0.007803)| [    0.000000] SRAT: PXM 2 -> APIC 0x00bb -> Node 2
  972.095005 (    0.004269)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c1 -> Node 3
  972.099150 (    0.004145)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c3 -> Node 3
  972.107092 (    0.007942)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c5 -> Node 3
  972.111250 (    0.004158)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c7 -> Node 3
  972.114960 (    0.003710)| [    0.000000] SRAT: PXM 3 -> APIC 0x00c9 -> Node 3
  972.119134 (    0.004174)| [    0.000000] SRAT: PXM 3 -> APIC 0x00cb -> Node 3
  972.126681 (    0.007547)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d1 -> Node 3
  972.130818 (    0.004137)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d3 -> Node 3
  972.134808 (    0.003990)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d5 -> Node 3
  972.142978 (    0.008170)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d7 -> Node 3
  972.146720 (    0.003742)| [    0.000000] SRAT: PXM 3 -> APIC 0x00d9 -> Node 3
  972.150697 (    0.003977)| [    0.000000] SRAT: PXM 3 -> APIC 0x00db -> Node 3
  972.154707 (    0.004010)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e1 -> Node 3
  972.162673 (    0.007966)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e3 -> Node 3
  972.167638 (    0.004965)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e5 -> Node 3
  972.170779 (    0.003141)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e7 -> Node 3
  972.178751 (    0.007972)| [    0.000000] SRAT: PXM 3 -> APIC 0x00e9 -> Node 3
  972.182727 (    0.003976)| [    0.000000] SRAT: PXM 3 -> APIC 0x00eb -> Node 3
  972.186831 (    0.004104)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f1 -> Node 3
  972.190734 (    0.003903)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f3 -> Node 3
  972.198655 (    0.007921)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f5 -> Node 3
  972.202687 (    0.004032)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f7 -> Node 3
  972.206703 (    0.004016)| [    0.000000] SRAT: PXM 3 -> APIC 0x00f9 -> Node 3
  972.214951 (    0.008248)| [    0.000000] SRAT: PXM 3 -> APIC 0x00fb -> Node 3
  972.219036 (    0.004085)| [    0.000000] SRAT: PXM 4 -> APIC 0x0101 -> Node 4
  972.223002 (    0.003966)| [    0.000000] SRAT: PXM 4 -> APIC 0x0103 -> Node 4
  972.226909 (    0.003907)| [    0.000000] SRAT: PXM 4 -> APIC 0x0105 -> Node 4
  972.234816 (    0.007907)| [    0.000000] SRAT: PXM 4 -> APIC 0x0107 -> Node 4
  972.238845 (    0.004029)| [    0.000000] SRAT: PXM 4 -> APIC 0x0109 -> Node 4
  972.242882 (    0.004037)| [    0.000000] SRAT: PXM 4 -> APIC 0x010b -> Node 4
  972.250936 (    0.008054)| [    0.000000] SRAT: PXM 4 -> APIC 0x0111 -> Node 4
  972.255074 (    0.004138)| [    0.000000] SRAT: PXM 4 -> APIC 0x0113 -> Node 4
  972.259008 (    0.003934)| [    0.000000] SRAT: PXM 4 -> APIC 0x0115 -> Node 4
  972.263016 (    0.004008)| [    0.000000] SRAT: PXM 4 -> APIC 0x0117 -> Node 4
  972.270820 (    0.007804)| [    0.000000] SRAT: PXM 4 -> APIC 0x0119 -> Node 4
  972.274893 (    0.004073)| [    0.000000] SRAT: PXM 4 -> APIC 0x011b -> Node 4
  972.278834 (    0.003941)| [    0.000000] SRAT: PXM 4 -> APIC 0x0121 -> Node 4
  972.286664 (    0.007830)| [    0.000000] SRAT: PXM 4 -> APIC 0x0123 -> Node 4
  972.290689 (    0.004025)| [    0.000000] SRAT: PXM 4 -> APIC 0x0125 -> Node 4
  972.294780 (    0.004091)| [    0.000000] SRAT: PXM 4 -> APIC 0x0127 -> Node 4
  972.298853 (    0.004073)| [    0.000000] SRAT: PXM 4 -> APIC 0x0129 -> Node 4
  972.306670 (    0.007817)| [    0.000000] SRAT: PXM 4 -> APIC 0x012b -> Node 4
  972.310707 (    0.004037)| [    0.000000] SRAT: PXM 4 -> APIC 0x0131 -> Node 4
  972.317269 (    0.006562)| [    0.000000] SRAT: PXM 4 -> APIC 0x0133 -> Node 4
  972.322894 (    0.005625)| [    0.000000] SRAT: PXM 4 -> APIC 0x0135 -> Node 4
  972.327314 (    0.004420)| [    0.000000] SRAT: PXM 4 -> APIC 0x0137 -> Node 4
  972.330842 (    0.003528)| [    0.000000] SRAT: PXM 4 -> APIC 0x0139 -> Node 4
  972.334978 (    0.004136)| [    0.000000] SRAT: PXM 4 -> APIC 0x013b -> Node 4
  972.342918 (    0.007940)| [    0.000000] SRAT: PXM 5 -> APIC 0x0141 -> Node 5
  972.346799 (    0.003881)| [    0.000000] SRAT: PXM 5 -> APIC 0x0143 -> Node 5
  972.350951 (    0.004152)| [    0.000000] SRAT: PXM 5 -> APIC 0x0145 -> Node 5
  972.358777 (    0.007826)| [    0.000000] SRAT: PXM 5 -> APIC 0x0147 -> Node 5
  972.362861 (    0.004084)| [    0.000000] SRAT: PXM 5 -> APIC 0x0149 -> Node 5
  972.366941 (    0.004080)| [    0.000000] SRAT: PXM 5 -> APIC 0x014b -> Node 5
  972.371236 (    0.004295)| [    0.000000] SRAT: PXM 5 -> APIC 0x0151 -> Node 5
  972.376178 (    0.004942)| [    0.000000] SRAT: PXM 5 -> APIC 0x0153 -> Node 5
  972.386731 (    0.010553)| [    0.000000] SRAT: PXM 5 -> APIC 0x0155 -> Node 5
  972.387544 (    0.000813)| [    0.000000] SRAT: PXM 5 -> APIC 0x0157 -> Node 5
  972.394900 (    0.007356)| [    0.000000] SRAT: PXM 5 -> APIC 0x0159 -> Node 5
  972.399522 (    0.004622)| [    0.000000] SRAT: PXM 5 -> APIC 0x015b -> Node 5
  972.403658 (    0.004136)| [    0.000000] SRAT: PXM 5 -> APIC 0x0161 -> Node 5
  972.406412 (    0.002754)| [    0.000000] SRAT: PXM 5 -> APIC 0x0163 -> Node 5
  972.415224 (    0.008812)| [    0.000000] SRAT: PXM 5 -> APIC 0x0165 -> Node 5
  972.419291 (    0.004067)| [    0.000000] SRAT: PXM 5 -> APIC 0x0167 -> Node 5
  972.423240 (    0.003949)| [    0.000000] SRAT: PXM 5 -> APIC 0x0169 -> Node 5
  972.431046 (    0.007806)| [    0.000000] SRAT: PXM 5 -> APIC 0x016b -> Node 5
  972.434844 (    0.003798)| [    0.000000] SRAT: PXM 5 -> APIC 0x0171 -> Node 5
  972.438827 (    0.003983)| [    0.000000] SRAT: PXM 5 -> APIC 0x0173 -> Node 5
  972.442270 (    0.003443)| [    0.000000] SRAT: PXM 5 -> APIC 0x0175 -> Node 5
  972.450968 (    0.008698)| [    0.000000] SRAT: PXM 5 -> APIC 0x0177 -> Node 5
  972.454771 (    0.003803)| [    0.000000] SRAT: PXM 5 -> APIC 0x0179 -> Node 5
  972.458785 (    0.004014)| [    0.000000] SRAT: PXM 5 -> APIC 0x017b -> Node 5
  972.466711 (    0.007926)| [    0.000000] SRAT: PXM 6 -> APIC 0x0181 -> Node 6
  972.470683 (    0.003972)| [    0.000000] SRAT: PXM 6 -> APIC 0x0183 -> Node 6
  972.474881 (    0.004198)| [    0.000000] SRAT: PXM 6 -> APIC 0x0185 -> Node 6
  972.479118 (    0.004237)| [    0.000000] SRAT: PXM 6 -> APIC 0x0187 -> Node 6
  972.486745 (    0.007627)| [    0.000000] SRAT: PXM 6 -> APIC 0x0189 -> Node 6
  972.491555 (    0.004810)| [    0.000000] SRAT: PXM 6 -> APIC 0x018b -> Node 6
  972.494892 (    0.003337)| [    0.000000] SRAT: PXM 6 -> APIC 0x0191 -> Node 6
  972.502759 (    0.007867)| [    0.000000] SRAT: PXM 6 -> APIC 0x0193 -> Node 6
  972.506721 (    0.003962)| [    0.000000] SRAT: PXM 6 -> APIC 0x0195 -> Node 6
  972.510872 (    0.004151)| [    0.000000] SRAT: PXM 6 -> APIC 0x0197 -> Node 6
  972.514979 (    0.004107)| [    0.000000] SRAT: PXM 6 -> APIC 0x0199 -> Node 6
  972.523328 (    0.008349)| [    0.000000] SRAT: PXM 6 -> APIC 0x019b -> Node 6
  972.527531 (    0.004203)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a1 -> Node 6
  972.531244 (    0.003713)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a3 -> Node 6
  972.538699 (    0.007455)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a5 -> Node 6
  972.542853 (    0.004154)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a7 -> Node 6
  972.546806 (    0.003953)| [    0.000000] SRAT: PXM 6 -> APIC 0x01a9 -> Node 6
  972.551154 (    0.004348)| [    0.000000] SRAT: PXM 6 -> APIC 0x01ab -> Node 6
  972.558976 (    0.007822)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b1 -> Node 6
  972.564040 (    0.005064)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b3 -> Node 6
  972.567914 (    0.003874)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b5 -> Node 6
  972.574717 (    0.006803)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b7 -> Node 6
  972.578707 (    0.003990)| [    0.000000] SRAT: PXM 6 -> APIC 0x01b9 -> Node 6
  972.583415 (    0.004708)| [    0.000000] SRAT: PXM 6 -> APIC 0x01bb -> Node 6
  972.586779 (    0.003364)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c1 -> Node 7
  972.594791 (    0.008012)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c3 -> Node 7
  972.601073 (    0.006282)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c5 -> Node 7
  972.603134 (    0.002061)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c7 -> Node 7
  972.610713 (    0.007579)| [    0.000000] SRAT: PXM 7 -> APIC 0x01c9 -> Node 7
  972.614818 (    0.004105)| [    0.000000] SRAT: PXM 7 -> APIC 0x01cb -> Node 7
  972.619300 (    0.004482)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d1 -> Node 7
  972.622745 (    0.003445)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d3 -> Node 7
  972.631010 (    0.008265)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d5 -> Node 7
  972.635103 (    0.004093)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d7 -> Node 7
  972.638934 (    0.003831)| [    0.000000] SRAT: PXM 7 -> APIC 0x01d9 -> Node 7
  972.647778 (    0.008844)| [    0.000000] SRAT: PXM 7 -> APIC 0x01db -> Node 7
  972.651036 (    0.003258)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e1 -> Node 7
  972.655004 (    0.003968)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e3 -> Node 7
  972.659572 (    0.004568)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e5 -> Node 7
  972.667601 (    0.008029)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e7 -> Node 7
  972.671573 (    0.003972)| [    0.000000] SRAT: PXM 7 -> APIC 0x01e9 -> Node 7
  972.675812 (    0.004239)| [    0.000000] SRAT: PXM 7 -> APIC 0x01eb -> Node 7
  972.683210 (    0.007398)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f1 -> Node 7
  972.687075 (    0.003865)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f3 -> Node 7
  972.690799 (    0.003724)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f5 -> Node 7
  972.695063 (    0.004264)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f7 -> Node 7
  972.702890 (    0.007827)| [    0.000000] SRAT: PXM 7 -> APIC 0x01f9 -> Node 7
  972.706789 (    0.003899)| [    0.000000] SRAT: PXM 7 -> APIC 0x01fb -> Node 7
  972.711475 (    0.004686)| [    0.000000] SRAT: PXM 8 -> APIC 0x0201 -> Node 8
  972.718964 (    0.007489)| [    0.000000] SRAT: PXM 8 -> APIC 0x0203 -> Node 8
  972.723232 (    0.004268)| [    0.000000] SRAT: PXM 8 -> APIC 0x0205 -> Node 8
  972.726758 (    0.003526)| [    0.000000] SRAT: PXM 8 -> APIC 0x0207 -> Node 8
  972.730777 (    0.004019)| [    0.000000] SRAT: PXM 8 -> APIC 0x0209 -> Node 8
  972.738830 (    0.008053)| [    0.000000] SRAT: PXM 8 -> APIC 0x020b -> Node 8
  972.743733 (    0.004903)| [    0.000000] SRAT: PXM 8 -> APIC 0x0211 -> Node 8
  972.747624 (    0.003891)| [    0.000000] SRAT: PXM 8 -> APIC 0x0213 -> Node 8
  972.756505 (    0.008881)| [    0.000000] SRAT: PXM 8 -> APIC 0x0215 -> Node 8
  972.758831 (    0.002326)| [    0.000000] SRAT: PXM 8 -> APIC 0x0217 -> Node 8
  972.763079 (    0.004248)| [    0.000000] SRAT: PXM 8 -> APIC 0x0219 -> Node 8
  972.766798 (    0.003719)| [    0.000000] SRAT: PXM 8 -> APIC 0x021b -> Node 8
  972.774780 (    0.007982)| [    0.000000] SRAT: PXM 8 -> APIC 0x0221 -> Node 8
  972.778866 (    0.004086)| [    0.000000] SRAT: PXM 8 -> APIC 0x0223 -> Node 8
  972.782954 (    0.004088)| [    0.000000] SRAT: PXM 8 -> APIC 0x0225 -> Node 8
  972.792569 (    0.009615)| [    0.000000] SRAT: PXM 8 -> APIC 0x0227 -> Node 8
  972.794828 (    0.002259)| [    0.000000] SRAT: PXM 8 -> APIC 0x0229 -> Node 8
  972.798922 (    0.004094)| [    0.000000] SRAT: PXM 8 -> APIC 0x022b -> Node 8
  972.803151 (    0.004229)| [    0.000000] SRAT: PXM 8 -> APIC 0x0231 -> Node 8
  972.810697 (    0.007546)| [    0.000000] SRAT: PXM 8 -> APIC 0x0233 -> Node 8
  972.814831 (    0.004134)| [    0.000000] SRAT: PXM 8 -> APIC 0x0235 -> Node 8
  972.818937 (    0.004106)| [    0.000000] SRAT: PXM 8 -> APIC 0x0237 -> Node 8
  972.822953 (    0.004016)| [    0.000000] SRAT: PXM 8 -> APIC 0x0239 -> Node 8
  972.831538 (    0.008585)| [    0.000000] SRAT: PXM 8 -> APIC 0x023b -> Node 8
  972.838819 (    0.007281)| [    0.000000] SRAT: PXM 9 -> APIC 0x0241 -> Node 9
  972.839842 (    0.001023)| [    0.000000] SRAT: PXM 9 -> APIC 0x0243 -> Node 9
  972.846843 (    0.007001)| [    0.000000] SRAT: PXM 9 -> APIC 0x0245 -> Node 9
  972.850831 (    0.003988)| [    0.000000] SRAT: PXM 9 -> APIC 0x0247 -> Node 9
  972.855064 (    0.004233)| [    0.000000] SRAT: PXM 9 -> APIC 0x0249 -> Node 9
  972.858999 (    0.003935)| [    0.000000] SRAT: PXM 9 -> APIC 0x024b -> Node 9
  972.867388 (    0.008389)| [    0.000000] SRAT: PXM 9 -> APIC 0x0251 -> Node 9
  972.871642 (    0.004254)| [    0.000000] SRAT: PXM 9 -> APIC 0x0253 -> Node 9
  972.876033 (    0.004391)| [    0.000000] SRAT: PXM 9 -> APIC 0x0255 -> Node 9
  972.886842 (    0.010809)| [    0.000000] SRAT: PXM 9 -> APIC 0x0257 -> Node 9
  972.887576 (    0.000734)| [    0.000000] SRAT: PXM 9 -> APIC 0x0259 -> Node 9
  972.891229 (    0.003653)| [    0.000000] SRAT: PXM 9 -> APIC 0x025b -> Node 9
  972.895297 (    0.004068)| [    0.000000] SRAT: PXM 9 -> APIC 0x0261 -> Node 9
  972.903546 (    0.008249)| [    0.000000] SRAT: PXM 9 -> APIC 0x0263 -> Node 9
  972.906952 (    0.003406)| [    0.000000] SRAT: PXM 9 -> APIC 0x0265 -> Node 9
  972.911603 (    0.004651)| [    0.000000] SRAT: PXM 9 -> APIC 0x0267 -> Node 9
  972.918967 (    0.007364)| [    0.000000] SRAT: PXM 9 -> APIC 0x0269 -> Node 9
  972.922850 (    0.003883)| [    0.000000] SRAT: PXM 9 -> APIC 0x026b -> Node 9
  972.926789 (    0.003939)| [    0.000000] SRAT: PXM 9 -> APIC 0x0271 -> Node 9
  972.930950 (    0.004161)| [    0.000000] SRAT: PXM 9 -> APIC 0x0273 -> Node 9
  972.938843 (    0.007893)| [    0.000000] SRAT: PXM 9 -> APIC 0x0275 -> Node 9
  972.942775 (    0.003932)| [    0.000000] SRAT: PXM 9 -> APIC 0x0277 -> Node 9
  972.947131 (    0.004356)| [    0.000000] SRAT: PXM 9 -> APIC 0x0279 -> Node 9
  972.957668 (    0.010537)| [    0.000000] SRAT: PXM 9 -> APIC 0x027b -> Node 9
  972.960733 (    0.003065)| [    0.000000] SRAT: PXM 10 -> APIC 0x0281 -> Node 10
  972.962936 (    0.002203)| [    0.000000] SRAT: PXM 10 -> APIC 0x0283 -> Node 10
  972.970855 (    0.007919)| [    0.000000] SRAT: PXM 10 -> APIC 0x0285 -> Node 10
  972.974901 (    0.004046)| [    0.000000] SRAT: PXM 10 -> APIC 0x0287 -> Node 10
  972.978378 (    0.003477)| [    0.000000] SRAT: PXM 10 -> APIC 0x0289 -> Node 10
  972.988574 (    0.010196)| [    0.000000] SRAT: PXM 10 -> APIC 0x028b -> Node 10
  972.991059 (    0.002485)| [    0.000000] SRAT: PXM 10 -> APIC 0x0291 -> Node 10
  972.996395 (    0.005336)| [    0.000000] SRAT: PXM 10 -> APIC 0x0293 -> Node 10
  973.002994 (    0.006599)| [    0.000000] SRAT: PXM 10 -> APIC 0x0295 -> Node 10
  973.007668 (    0.004674)| [    0.000000] SRAT: PXM 10 -> APIC 0x0297 -> Node 10
  973.011805 (    0.004137)| [    0.000000] SRAT: PXM 10 -> APIC 0x0299 -> Node 10
  973.020201 (    0.008396)| [    0.000000] SRAT: PXM 10 -> APIC 0x029b -> Node 10
  973.024636 (    0.004435)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a1 -> Node 10
  973.026962 (    0.002326)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a3 -> Node 10
  973.035745 (    0.008783)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a5 -> Node 10
  973.038989 (    0.003244)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a7 -> Node 10
  973.042843 (    0.003854)| [    0.000000] SRAT: PXM 10 -> APIC 0x02a9 -> Node 10
  973.051477 (    0.008634)| [    0.000000] SRAT: PXM 10 -> APIC 0x02ab -> Node 10
  973.054823 (    0.003346)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b1 -> Node 10
  973.059070 (    0.004247)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b3 -> Node 10
  973.067019 (    0.007949)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b5 -> Node 10
  973.071048 (    0.004029)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b7 -> Node 10
  973.077612 (    0.006564)| [    0.000000] SRAT: PXM 10 -> APIC 0x02b9 -> Node 10
  973.083228 (    0.005616)| [    0.000000] SRAT: PXM 10 -> APIC 0x02bb -> Node 10
  973.087121 (    0.003893)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c1 -> Node 11
  973.091954 (    0.004833)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c3 -> Node 11
  973.099597 (    0.007643)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c5 -> Node 11
  973.103516 (    0.003919)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c7 -> Node 11
  973.106623 (    0.003107)| [    0.000000] SRAT: PXM 11 -> APIC 0x02c9 -> Node 11
  973.115658 (    0.009035)| [    0.000000] SRAT: PXM 11 -> APIC 0x02cb -> Node 11
  973.119181 (    0.003523)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d1 -> Node 11
  973.123449 (    0.004268)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d3 -> Node 11
  973.131756 (    0.008307)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d5 -> Node 11
  973.136302 (    0.004546)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d7 -> Node 11
  973.140031 (    0.003729)| [    0.000000] SRAT: PXM 11 -> APIC 0x02d9 -> Node 11
  973.147496 (    0.007465)| [    0.000000] SRAT: PXM 11 -> APIC 0x02db -> Node 11
  973.151658 (    0.004162)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e1 -> Node 11
  973.156275 (    0.004617)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e3 -> Node 11
  973.163109 (    0.006834)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e5 -> Node 11
  973.167489 (    0.004380)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e7 -> Node 11
  973.171369 (    0.003880)| [    0.000000] SRAT: PXM 11 -> APIC 0x02e9 -> Node 11
  973.179873 (    0.008504)| [    0.000000] SRAT: PXM 11 -> APIC 0x02eb -> Node 11
  973.183456 (    0.003583)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f1 -> Node 11
  973.187395 (    0.003939)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f3 -> Node 11
  973.191746 (    0.004351)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f5 -> Node 11
  973.200231 (    0.008485)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f7 -> Node 11
  973.203764 (    0.003533)| [    0.000000] SRAT: PXM 11 -> APIC 0x02f9 -> Node 11
  973.211873 (    0.008109)| [    0.000000] SRAT: PXM 11 -> APIC 0x02fb -> Node 11
  973.215239 (    0.003366)| [    0.000000] SRAT: PXM 12 -> APIC 0x0301 -> Node 12
  973.220022 (    0.004783)| [    0.000000] SRAT: PXM 12 -> APIC 0x0303 -> Node 12
  973.225004 (    0.004982)| [    0.000000] SRAT: PXM 12 -> APIC 0x0305 -> Node 12
  973.232303 (    0.007299)| [    0.000000] SRAT: PXM 12 -> APIC 0x0307 -> Node 12
  973.235422 (    0.003119)| [    0.000000] SRAT: PXM 12 -> APIC 0x0309 -> Node 12
  973.243132 (    0.007710)| [    0.000000] SRAT: PXM 12 -> APIC 0x030b -> Node 12
  973.246925 (    0.003793)| [    0.000000] SRAT: PXM 12 -> APIC 0x0311 -> Node 12
  973.251507 (    0.004582)| [    0.000000] SRAT: PXM 12 -> APIC 0x0313 -> Node 12
  973.258988 (    0.007481)| [    0.000000] SRAT: PXM 12 -> APIC 0x0315 -> Node 12
  973.263550 (    0.004562)| [    0.000000] SRAT: PXM 12 -> APIC 0x0317 -> Node 12
  973.267437 (    0.003887)| [    0.000000] SRAT: PXM 12 -> APIC 0x0319 -> Node 12
  973.275267 (    0.007830)| [    0.000000] SRAT: PXM 12 -> APIC 0x031b -> Node 12
  973.279542 (    0.004275)| [    0.000000] SRAT: PXM 12 -> APIC 0x0321 -> Node 12
  973.283340 (    0.003798)| [    0.000000] SRAT: PXM 12 -> APIC 0x0323 -> Node 12
  973.290801 (    0.007461)| [    0.000000] SRAT: PXM 12 -> APIC 0x0325 -> Node 12
  973.295208 (    0.004407)| [    0.000000] SRAT: PXM 12 -> APIC 0x0327 -> Node 12
  973.299434 (    0.004226)| [    0.000000] SRAT: PXM 12 -> APIC 0x0329 -> Node 12
  973.307247 (    0.007813)| [    0.000000] SRAT: PXM 12 -> APIC 0x032b -> Node 12
  973.311193 (    0.003946)| [    0.000000] SRAT: PXM 12 -> APIC 0x0331 -> Node 12
  973.315399 (    0.004206)| [    0.000000] SRAT: PXM 12 -> APIC 0x0333 -> Node 12
  973.323232 (    0.007833)| [    0.000000] SRAT: PXM 12 -> APIC 0x0335 -> Node 12
  973.328023 (    0.004791)| [    0.000000] SRAT: PXM 12 -> APIC 0x0337 -> Node 12
  973.334757 (    0.006734)| [    0.000000] SRAT: PXM 12 -> APIC 0x0339 -> Node 12
  973.335623 (    0.000866)| [    0.000000] SRAT: PXM 12 -> APIC 0x033b -> Node 12
  973.343362 (    0.007739)| [    0.000000] SRAT: PXM 13 -> APIC 0x0341 -> Node 13
  973.348990 (    0.005628)| [    0.000000] SRAT: PXM 13 -> APIC 0x0343 -> Node 13
  973.355153 (    0.006163)| [    0.000000] SRAT: PXM 13 -> APIC 0x0345 -> Node 13
  973.359451 (    0.004298)| [    0.000000] SRAT: PXM 13 -> APIC 0x0347 -> Node 13
  973.366832 (    0.007381)| [    0.000000] SRAT: PXM 13 -> APIC 0x0349 -> Node 13
  973.371163 (    0.004331)| [    0.000000] SRAT: PXM 13 -> APIC 0x034b -> Node 13
  973.376618 (    0.005455)| [    0.000000] SRAT: PXM 13 -> APIC 0x0351 -> Node 13
  973.379802 (    0.003184)| [    0.000000] SRAT: PXM 13 -> APIC 0x0353 -> Node 13
  973.388943 (    0.009141)| [    0.000000] SRAT: PXM 13 -> APIC 0x0355 -> Node 13
  973.391362 (    0.002419)| [    0.000000] SRAT: PXM 13 -> APIC 0x0357 -> Node 13
  973.394862 (    0.003500)| [    0.000000] SRAT: PXM 13 -> APIC 0x0359 -> Node 13
  973.403189 (    0.008327)| [    0.000000] SRAT: PXM 13 -> APIC 0x035b -> Node 13
  973.407142 (    0.003953)| [    0.000000] SRAT: PXM 13 -> APIC 0x0361 -> Node 13
  973.411428 (    0.004286)| [    0.000000] SRAT: PXM 13 -> APIC 0x0363 -> Node 13
  973.414996 (    0.003568)| [    0.000000] SRAT: PXM 13 -> APIC 0x0365 -> Node 13
  973.423335 (    0.008339)| [    0.000000] SRAT: PXM 13 -> APIC 0x0367 -> Node 13
  973.428570 (    0.005235)| [    0.000000] SRAT: PXM 13 -> APIC 0x0369 -> Node 13
  973.435813 (    0.007243)| [    0.000000] SRAT: PXM 13 -> APIC 0x036b -> Node 13
  973.440531 (    0.004718)| [    0.000000] SRAT: PXM 13 -> APIC 0x0371 -> Node 13
  973.443210 (    0.002679)| [    0.000000] SRAT: PXM 13 -> APIC 0x0373 -> Node 13
  973.452233 (    0.009023)| [    0.000000] SRAT: PXM 13 -> APIC 0x0375 -> Node 13
  973.455145 (    0.002912)| [    0.000000] SRAT: PXM 13 -> APIC 0x0377 -> Node 13
  973.459220 (    0.004075)| [    0.000000] SRAT: PXM 13 -> APIC 0x0379 -> Node 13
  973.467611 (    0.008391)| [    0.000000] SRAT: PXM 13 -> APIC 0x037b -> Node 13
  973.471433 (    0.003822)| [    0.000000] SRAT: PXM 14 -> APIC 0x0381 -> Node 14
  973.475615 (    0.004182)| [    0.000000] SRAT: PXM 14 -> APIC 0x0383 -> Node 14
  973.483346 (    0.007731)| [    0.000000] SRAT: PXM 14 -> APIC 0x0385 -> Node 14
  973.487264 (    0.003918)| [    0.000000] SRAT: PXM 14 -> APIC 0x0387 -> Node 14
  973.491362 (    0.004098)| [    0.000000] SRAT: PXM 14 -> APIC 0x0389 -> Node 14
  973.499194 (    0.007832)| [    0.000000] SRAT: PXM 14 -> APIC 0x038b -> Node 14
  973.503155 (    0.003961)| [    0.000000] SRAT: PXM 14 -> APIC 0x0391 -> Node 14
  973.507217 (    0.004062)| [    0.000000] SRAT: PXM 14 -> APIC 0x0393 -> Node 14
  973.515349 (    0.008132)| [    0.000000] SRAT: PXM 14 -> APIC 0x0395 -> Node 14
  973.519590 (    0.004241)| [    0.000000] SRAT: PXM 14 -> APIC 0x0397 -> Node 14
  973.523800 (    0.004210)| [    0.000000] SRAT: PXM 14 -> APIC 0x0399 -> Node 14
  973.530869 (    0.007069)| [    0.000000] SRAT: PXM 14 -> APIC 0x039b -> Node 14
  973.535104 (    0.004235)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a1 -> Node 14
  973.539378 (    0.004274)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a3 -> Node 14
  973.547483 (    0.008105)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a5 -> Node 14
  973.551331 (    0.003848)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a7 -> Node 14
  973.555417 (    0.004086)| [    0.000000] SRAT: PXM 14 -> APIC 0x03a9 -> Node 14
  973.559468 (    0.004051)| [    0.000000] SRAT: PXM 14 -> APIC 0x03ab -> Node 14
  973.567504 (    0.008036)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b1 -> Node 14
  973.571362 (    0.003858)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b3 -> Node 14
  973.579789 (    0.008427)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b5 -> Node 14
  973.583758 (    0.003969)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b7 -> Node 14
  973.587569 (    0.003811)| [    0.000000] SRAT: PXM 14 -> APIC 0x03b9 -> Node 14
  973.595951 (    0.008382)| [    0.000000] SRAT: PXM 14 -> APIC 0x03bb -> Node 14
  973.599354 (    0.003403)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c1 -> Node 15
  973.603443 (    0.004089)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c3 -> Node 15
  973.607446 (    0.004003)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c5 -> Node 15
  973.615303 (    0.007857)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c7 -> Node 15
  973.619658 (    0.004355)| [    0.000000] SRAT: PXM 15 -> APIC 0x03c9 -> Node 15
  973.627252 (    0.007594)| [    0.000000] SRAT: PXM 15 -> APIC 0x03cb -> Node 15
  973.631495 (    0.004243)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d1 -> Node 15
  973.635486 (    0.003991)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d3 -> Node 15
  973.643181 (    0.007695)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d5 -> Node 15
  973.647318 (    0.004137)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d7 -> Node 15
  973.651802 (    0.004484)| [    0.000000] SRAT: PXM 15 -> APIC 0x03d9 -> Node 15
  973.658821 (    0.007019)| [    0.000000] SRAT: PXM 15 -> APIC 0x03db -> Node 15
  973.663383 (    0.004562)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e1 -> Node 15
  973.667411 (    0.004028)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e3 -> Node 15
  973.676069 (    0.008658)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e5 -> Node 15
  973.678872 (    0.002803)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e7 -> Node 15
  973.682832 (    0.003960)| [    0.000000] SRAT: PXM 15 -> APIC 0x03e9 -> Node 15
  973.691947 (    0.009115)| [    0.000000] SRAT: PXM 15 -> APIC 0x03eb -> Node 15
  973.695253 (    0.003306)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f1 -> Node 15
  973.699875 (    0.004622)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f3 -> Node 15
  973.707026 (    0.007151)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f5 -> Node 15
  973.711251 (    0.004225)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f7 -> Node 15
  973.715707 (    0.004456)| [    0.000000] SRAT: PXM 15 -> APIC 0x03f9 -> Node 15
  973.723343 (    0.007636)| [    0.000000] SRAT: PXM 15 -> APIC 0x03fb -> Node 15
  973.727534 (    0.004191)| [    0.000000] SRAT: PXM 16 -> APIC 0x0401 -> Node 16
  973.731840 (    0.004306)| [    0.000000] SRAT: PXM 16 -> APIC 0x0403 -> Node 16
  973.739310 (    0.007470)| [    0.000000] SRAT: PXM 16 -> APIC 0x0405 -> Node 16
  973.743498 (    0.004188)| [    0.000000] SRAT: PXM 16 -> APIC 0x0407 -> Node 16
  973.747361 (    0.003863)| [    0.000000] SRAT: PXM 16 -> APIC 0x0409 -> Node 16
  973.751452 (    0.004091)| [    0.000000] SRAT: PXM 16 -> APIC 0x040b -> Node 16
  973.759487 (    0.008035)| [    0.000000] SRAT: PXM 16 -> APIC 0x0411 -> Node 16
  973.763528 (    0.004041)| [    0.000000] SRAT: PXM 16 -> APIC 0x0413 -> Node 16
  973.768780 (    0.005252)| [    0.000000] SRAT: PXM 16 -> APIC 0x0415 -> Node 16
  973.776156 (    0.007376)| [    0.000000] SRAT: PXM 16 -> APIC 0x0417 -> Node 16
  973.779955 (    0.003799)| [    0.000000] SRAT: PXM 16 -> APIC 0x0419 -> Node 16
  973.785008 (    0.005053)| [    0.000000] SRAT: PXM 16 -> APIC 0x041b -> Node 16
  973.791953 (    0.006945)| [    0.000000] SRAT: PXM 16 -> APIC 0x0421 -> Node 16
  973.796212 (    0.004259)| [    0.000000] SRAT: PXM 16 -> APIC 0x0423 -> Node 16
  973.799898 (    0.003686)| [    0.000000] SRAT: PXM 16 -> APIC 0x0425 -> Node 16
  973.807657 (    0.007759)| [    0.000000] SRAT: PXM 16 -> APIC 0x0427 -> Node 16
  973.811200 (    0.003543)| [    0.000000] SRAT: PXM 16 -> APIC 0x0429 -> Node 16
  973.820745 (    0.009545)| [    0.000000] SRAT: PXM 16 -> APIC 0x042b -> Node 16
  973.825409 (    0.004664)| [    0.000000] SRAT: PXM 16 -> APIC 0x0431 -> Node 16
  973.827478 (    0.002069)| [    0.000000] SRAT: PXM 16 -> APIC 0x0433 -> Node 16
  973.835269 (    0.007791)| [    0.000000] SRAT: PXM 16 -> APIC 0x0435 -> Node 16
  973.839449 (    0.004180)| [    0.000000] SRAT: PXM 16 -> APIC 0x0437 -> Node 16
  973.843183 (    0.003734)| [    0.000000] SRAT: PXM 16 -> APIC 0x0439 -> Node 16
  973.851505 (    0.008322)| [    0.000000] SRAT: PXM 16 -> APIC 0x043b -> Node 16
  973.855503 (    0.003998)| [    0.000000] SRAT: PXM 17 -> APIC 0x0441 -> Node 17
  973.859802 (    0.004299)| [    0.000000] SRAT: PXM 17 -> APIC 0x0443 -> Node 17
  973.867597 (    0.007795)| [    0.000000] SRAT: PXM 17 -> APIC 0x0445 -> Node 17
  973.871380 (    0.003783)| [    0.000000] SRAT: PXM 17 -> APIC 0x0447 -> Node 17
  973.875407 (    0.004027)| [    0.000000] SRAT: PXM 17 -> APIC 0x0449 -> Node 17
  973.884008 (    0.008601)| [    0.000000] SRAT: PXM 17 -> APIC 0x044b -> Node 17
  973.887433 (    0.003425)| [    0.000000] SRAT: PXM 17 -> APIC 0x0451 -> Node 17
  973.891328 (    0.003895)| [    0.000000] SRAT: PXM 17 -> APIC 0x0453 -> Node 17
  973.899182 (    0.007854)| [    0.000000] SRAT: PXM 17 -> APIC 0x0455 -> Node 17
  973.903294 (    0.004112)| [    0.000000] SRAT: PXM 17 -> APIC 0x0457 -> Node 17
  973.907851 (    0.004557)| [    0.000000] SRAT: PXM 17 -> APIC 0x0459 -> Node 17
  973.915579 (    0.007728)| [    0.000000] SRAT: PXM 17 -> APIC 0x045b -> Node 17
  973.925290 (    0.009711)| [    0.000000] SRAT: PXM 17 -> APIC 0x0461 -> Node 17
  973.925404 (    0.000114)| [    0.000000] SRAT: PXM 17 -> APIC 0x0463 -> Node 17
  973.931276 (    0.005872)| [    0.000000] SRAT: PXM 17 -> APIC 0x0465 -> Node 17
  973.935981 (    0.004705)| [    0.000000] SRAT: PXM 17 -> APIC 0x0467 -> Node 17
  973.939764 (    0.003783)| [    0.000000] SRAT: PXM 17 -> APIC 0x0469 -> Node 17
  973.947274 (    0.007510)| [    0.000000] SRAT: PXM 17 -> APIC 0x046b -> Node 17
  973.951404 (    0.004130)| [    0.000000] SRAT: PXM 17 -> APIC 0x0471 -> Node 17
  973.955565 (    0.004161)| [    0.000000] SRAT: PXM 17 -> APIC 0x0473 -> Node 17
  973.959592 (    0.004027)| [    0.000000] SRAT: PXM 17 -> APIC 0x0475 -> Node 17
  973.967251 (    0.007659)| [    0.000000] SRAT: PXM 17 -> APIC 0x0477 -> Node 17
  973.971492 (    0.004241)| [    0.000000] SRAT: PXM 17 -> APIC 0x0479 -> Node 17
  973.980212 (    0.008720)| [    0.000000] SRAT: PXM 17 -> APIC 0x047b -> Node 17
  973.984041 (    0.003829)| [    0.000000] SRAT: PXM 18 -> APIC 0x0481 -> Node 18
  973.987354 (    0.003313)| [    0.000000] SRAT: PXM 18 -> APIC 0x0483 -> Node 18
  973.995932 (    0.008578)| [    0.000000] SRAT: PXM 18 -> APIC 0x0485 -> Node 18
  973.998905 (    0.002973)| [    0.000000] SRAT: PXM 18 -> APIC 0x0487 -> Node 18
  974.003095 (    0.004190)| [    0.000000] SRAT: PXM 18 -> APIC 0x0489 -> Node 18
  974.011533 (    0.008438)| [    0.000000] SRAT: PXM 18 -> APIC 0x048b -> Node 18
  974.015429 (    0.003896)| [    0.000000] SRAT: PXM 18 -> APIC 0x0491 -> Node 18
  974.019462 (    0.004033)| [    0.000000] SRAT: PXM 18 -> APIC 0x0493 -> Node 18
  974.027288 (    0.007826)| [    0.000000] SRAT: PXM 18 -> APIC 0x0495 -> Node 18
  974.031792 (    0.004504)| [    0.000000] SRAT: PXM 18 -> APIC 0x0497 -> Node 18
  974.039034 (    0.007242)| [    0.000000] SRAT: PXM 18 -> APIC 0x0499 -> Node 18
  974.040434 (    0.001400)| [    0.000000] SRAT: PXM 18 -> APIC 0x049b -> Node 18
  974.047268 (    0.006834)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a1 -> Node 18
  974.052958 (    0.005690)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a3 -> Node 18
  974.056086 (    0.003128)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a5 -> Node 18
  974.063384 (    0.007298)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a7 -> Node 18
  974.069074 (    0.005690)| [    0.000000] SRAT: PXM 18 -> APIC 0x04a9 -> Node 18
  974.078329 (    0.009255)| [    0.000000] SRAT: PXM 18 -> APIC 0x04ab -> Node 18
  974.080313 (    0.001984)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b1 -> Node 18
  974.083630 (    0.003317)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b3 -> Node 18
  974.093184 (    0.009554)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b5 -> Node 18
  974.096698 (    0.003514)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b7 -> Node 18
  974.100067 (    0.003369)| [    0.000000] SRAT: PXM 18 -> APIC 0x04b9 -> Node 18
  974.107050 (    0.006983)| [    0.000000] SRAT: PXM 18 -> APIC 0x04bb -> Node 18
  974.113057 (    0.006007)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c1 -> Node 19
  974.114902 (    0.001845)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c3 -> Node 19
  974.124703 (    0.009801)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c5 -> Node 19
  974.127748 (    0.003045)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c7 -> Node 19
  974.134073 (    0.006325)| [    0.000000] SRAT: PXM 19 -> APIC 0x04c9 -> Node 19
  974.140543 (    0.006470)| [    0.000000] SRAT: PXM 19 -> APIC 0x04cb -> Node 19
  974.144196 (    0.003653)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d1 -> Node 19
  974.147111 (    0.002915)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d3 -> Node 19
  974.156773 (    0.009662)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d5 -> Node 19
  974.161219 (    0.004446)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d7 -> Node 19
  974.165151 (    0.003932)| [    0.000000] SRAT: PXM 19 -> APIC 0x04d9 -> Node 19
  974.171898 (    0.006747)| [    0.000000] SRAT: PXM 19 -> APIC 0x04db -> Node 19
  974.175816 (    0.003918)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e1 -> Node 19
  974.179418 (    0.003602)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e3 -> Node 19
  974.185317 (    0.005899)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e5 -> Node 19
  974.191945 (    0.006628)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e7 -> Node 19
  974.195622 (    0.003677)| [    0.000000] SRAT: PXM 19 -> APIC 0x04e9 -> Node 19
  974.201075 (    0.005453)| [    0.000000] SRAT: PXM 19 -> APIC 0x04eb -> Node 19
  974.208017 (    0.006942)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f1 -> Node 19
  974.211268 (    0.003251)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f3 -> Node 19
  978.477155 (    4.265887)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f5 -> Node 19
  978.477198 (    0.000043)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f7 -> Node 19
  978.477233 (    0.000035)| [    0.000000] SRAT: PXM 19 -> APIC 0x04f9 -> Node 19
  978.477268 (    0.000035)| [    0.000000] SRAT: PXM 19 -> APIC 0x04fb -> Node 19
  978.477302 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0501 -> Node 20
  978.477337 (    0.000035)| [    0.000000] SRAT: PXM 20 -> APIC 0x0503 -> Node 20
  978.477371 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0505 -> Node 20
  978.477405 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0507 -> Node 20
  978.477440 (    0.000035)| [    0.000000] SRAT: PXM 20 -> APIC 0x0509 -> Node 20
  978.477474 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x050b -> Node 20
  978.477507 (    0.000033)| [    0.000000] SRAT: PXM 20 -> APIC 0x0511 -> Node 20
  978.477541 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0513 -> Node 20
  978.477575 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0515 -> Node 20
  978.477618 (    0.000043)| [    0.000000] SRAT: PXM 20 -> APIC 0x0517 -> Node 20
  978.477651 (    0.000033)| [    0.000000] SRAT: PXM 20 -> APIC 0x0519 -> Node 20
  978.477685 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x051b -> Node 20
  978.477719 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0521 -> Node 20
  978.477752 (    0.000033)| [    0.000000] SRAT: PXM 20 -> APIC 0x0523 -> Node 20
  978.477785 (    0.000033)| [    0.000000] SRAT: PXM 20 -> APIC 0x0525 -> Node 20
  978.477819 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0527 -> Node 20
  978.477853 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0529 -> Node 20
  978.477887 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x052b -> Node 20
  978.477921 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0531 -> Node 20
  978.477956 (    0.000035)| [    0.000000] SRAT: PXM 20 -> APIC 0x0533 -> Node 20
  978.477990 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0535 -> Node 20
  978.478024 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0537 -> Node 20
  978.478058 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x0539 -> Node 20
  978.478092 (    0.000034)| [    0.000000] SRAT: PXM 20 -> APIC 0x053b -> Node 20
  978.478126 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0541 -> Node 21
  978.478171 (    0.000045)| [    0.000000] SRAT: PXM 21 -> APIC 0x0543 -> Node 21
  978.478205 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0545 -> Node 21
  978.478238 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0547 -> Node 21
  978.478271 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0549 -> Node 21
  978.478305 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x054b -> Node 21
  978.478339 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0551 -> Node 21
  978.478372 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0553 -> Node 21
  978.478405 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0555 -> Node 21
  978.478439 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0557 -> Node 21
  978.478472 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0559 -> Node 21
  978.478506 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x055b -> Node 21
  978.478539 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0561 -> Node 21
  978.478573 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0563 -> Node 21
  978.478606 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0565 -> Node 21
  978.478640 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0567 -> Node 21
  978.478673 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0569 -> Node 21
  978.478706 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x056b -> Node 21
  978.478740 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0571 -> Node 21
  978.478773 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0573 -> Node 21
  978.478807 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0575 -> Node 21
  978.478840 (    0.000033)| [    0.000000] SRAT: PXM 21 -> APIC 0x0577 -> Node 21
  978.478874 (    0.000034)| [    0.000000] SRAT: PXM 21 -> APIC 0x0579 -> Node 21
  978.483111 (    0.004237)| [    0.000000] SRAT: PXM 21 -> APIC 0x057b -> Node 21
  978.483147 (    0.000036)| [    0.000000] SRAT: PXM 22 -> APIC 0x0581 -> Node 22
  978.483181 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x0583 -> Node 22
  978.483214 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x0585 -> Node 22
  978.483248 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x0587 -> Node 22
  978.483281 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x0589 -> Node 22
  978.483315 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x058b -> Node 22
  978.483348 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x0591 -> Node 22
  978.483382 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x0593 -> Node 22
  978.483415 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x0595 -> Node 22
  978.483448 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x0597 -> Node 22
  978.483482 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x0599 -> Node 22
  978.483515 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x059b -> Node 22
  978.483549 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a1 -> Node 22
  978.483582 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a3 -> Node 22
  978.483616 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a5 -> Node 22
  978.483649 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a7 -> Node 22
  978.483683 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x05a9 -> Node 22
  978.483716 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05ab -> Node 22
  978.483749 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b1 -> Node 22
  978.483783 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b3 -> Node 22
  978.483816 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b5 -> Node 22
  978.483850 (    0.000034)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b7 -> Node 22
  978.483887 (    0.000037)| [    0.000000] SRAT: PXM 22 -> APIC 0x05b9 -> Node 22
  978.483920 (    0.000033)| [    0.000000] SRAT: PXM 22 -> APIC 0x05bb -> Node 22
  978.483954 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c1 -> Node 23
  978.483987 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c3 -> Node 23
  978.484021 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c5 -> Node 23
  978.484062 (    0.000041)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c7 -> Node 23
  978.484096 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05c9 -> Node 23
  978.484130 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05cb -> Node 23
  978.484164 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d1 -> Node 23
  978.484198 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d3 -> Node 23
  978.484232 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d5 -> Node 23
  978.484265 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d7 -> Node 23
  978.484299 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05d9 -> Node 23
  978.484332 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05db -> Node 23
  978.484366 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e1 -> Node 23
  978.484399 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e3 -> Node 23
  978.484433 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e5 -> Node 23
  978.484466 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e7 -> Node 23
  978.484500 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05e9 -> Node 23
  978.484533 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05eb -> Node 23
  978.484567 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f1 -> Node 23
  978.484600 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f3 -> Node 23
  978.484634 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f5 -> Node 23
  978.484667 (    0.000033)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f7 -> Node 23
  978.484701 (    0.000034)| [    0.000000] SRAT: PXM 23 -> APIC 0x05f9 -> Node 23
  978.488627 (    0.003926)| [    0.000000] SRAT: PXM 23 -> APIC 0x05fb -> Node 23
  978.488662 (    0.000035)| [    0.000000] SRAT: PXM 24 -> APIC 0x0601 -> Node 24
  978.488695 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x0603 -> Node 24
  978.488729 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0605 -> Node 24
  978.488763 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0607 -> Node 24
  978.488797 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0609 -> Node 24
  978.488830 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x060b -> Node 24
  978.488864 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0611 -> Node 24
  978.488898 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0613 -> Node 24
  978.488932 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0615 -> Node 24
  978.488967 (    0.000035)| [    0.000000] SRAT: PXM 24 -> APIC 0x0617 -> Node 24
  978.489002 (    0.000035)| [    0.000000] SRAT: PXM 24 -> APIC 0x0619 -> Node 24
  978.489036 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x061b -> Node 24
  978.489069 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x0621 -> Node 24
  978.489104 (    0.000035)| [    0.000000] SRAT: PXM 24 -> APIC 0x0623 -> Node 24
  978.489138 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0625 -> Node 24
  978.489171 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x0627 -> Node 24
  978.489205 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0629 -> Node 24
  978.489239 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x062b -> Node 24
  978.489273 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0631 -> Node 24
  978.489306 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x0633 -> Node 24
  978.489340 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0635 -> Node 24
  978.489374 (    0.000034)| [    0.000000] SRAT: PXM 24 -> APIC 0x0637 -> Node 24
  978.489407 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x0639 -> Node 24
  978.489440 (    0.000033)| [    0.000000] SRAT: PXM 24 -> APIC 0x063b -> Node 24
  978.489474 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0641 -> Node 25
  978.489507 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x0643 -> Node 25
  978.489541 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0645 -> Node 25
  978.489575 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0647 -> Node 25
  978.489627 (    0.000052)| [    0.000000] SRAT: PXM 25 -> APIC 0x0649 -> Node 25
  978.489661 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x064b -> Node 25
  978.489695 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0651 -> Node 25
  978.489728 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x0653 -> Node 25
  978.489762 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0655 -> Node 25
  978.489795 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x0657 -> Node 25
  978.489829 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0659 -> Node 25
  978.489862 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x065b -> Node 25
  978.489896 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0661 -> Node 25
  978.489929 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x0663 -> Node 25
  978.489963 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0665 -> Node 25
  978.489997 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0667 -> Node 25
  978.490031 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0669 -> Node 25
  978.490065 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x066b -> Node 25
  978.490098 (    0.000033)| [    0.000000] SRAT: PXM 25 -> APIC 0x0671 -> Node 25
  978.490132 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0673 -> Node 25
  978.490166 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0675 -> Node 25
  978.490200 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0677 -> Node 25
  978.490234 (    0.000034)| [    0.000000] SRAT: PXM 25 -> APIC 0x0679 -> Node 25
  978.495376 (    0.005142)| [    0.000000] SRAT: PXM 25 -> APIC 0x067b -> Node 25
  978.495412 (    0.000036)| [    0.000000] SRAT: PXM 26 -> APIC 0x0681 -> Node 26
  978.495449 (    0.000037)| [    0.000000] SRAT: PXM 26 -> APIC 0x0683 -> Node 26
  978.495483 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x0685 -> Node 26
  978.495516 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x0687 -> Node 26
  978.495550 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x0689 -> Node 26
  978.495583 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x068b -> Node 26
  978.495616 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x0691 -> Node 26
  978.495648 (    0.000032)| [    0.000000] SRAT: PXM 26 -> APIC 0x0693 -> Node 26
  978.495681 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x0695 -> Node 26
  978.495714 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x0697 -> Node 26
  978.495748 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x0699 -> Node 26
  978.495781 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x069b -> Node 26
  978.495815 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a1 -> Node 26
  978.495848 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a3 -> Node 26
  978.495882 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a5 -> Node 26
  978.495915 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a7 -> Node 26
  978.495949 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06a9 -> Node 26
  978.495982 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06ab -> Node 26
  978.496016 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b1 -> Node 26
  978.496049 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b3 -> Node 26
  978.496083 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b5 -> Node 26
  978.496116 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b7 -> Node 26
  978.496150 (    0.000034)| [    0.000000] SRAT: PXM 26 -> APIC 0x06b9 -> Node 26
  978.496183 (    0.000033)| [    0.000000] SRAT: PXM 26 -> APIC 0x06bb -> Node 26
  978.496217 (    0.000034)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c1 -> Node 27
  978.496250 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c3 -> Node 27
  978.496284 (    0.000034)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c5 -> Node 27
  978.496317 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c7 -> Node 27
  978.496351 (    0.000034)| [    0.000000] SRAT: PXM 27 -> APIC 0x06c9 -> Node 27
  978.496395 (    0.000044)| [    0.000000] SRAT: PXM 27 -> APIC 0x06cb -> Node 27
  978.496429 (    0.000034)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d1 -> Node 27
  978.496462 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d3 -> Node 27
  978.496496 (    0.000034)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d5 -> Node 27
  978.496529 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d7 -> Node 27
  978.496562 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06d9 -> Node 27
  978.496595 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06db -> Node 27
  978.496627 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e1 -> Node 27
  978.496660 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e3 -> Node 27
  978.496692 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e5 -> Node 27
  978.496725 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e7 -> Node 27
  978.496757 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06e9 -> Node 27
  978.496790 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06eb -> Node 27
  978.496822 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f1 -> Node 27
  978.496855 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f3 -> Node 27
  978.496887 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f5 -> Node 27
  978.496919 (    0.000032)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f7 -> Node 27
  978.496952 (    0.000033)| [    0.000000] SRAT: PXM 27 -> APIC 0x06f9 -> Node 27
  978.502418 (    0.005466)| [    0.000000] SRAT: PXM 27 -> APIC 0x06fb -> Node 27
  978.502452 (    0.000034)| [    0.000000] SRAT: PXM 28 -> APIC 0x0701 -> Node 28
  978.502486 (    0.000034)| [    0.000000] SRAT: PXM 28 -> APIC 0x0703 -> Node 28
  978.502518 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0705 -> Node 28
  978.502551 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0707 -> Node 28
  978.502583 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0709 -> Node 28
  978.502616 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x070b -> Node 28
  978.502648 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0711 -> Node 28
  978.502681 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0713 -> Node 28
  978.502714 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0715 -> Node 28
  978.502746 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0717 -> Node 28
  978.502778 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0719 -> Node 28
  978.502811 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x071b -> Node 28
  978.502843 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0721 -> Node 28
  978.502876 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0723 -> Node 28
  978.502909 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0725 -> Node 28
  978.502941 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0727 -> Node 28
  978.502974 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0729 -> Node 28
  978.503007 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x072b -> Node 28
  978.503039 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0731 -> Node 28
  978.503072 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0733 -> Node 28
  978.503104 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0735 -> Node 28
  978.503137 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x0737 -> Node 28
  978.503169 (    0.000032)| [    0.000000] SRAT: PXM 28 -> APIC 0x0739 -> Node 28
  978.503202 (    0.000033)| [    0.000000] SRAT: PXM 28 -> APIC 0x073b -> Node 28
  978.503234 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0741 -> Node 29
  978.503267 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0743 -> Node 29
  978.503300 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0745 -> Node 29
  978.503336 (    0.000036)| [    0.000000] SRAT: PXM 29 -> APIC 0x0747 -> Node 29
  978.503369 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0749 -> Node 29
  978.503401 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x074b -> Node 29
  978.503434 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0751 -> Node 29
  978.503474 (    0.000040)| [    0.000000] SRAT: PXM 29 -> APIC 0x0753 -> Node 29
  978.503507 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0755 -> Node 29
  978.503539 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0757 -> Node 29
  978.503572 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0759 -> Node 29
  978.503612 (    0.000040)| [    0.000000] SRAT: PXM 29 -> APIC 0x075b -> Node 29
  978.503645 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0761 -> Node 29
  978.503677 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0763 -> Node 29
  978.503710 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0765 -> Node 29
  978.503742 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0767 -> Node 29
  978.503775 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0769 -> Node 29
  978.503807 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x076b -> Node 29
  978.503840 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0771 -> Node 29
  978.503872 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0773 -> Node 29
  978.503904 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0775 -> Node 29
  978.503937 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x0777 -> Node 29
  978.503969 (    0.000032)| [    0.000000] SRAT: PXM 29 -> APIC 0x0779 -> Node 29
  978.504002 (    0.000033)| [    0.000000] SRAT: PXM 29 -> APIC 0x077b -> Node 29
  978.504034 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x0781 -> Node 30
  978.504067 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0783 -> Node 30
  978.504100 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0785 -> Node 30
  978.504132 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x0787 -> Node 30
  978.504165 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0789 -> Node 30
  978.504197 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x078b -> Node 30
  978.504229 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x0791 -> Node 30
  978.504262 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0793 -> Node 30
  978.504295 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0795 -> Node 30
  978.504327 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x0797 -> Node 30
  978.504360 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x0799 -> Node 30
  978.504392 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x079b -> Node 30
  978.504425 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a1 -> Node 30
  978.504458 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a3 -> Node 30
  978.504490 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a5 -> Node 30
  978.504523 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a7 -> Node 30
  978.504555 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x07a9 -> Node 30
  978.504588 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07ab -> Node 30
  978.504620 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b1 -> Node 30
  978.504653 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b3 -> Node 30
  978.504686 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b5 -> Node 30
  978.504718 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b7 -> Node 30
  978.504751 (    0.000033)| [    0.000000] SRAT: PXM 30 -> APIC 0x07b9 -> Node 30
  978.504783 (    0.000032)| [    0.000000] SRAT: PXM 30 -> APIC 0x07bb -> Node 30
  978.504816 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c1 -> Node 31
  978.504848 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c3 -> Node 31
  978.504881 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c5 -> Node 31
  978.504913 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c7 -> Node 31
  978.504946 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07c9 -> Node 31
  978.504978 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07cb -> Node 31
  978.505011 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d1 -> Node 31
  978.505043 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d3 -> Node 31
  978.505084 (    0.000041)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d5 -> Node 31
  978.505117 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d7 -> Node 31
  978.505149 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07d9 -> Node 31
  978.505182 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07db -> Node 31
  978.505214 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e1 -> Node 31
  978.505247 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e3 -> Node 31
  978.505279 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e5 -> Node 31
  978.505312 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e7 -> Node 31
  978.505344 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07e9 -> Node 31
  978.505377 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07eb -> Node 31
  978.505409 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f1 -> Node 31
  978.505442 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f3 -> Node 31
  978.505474 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f5 -> Node 31
  978.505507 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f7 -> Node 31
  978.505539 (    0.000032)| [    0.000000] SRAT: PXM 31 -> APIC 0x07f9 -> Node 31
  978.505572 (    0.000033)| [    0.000000] SRAT: PXM 31 -> APIC 0x07fb -> Node 31
  978.505612 (    0.000040)| [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
  978.506054 (    0.000442)| [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x807fffffff]
  978.506200 (    0.000146)| [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x10000000000-0x17fffffffff]
  978.506350 (    0.000150)| [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x20000000000-0x27fffffffff]
  978.506505 (    0.000155)| [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0x30000000000-0x37fffffffff]
  978.506657 (    0.000152)| [    0.000000] ACPI: SRAT: Node 4 PXM 4 [mem 0x40000000000-0x47fffffffff]
  978.506807 (    0.000150)| [    0.000000] ACPI: SRAT: Node 5 PXM 5 [mem 0x50000000000-0x57fffffffff]
  978.506958 (    0.000151)| [    0.000000] ACPI: SRAT: Node 6 PXM 6 [mem 0x60000000000-0x67fffffffff]
  978.507109 (    0.000151)| [    0.000000] ACPI: SRAT: Node 7 PXM 7 [mem 0x70000000000-0x77fffffffff]
  978.507260 (    0.000151)| [    0.000000] ACPI: SRAT: Node 8 PXM 8 [mem 0x80000000000-0x87fffffffff]
  978.507411 (    0.000151)| [    0.000000] ACPI: SRAT: Node 9 PXM 9 [mem 0x90000000000-0x97fffffffff]
  978.507563 (    0.000152)| [    0.000000] ACPI: SRAT: Node 10 PXM 10 [mem 0xa0000000000-0xa7fffffffff]
  978.507718 (    0.000155)| [    0.000000] ACPI: SRAT: Node 11 PXM 11 [mem 0xb0000000000-0xb7fffffffff]
  978.507874 (    0.000156)| [    0.000000] ACPI: SRAT: Node 12 PXM 12 [mem 0xc0000000000-0xc7fffffffff]
  978.508029 (    0.000155)| [    0.000000] ACPI: SRAT: Node 13 PXM 13 [mem 0xd0000000000-0xd7fffffffff]
  978.508185 (    0.000156)| [    0.000000] ACPI: SRAT: Node 14 PXM 14 [mem 0xe0000000000-0xe7fffffffff]
  978.508343 (    0.000158)| [    0.000000] ACPI: SRAT: Node 15 PXM 15 [mem 0xf0000000000-0xf7fffffffff]
  978.508499 (    0.000156)| [    0.000000] ACPI: SRAT: Node 16 PXM 16 [mem 0x100000000000-0x107fffffffff]
  978.508657 (    0.000158)| [    0.000000] ACPI: SRAT: Node 17 PXM 17 [mem 0x110000000000-0x117fffffffff]
  978.508815 (    0.000158)| [    0.000000] ACPI: SRAT: Node 18 PXM 18 [mem 0x120000000000-0x127fffffffff]
  978.508973 (    0.000158)| [    0.000000] ACPI: SRAT: Node 19 PXM 19 [mem 0x130000000000-0x137fffffffff]
  978.509132 (    0.000159)| [    0.000000] ACPI: SRAT: Node 20 PXM 20 [mem 0x140000000000-0x147fffffffff]
  978.509291 (    0.000159)| [    0.000000] ACPI: SRAT: Node 21 PXM 21 [mem 0x150000000000-0x157fffffffff]
  978.509451 (    0.000160)| [    0.000000] ACPI: SRAT: Node 22 PXM 22 [mem 0x160000000000-0x167fffffffff]
  978.509616 (    0.000165)| [    0.000000] ACPI: SRAT: Node 23 PXM 23 [mem 0x170000000000-0x177fffffffff]
  978.509777 (    0.000161)| [    0.000000] ACPI: SRAT: Node 24 PXM 24 [mem 0x180000000000-0x187fffffffff]
  978.509937 (    0.000160)| [    0.000000] ACPI: SRAT: Node 25 PXM 25 [mem 0x190000000000-0x197fffffffff]
  978.510107 (    0.000170)| [    0.000000] ACPI: SRAT: Node 26 PXM 26 [mem 0x1a0000000000-0x1a7fffffffff]
  978.510266 (    0.000159)| [    0.000000] ACPI: SRAT: Node 27 PXM 27 [mem 0x1b0000000000-0x1b7fffffffff]
  978.510434 (    0.000168)| [    0.000000] ACPI: SRAT: Node 28 PXM 28 [mem 0x1c0000000000-0x1c7fffffffff]
  978.510594 (    0.000160)| [    0.000000] ACPI: SRAT: Node 29 PXM 29 [mem 0x1d0000000000-0x1d7fffffffff]
  978.510753 (    0.000159)| [    0.000000] ACPI: SRAT: Node 30 PXM 30 [mem 0x1e0000000000-0x1e7fffffffff]
  978.510911 (    0.000158)| [    0.000000] ACPI: SRAT: Node 31 PXM 31 [mem 0x1f0000000000-0x1f7fffffffff]
  978.511070 (    0.000159)| [    0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x807fffffff] -> [mem 0x00000000-0x807fffffff]
  978.511314 (    0.000244)| [    0.000000] NODE_DATA(0) allocated [mem 0x807ffde000-0x807fffffff]
  978.511457 (    0.000143)| [    0.000000] NODE_DATA(1) allocated [mem 0x17bfefde000-0x17bfeffffff]
  978.511604 (    0.000147)| [    0.000000] NODE_DATA(2) allocated [mem 0x27bfefde000-0x27bfeffffff]
  978.511752 (    0.000148)| [    0.000000] NODE_DATA(3) allocated [mem 0x37bfefde000-0x37bfeffffff]
  978.511899 (    0.000147)| [    0.000000] NODE_DATA(4) allocated [mem 0x47bfefde000-0x47bfeffffff]
  978.512046 (    0.000147)| [    0.000000] NODE_DATA(5) allocated [mem 0x57bfefde000-0x57bfeffffff]
  978.512191 (    0.000145)| [    0.000000] NODE_DATA(6) allocated [mem 0x67bfefde000-0x67bfeffffff]
  978.512338 (    0.000147)| [    0.000000] NODE_DATA(7) allocated [mem 0x77bfefde000-0x77bfeffffff]
  978.512487 (    0.000149)| [    0.000000] NODE_DATA(8) allocated [mem 0x87bfefde000-0x87bfeffffff]
  978.512634 (    0.000147)| [    0.000000] NODE_DATA(9) allocated [mem 0x97bfefde000-0x97bfeffffff]
  978.512780 (    0.000146)| [    0.000000] NODE_DATA(10) allocated [mem 0xa7bfefde000-0xa7bfeffffff]
  978.512929 (    0.000149)| [    0.000000] NODE_DATA(11) allocated [mem 0xb7bfefde000-0xb7bfeffffff]
  978.513076 (    0.000147)| [    0.000000] NODE_DATA(12) allocated [mem 0xc7bfefde000-0xc7bfeffffff]
  978.513225 (    0.000149)| [    0.000000] NODE_DATA(13) allocated [mem 0xd7bfefde000-0xd7bfeffffff]
  978.513374 (    0.000149)| [    0.000000] NODE_DATA(14) allocated [mem 0xe7bfefde000-0xe7bfeffffff]
  978.513526 (    0.000152)| [    0.000000] NODE_DATA(15) allocated [mem 0xf7bfefde000-0xf7bfeffffff]
  978.513728 (    0.000202)| [    0.000000] NODE_DATA(16) allocated [mem 0x107bfefde000-0x107bfeffffff]
  978.513890 (    0.000162)| [    0.000000] NODE_DATA(17) allocated [mem 0x117bfefde000-0x117bfeffffff]
  978.514044 (    0.000154)| [    0.000000] NODE_DATA(18) allocated [mem 0x127bfefde000-0x127bfeffffff]
  978.514197 (    0.000153)| [    0.000000] NODE_DATA(19) allocated [mem 0x137bfefde000-0x137bfeffffff]
  978.514357 (    0.000160)| [    0.000000] NODE_DATA(20) allocated [mem 0x147bfefde000-0x147bfeffffff]
  978.514511 (    0.000154)| [    0.000000] NODE_DATA(21) allocated [mem 0x157bfefde000-0x157bfeffffff]
  978.514664 (    0.000153)| [    0.000000] NODE_DATA(22) allocated [mem 0x167bfefde000-0x167bfeffffff]
  978.514818 (    0.000154)| [    0.000000] NODE_DATA(23) allocated [mem 0x177bfefde000-0x177bfeffffff]
  978.514972 (    0.000154)| [    0.000000] NODE_DATA(24) allocated [mem 0x187bfefde000-0x187bfeffffff]
  978.515125 (    0.000153)| [    0.000000] NODE_DATA(25) allocated [mem 0x197bfefde000-0x197bfeffffff]
  978.515278 (    0.000153)| [    0.000000] NODE_DATA(26) allocated [mem 0x1a7bfefde000-0x1a7bfeffffff]
  978.515431 (    0.000153)| [    0.000000] NODE_DATA(27) allocated [mem 0x1b7bfefde000-0x1b7bfeffffff]
  978.515583 (    0.000152)| [    0.000000] NODE_DATA(28) allocated [mem 0x1c7bfefde000-0x1c7bfeffffff]
  978.515736 (    0.000153)| [    0.000000] NODE_DATA(29) allocated [mem 0x1d7bfefde000-0x1d7bfeffffff]
  978.515889 (    0.000153)| [    0.000000] NODE_DATA(30) allocated [mem 0x1e7bfefde000-0x1e7bfeffffff]
  978.516041 (    0.000152)| [    0.000000] NODE_DATA(31) allocated [mem 0x1f7bfe798000-0x1f7bfe7b9fff]
  978.516204 (    0.000163)| [    0.000000] Reserving 450MB of memory at 432MB for crashkernel (System RAM: 16252376MB)
 1077.600637 (   99.084433)| [    0.000000] Zone ranges:
 1077.600690 (    0.000053)| [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
 1077.608214 (    0.007524)| [    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
 1077.616501 (    0.008287)| [    0.000000]   Normal   [mem 0x0000000100000000-0x00001f7bfeffffff]
 1077.620247 (    0.003746)| [    0.000000] Movable zone start for each node
 1077.628266 (    0.008019)| [    0.000000] Early memory node ranges
 1077.632204 (    0.003938)| [    0.000000]   node   0: [mem 0x0000000000001000-0x0000000000051fff]
 1077.636261 (    0.004057)| [    0.000000]   node   0: [mem 0x000000000005b000-0x000000000009ffff]
 1077.644137 (    0.007876)| [    0.000000]   node   0: [mem 0x0000000000100000-0x0000000061ea1fff]
 1077.652155 (    0.008018)| [    0.000000]   node   0: [mem 0x00000000620ca000-0x00000000622e4fff]
 1077.660769 (    0.008614)| [    0.000000]   node   0: [mem 0x0000000062735000-0x0000000072829fff]
 1077.664179 (    0.003410)| [    0.000000]   node   0: [mem 0x000000007282b000-0x0000000072837fff]
 1077.672380 (    0.008201)| [    0.000000]   node   0: [mem 0x0000000072a78000-0x00000000751fefff]
 1077.680147 (    0.007767)| [    0.000000]   node   0: [mem 0x0000000076fff000-0x000000007effffff]
 1077.688877 (    0.008730)| [    0.000000]   node   0: [mem 0x0000000100000000-0x0000007bffffffff]
 1077.692232 (    0.003355)| [    0.000000]   node   0: [mem 0x0000008000000000-0x000000807fffffff]
 1077.700377 (    0.008145)| [    0.000000]   node   1: [mem 0x0000010000000000-0x0000017bfeffffff]
 1077.708164 (    0.007787)| [    0.000000]   node   2: [mem 0x0000020000000000-0x0000027bfeffffff]
 1077.716274 (    0.008110)| [    0.000000]   node   3: [mem 0x0000030000000000-0x0000037bfeffffff]
 1077.720257 (    0.003983)| [    0.000000]   node   4: [mem 0x0000040000000000-0x0000047bfeffffff]
 1077.728980 (    0.008723)| [    0.000000]   node   5: [mem 0x0000050000000000-0x0000057bfeffffff]
 1077.736395 (    0.007415)| [    0.000000]   node   6: [mem 0x0000060000000000-0x0000067bfeffffff]
 1077.740632 (    0.004237)| [    0.000000]   node   7: [mem 0x0000070000000000-0x0000077bfeffffff]
 1077.748468 (    0.007836)| [    0.000000]   node   8: [mem 0x0000080000000000-0x0000087bfeffffff]
 1077.756372 (    0.007904)| [    0.000000]   node   9: [mem 0x0000090000000000-0x0000097bfeffffff]
 1077.764658 (    0.008286)| [    0.000000]   node  10: [mem 0x00000a0000000000-0x00000a7bfeffffff]
 1077.768579 (    0.003921)| [    0.000000]   node  11: [mem 0x00000b0000000000-0x00000b7bfeffffff]
 1077.780485 (    0.011906)| [    0.000000]   node  12: [mem 0x00000c0000000000-0x00000c7bfeffffff]
 1077.784811 (    0.004326)| [    0.000000]   node  13: [mem 0x00000d0000000000-0x00000d7bfeffffff]
 1077.792454 (    0.007643)| [    0.000000]   node  14: [mem 0x00000e0000000000-0x00000e7bfeffffff]
 1077.796944 (    0.004490)| [    0.000000]   node  15: [mem 0x00000f0000000000-0x00000f7bfeffffff]
 1077.805062 (    0.008118)| [    0.000000]   node  16: [mem 0x0000100000000000-0x0000107bfeffffff]
 1077.812495 (    0.007433)| [    0.000000]   node  17: [mem 0x0000110000000000-0x0000117bfeffffff]
 1077.820360 (    0.007865)| [    0.000000]   node  18: [mem 0x0000120000000000-0x0000127bfeffffff]
 1077.824444 (    0.004084)| [    0.000000]   node  19: [mem 0x0000130000000000-0x0000137bfeffffff]
 1077.832378 (    0.007934)| [    0.000000]   node  20: [mem 0x0000140000000000-0x0000147bfeffffff]
 1077.840466 (    0.008088)| [    0.000000]   node  21: [mem 0x0000150000000000-0x0000157bfeffffff]
 1077.848678 (    0.008212)| [    0.000000]   node  22: [mem 0x0000160000000000-0x0000167bfeffffff]
 1077.852714 (    0.004036)| [    0.000000]   node  23: [mem 0x0000170000000000-0x0000177bfeffffff]
 1077.860468 (    0.007754)| [    0.000000]   node  24: [mem 0x0000180000000000-0x0000187bfeffffff]
 1077.868386 (    0.007918)| [    0.000000]   node  25: [mem 0x0000190000000000-0x0000197bfeffffff]
 1077.876614 (    0.008228)| [    0.000000]   node  26: [mem 0x00001a0000000000-0x00001a7bfeffffff]
 1077.882268 (    0.005654)| [    0.000000]   node  27: [mem 0x00001b0000000000-0x00001b7bfeffffff]
 1077.888584 (    0.006316)| [    0.000000]   node  28: [mem 0x00001c0000000000-0x00001c7bfeffffff]
 1077.896742 (    0.008158)| [    0.000000]   node  29: [mem 0x00001d0000000000-0x00001d7bfeffffff]
 1077.904613 (    0.007871)| [    0.000000]   node  30: [mem 0x00001e0000000000-0x00001e7bfeffffff]
 1077.908673 (    0.004060)| [    0.000000]   node  31: [mem 0x00001f0000000000-0x00001f7bfeffffff]
 1077.917339 (    0.008666)| [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000807fffffff]
 1077.944326 (    0.026987)| [    0.000000] Initmem setup node 1 [mem 0x0000010000000000-0x0000017bfeffffff]
 1077.960590 (    0.016264)| [    0.000000] Initmem setup node 2 [mem 0x0000020000000000-0x0000027bfeffffff]
 1077.976422 (    0.015832)| [    0.000000] Initmem setup node 3 [mem 0x0000030000000000-0x0000037bfeffffff]
 1077.992957 (    0.016535)| [    0.000000] Initmem setup node 4 [mem 0x0000040000000000-0x0000047bfeffffff]
 1078.013201 (    0.020244)| [    0.000000] Initmem setup node 5 [mem 0x0000050000000000-0x0000057bfeffffff]
 1078.037070 (    0.023869)| [    0.000000] Initmem setup node 6 [mem 0x0000060000000000-0x0000067bfeffffff]
 1078.060445 (    0.023375)| [    0.000000] Initmem setup node 7 [mem 0x0000070000000000-0x0000077bfeffffff]
 1078.080623 (    0.020178)| [    0.000000] Initmem setup node 8 [mem 0x0000080000000000-0x0000087bfeffffff]
 1078.104569 (    0.023946)| [    0.000000] Initmem setup node 9 [mem 0x0000090000000000-0x0000097bfeffffff]
 1078.128244 (    0.023675)| [    0.000000] Initmem setup node 10 [mem 0x00000a0000000000-0x00000a7bfeffffff]
 1078.148396 (    0.020152)| [    0.000000] Initmem setup node 11 [mem 0x00000b0000000000-0x00000b7bfeffffff]
 1078.172279 (    0.023883)| [    0.000000] Initmem setup node 12 [mem 0x00000c0000000000-0x00000c7bfeffffff]
 1078.196191 (    0.023912)| [    0.000000] Initmem setup node 13 [mem 0x00000d0000000000-0x00000d7bfeffffff]
 1078.220328 (    0.024137)| [    0.000000] Initmem setup node 14 [mem 0x00000e0000000000-0x00000e7bfeffffff]
 1078.240727 (    0.020399)| [    0.000000] Initmem setup node 15 [mem 0x00000f0000000000-0x00000f7bfeffffff]
 1078.264287 (    0.023560)| [    0.000000] Initmem setup node 16 [mem 0x0000100000000000-0x0000107bfeffffff]
 1078.288579 (    0.024292)| [    0.000000] Initmem setup node 17 [mem 0x0000110000000000-0x0000117bfeffffff]
 1078.308143 (    0.019564)| [    0.000000] Initmem setup node 18 [mem 0x0000120000000000-0x0000127bfeffffff]
 1078.332258 (    0.024115)| [    0.000000] Initmem setup node 19 [mem 0x0000130000000000-0x0000137bfeffffff]
 1078.356824 (    0.024566)| [    0.000000] Initmem setup node 20 [mem 0x0000140000000000-0x0000147bfeffffff]
 1078.380407 (    0.023583)| [    0.000000] Initmem setup node 21 [mem 0x0000150000000000-0x0000157bfeffffff]
 1078.400322 (    0.019915)| [    0.000000] Initmem setup node 22 [mem 0x0000160000000000-0x0000167bfeffffff]
 1078.424815 (    0.024493)| [    0.000000] Initmem setup node 23 [mem 0x0000170000000000-0x0000177bfeffffff]
 1078.448385 (    0.023570)| [    0.000000] Initmem setup node 24 [mem 0x0000180000000000-0x0000187bfeffffff]
 1078.468274 (    0.019889)| [    0.000000] Initmem setup node 25 [mem 0x0000190000000000-0x0000197bfeffffff]
 1078.492480 (    0.024206)| [    0.000000] Initmem setup node 26 [mem 0x00001a0000000000-0x00001a7bfeffffff]
 1078.516618 (    0.024138)| [    0.000000] Initmem setup node 27 [mem 0x00001b0000000000-0x00001b7bfeffffff]
 1078.542553 (    0.025935)| [    0.000000] Initmem setup node 28 [mem 0x00001c0000000000-0x00001c7bfeffffff]
 1078.560505 (    0.017952)| [    0.000000] Initmem setup node 29 [mem 0x00001d0000000000-0x00001d7bfeffffff]
 1078.584483 (    0.023978)| [    0.000000] Initmem setup node 30 [mem 0x00001e0000000000-0x00001e7bfeffffff]
 1078.608453 (    0.023970)| [    0.000000] Initmem setup node 31 [mem 0x00001f0000000000-0x00001f7bfeffffff]
 1078.632351 (    0.023898)| [    0.000000] tboot: non-0 tboot_addr but it is not of type E820_RESERVED
 1078.640422 (    0.008071)| [    0.000000] ACPI: PM-Timer IO Port: 0x408
 1078.644641 (    0.004219)| [    0.000000] UV: rev:1 part#:9578 nodeid:0000 n_skt:6 pnmsk:3f pn:0
 1078.648597 (    0.003956)| [    0.000000] UV: apicid_shift:0 apicid_mask:0x3f
 1078.656380 (    0.007783)| [    0.000000] UV: socketid_shift:6 pnode_mask:0x3f
 1078.660369 (    0.003989)| [    0.000000] UV: OEM IDs SGI3X/UVX, System/HUB Types 2/6, uv_apic 0
 1078.668460 (    0.008091)| [    0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
 1078.676337 (    0.007877)| [    0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
 1078.684482 (    0.008145)| [    0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec02000, GSI 48-71
 1078.692462 (    0.007980)| [    0.000000] IOAPIC[3]: apic_id 11, version 32, address 0xfec03000, GSI 72-95
 1078.700358 (    0.007896)| [    0.000000] IOAPIC[4]: apic_id 12, version 32, address 0xfec04000, GSI 96-119
 1078.708529 (    0.008171)| [    0.000000] IOAPIC[5]: apic_id 13, version 32, address 0xfec05000, GSI 120-143
 1078.716554 (    0.008025)| [    0.000000] IOAPIC[6]: apic_id 14, version 32, address 0xfec06000, GSI 144-167
 1078.724318 (    0.007764)| [    0.000000] IOAPIC[7]: apic_id 15, version 32, address 0xfec07000, GSI 168-191
 1078.733018 (    0.008700)| [    0.000000] IOAPIC[8]: apic_id 16, version 32, address 0xfec08000, GSI 192-215
 1078.740537 (    0.007519)| [    0.000000] IOAPIC[9]: apic_id 17, version 32, address 0xfec09000, GSI 216-239
 1078.748869 (    0.008332)| [    0.000000] IOAPIC[10]: apic_id 18, version 32, address 0xfec0a000, GSI 240-263
 1078.760593 (    0.011724)| [    0.000000] IOAPIC[11]: apic_id 19, version 32, address 0xfec0b000, GSI 264-287
 1078.764669 (    0.004076)| [    0.000000] IOAPIC[12]: apic_id 20, version 32, address 0xfec0c000, GSI 288-311
 1078.772548 (    0.007879)| [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
 1078.780584 (    0.008036)| [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
 1078.788496 (    0.007912)| [    0.000000] Using ACPI (MADT) for SMP configuration information
 1078.792586 (    0.004090)| [    0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
 1078.800660 (    0.008074)| [    0.000000] smpboot: Allowing 1536 CPUs, 0 hotplug CPUs
 1078.805486 (    0.004826)| [    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
 1078.812401 (    0.006915)| [    0.000000] PM: Registered nosave memory: [mem 0x00052000-0x0005afff]
 1078.820543 (    0.008142)| [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
 1078.824724 (    0.004181)| [    0.000000] PM: Registered nosave memory: [mem 0x61ea2000-0x620c9fff]
 1078.832411 (    0.007687)| [    0.000000] PM: Registered nosave memory: [mem 0x622e5000-0x62734fff]
 1078.840351 (    0.007940)| [    0.000000] PM: Registered nosave memory: [mem 0x7282a000-0x7282afff]
 1078.848461 (    0.008110)| [    0.000000] PM: Registered nosave memory: [mem 0x72838000-0x72a77fff]
 1078.856388 (    0.007927)| [    0.000000] PM: Registered nosave memory: [mem 0x751ff000-0x75efefff]
 1078.860483 (    0.004095)| [    0.000000] PM: Registered nosave memory: [mem 0x75eff000-0x76efefff]
 1078.868734 (    0.008251)| [    0.000000] PM: Registered nosave memory: [mem 0x76eff000-0x76ffefff]
 1078.876220 (    0.007486)| [    0.000000] PM: Registered nosave memory: [mem 0x7f000000-0x7fffffff]
 1078.883977 (    0.007757)| [    0.000000] PM: Registered nosave memory: [mem 0x80000000-0x8fffffff]
 1078.894171 (    0.010194)| [    0.000000] PM: Registered nosave memory: [mem 0x90000000-0xf7ffffff]
 1078.896900 (    0.002729)| [    0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff]
 1078.904482 (    0.007582)| [    0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
 1078.913069 (    0.008587)| [    0.000000] PM: Registered nosave memory: [mem 0x7c00000000-0x7fffffffff]
 1078.920594 (    0.007525)| [    0.000000] PM: Registered nosave memory: [mem 0x8080000000-0xffffffffff]
 1078.928767 (    0.008173)| [    0.000000] PM: Registered nosave memory: [mem 0x17bff000000-0x1ffffffffff]
 1078.938580 (    0.009813)| [    0.000000] PM: Registered nosave memory: [mem 0x27bff000000-0x2ffffffffff]
 1078.950329 (    0.011749)| [    0.000000] PM: Registered nosave memory: [mem 0x37bff000000-0x3ffffffffff]
 1078.953115 (    0.002786)| [    0.000000] PM: Registered nosave memory: [mem 0x47bff000000-0x4ffffffffff]
 1078.960632 (    0.007517)| [    0.000000] PM: Registered nosave memory: [mem 0x57bff000000-0x5ffffffffff]
 1078.968393 (    0.007761)| [    0.000000] PM: Registered nosave memory: [mem 0x67bff000000-0x6ffffffffff]
 1078.972777 (    0.004384)| [    0.000000] PM: Registered nosave memory: [mem 0x77bff000000-0x7ffffffffff]
 1078.980761 (    0.007984)| [    0.000000] PM: Registered nosave memory: [mem 0x87bff000000-0x8ffffffffff]
 1078.988464 (    0.007703)| [    0.000000] PM: Registered nosave memory: [mem 0x97bff000000-0x9ffffffffff]
 1078.996414 (    0.007950)| [    0.000000] PM: Registered nosave memory: [mem 0xa7bff000000-0xaffffffffff]
 1079.004346 (    0.007932)| [    0.000000] PM: Registered nosave memory: [mem 0xb7bff000000-0xbffffffffff]
 1079.012457 (    0.008111)| [    0.000000] PM: Registered nosave memory: [mem 0xc7bff000000-0xcffffffffff]
 1079.021716 (    0.009259)| [    0.000000] PM: Registered nosave memory: [mem 0xd7bff000000-0xdffffffffff]
 1079.028873 (    0.007157)| [    0.000000] PM: Registered nosave memory: [mem 0xe7bff000000-0xeffffffffff]
 1079.036710 (    0.007837)| [    0.000000] PM: Registered nosave memory: [mem 0xf7bff000000-0xfffffffffff]
 1079.044390 (    0.007680)| [    0.000000] PM: Registered nosave memory: [mem 0x107bff000000-0x10ffffffffff]
 1079.052432 (    0.008042)| [    0.000000] PM: Registered nosave memory: [mem 0x117bff000000-0x11ffffffffff]
 1079.060690 (    0.008258)| [    0.000000] PM: Registered nosave memory: [mem 0x127bff000000-0x12ffffffffff]
 1079.070248 (    0.009558)| [    0.000000] PM: Registered nosave memory: [mem 0x137bff000000-0x13ffffffffff]
 1079.076531 (    0.006283)| [    0.000000] PM: Registered nosave memory: [mem 0x147bff000000-0x14ffffffffff]
 1079.084493 (    0.007962)| [    0.000000] PM: Registered nosave memory: [mem 0x157bff000000-0x15ffffffffff]
 1079.092716 (    0.008223)| [    0.000000] PM: Registered nosave memory: [mem 0x167bff000000-0x16ffffffffff]
 1079.101909 (    0.009193)| [    0.000000] PM: Registered nosave memory: [mem 0x177bff000000-0x17ffffffffff]
 1079.109741 (    0.007832)| [    0.000000] PM: Registered nosave memory: [mem 0x187bff000000-0x18ffffffffff]
 1079.116717 (    0.006976)| [    0.000000] PM: Registered nosave memory: [mem 0x197bff000000-0x19ffffffffff]
 1079.124926 (    0.008209)| [    0.000000] PM: Registered nosave memory: [mem 0x1a7bff000000-0x1affffffffff]
 1079.132665 (    0.007739)| [    0.000000] PM: Registered nosave memory: [mem 0x1b7bff000000-0x1bffffffffff]
 1079.141019 (    0.008354)| [    0.000000] PM: Registered nosave memory: [mem 0x1c7bff000000-0x1cffffffffff]
 1079.148525 (    0.007506)| [    0.000000] PM: Registered nosave memory: [mem 0x1d7bff000000-0x1dffffffffff]
 1079.156472 (    0.007947)| [    0.000000] PM: Registered nosave memory: [mem 0x1e7bff000000-0x1effffffffff]
 1079.164470 (    0.007998)| [    0.000000] e820: [mem 0x90000000-0xf7ffffff] available for PCI devices
 1079.172450 (    0.007980)| [    0.000000] Booting paravirtualized kernel on bare hardware
 1079.176809 (    0.004359)| [    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
 1079.189101 (    0.012292)| [    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1536 nr_cpu_ids:1536 nr_node_ids:32
 1079.405985 (    0.216884)| [    0.000000] percpu: max_distance=0x1ef98ec00000 too large for vmalloc space 0x1fffffffffff
 1079.412485 (    0.006500)| [    0.000000] setup_percpu: auto allocator failed (-22), falling back to page size
 1079.740769 (    0.328284)| [    0.000000] percpu: 30 4K pages/cpu @ffffc90000000000 s99928 r8192 d14760
 1079.753279 (    0.012510)| [    0.000000] Built 32 zonelists in Node order, mobility grouping on.  Total pages: 4095533409
 1079.765407 (    0.012128)| [    0.000000] Policy zone: Normal
 1079.765435 (    0.000028)| [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-rc5-clean+ root=UUID=5916572d-2a57-447a-bf73-63cdf13310dd ro nowatchdog nortsched udev.children-max=512 add_efi_memmap intel_idle.max_cstate=1 crashkernel=450M nobau rcupdate.rcu_cpu_stall_suppress=1 processor.max_cstate=1 video=efifb:off cpuidle_sysfs_switch numa_balancing=disable relax_domain_level=2 pcie_aspm=on earlyprintk=ttyS0,115200 skew_tick=1 log_buf_len=8M pci=hpiosize=0,hpmemsize=0,nobar cgroup_disable=memory kgdboc=ttyS0,115200n8 console=ttyS0,115200 resume=/dev/sda3 splash=silent showopts
 1079.824668 (    0.059233)| [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
 1086.821870 (    6.997202)| [    0.000000] Memory: 68385804K/16642433908K available (6346K kernel code, 1399K rwdata, 3060K rodata, 2272K init, 10520K bss, 261148632K reserved, 0K cma-reserved)
 1086.870128 (    0.048258)| [    0.000000] Hierarchical RCU implementation.
 1086.874693 (    0.004565)| [    0.000000] 	Build-time adjustment of leaf fanout to 64.
 1086.878407 (    0.003714)| [    0.000000] 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1536.
 1086.886010 (    0.007603)| [    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1536
 1086.905827 (    0.019817)| [    0.000000] NR_IRQS:524544 nr_irqs:17608 16
 1086.973804 (    0.067977)| [    0.000000] Console: colour dummy device 80x25
 1086.977754 (    0.003950)| [    0.000000] console [ttyS0] enabled
 1086.981888 (    0.004134)| [    0.000000] console [ttyS0] enabled
 1086.985886 (    0.003998)| [    0.000000] bootconsole [earlyser0] disabled
 1086.993337 (    0.007451)| [    0.000000] bootconsole [earlyser0] disabled
 1087.030180 (    0.036843)| [    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
 1087.070106 (    0.039926)| [    0.000000] tsc: Fast TSC calibration using PIT
 1087.073752 (    0.003646)| [    0.000000] tsc: Detected 2199.981 MHz processor
 1087.081772 (    0.008020)| [    0.001724] Calibrating delay loop (skipped), value calculated using timer frequency.. 4399.96 BogoMIPS (lpj=8799924)
 1087.094061 (    0.012289)| [    0.013587] pid_max: default: 1572864 minimum: 12288
 1087.106036 (    0.011975)| [    0.025062] ACPI: Core revision 20160831
 1087.942150 (    0.836114)| [    0.864321] ACPI: 2 ACPI AML tables successfully acquired and loaded
 1087.990647 (    0.048497)| [    0.912028] Security Framework initialized
 1087.996222 (    0.005575)| [    0.916614] AppArmor: AppArmor initialized
 1088.114046 (    0.117824)| [    1.034295] random: fast init done
 1090.038392 (    1.924346)| [    2.960466] Dentry cache hash table entries: 2147483648 (order: 22, 17179869184 bytes)
 1099.684521 (    9.646129)| [   12.604001] Inode-cache hash table entries: 1073741824 (order: 21, 8589934592 bytes)
 1104.033910 (    4.349389)| [   16.952226] Mount-cache hash table entries: 33554432 (order: 16, 268435456 bytes)
 1104.069240 (    0.035330)| [   16.988733] Mountpoint-cache hash table entries: 33554432 (order: 16, 268435456 bytes)
 1104.365135 (    0.295895)| [   17.283969] Disabling memory control group subsystem
 1104.373156 (    0.008021)| [   17.292816] CPU: Physical Processor ID: 0
 1104.377359 (    0.004203)| [   17.297299] CPU: Processor Core ID: 0
 1104.380915 (    0.003556)| [   17.301394] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
 1104.391118 (    0.010203)| [   17.308104] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
 1104.396829 (    0.005711)| [   17.315981] mce: CPU supports 22 MCE banks
 1104.400902 (    0.004073)| [   17.320578] CPU0: Thermal monitoring enabled (TM1)
 1104.404840 (    0.003938)| [   17.325985] process: using mwait in idle threads
 1104.410533 (    0.005693)| [   17.331148] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
 1104.417091 (    0.006558)| [   17.337180] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
 1104.424908 (    0.007817)| [   17.344633] Freeing SMP alternatives memory: 24K (ffffffff81f97000 - ffffffff81f9d000)
 1104.444991 (    0.020083)| [   17.365573] ftrace: allocating 25434 entries in 100 pages
 1104.465219 (    0.020228)| [   17.386668] smpboot: APIC(0) Converting physical 0 to logical package 0
 1104.473304 (    0.008085)| [   17.394062] smpboot: APIC(40) Converting physical 1 to logical package 1
 1104.480949 (    0.007645)| [   17.401554] smpboot: APIC(80) Converting physical 2 to logical package 2
 1104.489360 (    0.008411)| [   17.409036] smpboot: APIC(c0) Converting physical 3 to logical package 3
 1104.496949 (    0.007589)| [   17.416519] smpboot: APIC(100) Converting physical 4 to logical package 4
 1104.505785 (    0.008836)| [   17.424117] smpboot: APIC(140) Converting physical 5 to logical package 5
 1104.511178 (    0.005393)| [   17.431705] smpboot: APIC(180) Converting physical 6 to logical package 6
 1104.518776 (    0.007598)| [   17.439292] smpboot: APIC(1c0) Converting physical 7 to logical package 7
 1104.524574 (    0.005798)| [   17.446887] smpboot: APIC(200) Converting physical 8 to logical package 8
 1104.532553 (    0.007979)| [   17.454484] smpboot: APIC(240) Converting physical 9 to logical package 9
 1104.541485 (    0.008932)| [   17.462081] smpboot: APIC(280) Converting physical 10 to logical package 10
 1104.549574 (    0.008089)| [   17.469862] smpboot: APIC(2c0) Converting physical 11 to logical package 11
 1104.557783 (    0.008209)| [   17.477650] smpboot: APIC(300) Converting physical 12 to logical package 12
 1104.567403 (    0.009620)| [   17.485439] smpboot: APIC(340) Converting physical 13 to logical package 13
 1104.572936 (    0.005533)| [   17.493229] smpboot: APIC(380) Converting physical 14 to logical package 14
 1104.580919 (    0.007983)| [   17.501019] smpboot: APIC(3c0) Converting physical 15 to logical package 15
 1104.589146 (    0.008227)| [   17.508802] smpboot: APIC(400) Converting physical 16 to logical package 16
 1104.598256 (    0.009110)| [   17.516583] smpboot: APIC(440) Converting physical 17 to logical package 17
 1104.604955 (    0.006699)| [   17.524373] smpboot: APIC(480) Converting physical 18 to logical package 18
 1104.612981 (    0.008026)| [   17.532163] smpboot: APIC(4c0) Converting physical 19 to logical package 19
 1104.616900 (    0.003919)| [   17.539945] smpboot: APIC(500) Converting physical 20 to logical package 20
 1104.624889 (    0.007989)| [   17.547734] smpboot: APIC(540) Converting physical 21 to logical package 21
 1104.633332 (    0.008443)| [   17.555521] smpboot: APIC(580) Converting physical 22 to logical package 22
 1104.641183 (    0.007851)| [   17.563311] smpboot: APIC(5c0) Converting physical 23 to logical package 23
 1104.649025 (    0.007842)| [   17.571099] smpboot: APIC(600) Converting physical 24 to logical package 24
 1104.656985 (    0.007960)| [   17.578890] smpboot: APIC(640) Converting physical 25 to logical package 25
 1104.665229 (    0.008244)| [   17.586681] smpboot: APIC(680) Converting physical 26 to logical package 26
 1104.673132 (    0.007903)| [   17.594464] smpboot: APIC(6c0) Converting physical 27 to logical package 27
 1104.681004 (    0.007872)| [   17.602247] smpboot: APIC(700) Converting physical 28 to logical package 28
 1104.689011 (    0.008007)| [   17.610036] smpboot: APIC(740) Converting physical 29 to logical package 29
 1104.697181 (    0.008170)| [   17.617825] smpboot: APIC(780) Converting physical 30 to logical package 30
 1104.705118 (    0.007937)| [   17.625616] smpboot: APIC(7c0) Converting physical 31 to logical package 31
 1104.713202 (    0.008084)| [   17.633737] smpboot: Max logical packages: 64
 1104.716940 (    0.003738)| [   17.639309] DMAR: Host address width 46
 1104.721144 (    0.004204)| [   17.643597] DMAR: DRHD base: 0x000000f6020000 flags: 0x1
 1104.729017 (    0.007873)| [   17.649548] DMAR: dmar0: reg_base_addr f6020000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.737025 (    0.008008)| [   17.658490] DMAR: DRHD base: 0x000000f6040000 flags: 0x1
 1104.745160 (    0.008135)| [   17.664442] DMAR: dmar1: reg_base_addr f6040000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.752996 (    0.007836)| [   17.673385] DMAR: DRHD base: 0x000000f6060000 flags: 0x1
 1104.757118 (    0.004122)| [   17.679335] DMAR: dmar2: reg_base_addr f6060000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.769392 (    0.012274)| [   17.688276] DMAR: DRHD base: 0x000000f6080000 flags: 0x1
 1104.773005 (    0.003613)| [   17.694234] DMAR: dmar3: reg_base_addr f6080000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.782273 (    0.009268)| [   17.703178] DMAR: DRHD base: 0x000000f60a0000 flags: 0x1
 1104.789363 (    0.007090)| [   17.709124] DMAR: dmar4: reg_base_addr f60a0000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.797839 (    0.008476)| [   17.718066] DMAR: DRHD base: 0x000000f60c0000 flags: 0x1
 1104.801972 (    0.004133)| [   17.724011] DMAR: dmar5: reg_base_addr f60c0000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.813553 (    0.011581)| [   17.732953] DMAR: DRHD base: 0x000000f60e0000 flags: 0x1
 1104.817531 (    0.003978)| [   17.738901] DMAR: dmar6: reg_base_addr f60e0000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.825429 (    0.007898)| [   17.747844] DMAR: DRHD base: 0x000000f6100000 flags: 0x1
 1104.833123 (    0.007694)| [   17.753789] DMAR: dmar7: reg_base_addr f6100000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.841103 (    0.007980)| [   17.762731] DMAR: DRHD base: 0x000000f6120000 flags: 0x1
 1104.850291 (    0.009188)| [   17.768676] DMAR: dmar8: reg_base_addr f6120000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.857681 (    0.007390)| [   17.777617] DMAR: DRHD base: 0x000000f6140000 flags: 0x1
 1104.860931 (    0.003250)| [   17.783572] DMAR: dmar9: reg_base_addr f6140000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.873089 (    0.012158)| [   17.792515] DMAR: DRHD base: 0x000000f6160000 flags: 0x1
 1104.879851 (    0.006762)| [   17.798462] DMAR: dmar10: reg_base_addr f6160000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.885525 (    0.005674)| [   17.807501] DMAR: DRHD base: 0x000000f6180000 flags: 0x1
 1104.894980 (    0.009455)| [   17.813444] DMAR: dmar11: reg_base_addr f6180000 ver 1:0 cap 8d2078c106f0466 ecap f020df
 1104.901175 (    0.006195)| [   17.822483] DMAR: RMRR base: 0x00000075efa000 end: 0x00000075efcfff
 1104.908999 (    0.007824)| [   17.829486] DMAR: ATSR flags: 0x0
 1104.913133 (    0.004134)| [   17.833192] DMAR: ATSR flags: 0x0
 1104.917083 (    0.003950)| [   17.836897] DMAR: ATSR flags: 0x0
 1104.920970 (    0.003887)| [   17.840604] DMAR: ATSR flags: 0x0
 1104.926331 (    0.005361)| [   17.844313] DMAR: ATSR flags: 0x0
 1104.926358 (    0.000027)| [   17.848019] DMAR: ATSR flags: 0x0
 1104.929122 (    0.002764)| [   17.851726] DMAR: ATSR flags: 0x0
 1104.933384 (    0.004262)| [   17.855435] DMAR: ATSR flags: 0x0
 1104.937001 (    0.003617)| [   17.859142] DMAR: ATSR flags: 0x0
 1104.941482 (    0.004481)| [   17.862848] DMAR: ATSR flags: 0x0
 1104.945117 (    0.003635)| [   17.866554] DMAR: ATSR flags: 0x0
 1104.949000 (    0.003883)| [   17.870260] DMAR: ATSR flags: 0x0
 1104.953019 (    0.004019)| [   17.873959] DMAR: RHSA base: 0x000000f6020000 proximity domain: 0x0
 1104.962870 (    0.009851)| [   17.880959] DMAR: RHSA base: 0x000000f6040000 proximity domain: 0x1
 1104.965912 (    0.003042)| [   17.887961] DMAR: RHSA base: 0x000000f6060000 proximity domain: 0x2
 1104.973470 (    0.007558)| [   17.894964] DMAR: RHSA base: 0x000000f6080000 proximity domain: 0x4
 1104.981683 (    0.008213)| [   17.901968] DMAR: RHSA base: 0x000000f60a0000 proximity domain: 0x5
 1104.989553 (    0.007870)| [   17.908971] DMAR: RHSA base: 0x000000f60c0000 proximity domain: 0x6
 1104.993369 (    0.003816)| [   17.915973] DMAR: RHSA base: 0x000000f60e0000 proximity domain: 0x8
 1105.001530 (    0.008161)| [   17.922974] DMAR: RHSA base: 0x000000f6100000 proximity domain: 0x9
 1105.009312 (    0.007782)| [   17.929976] DMAR: RHSA base: 0x000000f6120000 proximity domain: 0xa
 1105.016980 (    0.007668)| [   17.936979] DMAR: RHSA base: 0x000000f6140000 proximity domain: 0xc
 1105.024123 (    0.007143)| [   17.943980] DMAR: RHSA base: 0x000000f6160000 proximity domain: 0xd
 1105.029117 (    0.004994)| [   17.950981] DMAR: RHSA base: 0x000000f6180000 proximity domain: 0xe
 1105.037044 (    0.007927)| [   17.957985] DMAR-IR: IOAPIC id 8 under DRHD base  0xf6020000 IOMMU 0
 1105.045077 (    0.008033)| [   17.965085] DMAR-IR: IOAPIC id 9 under DRHD base  0xf6020000 IOMMU 0
 1105.053006 (    0.007929)| [   17.972184] DMAR-IR: IOAPIC id 10 under DRHD base  0xf6040000 IOMMU 1
 1105.057308 (    0.004302)| [   17.979382] DMAR-IR: IOAPIC id 11 under DRHD base  0xf6060000 IOMMU 2
 1105.066551 (    0.009243)| [   17.986577] DMAR-IR: IOAPIC id 12 under DRHD base  0xf6080000 IOMMU 3
 1105.073480 (    0.006929)| [   17.993771] DMAR-IR: IOAPIC id 13 under DRHD base  0xf60a0000 IOMMU 4
 1105.083688 (    0.010208)| [   18.000966] DMAR-IR: IOAPIC id 14 under DRHD base  0xf60c0000 IOMMU 5
 1105.089256 (    0.005568)| [   18.008163] DMAR-IR: IOAPIC id 15 under DRHD base  0xf60e0000 IOMMU 6
 1105.093064 (    0.003808)| [   18.015359] DMAR-IR: IOAPIC id 16 under DRHD base  0xf6100000 IOMMU 7
 1105.101423 (    0.008359)| [   18.022554] DMAR-IR: IOAPIC id 17 under DRHD base  0xf6120000 IOMMU 8
 1105.109935 (    0.008512)| [   18.029749] DMAR-IR: IOAPIC id 18 under DRHD base  0xf6140000 IOMMU 9
 1105.117136 (    0.007201)| [   18.036946] DMAR-IR: IOAPIC id 19 under DRHD base  0xf6160000 IOMMU 10
 1105.125097 (    0.007961)| [   18.044239] DMAR-IR: IOAPIC id 20 under DRHD base  0xf6180000 IOMMU 11
 1105.129227 (    0.004130)| [   18.051533] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
 1105.148968 (    0.019741)| [   18.071201] DMAR-IR: Enabled IRQ remapping in x2apic mode
 1105.160985 (    0.012017)| [   18.083535] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
 1105.211975 (    0.050990)| [   18.130272] smpboot: CPU0: Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz (family: 0x6, model: 0x4f, stepping: 0x1)
 1105.221134 (    0.009159)| [   18.141463] UV: Found UV300 hub
 1105.224984 (    0.003850)| [   18.144977] UV: UVsystab: Revision:1
 1105.229237 (    0.004253)| [   18.148967] UV: No UVsystab socket table, ignoring
 1105.233296 (    0.004059)| [   18.154316] UV: N:6 M:40 m_shift:24 n_lshift:43
 1105.237906 (    0.004610)| [   18.159364] UV: gpa_mask/shift:0x3fffffffffff/0 pnode_mask:0x3f apic_pns:6
 1105.245452 (    0.007546)| [   18.167037] UV: mmr_base/shift:0x2ffd00000000/26 gru_base/shift:0x0/0
 1105.256942 (    0.011490)| [   18.174227] UV: gnode_upper:0x0 gnode_extra:0x0
 1105.259289 (    0.002347)| [   18.179283] UV: NODE_PRESENT_DEPTH = 16
 1105.265990 (    0.006701)| [   18.183564] UV: NODE_PRESENT(0) = 0x00000000ffffffff
 1105.276910 (    0.010920)| [   18.189108] UV: Found 32 hubs, 32 nodes, 1536 cpus
 1105.278034 (    0.001124)| [   18.196538] UV: UVHUB node: 0 pn:00 nrcpus:48
 1105.281208 (    0.003174)| [   18.201401] UV: UVHUB node: 1 pn:01 nrcpus:48
 1105.285079 (    0.003871)| [   18.206261] UV: UVHUB node: 2 pn:02 nrcpus:48
 1105.289327 (    0.004248)| [   18.211124] UV: UVHUB node: 3 pn:03 nrcpus:48
 1105.293411 (    0.004084)| [   18.215985] UV: UVHUB node: 4 pn:04 nrcpus:48
 1105.302130 (    0.008719)| [   18.220845] UV: UVHUB node: 5 pn:05 nrcpus:48
 1105.305809 (    0.003679)| [   18.225707] UV: UVHUB node: 6 pn:06 nrcpus:48
 1105.311455 (    0.005646)| [   18.230567] UV: UVHUB node: 7 pn:07 nrcpus:48
 1105.318411 (    0.006956)| [   18.235427] UV: UVHUB node: 8 pn:08 nrcpus:48
 1105.325055 (    0.006644)| [   18.240288] UV: UVHUB node: 9 pn:09 nrcpus:48
 1105.327932 (    0.002877)| [   18.245149] UV: UVHUB node:10 pn:0a nrcpus:48
 1105.336575 (    0.008643)| [   18.250009] UV: UVHUB node:11 pn:0b nrcpus:48
 1105.338620 (    0.002045)| [   18.254870] UV: UVHUB node:12 pn:0c nrcpus:48
 1105.348074 (    0.009454)| [   18.259730] UV: UVHUB node:13 pn:0d nrcpus:48
 1105.348132 (    0.000058)| [   18.264591] UV: UVHUB node:14 pn:0e nrcpus:48
 1105.352113 (    0.003981)| [   18.269452] UV: UVHUB node:15 pn:0f nrcpus:48
 1105.353919 (    0.001806)| [   18.274313] UV: UVHUB node:16 pn:10 nrcpus:48
 1105.357541 (    0.003622)| [   18.279173] UV: UVHUB node:17 pn:11 nrcpus:48
 1105.361683 (    0.004142)| [   18.284036] UV: UVHUB node:18 pn:12 nrcpus:48
 1105.373239 (    0.011556)| [   18.288896] UV: UVHUB node:19 pn:13 nrcpus:48
 1105.374222 (    0.000983)| [   18.293756] UV: UVHUB node:20 pn:14 nrcpus:48
 1105.381033 (    0.006811)| [   18.298617] UV: UVHUB node:21 pn:15 nrcpus:48
 1105.387737 (    0.006704)| [   18.303478] UV: UVHUB node:22 pn:16 nrcpus:48
 1105.394342 (    0.006605)| [   18.308338] UV: UVHUB node:23 pn:17 nrcpus:48
 1105.410702 (    0.016360)| [   18.313199] UV: UVHUB node:24 pn:18 nrcpus:48
 1105.414792 (    0.004090)| [   18.318059] UV: UVHUB node:25 pn:19 nrcpus:48
 1105.420521 (    0.005729)| [   18.322920] UV: UVHUB node:26 pn:1a nrcpus:48
 1105.423266 (    0.002745)| [   18.327782] UV: UVHUB node:27 pn:1b nrcpus:48
 1105.423297 (    0.000031)| [   18.332643] UV: UVHUB node:28 pn:1c nrcpus:48
 1105.428532 (    0.005235)| [   18.337505] UV: UVHUB node:29 pn:1d nrcpus:48
 1105.432056 (    0.003524)| [   18.342365] UV: UVHUB node:30 pn:1e nrcpus:48
 1105.432087 (    0.000031)| [   18.347228] UV: UVHUB node:31 pn:1f nrcpus:48
 1105.434915 (    0.002828)| [   18.352090] UV: min_pnode:00 max_pnode:1f
 1105.441619 (    0.006704)| [   18.356672] UV: MMIOH0 overlay 0x8006affe00000000 base:0xbff80 m_io:26
 1105.447408 (    0.005789)| [   18.363960] UV: MMIOH0[000..000] NASID 0x0000 ADDR 0x00002ffe00000000 - 0x00002ffe04000000
 1105.464565 (    0.017157)| [   18.373187] UV: MMIOH0[001..001] NASID 0x0002 ADDR 0x00002ffe04000000 - 0x00002ffe08000000
 1105.469968 (    0.005403)| [   18.382411] UV: MMIOH0[002..002] NASID 0x0004 ADDR 0x00002ffe08000000 - 0x00002ffe0c000000
 1105.476807 (    0.006839)| [   18.391637] UV: MMIOH0[003..003] NASID 0x0008 ADDR 0x00002ffe0c000000 - 0x00002ffe10000000
 1105.490941 (    0.014134)| [   18.400864] UV: MMIOH0[004..004] NASID 0x000a ADDR 0x00002ffe10000000 - 0x00002ffe14000000
 1105.499586 (    0.008645)| [   18.410091] UV: MMIOH0[005..005] NASID 0x000c ADDR 0x00002ffe14000000 - 0x00002ffe18000000
 1105.505470 (    0.005884)| [   18.419316] UV: MMIOH0[006..006] NASID 0x0010 ADDR 0x00002ffe18000000 - 0x00002ffe1c000000
 1105.528649 (    0.023179)| [   18.428543] UV: MMIOH0[007..007] NASID 0x0012 ADDR 0x00002ffe1c000000 - 0x00002ffe20000000
 1105.540855 (    0.012206)| [   18.437769] UV: MMIOH0[008..008] NASID 0x0014 ADDR 0x00002ffe20000000 - 0x00002ffe24000000
 1105.552146 (    0.011291)| [   18.446995] UV: MMIOH0[009..009] NASID 0x0018 ADDR 0x00002ffe24000000 - 0x00002ffe28000000
 1105.564900 (    0.012754)| [   18.456220] UV: MMIOH0[010..010] NASID 0x001a ADDR 0x00002ffe28000000 - 0x00002ffe2c000000
 1105.577262 (    0.012362)| [   18.465446] UV: MMIOH0[011..011] NASID 0x001c ADDR 0x00002ffe2c000000 - 0x00002ffe30000000
 1105.583191 (    0.005929)| [   18.474694] UV: MMIOH0[012..127] NASID 0x0000 ADDR 0x00002ffe30000000 - 0x0000300000000000
 1105.591195 (    0.008004)| [   18.483919] UV: MMIOH0 base:0xbff80 shift:26 M_IO:26 MAX_IO:127
 1105.600334 (    0.009139)| [   18.490604] UV: MMIOH1 overlay 0x8009700000000000 base:0xc0000 m_io:37
 1105.605850 (    0.005516)| [   18.497916] UV: MMIOH1[000..127] NASID 0x0000 ADDR 0x0000300000000000 - 0x0000300000001000
 1105.618475 (    0.012625)| [   18.507142] UV: MMIOH1 base:0xc0000 shift:26 M_IO:37 MAX_IO:127
 1105.779214 (    0.160739)| [   18.657622] UV: SMI NMI support: EXTIO_INT0
 1105.784437 (    0.005223)| [   18.662301] UV: NMI hub list @ 0xffff9f004eb66000 (8192)
 1105.789556 (    0.005119)| [   18.669131] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
 1105.805985 (    0.016429)| [   18.680665] ... version:                3
 1105.807267 (    0.001282)| [   18.685140] ... bit width:              48
 1105.813421 (    0.006154)| [   18.689709] ... generic registers:      4
 1105.818949 (    0.005528)| [   18.694182] ... value mask:             0000ffffffffffff
 1105.821873 (    0.002924)| [   18.700110] ... max period:             0000ffffffffffff
 1105.830831 (    0.008958)| [   18.706039] ... fixed-purpose events:   3
 1105.965675 (    0.134844)| [   18.710512] ... event mask:             000000070000000f
 1105.965721 (    0.000046)| [   18.797444] x86: Booting SMP configuration:
 1105.965751 (    0.000030)| [   18.802113] .... node   #0, CPUs:            #1    #2    #3    #4    #5    #6    #7    #8    #9   #10   #11   #12   #13   #14   #15   #16   #17   #18   #19   #20   #21   #22   #23
 1107.943835 (    1.978084)| [   20.833113] .... node   #1, CPUs:     #24   #25   #26   #27   #28   #29   #30   #31   #32   #33   #34   #35   #36   #37   #38   #39   #40   #41   #42   #43   #44   #45   #46   #47
 1110.016260 (    2.072425)| [   22.926988] .... node   #2, CPUs:     #48   #49   #50   #51   #52   #53   #54   #55   #56   #57   #58   #59   #60   #61   #62   #63   #64   #65   #66   #67   #68   #69   #70   #71
 1112.114005 (    2.097745)| [   25.033504] .... node   #3, CPUs:     #72   #73   #74   #75   #76   #77   #78   #79   #80   #81   #82   #83   #84   #85   #86   #87   #88   #89   #90   #91   #92   #93   #94   #95
 1114.222713 (    2.108708)| [   27.143948] .... node   #4, CPUs:     #96   #97   #98   #99  #100  #101  #102  #103  #104  #105  #106  #107  #108  #109  #110  #111  #112  #113  #114  #115  #116  #117  #118  #119
 1116.358929 (    2.136216)| [   29.278887] .... node   #5, CPUs:    #120  #121  #122  #123  #124  #125  #126  #127  #128  #129  #130  #131  #132  #133  #134  #135  #136  #137  #138  #139  #140  #141  #142  #143
 1118.491056 (    2.132127)| [   31.410679] .... node   #6, CPUs:    #144  #145  #146  #147  #148  #149  #150  #151  #152  #153  #154  #155  #156  #157  #158  #159  #160  #161  #162  #163  #164  #165  #166  #167
 1120.620037 (    2.128981)| [   33.542431] .... node   #7, CPUs:    #168  #169  #170  #171  #172  #173  #174  #175  #176  #177  #178  #179  #180  #181  #182  #183  #184  #185  #186  #187  #188  #189  #190  #191
 1122.752015 (    2.131978)| [   35.674297] .... node   #8, CPUs:    #192  #193  #194  #195  #196  #197  #198  #199  #200  #201  #202  #203  #204  #205  #206  #207  #208  #209  #210  #211  #212  #213  #214  #215
 1124.888391 (    2.136376)| [   37.809908] .... node   #9, CPUs:    #216  #217  #218  #219  #220  #221  #222  #223  #224  #225  #226  #227  #228  #229  #230  #231  #232  #233  #234  #235  #236  #237  #238  #239
 1127.023762 (    2.135371)| [   39.941906] .... node  #10, CPUs:    #240  #241  #242  #243  #244  #245  #246  #247  #248  #249  #250  #251  #252  #253  #254  #255  #256  #257  #258  #259  #260  #261  #262  #263
 1129.152923 (    2.129161)| [   42.074104] .... node  #11, CPUs:    #264  #265  #266  #267  #268  #269  #270  #271  #272  #273  #274  #275  #276  #277  #278  #279  #280  #281  #282  #283  #284  #285  #286  #287
 1131.285935 (    2.133012)| [   44.206360] .... node  #12, CPUs:    #288  #289  #290  #291  #292  #293  #294  #295  #296  #297  #298  #299  #300  #301  #302  #303  #304  #305  #306  #307  #308  #309  #310  #311
 1133.423620 (    2.137685)| [   46.342104] .... node  #13, CPUs:    #312  #313  #314  #315  #316  #317  #318  #319  #320  #321  #322  #323  #324  #325  #326  #327  #328  #329  #330  #331  #332  #333  #334  #335
 1135.554841 (    2.131221)| [   48.474280] .... node  #14, CPUs:    #336  #337  #338  #339  #340  #341  #342  #343  #344  #345  #346  #347  #348  #349  #350  #351  #352  #353  #354  #355  #356  #357  #358  #359
 1137.691813 (    2.136972)| [   50.606641] .... node  #15, CPUs:    #360  #361  #362  #363  #364  #365  #366  #367  #368  #369  #370  #371  #372  #373  #374  #375  #376  #377  #378  #379  #380  #381  #382  #383
 1139.818735 (    2.126922)| [   52.738905] .... node  #16, CPUs:    #384  #385  #386  #387  #388  #389  #390  #391  #392  #393  #394  #395  #396  #397  #398  #399  #400  #401  #402  #403  #404  #405  #406  #407
 1141.955272 (    2.136537)| [   54.875430] .... node  #17, CPUs:    #408  #409  #410  #411  #412  #413  #414  #415  #416  #417  #418  #419  #420  #421  #422  #423  #424  #425  #426  #427  #428  #429  #430  #431
 1144.088559 (    2.133287)| [   57.008038] .... node  #18, CPUs:    #432  #433  #434  #435  #436  #437  #438  #439  #440  #441  #442  #443  #444  #445  #446  #447  #448  #449  #450  #451  #452  #453  #454  #455
 1146.223643 (    2.135084)| [   59.140725] .... node  #19, CPUs:    #456  #457  #458  #459  #460  #461  #462  #463  #464  #465  #466  #467  #468  #469  #470  #471  #472  #473  #474  #475  #476  #477  #478  #479
 1148.352056 (    2.128413)| [   61.273414] .... node  #20, CPUs:    #480  #481  #482  #483  #484  #485  #486  #487  #488  #489  #490  #491  #492  #493  #494  #495  #496  #497  #498  #499  #500  #501  #502  #503
 1150.488436 (    2.136380)| [   63.409736] .... node  #21, CPUs:    #504  #505  #506  #507  #508  #509  #510  #511  #512  #513  #514  #515  #516  #517  #518  #519  #520  #521  #522  #523  #524  #525  #526  #527
 1152.620960 (    2.132524)| [   65.542395] .... node  #22, CPUs:    #528  #529  #530  #531  #532  #533  #534  #535  #536  #537  #538  #539  #540  #541  #542  #543  #544  #545  #546  #547  #548  #549  #550  #551
 1154.754464 (    2.133504)| [   67.675078] .... node  #23, CPUs:    #552  #553  #554  #555  #556  #557  #558  #559  #560  #561  #562  #563  #564  #565  #566  #567  #568  #569  #570  #571  #572  #573  #574  #575
 1156.886450 (    2.131986)| [   69.807860] .... node  #24, CPUs:    #576  #577  #578  #579  #580  #581  #582  #583  #584  #585  #586  #587  #588  #589  #590  #591  #592  #593  #594  #595  #596  #597  #598  #599
 1159.022721 (    2.136271)| [   71.944152] .... node  #25, CPUs:    #600  #601  #602  #603  #604  #605  #606  #607  #608  #609  #610  #611  #612  #613  #614  #615  #616  #617  #618  #619  #620  #621  #622  #623
 1161.154441 (    2.131720)| [   74.076986] .... node  #26, CPUs:    #624  #625  #626  #627  #628  #629  #630  #631  #632  #633  #634  #635  #636  #637  #638  #639  #640  #641  #642  #643  #644  #645  #646  #647
 1163.287666 (    2.133225)| [   76.209707] .... node  #27, CPUs:    #648  #649  #650  #651  #652  #653  #654  #655  #656  #657  #658  #659  #660  #661  #662  #663  #664  #665  #666  #667  #668  #669  #670  #671
 1165.420380 (    2.132714)| [   78.342463] .... node  #28, CPUs:    #672  #673  #674  #675  #676  #677  #678  #679  #680  #681  #682  #683  #684  #685  #686  #687  #688  #689  #690  #691  #692  #693  #694  #695
 1167.552287 (    2.131907)| [   80.474866] .... node  #29, CPUs:    #696  #697  #698  #699  #700  #701  #702  #703  #704  #705  #706  #707  #708  #709  #710  #711  #712  #713  #714  #715  #716  #717  #718  #719
 1169.684889 (    2.132602)| [   82.607082] .... node  #30, CPUs:    #720  #721  #722  #723  #724  #725  #726  #727  #728  #729  #730  #731  #732  #733  #734  #735  #736  #737  #738  #739  #740  #741  #742  #743
 1171.817135 (    2.132246)| [   84.739137] .... node  #31, CPUs:    #744  #745  #746  #747  #748  #749  #750  #751  #752  #753  #754  #755  #756  #757  #758  #759  #760  #761  #762  #763  #764  #765  #766  #767
 1173.944376 (    2.127241)| [   86.866750] .... node   #0, CPUs:    #768  #769  #770  #771  #772  #773  #774  #775  #776  #777  #778  #779  #780  #781  #782  #783  #784  #785  #786  #787  #788  #789  #790  #791
 1176.068761 (    2.124385)| [   88.988730] .... node   #1, CPUs:    #792  #793  #794  #795  #796  #797  #798  #799  #800  #801  #802  #803  #804  #805  #806  #807  #808  #809  #810  #811  #812  #813  #814  #815
 1178.181126 (    2.112365)| [   91.104198] .... node   #2, CPUs:    #816  #817  #818  #819  #820  #821  #822  #823  #824  #825  #826  #827  #828  #829  #830  #831  #832  #833  #834  #835  #836  #837  #838  #839
 1180.298686 (    2.117560)| [   93.220202] .... node   #3, CPUs:    #840  #841  #842  #843  #844  #845  #846  #847  #848  #849  #850  #851  #852  #853  #854  #855  #856  #857  #858  #859  #860  #861  #862  #863
 1182.413284 (    2.114598)| [   95.335836] .... node   #4, CPUs:    #864  #865  #866  #867  #868  #869  #870  #871  #872  #873  #874  #875  #876  #877  #878  #879  #880  #881  #882  #883  #884  #885  #886  #887
 1184.532510 (    2.119226)| [   97.455721] .... node   #5, CPUs:    #888  #889  #890  #891  #892  #893  #894  #895  #896  #897  #898  #899  #900  #901  #902  #903  #904  #905  #906  #907  #908  #909  #910  #911
 1186.651548 (    2.119038)| [   99.571933] .... node   #6, CPUs:    #912  #913  #914  #915  #916  #917  #918  #919  #920  #921  #922  #923  #924  #925  #926  #927  #928  #929  #930  #931  #932  #933  #934  #935
 1188.767177 (    2.115629)| [  101.688172] .... node   #7, CPUs:    #936  #937  #938  #939  #940  #941  #942  #943  #944  #945  #946  #947  #948  #949  #950  #951  #952  #953  #954  #955  #956  #957  #958  #959
 1190.882319 (    2.115142)| [  103.804377] .... node   #8, CPUs:    #960  #961  #962  #963  #964  #965  #966  #967  #968  #969  #970  #971  #972  #973  #974  #975  #976  #977  #978  #979  #980  #981  #982  #983
 1193.001497 (    2.119178)| [  105.924711] .... node   #9, CPUs:    #984  #985  #986  #987  #988  #989  #990  #991  #992  #993  #994  #995  #996  #997  #998  #999 #1000 #1001 #1002 #1003 #1004 #1005 #1006 #1007
 1195.120400 (    2.118903)| [  108.041342] .... node  #10, CPUs:   #1008 #1009 #1010 #1011 #1012 #1013 #1014 #1015 #1016 #1017 #1018 #1019 #1020 #1021 #1022 #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031
 1197.236108 (    2.115708)| [  110.158006] .... node  #11, CPUs:   #1032 #1033 #1034 #1035 #1036 #1037 #1038 #1039 #1040 #1041 #1042 #1043 #1044 #1045 #1046 #1047 #1048 #1049 #1050 #1051 #1052 #1053 #1054 #1055
 1199.351128 (    2.115020)| [  112.274547] .... node  #12, CPUs:   #1056 #1057 #1058 #1059 #1060 #1061 #1062 #1063 #1064 #1065 #1066 #1067 #1068 #1069 #1070 #1071 #1072 #1073 #1074 #1075 #1076 #1077 #1078 #1079
 1201.474191 (    2.123063)| [  114.394927] .... node  #13, CPUs:   #1080 #1081 #1082 #1083 #1084 #1085 #1086 #1087 #1088 #1089 #1090 #1091 #1092 #1093 #1094 #1095 #1096 #1097 #1098 #1099 #1100 #1101 #1102 #1103
 1203.589459 (    2.115268)| [  116.511346] .... node  #14, CPUs:   #1104 #1105 #1106 #1107 #1108 #1109 #1110 #1111 #1112 #1113 #1114 #1115 #1116 #1117 #1118 #1119 #1120 #1121 #1122 #1123 #1124 #1125 #1126 #1127
 1205.705613 (    2.116154)| [  118.628111] .... node  #15, CPUs:   #1128 #1129 #1130 #1131 #1132 #1133 #1134 #1135 #1136 #1137 #1138 #1139 #1140 #1141 #1142 #1143 #1144 #1145 #1146 #1147 #1148 #1149 #1150 #1151
 1207.830410 (    2.124797)| [  120.744769] .... node  #16, CPUs:   #1152 #1153 #1154 #1155 #1156 #1157 #1158 #1159 #1160 #1161 #1162 #1163 #1164 #1165 #1166 #1167 #1168 #1169 #1170 #1171 #1172 #1173 #1174 #1175
 1209.950918 (    2.120508)| [  122.865643] .... node  #17, CPUs:   #1176 #1177 #1178 #1179 #1180 #1181 #1182 #1183 #1184 #1185 #1186 #1187 #1188 #1189 #1190 #1191 #1192 #1193 #1194 #1195 #1196 #1197 #1198 #1199
 1212.058744 (    2.107826)| [  124.982335] .... node  #18, CPUs:   #1200 #1201 #1202 #1203 #1204 #1205 #1206 #1207 #1208 #1209 #1210 #1211 #1212 #1213 #1214 #1215 #1216 #1217 #1218 #1219 #1220 #1221 #1222 #1223
 1214.179620 (    2.120876)| [  127.099123] .... node  #19, CPUs:   #1224 #1225 #1226 #1227 #1228 #1229 #1230 #1231 #1232 #1233 #1234 #1235 #1236 #1237 #1238 #1239 #1240 #1241 #1242 #1243 #1244 #1245 #1246 #1247
 1216.293639 (    2.114019)| [  129.216082] .... node  #20, CPUs:   #1248 #1249 #1250 #1251 #1252 #1253 #1254 #1255 #1256 #1257 #1258 #1259 #1260 #1261 #1262 #1263 #1264 #1265 #1266 #1267 #1268 #1269 #1270 #1271
 1218.417350 (    2.123711)| [  131.337029] .... node  #21, CPUs:   #1272 #1273 #1274 #1275 #1276 #1277 #1278 #1279 #1280 #1281 #1282 #1283 #1284 #1285 #1286 #1287 #1288 #1289 #1290 #1291 #1292 #1293 #1294 #1295
 1220.533149 (    2.115799)| [  133.453678] .... node  #22, CPUs:   #1296 #1297 #1298 #1299 #1300 #1301 #1302 #1303 #1304 #1305 #1306 #1307 #1308 #1309 #1310 #1311 #1312 #1313 #1314 #1315 #1316 #1317 #1318 #1319
 1222.647314 (    2.114165)| [  135.570562] .... node  #23, CPUs:   #1320 #1321 #1322 #1323 #1324 #1325 #1326 #1327 #1328 #1329 #1330 #1331 #1332 #1333 #1334 #1335 #1336 #1337 #1338 #1339 #1340 #1341 #1342 #1343
 1224.767453 (    2.120139)| [  137.687562] .... node  #24, CPUs:   #1344 #1345 #1346 #1347 #1348 #1349 #1350 #1351 #1352 #1353 #1354 #1355 #1356 #1357 #1358 #1359 #1360 #1361 #1362 #1363 #1364 #1365 #1366 #1367
 1226.887108 (    2.119655)| [  139.808334] .... node  #25, CPUs:   #1368 #1369 #1370 #1371 #1372 #1373 #1374 #1375 #1376 #1377 #1378 #1379 #1380 #1381 #1382 #1383 #1384 #1385 #1386 #1387 #1388 #1389 #1390 #1391
 1229.001400 (    2.114292)| [  141.925110] .... node  #26, CPUs:   #1392 #1393 #1394 #1395 #1396 #1397 #1398 #1399 #1400 #1401 #1402 #1403 #1404 #1405 #1406 #1407 #1408 #1409 #1410 #1411 #1412 #1413 #1414 #1415
 1231.121134 (    2.119734)| [  144.042112] .... node  #27, CPUs:   #1416 #1417 #1418 #1419 #1420 #1421 #1422 #1423 #1424 #1425 #1426 #1427 #1428 #1429 #1430 #1431 #1432 #1433 #1434 #1435 #1436 #1437 #1438 #1439
 1233.235975 (    2.114841)| [  146.159050] .... node  #28, CPUs:   #1440 #1441 #1442 #1443 #1444 #1445 #1446 #1447 #1448 #1449 #1450 #1451 #1452 #1453 #1454 #1455 #1456 #1457 #1458 #1459 #1460 #1461 #1462 #1463
 1235.359490 (    2.123515)| [  148.279795] .... node  #29, CPUs:   #1464 #1465 #1466 #1467 #1468 #1469 #1470 #1471 #1472 #1473 #1474 #1475 #1476 #1477 #1478 #1479 #1480 #1481 #1482 #1483 #1484 #1485 #1486 #1487
 1237.475090 (    2.115600)| [  150.396583] .... node  #30, CPUs:   #1488 #1489 #1490 #1491 #1492 #1493 #1494 #1495 #1496 #1497 #1498 #1499 #1500 #1501 #1502 #1503 #1504 #1505 #1506 #1507 #1508 #1509 #1510 #1511
 1239.589471 (    2.114381)| [  152.513170] .... node  #31, CPUs:   #1512 #1513 #1514 #1515 #1516 #1517 #1518 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1526 #1527 #1528 #1529 #1530 #1531 #1532 #1533 #1534 #1535[  154.628710] x86: Booted up 32 nodes, 1536 CPUs
 1241.712788 (    2.123317)| [  154.633995] smpboot: Total of 1536 processors activated (6767256.01 BogoMIPS)
 1267.288527 (   25.575739)| [  180.208804] node 1 initialised, 127461880 pages in 2500ms
 1267.292121 (    0.003594)| [  180.213946] node 2 initialised, 127461880 pages in 2504ms
 1267.296029 (    0.003908)| [  180.214085] node 0 initialised, 126908917 pages in 2504ms
 1267.304114 (    0.008085)| [  180.226019] node 3 initialised, 127461880 pages in 2516ms
 1268.672015 (    1.367901)| [  181.596070] node 7 initialised, 127461880 pages in 3884ms
 1268.679582 (    0.007567)| [  181.600541] node 8 initialised, 127461880 pages in 3888ms
 1268.683687 (    0.004105)| [  181.602855] node 9 initialised, 127461880 pages in 3888ms
 1268.705012 (    0.021325)| [  181.617240] node 4 initialised, 127461880 pages in 3904ms
 1268.705053 (    0.000041)| [  181.617258] node 6 initialised, 127461880 pages in 3904ms
 1268.708587 (    0.003534)| [  181.619499] node 5 initialised, 127461880 pages in 3908ms
 1268.723669 (    0.015082)| [  181.645866] node 12 initialised, 127461880 pages in 3932ms
 1268.727740 (    0.004071)| [  181.646572] node 13 initialised, 127461880 pages in 3932ms
 1268.747930 (    0.020190)| [  181.668562] node 16 initialised, 127461880 pages in 3956ms
 1268.751740 (    0.003810)| [  181.669073] node 17 initialised, 127461880 pages in 3956ms
 1268.759814 (    0.008074)| [  181.671882] node 11 initialised, 127461880 pages in 3960ms
 1268.763702 (    0.003888)| [  181.677128] node 10 initialised, 127461880 pages in 3964ms
 1268.772390 (    0.008688)| [  181.682475] node 15 initialised, 127461880 pages in 3968ms
 1268.775719 (    0.003329)| [  181.693022] node 14 initialised, 127461880 pages in 3980ms
 1268.837129 (    0.061410)| [  181.759317] node 21 initialised, 127461880 pages in 4044ms
 1268.844447 (    0.007318)| [  181.759344] node 20 initialised, 127461880 pages in 4044ms
 1268.855829 (    0.011382)| [  181.780179] node 24 initialised, 127461880 pages in 4068ms
 1268.864094 (    0.008265)| [  181.787007] node 25 initialised, 127461880 pages in 4072ms
 1268.879919 (    0.015825)| [  181.801148] node 18 initialised, 127461880 pages in 4088ms
 1268.884218 (    0.004299)| [  181.804566] node 19 initialised, 127461880 pages in 4092ms
 1268.892441 (    0.008223)| [  181.806148] node 23 initialised, 127461880 pages in 4092ms
 1268.896415 (    0.003974)| [  181.811363] node 22 initialised, 127461880 pages in 4096ms
 1268.983701 (    0.087286)| [  181.908076] node 27 initialised, 127461880 pages in 4196ms
 1268.991608 (    0.007907)| [  181.915784] node 26 initialised, 127461880 pages in 4200ms
 1269.011913 (    0.020305)| [  181.933824] node 29 initialised, 127461880 pages in 4220ms
 1269.016187 (    0.004274)| [  181.939037] node 28 initialised, 127461880 pages in 4224ms
 1269.059599 (    0.043412)| [  181.983963] node 30 initialised, 127461880 pages in 4268ms
 1269.069120 (    0.009521)| [  181.985331] node 31 initialised, 127459549 pages in 4272ms
 1270.703290 (    1.634170)| [  183.624092] devtmpfs: initialized
 1270.707723 (    0.004433)| [  183.628478] x86/mm: Memory block size: 2048MB
 1271.043484 (    0.335761)| [  183.966803] evm: security.selinux
 1271.046981 (    0.003497)| [  183.970503] evm: security.ima
 1271.050962 (    0.003981)| [  183.973813] evm: security.capability
 1271.058814 (    0.007852)| [  183.982255] PM: Registering ACPI NVS region [mem 0x75eff000-0x76efefff] (16777216 bytes)
 1271.090831 (    0.032017)| [  184.014639] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
 1271.170806 (    0.079975)| [  184.094954] pinctrl core: initialized pinctrl subsystem
 1271.179070 (    0.008264)| [  184.101890] RTC time: 20:19:04, date: 11/17/16
 1271.195015 (    0.015945)| [  184.118760] NET: Registered protocol family 16
 1271.242910 (    0.047895)| [  184.163758] cpuidle: using governor ladder
 1271.331318 (    0.088408)| [  184.252386] cpuidle: using governor menu
 1271.343398 (    0.012080)| [  184.266515] clocksource: sgi_rtc: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
 1271.355558 (    0.012160)| [  184.277786] UV RTC clocksource registered freq 200 MHz
 1271.359051 (    0.003493)| [  184.283529] ACPI: bus type PCI registered
 1271.367678 (    0.008627)| [  184.288010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
 1271.382914 (    0.015236)| [  184.305551] PCI: MMCONFIG for domain 0000 [bus 00-0b] at [mem 0x80000000-0x80bfffff] (base 0x80000000)
 1271.392033 (    0.009119)| [  184.315947] PCI: MMCONFIG for domain 0001 [bus 00-04] at [mem 0x2ffc04000000-0x2ffc044fffff] (base 0x2ffc04000000)
 1271.403097 (    0.011064)| [  184.327505] PCI: MMCONFIG for domain 0002 [bus 00-04] at [mem 0x2ffc08000000-0x2ffc084fffff] (base 0x2ffc08000000)
 1271.415365 (    0.012268)| [  184.339062] PCI: MMCONFIG for domain 0003 [bus 00-03] at [mem 0x2ffc10000000-0x2ffc103fffff] (base 0x2ffc10000000)
 1271.426982 (    0.011617)| [  184.350620] PCI: MMCONFIG for domain 0004 [bus 00-04] at [mem 0x2ffc14000000-0x2ffc144fffff] (base 0x2ffc14000000)
 1271.439794 (    0.012812)| [  184.362177] PCI: MMCONFIG for domain 0005 [bus 00-04] at [mem 0x2ffc18000000-0x2ffc184fffff] (base 0x2ffc18000000)
 1271.450873 (    0.011079)| [  184.373734] PCI: MMCONFIG for domain 0006 [bus 00-03] at [mem 0x2ffc20000000-0x2ffc203fffff] (base 0x2ffc20000000)
 1271.462870 (    0.011997)| [  184.385294] PCI: MMCONFIG for domain 0007 [bus 00-04] at [mem 0x2ffc24000000-0x2ffc244fffff] (base 0x2ffc24000000)
 1271.475414 (    0.012544)| [  184.396851] PCI: MMCONFIG for domain 0008 [bus 00-04] at [mem 0x2ffc28000000-0x2ffc284fffff] (base 0x2ffc28000000)
 1271.487277 (    0.011863)| [  184.408409] PCI: MMCONFIG for domain 0009 [bus 00-03] at [mem 0x2ffc30000000-0x2ffc303fffff] (base 0x2ffc30000000)
 1271.498819 (    0.011542)| [  184.419965] PCI: MMCONFIG for domain 000a [bus 00-04] at [mem 0x2ffc34000000-0x2ffc344fffff] (base 0x2ffc34000000)
 1271.506808 (    0.007989)| [  184.431523] PCI: MMCONFIG for domain 000b [bus 00-04] at [mem 0x2ffc38000000-0x2ffc384fffff] (base 0x2ffc38000000)
 1271.519653 (    0.012845)| [  184.443078] PCI: MMCONFIG for domain 1000 [bus 3f-3f] at [mem 0x2ffc03f00000-0x2ffc03ffffff] (base 0x2ffc00000000)
 1271.531821 (    0.012168)| [  184.454636] PCI: MMCONFIG for domain 1001 [bus 3f-3f] at [mem 0x2ffc07f00000-0x2ffc07ffffff] (base 0x2ffc04000000)
 1271.543411 (    0.011590)| [  184.466194] PCI: MMCONFIG for domain 1002 [bus 3f-3f] at [mem 0x2ffc0bf00000-0x2ffc0bffffff] (base 0x2ffc08000000)
 1271.554871 (    0.011460)| [  184.477751] PCI: MMCONFIG for domain 1003 [bus 3f-3f] at [mem 0x2ffc0ff00000-0x2ffc0fffffff] (base 0x2ffc0c000000)
 1271.567525 (    0.012654)| [  184.489308] PCI: MMCONFIG for domain 1004 [bus 3f-3f] at [mem 0x2ffc13f00000-0x2ffc13ffffff] (base 0x2ffc10000000)
 1271.579544 (    0.012019)| [  184.500867] PCI: MMCONFIG for domain 1005 [bus 3f-3f] at [mem 0x2ffc17f00000-0x2ffc17ffffff] (base 0x2ffc14000000)
 1271.591084 (    0.011540)| [  184.512428] PCI: MMCONFIG for domain 1006 [bus 3f-3f] at [mem 0x2ffc1bf00000-0x2ffc1bffffff] (base 0x2ffc18000000)
 1271.603260 (    0.012176)| [  184.523986] PCI: MMCONFIG for domain 1007 [bus 3f-3f] at [mem 0x2ffc1ff00000-0x2ffc1fffffff] (base 0x2ffc1c000000)
 1271.617100 (    0.013840)| [  184.535543] PCI: MMCONFIG for domain 1008 [bus 3f-3f] at [mem 0x2ffc23f00000-0x2ffc23ffffff] (base 0x2ffc20000000)
 1271.623170 (    0.006070)| [  184.547101] PCI: MMCONFIG for domain 1009 [bus 3f-3f] at [mem 0x2ffc27f00000-0x2ffc27ffffff] (base 0x2ffc24000000)
 1271.634818 (    0.011648)| [  184.558656] PCI: MMCONFIG for domain 100a [bus 3f-3f] at [mem 0x2ffc2bf00000-0x2ffc2bffffff] (base 0x2ffc28000000)
 1271.646947 (    0.012129)| [  184.570215] PCI: MMCONFIG for domain 100b [bus 3f-3f] at [mem 0x2ffc2ff00000-0x2ffc2fffffff] (base 0x2ffc2c000000)
 1271.660478 (    0.013531)| [  184.581773] PCI: MMCONFIG for domain 100c [bus 3f-3f] at [mem 0x2ffc33f00000-0x2ffc33ffffff] (base 0x2ffc30000000)
 1271.670847 (    0.010369)| [  184.593331] PCI: MMCONFIG for domain 100d [bus 3f-3f] at [mem 0x2ffc37f00000-0x2ffc37ffffff] (base 0x2ffc34000000)
 1271.683548 (    0.012701)| [  184.604888] PCI: MMCONFIG for domain 100e [bus 3f-3f] at [mem 0x2ffc3bf00000-0x2ffc3bffffff] (base 0x2ffc38000000)
 1271.696770 (    0.013222)| [  184.616444] PCI: MMCONFIG for domain 100f [bus 3f-3f] at [mem 0x2ffc3ff00000-0x2ffc3fffffff] (base 0x2ffc3c000000)
 1271.706665 (    0.009895)| [  184.627993] PCI: MMCONFIG for domain 1010 [bus 3f-3f] at [mem 0x2ffc43f00000-0x2ffc43ffffff] (base 0x2ffc40000000)
 1271.719101 (    0.012436)| [  184.639553] PCI: MMCONFIG for domain 1011 [bus 3f-3f] at [mem 0x2ffc47f00000-0x2ffc47ffffff] (base 0x2ffc44000000)
 1271.727287 (    0.008186)| [  184.651109] PCI: MMCONFIG for domain 1012 [bus 3f-3f] at [mem 0x2ffc4bf00000-0x2ffc4bffffff] (base 0x2ffc48000000)
 1271.739324 (    0.012037)| [  184.662668] PCI: MMCONFIG for domain 1013 [bus 3f-3f] at [mem 0x2ffc4ff00000-0x2ffc4fffffff] (base 0x2ffc4c000000)
 1271.750753 (    0.011429)| [  184.674225] PCI: MMCONFIG for domain 1014 [bus 3f-3f] at [mem 0x2ffc53f00000-0x2ffc53ffffff] (base 0x2ffc50000000)
 1271.762798 (    0.012045)| [  184.685784] PCI: MMCONFIG for domain 1015 [bus 3f-3f] at [mem 0x2ffc57f00000-0x2ffc57ffffff] (base 0x2ffc54000000)
 1271.774816 (    0.012018)| [  184.697342] PCI: MMCONFIG for domain 1016 [bus 3f-3f] at [mem 0x2ffc5bf00000-0x2ffc5bffffff] (base 0x2ffc58000000)
 1271.787213 (    0.012397)| [  184.708900] PCI: MMCONFIG for domain 1017 [bus 3f-3f] at [mem 0x2ffc5ff00000-0x2ffc5fffffff] (base 0x2ffc5c000000)
 1271.798971 (    0.011758)| [  184.720458] PCI: MMCONFIG for domain 1018 [bus 3f-3f] at [mem 0x2ffc63f00000-0x2ffc63ffffff] (base 0x2ffc60000000)
 1271.812102 (    0.013131)| [  184.732018] PCI: MMCONFIG for domain 1019 [bus 3f-3f] at [mem 0x2ffc67f00000-0x2ffc67ffffff] (base 0x2ffc64000000)
 1271.822952 (    0.010850)| [  184.743574] PCI: MMCONFIG for domain 101a [bus 3f-3f] at [mem 0x2ffc6bf00000-0x2ffc6bffffff] (base 0x2ffc68000000)
 1271.831780 (    0.008828)| [  184.755133] PCI: MMCONFIG for domain 101b [bus 3f-3f] at [mem 0x2ffc6ff00000-0x2ffc6fffffff] (base 0x2ffc6c000000)
 1271.843246 (    0.011466)| [  184.766692] PCI: MMCONFIG for domain 101c [bus 3f-3f] at [mem 0x2ffc73f00000-0x2ffc73ffffff] (base 0x2ffc70000000)
 1271.855337 (    0.012091)| [  184.778249] PCI: MMCONFIG for domain 101d [bus 3f-3f] at [mem 0x2ffc77f00000-0x2ffc77ffffff] (base 0x2ffc74000000)
 1271.867129 (    0.011792)| [  184.789806] PCI: MMCONFIG for domain 101e [bus 3f-3f] at [mem 0x2ffc7bf00000-0x2ffc7bffffff] (base 0x2ffc78000000)
 1271.879153 (    0.012024)| [  184.801365] PCI: MMCONFIG for domain 101f [bus 3f-3f] at [mem 0x2ffc7ff00000-0x2ffc7fffffff] (base 0x2ffc7c000000)
 1271.891178 (    0.012025)| [  184.812925] PCI: MMCONFIG at [mem 0x80000000-0x80bfffff] reserved in E820
 1271.900139 (    0.008961)| [  184.820504] PCI: MMCONFIG at [mem 0x2ffc04000000-0x2ffc044fffff] reserved in E820
 1271.907934 (    0.007795)| [  184.828859] PCI: MMCONFIG at [mem 0x2ffc08000000-0x2ffc084fffff] reserved in E820
 1271.917366 (    0.009432)| [  184.837213] PCI: MMCONFIG at [mem 0x2ffc10000000-0x2ffc103fffff] reserved in E820
 1271.922785 (    0.005419)| [  184.845568] PCI: MMCONFIG at [mem 0x2ffc14000000-0x2ffc144fffff] reserved in E820
 1271.930770 (    0.007985)| [  184.853923] PCI: MMCONFIG at [mem 0x2ffc18000000-0x2ffc184fffff] reserved in E820
 1271.938749 (    0.007979)| [  184.862276] PCI: MMCONFIG at [mem 0x2ffc20000000-0x2ffc203fffff] reserved in E820
 1271.947505 (    0.008756)| [  184.870632] PCI: MMCONFIG at [mem 0x2ffc24000000-0x2ffc244fffff] reserved in E820
 1271.954710 (    0.007205)| [  184.878988] PCI: MMCONFIG at [mem 0x2ffc28000000-0x2ffc284fffff] reserved in E820
 1271.962675 (    0.007965)| [  184.887334] PCI: MMCONFIG at [mem 0x2ffc30000000-0x2ffc303fffff] reserved in E820
 1271.974777 (    0.012102)| [  184.895688] PCI: MMCONFIG at [mem 0x2ffc34000000-0x2ffc344fffff] reserved in E820
 1271.982686 (    0.007909)| [  184.904044] PCI: MMCONFIG at [mem 0x2ffc38000000-0x2ffc384fffff] reserved in E820
 1271.991150 (    0.008464)| [  184.912389] PCI: MMCONFIG at [mem 0x2ffc03f00000-0x2ffc03ffffff] reserved in E820
 1271.998734 (    0.007584)| [  184.920746] PCI: MMCONFIG at [mem 0x2ffc07f00000-0x2ffc07ffffff] reserved in E820
 1272.006534 (    0.007800)| [  184.929101] PCI: MMCONFIG at [mem 0x2ffc0bf00000-0x2ffc0bffffff] reserved in E820
 1272.015570 (    0.009036)| [  184.937455] PCI: MMCONFIG at [mem 0x2ffc0ff00000-0x2ffc0fffffff] reserved in E820
 1272.022965 (    0.007395)| [  184.945810] PCI: MMCONFIG at [mem 0x2ffc13f00000-0x2ffc13ffffff] reserved in E820
 1272.030732 (    0.007767)| [  184.954165] PCI: MMCONFIG at [mem 0x2ffc17f00000-0x2ffc17ffffff] reserved in E820
 1272.038613 (    0.007881)| [  184.962519] PCI: MMCONFIG at [mem 0x2ffc1bf00000-0x2ffc1bffffff] reserved in E820
 1272.046673 (    0.008060)| [  184.970873] PCI: MMCONFIG at [mem 0x2ffc1ff00000-0x2ffc1fffffff] reserved in E820
 1272.055958 (    0.009285)| [  184.979228] PCI: MMCONFIG at [mem 0x2ffc23f00000-0x2ffc23ffffff] reserved in E820
 1272.068177 (    0.012219)| [  184.987581] PCI: MMCONFIG at [mem 0x2ffc27f00000-0x2ffc27ffffff] reserved in E820
 1272.074767 (    0.006590)| [  184.995934] PCI: MMCONFIG at [mem 0x2ffc2bf00000-0x2ffc2bffffff] reserved in E820
 1272.082778 (    0.008011)| [  185.004289] PCI: MMCONFIG at [mem 0x2ffc2ff00000-0x2ffc2fffffff] reserved in E820
 1272.090723 (    0.007945)| [  185.012644] PCI: MMCONFIG at [mem 0x2ffc33f00000-0x2ffc33ffffff] reserved in E820
 1272.099186 (    0.008463)| [  185.020998] PCI: MMCONFIG at [mem 0x2ffc37f00000-0x2ffc37ffffff] reserved in E820
 1272.106533 (    0.007347)| [  185.029355] PCI: MMCONFIG at [mem 0x2ffc3bf00000-0x2ffc3bffffff] reserved in E820
 1272.114564 (    0.008031)| [  185.037709] PCI: MMCONFIG at [mem 0x2ffc3ff00000-0x2ffc3fffffff] reserved in E820
 1272.122698 (    0.008134)| [  185.046063] PCI: MMCONFIG at [mem 0x2ffc43f00000-0x2ffc43ffffff] reserved in E820
 1272.130630 (    0.007932)| [  185.054417] PCI: MMCONFIG at [mem 0x2ffc47f00000-0x2ffc47ffffff] reserved in E820
 1272.139324 (    0.008694)| [  185.062770] PCI: MMCONFIG at [mem 0x2ffc4bf00000-0x2ffc4bffffff] reserved in E820
 1272.147482 (    0.008158)| [  185.071125] PCI: MMCONFIG at [mem 0x2ffc4ff00000-0x2ffc4fffffff] reserved in E820
 1272.158877 (    0.011395)| [  185.079481] PCI: MMCONFIG at [mem 0x2ffc53f00000-0x2ffc53ffffff] reserved in E820
 1272.172341 (    0.013464)| [  185.087827] PCI: MMCONFIG at [mem 0x2ffc57f00000-0x2ffc57ffffff] reserved in E820
 1272.174413 (    0.002072)| [  185.096180] PCI: MMCONFIG at [mem 0x2ffc5bf00000-0x2ffc5bffffff] reserved in E820
 1272.182776 (    0.008363)| [  185.104535] PCI: MMCONFIG at [mem 0x2ffc5ff00000-0x2ffc5fffffff] reserved in E820
 1272.190553 (    0.007777)| [  185.112881] PCI: MMCONFIG at [mem 0x2ffc63f00000-0x2ffc63ffffff] reserved in E820
 1272.199032 (    0.008479)| [  185.121236] PCI: MMCONFIG at [mem 0x2ffc67f00000-0x2ffc67ffffff] reserved in E820
 1272.207059 (    0.008027)| [  185.129590] PCI: MMCONFIG at [mem 0x2ffc6bf00000-0x2ffc6bffffff] reserved in E820
 1272.214612 (    0.007553)| [  185.137943] PCI: MMCONFIG at [mem 0x2ffc6ff00000-0x2ffc6fffffff] reserved in E820
 1272.222552 (    0.007940)| [  185.146297] PCI: MMCONFIG at [mem 0x2ffc73f00000-0x2ffc73ffffff] reserved in E820
 1272.232001 (    0.009449)| [  185.154651] PCI: MMCONFIG at [mem 0x2ffc77f00000-0x2ffc77ffffff] reserved in E820
 1272.238546 (    0.006545)| [  185.163006] PCI: MMCONFIG at [mem 0x2ffc7bf00000-0x2ffc7bffffff] reserved in E820
 1272.250603 (    0.012057)| [  185.171359] PCI: MMCONFIG at [mem 0x2ffc7ff00000-0x2ffc7fffffff] reserved in E820
 1272.259156 (    0.008553)| [  185.181432] PCI: Using configuration type 1 for base access
 1272.522552 (    0.263396)| [  185.444405] HugeTLB registered 1 GB page size, pre-allocated 0 pages
 1272.530727 (    0.008175)| [  185.451519] HugeTLB registered 2 MB page size, pre-allocated 0 pages
 1272.594319 (    0.063592)| [  185.517821] ACPI: Added _OSI(Module Device)
 1272.598378 (    0.004059)| [  185.522495] ACPI: Added _OSI(Processor Device)
 1272.606470 (    0.008092)| [  185.527455] ACPI: Added _OSI(3.0 _SCP Extensions)
 1272.610346 (    0.003876)| [  185.532706] ACPI: Added _OSI(Processor Aggregator Device)
 1274.414854 (    1.804508)| [  187.337713] ACPI: Interpreter enabled
 1274.418199 (    0.003345)| [  187.341859] ACPI: (supports S0 S5)
 1274.422250 (    0.004051)| [  187.345658] ACPI: Using IOAPIC for interrupt routing
 1274.432014 (    0.009764)| [  187.351279] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
 1274.638201 (    0.206187)| [  187.558583] ACPI: PCI Root Bridge [IIO0] (domain 0000 [bus 00-fe])
 1274.641914 (    0.003713)| [  187.565500] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1274.654077 (    0.012163)| [  187.575639] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1274.662339 (    0.008262)| [  187.584596] acpi PNP0A08:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-0b] only partially covers this bridge
 1274.673626 (    0.011287)| [  187.597453] PCI host bridge to bus 0000:00
 1274.677903 (    0.004277)| [  187.602030] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
 1274.685945 (    0.008042)| [  187.609609] pci_bus 0000:00: root bus resource [io  0x1000-0x33ff window]
 1274.693713 (    0.007768)| [  187.617190] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
 1274.702100 (    0.008387)| [  187.625546] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000cbfff window]
 1274.709715 (    0.007615)| [  187.633904] pci_bus 0000:00: root bus resource [mem 0x2ffe00000000-0x2ffe03ffffff window]
 1274.722157 (    0.012442)| [  187.643036] pci_bus 0000:00: root bus resource [mem 0x90000000-0x91ffffff window]
 1274.730441 (    0.008284)| [  187.651392] pci_bus 0000:00: root bus resource [bus 00-fe]
 1274.734030 (    0.003589)| [  187.658125] pci 0000:00:02.0: System wakeup disabled by ACPI
 1274.741973 (    0.007943)| [  187.664784] pci 0000:00:03.0: System wakeup disabled by ACPI
 1274.754649 (    0.012676)| [  187.675431] pci 0000:00:1c.0: System wakeup disabled by ACPI
 1274.758845 (    0.004196)| [  187.682106] pci 0000:00:1c.7: System wakeup disabled by ACPI
 1274.766000 (    0.007155)| [  187.689074] pci 0000:00:1e.0: System wakeup disabled by ACPI
 1274.773965 (    0.007965)| [  187.696843] acpiphp: Slot [1] registered
 1274.777422 (    0.003457)| [  187.701265] pci 0000:00:02.0: PCI bridge to [bus 01]
 1274.785918 (    0.008496)| [  187.707205] acpiphp: Slot [2] registered
 1274.789651 (    0.003733)| [  187.711790] pci 0000:02:00.0: VF(n) BAR0 space: [mem 0x91b00000-0x91bfffff 64bit] (contains BAR0 for 64 VFs)
 1274.802321 (    0.012670)| [  187.722779] pci 0000:02:00.0: VF(n) BAR3 space: [mem 0x91c00000-0x91cfffff 64bit] (contains BAR3 for 64 VFs)
 1274.810361 (    0.008040)| [  187.734204] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0x91d00000-0x91dfffff 64bit] (contains BAR0 for 64 VFs)
 1274.822086 (    0.011725)| [  187.745190] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0x91e00000-0x91efffff 64bit] (contains BAR3 for 64 VFs)
 1274.846016 (    0.023930)| [  187.768199] pci 0000:00:03.0: PCI bridge to [bus 02-03]
 1274.849985 (    0.003969)| [  187.774164] pci 0000:00:11.0: PCI bridge to [bus 04]
 1274.870566 (    0.020581)| [  187.791750] pci 0000:00:1c.0: PCI bridge to [bus 05]
 1274.890249 (    0.019683)| [  187.813401] pci 0000:00:1c.7: PCI bridge to [bus 06-0a]
 1274.907689 (    0.017440)| [  187.828988] pci 0000:06:00.0: PCI bridge to [bus 07-0a]
 1274.917864 (    0.010175)| [  187.839920] pci 0000:07:00.0: PCI bridge to [bus 08-09]
 1274.929932 (    0.012068)| [  187.852110] pci 0000:08:00.0: PCI bridge to [bus 09]
 1274.942633 (    0.012701)| [  187.863712] pci 0000:07:01.0: PCI bridge to [bus 0a]
 1274.951318 (    0.008685)| [  187.870691] pci 0000:00:1e.0: PCI bridge to [bus 0b] (subtractive decode)
 1274.958222 (    0.006904)| [  187.878799] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1274.966020 (    0.007798)| [  187.888408] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1274.974150 (    0.008130)| [  187.898013] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1274.986490 (    0.012340)| [  187.907607] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1274.993978 (    0.007488)| [  187.917202] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1275.006012 (    0.012034)| [  187.926809] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1275.014126 (    0.008114)| [  187.936410] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1275.022570 (    0.008444)| [  187.946023] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
 1275.082317 (    0.059747)| [  188.005966] ACPI: PCI Root Bridge [I001] (domain 0001 [bus 00-3d])
 1275.090176 (    0.007859)| [  188.012872] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.101831 (    0.011655)| [  188.023008] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.110068 (    0.008237)| [  188.031957] acpi PNP0A08:01: [Firmware Info]: MMCONFIG for domain 0001 [bus 00-04] only partially covers this bridge
 1275.124090 (    0.014022)| [  188.043862] PCI host bridge to bus 0001:00
 1275.126028 (    0.001938)| [  188.048438] pci_bus 0001:00: root bus resource [mem 0x2ffe04000000-0x2ffe07ffffff window]
 1275.137998 (    0.011970)| [  188.057570] pci_bus 0001:00: root bus resource [mem 0x92000000-0x923fffff window]
 1275.142126 (    0.004128)| [  188.065928] pci_bus 0001:00: root bus resource [bus 00-3d]
 1275.154140 (    0.012014)| [  188.075844] pci 0001:01:00.0: VF(n) BAR0 space: [mem 0x2ffe04010000-0x2ffe04017fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.167050 (    0.012910)| [  188.089868] pci 0001:01:00.1: VF(n) BAR0 space: [mem 0x2ffe04000000-0x2ffe04007fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.193923 (    0.026873)| [  188.118054] pci 0001:00:02.0: PCI bridge to [bus 01-02]
 1275.201984 (    0.008061)| [  188.123964] pci 0001:00:03.0: PCI bridge to [bus 03]
 1275.205999 (    0.004015)| [  188.129574] pci 0001:00:03.2: PCI bridge to [bus 04]
 1275.213862 (    0.007863)| [  188.135197] ACPI: PCI Root Bridge [I002] (domain 0002 [bus 00-3d])
 1275.218208 (    0.004346)| [  188.142101] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.230108 (    0.011900)| [  188.152210] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.237853 (    0.007745)| [  188.161160] acpi PNP0A08:02: [Firmware Info]: MMCONFIG for domain 0002 [bus 00-04] only partially covers this bridge
 1275.250284 (    0.012431)| [  188.173077] PCI host bridge to bus 0002:00
 1275.253771 (    0.003487)| [  188.177650] pci_bus 0002:00: root bus resource [mem 0x2ffe08000000-0x2ffe0bffffff window]
 1275.265914 (    0.012143)| [  188.186784] pci_bus 0002:00: root bus resource [mem 0x92400000-0x927fffff window]
 1275.274235 (    0.008321)| [  188.195140] pci_bus 0002:00: root bus resource [bus 00-3d]
 1275.282347 (    0.008112)| [  188.203684] pci 0002:00:02.0: PCI bridge to [bus 01]
 1275.286422 (    0.004075)| [  188.209301] pci 0002:00:02.2: PCI bridge to [bus 02]
 1275.294039 (    0.007617)| [  188.216288] pci 0002:03:00.0: VF(n) BAR0 space: [mem 0x2ffe08010000-0x2ffe08017fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.306322 (    0.012283)| [  188.230298] pci 0002:03:00.1: VF(n) BAR0 space: [mem 0x2ffe08000000-0x2ffe08007fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.337891 (    0.031569)| [  188.258472] pci 0002:00:03.0: PCI bridge to [bus 03-04]
 1275.341847 (    0.003956)| [  188.264387] ACPI: PCI Root Bridge [I003] (domain 0003 [bus 00-3d])
 1275.349881 (    0.008034)| [  188.271292] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.358700 (    0.008819)| [  188.281420] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.366003 (    0.007303)| [  188.290358] acpi PNP0A08:03: [Firmware Info]: MMCONFIG for domain 0003 [bus 00-03] only partially covers this bridge
 1275.378283 (    0.012280)| [  188.302268] PCI host bridge to bus 0003:00
 1275.385454 (    0.007171)| [  188.306842] pci_bus 0003:00: root bus resource [io  0x4000-0x4fff window]
 1275.391238 (    0.005784)| [  188.314422] pci_bus 0003:00: root bus resource [mem 0x2ffe0c000000-0x2ffe0fffffff window]
 1275.401857 (    0.010619)| [  188.323556] pci_bus 0003:00: root bus resource [mem 0x92800000-0x92ffffff window]
 1275.410614 (    0.008757)| [  188.331913] pci_bus 0003:00: root bus resource [bus 00-3d]
 1275.417328 (    0.006714)| [  188.340288] pci 0003:00:02.0: PCI bridge to [bus 01]
 1275.421847 (    0.004519)| [  188.346030] pci 0003:02:00.0: VF(n) BAR0 space: [mem 0x92800000-0x928fffff 64bit] (contains BAR0 for 64 VFs)
 1275.433811 (    0.011964)| [  188.357016] pci 0003:02:00.0: VF(n) BAR3 space: [mem 0x92900000-0x929fffff 64bit] (contains BAR3 for 64 VFs)
 1275.446024 (    0.012213)| [  188.368373] pci 0003:02:00.1: VF(n) BAR0 space: [mem 0x92a00000-0x92afffff 64bit] (contains BAR0 for 64 VFs)
 1275.458023 (    0.011999)| [  188.379362] pci 0003:02:00.1: VF(n) BAR3 space: [mem 0x92b00000-0x92bfffff 64bit] (contains BAR3 for 64 VFs)
 1275.484594 (    0.026571)| [  188.402356] pci 0003:00:03.0: PCI bridge to [bus 02-03]
 1275.485206 (    0.000612)| [  188.408274] ACPI: PCI Root Bridge [I004] (domain 0004 [bus 00-3d])
 1275.493587 (    0.008381)| [  188.415179] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.503231 (    0.009644)| [  188.425318] acpi PNP0A08:04: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.509821 (    0.006590)| [  188.434266] acpi PNP0A08:04: [Firmware Info]: MMCONFIG for domain 0004 [bus 00-04] only partially covers this bridge
 1275.522028 (    0.012207)| [  188.446185] PCI host bridge to bus 0004:00
 1275.530432 (    0.008404)| [  188.450758] pci_bus 0004:00: root bus resource [mem 0x2ffe10000000-0x2ffe13ffffff window]
 1275.537772 (    0.007340)| [  188.459882] pci_bus 0004:00: root bus resource [mem 0x93000000-0x933fffff window]
 1275.547021 (    0.009249)| [  188.468241] pci_bus 0004:00: root bus resource [bus 00-3d]
 1275.558111 (    0.011090)| [  188.478270] pci 0004:01:00.0: VF(n) BAR0 space: [mem 0x2ffe10010000-0x2ffe10017fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.569837 (    0.011726)| [  188.492286] pci 0004:01:00.1: VF(n) BAR0 space: [mem 0x2ffe10000000-0x2ffe10007fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.593820 (    0.023983)| [  188.516469] pci 0004:00:02.0: PCI bridge to [bus 01-02]
 1275.602131 (    0.008311)| [  188.522381] pci 0004:00:03.0: PCI bridge to [bus 03]
 1275.606036 (    0.003905)| [  188.527995] pci 0004:00:03.2: PCI bridge to [bus 04]
 1275.610232 (    0.004196)| [  188.533621] ACPI: PCI Root Bridge [I005] (domain 0005 [bus 00-3d])
 1275.617851 (    0.007619)| [  188.540524] acpi PNP0A08:05: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.629669 (    0.011818)| [  188.550636] acpi PNP0A08:05: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.638917 (    0.009248)| [  188.559584] acpi PNP0A08:05: [Firmware Info]: MMCONFIG for domain 0005 [bus 00-04] only partially covers this bridge
 1275.650009 (    0.011092)| [  188.571497] PCI host bridge to bus 0005:00
 1275.653869 (    0.003860)| [  188.576074] pci_bus 0005:00: root bus resource [mem 0x2ffe14000000-0x2ffe17ffffff window]
 1275.661647 (    0.007778)| [  188.585204] pci_bus 0005:00: root bus resource [mem 0x93400000-0x937fffff window]
 1275.670064 (    0.008417)| [  188.593560] pci_bus 0005:00: root bus resource [bus 00-3d]
 1275.678056 (    0.007992)| [  188.602171] pci 0005:00:02.0: PCI bridge to [bus 01]
 1275.685692 (    0.007636)| [  188.607791] pci 0005:00:02.2: PCI bridge to [bus 02]
 1275.728073 (    0.042381)| [  188.614790] pci 0005:03:00.0: VF(n) BAR0 space: [mem 0x2ffe14010000-0x2ffe14017fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.728150 (    0.000077)| [  188.628806] pci 0005:03:00.1: VF(n) BAR0 space: [mem 0x2ffe14000000-0x2ffe14007fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.729974 (    0.001824)| [  188.652989] pci 0005:00:03.0: PCI bridge to [bus 03-04]
 1275.738296 (    0.008322)| [  188.658909] ACPI: PCI Root Bridge [I006] (domain 0006 [bus 00-3d])
 1275.742244 (    0.003948)| [  188.665815] acpi PNP0A08:06: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.754399 (    0.012155)| [  188.675909] acpi PNP0A08:06: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.762126 (    0.007727)| [  188.684856] acpi PNP0A08:06: [Firmware Info]: MMCONFIG for domain 0006 [bus 00-03] only partially covers this bridge
 1275.773927 (    0.011801)| [  188.696773] PCI host bridge to bus 0006:00
 1275.778538 (    0.004611)| [  188.701349] pci_bus 0006:00: root bus resource [io  0x5000-0x5fff window]
 1275.786292 (    0.007754)| [  188.708929] pci_bus 0006:00: root bus resource [mem 0x2ffe18000000-0x2ffe1bffffff window]
 1275.793402 (    0.007110)| [  188.718060] pci_bus 0006:00: root bus resource [mem 0x93800000-0x93ffffff window]
 1275.806441 (    0.013039)| [  188.726415] pci_bus 0006:00: root bus resource [bus 00-3d]
 1275.814543 (    0.008102)| [  188.734946] pci 0006:00:02.0: PCI bridge to [bus 01]
 1275.818126 (    0.003583)| [  188.740707] pci 0006:02:00.0: VF(n) BAR0 space: [mem 0x93800000-0x938fffff 64bit] (contains BAR0 for 64 VFs)
 1275.829890 (    0.011764)| [  188.751697] pci 0006:02:00.0: VF(n) BAR3 space: [mem 0x93900000-0x939fffff 64bit] (contains BAR3 for 64 VFs)
 1275.841632 (    0.011742)| [  188.763079] pci 0006:02:00.1: VF(n) BAR0 space: [mem 0x93a00000-0x93afffff 64bit] (contains BAR0 for 64 VFs)
 1275.850057 (    0.008425)| [  188.774067] pci 0006:02:00.1: VF(n) BAR3 space: [mem 0x93b00000-0x93bfffff 64bit] (contains BAR3 for 64 VFs)
 1275.874285 (    0.024228)| [  188.797075] pci 0006:00:03.0: PCI bridge to [bus 02-03]
 1275.882632 (    0.008347)| [  188.802991] ACPI: PCI Root Bridge [I007] (domain 0007 [bus 00-3d])
 1275.885748 (    0.003116)| [  188.809897] acpi PNP0A08:07: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1275.897966 (    0.012218)| [  188.820002] acpi PNP0A08:07: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1275.906304 (    0.008338)| [  188.828947] acpi PNP0A08:07: [Firmware Info]: MMCONFIG for domain 0007 [bus 00-04] only partially covers this bridge
 1275.918427 (    0.012123)| [  188.840862] PCI host bridge to bus 0007:00
 1275.921735 (    0.003308)| [  188.845438] pci_bus 0007:00: root bus resource [mem 0x2ffe1c000000-0x2ffe1fffffff window]
 1275.933921 (    0.012186)| [  188.854571] pci_bus 0007:00: root bus resource [mem 0x94000000-0x943fffff window]
 1275.942952 (    0.009031)| [  188.862930] pci_bus 0007:00: root bus resource [bus 00-3d]
 1275.949705 (    0.006753)| [  188.873113] pci 0007:01:00.0: VF(n) BAR0 space: [mem 0x2ffe1c010000-0x2ffe1c017fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.966033 (    0.016328)| [  188.887141] pci 0007:01:00.1: VF(n) BAR0 space: [mem 0x2ffe1c000000-0x2ffe1c007fff 64bit pref] (contains BAR0 for 1 VFs)
 1275.990091 (    0.024058)| [  188.911316] pci 0007:00:02.0: PCI bridge to [bus 01-02]
 1275.993687 (    0.003596)| [  188.917232] pci 0007:00:03.0: PCI bridge to [bus 03]
 1276.001519 (    0.007832)| [  188.922848] pci 0007:00:03.2: PCI bridge to [bus 04]
 1276.005761 (    0.004242)| [  188.928472] ACPI: PCI Root Bridge [I008] (domain 0008 [bus 00-3d])
 1276.013426 (    0.007665)| [  188.935378] acpi PNP0A08:08: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.021679 (    0.008253)| [  188.945492] acpi PNP0A08:08: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1276.033526 (    0.011847)| [  188.954441] acpi PNP0A08:08: [Firmware Info]: MMCONFIG for domain 0008 [bus 00-04] only partially covers this bridge
 1276.045502 (    0.011976)| [  188.966354] PCI host bridge to bus 0008:00
 1276.049445 (    0.003943)| [  188.970928] pci_bus 0008:00: root bus resource [mem 0x2ffe20000000-0x2ffe23ffffff window]
 1276.057513 (    0.008068)| [  188.980060] pci_bus 0008:00: root bus resource [mem 0x94400000-0x947fffff window]
 1276.065559 (    0.008046)| [  188.988417] pci_bus 0008:00: root bus resource [bus 00-3d]
 1276.073562 (    0.008003)| [  188.997200] pci 0008:00:02.0: PCI bridge to [bus 01]
 1276.081440 (    0.007878)| [  189.002815] pci 0008:00:02.2: PCI bridge to [bus 02]
 1276.085494 (    0.004054)| [  189.009827] pci 0008:03:00.0: VF(n) BAR0 space: [mem 0x2ffe20010000-0x2ffe20017fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.101442 (    0.015948)| [  189.023865] pci 0008:03:00.1: VF(n) BAR0 space: [mem 0x2ffe20000000-0x2ffe20007fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.129567 (    0.028125)| [  189.052051] pci 0008:00:03.0: PCI bridge to [bus 03-04]
 1276.134131 (    0.004564)| [  189.057973] ACPI: PCI Root Bridge [I009] (domain 0009 [bus 00-3d])
 1276.141724 (    0.007593)| [  189.064867] acpi PNP0A08:09: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.153457 (    0.011733)| [  189.074998] acpi PNP0A08:09: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1276.161353 (    0.007896)| [  189.083945] acpi PNP0A08:09: [Firmware Info]: MMCONFIG for domain 0009 [bus 00-03] only partially covers this bridge
 1276.174429 (    0.013076)| [  189.095859] PCI host bridge to bus 0009:00
 1276.178353 (    0.003924)| [  189.100435] pci_bus 0009:00: root bus resource [io  0x6000-0x6fff window]
 1276.185444 (    0.007091)| [  189.108016] pci_bus 0009:00: root bus resource [mem 0x2ffe24000000-0x2ffe27ffffff window]
 1276.193423 (    0.007979)| [  189.117148] pci_bus 0009:00: root bus resource [mem 0x94800000-0x94ffffff window]
 1276.201478 (    0.008055)| [  189.125496] pci_bus 0009:00: root bus resource [bus 00-3d]
 1276.209522 (    0.008044)| [  189.134087] pci 0009:00:02.0: PCI bridge to [bus 01]
 1276.217610 (    0.008088)| [  189.139859] pci 0009:02:00.0: VF(n) BAR0 space: [mem 0x94800000-0x948fffff 64bit] (contains BAR0 for 64 VFs)
 1276.230356 (    0.012746)| [  189.150849] pci 0009:02:00.0: VF(n) BAR3 space: [mem 0x94900000-0x949fffff 64bit] (contains BAR3 for 64 VFs)
 1276.241648 (    0.011292)| [  189.162263] pci 0009:02:00.1: VF(n) BAR0 space: [mem 0x94a00000-0x94afffff 64bit] (contains BAR0 for 64 VFs)
 1276.249590 (    0.007942)| [  189.173251] pci 0009:02:00.1: VF(n) BAR3 space: [mem 0x94b00000-0x94bfffff 64bit] (contains BAR3 for 64 VFs)
 1276.273595 (    0.024005)| [  189.196248] pci 0009:00:03.0: PCI bridge to [bus 02-03]
 1276.281468 (    0.007873)| [  189.202154] ACPI: PCI Root Bridge [I00A] (domain 000a [bus 00-3d])
 1276.285440 (    0.003972)| [  189.209058] acpi PNP0A08:0a: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.298298 (    0.012858)| [  189.219204] acpi PNP0A08:0a: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1276.305524 (    0.007226)| [  189.228153] acpi PNP0A08:0a: [Firmware Info]: MMCONFIG for domain 000a [bus 00-04] only partially covers this bridge
 1276.317492 (    0.011968)| [  189.240065] PCI host bridge to bus 000a:00
 1276.321494 (    0.004002)| [  189.244641] pci_bus 000a:00: root bus resource [mem 0x2ffe28000000-0x2ffe2bffffff window]
 1276.329495 (    0.008001)| [  189.253772] pci_bus 000a:00: root bus resource [mem 0x95000000-0x953fffff window]
 1276.341447 (    0.011952)| [  189.262127] pci_bus 000a:00: root bus resource [bus 00-3d]
 1276.350473 (    0.009026)| [  189.272381] pci 000a:01:00.0: VF(n) BAR0 space: [mem 0x2ffe28010000-0x2ffe28017fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.365502 (    0.015029)| [  189.286418] pci 000a:01:00.1: VF(n) BAR0 space: [mem 0x2ffe28000000-0x2ffe28007fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.392155 (    0.026653)| [  189.310595] pci 000a:00:02.0: PCI bridge to [bus 01-02]
 1276.394447 (    0.002292)| [  189.316507] pci 000a:00:03.0: PCI bridge to [bus 03]
 1276.401927 (    0.007480)| [  189.322130] pci 000a:00:03.2: PCI bridge to [bus 04]
 1276.405637 (    0.003710)| [  189.327758] ACPI: PCI Root Bridge [I00B] (domain 000b [bus 00-3d])
 1276.415260 (    0.009623)| [  189.334663] acpi PNP0A08:0b: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.435570 (    0.020310)| [  189.344784] acpi PNP0A08:0b: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
 1276.435627 (    0.000057)| [  189.353734] acpi PNP0A08:0b: [Firmware Info]: MMCONFIG for domain 000b [bus 00-04] only partially covers this bridge
 1276.442504 (    0.006877)| [  189.365642] PCI host bridge to bus 000b:00
 1276.450243 (    0.007739)| [  189.370218] pci_bus 000b:00: root bus resource [mem 0x2ffe2c000000-0x2ffe2fffffff window]
 1276.458078 (    0.007835)| [  189.379349] pci_bus 000b:00: root bus resource [mem 0x95400000-0x957fffff window]
 1276.467921 (    0.009843)| [  189.387706] pci_bus 000b:00: root bus resource [bus 00-3d]
 1276.473594 (    0.005673)| [  189.396572] pci 000b:00:02.0: PCI bridge to [bus 01]
 1276.481673 (    0.008079)| [  189.402196] pci 000b:00:02.2: PCI bridge to [bus 02]
 1276.487509 (    0.005836)| [  189.409222] pci 000b:03:00.0: VF(n) BAR0 space: [mem 0x2ffe2c010000-0x2ffe2c017fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.501833 (    0.014324)| [  189.423263] pci 000b:03:00.1: VF(n) BAR0 space: [mem 0x2ffe2c000000-0x2ffe2c007fff 64bit pref] (contains BAR0 for 1 VFs)
 1276.525350 (    0.023517)| [  189.447460] pci 000b:00:03.0: PCI bridge to [bus 03-04]
 1276.529843 (    0.004493)| [  189.453360] ACPI: PCI Root Bridge [S000] (domain 1000 [bus 3f])
 1276.537222 (    0.007379)| [  189.459974] acpi PNP0A08:0c: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.545696 (    0.008474)| [  189.469324] acpi PNP0A08:0c: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.554617 (    0.008921)| [  189.477984] acpi PNP0A08:0c: _OSC: OS now controls [PCIeCapability]
 1276.561434 (    0.006817)| [  189.485205] PCI host bridge to bus 1000:3f
 1276.565695 (    0.004261)| [  189.489779] pci_bus 1000:3f: root bus resource [bus 3f]
 1276.589715 (    0.024020)| [  189.512856] ACPI: PCI Root Bridge [S001] (domain 1001 [bus 3f])
 1276.597690 (    0.007975)| [  189.519469] acpi PNP0A08:0d: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.607593 (    0.009903)| [  189.528819] acpi PNP0A08:0d: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.615240 (    0.007647)| [  189.537459] acpi PNP0A08:0d: _OSC: OS now controls [PCIeCapability]
 1276.621278 (    0.006038)| [  189.544673] PCI host bridge to bus 1001:3f
 1276.625720 (    0.004442)| [  189.549247] pci_bus 1001:3f: root bus resource [bus 3f]
 1276.645662 (    0.019942)| [  189.568217] ACPI: PCI Root Bridge [S002] (domain 1002 [bus 3f])
 1276.653810 (    0.008148)| [  189.574833] acpi PNP0A08:0e: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.662043 (    0.008233)| [  189.584180] acpi PNP0A08:0e: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.671293 (    0.009250)| [  189.592824] acpi PNP0A08:0e: _OSC: OS now controls [PCIeCapability]
 1276.677390 (    0.006097)| [  189.600042] PCI host bridge to bus 1002:3f
 1276.681923 (    0.004533)| [  189.604620] pci_bus 1002:3f: root bus resource [bus 3f]
 1276.702670 (    0.020747)| [  189.623725] ACPI: PCI Root Bridge [S003] (domain 1003 [bus 3f])
 1276.713242 (    0.010572)| [  189.630341] acpi PNP0A08:0f: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.718562 (    0.005320)| [  189.639690] acpi PNP0A08:0f: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.726558 (    0.007996)| [  189.648333] acpi PNP0A08:0f: _OSC: OS now controls [PCIeCapability]
 1276.739573 (    0.013015)| [  189.655551] PCI host bridge to bus 1003:3f
 1276.739605 (    0.000032)| [  189.660126] pci_bus 1003:3f: root bus resource [bus 3f]
 1276.758060 (    0.018455)| [  189.679309] ACPI: PCI Root Bridge [S004] (domain 1004 [bus 3f])
 1276.761405 (    0.003345)| [  189.685923] acpi PNP0A08:10: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.773362 (    0.011957)| [  189.695272] acpi PNP0A08:10: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.783437 (    0.010075)| [  189.703921] acpi PNP0A08:10: _OSC: OS now controls [PCIeCapability]
 1276.790503 (    0.007066)| [  189.711142] PCI host bridge to bus 1004:3f
 1276.793289 (    0.002786)| [  189.715716] pci_bus 1004:3f: root bus resource [bus 3f]
 1276.814138 (    0.020849)| [  189.735008] ACPI: PCI Root Bridge [S005] (domain 1005 [bus 3f])
 1276.817381 (    0.003243)| [  189.741621] acpi PNP0A08:11: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.829215 (    0.011834)| [  189.750967] acpi PNP0A08:11: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.836778 (    0.007563)| [  189.759611] acpi PNP0A08:11: _OSC: OS now controls [PCIeCapability]
 1276.845357 (    0.008579)| [  189.766833] PCI host bridge to bus 1005:3f
 1276.849448 (    0.004091)| [  189.771407] pci_bus 1005:3f: root bus resource [bus 3f]
 1276.869395 (    0.019947)| [  189.790808] ACPI: PCI Root Bridge [S006] (domain 1006 [bus 3f])
 1276.873398 (    0.004003)| [  189.797422] acpi PNP0A08:12: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.885959 (    0.012561)| [  189.806771] acpi PNP0A08:12: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.894886 (    0.008927)| [  189.815415] acpi PNP0A08:12: _OSC: OS now controls [PCIeCapability]
 1276.902463 (    0.007577)| [  189.822632] PCI host bridge to bus 1006:3f
 1276.905301 (    0.002838)| [  189.827206] pci_bus 1006:3f: root bus resource [bus 3f]
 1276.925652 (    0.020351)| [  189.846679] ACPI: PCI Root Bridge [S007] (domain 1007 [bus 3f])
 1276.929295 (    0.003643)| [  189.853292] acpi PNP0A08:13: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.941491 (    0.012196)| [  189.862637] acpi PNP0A08:13: _OSC: platform does not support [PCIeHotplug PME AER]
 1276.949223 (    0.007732)| [  189.871284] acpi PNP0A08:13: _OSC: OS now controls [PCIeCapability]
 1276.957510 (    0.008287)| [  189.878507] PCI host bridge to bus 1007:3f
 1276.962554 (    0.005044)| [  189.883085] pci_bus 1007:3f: root bus resource [bus 3f]
 1276.981602 (    0.019048)| [  189.902764] ACPI: PCI Root Bridge [S008] (domain 1008 [bus 3f])
 1276.985427 (    0.003825)| [  189.909378] acpi PNP0A08:14: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1276.999454 (    0.014027)| [  189.918721] acpi PNP0A08:14: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.005862 (    0.006408)| [  189.927368] acpi PNP0A08:14: _OSC: OS now controls [PCIeCapability]
 1277.013764 (    0.007902)| [  189.934590] PCI host bridge to bus 1008:3f
 1277.017934 (    0.004170)| [  189.939169] pci_bus 1008:3f: root bus resource [bus 3f]
 1277.222281 (    0.204347)| [  189.958936] ACPI: PCI Root Bridge [S009] (domain 1009 [bus 3f])
 1277.222342 (    0.000061)| [  189.965552] acpi PNP0A08:15: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.222399 (    0.000057)| [  189.974896] acpi PNP0A08:15: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.222452 (    0.000053)| [  189.983546] acpi PNP0A08:15: _OSC: OS now controls [PCIeCapability]
 1277.222496 (    0.000044)| [  189.990770] PCI host bridge to bus 1009:3f
 1277.222526 (    0.000030)| [  189.995347] pci_bus 1009:3f: root bus resource [bus 3f]
 1277.222562 (    0.000036)| [  190.015111] ACPI: PCI Root Bridge [S00A] (domain 100a [bus 3f])
 1277.222603 (    0.000041)| [  190.021722] acpi PNP0A08:16: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.222659 (    0.000056)| [  190.031065] acpi PNP0A08:16: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.222710 (    0.000051)| [  190.039721] acpi PNP0A08:16: _OSC: OS now controls [PCIeCapability]
 1277.222754 (    0.000044)| [  190.046941] PCI host bridge to bus 100a:3f
 1277.222783 (    0.000029)| [  190.051521] pci_bus 100a:3f: root bus resource [bus 3f]
 1277.222820 (    0.000037)| [  190.071424] ACPI: PCI Root Bridge [S00B] (domain 100b [bus 3f])
 1277.222862 (    0.000042)| [  190.078038] acpi PNP0A08:17: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.222919 (    0.000057)| [  190.087383] acpi PNP0A08:17: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.222973 (    0.000054)| [  190.096027] acpi PNP0A08:17: _OSC: OS now controls [PCIeCapability]
 1277.223016 (    0.000043)| [  190.103252] PCI host bridge to bus 100b:3f
 1277.223045 (    0.000029)| [  190.107827] pci_bus 100b:3f: root bus resource [bus 3f]
 1277.223081 (    0.000036)| [  190.127926] ACPI: PCI Root Bridge [S00C] (domain 100c [bus 3f])
 1277.223122 (    0.000041)| [  190.134539] acpi PNP0A08:18: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.237500 (    0.014378)| [  190.143886] acpi PNP0A08:18: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.237554 (    0.000054)| [  190.152529] acpi PNP0A08:18: _OSC: OS now controls [PCIeCapability]
 1277.238469 (    0.000915)| [  190.159753] PCI host bridge to bus 100c:3f
 1277.241488 (    0.003019)| [  190.164329] pci_bus 100c:3f: root bus resource [bus 3f]
 1277.261024 (    0.019536)| [  190.184549] ACPI: PCI Root Bridge [S00D] (domain 100d [bus 3f])
 1277.269199 (    0.008175)| [  190.191163] acpi PNP0A08:19: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.276983 (    0.007784)| [  190.200508] acpi PNP0A08:19: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.285535 (    0.008552)| [  190.209149] acpi PNP0A08:19: _OSC: OS now controls [PCIeCapability]
 1277.293489 (    0.007954)| [  190.216365] PCI host bridge to bus 100d:3f
 1277.297359 (    0.003870)| [  190.220942] pci_bus 100d:3f: root bus resource [bus 3f]
 1277.317378 (    0.020019)| [  190.241190] ACPI: PCI Root Bridge [S00E] (domain 100e [bus 3f])
 1277.325630 (    0.008252)| [  190.247803] acpi PNP0A08:1a: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.333134 (    0.007504)| [  190.257150] acpi PNP0A08:1a: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.345146 (    0.012012)| [  190.265794] acpi PNP0A08:1a: _OSC: OS now controls [PCIeCapability]
 1277.350093 (    0.004947)| [  190.273017] PCI host bridge to bus 100e:3f
 1277.353108 (    0.003015)| [  190.277591] pci_bus 100e:3f: root bus resource [bus 3f]
 1277.377742 (    0.024634)| [  190.297893] ACPI: PCI Root Bridge [S00F] (domain 100f [bus 3f])
 1277.380649 (    0.002907)| [  190.304508] acpi PNP0A08:1b: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.390755 (    0.010106)| [  190.313852] acpi PNP0A08:1b: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.401211 (    0.010456)| [  190.322496] acpi PNP0A08:1b: _OSC: OS now controls [PCIeCapability]
 1277.405206 (    0.003995)| [  190.329723] PCI host bridge to bus 100f:3f
 1277.413216 (    0.008010)| [  190.334297] pci_bus 100f:3f: root bus resource [bus 3f]
 1277.433489 (    0.020273)| [  190.354973] ACPI: PCI Root Bridge [S010] (domain 1010 [bus 3f])
 1277.437507 (    0.004018)| [  190.361584] acpi PNP0A08:1c: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.448999 (    0.011492)| [  190.370934] acpi PNP0A08:1c: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.457319 (    0.008320)| [  190.379578] acpi PNP0A08:1c: _OSC: OS now controls [PCIeCapability]
 1277.465465 (    0.008146)| [  190.386794] PCI host bridge to bus 1010:3f
 1277.470546 (    0.005081)| [  190.391370] pci_bus 1010:3f: root bus resource [bus 3f]
 1277.489665 (    0.019119)| [  190.412131] ACPI: PCI Root Bridge [S011] (domain 1011 [bus 3f])
 1277.497274 (    0.007609)| [  190.418744] acpi PNP0A08:1d: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.506872 (    0.009598)| [  190.428092] acpi PNP0A08:1d: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.514764 (    0.007892)| [  190.436739] acpi PNP0A08:1d: _OSC: OS now controls [PCIeCapability]
 1277.521847 (    0.007083)| [  190.443960] PCI host bridge to bus 1011:3f
 1277.525374 (    0.003527)| [  190.448534] pci_bus 1011:3f: root bus resource [bus 3f]
 1277.546312 (    0.020938)| [  190.469354] ACPI: PCI Root Bridge [S012] (domain 1012 [bus 3f])
 1277.553730 (    0.007418)| [  190.475970] acpi PNP0A08:1e: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.561331 (    0.007601)| [  190.485318] acpi PNP0A08:1e: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.571138 (    0.009807)| [  190.493962] acpi PNP0A08:1e: _OSC: OS now controls [PCIeCapability]
 1277.577919 (    0.006781)| [  190.501179] PCI host bridge to bus 1012:3f
 1277.584549 (    0.006630)| [  190.505756] pci_bus 1012:3f: root bus resource [bus 3f]
 1277.605269 (    0.020720)| [  190.526904] ACPI: PCI Root Bridge [S013] (domain 1013 [bus 3f])
 1277.609233 (    0.003964)| [  190.533517] acpi PNP0A08:1f: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.621589 (    0.012356)| [  190.542867] acpi PNP0A08:1f: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.630151 (    0.008562)| [  190.551508] acpi PNP0A08:1f: _OSC: OS now controls [PCIeCapability]
 1277.637230 (    0.007079)| [  190.558729] PCI host bridge to bus 1013:3f
 1277.641394 (    0.004164)| [  190.563304] pci_bus 1013:3f: root bus resource [bus 3f]
 1277.660751 (    0.019357)| [  190.584462] ACPI: PCI Root Bridge [S014] (domain 1014 [bus 3f])
 1277.669030 (    0.008279)| [  190.591076] acpi PNP0A08:20: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.677127 (    0.008097)| [  190.600421] acpi PNP0A08:20: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.685844 (    0.008717)| [  190.609063] acpi PNP0A08:20: _OSC: OS now controls [PCIeCapability]
 1277.693601 (    0.007757)| [  190.616281] PCI host bridge to bus 1014:3f
 1277.698002 (    0.004401)| [  190.620858] pci_bus 1014:3f: root bus resource [bus 3f]
 1277.721171 (    0.023169)| [  190.641996] ACPI: PCI Root Bridge [S015] (domain 1015 [bus 3f])
 1277.725273 (    0.004102)| [  190.648611] acpi PNP0A08:21: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.737048 (    0.011775)| [  190.657956] acpi PNP0A08:21: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.744734 (    0.007686)| [  190.666593] acpi PNP0A08:21: _OSC: OS now controls [PCIeCapability]
 1277.753601 (    0.008867)| [  190.673814] PCI host bridge to bus 1015:3f
 1277.757508 (    0.003907)| [  190.678388] pci_bus 1015:3f: root bus resource [bus 3f]
 1277.776973 (    0.019465)| [  190.699660] ACPI: PCI Root Bridge [S016] (domain 1016 [bus 3f])
 1277.785469 (    0.008496)| [  190.706273] acpi PNP0A08:22: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.793756 (    0.008287)| [  190.715617] acpi PNP0A08:22: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.800542 (    0.006786)| [  190.724266] acpi PNP0A08:22: _OSC: OS now controls [PCIeCapability]
 1277.809863 (    0.009321)| [  190.731486] PCI host bridge to bus 1016:3f
 1277.812861 (    0.002998)| [  190.736063] pci_bus 1016:3f: root bus resource [bus 3f]
 1277.832915 (    0.020054)| [  190.757260] ACPI: PCI Root Bridge [S017] (domain 1017 [bus 3f])
 1277.841468 (    0.008553)| [  190.763873] acpi PNP0A08:23: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.849254 (    0.007786)| [  190.773222] acpi PNP0A08:23: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.861004 (    0.011750)| [  190.781867] acpi PNP0A08:23: _OSC: OS now controls [PCIeCapability]
 1277.866041 (    0.005037)| [  190.789085] PCI host bridge to bus 1017:3f
 1277.869326 (    0.003285)| [  190.793662] pci_bus 1017:3f: root bus resource [bus 3f]
 1277.893181 (    0.023855)| [  190.815098] ACPI: PCI Root Bridge [S018] (domain 1018 [bus 3f])
 1277.897134 (    0.003953)| [  190.821712] acpi PNP0A08:24: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.909110 (    0.011976)| [  190.831052] acpi PNP0A08:24: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.917022 (    0.007912)| [  190.839697] acpi PNP0A08:24: _OSC: OS now controls [PCIeCapability]
 1277.925165 (    0.008143)| [  190.846921] PCI host bridge to bus 1018:3f
 1277.929088 (    0.003923)| [  190.851499] pci_bus 1018:3f: root bus resource [bus 3f]
 1277.948889 (    0.019801)| [  190.872888] ACPI: PCI Root Bridge [S019] (domain 1019 [bus 3f])
 1277.958107 (    0.009218)| [  190.879501] acpi PNP0A08:25: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1277.964809 (    0.006702)| [  190.888843] acpi PNP0A08:25: _OSC: platform does not support [PCIeHotplug PME AER]
 1277.973718 (    0.008909)| [  190.897490] acpi PNP0A08:25: _OSC: OS now controls [PCIeCapability]
 1277.981157 (    0.007439)| [  190.904713] PCI host bridge to bus 1019:3f
 1277.984933 (    0.003776)| [  190.909291] pci_bus 1019:3f: root bus resource [bus 3f]
 1278.009020 (    0.024087)| [  190.931012] ACPI: PCI Root Bridge [S01A] (domain 101a [bus 3f])
 1278.017429 (    0.008409)| [  190.937628] acpi PNP0A08:26: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.029991 (    0.012562)| [  190.946984] acpi PNP0A08:26: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.035236 (    0.005245)| [  190.955617] acpi PNP0A08:26: _OSC: OS now controls [PCIeCapability]
 1278.045219 (    0.009983)| [  190.962836] PCI host bridge to bus 101a:3f
 1278.045325 (    0.000106)| [  190.967412] pci_bus 101a:3f: root bus resource [bus 3f]
 1278.067197 (    0.021872)| [  190.989161] ACPI: PCI Root Bridge [S01B] (domain 101b [bus 3f])
 1278.074775 (    0.007578)| [  190.995774] acpi PNP0A08:27: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.080911 (    0.006136)| [  191.005129] acpi PNP0A08:27: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.093141 (    0.012230)| [  191.013770] acpi PNP0A08:27: _OSC: OS now controls [PCIeCapability]
 1278.096750 (    0.003609)| [  191.020988] PCI host bridge to bus 101b:3f
 1278.111518 (    0.014768)| [  191.025565] pci_bus 101b:3f: root bus resource [bus 3f]
 1278.126657 (    0.015139)| [  191.047473] ACPI: PCI Root Bridge [S01C] (domain 101c [bus 3f])
 1278.133377 (    0.006720)| [  191.054086] acpi PNP0A08:28: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.142774 (    0.009397)| [  191.063432] acpi PNP0A08:28: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.149763 (    0.006989)| [  191.072076] acpi PNP0A08:28: _OSC: OS now controls [PCIeCapability]
 1278.157309 (    0.007546)| [  191.079284] PCI host bridge to bus 101c:3f
 1278.161473 (    0.004164)| [  191.083859] pci_bus 101c:3f: root bus resource [bus 3f]
 1278.184872 (    0.023399)| [  191.105283] ACPI: PCI Root Bridge [S01D] (domain 101d [bus 3f])
 1278.189411 (    0.004539)| [  191.111896] acpi PNP0A08:29: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.196647 (    0.007236)| [  191.121244] acpi PNP0A08:29: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.208664 (    0.012017)| [  191.129875] acpi PNP0A08:29: _OSC: OS now controls [PCIeCapability]
 1278.213608 (    0.004944)| [  191.137098] PCI host bridge to bus 101d:3f
 1278.221243 (    0.007635)| [  191.141673] pci_bus 101d:3f: root bus resource [bus 3f]
 1278.240655 (    0.019412)| [  191.163326] ACPI: PCI Root Bridge [S01E] (domain 101e [bus 3f])
 1278.248779 (    0.008124)| [  191.169940] acpi PNP0A08:2a: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.256640 (    0.007861)| [  191.179284] acpi PNP0A08:2a: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.264728 (    0.008088)| [  191.187930] acpi PNP0A08:2a: _OSC: OS now controls [PCIeCapability]
 1278.273158 (    0.008430)| [  191.195151] PCI host bridge to bus 101e:3f
 1278.276714 (    0.003556)| [  191.199726] pci_bus 101e:3f: root bus resource [bus 3f]
 1278.296752 (    0.020038)| [  191.220815] ACPI: PCI Root Bridge [S01F] (domain 101f [bus 3f])
 1278.304926 (    0.008174)| [  191.227430] acpi PNP0A08:2b: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
 1278.313073 (    0.008147)| [  191.236783] acpi PNP0A08:2b: _OSC: platform does not support [PCIeHotplug PME AER]
 1278.320836 (    0.007763)| [  191.245426] acpi PNP0A08:2b: _OSC: OS now controls [PCIeCapability]
 1278.328633 (    0.007797)| [  191.252652] PCI host bridge to bus 101f:3f
 1278.332901 (    0.004268)| [  191.257229] pci_bus 101f:3f: root bus resource [bus 3f]
 1278.356878 (    0.023977)| [  191.278519] ACPI: Enabled 3 GPEs in block 00 to 3F
 1278.373450 (    0.016572)| [  191.293844] vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=none,locks=none
 1278.384250 (    0.010800)| [  191.308013] vgaarb: loaded
 1278.390095 (    0.005845)| [  191.311028] vgaarb: setting as boot device: PCI:0000:09:00.0
 1278.392924 (    0.002829)| [  191.317342] vgaarb: bridge control possible 0000:09:00.0
 1278.401050 (    0.008126)| [  191.324837] Registered efivars operations
 1278.412763 (    0.011713)| [  191.333525] PCI: Using ACPI for IRQ routing
 1278.432610 (    0.019847)| [  191.353883] NetLabel: Initializing
 1278.436789 (    0.004179)| [  191.357683] NetLabel:  domain hash size = 128
 1278.440865 (    0.004076)| [  191.362545] NetLabel:  protocols = UNLABELED CIPSOv4
 1278.445018 (    0.004153)| [  191.368121] NetLabel:  unlabeled traffic allowed by default
 1278.461547 (    0.016529)| [  191.382005] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
 1278.466150 (    0.004603)| [  191.388991] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
 1278.500631 (    0.034481)| [  191.423420] clocksource: Switched to clocksource sgi_rtc
 1278.612531 (    0.111900)| [  191.534355] VFS: Disk quotas dquot_6.6.0
 1278.620479 (    0.007948)| [  191.544661] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
 1278.636719 (    0.016240)| [  191.558183] AppArmor: AppArmor Filesystem Enabled
 1278.640799 (    0.004080)| [  191.563703] pnp: PnP ACPI init
 1278.656342 (    0.015543)| [  191.579776] system 00:01: [io  0x0500-0x053f] has been reserved
 1278.664447 (    0.008105)| [  191.586394] system 00:01: [io  0x0400-0x047f] has been reserved
 1278.668679 (    0.004232)| [  191.593010] system 00:01: [io  0x0540-0x057f] has been reserved
 1278.676568 (    0.007889)| [  191.599625] system 00:01: [io  0x0600-0x061f] has been reserved
 1278.684788 (    0.008220)| [  191.606241] system 00:01: [io  0x0ca0-0x0ca5] has been reserved
 1278.688719 (    0.003931)| [  191.612860] system 00:01: [io  0x0880-0x0883] has been reserved
 1278.696885 (    0.008166)| [  191.619468] system 00:01: [io  0x0800-0x081f] has been reserved
 1278.704981 (    0.008096)| [  191.626092] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved
 1278.714407 (    0.009426)| [  191.633487] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved
 1278.717651 (    0.003244)| [  191.640881] system 00:01: [mem 0xff000000-0xffffffff] has been reserved
 1278.725182 (    0.007531)| [  191.648273] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved
 1278.736990 (    0.011808)| [  191.655668] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved
 1278.740747 (    0.003757)| [  191.663061] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved
 1278.748508 (    0.007761)| [  191.670462] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved
 1278.772579 (    0.024071)| [  191.695072] pnp: PnP ACPI: found 4 devices
 1278.788488 (    0.015909)| [  191.709197] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
 1278.797800 (    0.009312)| [  191.720321] pci 0000:00:02.0: PCI bridge to [bus 01]
 1278.804625 (    0.006825)| [  191.725886] pci 0000:00:03.0: PCI bridge to [bus 02-03]
 1278.808536 (    0.003911)| [  191.731729] pci 0000:00:03.0:   bridge window [io  0x2000-0x2fff]
 1278.816586 (    0.008050)| [  191.738533] pci 0000:00:03.0:   bridge window [mem 0x91b00000-0x91efffff]
 1278.824497 (    0.007911)| [  191.746110] pci 0000:00:03.0:   bridge window [mem 0x2ffe00000000-0x2ffe000fffff 64bit pref]
 1278.832589 (    0.008092)| [  191.755543] pci 0000:00:11.0: PCI bridge to [bus 04]
 1278.836647 (    0.004058)| [  191.761108] pci 0000:00:1c.0: PCI bridge to [bus 05]
 1278.844649 (    0.008002)| [  191.766659] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
 1278.852471 (    0.007822)| [  191.773472] pci 0000:00:1c.0:   bridge window [mem 0x91a00000-0x91afffff]
 1278.858258 (    0.005787)| [  191.781069] pci 0000:08:00.0: PCI bridge to [bus 09]
 1278.864640 (    0.006382)| [  191.786759] pci 0000:08:00.0:   bridge window [mem 0x91000000-0x918fffff]
 1278.872497 (    0.007857)| [  191.794439] pci 0000:08:00.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
 1278.880469 (    0.007972)| [  191.803283] pci 0000:07:00.0: PCI bridge to [bus 08-09]
 1278.888687 (    0.008218)| [  191.809262] pci 0000:07:00.0:   bridge window [mem 0x91000000-0x918fffff]
 1278.892624 (    0.003937)| [  191.816937] pci 0000:07:00.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
 1278.904196 (    0.011572)| [  191.825781] pci 0000:07:01.0: PCI bridge to [bus 0a]
 1278.908828 (    0.004632)| [  191.831472] pci 0000:07:01.0:   bridge window [mem 0x91900000-0x919fffff]
 1278.916573 (    0.007745)| [  191.839400] pci 0000:06:00.0: PCI bridge to [bus 07-0a]
 1278.925090 (    0.008517)| [  191.845366] pci 0000:06:00.0:   bridge window [mem 0x91000000-0x919fffff]
 1278.928568 (    0.003478)| [  191.853031] pci 0000:06:00.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
 1278.940760 (    0.012192)| [  191.861920] pci 0000:00:1c.7: PCI bridge to [bus 06-0a]
 1278.945097 (    0.004337)| [  191.867764] pci 0000:00:1c.7:   bridge window [mem 0x91000000-0x919fffff]
 1278.954153 (    0.009056)| [  191.875351] pci 0000:00:1c.7:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
 1278.960724 (    0.006571)| [  191.884009] pci 0000:00:1e.0: PCI bridge to [bus 0b]
 1278.969309 (    0.008585)| [  191.889837] pci 0001:00:02.0: PCI bridge to [bus 01-02]
 1278.973162 (    0.003853)| [  191.895681] pci 0001:00:02.0:   bridge window [mem 0x92000000-0x920fffff]
 1278.980945 (    0.007783)| [  191.903266] pci 0001:00:02.0:   bridge window [mem 0x2ffe04000000-0x2ffe040fffff 64bit pref]
 1278.988525 (    0.007580)| [  191.912699] pci 0001:00:03.0: PCI bridge to [bus 03]
 1278.997260 (    0.008735)| [  191.918254] pci 0001:00:03.2: PCI bridge to [bus 04]
 1279.000578 (    0.003318)| [  191.923853] pci 0002:00:02.0: PCI bridge to [bus 01]
 1279.013483 (    0.012905)| [  191.929410] pci 0002:00:02.2: PCI bridge to [bus 02]
 1279.013621 (    0.000138)| [  191.934964] pci 0002:00:03.0: PCI bridge to [bus 03-04]
 1279.016487 (    0.002866)| [  191.940802] pci 0002:00:03.0:   bridge window [mem 0x92400000-0x924fffff]
 1279.024600 (    0.008113)| [  191.948387] pci 0002:00:03.0:   bridge window [mem 0x2ffe08000000-0x2ffe080fffff 64bit pref]
 1279.036532 (    0.011932)| [  191.957856] pci 0003:00:02.0: PCI bridge to [bus 01]
 1279.040598 (    0.004066)| [  191.963413] pci 0003:00:03.0: PCI bridge to [bus 02-03]
 1279.048588 (    0.007990)| [  191.969251] pci 0003:00:03.0:   bridge window [io  0x4000-0x4fff]
 1279.052787 (    0.004199)| [  191.976064] pci 0003:00:03.0:   bridge window [mem 0x92800000-0x92bfffff]
 1279.060467 (    0.007680)| [  191.983651] pci 0003:00:03.0:   bridge window [mem 0x2ffe0c000000-0x2ffe0c0fffff 64bit pref]
 1279.068738 (    0.008271)| [  191.993133] pci 0004:00:02.0: PCI bridge to [bus 01-02]
 1279.077072 (    0.008334)| [  191.998971] pci 0004:00:02.0:   bridge window [mem 0x93000000-0x930fffff]
 1279.084914 (    0.007842)| [  192.006559] pci 0004:00:02.0:   bridge window [mem 0x2ffe10000000-0x2ffe100fffff 64bit pref]
 1279.092849 (    0.007935)| [  192.015983] pci 0004:00:03.0: PCI bridge to [bus 03]
 1279.101038 (    0.008189)| [  192.021538] pci 0004:00:03.2: PCI bridge to [bus 04]
 1279.104563 (    0.003525)| [  192.027139] pci 0005:00:02.0: PCI bridge to [bus 01]
 1279.108773 (    0.004210)| [  192.032697] pci 0005:00:02.2: PCI bridge to [bus 02]
 1279.116354 (    0.007581)| [  192.038251] pci 0005:00:03.0: PCI bridge to [bus 03-04]
 1279.120411 (    0.004057)| [  192.044091] pci 0005:00:03.0:   bridge window [mem 0x93400000-0x934fffff]
 1279.128635 (    0.008224)| [  192.051680] pci 0005:00:03.0:   bridge window [mem 0x2ffe14000000-0x2ffe140fffff 64bit pref]
 1279.140490 (    0.011855)| [  192.061142] pci 0006:00:02.0: PCI bridge to [bus 01]
 1279.144449 (    0.003959)| [  192.066699] pci 0006:00:03.0: PCI bridge to [bus 02-03]
 1279.149266 (    0.004817)| [  192.072538] pci 0006:00:03.0:   bridge window [io  0x5000-0x5fff]
 1279.156437 (    0.007171)| [  192.079353] pci 0006:00:03.0:   bridge window [mem 0x93800000-0x93bfffff]
 1279.165302 (    0.008865)| [  192.086939] pci 0006:00:03.0:   bridge window [mem 0x2ffe18000000-0x2ffe180fffff 64bit pref]
 1279.172764 (    0.007462)| [  192.096417] pci 0007:00:02.0: PCI bridge to [bus 01-02]
 1279.181644 (    0.008880)| [  192.102262] pci 0007:00:02.0:   bridge window [mem 0x94000000-0x940fffff]
 1279.188391 (    0.006747)| [  192.109851] pci 0007:00:02.0:   bridge window [mem 0x2ffe1c000000-0x2ffe1c0fffff 64bit pref]
 1279.196692 (    0.008301)| [  192.119285] pci 0007:00:03.0: PCI bridge to [bus 03]
 1279.200487 (    0.003795)| [  192.124839] pci 0007:00:03.2: PCI bridge to [bus 04]
 1279.208416 (    0.007929)| [  192.130441] pci 0008:00:02.0: PCI bridge to [bus 01]
 1279.212489 (    0.004073)| [  192.135999] pci 0008:00:02.2: PCI bridge to [bus 02]
 1279.220792 (    0.008303)| [  192.141555] pci 0008:00:03.0: PCI bridge to [bus 03-04]
 1279.224500 (    0.003708)| [  192.147395] pci 0008:00:03.0:   bridge window [mem 0x94400000-0x944fffff]
 1279.232519 (    0.008019)| [  192.154978] pci 0008:00:03.0:   bridge window [mem 0x2ffe20000000-0x2ffe200fffff 64bit pref]
 1279.240450 (    0.007931)| [  192.164451] pci 0009:00:02.0: PCI bridge to [bus 01]
 1279.249298 (    0.008848)| [  192.170009] pci 0009:00:03.0: PCI bridge to [bus 02-03]
 1279.252957 (    0.003659)| [  192.175850] pci 0009:00:03.0:   bridge window [io  0x6000-0x6fff]
 1279.260536 (    0.007579)| [  192.182659] pci 0009:00:03.0:   bridge window [mem 0x94800000-0x94bfffff]
 1279.268966 (    0.008430)| [  192.190246] pci 0009:00:03.0:   bridge window [mem 0x2ffe24000000-0x2ffe240fffff 64bit pref]
 1279.276976 (    0.008010)| [  192.199728] pci 000a:00:02.0: PCI bridge to [bus 01-02]
 1279.285251 (    0.008275)| [  192.205572] pci 000a:00:02.0:   bridge window [mem 0x95000000-0x950fffff]
 1279.289023 (    0.003772)| [  192.213157] pci 000a:00:02.0:   bridge window [mem 0x2ffe28000000-0x2ffe280fffff 64bit pref]
 1279.300695 (    0.011672)| [  192.222589] pci 000a:00:03.0: PCI bridge to [bus 03]
 1279.306262 (    0.005567)| [  192.228144] pci 000a:00:03.2: PCI bridge to [bus 04]
 1279.311951 (    0.005689)| [  192.233744] pci 000b:00:02.0: PCI bridge to [bus 01]
 1279.316507 (    0.004556)| [  192.239298] pci 000b:00:02.2: PCI bridge to [bus 02]
 1279.320476 (    0.003969)| [  192.244854] pci 000b:00:03.0: PCI bridge to [bus 03-04]
 1279.328537 (    0.008061)| [  192.250693] pci 000b:00:03.0:   bridge window [mem 0x95400000-0x954fffff]
 1279.337154 (    0.008617)| [  192.258277] pci 000b:00:03.0:   bridge window [mem 0x2ffe2c000000-0x2ffe2c0fffff 64bit pref]
 1279.368549 (    0.031395)| [  192.290473] NET: Registered protocol family 2
 1279.405024 (    0.036475)| [  192.325992] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
 1279.412769 (    0.007745)| [  192.336828] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
 1279.424201 (    0.011432)| [  192.344846] TCP: Hash tables configured (established 524288 bind 65536)
 1279.432474 (    0.008273)| [  192.354105] UDP hash table entries: 65536 (order: 9, 2097152 bytes)
 1279.440416 (    0.007942)| [  192.362340] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes)
 1279.465132 (    0.024716)| [  192.388267] NET: Registered protocol family 1
 1279.484412 (    0.019280)| [  192.405017] Unpacking initramfs...
 1281.048077 (    1.563665)| [  193.971926] Freeing initrd memory: 17500K (ffff88003eee4000 - ffff88003fffb000)
 1281.071649 (    0.023572)| [  193.993881] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
 1281.079792 (    0.008143)| [  194.001078] software IO TLB [mem 0x78983000-0x7c983000] (64MB) mapped at [ffff880078983000-ffff88007c982fff]
 1281.092023 (    0.012231)| [  194.013524] Intel CQM monitoring enabled
 1281.095826 (    0.003803)| [  194.017911] Intel MBM enabled
 1281.139707 (    0.043881)| [  194.063284] RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer
 1281.147816 (    0.008109)| [  194.072131] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
 1281.155973 (    0.008157)| [  194.078548] RAPL PMU: hw unit of domain package 2^-14 Joules
 1281.163883 (    0.007910)| [  194.084866] RAPL PMU: hw unit of domain dram 2^-14 Joules
 1281.167982 (    0.004099)| [  194.090901] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
 1281.332134 (    0.164152)| [  194.255556] Scanning for low memory corruption every 60 seconds
 1281.424367 (    0.092233)| [  194.344724] futex hash table entries: 524288 (order: 13, 33554432 bytes)
 1281.451972 (    0.027605)| [  194.373367] audit: initializing netlink subsys (disabled)
 1281.456255 (    0.004283)| [  194.379469] audit: type=2000 audit(1479413930.172:1): initialized
 1281.467698 (    0.011443)| [  194.389581] Initialise system trusted keyrings
 1281.491761 (    0.024063)| [  194.412800] workingset: timestamp_bits=36 max_order=32 bucket_order=0
 1281.516011 (    0.024250)| [  194.438093] zbud: loaded
 1281.579616 (    0.063605)| [  194.500285] Key type asymmetric registered
 1281.583650 (    0.004034)| [  194.504867] Asymmetric key parser 'x509' registered
 1281.592175 (    0.008525)| [  194.514190] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
 1281.607566 (    0.015391)| [  194.529948] io scheduler noop registered
 1281.612158 (    0.004592)| [  194.534333] io scheduler deadline registered
 1281.615579 (    0.003421)| [  194.539115] io scheduler cfq registered (default)
 1282.279406 (    0.663827)| [  195.201256] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt
 1282.288971 (    0.009565)| [  195.209097] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
 1282.295570 (    0.006599)| [  195.216877] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt
 1282.300954 (    0.005384)| [  195.224174] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt
 1282.327314 (    0.026360)| [  195.250628] pcieport 0000:00:11.0: Signaling PME through PCIe PME interrupt
 1282.335374 (    0.008060)| [  195.258552] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt
 1282.344474 (    0.009100)| [  195.266331] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt
 1282.351305 (    0.006831)| [  195.273727] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt
 1282.359783 (    0.008478)| [  195.281511] pcieport 0000:06:00.0: Signaling PME through PCIe PME interrupt
 1282.367596 (    0.007813)| [  195.289290] pcieport 0000:07:00.0: Signaling PME through PCIe PME interrupt
 1282.374575 (    0.006979)| [  195.297076] pci 0000:08:00.0: Signaling PME through PCIe PME interrupt
 1282.383452 (    0.008877)| [  195.304373] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt
 1282.388807 (    0.005355)| [  195.305124] tsc: Refined TSC clocksource calibration: 2200.000 MHz
 1282.397718 (    0.008911)| [  195.306485] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fb633008a4, max_idle_ns: 440795292230 ns
 1282.407646 (    0.009928)| [  195.329755] pcieport 0000:07:01.0: Signaling PME through PCIe PME interrupt
 1282.415554 (    0.007908)| [  195.337537] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt
 1282.423572 (    0.008018)| [  195.344871] pcieport 0001:00:02.0: Signaling PME through PCIe PME interrupt
 1282.431791 (    0.008219)| [  195.352653] pci 0001:01:00.0: Signaling PME through PCIe PME interrupt
 1282.435465 (    0.003674)| [  195.359954] pci 0001:01:00.1: Signaling PME through PCIe PME interrupt
 1282.463441 (    0.027976)| [  195.385214] pcieport 0001:00:03.0: Signaling PME through PCIe PME interrupt
 1282.492070 (    0.028629)| [  195.409174] pcieport 0001:00:03.2: Signaling PME through PCIe PME interrupt
 1282.511493 (    0.019423)| [  195.433184] pcieport 0002:00:02.0: Signaling PME through PCIe PME interrupt
 1282.535865 (    0.024372)| [  195.457179] pcieport 0002:00:02.2: Signaling PME through PCIe PME interrupt
 1282.543571 (    0.007706)| [  195.464998] pcieport 0002:00:03.0: Signaling PME through PCIe PME interrupt
 1282.551962 (    0.008391)| [  195.472778] pci 0002:03:00.0: Signaling PME through PCIe PME interrupt
 1282.559639 (    0.007677)| [  195.480075] pci 0002:03:00.1: Signaling PME through PCIe PME interrupt
 1282.583600 (    0.023961)| [  195.505197] pcieport 0003:00:02.0: Signaling PME through PCIe PME interrupt
 1282.591506 (    0.007906)| [  195.513011] pcieport 0003:00:03.0: Signaling PME through PCIe PME interrupt
 1282.600336 (    0.008830)| [  195.520788] pci 0003:02:00.0: Signaling PME through PCIe PME interrupt
 1282.604488 (    0.004152)| [  195.528087] pci 0003:02:00.1: Signaling PME through PCIe PME interrupt
 1282.612326 (    0.007838)| [  195.535412] pcieport 0004:00:02.0: Signaling PME through PCIe PME interrupt
 1282.620267 (    0.007941)| [  195.543194] pci 0004:01:00.0: Signaling PME through PCIe PME interrupt
 1282.628113 (    0.007846)| [  195.550491] pci 0004:01:00.1: Signaling PME through PCIe PME interrupt
 1282.655561 (    0.027448)| [  195.577202] pcieport 0004:00:03.0: Signaling PME through PCIe PME interrupt
 1282.680552 (    0.024991)| [  195.601192] pcieport 0004:00:03.2: Signaling PME through PCIe PME interrupt
 1282.703298 (    0.022746)| [  195.625181] pcieport 0005:00:02.0: Signaling PME through PCIe PME interrupt
 1282.727311 (    0.024013)| [  195.649178] pcieport 0005:00:02.2: Signaling PME through PCIe PME interrupt
 1282.735341 (    0.008030)| [  195.657003] pcieport 0005:00:03.0: Signaling PME through PCIe PME interrupt
 1282.743570 (    0.008229)| [  195.664778] pci 0005:03:00.0: Signaling PME through PCIe PME interrupt
 1282.751536 (    0.007966)| [  195.672073] pci 0005:03:00.1: Signaling PME through PCIe PME interrupt
 1282.775357 (    0.023821)| [  195.697189] pcieport 0006:00:02.0: Signaling PME through PCIe PME interrupt
 1282.783485 (    0.008128)| [  195.704996] pcieport 0006:00:03.0: Signaling PME through PCIe PME interrupt
 1282.791413 (    0.007928)| [  195.712772] pci 0006:02:00.0: Signaling PME through PCIe PME interrupt
 1282.799218 (    0.007805)| [  195.720069] pci 0006:02:00.1: Signaling PME through PCIe PME interrupt
 1282.803532 (    0.004314)| [  195.727397] pcieport 0007:00:02.0: Signaling PME through PCIe PME interrupt
 1282.812977 (    0.009445)| [  195.735177] pci 0007:01:00.0: Signaling PME through PCIe PME interrupt
 1282.819334 (    0.006357)| [  195.742474] pci 0007:01:00.1: Signaling PME through PCIe PME interrupt
 1282.848144 (    0.028810)| [  195.769218] pcieport 0007:00:03.0: Signaling PME through PCIe PME interrupt
 1282.871313 (    0.023169)| [  195.793200] pcieport 0007:00:03.2: Signaling PME through PCIe PME interrupt
 1282.895523 (    0.024210)| [  195.817202] pcieport 0008:00:02.0: Signaling PME through PCIe PME interrupt
 1282.919556 (    0.024033)| [  195.841188] pcieport 0008:00:02.2: Signaling PME through PCIe PME interrupt
 1282.927248 (    0.007692)| [  195.849001] pcieport 0008:00:03.0: Signaling PME through PCIe PME interrupt
 1282.935603 (    0.008355)| [  195.856777] pci 0008:03:00.0: Signaling PME through PCIe PME interrupt
 1282.942677 (    0.007074)| [  195.864075] pci 0008:03:00.1: Signaling PME through PCIe PME interrupt
 1282.967423 (    0.024746)| [  195.889172] pcieport 0009:00:02.0: Signaling PME through PCIe PME interrupt
 1282.975334 (    0.007911)| [  195.896977] pcieport 0009:00:03.0: Signaling PME through PCIe PME interrupt
 1282.983156 (    0.007822)| [  195.904753] pci 0009:02:00.0: Signaling PME through PCIe PME interrupt
 1282.991332 (    0.008176)| [  195.912052] pci 0009:02:00.1: Signaling PME through PCIe PME interrupt
 1282.995439 (    0.004107)| [  195.919371] pcieport 000a:00:02.0: Signaling PME through PCIe PME interrupt
 1283.003139 (    0.007700)| [  195.927152] pci 000a:01:00.0: Signaling PME through PCIe PME interrupt
 1283.011373 (    0.008234)| [  195.934447] pci 000a:01:00.1: Signaling PME through PCIe PME interrupt
 1283.039076 (    0.027703)| [  195.961203] pcieport 000a:00:03.0: Signaling PME through PCIe PME interrupt
 1283.063643 (    0.024567)| [  195.985207] pcieport 000a:00:03.2: Signaling PME through PCIe PME interrupt
 1283.087296 (    0.023653)| [  196.009192] pcieport 000b:00:02.0: Signaling PME through PCIe PME interrupt
 1283.112083 (    0.024787)| [  196.033190] pcieport 000b:00:02.2: Signaling PME through PCIe PME interrupt
 1283.119508 (    0.007425)| [  196.040998] pcieport 000b:00:03.0: Signaling PME through PCIe PME interrupt
 1283.127696 (    0.008188)| [  196.048774] pci 000b:03:00.0: Signaling PME through PCIe PME interrupt
 1283.135612 (    0.007916)| [  196.056071] pci 000b:03:00.1: Signaling PME through PCIe PME interrupt
 1283.139274 (    0.003662)| [  196.063404] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
 1283.147477 (    0.008203)| [  196.069655] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
 1283.159492 (    0.012015)| [  196.080208] intel_idle: max_cstate 1 reached
 1283.415437 (    0.255945)| [  196.335981] clocksource: Switched to clocksource tsc
 1283.887372 (    0.471935)| [  196.807610] GHES: HEST is not enabled!
 1283.890717 (    0.003345)| [  196.815730] Serial: 8250/16550 driver, 32 ports, IRQ sharing disabled
 1283.923070 (    0.032353)| [  196.843544] 00:02: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
 1283.951216 (    0.028146)| [  196.872407] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
 1283.962540 (    0.011324)| [  196.886297] KGDB: Registered I/O driver kgdboc
 1283.967019 (    0.004479)| [  196.891263] ------------[ cut here ]------------
 1283.974809 (    0.007790)| [  196.896422] WARNING: CPU: 1535 PID: 1 at arch/x86/kernel/nmi.c:171 __register_nmi_handler+0x150/0x190
 1283.982966 (    0.008157)| [  196.906715] Modules linked in:
 1283.986890 (    0.003924)| [  196.910140] CPU: 1535 PID: 1 Comm: swapper/0 Not tainted 4.9.0-rc5-clean+ #104
 1283.995206 (    0.008316)| [  196.918202] Hardware name: SGI UV300/UV300, BIOS SGI UV 300 series BIOS 08/09/2016
 1284.003374 (    0.008168)| [  196.926654]  ffffc9000b58bd10 ffffffff81333e6c 0000000000000000 0000000000000000
 1284.011599 (    0.008225)| [  196.934957]  ffffc9000b58bd50 ffffffff81071a91 000000ab009c9aa0 ffffffff81c302e0
 1284.019668 (    0.008069)| [  196.943260]  0000000000000018 0000000000000001 ffffffff81c22838 ffffffff81c22840
 1284.030869 (    0.011201)| [  196.951563] Call Trace:
 1284.030941 (    0.000072)| [  196.954306]  [<ffffffff81333e6c>] dump_stack+0x63/0x87
 1284.039420 (    0.008479)| [  196.960049]  [<ffffffff81071a91>] __warn+0xd1/0xf0
 1284.042566 (    0.003146)| [  196.965395]  [<ffffffff81071b7d>] warn_slowpath_null+0x1d/0x20
 1284.050933 (    0.008367)| [  196.971903]  [<ffffffff810200c0>] __register_nmi_handler+0x150/0x190
 1284.054886 (    0.003953)| [  196.979006]  [<ffffffff8104da76>] kgdb_arch_init+0x46/0x70
 1284.062922 (    0.008036)| [  196.985135]  [<ffffffff811274d9>] kgdb_register_io_module+0xc9/0x150
 1284.069284 (    0.006362)| [  196.992237]  [<ffffffff8144cbf8>] configure_kgdboc+0x108/0x1a0
 1284.075085 (    0.005801)| [  196.998762]  [<ffffffff81dc1a37>] ? lpss8250_pci_driver_init+0x1b/0x1b
 1284.082964 (    0.007879)| [  197.006049]  [<ffffffff81dc1a4b>] init_kgdboc+0x14/0x16
 1284.090978 (    0.008014)| [  197.011881]  [<ffffffff81000450>] do_one_initcall+0x50/0x190
 1284.094947 (    0.003969)| [  197.018204]  [<ffffffff81090d7b>] ? parse_args+0x27b/0x460
 1284.103261 (    0.008314)| [  197.024338]  [<ffffffff81d792a3>] kernel_init_freeable+0x196/0x226
 1284.107005 (    0.003744)| [  197.031237]  [<ffffffff81d789aa>] ? set_debug_rodata+0x12/0x12
 1284.115221 (    0.008216)| [  197.037758]  [<ffffffff81621790>] ? rest_init+0x80/0x80
 1284.123070 (    0.007849)| [  197.043581]  [<ffffffff8162179e>] kernel_init+0xe/0x110
 1284.126966 (    0.003896)| [  197.049422]  [<ffffffff8162e295>] ret_from_fork+0x25/0x30
 1284.131008 (    0.004042)| [  197.055449] ---[ end trace 552967228ff7cc5d ]---
 1284.259325 (    0.128317)| [  197.179957] Non-volatile memory driver v1.3
 1284.263194 (    0.003869)| [  197.184694] Linux agpgart interface v0.103
 1284.750691 (    0.487497)| [  197.674342] i8042: PNP: No PS/2 controller found. Probing ports directly.
 1285.794794 (    1.044103)| [  198.718197] i8042: No controller found
 1285.798472 (    0.003678)| [  198.722610] mousedev: PS/2 mouse device common for all mice
 1285.806587 (    0.008115)| [  198.729015] rtc_cmos 00:00: RTC can wake from S4
 1285.810247 (    0.003660)| [  198.734457] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
 1285.818544 (    0.008297)| [  198.741364] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
 1285.826355 (    0.007811)| [  198.749947] intel_pstate: Intel P-state driver initializing
 1285.951469 (    0.125114)| [  198.873363] ledtrig-cpu: registered to indicate activity on CPUs
 1285.958360 (    0.006891)| [  198.880084] EFI Variables Facility v0.08 2004-May-17
 1285.966465 (    0.008105)| [  198.887412] hidraw: raw HID events driver (C) Jiri Kosina
 1285.995266 (    0.028801)| [  198.918596] NET: Registered protocol family 10
 1286.170524 (    0.175258)| [  199.094830] microcode: sig=0x406f1, pf=0x80, revision=0xb00001d
 1286.206566 (    0.036042)| [  199.128973] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
 1286.219689 (    0.013123)| [  199.139346] registered taskstats version 1
 1286.225491 (    0.005802)| [  199.143939] Loading compiled-in X.509 certificates
 1286.230432 (    0.004941)| [  199.153672] alg: No test for pkcs1pad(rsa,sha256) (pkcs1pad(rsa-generic,sha256))
 1286.242449 (    0.012017)| [  199.163348] Loaded X.509 cert 'Build time autogenerated kernel key: 7b47db3b0870d4f59ebefb457b7103770bbfd5f9'
 1286.258651 (    0.016202)| [  199.182058] zswap: loaded using pool lzo/zbud
 1286.286451 (    0.027800)| [  199.209944] page_owner is disabled
 1286.294772 (    0.008321)| [  199.216155] Key type trusted registered
 1286.302949 (    0.008177)| [  199.225613] Key type encrypted registered
 1286.307022 (    0.004073)| [  199.230094] AppArmor: AppArmor sha1 policy hashing enabled
 1286.314520 (    0.007498)| [  199.236224] ima: No TPM chip found, activating TPM-bypass!
 1286.318887 (    0.004367)| [  199.242439] evm: HMAC attrs: 0x1
 1286.370122 (    0.051235)| [  199.289890]   Magic number: 8:701:346
 1286.371291 (    0.001169)| [  199.294146] machinecheck machinecheck798: hash matches
 1286.378769 (    0.007478)| [  199.300040] machinecheck machinecheck117: hash matches
 1286.390511 (    0.011742)| [  199.311433] thermal cooling_device1043: hash matches
 1286.394708 (    0.004197)| [  199.318625] thermal cooling_device126: hash matches
 1286.403014 (    0.008306)| [  199.325699] clockevents clockevent756: hash matches
 1286.410508 (    0.007494)| [  199.332165] tty tty63: hash matches
 1286.414289 (    0.003781)| [  199.336323] pci 101a:3f:15.3: hash matches
 1286.418662 (    0.004373)| [  199.341152] pci 1013:3f:14.5: hash matches
 1286.422387 (    0.003725)| [  199.345941] pci 100d:3f:10.6: hash matches
 1286.426453 (    0.004066)| [  199.350543] pci 100c:3f:13.0: hash matches
 1286.434451 (    0.007998)| [  199.355378] pci 1005:3f:12.2: hash matches
 1286.438287 (    0.003836)| [  199.362243] acpi ACPI0007:08: hash matches
 1286.443064 (    0.004777)| [  199.367199] processor cpu1256: hash matches
 1286.450014 (    0.006950)| [  199.372323] processor cpu764: hash matches
 1286.454397 (    0.004383)| [  199.378570] memory memory14342: hash matches
 1286.462329 (    0.007932)| [  199.384230] memory memory12415: hash matches
 1286.466581 (    0.004252)| [  199.389642] memory memory10974: hash matches
 1286.470337 (    0.003756)| [  199.394900] memory memory9973: hash matches
 1286.478458 (    0.008121)| [  199.401223] memory memory6760: hash matches
 1286.482394 (    0.003936)| [  199.407188] memory memory4192: hash matches
 1286.490434 (    0.008040)| [  199.411987] memory memory3802: hash matches
 1286.494477 (    0.004043)| [  199.417247] memory memory2711: hash matches
 1286.498736 (    0.004259)| [  199.422097] memory memory2265: hash matches
 1286.506815 (    0.008079)| [  199.427782] memory memory192: hash matches
 1286.518216 (    0.011401)| [  199.442569] Freeing unused kernel memory: 2272K (ffffffff81d5f000 - ffffffff81f97000)
 1286.530634 (    0.012418)| [  199.451315] Write protecting the kernel read-only data: 12288k
 1286.542644 (    0.012010)| [  199.466909] Freeing unused kernel memory: 1832K (ffff880001636000 - ffff880001800000)
 1286.558372 (    0.015728)| [  199.481757] Freeing unused kernel memory: 1036K (ffff880001afd000 - ffff880001c00000)
 1286.574490 (    0.016118)| [  199.499002] autofs4: module verification failed: signature and/or required key missing - tainting kernel
 1286.763154 (    0.188664)| [  199.686803] systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
 1286.810915 (    0.047761)| [  199.734995] systemd[1]: Detected architecture x86-64.
 1286.819217 (    0.008302)| [  199.740638] systemd[1]: Running in initial RAM disk.
 1286.822698 (    0.003481)| 
 1286.822705 (    0.000007)| Welcome to ^[[0;34mSUSE Linux Enterprise Server 12 SP2 dracut-044-87.1 (Initramfs)^[[0m!
 1286.830329 (    0.007624)| 
 1286.842581 (    0.012252)| [  199.765464] systemd[1]: Set hostname to <oracle>.
 1287.066559 (    0.223978)| [  199.988943] systemd[1]: Listening on Journal Audit Socket.
 1287.074371 (    0.007812)| [^[[0;32m  OK  ^[[0m] Listening on Journal Audit Socket.
 1287.086249 (    0.011878)| [  200.010296] systemd[1]: Listening on Journal Socket (/dev/log).
 1287.094424 (    0.008175)| [^[[0;32m  OK  ^[[0m] Listening on Journal Socket (/dev/log).
 1287.114963 (    0.020539)| [  200.035927] systemd[1]: Created slice System Slice.
 1287.118722 (    0.003759)| [^[[0;32m  OK  ^[[0m] Created slice System Slice.
 1287.138399 (    0.019677)| [  200.059454] systemd[1]: Created slice system-systemd\x2dhibernate\x2dresume.slice.
 1287.146640 (    0.008241)| [^[[0;32m  OK  ^[[0m] Created slice system-systemd\x2dhibernate\x2dresume.slice.
 1287.162919 (    0.016279)| [  200.086233] systemd[1]: Reached target Slices.
 1287.167567 (    0.004648)| [^[[0;32m  OK  ^[[0m] Reached target Slices.
 1287.182625 (    0.015058)| [  200.106269] systemd[1]: Listening on udev Control Socket.
 1287.190754 (    0.008129)| [^[[0;32m  OK  ^[[0m] Listening on udev Control Socket.
 1287.202684 (    0.011930)| [^[[0;32m  OK  ^[[0m] Reached target Swap.
 1287.220002 (    0.017318)| [^[[0;32m  OK  ^[[0m] Listening on Journal Socket.
 1287.238761 (    0.018759)|          Starting Load Kernel Modules...
 1287.255393 (    0.016632)|          Starting Create list of required st... nodes for the current kernel...
 1287.274622 (    0.019229)|          Starting Setup Virtual Console...
 1287.282725 (    0.008103)| [  200.206401] SCSI subsystem initialized
 1287.294871 (    0.012146)| [^[[0;32m  OK  ^[[0m] Started Entropy Daemon based on the HAVEGE algorithm.
 1287.318567 (    0.023696)|          Starting Journal Service...
 1287.322784 (    0.004217)| [  200.246814] alua: device handler registered
 1287.342720 (    0.019936)|          Starting dracut cmdline hook...
 1287.346631 (    0.003911)| [  200.271323] emc: device handler registered
 1287.358425 (    0.011794)| [^[[0;32m  OK  ^[[0m] Reached target Timers.
 1287.370799 (    0.012374)| [  200.293035] rdac: device handler registered
 1287.378911 (    0.008112)| [^[[0;32m  OK  ^[[0m] Listening on udev Kernel Socket.
 1287.398836 (    0.019925)| [^[[0;32m  OK  ^[[0m] Reached target Sockets.
 1287.406834 (    0.007998)| [  200.330595] device-mapper: uevent: version 1.0.3
 1287.418668 (    0.011834)| [  200.341043] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
 1287.426577 (    0.007909)| [^[[0;32m  OK  ^[[0m] Started Create list of required sta...ce nodes for the current kernel.
 1287.450428 (    0.023851)| [^[[0;32m  OK  ^[[0m] Started Setup Virtual Console.
 1287.470445 (    0.020017)| [^[[0;32m  OK  ^[[0m] Started dracut cmdline hook.
 1287.478554 (    0.008109)| [  200.399660] hwperf: loading out-of-tree module taints kernel.
 1287.498741 (    0.020187)|          Starting Create Static Device Nodes in /dev...
 1287.522546 (    0.023805)| [^[[0;32m  OK  ^[[0m] Started Create Static Device Nodes in /dev.
 1287.546425 (    0.023879)|          Starting udev Kernel Device Manager...
 1287.574596 (    0.028171)| [^[[0;32m  OK  ^[[0m] Started udev Kernel Device Manager.
 1287.582785 (    0.008189)| [  200.504356] SGI UV Memory Mapped RTC Timer: v1.0, 200 MHz
 1287.610523 (    0.027738)|          Starting dracut pre-trigger hook...
 1287.646687 (    0.036164)| [^[[0;32m  OK  ^[[0m] Started Load Kernel Modules.
 1287.675380 (    0.028693)|          Starting Apply Kernel Variables...
 1287.683324 (    0.007944)| [  200.605618] perf: Dynamic interrupt throttling disabled, can hang your system!
 1287.706450 (    0.023126)| [^[[0;1;31mFAILED^[[0m] Failed to start Apply Kernel Variables.
 1287.719700 (    0.013250)| See 'systemctl status systemd-sysctl.service' for details.
 1287.734565 (    0.014865)| [^[[0;32m  OK  ^[[0m] Started dracut pre-trigger hook.
 1287.738467 (    0.003902)| [  200.662388] random: crng init done
 1287.759294 (    0.020827)|          Starting udev Coldplug all Devices...
 1288.006669 (    0.247375)| [^[[0;32m  OK  ^[[0m] Started Journal Service.
 1289.246856 (    1.240187)| [  202.171078] ACPI: bus type USB registered
 1289.250718 (    0.003862)| [  202.171269] [drm] Initialized
 1289.257551 (    0.006833)| [  202.179144] usbcore: registered new interface driver usbfs
 1289.267666 (    0.010115)| [  202.185385] usbcore: registered new interface driver hub
 1289.278926 (    0.011260)| [  202.201950] Emulex LightPulse Fibre Channel SCSI driver 11.2.0.0.
 1289.286860 (    0.007934)| [  202.202978] usbcore: registered new device driver usb
 1289.291554 (    0.004694)| [  202.214403] Copyright(c) 2004-2016 Emulex.  All rights reserved.
 1289.298719 (    0.007165)| [  202.221603] lpfc 0001:01:00.0: enabling device (0140 -> 0142)
 1289.380105 (    0.081386)| [  202.303416] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x7 impl RAID mode
 1289.388892 (    0.008787)| [  202.304962] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
 1289.399926 (    0.011034)| [  202.319752] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
 1289.448193 (    0.048267)| [  202.367909] ehci-pci: EHCI PCI platform driver
 1289.451579 (    0.003386)| [  202.373446] scsi host0: ahci
 1289.455291 (    0.003712)| [  202.377130] scsi host1: ahci
 1289.460305 (    0.005014)| [  202.381501] scsi host2: ahci
 1289.464530 (    0.004225)| [  202.386101] scsi host3: ahci
 1289.470047 (    0.005517)| [  202.389993] scsi host4: ahci
 1289.473546 (    0.003499)| [  202.393939] scsi host5: ahci
 1289.475037 (    0.001491)| [  202.397253] ata1: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00100 irq 19
 1289.483264 (    0.008227)| [  202.405519] ata2: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00180 irq 19
 1289.490783 (    0.007519)| [  202.411952] scsi host6: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 00 irq 74
 1289.503588 (    0.012805)| [  202.424093] ata3: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00200 irq 19
 1289.511393 (    0.007805)| [  202.432356] ata4: DUMMY
 1289.511412 (    0.000019)| [  202.435089] ata5: DUMMY
 1289.516128 (    0.004716)| [  202.437822] ata6: DUMMY
 1289.532926 (    0.016798)| [  202.455018] ehci-pci 0000:00:1a.0: EHCI Host Controller
 1289.539060 (    0.006134)| [  202.460899] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1
 1289.547117 (    0.008057)| [  202.469172] ehci-pci 0000:00:1a.0: debug port 2
 1289.555969 (    0.008852)| [  202.478193] ehci-pci 0000:00:1a.0: irq 16, io mem 0x91f02000
 1289.576106 (    0.020137)| [  202.497161] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
 1289.580528 (    0.004422)| [  202.503665] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
 1289.588088 (    0.007560)| [  202.511250] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
 1289.596722 (    0.008634)| [  202.519332] usb usb1: Product: EHCI Host Controller
 1289.602816 (    0.006094)| [  202.524789] usb usb1: Manufacturer: Linux 4.9.0-rc5-clean+ ehci_hcd
 1289.610750 (    0.007934)| [  202.531791] usb usb1: SerialNumber: 0000:00:1a.0
 1289.615207 (    0.004457)| [  202.537187] hub 1-0:1.0: USB hub found
 1289.619563 (    0.004356)| [  202.541379] hub 1-0:1.0: 2 ports detected
 1289.631365 (    0.011802)| [  202.553623] ehci-pci 0000:00:1d.0: EHCI Host Controller
 1289.636034 (    0.004669)| [  202.559472] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2
 1289.648109 (    0.012075)| [  202.567746] ehci-pci 0000:00:1d.0: debug port 2
 1289.655521 (    0.007412)| [  202.576754] ehci-pci 0000:00:1d.0: irq 23, io mem 0x91f01000
 1289.678710 (    0.023189)| [  202.600516] [TTM] Zone  kernel: Available graphics memory: 8190816120 kiB
 1289.687386 (    0.008676)| [  202.608112] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
 1289.693549 (    0.006163)| [  202.615401] [TTM] Initializing pool allocator
 1289.701048 (    0.007499)| [  202.620280] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00
 1289.703779 (    0.002731)| [  202.626775] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
 1289.712273 (    0.008494)| [  202.634377] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
 1289.719192 (    0.006919)| [  202.642454] usb usb2: Product: EHCI Host Controller
 1289.729749 (    0.010557)| [  202.647929] usb usb2: Manufacturer: Linux 4.9.0-rc5-clean+ ehci_hcd
 1289.731640 (    0.001891)| [  202.654935] usb usb2: SerialNumber: 0000:00:1d.0
 1289.739345 (    0.007705)| [  202.660178] [TTM] Initializing DMA pool allocator
 1289.747680 (    0.008335)| [  202.667889] hub 2-0:1.0: USB hub found
 1289.752150 (    0.004470)| [  202.672101] hub 2-0:1.0: 2 ports detected
 1289.936330 (    0.184180)| [  202.702241] fbcon: mgadrmfb (fb0) is primary device
 1289.936372 (    0.000042)| [  202.703702] Console: switching to colour frame buffer device 128x48
 1289.936414 (    0.000042)| [  202.755809] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
 1289.936456 (    0.000042)| [  202.755873] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
 1289.938322 (    0.001866)| [  202.755909] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
 1289.938365 (    0.000043)| [  202.756179] ata1.00: ATA-9: INTEL SSDSC1NB800G4, D2010370, max UDMA/133
 1289.938410 (    0.000045)| [  202.756181] ata1.00: 1562824368 sectors, multi 1: LBA48 NCQ (depth 31/32)
 1289.939540 (    0.001130)| [  202.756191] ata2.00: ATA-9: INTEL SSDSC1NB800G4, D2010370, max UDMA/133
 1289.944592 (    0.005052)| [  202.756192] ata2.00: 1562824368 sectors, multi 1: LBA48 NCQ (depth 31/32)
 1290.047101 (    0.102509)| [  202.756447] ata2.00: configured for UDMA/133
 1290.047136 (    0.000035)| [  202.756478] ata1.00: configured for UDMA/133
 1290.047167 (    0.000031)| [  202.756819] ata3.00: ATA-10: ST1000NX0423, TN03, max UDMA/133
 1290.047205 (    0.000038)| [  202.756820] ata3.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
 1290.048237 (    0.001032)| [  202.757404] ata3.00: configured for UDMA/133
 1290.048267 (    0.000030)| [  202.772910] scsi 0:0:0:0: Direct-Access     ATA      INTEL SSDSC1NB80 0370 PQ: 0 ANSI: 5
 1290.048322 (    0.000055)| [  202.913143] mgag200 0000:09:00.0: fb0: mgadrmfb frame buffer device
 1290.048365 (    0.000043)| [^[[0;32m  OK  ^[[0m] Started udev Coldplug all Devices.
 1290.048399 (    0.000034)| [^[[0;32m  OK  ^[[0m] Reached target Remote File Systems (Pre).
 1290.054718 (    0.006319)| [^[[0;32m  OK  ^[[0m] Reached target Remote File Systems.
 1290.156872 (    0.102154)| [  202.988701] [drm] Initialized mgag200 1.0.0 20110418 for 0000:09:00.0 on minor 0
 1290.156935 (    0.000063)|          Starting Show Plymouth Boot Screen...
 1290.156966 (    0.000031)| [  203.041155] usb 1-1: new high-speed USB device number 2 using ehci-pci
 1290.157012 (    0.000046)| [  203.054858] scsi 0:0:0:0: Attached scsi generic sg0 type 0
 1290.157686 (    0.000674)| [  203.077862] scsi 1:0:0:0: Direct-Access     ATA      INTEL SSDSC1NB80 0370 PQ: 0 ANSI: 5
 1290.267759 (    0.110073)| [  203.152223] usb 2-1: new high-speed USB device number 2 using ehci-pci
 1290.376630 (    0.108871)| [  203.228211] usb 1-1: New USB device found, idVendor=8087, idProduct=0024
 1290.376703 (    0.000073)| [  203.235708] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
 1290.376754 (    0.000051)| [  203.244258] hub 1-1:1.0: USB hub found
 1290.376782 (    0.000028)| [  203.248566] hub 1-1:1.0: 6 ports detected
 1290.376811 (    0.000029)| [  203.264179] scsi 1:0:0:0: Attached scsi generic sg1 type 0
 1290.377603 (    0.000792)| [  203.283524] scsi 2:0:0:0: Direct-Access     ATA      ST1000NX0423     TN03 PQ: 0 ANSI: 5
 1290.511120 (    0.133517)| [  203.340788] usb 2-1: New USB device found, idVendor=8087, idProduct=0024
 1290.511188 (    0.000068)| [  203.348291] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
 1290.511239 (    0.000051)| [  203.357239] hub 2-1:1.0: USB hub found
 1290.511267 (    0.000028)| [  203.361562] hub 2-1:1.0: 8 ports detected
 1290.511297 (    0.000030)| [  203.423829] lpfc 0001:01:00.0: 0:3176 Port Name 0 Physical Link is functional
 1290.512732 (    0.001435)| [  203.432368] lpfc 0001:01:00.1: enabling device (0140 -> 0142)
 1290.749995 (    0.237263)| [  203.471810] scsi 2:0:0:0: Attached scsi generic sg2 type 0
 1290.750047 (    0.000052)| [  203.573173] usb 1-1.3: new high-speed USB device number 3 using ehci-pci
 1290.750094 (    0.000047)| [  203.662062] scsi host7: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 01 irq 79
 1290.750157 (    0.000063)| [  203.669165] usb 2-1.1: new high-speed USB device number 3 using ehci-pci
 1290.869455 (    0.119298)| [  203.705832] usb 1-1.3: New USB device found, idVendor=046b, idProduct=ff01
 1290.869508 (    0.000053)| [  203.713533] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
 1290.869558 (    0.000050)| [  203.713535] usb 1-1.3: Product: Generic Hub
 1290.869599 (    0.000041)| [  203.713536] usb 1-1.3: Manufacturer: American Megatrends Inc.
 1290.870550 (    0.000951)| [  203.713538] usb 1-1.3: SerialNumber: AAAABBBBCCCC4
 1290.870584 (    0.000034)| [  203.730355] hub 1-1.3:1.0: USB hub found
 1290.870613 (    0.000029)| [  203.730442] hub 1-1.3:1.0: 6 ports detected
 1290.887398 (    0.016785)| [  203.795068] usb 2-1.1: New USB device found, idVendor=05e3, idProduct=0719
 1290.887447 (    0.000049)| [  203.802755] usb 2-1.1: New USB device strings: Mfr=0, Product=1, SerialNumber=2
 1290.887497 (    0.000050)| [  203.810923] usb 2-1.1: Product: USB Storage
 1290.988702 (    0.101205)| [  203.815610] usb 2-1.1: SerialNumber: 000000000033
 1291.602392 (    0.613690)| [  204.149162] usb 1-1.3.1: new high-speed USB device number 4 using ehci-pci
 1291.602455 (    0.000063)| [  204.158635] ata1.00: Enabling discard_zeroes_data
 1291.602489 (    0.000034)| [  204.158655] sd 0:0:0:0: [sda] 1562824368 512-byte logical blocks: (800 GB/745 GiB)
 1291.602541 (    0.000052)| [  204.158658] sd 0:0:0:0: [sda] 4096-byte physical blocks
 1291.604056 (    0.001515)| [  204.158788] sd 0:0:0:0: [sda] Write Protect is off
 1291.604092 (    0.000036)| [  204.158834] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 1291.604155 (    0.000063)| [  204.159164] ata1.00: Enabling discard_zeroes_data
 1291.604188 (    0.000033)| [  204.160030] ata2.00: Enabling discard_zeroes_data
 1291.604222 (    0.000034)| [  204.160041] sd 1:0:0:0: [sdb] 1562824368 512-byte logical blocks: (800 GB/745 GiB)
 1291.604274 (    0.000052)| [  204.160044] sd 1:0:0:0: [sdb] 4096-byte physical blocks
 1291.604311 (    0.000037)| [  204.160107] sd 1:0:0:0: [sdb] Write Protect is off
 1291.604345 (    0.000034)| [  204.160137] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 1291.604406 (    0.000061)| [  204.160267] ata2.00: Enabling discard_zeroes_data
 1291.604440 (    0.000034)| [  204.160692] sd 2:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
 1291.604492 (    0.000052)| [  204.160738] sd 2:0:0:0: [sdc] Write Protect is off
 1291.604525 (    0.000033)| [  204.160771] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 1291.604586 (    0.000061)| [  204.161165]  sda: sda1 sda2 sda3 sda4
 1291.604612 (    0.000026)| [  204.162247]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5
 1291.604641 (    0.000029)| [  204.167274] ata1.00: Enabling discard_zeroes_data
 1291.604675 (    0.000034)| [  204.167576] sd 0:0:0:0: [sda] Attached SCSI disk
 1291.604709 (    0.000034)| [  204.169000] ata2.00: Enabling discard_zeroes_data
 1291.604741 (    0.000032)| [  204.169186] sd 1:0:0:0: [sdb] Attached SCSI disk
 1291.604775 (    0.000034)| [  204.207472]  sdc: sdc1
 1291.604793 (    0.000018)| [  204.210513] sd 2:0:0:0: [sdc] Attached SCSI disk
 1291.604825 (    0.000032)| [  204.232961] usb-storage 2-1.1:1.0: USB Mass Storage device detected
 1291.604869 (    0.000044)| [  204.233590] scsi host8: usb-storage 2-1.1:1.0
 1291.604899 (    0.000030)| [  204.234433] usbcore: registered new interface driver usb-storage
 1291.604942 (    0.000043)| [  204.309565] usbcore: registered new interface driver uas
 1291.604978 (    0.000036)| [  204.351879] PM: Starting manual resume from disk
 1291.605010 (    0.000032)| [  204.357951] usb 1-1.3.1: New USB device found, idVendor=046b, idProduct=ff10
 1291.605059 (    0.000049)| [  204.365843] usb 1-1.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
 1291.605110 (    0.000051)| [  204.365848] usb 1-1.3.1: Product: Virtual Keyboard and Mouse
 1291.605150 (    0.000040)| [  204.365850] usb 1-1.3.1\0: Manufacturer: [  204.479448] SGI XFS with ACLs, security attributes, realtime, no debug enabled
 1291.839239 (    0.234089)| [  204.549464] lpfc 0001:01:00.1: 1:3176 Port Name 1 Physical Link is functional
 1291.839298 (    0.000059)| [  204.557594] lpfc 0002:03:00.0: enabling device (0140 -> 0142)
 1291.839339 (    0.000041)| [  204.581171] usb 1-1.3.2: new high-speed USB device number 5 using ehci-pci
 1291.839385 (    0.000046)| [  204.680533] XFS (sda2): Mounting V5 Filesystem
 1291.841053 (    0.001668)| [  204.705538] XFS (sda2): Ending clean mount
 1291.841083 (    0.000030)| [  204.743240] scsi host9: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 00 irq 37
 1291.961752 (    0.120669)| [  204.798079] usb 1-1.3.2: New USB device found, idVendor=046b, idProduct=ff20
 1291.961811 (    0.000059)| [  204.805958] usb 1-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
 1291.961863 (    0.000052)| [  204.814326] usb 1-1.3.2: Product: Virtual Cdrom Device
 1291.961916 (    0.000053)| [  204.820071] usb 1-1.3.2: Manufacturer: American Megatrends Inc.
 1291.962926 (    0.001010)| [  204.826689] usb 1-1.3.2: SerialNumber: AAAABBBBCCCC1
 1291.962961 (    0.000035)| [  204.848994] usb-storage 1-1.3.2:1.0: USB Mass Storage device detected
 1291.963005 (    0.000044)| [  204.856782] scsi host10: usb-storage 1-1.3.2:1.0
 1291.963037 (    0.000032)| [  204.885207] usbcore: registered new interface driver usbhid
 1292.064221 (    0.101184)| [  204.891433] usbhid: USB HID core driver
 1292.064254 (    0.000033)| [  204.918743] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3.1/1-1.3.1:1.0/0003:046B:FF10.0001/input/input0
 1292.170667 (    0.106413)| [  204.997281] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.0-1.3.1/input0
 1292.170777 (    0.000110)| [  205.014664] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3.1/1-1.3.1:1.1/0003:046B:FF10.0002/input/input1
 1292.171334 (    0.000557)| [  205.032224] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.0-1.3.1/input1
 1292.372894 (    0.201560)| [  205.278262] scsi 8:0:0:0: CD-ROM            TEAC     DV-W28SS-B       1.2A PQ: 0 ANSI: 0
 1292.372957 (    0.000063)| [  205.288677] scsi 8:0:0:0: Attached scsi generic sg3 type 5
 1292.675510 (    0.302553)| [  205.581470] lpfc 0002:03:00.0: 2:3176 Port Name 0 Physical Link is functional
 1292.675569 (    0.000059)| [  205.589571] lpfc 0002:03:00.1: enabling device (0140 -> 0142)
 1292.884106 (    0.208537)| [  205.768167] scsi host11: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 01 irq 88
 1292.991759 (    0.107653)| [  205.912379] scsi 10:0:0:0: CD-ROM            AMI      Virtual CDROM0   1.00 PQ: 0 ANSI: 0 CCS
 1293.092969 (    0.101210)| [  205.923259] scsi 10:0:0:0: Attached scsi generic sg4 type 5
 1293.704598 (    0.611629)| [  206.585473] lpfc 0002:03:00.1: 3:3176 Port Name 1 Physical Link is functional
 1293.704672 (    0.000074)| [  206.593636] lpfc 0004:01:00.0: enabling device (0140 -> 0142)
 1293.908218 (    0.203546)| [  206.778021] scsi host12: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 00 irq 93
 1294.753435 (    0.845217)| [  207.601517] lpfc 0004:01:00.0: 4:3176 Port Name 0 Physical Link is functional
 1294.753513 (    0.000078)| [  207.609645] lpfc 0004:01:00.1: enabling device (0140 -> 0142)
 1294.921075 (    0.167562)| [  207.792619] scsi host13: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 01 irq 98
 1295.140289 (    0.219214)| [^[[0;32m  OK  ^[[0m] Started Show Plymouth Boot Screen.
 1295.140338 (    0.000049)| [^[[0;32m  OK  ^[[0m] Started Forward Password Requests to Plymouth Directory Watch.
 1295.140389 (    0.000051)| [^[[0;32m  OK  ^[[0m] Reached target Paths.
 1295.140418 (    0.000029)| [^[[0;32m  OK  ^[[0m] Found device INTEL_SSDSC1NB80 primary.
 1295.140455 (    0.000037)| [^[[0;32m  OK  ^[[0m] Found device INTEL_SSDSC1NB80 primary.
 1295.141238 (    0.000783)|          Starting Resume from hibernation using device /dev/sda3...
 1295.141281 (    0.000043)| [^[[0;32m  OK  ^[[0m] Started Resume from hibernation using device /dev/sda3.
 1295.141328 (    0.000047)| [^[[0;32m  OK  ^[[0m] Reached target Local File Systems (Pre).
 1295.141367 (    0.000039)|          Starting File System Check on /dev/...d-2a57-447a-bf73-63cdf13310dd...
 1295.143867 (    0.002500)| [^[[0;32m  OK  ^[[0m] Reached target Local File Systems.
 1295.247192 (    0.103325)| [^[[0;32m  OK  ^[[0m] Reached target System Initialization.
 1295.247237 (    0.000045)| [^[[0;32m  OK  ^[[0m] Reached target Basic System.
 1295.247270 (    0.000033)| [^[[0;32m  OK  ^[[0m] Started File System Check on /dev/d...72d-2a57-447a-bf73-63cdf13310dd.
 1295.247325 (    0.000055)|          Mounting /sysroot...
 1295.247853 (    0.000528)| [^[[0;32m  OK  ^[[0m] Mounted /sysroot.
 1295.247879 (    0.000026)| [^[[0;32m  OK  ^[[0m] Reached target Initrd Root File System.
 1295.247917 (    0.000038)|          Starting Reload Configuration from the Real Root...
 1295.247955 (    0.000038)| [^[[0;32m  OK  ^[[0m] Started Reload Configuration from the Real Root.
 1295.249612 (    0.001657)| [^[[0;32m  OK  ^[[0m] Reached target Initrd File Systems.
 1295.249648 (    0.000036)| [^[[0;32m  OK  ^[[0m] Reached target Initrd Default Target.
 1295.249684 (    0.000036)|          Starting Cleaning Up and Shutting Down Daemons...
 1295.249720 (    0.000036)| [^[[0;32m  OK  ^[[0m] Stopped target Timers.
 1295.249748 (    0.000028)| [^[[0;32m  OK  ^[[0m] Stopped target Remote File Systems.
 1295.249782 (    0.000034)| [^[[0;32m  OK  ^[[0m] Stopped target Remote File Systems (Pre).
 1295.249820 (    0.000038)| [^[[0;32m  OK  ^[[0m] Stopped dracut cmdline hook.
 1295.249852 (    0.000032)|          Starting Plymouth switch root service...
 1295.249883 (    0.000031)| [^[[0;32m  OK  ^[[0m] Stopped Cleaning Up and Shutting Down Daemons.
 1295.249924 (    0.000041)| [^[[0;32m  OK  ^[[0m] Stopped target Initrd Default Target.
 1295.351331 (    0.101407)| [^[[0;32m  OK  ^[[0m] Stopped target Basic System.
 1295.351364 (    0.000033)| [^[[0;32m  OK  ^[[0m] Stopped target Paths.
 1295.351391 (    0.000027)| [^[[0;32m  OK  ^[[0m] Stopped target System Initialization.
 1295.351427 (    0.000036)| [^[[0;32m  OK  ^[[0m] Stopped udev Coldplug all Devices.
 1295.351461 (    0.000034)| [^[[0;32m  OK  ^[[0m] Stopped dracut pre-trigger hook.
 1295.352454 (    0.000993)|          Stopping udev Kernel Device Manager...
 1295.352485 (    0.000031)| [^[[0;32m  OK  ^[[0m] Stopped target Swap.
 1295.352512 (    0.000027)| [^[[0;32m  OK  ^[[0m] Stopped target Local File Systems.
 1295.352546 (    0.000034)| [^[[0;32m  OK  ^[[0m] Stopped target Local File Systems (Pre).
 1295.352584 (    0.000038)| [^[[0;32m  OK  ^[[0m] Stopped Load Kernel Modules.
 1295.352615 (    0.000031)| [^[[0;32m  OK  ^[[0m] Stopped target Slices.
 1295.352643 (    0.000028)| [^[[0;32m  OK  ^[[0m] Stopped target Sockets.
 1295.352671 (    0.000028)| [^[[0;32m  OK  ^[[0m] Started Plymouth switch root service.
 1295.692401 (    0.339730)| [  208.613524] lpfc 0004:01:00.1: 5:3176 Port Name 1 Physical Link is functional
 1295.699960 (    0.007559)| [  208.621705] lpfc 0005:03:00.0: enabling device (0140 -> 0142)
 1295.956219 (    0.256259)| [  208.805430] scsi host14: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 00 irq 48
 1296.782770 (    0.826551)| [  209.633526] lpfc 0005:03:00.0: 6:3176 Port Name 0 Physical Link is functional
 1296.782843 (    0.000073)| [  209.641646] lpfc 0005:03:00.1: enabling device (0140 -> 0142)
 1296.900596 (    0.117753)| [  209.824228] scsi host15: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 01 irq 107
 1297.798113 (    0.897517)| [  210.645549] lpfc 0005:03:00.1: 7:3176 Port Name 1 Physical Link is functional
 1297.798183 (    0.000070)| [  210.653742] lpfc 0007:01:00.0: enabling device (0140 -> 0142)
 1297.916683 (    0.118500)| [  210.837634] scsi host16: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 00 irq 112
 1298.740856 (    0.824173)| [  211.661548] lpfc 0007:01:00.0: 8:3176 Port Name 0 Physical Link is functional
 1298.748323 (    0.007467)| [  211.669682] lpfc 0007:01:00.1: enabling device (0140 -> 0142)
 1298.924601 (    0.176278)| [  211.848327] scsi host17: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 01 irq 117
 1299.748792 (    0.824191)| [  212.669551] lpfc 0007:01:00.1: 9:3176 Port Name 1 Physical Link is functional
 1299.756608 (    0.007816)| [  212.677720] lpfc 0008:03:00.0: enabling device (0140 -> 0142)
 1299.937792 (    0.181184)| [  212.860410] scsi host18: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 00 irq 59
 1300.761487 (    0.823695)| [  213.685558] lpfc 0008:03:00.0: 10:3176 Port Name 0 Physical Link is functional
 1300.768983 (    0.007496)| [  213.693796] lpfc 0008:03:00.1: enabling device (0140 -> 0142)
 1300.952831 (    0.183848)| [  213.876181] scsi host19: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 01 irq 126
 1301.777360 (    0.824529)| [  214.701577] lpfc 0008:03:00.1: 11:3176 Port Name 1 Physical Link is functional
 1301.785240 (    0.007880)| [  214.709884] lpfc 000a:01:00.0: enabling device (0140 -> 0142)
 1301.969654 (    0.184414)| [  214.893763] scsi host20: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 00 irq 131
 1302.795083 (    0.825429)| [  215.717584] lpfc 000a:01:00.0: 12:3176 Port Name 0 Physical Link is functional
 1302.801225 (    0.006142)| [  215.725831] lpfc 000a:01:00.1: enabling device (0140 -> 0142)
 1302.985495 (    0.184270)| [  215.908335] scsi host21: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 01 device 01 irq 136
 1303.806872 (    0.821377)| [  216.729587] lpfc 000a:01:00.1: 13:3176 Port Name 1 Physical Link is functional
 1303.816194 (    0.009322)| [  216.737860] lpfc 000b:03:00.0: enabling device (0140 -> 0142)
 1303.999495 (    0.183301)| [  216.920488] scsi host22: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 00 irq 70
 1304.801387 (    0.801892)| [  217.717614] lpfc 000b:03:00.0: 14:3176 Port Name 0 Physical Link is functional
 1304.878233 (    0.076846)| [  217.725864] lpfc 000b:03:00.1: enabling device (0140 -> 0142)
 1304.986324 (    0.108091)| [  217.908302] scsi host23: Emulex LPe16000 8Gb PCIe Fibre Channel Adapter on PCI bus 03 device 01 irq 145
 1305.784659 (    0.798335)| [  218.705612] lpfc 000b:03:00.1: 15:3176 Port Name 1 Physical Link is functional
 1305.903059 (    0.118400)| [^[[0;32m  OK  ^[[0m] Stopped udev Kernel Device Manager.
 1305.905440 (    0.002381)| [^[[0;32m  OK  ^[[0m] Stopped Create Static Device Nodes in /dev.
 1305.909713 (    0.004273)| [^[[0;32m  OK  ^[[0m] Stopped Create list of required sta...ce nodes for the current kernel.
 1305.921923 (    0.012210)| [^[[0;32m  OK  ^[[0m] Closed udev Control Socket.
 1305.926449 (    0.004526)| [^[[0;32m  OK  ^[[0m] Closed udev Kernel Socket.
 1305.930374 (    0.003925)|          Starting Cleanup udevd DB...
 1305.934138 (    0.003764)| [^[[0;32m  OK  ^[[0m] Started Cleanup udevd DB.
 1305.938386 (    0.004248)| [^[[0;32m  OK  ^[[0m] Reached target Switch Root.
 1305.960174 (    0.021788)|          Starting Switch Root...
 1306.363432 (    0.403258)| [  219.284868] systemd-journald[9461]: Received SIGTERM from PID 1 (systemd).
 1306.389897 (    0.026465)| [  219.308970] systemd: 28 output lines suppressed due to ratelimiting
 1306.474658 (    0.084761)| 
 1306.474666 (    0.000008)| Welcome to ^[[0;32mSUSE Linux Enterprise Server 12 SP2^[[0m!
 1306.479774 (    0.005108)| 
 1306.617815 (    0.138041)| [  219.540331] systemd-sysv-generator[10527]: Overwriting existing symlink /run/systemd/generator.late/inputattach.service with real service.
 1306.629822 (    0.012007)| [  219.554610] systemd-sysv-generator[10527]: Overwriting existing symlink /run/systemd/generator.late/snoop_filter_sfs.service with real service.
 1306.750236 (    0.120414)| [^[[0;32m  OK  ^[[0m] Stopped Switch Root.
 1306.753783 (    0.003547)| [^[[0;32m  OK  ^[[0m] Stopped Journal Service.
 1306.759532 (    0.005749)|          Starting Create list of required st... nodes for the current kernel...
 1306.766033 (    0.006501)| [^[[0;32m  OK  ^[[0m] Set up automount Arbitrary Executab...ats File System Automount Point.
 1306.773998 (    0.007965)|          Mounting Debug File System...
 1306.778520 (    0.004522)|          Starting Rule generator for /dev/root symlink...
 1306.786332 (    0.007812)| [^[[0;32m  OK  ^[[0m] Stopped target Switch Root.
 1306.790383 (    0.004051)|          Mounting POSIX Message Queue File System...
 1306.795666 (    0.005283)| [^[[0;32m  OK  ^[[0m] Listening on LVM2 metadata daemon socket.
 1306.802020 (    0.006354)|          Starting LSB: Disables the "snoop f...S setting on Clovertown CPUs....
 1306.808491 (    0.006471)|          Starting Remount Root and Kernel File Systems...
 1306.819413 (    0.010922)| [^[[0;32m  OK  ^[[0m] Listening on Syslog Socket.
 1306.819444 (    0.000031)|          Starting Journal Service...
 1306.829958 (    0.010514)| [^[[0;32m  OK  ^[[0m] Created slice system-serial\x2dgetty.slice.
 1306.829999 (    0.000041)|          Mounting Huge Pages File System...
 1306.834771 (    0.004772)| [^[[0;32m  OK  ^[[0m] Reached target System Time Synchronized.
 1306.842723 (    0.007952)| [^[[0;32m  OK  ^[[0m] Created slice system-getty.slice.
 1306.846047 (    0.003324)| [^[[0;32m  OK  ^[[0m] Listening on udev Kernel Socket.
 1306.853921 (    0.007874)| [^[[0;32m  OK  ^[[0m] Created slice system-systemd\x2dfsck.slice.
 1306.857949 (    0.004028)| [^[[0;32m  OK  ^[[0m] Reached target Remote File Systems (Pre).
 1306.866150 (    0.008201)| [^[[0;32m  OK  ^[[0m] Listening on udev Control Socket.
 1306.870561 (    0.004411)| [^[[0;32m  OK  ^[[0m] Stopped target Initrd Root File System.
 1306.878397 (    0.007836)| [^[[0;32m  OK  ^[[0m] Reached target Remote File Systems.
 1306.882591 (    0.004194)| [^[[0;32m  OK  ^[[0m] Reached target Paths.
 1306.887253 (    0.004662)| [^[[0;32m  OK  ^[[0m] Reached target Encrypted Volumes.
 1306.890975 (    0.003722)| [^[[0;32m  OK  ^[[0m] Listening on /dev/initctl Compatibility Named Pipe.
 1306.898656 (    0.007681)| [^[[0;32m  OK  ^[[0m] Stopped target Initrd File Systems.
 1306.903324 (    0.004668)| [^[[0;32m  OK  ^[[0m] Created slice User and Session Slice.
 1306.911314 (    0.007990)| [^[[0;32m  OK  ^[[0m] Reached target Slices.
 1306.919508 (    0.008194)|          Starting Load Kernel Modules...
 1306.919534 (    0.000026)| [^[[0;32m  OK  ^[[0m] Mounted Debug File System.
 1306.921911 (    0.002377)| [^[[0;32m  OK  ^[[0m] Mounted POSIX Message Queue File System.
 1306.930643 (    0.008732)| [^[[0;32m  OK  ^[[0m] Mounted Huge Pages File System.
 1306.937224 (    0.006581)| [^[[0;32m  OK  ^[[0m] Started Journal Service.
 1306.938184 (    0.000960)| [^[[0;32m  OK  ^[[0m] Started Create list of required sta...ce nodes for the current kernel.
 1306.950173 (    0.011989)| [^[[0;32m  OK  ^[[0m] Started Rule generator for /dev/root symlink.
 1306.953908 (    0.003735)| [^[[0;32m  OK  ^[[0m] Started LSB: Disables the "snoop fi...IOS setting on Clovertown CPUs..
 1306.967573 (    0.013665)| [^[[0;32m  OK [  219.884732] perf: Dynamic interrupt throttling disabled, can hang your system!
 1306.974202 (    0.006629)|  ^[[0m] Started Remount Root and Kernel File Systems.
 1306.979659 (    0.005457)| [^[[0;32m  OK  ^[[0m] Started Load Kernel Modules.
 1306.981985 (    0.002326)| [^[[0;32m  OK  ^[[0m] Stopped Entropy Daemon based on the HAVEGE algorithm.
 1306.996562 (    0.014577)| [^[[0;32m  [  219.911813] systemd-journald[10560]: Received request to flush runtime journal from PID 1
 1307.006263 (    0.009701)| OK  ^[[0m] Started Entropy Daemon based on the HAVEGE algorithm.
 1307.006303 (    0.000040)|          Starting Apply Kernel Variables...
 1307.009883 (    0.003580)|          Starting udev Coldplug all Devices...
 1307.022638 (    0.012755)|          Starting Load/Save Random Seed...
 1307.022668 (    0.000030)|          Starting Create Static Device Nodes in /dev...
 1307.026939 (    0.004271)|          Starting Flush Journal to Persistent Storage...
 1307.034916 (    0.007977)| [^[[0;32m  OK  ^[[0m] Started Load/Save Random Seed.
 1307.037941 (    0.003025)| [^[[0;32m  OK  ^[[0m] Started Create Static Device Nodes in /dev.
 1307.041949 (    0.004008)| [^[[0;1;3mFAILED^[[0m] Failed to start Apply Kernel Variables.
 1307.050028 (    0.008079)| See 'systemctl status systemd-sysctl.service' for details.
 1307.054996 (    0.004968)| [^[[0;32m  OK  ^[[0m] Started Flush Journal to Persistent Storage.
 1307.062623 (    0.007627)|          Starting udev Kernel Device Manager...
 1307.066001 (    0.003378)| [^[[0;32m  OK  ^[[0m] Reached target Local File Systems (Pre).
 1307.073912 (    0.007911)| [^[[0;32m  OK  ^[[0m] Started udev Kernel Device Manager.
 1308.122214 (    1.048302)| [  221.045689] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input2
 1308.134881 (    0.012667)| [  221.055156] ACPI: Sleep Button [SLPB]
 1308.138109 (    0.003228)| [  221.062469] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
 1308.147479 (    0.009370)| [  221.070797] ACPI: Power Button [PWRF]
 1308.686435 (    0.538956)| [^[[0;32m  OK  ^[[0m] Started udev Coldplug all Devices.
 1308.817996 (    0.131561)| [  221.739287] dca service started, version 1.12.1
 1308.822668 (    0.004672)| [  221.742222] wmi: Mapper loaded
 1308.863818 (    0.041150)| [  221.786257] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
 1308.986687 (    0.122869)| [  221.908668] FUJITSU Extended Socket Network Device Driver - version 1.1 - Copyright (c) 2015 FUJITSU LIMITED
 1309.106493 (    0.119806)| [  222.030983] pps_core: LinuxPPS API ver. 1 registered
 1309.116115 (    0.009622)| [  222.036540] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
 1309.138221 (    0.022106)| [  222.062078] ioatdma: Intel(R) QuickData Technology Driver 4.00
 1309.146520 (    0.008299)| [  222.068707] ioatdma 0000:00:04.0: enabling device (0004 -> 0006)
 1309.172523 (    0.026003)| [  222.094287] ioatdma 0000:00:04.1: enabling device (0004 -> 0006)
 1309.194272 (    0.021749)| [  222.118157] ioatdma 0000:00:04.2: enabling device (0004 -> 0006)
 1309.206489 (    0.012217)| [  222.128961] PTP clock support registered
 1309.218833 (    0.012344)| [  222.140825] ioatdma 0000:00:04.3: enabling device (0004 -> 0006)
 1309.243028 (    0.024195)| [  222.163496] ioatdma 0000:00:04.4: enabling device (0004 -> 0006)
 1309.267474 (    0.024446)| [  222.189807] ioatdma 0000:00:04.5: enabling device (0004 -> 0006)
 1309.291043 (    0.023569)| [  222.214626] i801_smbus 0000:00:1f.3: enabling device (0140 -> 0143)
 1309.299620 (    0.008577)| [  222.223347] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
 1309.315026 (    0.015406)| [  222.235634] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
 1309.319024 (    0.003998)| [  222.243416] igb: Copyright (c) 2007-2014 Intel Corporation.
 1309.330967 (    0.011943)| [  222.254087] ioatdma 0000:00:04.6: enabling device (0004 -> 0006)
 1309.367652 (    0.036685)| [  222.287295] ioatdma 0000:00:04.7: enabling device (0004 -> 0006)
 1309.386628 (    0.018976)| [  222.309486] ioatdma 0001:00:04.0: enabling device (0004 -> 0006)
 1309.391208 (    0.004580)| [  222.313847] igb 0000:05:00.0: DCA enabled
 1309.397766 (    0.006558)| [  222.319455] pps pps0: new PPS source ptp0
 1309.403527 (    0.005761)| [  222.319459] igb 0000:05:00.0: added PHC on eth0
 1309.407312 (    0.003785)| [  222.319461] igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
 1309.415712 (    0.008400)| [  222.319464] igb 0000:05:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 08:00:69:17:90:55
 1309.423230 (    0.007518)| [  222.319552] igb 0000:05:00.0: eth0: PBA No: 000300-000
 1309.431319 (    0.008089)| [  222.319554] igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
 1309.438609 (    0.007290)| [  222.325684] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.4.0-k
 1309.446387 (    0.007778)| [  222.325686] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
 1309.450753 (    0.004366)| [  222.326043] ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
 1309.459602 (    0.008849)| [^[[0;32m  OK  ^[[0m] Found device INTEL_SSDSC1NB80 primary.
 1309.467438 (    0.007836)| [^[[0;32m  OK  ^[[0m] Found device INTEL_SSDSC1NB80 primary.
 1309.471448 (    0.004010)| [^[[0;32m  OK  ^[[0m] Found device INTEL_SSDSC1NB80 primary.
 1309.688909 (    0.217461)| [  222.404547] input: PC Speaker as /devices/platform/pcspkr/input/input4
 1309.688966 (    0.000057)| [  222.410420] ioatdma 0001:00:04.1: enabling device (0004 -> 0006)
 1309.689009 (    0.000043)|          Mounting /boot/efi...
 1309.689032 (    0.000023)| [  222.432496] ioatdma 0001:00:04.2: enabling device (0004 -> 0006)
 1309.689074 (    0.000042)|          Activating swap /dev/disk/by-uuid/0...c-291f-4928-bcf3-ce5e9930810b...
 1309.689123 (    0.000049)| [  222.459908] ioatdma 0001:00:04.3: enabling device (0004 -> 0006)
 1309.689178 (    0.000055)|          Starting File System Check on /dev/...d-84ed-4b95-a1d7-[  222.468264] Adding 8385532k swap on /dev/sda3.  Priority:-1 extents:1 across:8385532k SSFS
 1309.689269 (    0.000091)| 711feceb0530...
 1309.689282 (    0.000013)| [^[[0;32m  OK  ^[[0m] Activated swap /dev/disk/by-uuid/0587fd9c-291f-4928-bcf3-ce5e9930810b.
 1309.689337 (    0.000055)| [^[[0;32m  OK  ^[[0[  222.489804] ioatdma 0001:00:04.4: enabling device (0004 -> 0006)
 1309.689387 (    0.000050)| m] Found device /dev/ttyS0.
 1309.689407 (    0.000020)| [^[[0;32m  OK  ^[[0m] Started File System Check on /[  222.507795] ioatdma 0001:00:04.5: enabling device (0004 -> 0006)
 1309.689476 (    0.000069)| dev/d...bed-84ed-4b95-a1d7-711feceb0530.
 1309.689502 (    0.000026)|          Mounting /home...
 1309.689522 (    0.000020)| [  222.535835] ioatdma 0001:00:04.6: enabling device (0004 -> 0006)
 1309.689563 (    0.000041)| [^[[0;32m  OK  ^[[0m] Reached target Swap.
 1309.689600 (    0.000037)| [  222.560453] ioatdma 0001:00:04.7: enabling device (0004 -> 0006)
 1309.689642 (    0.000042)| [  222.581686] ioatdma 0002:00:04.0: enabling device (0004 -> 0006)
 1309.689683 (    0.000041)| [  222.583221] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1309.689740 (    0.000057)| [  222.583358] ixgbe 0000:02:00.0: PCI Express bandwidth of 32GT/s available
 1309.689786 (    0.000046)| [  222.583361] ixgbe 0000:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1309.696502 (    0.006716)| [  222.583444] ixgbe 0000:02:00.0: MAC: 2, PHY: 15, SFP+: 5, PBA No: FFFFFF-0FF
 1309.697565 (    0.001063)| [  222.583446] ixgbe 0000:02:00.0: 00:e0:ed:65:0c:22
 1309.704318 (    0.006753)| [  222.598800] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
 1309.714724 (    0.010406)| [  222.598885] ixgbe 0000:02:00.1: enabling device (0140 -> 0142)
 1309.739870 (    0.025146)| [  222.661240] ioatdma 0002:00:04.1: enabling device (0004 -> 0006)
 1309.748035 (    0.008165)| [  222.662102] XFS (sda4): Mounting V5 Filesystem
 1309.750691 (    0.002656)| [  222.674923] XFS (sda4): Ending clean mount
 1309.759242 (    0.008551)| [^[[0;32m  OK  ^[[0m] Mounted /home.
 1309.763483 (    0.004241)| [  222.685773] ioatdma 0002:00:04.2: enabling device (0004 -> 0006)
 1309.774419 (    0.010936)| [  222.696343] ioatdma 0002:00:04.3: enabling device (0004 -> 0006)
 1309.790594 (    0.016175)| [  222.714472] ioatdma 0002:00:04.4: enabling device (0004 -> 0006)
 1309.810438 (    0.019844)| [  222.732411] ioatdma 0002:00:04.5: enabling device (0004 -> 0006)
 1309.814484 (    0.004046)| [  222.734997] sr 8:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
 1309.827244 (    0.012760)| [  222.735000] cdrom: Uniform CD-ROM driver Revision: 3.20
 1309.842765 (    0.015521)| [  222.766447] ioatdma 0002:00:04.6: enabling device (0004 -> 0006)
 1309.867014 (    0.024249)| [^[[0;32m  OK  ^[[0m] Mounted /boot/efi.
 1309.875387 (    0.008373)| [  222.795175] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1309.888341 (    0.012954)| [  222.795304] ixgbe 0000:02:00.1: PCI Express bandwidth of 32GT/s available
 1309.891070 (    0.002729)| [  222.795308] ixgbe 0000:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1309.906897 (    0.015827)| [  222.795392] ixgbe 0000:02:00.1: MAC: 2, PHY: 15, SFP+: 6, PBA No: FFFFFF-0FF
 1309.906947 (    0.000050)| [  222.795395] ixgbe 0000:02:00.1: 00:e0:ed:65:0c:23
 1309.915181 (    0.008234)| [  222.800389] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
 1309.923606 (    0.008425)| [  222.800550] ixgbe 0003:02:00.0: enabling device (0140 -> 0142)
 1309.927477 (    0.003871)| [  222.801395] ioatdma 0002:00:04.7: enabling device (0004 -> 0006)
 1309.955788 (    0.028311)| [  222.832809] ioatdma 0003:00:04.0: enabling device (0004 -> 0006)
 1309.955831 (    0.000043)| [  222.857147] ioatdma 0003:00:04.1: enabling device (0004 -> 0006)
 1309.955873 (    0.000042)| [^[[0;32m  OK  ^[[0m] Reached target Local File Systems.
 1309.955917 (    0.000044)|          Starting Create Volatile Files and Directories...
 1309.959500 (    0.003583)|    [  222.881710] ioatdma 0003:00:04.2: enabling device (0004 -> 0006)
 1309.967747 (    0.008247)|       Starting Tell Plymouth To Write Out Runtime Data...
 1309.974470 (    0.006723)|          Starting Restore /run/initramfs on shutdown...
 1309.978716 (    0.004246)| [^[[0;32m  OK  ^[[0m] Started Shadow /etc/init.d/boot.localfs.
 1309.986981 (    0.008265)| [  222.906675] ioatdma 0003:00:04.3: enabling device (0004 -> 0006)
 1309.990779 (    0.003798)| [^[[0;32m  OK  ^[[0m] Started Create Volatile Files and Directories.
 1309.999482 (    0.008703)| [^[[0;32m  O[  222.922394] ioatdma 0003:00:04.4: enabling device (0004 -> 0006)
 1310.016631 (    0.017149)| K  ^[[0m] Started Restore /run/initramfs on shutdown.
 1310.022434 (    0.005803)| [^[[0;32m  OK  ^[[0m] Started Tell Plymouth To Write Out Runtime Data.
 1310.022478 (    0.000044)|          Starting Update UTMP about System Boot/Shutdown...
 1310.026688 (    0.004210)|          Starting LSB: AppArmor initialization...
 1310.030688 (    0.004000)| [  222.952806] ioatdma 0003:00:04.5: enabling device (0004 -> 0006)
 1310.039270 (    0.008582)| [^[[0;32m  OK  ^[[0m] Started Update UTMP about System Boot/Shutdown.
 1310.051840 (    0.012570)| [  222.974190] ioatdma 0003:00:04.6: enabling device (0004 -> 0006)
 1310.059307 (    0.007467)| [  222.976624] sr 10:0:0:0: [sr1] scsi-1 drive
 1310.062602 (    0.003295)| [  222.981087] ixgbe 0003:02:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1310.071214 (    0.008612)| [  222.981217] ixgbe 0003:02:00.0: PCI Express bandwidth of 32GT/s available
 1310.080245 (    0.009031)| [  222.981220] ixgbe 0003:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1310.086565 (    0.006320)| [  222.981304] ixgbe 0003:02:00.0: MAC: 2, PHY: 15, SFP+: 5, PBA No: FFFFFF-0FF
 1310.116164 (    0.029599)| [  222.981306] ixgbe 0003:02:00.0: 00:e0:ed:42:f3:ae
 1310.116200 (    0.000036)| [  222.985254] ixgbe 0003:02:00.0: Intel(R) 10 Gigabit Network Connection
 1310.116246 (    0.000046)| [  222.985398] ixgbe 0003:02:00.1: enabling device (0140 -> 0142)
 1310.131184 (    0.014938)| [  223.054928] ioatdma 0003:00:04.7: enabling device (0004 -> 0006)
 1310.144636 (    0.013452)| [  223.066844] AVX2 version of gcm_enc/dec engaged.
 1310.154492 (    0.009856)| [  223.072003] AES CTR mode by8 optimization enabled
 1310.170605 (    0.016113)| [  223.093252] ioatdma 0004:00:04.0: enabling device (0004 -> 0006)
 1310.203449 (    0.032844)| [  223.125537] ioatdma 0004:00:04.1: enabling device (0004 -> 0006)
 1310.230711 (    0.027262)| [  223.151937] ioatdma 0004:00:04.2: enabling device (0004 -> 0006)
 1310.242981 (    0.012270)| [  223.165801] ixgbe 0003:02:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1310.257128 (    0.014147)| [  223.168716] audit: type=1400 audit(1479413958.957:2): apparmor="STATUS" operation="profile_load" name="/{usr/,}bin/ping" pid=12661 comm="apparmor_parser"
 1310.267833 (    0.010705)| [  223.180874] ioatdma 0004:00:04.3: enabling device (0004 -> 0006)
 1310.277846 (    0.010013)| [  223.194829] ioatdma 0004:00:04.4: enabling device (0004 -> 0006)
 1310.283291 (    0.005445)| [  223.204121] ixgbe 0003:02:00.1: PCI Express bandwidth of 32GT/s available
 1310.290528 (    0.007237)| [  223.208035] ioatdma 0004:00:04.5: enabling device (0004 -> 0006)
 1310.294462 (    0.003934)| [  223.218427] ixgbe 0003:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1310.303133 (    0.008671)| [  223.221628] ioatdma 0004:00:04.6: enabling device (0004 -> 0006)
 1310.311209 (    0.008076)| [  223.233201] ixgbe 0003:02:00.1: MAC: 2, PHY: 15, SFP+: 6, PBA No: FFFFFF-0FF
 1310.318569 (    0.007360)| [  223.234194] ioatdma 0004:00:04.7: enabling device (0004 -> 0006)
 1310.326969 (    0.008400)| [  223.246757] ioatdma 0005:00:04.0: enabling device (0004 -> 0006)
 1310.330596 (    0.003627)| [  223.254510] ixgbe 0003:02:00.1: 00:e0:ed:42:f3:af
 1310.338612 (    0.008016)| [  223.263440] ixgbe 0003:02:00.1: Intel(R) 10 Gigabit Network Connection
 1310.349943 (    0.011331)| [  223.270471] audit: type=1400 audit(1479413959.061:3): apparmor="STATUS" operation="profile_load" name="/sbin/klogd" pid=12698 comm="apparmor_parser"
 1310.365566 (    0.015623)| [  223.275205] ioatdma 0005:00:04.1: enabling device (0004 -> 0006)
 1310.370592 (    0.005026)| [  223.292424] ixgbe 0006:02:00.0: enabling device (0140 -> 0142)
 1310.374552 (    0.003960)| [  223.295536] ioatdma 0005:00:04.2: enabling device (0004 -> 0006)
 1310.391135 (    0.016583)| [  223.314004] ioatdma 0005:00:04.3: enabling device (0004 -> 0006)
 1310.415136 (    0.024001)| [  223.338674] ioatdma 0005:00:04.4: enabling device (0004 -> 0006)
 1310.438789 (    0.023653)| [  223.359309] ioatdma 0005:00:04.5: enabling device (0004 -> 0006)
 1310.459132 (    0.020343)| [  223.381740] ioatdma 0005:00:04.6: enabling device (0004 -> 0006)
 1310.479067 (    0.019935)| [  223.401987] ioatdma 0005:00:04.7: enabling device (0004 -> 0006)
 1310.498681 (    0.019614)| [  223.422260] ioatdma 0006:00:04.0: enabling device (0004 -> 0006)
 1310.508140 (    0.009459)| [  223.427189] audit: type=1400 audit(1479413959.217:4): apparmor="STATUS" operation="profile_load" name="/sbin/syslog-ng" pid=12703 comm="apparmor_parser"
 1310.583025 (    0.074885)| [  223.505655] ixgbe 0006:02:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1310.590675 (    0.007650)| [  223.515203] ixgbe 0006:02:00.0: PCI Express bandwidth of 32GT/s available
 1310.598697 (    0.008022)| [  223.522793] ixgbe 0006:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1310.606866 (    0.008169)| [  223.530852] ixgbe 0006:02:00.0: MAC: 2, PHY: 15, SFP+: 5, PBA No: FFFFFF-0FF
 1310.614666 (    0.007800)| [  223.538729] ixgbe 0006:02:00.0: 00:e0:ed:46:3e:a6
 1310.627357 (    0.012691)| [  223.548753] ixgbe 0006:02:00.0: Intel(R) 10 Gigabit Network Connection
 1310.639144 (    0.011787)| [  223.562172] ioatdma 0006:00:04.1: enabling device (0004 -> 0006)
 1310.647237 (    0.008093)| [  223.562188] ixgbe 0006:02:00.1: enabling device (0140 -> 0142)
 1310.662567 (    0.015330)| [  223.584806] audit: type=1400 audit(1479413959.373:5): apparmor="STATUS" operation="profile_load" name="/sbin/syslogd" pid=12711 comm="apparmor_parser"
 1310.679934 (    0.017367)| [  223.591586] ioatdma 0006:00:04.2: enabling device (0004 -> 0006)
 1310.685325 (    0.005391)| [  223.605014] ioatdma 0006:00:04.3: enabling device (0004 -> 0006)
 1310.699146 (    0.013821)| [  223.621647] ioatdma 0006:00:04.4: enabling device (0004 -> 0006)
 1310.727202 (    0.028056)| [  223.649603] ioatdma 0006:00:04.5: enabling device (0004 -> 0006)
 1310.754752 (    0.027550)| [  223.677718] ioatdma 0006:00:04.6: enabling device (0004 -> 0006)
 1310.783023 (    0.028271)| [  223.707034] ioatdma 0006:00:04.7: enabling device (0004 -> 0006)
 1310.814772 (    0.031749)| [  223.736157] ioatdma 0007:00:04.0: enabling device (0004 -> 0006)
 1310.819112 (    0.004340)| [  223.738582] ixgbe 0006:02:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1310.831391 (    0.012279)| [  223.738712] ixgbe 0006:02:00.1: PCI Express bandwidth of 32GT/s available
 1310.836359 (    0.004968)| [  223.738714] ixgbe 0006:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1310.847063 (    0.010704)| [  223.738797] ixgbe 0006:02:00.1: MAC: 2, PHY: 15, SFP+: 6, PBA No: FFFFFF-0FF
 1310.854838 (    0.007775)| [  223.738799] ixgbe 0006:02:00.1: 00:e0:ed:46:3e:a7
 1310.858689 (    0.003851)| [  223.743155] ixgbe 0006:02:00.1: Intel(R) 10 Gigabit Network Connection
 1310.867220 (    0.008531)| [  223.743307] ixgbe 0009:02:00.0: enabling device (0140 -> 0142)
 1310.895878 (    0.028658)| [  223.817299] ioatdma 0007:00:04.1: enabling device (0004 -> 0006)
 1310.920479 (    0.024601)| [  223.840554] ioatdma 0007:00:04.2: enabling device (0004 -> 0006)
 1310.939174 (    0.018695)| [  223.862222] ioatdma 0007:00:04.3: enabling device (0004 -> 0006)
 1310.958918 (    0.019744)| [  223.881713] ioatdma 0007:00:04.4: enabling device (0004 -> 0006)
 1310.979229 (    0.020311)| [  223.901092] ioatdma 0007:00:04.5: enabling device (0004 -> 0006)
 1310.998816 (    0.019587)| [  223.922778] ioatdma 0007:00:04.6: enabling device (0004 -> 0006)
 1311.006693 (    0.007877)| [  223.926524] ixgbe 0009:02:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1311.015482 (    0.008789)| [  223.926657] ixgbe 0009:02:00.0: PCI Express bandwidth of 32GT/s available
 1311.023462 (    0.007980)| [  223.926660] ixgbe 0009:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1311.031131 (    0.007669)| [  223.926746] ixgbe 0009:02:00.0: MAC: 2, PHY: 15, SFP+: 5, PBA No: FFFFFF-0FF
 1311.040116 (    0.008985)| [  223.926749] ixgbe 0009:02:00.0: 00:e0:ed:43:54:44
 1311.046639 (    0.006523)| [  223.931273] ixgbe 0009:02:00.0: Intel(R) 10 Gigabit Network Connection
 1311.050703 (    0.004064)| [  223.931452] ixgbe 0009:02:00.1: enabling device (0140 -> 0142)
 1311.070954 (    0.020251)| [  223.994850] ioatdma 0007:00:04.7: enabling device (0004 -> 0006)
 1311.090660 (    0.019706)| [  224.015027] ioatdma 0008:00:04.0: enabling device (0004 -> 0006)
 1311.127559 (    0.036899)| [  224.048507] ioatdma 0008:00:04.1: enabling device (0004 -> 0006)
 1311.150807 (    0.023248)| [  224.074067] ioatdma 0008:00:04.2: enabling device (0004 -> 0006)
 1311.187240 (    0.036433)| [  224.097414] ioatdma 0008:00:04.3: enabling device (0004 -> 0006)
 1311.187287 (    0.000047)| [  224.108222] ixgbe 0009:02:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
 1311.194697 (    0.007410)| [  224.117785] ixgbe 0009:02:00.1: PCI Express bandwidth of 32GT/s available
 1311.202901 (    0.008204)| [  224.119155] ioatdma 0008:00:04.4: enabling device (0004 -> 0006)
 1311.211288 (    0.008387)| [  224.122007] audit: type=1400 audit(1479413959.913:6): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2" pid=12716 comm="apparmor_parser"
 1311.227234 (    0.015946)| [  224.122197] audit: type=1400 audit(1479413959.913:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//DEFAULT_URI" pid=12716 comm="apparmor_parser"
 1311.372738 (    0.145504)| [  224.122324] audit: type=1400 audit(1479413959.913:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//HANDLING_UNTRUSTED_INPUT" pid=12716 comm="apparmor_parser"
 1311.372859 (    0.000121)| [  224.122571] audit: type=1400 audit(1479413959.913:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//phpsysinfo" pid=12716 comm="apparmor_parser"
 1311.372968 (    0.000109)| [  224.132413] ioatdma 0008:00:04.5: enabling device (0004 -> 0006)
 1311.373009 (    0.000041)| [  224.145760] ioatdma 0008:00:04.6: enabling device (0004 -> 0006)
 1311.373051 (    0.000042)| [  224.158225] ioatdma 0008:00:04.7: enabling device (0004 -> 0006)
 1311.373092 (    0.000041)| [  224.170857] ioatdma 0009:00:04.0: enabling device (0004 -> 0006)
 1311.373133 (    0.000041)| [  224.198475] ioatdma 0009:00:04.1: enabling device (0004 -> 0006)
 1311.373174 (    0.000041)| [  224.208006] audit: type=1400 audit(1479413959.997:10): apparmor="STATUS" operation="profile_load" name="/usr/lib/dovecot/anvil" pid=12721 comm="apparmor_parser"
 1311.373271 (    0.000097)| [  224.220315] ioatdma 0009:00:04.2: enabling device (0004 -> 0006)
 1311.373312 (    0.000041)| [  224.239462] ioatdma 0009:00:04.3: enabling device (0004 -> 0006)
 1311.373353 (    0.000041)| [  224.262191] ioatdma 0009:00:04.4: enabling device (0004 -> 0006)
 1311.373394 (    0.000041)| [  224.275830] ixgbe 0009:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
 1311.373443 (    0.000049)| [  224.283674] ioatdma 0009:00:04.5: enabling device (0004 -> 0006)
 1311.373484 (    0.000041)| [  224.290611] ixgbe 0009:02:00.1: MAC: 2, PHY: 15, SFP+: 6, PBA No: FFFFFF-0FF
 1311.410978 (    0.037494)| [  224.298487] ixgbe 0009:02:00.1: 00:e0:ed:43:54:45
 1311.411014 (    0.000036)| [  224.304949] ioatdma 0009:00:04.6: enabling device (0004 -> 0006)
 1311.411055 (    0.000041)| [  224.308250] ixgbe 0009:02:00.1: Intel(R) 10 Gigabit Network Connection
 1311.415905 (    0.004850)| [  224.336220] ioatdma 0009:00:04.7: enabling device (0004 -> 0006)
 1311.439466 (    0.023561)| [  224.360190] ioatdma 000a:00:04.0: enabling device (0004 -> 0006)
 1311.471051 (    0.031585)| [  224.391523] audit: type=1400 audit(1479413960.181:11): apparmor="STATUS" operation="profile_load" name="/usr/lib/dovecot/auth" pid=12725 comm="apparmor_parser"
 1311.483167 (    0.012116)| [  224.394807] ioatdma 000a:00:04.1: enabling device (0004 -> 0006)
 1311.490946 (    0.007779)| [  224.414305] ioatdma 000a:00:04.2: enabling device (0004 -> 0006)
 1311.514768 (    0.023822)| [  224.437758] ioatdma 000a:00:04.3: enabling device (0004 -> 0006)
 1311.536652 (    0.021884)| [  224.458876] ioatdma 000a:00:04.4: enabling device (0004 -> 0006)
 1311.554744 (    0.018092)| [  224.478931] ioatdma 000a:00:04.5: enabling device (0004 -> 0006)
 1311.574889 (    0.020145)| [  224.499128] ioatdma 000a:00:04.6: enabling device (0004 -> 0006)
 1311.595241 (    0.020352)| [  224.518443] ioatdma 000a:00:04.7: enabling device (0004 -> 0006)
 1311.615123 (    0.019882)| [  224.537803] ioatdma 000b:00:04.0: enabling device (0004 -> 0006)
 1311.651001 (    0.035878)| [  224.572139] ioatdma 000b:00:04.1: enabling device (0004 -> 0006)
 1311.674785 (    0.023784)| [  224.596733] ioatdma 000b:00:04.2: enabling device (0004 -> 0006)
 1311.695201 (    0.020416)| [  224.619155] ioatdma 000b:00:04.3: enabling device (0004 -> 0006)
 1311.724736 (    0.029535)| [  224.641273] ioatdma 000b:00:04.4: enabling device (0004 -> 0006)
 1311.739361 (    0.014625)| [  224.661283] ioatdma 000b:00:04.5: enabling device (0004 -> 0006)
 1311.758989 (    0.019628)| [  224.680615] ioatdma 000b:00:04.6: enabling device (0004 -> 0006)
 1311.779254 (    0.020265)| [  224.699960] ioatdma 000b:00:04.7: enabling device (0004 -> 0006)
 1312.796630 (    1.017376)| [  225.719123] intel_rapl: Found RAPL domain package
 1312.802777 (    0.006147)| [  225.724419] intel_rapl: Found RAPL domain dram
 1312.806726 (    0.003949)| [  225.729424] intel_rapl: DRAM domain energy unit 15300pj
 1312.810899 (    0.004173)| [  225.735311] intel_rapl: Found RAPL domain package
 1312.819443 (    0.008544)| [  225.740590] intel_rapl: Found RAPL domain dram
 1312.823860 (    0.004417)| [  225.745553] intel_rapl: DRAM domain energy unit 15300pj
 1312.827646 (    0.003786)| [  225.751447] intel_rapl: Found RAPL domain package
 1312.835616 (    0.007970)| [  225.756723] intel_rapl: Found RAPL domain dram
 1312.839242 (    0.003626)| [  225.761687] intel_rapl: DRAM domain energy unit 15300pj
 1312.844527 (    0.005285)| [  225.767578] intel_rapl: Found RAPL domain package
 1312.850854 (    0.006327)| [  225.772855] intel_rapl: Found RAPL domain dram
 1312.855138 (    0.004284)| [  225.777820] intel_rapl: DRAM domain energy unit 15300pj
 1312.858950 (    0.003812)| [  225.783714] intel_rapl: Found RAPL domain package
 1312.867293 (    0.008343)| [  225.788996] intel_rapl: Found RAPL domain dram
 1312.871016 (    0.003723)| [  225.793958] intel_rapl: DRAM domain energy unit 15300pj
 1312.875122 (    0.004106)| [  225.799857] intel_rapl: Found RAPL domain package
 1312.881069 (    0.005947)| [  225.805139] intel_rapl: Found RAPL domain dram
 1312.887128 (    0.006059)| [  225.810106] intel_rapl: DRAM domain energy unit 15300pj
 1312.894977 (    0.007849)| [  225.816011] intel_rapl: Found RAPL domain package
 1312.899481 (    0.004504)| [  225.821294] intel_rapl: Found RAPL domain dram
 1313.046102 (    0.146621)| [  225.826259] intel_rapl: DRAM domain energy unit 15300pj
 1313.046146 (    0.000044)| [  225.832160] intel_rapl: Found RAPL domain package
 1313.046180 (    0.000034)| [  225.837449] intel_rapl: Found RAPL domain dram
 1313.046213 (    0.000033)| [  225.842414] intel_rapl: DRAM domain energy unit 15300pj
 1313.046250 (    0.000037)| [  225.848322] intel_rapl: Found RAPL domain package
 1313.046284 (    0.000034)| [  225.853625] intel_rapl: Found RAPL domain dram
 1313.046336 (    0.000052)| [  225.858592] intel_rapl: DRAM domain energy unit 15300pj
 1313.046373 (    0.000037)| [  225.864493] intel_rapl: Found RAPL domain package
 1313.046407 (    0.000034)| [  225.869797] intel_rapl: Found RAPL domain dram
 1313.046439 (    0.000032)| [  225.874764] intel_rapl: DRAM domain energy unit 15300pj
 1313.046475 (    0.000036)| [  225.880670] intel_rapl: Found RAPL domain package
 1313.046509 (    0.000034)| [  225.885959] intel_rapl: Found RAPL domain dram
 1313.046540 (    0.000031)| [  225.890927] intel_rapl: DRAM domain energy unit 15300pj
 1313.046577 (    0.000037)| [  225.896832] intel_rapl: Found RAPL domain package
 1313.046610 (    0.000033)| [  225.902127] intel_rapl: Found RAPL domain dram
 1313.046642 (    0.000032)| [  225.907094] intel_rapl: DRAM domain energy unit 15300pj
 1313.046679 (    0.000037)| [  225.912996] intel_rapl: Found RAPL domain package
 1313.046712 (    0.000033)| [  225.918283] intel_rapl: Found RAPL domain dram
 1313.046744 (    0.000032)| [  225.923242] intel_rapl: DRAM domain energy unit 15300pj
 1313.046782 (    0.000038)| [  225.929147] intel_rapl: Found RAPL domain package
 1313.046816 (    0.000034)| [  225.934434] intel_rapl: Found RAPL domain dram
 1313.046848 (    0.000032)| [  225.939399] intel_rapl: DRAM domain energy unit 15300pj
 1313.046884 (    0.000036)| [  225.945303] intel_rapl: Found RAPL domain package
 1313.046918 (    0.000034)| [  225.950583] intel_rapl: Found RAPL domain dram
 1313.046949 (    0.000031)| [  225.955549] intel_rapl: DRAM domain energy unit 15300pj
 1313.046986 (    0.000037)| [  225.961451] intel_rapl: Found RAPL domain package
 1313.086458 (    0.039472)| [  225.966739] intel_rapl: Found RAPL domain dram
 1313.086493 (    0.000035)| [  225.971705] intel_rapl: DRAM domain energy unit 15300pj
 1313.086530 (    0.000037)| [  225.977606] intel_rapl: Found RAPL domain package
 1313.086563 (    0.000033)| [  225.982887] intel_rapl: Found RAPL domain dram
 1313.086595 (    0.000032)| [  225.987852] intel_rapl: DRAM domain energy unit 15300pj
 1313.086631 (    0.000036)| [  225.993739] intel_rapl: Found RAPL domain package
 1313.086664 (    0.000033)| [  225.999028] intel_rapl: Found RAPL domain dram
 1313.086696 (    0.000032)| [  226.003992] intel_rapl: DRAM domain energy unit 15300pj
 1313.088593 (    0.001897)| [  226.009894] intel_rapl: Found RAPL domain package
 1313.091653 (    0.003060)| [  226.015181] intel_rapl: Found RAPL domain dram
 1313.099364 (    0.007711)| [  226.020160] intel_rapl: DRAM domain energy unit 15300pj
 1313.103488 (    0.004124)| [  226.026065] intel_rapl: Found RAPL domain package
 1313.249114 (    0.145626)| [  226.031349] intel_rapl: Found RAPL domain dram
 1313.249153 (    0.000039)| [  226.036316] intel_rapl: DRAM domain energy unit 15300pj
 1313.249190 (    0.000037)| [  226.042217] intel_rapl: Found RAPL domain package
 1313.249224 (    0.000034)| [  226.047505] intel_rapl: Found RAPL domain dram
 1313.249256 (    0.000032)| [  226.052472] intel_rapl: DRAM domain energy unit 15300pj
 1313.249292 (    0.000036)| [  226.058374] intel_rapl: Found RAPL domain package
 1313.249325 (    0.000033)| [  226.063658] intel_rapl: Found RAPL domain dram
 1313.249357 (    0.000032)| [  226.068624] intel_rapl: DRAM domain energy unit 15300pj
 1313.249393 (    0.000036)| [  226.074533] intel_rapl: Found RAPL domain package
 1313.249427 (    0.000034)| [  226.079822] intel_rapl: Found RAPL domain dram
 1313.249458 (    0.000031)| [  226.084786] intel_rapl: DRAM domain energy unit 15300pj
 1313.249494 (    0.000036)| [  226.090693] intel_rapl: Found RAPL domain package
 1313.249527 (    0.000033)| [  226.095982] intel_rapl: Found RAPL domain dram
 1313.249559 (    0.000032)| [  226.100947] intel_rapl: DRAM domain energy unit 15300pj
 1313.249604 (    0.000045)| [  226.106854] intel_rapl: Found RAPL domain package
 1313.249638 (    0.000034)| [  226.112140] intel_rapl: Found RAPL domain dram
 1313.249669 (    0.000031)| [  226.117104] intel_rapl: DRAM domain energy unit 15300pj
 1313.249706 (    0.000037)| [  226.123012] intel_rapl: Found RAPL domain package
 1313.249749 (    0.000043)| [  226.128300] intel_rapl: Found RAPL domain dram
 1313.249782 (    0.000033)| [  226.133264] intel_rapl: DRAM domain energy unit 15300pj
 1313.249819 (    0.000037)| [  226.139170] intel_rapl: Found RAPL domain package
 1313.249853 (    0.000034)| [  226.144456] intel_rapl: Found RAPL domain dram
 1313.249884 (    0.000031)| [  226.149420] intel_rapl: DRAM domain energy unit 15300pj
 1313.249920 (    0.000036)| [  226.155327] intel_rapl: Found RAPL domain package
 1313.249954 (    0.000034)| [  226.160615] intel_rapl: Found RAPL domain dram
 1313.249985 (    0.000031)| [  226.165585] intel_rapl: DRAM domain energy unit 15300pj
 1313.287118 (    0.037133)| [  226.171499] intel_rapl: Found RAPL domain package
 1313.287153 (    0.000035)| [  226.176782] intel_rapl: Found RAPL domain dram
 1313.287185 (    0.000032)| [  226.181747] intel_rapl: DRAM domain energy unit 15300pj
 1313.287221 (    0.000036)| [  226.187640] intel_rapl: Found RAPL domain package
 1313.287254 (    0.000033)| [  226.192920] intel_rapl: Found RAPL domain dram
 1313.287286 (    0.000032)| [  226.197886] intel_rapl: DRAM domain energy unit 15300pj
 1313.287322 (    0.000036)| [  226.203778] intel_rapl: Found RAPL domain package
 1313.287671 (    0.000349)| [  226.209058] intel_rapl: Found RAPL domain dram
 1313.292435 (    0.004764)| [  226.214025] intel_rapl: DRAM domain energy unit 15300pj
 1313.295894 (    0.003459)| [  226.219906] intel_rapl: Found RAPL domain package
 1313.303481 (    0.007587)| [  226.225176] intel_rapl: Found RAPL domain dram
 1313.307527 (    0.004046)| [  226.230140] intel_rapl: DRAM domain energy unit 15300pj
 1315.211784 (    1.904257)| [^[[0m^[[0;31m*     ^[[0m] A start job is running for LSB: AppArmor initialization (8s / 5min 3s)[  228.242048] audit_printk_skb: 75 callbacks suppressed
 1315.323377 (    0.111593)| [  228.247706] audit: type=1400 audit(1479413964.033:37): apparmor="STATUS" operation="profile_load" name="/usr/{sbin/traceroute,bin/traceroute.db}" pid=20035 comm="apparmor_parser"
 1315.523631 (    0.200254)| [  228.444888] audit: type=1400 audit(1479413964.233:38): apparmor="STATUS" operation="profile_load" name="/usr/sbin/winbindd" pid=20765 comm="apparmor_parser"
 1315.543824 (    0.020193)| [^[[0;32m  OK  ^[[0m] Started LSB: AppArmor initialization.
 1315.547856 (    0.004032)| [^[[0;32m  OK  ^[[0m] Reached target System Initialization.
 1315.555525 (    0.007669)| [^[[0;32m  OK  ^[[0m] Started Daily Cleanup of Temporary Directories.
 1315.563495 (    0.007970)| [^[[0;32m  OK  ^[[0m] Listening on Open-iSCSI iscsid Socket.
 1315.567956 (    0.004461)| [^[[0;32m  OK  ^[[0m] Listening on PC/SC Smart Card Daemon Activation Socket.
 1315.575388 (    0.007432)| [^[[0;32m  OK  ^[[0m] Started Discard unused blocks once a week.
 1315.583499 (    0.008111)| [^[[0;32m  OK  ^[[0m] Reached target Timers.
 1315.587771 (    0.004272)| [^[[0;32m  OK  ^[[0m] Listening on D-Bus System Message Bus Socket.
 1315.591850 (    0.004079)| [^[[0;32m  OK  ^[[0m] Reached target Sockets.
 1315.597706 (    0.005856)| [^[[0;32m  OK  ^[[0m] Reached target Basic System.
 1315.603949 (    0.006243)|          Starting System Logging Service...
 1315.611715 (    0.007766)|          Starting Load kdump kernel on startup...
 1315.617813 (    0.006098)|          Starting /etc/init.d/boot.local Compatibility...
 1315.637881 (    0.020068)|          Starting Name Service Cache Daemon...
 1315.652786 (    0.014905)|          Starting Corrected machine check interrupt manager daemon...
 1315.655384 (    0.002598)|          Starting SuSEfirewall2 phase 1...
 1315.666481 (    0.011097)| [^[[0;32m  OK  ^[[0m] Started D-Bus System Message Bus.
 1315.728753 (    0.062272)| [^[[0;32m  OK  ^[[0m] Started Self Monitoring and Reporting Technology (SMART) Daemon.
 1315.728807 (    0.000054)|          Starting LSB: gr_systat setup...
 1315.728835 (    0.000028)|          Starting Update cron periods from /etc/sysconfig/btrfsmaintenance...
 1315.776191 (    0.047356)| [  228.695471] iTCO_vendor_support: vendor-support=0
 1315.800740 (    0.024549)| [^[[0;1;3mFAILED^[[0m] Failed to start Load kdump kernel on startup.
 1315.808077 (    0.007337)| See 'systemctl status kdump.service' for details.
 1315.812739 (    0.004662)| [^[[0;32m  OK  ^[[0m] Started /etc/init.d/boot.local Compatibility.
 1315.819524 (    0.006785)| [^[[0;32m  OK  ^[[0m] Started Name Service Cache Daemon.
 1315.875962 (    0.056438)| [^[[0;32m  OK  ^[[0m] Started Update cron periods from /etc/sysconfig/btrfsmaintenance.
 1315.876019 (    0.000057)| [^[[0;32m  OK  ^[[0m] Started System Logging Service.
 1315.976987 (    0.100968)|          Starting SGI Automated Configuration Utilities...
 1316.004232 (    0.027245)| [^[[0;32m  OK  ^[[0m] Reached target User and Group Name Lookups.
 1316.019680 (    0.015448)|          Starting Permit User Sessions...
 1316.048196 (    0.028516)|          Starting Login Service...
 1316.080325 (    0.032129)| [^[[0;32m  OK  ^[[0m] Reached target Host and Network Name Lookups.
 1316.131735 (    0.051410)| [^[[0;32m  OK  ^[[0m] Started Permit User Sessions.
 1316.156383 (    0.024648)| [^[[0;32m  OK  ^[[0m] Started Login Service.
 1316.196182 (    0.039799)|          Starting Hold until boot process finishes up...
 1316.217572 (    0.021390)|          Starting Terminate Plymouth Boot Screen...
 1322.332633 (    6.115061)| 
 1322.344430 (    0.011797)| 
 1322.344445 (    0.000015)| Welcome to SUSE Linux Enterprise Server 12 SP2  (x86_64) - Kernel 4.9.0-rc5-clean+ (ttyS0).
 1322.352698 (    0.008253)| 
 1322.352708 (    0.000010)| 
 1322.368454 (    0.015746)| oracle login: 
 2697.304278 ( 1374.935824)| uvcon: interrupted via signal (3), exiting
 2697.304328 (    0.000050)| uv300-or-rmc RMC:r001i21c> logout
 2698.620389 (    1.316061)| Connection to 137.38.153.224 closed.
 2698.620651 (    0.000262)| athorlton@stormcage:~/notes/11-16/ovm-logs$ exit

Script done on Thu Nov 17 17:59:43 2016

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 15/11/16 01:11, Alex Thorlton wrote:
> Hey everyone,
> 
> We're having problems with large systems hitting a BUG in
> xen_memory_setup, due to extra e820 entries created in the
> XENMEM_machine_memory_map callback.  The change in the patch gets things
> working, but Boris and I wanted to get opinions on whether or not this
> is the appropriate/entire solution, which is why I've sent it as an RFC
> for now.
> 
> Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
> which is a detail worth discussig.  He proposed possibly adding
> CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
> value than E820MAX, since the Xen e820 table isn't bound by the
> zero-page memory limitations.
> 
> I do *slightly* question the use of E820_X_MAX here, only from a
> cosmetic prospective, as I believe this macro is intended to describe
> the maximum size of the extended e820 table, which, AFAIK, is not used
> by the Xen HV.  That being said, there isn't exactly a "more
> appropriate" macro/variable to use, so this may not really be an issue.
> 
> Any input on the patch, or the questions I've raised above is greatly
> appreciated!

While I think extending the e820 table is the right thing to do I'm
questioning the assumptions here.

Looking briefly through the Xen hypervisor sources I think it isn't
yet ready for such large machines: the hypervisor's e820 map seems to
be still limited to 128 e820 entries. Jan, did I overlook an EFI
specific path extending this limitation?

In case I'm right the Xen hypervisor should be prepared for a larger
e820 map, but this won't help alone as there would still be additional
entries for the IOAPICs created.

So I think we need something like:

#define E820_XEN_MAX (E820_X_MAX + MAX_IO_APICS)

and use this for sizing xen_e820_map[].


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[RFC PATCH v2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

Here's the second round of my patches to fix up the problems that we're
seeing with the XENMEM_machine_memory_map hypercall.  These few simple
changes, to give xen_e820_map some extra space, get things working fine
on our large machine.

This version of the patchset adds a patch to remove the #ifdef
CONFIG_EFI conditional around the definition of E820_X_MAX, so that it's
always slightly larger than E820MAX.  I've also tweaked the code that
works on xen_e820_map to use ARRAY_SIZE(xen_e820_map) instead of using
E820_X_MAX directly, as suggested by Boris.

As always, I appreciate any input that others can give!

- Alex

Alex Thorlton (2):
  x86: Make E820_X_MAX unconditionally larger than E820MAX
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/include/asm/e820.h | 8 +++-----
 arch/x86/xen/setup.c        | 6 +++---
 2 files changed, 6 insertions(+), 8 deletions(-)

-- 
1.8.5.6

[PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
E820_X_MAX is always at least slightly larger than E820MAX.

The real motivation behind this is actually to prevent some issues in
the Xen kernel, where the XENMEM_machine_memory_map hypercall can
produce an e820 map larger than 128 entries, even on systems where the
original e820 table was quite a bit smaller than that, depending on how
many IOAPICs are installed on the system.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Ingo Molnar <mingo@redhat.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/include/asm/e820.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
index 476b574..aa00d33 100644
--- a/arch/x86/include/asm/e820.h
+++ b/arch/x86/include/asm/e820.h
@@ -1,13 +1,15 @@
 #ifndef _ASM_X86_E820_H
 #define _ASM_X86_E820_H
 
-#ifdef CONFIG_EFI
+/*
+ * We need to make sure that E820_X_MAX is defined
+ * before we include uapi/asm/e820.h
+ */
 #include <linux/numa.h>
 #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
-#else	/* ! CONFIG_EFI */
-#define E820_X_MAX E820MAX
-#endif
+
 #include <uapi/asm/e820.h>
+
 #ifndef __ASSEMBLY__
 /* see comment in arch/x86/kernel/e820.c */
 extern struct e820map *e820;
-- 
1.8.5.6

[PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
E820_X_MAX is always at least slightly larger than E820MAX.

The real motivation behind this is actually to prevent some issues in
the Xen kernel, where the XENMEM_machine_memory_map hypercall can
produce an e820 map larger than 128 entries, even on systems where the
original e820 table was quite a bit smaller than that, depending on how
many IOAPICs are installed on the system.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Ingo Molnar <mingo@redhat.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/include/asm/e820.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
index 476b574..aa00d33 100644
--- a/arch/x86/include/asm/e820.h
+++ b/arch/x86/include/asm/e820.h
@@ -1,13 +1,15 @@
 #ifndef _ASM_X86_E820_H
 #define _ASM_X86_E820_H
 
-#ifdef CONFIG_EFI
+/*
+ * We need to make sure that E820_X_MAX is defined
+ * before we include uapi/asm/e820.h
+ */
 #include <linux/numa.h>
 #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
-#else	/* ! CONFIG_EFI */
-#define E820_X_MAX E820MAX
-#endif
+
 #include <uapi/asm/e820.h>
+
 #ifndef __ASSEMBLY__
 /* see comment in arch/x86/kernel/e820.c */
 extern struct e820map *e820;
-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Ingo Molnar]

* Alex Thorlton <athorlton@sgi.com> wrote:

> It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> E820_X_MAX is always at least slightly larger than E820MAX.
> 
> The real motivation behind this is actually to prevent some issues in
> the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> produce an e820 map larger than 128 entries, even on systems where the
> original e820 table was quite a bit smaller than that, depending on how
> many IOAPICs are installed on the system.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Ingo Molnar <mingo@redhat.com>
> Cc: Russ Anderson <rja@sgi.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: x86@kernel.org
> Cc: xen-devel@lists.xenproject.org
> ---
>  arch/x86/include/asm/e820.h | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
> index 476b574..aa00d33 100644
> --- a/arch/x86/include/asm/e820.h
> +++ b/arch/x86/include/asm/e820.h
> @@ -1,13 +1,15 @@
>  #ifndef _ASM_X86_E820_H
>  #define _ASM_X86_E820_H
>  
> -#ifdef CONFIG_EFI
> +/*
> + * We need to make sure that E820_X_MAX is defined
> + * before we include uapi/asm/e820.h
> + */
>  #include <linux/numa.h>
>  #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)

What we need an explanation for in the comment is what does this stand for (what 
does the 'X' mean?), and what is the magic 3*MAX_NUMNODES about?

Thanks,

	Ingo

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Ingo Molnar]

* Alex Thorlton <athorlton@sgi.com> wrote:

> It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> E820_X_MAX is always at least slightly larger than E820MAX.
> 
> The real motivation behind this is actually to prevent some issues in
> the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> produce an e820 map larger than 128 entries, even on systems where the
> original e820 table was quite a bit smaller than that, depending on how
> many IOAPICs are installed on the system.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Ingo Molnar <mingo@redhat.com>
> Cc: Russ Anderson <rja@sgi.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: x86@kernel.org
> Cc: xen-devel@lists.xenproject.org
> ---
>  arch/x86/include/asm/e820.h | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
> index 476b574..aa00d33 100644
> --- a/arch/x86/include/asm/e820.h
> +++ b/arch/x86/include/asm/e820.h
> @@ -1,13 +1,15 @@
>  #ifndef _ASM_X86_E820_H
>  #define _ASM_X86_E820_H
>  
> -#ifdef CONFIG_EFI
> +/*
> + * We need to make sure that E820_X_MAX is defined
> + * before we include uapi/asm/e820.h
> + */
>  #include <linux/numa.h>
>  #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)

What we need an explanation for in the comment is what does this stand for (what 
does the 'X' mean?), and what is the magic 3*MAX_NUMNODES about?

Thanks,

	Ingo

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

On Wed, Nov 30, 2016 at 07:21:48AM +0100, Ingo Molnar wrote:
> 
> * Alex Thorlton <athorlton@sgi.com> wrote:
> 
> > It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> > case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> > E820_X_MAX is always at least slightly larger than E820MAX.
> > 
> > The real motivation behind this is actually to prevent some issues in
> > the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> > produce an e820 map larger than 128 entries, even on systems where the
> > original e820 table was quite a bit smaller than that, depending on how
> > many IOAPICs are installed on the system.
> > 
> > Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> > Suggested-by: Ingo Molnar <mingo@redhat.com>
> > Cc: Russ Anderson <rja@sgi.com>
> > Cc: David Vrabel <david.vrabel@citrix.com>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: Juergen Gross <jgross@suse.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: "H. Peter Anvin" <hpa@zytor.com>
> > Cc: Denys Vlasenko <dvlasenk@redhat.com>
> > Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> > Cc: x86@kernel.org
> > Cc: xen-devel@lists.xenproject.org
> > ---
> >  arch/x86/include/asm/e820.h | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
> > index 476b574..aa00d33 100644
> > --- a/arch/x86/include/asm/e820.h
> > +++ b/arch/x86/include/asm/e820.h
> > @@ -1,13 +1,15 @@
> >  #ifndef _ASM_X86_E820_H
> >  #define _ASM_X86_E820_H
> >  
> > -#ifdef CONFIG_EFI
> > +/*
> > + * We need to make sure that E820_X_MAX is defined
> > + * before we include uapi/asm/e820.h
> > + */
> >  #include <linux/numa.h>
> >  #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
> 
> What we need an explanation for in the comment is what does this stand for (what 
> does the 'X' mean?), and what is the magic 3*MAX_NUMNODES about?

I'm not actually certain why 3*MAX_NUMNODES was chosen as the max size
for this table, but it was written by somebody here at SGI, so I'm sure
I can find out :)

As for the 'X,' I'm assuming that's meant to imply that this is the
maxmimum size for the 'eXtended' table, but that could be made more
clear in the comment as well.

I'll come up with a better comment for this and submit a v3.

Thanks, Ingo!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

On Wed, Nov 30, 2016 at 07:21:48AM +0100, Ingo Molnar wrote:
> 
> * Alex Thorlton <athorlton@sgi.com> wrote:
> 
> > It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> > case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> > E820_X_MAX is always at least slightly larger than E820MAX.
> > 
> > The real motivation behind this is actually to prevent some issues in
> > the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> > produce an e820 map larger than 128 entries, even on systems where the
> > original e820 table was quite a bit smaller than that, depending on how
> > many IOAPICs are installed on the system.
> > 
> > Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> > Suggested-by: Ingo Molnar <mingo@redhat.com>
> > Cc: Russ Anderson <rja@sgi.com>
> > Cc: David Vrabel <david.vrabel@citrix.com>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: Juergen Gross <jgross@suse.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: "H. Peter Anvin" <hpa@zytor.com>
> > Cc: Denys Vlasenko <dvlasenk@redhat.com>
> > Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> > Cc: x86@kernel.org
> > Cc: xen-devel@lists.xenproject.org
> > ---
> >  arch/x86/include/asm/e820.h | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
> > index 476b574..aa00d33 100644
> > --- a/arch/x86/include/asm/e820.h
> > +++ b/arch/x86/include/asm/e820.h
> > @@ -1,13 +1,15 @@
> >  #ifndef _ASM_X86_E820_H
> >  #define _ASM_X86_E820_H
> >  
> > -#ifdef CONFIG_EFI
> > +/*
> > + * We need to make sure that E820_X_MAX is defined
> > + * before we include uapi/asm/e820.h
> > + */
> >  #include <linux/numa.h>
> >  #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
> 
> What we need an explanation for in the comment is what does this stand for (what 
> does the 'X' mean?), and what is the magic 3*MAX_NUMNODES about?

I'm not actually certain why 3*MAX_NUMNODES was chosen as the max size
for this table, but it was written by somebody here at SGI, so I'm sure
I can find out :)

As for the 'X,' I'm assuming that's meant to imply that this is the
maxmimum size for the 'eXtended' table, but that could be made more
clear in the comment as well.

I'll come up with a better comment for this and submit a v3.

Thanks, Ingo!

[PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On systems with sufficiently large e820 tables, and several IOAPICs, it
is possible for the XENMEM_machine_memory_map callback (and its
counterpart, XENMEM_memory_map) to attempt to return an e820 table with
more than 128 entries.  This callback adds entries to the BIOS-provided
e820 table to account for IOAPIC registers, which, on sufficiently large
systems, can result in an e820 table that is too large to copy back into
xen_e820_map.

This change simply increases the size of xen_e820_map to E820_X_MAX to
ensure that there is enough room to store the entire e820 map returned
from this callback.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index f8960fc..8c394e3 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -41,7 +41,7 @@
 unsigned long xen_released_pages;
 
 /* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820MAX] __initdata;
+static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
 static u32 xen_e820_map_entries __initdata;
 
 /*
@@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
 	max_pfn = min(max_pfn, xen_start_info->nr_pages);
 	mem_end = PFN_PHYS(max_pfn);
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	op = xen_initial_domain() ?
@@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
 	int i;
 	int rc;
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On systems with sufficiently large e820 tables, and several IOAPICs, it
is possible for the XENMEM_machine_memory_map callback (and its
counterpart, XENMEM_memory_map) to attempt to return an e820 table with
more than 128 entries.  This callback adds entries to the BIOS-provided
e820 table to account for IOAPIC registers, which, on sufficiently large
systems, can result in an e820 table that is too large to copy back into
xen_e820_map.

This change simply increases the size of xen_e820_map to E820_X_MAX to
ensure that there is enough room to store the entire e820 map returned
from this callback.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index f8960fc..8c394e3 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -41,7 +41,7 @@
 unsigned long xen_released_pages;
 
 /* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820MAX] __initdata;
+static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
 static u32 xen_e820_map_entries __initdata;
 
 /*
@@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
 	max_pfn = min(max_pfn, xen_start_info->nr_pages);
 	mem_end = PFN_PHYS(max_pfn);
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	op = xen_initial_domain() ?
@@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
 	int i;
 	int rc;
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
-- 
1.8.5.6

Re: [PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 30/11/16 04:24, Alex Thorlton wrote:
> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>

Reviewed-by: Juergen Gross <jgross@suse.com>


Juergen

> Cc: Russ Anderson <rja@sgi.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: x86@kernel.org
> Cc: xen-devel@lists.xenproject.org
> ---
>  arch/x86/xen/setup.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index f8960fc..8c394e3 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -41,7 +41,7 @@
>  unsigned long xen_released_pages;
>  
>  /* E820 map used during setting up memory. */
> -static struct e820entry xen_e820_map[E820MAX] __initdata;
> +static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
>  static u32 xen_e820_map_entries __initdata;
>  
>  /*
> @@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
>  	max_pfn = min(max_pfn, xen_start_info->nr_pages);
>  	mem_end = PFN_PHYS(max_pfn);
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
>  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
>  
>  	op = xen_initial_domain() ?
> @@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
>  	int i;
>  	int rc;
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
>  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
>  
>  	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
> 

Re: [PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 30/11/16 04:24, Alex Thorlton wrote:
> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>

Reviewed-by: Juergen Gross <jgross@suse.com>


Juergen

> Cc: Russ Anderson <rja@sgi.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: x86@kernel.org
> Cc: xen-devel@lists.xenproject.org
> ---
>  arch/x86/xen/setup.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index f8960fc..8c394e3 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -41,7 +41,7 @@
>  unsigned long xen_released_pages;
>  
>  /* E820 map used during setting up memory. */
> -static struct e820entry xen_e820_map[E820MAX] __initdata;
> +static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
>  static u32 xen_e820_map_entries __initdata;
>  
>  /*
> @@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
>  	max_pfn = min(max_pfn, xen_start_info->nr_pages);
>  	mem_end = PFN_PHYS(max_pfn);
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
>  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
>  
>  	op = xen_initial_domain() ?
> @@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
>  	int i;
>  	int rc;
>  
> -	memmap.nr_entries = E820MAX;
> +	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
>  	set_xen_guest_handle(memmap.buffer, xen_e820_map);
>  
>  	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[RFC PATCH v2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

Here's the second round of my patches to fix up the problems that we're
seeing with the XENMEM_machine_memory_map hypercall.  These few simple
changes, to give xen_e820_map some extra space, get things working fine
on our large machine.

This version of the patchset adds a patch to remove the #ifdef
CONFIG_EFI conditional around the definition of E820_X_MAX, so that it's
always slightly larger than E820MAX.  I've also tweaked the code that
works on xen_e820_map to use ARRAY_SIZE(xen_e820_map) instead of using
E820_X_MAX directly, as suggested by Boris.

As always, I appreciate any input that others can give!

- Alex

Alex Thorlton (2):
  x86: Make E820_X_MAX unconditionally larger than E820MAX
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/include/asm/e820.h | 8 +++-----
 arch/x86/xen/setup.c        | 6 +++---
 2 files changed, 6 insertions(+), 8 deletions(-)

-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

This is the third pass at my patchset to fix up our problems with
XENMEM_machine_memory_map on large systems.  The only changes on this
pass were to flesh out the comment above the E820_X_MAX definition, and
to add Juergen's Reviewed-by to the second patch.

Let me know if anyone has any questions/comments!

Alex Thorlton (2):
  x86: Make E820_X_MAX unconditionally larger than E820MAX
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/include/asm/e820.h | 12 ++++++++----
 arch/x86/xen/setup.c        |  6 +++---
 2 files changed, 11 insertions(+), 7 deletions(-)

-- 
1.8.5.6

[PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
E820_X_MAX is always at least slightly larger than E820MAX.

The real motivation behind this is actually to prevent some issues in
the Xen kernel, where the XENMEM_machine_memory_map hypercall can
produce an e820 map larger than 128 entries, even on systems where the
original e820 table was quite a bit smaller than that, depending on how
many IOAPICs are installed on the system.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Ingo Molnar <mingo@redhat.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/include/asm/e820.h | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
index 476b574..ec23d8e 100644
--- a/arch/x86/include/asm/e820.h
+++ b/arch/x86/include/asm/e820.h
@@ -1,13 +1,17 @@
 #ifndef _ASM_X86_E820_H
 #define _ASM_X86_E820_H
 
-#ifdef CONFIG_EFI
+/*
+ * E820_X_MAX is the maximum size of the extended E820 table.  The extended
+ * table may contain up to 3 extra E820 entries per possible NUMA node, so we
+ * make room for 3 * MAX_NUMNODES possible entries, beyond the standard 128.
+ * Also note that E820_X_MAX *must* be defined before we include uapi/asm/e820.h.
+ */
 #include <linux/numa.h>
 #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
-#else	/* ! CONFIG_EFI */
-#define E820_X_MAX E820MAX
-#endif
+
 #include <uapi/asm/e820.h>
+
 #ifndef __ASSEMBLY__
 /* see comment in arch/x86/kernel/e820.c */
 extern struct e820map *e820;
-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Alex Thorlton]

It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
E820_X_MAX is always at least slightly larger than E820MAX.

The real motivation behind this is actually to prevent some issues in
the Xen kernel, where the XENMEM_machine_memory_map hypercall can
produce an e820 map larger than 128 entries, even on systems where the
original e820 table was quite a bit smaller than that, depending on how
many IOAPICs are installed on the system.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Ingo Molnar <mingo@redhat.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/include/asm/e820.h | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
index 476b574..ec23d8e 100644
--- a/arch/x86/include/asm/e820.h
+++ b/arch/x86/include/asm/e820.h
@@ -1,13 +1,17 @@
 #ifndef _ASM_X86_E820_H
 #define _ASM_X86_E820_H
 
-#ifdef CONFIG_EFI
+/*
+ * E820_X_MAX is the maximum size of the extended E820 table.  The extended
+ * table may contain up to 3 extra E820 entries per possible NUMA node, so we
+ * make room for 3 * MAX_NUMNODES possible entries, beyond the standard 128.
+ * Also note that E820_X_MAX *must* be defined before we include uapi/asm/e820.h.
+ */
 #include <linux/numa.h>
 #define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
-#else	/* ! CONFIG_EFI */
-#define E820_X_MAX E820MAX
-#endif
+
 #include <uapi/asm/e820.h>
+
 #ifndef __ASSEMBLY__
 /* see comment in arch/x86/kernel/e820.c */
 extern struct e820map *e820;
-- 
1.8.5.6

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> E820_X_MAX is always at least slightly larger than E820MAX.
> 
> The real motivation behind this is actually to prevent some issues in
> the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> produce an e820 map larger than 128 entries, even on systems where the
> original e820 table was quite a bit smaller than that, depending on how
> many IOAPICs are installed on the system.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Ingo Molnar <mingo@redhat.com>


Commited to xen/tip.git for-linus-4.10


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 1/2] x86: Make E820_X_MAX unconditionally larger than E820MAX

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> It's really not necessary to limit E820_X_MAX to 128 in the non-EFI
> case.  This commit drops E820_X_MAX's dependency on CONFIG_EFI, so that
> E820_X_MAX is always at least slightly larger than E820MAX.
> 
> The real motivation behind this is actually to prevent some issues in
> the Xen kernel, where the XENMEM_machine_memory_map hypercall can
> produce an e820 map larger than 128 entries, even on systems where the
> original e820 table was quite a bit smaller than that, depending on how
> many IOAPICs are installed on the system.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Ingo Molnar <mingo@redhat.com>


Commited to xen/tip.git for-linus-4.10


Juergen

[PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On systems with sufficiently large e820 tables, and several IOAPICs, it
is possible for the XENMEM_machine_memory_map callback (and its
counterpart, XENMEM_memory_map) to attempt to return an e820 table with
more than 128 entries.  This callback adds entries to the BIOS-provided
e820 table to account for IOAPIC registers, which, on sufficiently large
systems, can result in an e820 table that is too large to copy back into
xen_e820_map.

This change simply increases the size of xen_e820_map to E820_X_MAX to
ensure that there is enough room to store the entire e820 map returned
from this callback.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index f8960fc..8c394e3 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -41,7 +41,7 @@
 unsigned long xen_released_pages;
 
 /* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820MAX] __initdata;
+static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
 static u32 xen_e820_map_entries __initdata;
 
 /*
@@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
 	max_pfn = min(max_pfn, xen_start_info->nr_pages);
 	mem_end = PFN_PHYS(max_pfn);
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	op = xen_initial_domain() ?
@@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
 	int i;
 	int rc;
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
-- 
1.8.5.6

Re: [PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Reviewed-by: Juergen Gross <jgross@suse.com>


Commited to xen/tip.git for-linus-4.10


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> On systems with sufficiently large e820 tables, and several IOAPICs, it
> is possible for the XENMEM_machine_memory_map callback (and its
> counterpart, XENMEM_memory_map) to attempt to return an e820 table with
> more than 128 entries.  This callback adds entries to the BIOS-provided
> e820 table to account for IOAPIC registers, which, on sufficiently large
> systems, can result in an e820 table that is too large to copy back into
> xen_e820_map.
> 
> This change simply increases the size of xen_e820_map to E820_X_MAX to
> ensure that there is enough room to store the entire e820 map returned
> from this callback.
> 
> Signed-off-by: Alex Thorlton <athorlton@sgi.com>
> Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Reviewed-by: Juergen Gross <jgross@suse.com>


Commited to xen/tip.git for-linus-4.10


Juergen

[PATCH 2/2] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

On systems with sufficiently large e820 tables, and several IOAPICs, it
is possible for the XENMEM_machine_memory_map callback (and its
counterpart, XENMEM_memory_map) to attempt to return an e820 table with
more than 128 entries.  This callback adds entries to the BIOS-provided
e820 table to account for IOAPIC registers, which, on sufficiently large
systems, can result in an e820 table that is too large to copy back into
xen_e820_map.

This change simply increases the size of xen_e820_map to E820_X_MAX to
ensure that there is enough room to store the entire e820 map returned
from this callback.

Signed-off-by: Alex Thorlton <athorlton@sgi.com>
Suggested-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Cc: Russ Anderson <rja@sgi.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: x86@kernel.org
Cc: xen-devel@lists.xenproject.org
---
 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index f8960fc..8c394e3 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -41,7 +41,7 @@
 unsigned long xen_released_pages;
 
 /* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820MAX] __initdata;
+static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
 static u32 xen_e820_map_entries __initdata;
 
 /*
@@ -750,7 +750,7 @@ char * __init xen_memory_setup(void)
 	max_pfn = min(max_pfn, xen_start_info->nr_pages);
 	mem_end = PFN_PHYS(max_pfn);
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	op = xen_initial_domain() ?
@@ -923,7 +923,7 @@ char * __init xen_auto_xlated_memory_setup(void)
 	int i;
 	int rc;
 
-	memmap.nr_entries = E820MAX;
+	memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
 	set_xen_guest_handle(memmap.buffer, xen_e820_map);
 
 	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> This is the third pass at my patchset to fix up our problems with
> XENMEM_machine_memory_map on large systems.  The only changes on this
> pass were to flesh out the comment above the E820_X_MAX definition, and
> to add Juergen's Reviewed-by to the second patch.
> 
> Let me know if anyone has any questions/comments!
> 
> Alex Thorlton (2):
>   x86: Make E820_X_MAX unconditionally larger than E820MAX
>   xen/x86: Increase xen_e820_map to E820_X_MAX possible entries
> 
>  arch/x86/include/asm/e820.h | 12 ++++++++----
>  arch/x86/xen/setup.c        |  6 +++---
>  2 files changed, 11 insertions(+), 7 deletions(-)
> 

Ingo, do you have any preferences through which tree those patches
should go? I'd like to have at least patch 2 in 4.10, so I could take
it through the Xen tree.


Juergen

Re: [RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Ingo Molnar]

* Juergen Gross <jgross@suse.com> wrote:

> On 05/12/16 18:49, Alex Thorlton wrote:
> > This is the third pass at my patchset to fix up our problems with
> > XENMEM_machine_memory_map on large systems.  The only changes on this
> > pass were to flesh out the comment above the E820_X_MAX definition, and
> > to add Juergen's Reviewed-by to the second patch.
> > 
> > Let me know if anyone has any questions/comments!
> > 
> > Alex Thorlton (2):
> >   x86: Make E820_X_MAX unconditionally larger than E820MAX
> >   xen/x86: Increase xen_e820_map to E820_X_MAX possible entries
> > 
> >  arch/x86/include/asm/e820.h | 12 ++++++++----
> >  arch/x86/xen/setup.c        |  6 +++---
> >  2 files changed, 11 insertions(+), 7 deletions(-)
> > 
> 
> Ingo, do you have any preferences through which tree those patches
> should go? I'd like to have at least patch 2 in 4.10, so I could take
> it through the Xen tree.

Sure, both are fine to me:

  Acked-by: Ingo Molnar <mingo@kernel.org>

Thanks,

	Ingo

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Ingo Molnar]

* Juergen Gross <jgross@suse.com> wrote:

> On 05/12/16 18:49, Alex Thorlton wrote:
> > This is the third pass at my patchset to fix up our problems with
> > XENMEM_machine_memory_map on large systems.  The only changes on this
> > pass were to flesh out the comment above the E820_X_MAX definition, and
> > to add Juergen's Reviewed-by to the second patch.
> > 
> > Let me know if anyone has any questions/comments!
> > 
> > Alex Thorlton (2):
> >   x86: Make E820_X_MAX unconditionally larger than E820MAX
> >   xen/x86: Increase xen_e820_map to E820_X_MAX possible entries
> > 
> >  arch/x86/include/asm/e820.h | 12 ++++++++----
> >  arch/x86/xen/setup.c        |  6 +++---
> >  2 files changed, 11 insertions(+), 7 deletions(-)
> > 
> 
> Ingo, do you have any preferences through which tree those patches
> should go? I'd like to have at least patch 2 in 4.10, so I could take
> it through the Xen tree.

Sure, both are fine to me:

  Acked-by: Ingo Molnar <mingo@kernel.org>

Thanks,

	Ingo

Re: [RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Juergen Gross]

On 05/12/16 18:49, Alex Thorlton wrote:
> This is the third pass at my patchset to fix up our problems with
> XENMEM_machine_memory_map on large systems.  The only changes on this
> pass were to flesh out the comment above the E820_X_MAX definition, and
> to add Juergen's Reviewed-by to the second patch.
> 
> Let me know if anyone has any questions/comments!
> 
> Alex Thorlton (2):
>   x86: Make E820_X_MAX unconditionally larger than E820MAX
>   xen/x86: Increase xen_e820_map to E820_X_MAX possible entries
> 
>  arch/x86/include/asm/e820.h | 12 ++++++++----
>  arch/x86/xen/setup.c        |  6 +++---
>  2 files changed, 11 insertions(+), 7 deletions(-)
> 

Ingo, do you have any preferences through which tree those patches
should go? I'd like to have at least patch 2 in 4.10, so I could take
it through the Xen tree.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[RFC PATCH v3] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

This is the third pass at my patchset to fix up our problems with
XENMEM_machine_memory_map on large systems.  The only changes on this
pass were to flesh out the comment above the E820_X_MAX definition, and
to add Juergen's Reviewed-by to the second patch.

Let me know if anyone has any questions/comments!

Alex Thorlton (2):
  x86: Make E820_X_MAX unconditionally larger than E820MAX
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/include/asm/e820.h | 12 ++++++++----
 arch/x86/xen/setup.c        |  6 +++---
 2 files changed, 11 insertions(+), 7 deletions(-)

-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[RFC PATCH] xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

[Alex Thorlton]

Hey everyone,

We're having problems with large systems hitting a BUG in
xen_memory_setup, due to extra e820 entries created in the
XENMEM_machine_memory_map callback.  The change in the patch gets things
working, but Boris and I wanted to get opinions on whether or not this
is the appropriate/entire solution, which is why I've sent it as an RFC
for now.

Boris pointed out to me that E820_X_MAX is only large when CONFIG_EFI=y,
which is a detail worth discussig.  He proposed possibly adding
CONFIG_XEN to the conditions under which we set E820_X_MAX to a larger
value than E820MAX, since the Xen e820 table isn't bound by the
zero-page memory limitations.

I do *slightly* question the use of E820_X_MAX here, only from a
cosmetic prospective, as I believe this macro is intended to describe
the maximum size of the extended e820 table, which, AFAIK, is not used
by the Xen HV.  That being said, there isn't exactly a "more
appropriate" macro/variable to use, so this may not really be an issue.

Any input on the patch, or the questions I've raised above is greatly
appreciated!

- Alex

Alex Thorlton (1):
  xen/x86: Increase xen_e820_map to E820_X_MAX possible entries

 arch/x86/xen/setup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
1.8.5.6


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel