* RFE: Detect NUMA misconfigurations and prevent machine freezes
@ 2018-08-29 5:33 Steven Haigh
2018-08-29 5:49 ` Juergen Gross
2018-08-29 6:39 ` Jan Beulich
0 siblings, 2 replies; 11+ messages in thread
From: Steven Haigh @ 2018-08-29 5:33 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1433 bytes --]
When playing with NUMA support recently, I noticed a host would always hang
when trying to create a cpupool for the second NUMA node in the system.
I was using the following commands:
# xl cpupool-create name=\"Pool-1\" sched=\"credit2\
# xl cpupool-cpu-remove Pool-0 node:1
# xl cpupool-cpu-add Pool-1 node:1
After the last command, the system would hang - requiring a hard reset of the
machine to fix.
I tried a different variation with the same result:
# xl cpupool-create name=\"Pool-1\" sched=\"credit2\
# xl cpupool-cpu-remove Pool-0 node:1
# xl cpupool-cpu-add Pool-1 12
It turns out that the RAM was installed sub-optimally in this machine. A
partial output from 'xl info -n' shows:
numa_info :
node: memsize memfree distances
0: 67584 62608 10,21
1: 0 0 21,10
A machine where we could get this working every time shows:
node: memsize memfree distances
0: 34816 30483 10,21
1: 32768 32125 21,10
As we can deduce RAM misconfigurations in this scenario, I believe we should
check to ensure that RAM configuration / layout is sane *before* attempting to
split the system and print a warning.
This would prevent a hard system freeze in this scenario.
--
Steven Haigh
📧 netwiz@crc.id.au 💻 https://www.crc.id.au
📞 +61 (3) 9001 6090 📱 0412 935 897
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-29 5:33 RFE: Detect NUMA misconfigurations and prevent machine freezes Steven Haigh
@ 2018-08-29 5:49 ` Juergen Gross
2018-08-30 4:01 ` Steven Haigh
2018-08-29 6:39 ` Jan Beulich
1 sibling, 1 reply; 11+ messages in thread
From: Juergen Gross @ 2018-08-29 5:49 UTC (permalink / raw)
To: Steven Haigh, xen-devel
On 29/08/18 07:33, Steven Haigh wrote:
> When playing with NUMA support recently, I noticed a host would always hang
> when trying to create a cpupool for the second NUMA node in the system.
>
> I was using the following commands:
> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
> # xl cpupool-cpu-remove Pool-0 node:1
> # xl cpupool-cpu-add Pool-1 node:1
>
> After the last command, the system would hang - requiring a hard reset of the
> machine to fix.
>
> I tried a different variation with the same result:
> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
> # xl cpupool-cpu-remove Pool-0 node:1
> # xl cpupool-cpu-add Pool-1 12
>
> It turns out that the RAM was installed sub-optimally in this machine. A
> partial output from 'xl info -n' shows:
> numa_info :
> node: memsize memfree distances
> 0: 67584 62608 10,21
> 1: 0 0 21,10
>
> A machine where we could get this working every time shows:
> node: memsize memfree distances
> 0: 34816 30483 10,21
> 1: 32768 32125 21,10
>
> As we can deduce RAM misconfigurations in this scenario, I believe we should
> check to ensure that RAM configuration / layout is sane *before* attempting to
> split the system and print a warning.
>
> This would prevent a hard system freeze in this scenario.
RAM placement should not matter here. As the name already suggests
cpupools do assignment of cpus. RAM allocated will be preferred taken
from a local node, but this shouldn't be mandatory for success.
Would it be possible to use a debug hypervisor (e.g. 4.12-unstable) for
generating a verbose log (hypervisor boot parameter "loglvl=all") and
sending the complete hypervisor log?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-29 5:33 RFE: Detect NUMA misconfigurations and prevent machine freezes Steven Haigh
2018-08-29 5:49 ` Juergen Gross
@ 2018-08-29 6:39 ` Jan Beulich
1 sibling, 0 replies; 11+ messages in thread
From: Jan Beulich @ 2018-08-29 6:39 UTC (permalink / raw)
To: Steven Haigh; +Cc: xen-devel
>>> On 29.08.18 at 07:33, <netwiz@crc.id.au> wrote:
> When playing with NUMA support recently, I noticed a host would always hang
> when trying to create a cpupool for the second NUMA node in the system.
>
> I was using the following commands:
> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
> # xl cpupool-cpu-remove Pool-0 node:1
> # xl cpupool-cpu-add Pool-1 node:1
>
> After the last command, the system would hang - requiring a hard reset of the
> machine to fix.
Yeah, as Jürgen says - we need to get to the bottom of this,
instead of trying to cure just the symptom.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-29 5:49 ` Juergen Gross
@ 2018-08-30 4:01 ` Steven Haigh
2018-08-30 7:13 ` Juergen Gross
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Steven Haigh @ 2018-08-30 4:01 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
On 2018-08-29 15:49, Juergen Gross wrote:
> On 29/08/18 07:33, Steven Haigh wrote:
>> When playing with NUMA support recently, I noticed a host would always
>> hang
>> when trying to create a cpupool for the second NUMA node in the
>> system.
>>
>> I was using the following commands:
>> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
>> # xl cpupool-cpu-remove Pool-0 node:1
>> # xl cpupool-cpu-add Pool-1 node:1
>>
>> After the last command, the system would hang - requiring a hard reset
>> of the
>> machine to fix.
>>
>> I tried a different variation with the same result:
>> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
>> # xl cpupool-cpu-remove Pool-0 node:1
>> # xl cpupool-cpu-add Pool-1 12
>>
>> It turns out that the RAM was installed sub-optimally in this machine.
>> A
>> partial output from 'xl info -n' shows:
>> numa_info :
>> node: memsize memfree distances
>> 0: 67584 62608 10,21
>> 1: 0 0 21,10
>>
>> A machine where we could get this working every time shows:
>> node: memsize memfree distances
>> 0: 34816 30483 10,21
>> 1: 32768 32125 21,10
>>
>> As we can deduce RAM misconfigurations in this scenario, I believe we
>> should
>> check to ensure that RAM configuration / layout is sane *before*
>> attempting to
>> split the system and print a warning.
>>
>> This would prevent a hard system freeze in this scenario.
>
> RAM placement should not matter here. As the name already suggests
> cpupools do assignment of cpus. RAM allocated will be preferred taken
> from a local node, but this shouldn't be mandatory for success.
>
> Would it be possible to use a debug hypervisor (e.g. 4.12-unstable) for
> generating a verbose log (hypervisor boot parameter "loglvl=all") and
> sending the complete hypervisor log?
I don't have a package for 4.11 or 4.12 built at all - but I did this on
4.10.2-pre (built from staging-4.10).
Managed to get the same crash log when adding CPUs to Pool-1 as follows:
Create the pool:
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
Add the CPUs:
(XEN) Adding cpu 12 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Removing cpu 12 from runqueue 0
(XEN) Adding cpu 13 to runqueue 0
(XEN) Removing cpu 13 from runqueue 0
(XEN) Adding cpu 14 to runqueue 0
(XEN) Removing cpu 14 from runqueue 0
(XEN) Xen BUG at sched_credit2.c:3452
(XEN) Adding cpu 15 to runqueue 0
(XEN) ----[ Xen-4.10.2-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 13
(XEN) RIP: e008:[<ffff82d08022fe40>]
sched_credit2.c#csched2_schedule+0xf60/0x1330
(XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff82d080562500 rcx:
fffffffffffffffd
(XEN) rdx: ffff831040ae1f80 rsi: 000000000000000d rdi:
ffff831050abfe38
(XEN) rbp: ffff831044ba7f30 rsp: ffff831050abfd20 r8:
ffff831050ac61a0
(XEN) r9: ffff82d08022eee0 r10: 0000000000000001 r11:
ffff83007dc11060
(XEN) r12: 000000000000000d r13: ffff831050aefec0 r14:
ffff831050ac6180
(XEN) r15: ffff82d080562500 cr0: 000000008005003b cr4:
00000000001526e0
(XEN) cr3: 000000007dc32000 cr2: 00007fff9cb47838
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss:
0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen code around <ffff82d08022fe40>
(sched_credit2.c#csched2_schedule+0xf60/0x1330):
(XEN) ff ff 0f 0b 0f 1f 40 00 <0f> 0b 0f 0b 4c 89 ef e8 d4 b6 ff ff e9
87 f1 ff
(XEN) Xen stack trace from rsp=ffff831050abfd20:
(XEN) ffff83107f5f6010 ffff831050abfe38 0000002dd914464a
000000000000000d
(XEN) 0000000002f8d264 ffff831000000000 0000000000000000
ffff8310419efdd0
(XEN) ffff831050ac6400 ffff82d0802344fa 0000000000000000
0000001bc97d2ea7
(XEN) 0000002d9f005422 ffff831050ac63c0 ffff82d080562500
ffff82d080577280
(XEN) 000000000000000d 0000000000000046 0000000000000282
0000000000000082
(XEN) 000000000000000d ffff831050ac61c8 ffff82d080576fac
000000000000000d
(XEN) ffff83007dd7f000 ffff831044ba8000 0000002dd914464a
ffff831050ac6180
(XEN) ffff82d080562500 ffff82d080233eed 0000002d00000000
ffff831050ac61a0
(XEN) ffff831050ab6010 ffff82d080267cf5 ffff82d0802bffe0
ffff82d0802c4505
(XEN) 0000002dd61b7fed ffff82d08023ae48 000038000000380b
00000000ffffffff
(XEN) ffffffffffffffff ffff831050abffff ffff82d08054bc80
ffff82d080562500
(XEN) ffff831050abffff ffff82d0802375b2 ffff82d0805771f0
ffff82d08054c300
(XEN) ffff82d0805771f0 000000000000000d 000000000000000d
ffff82d08026d2d5
(XEN) ffff83007dd7f000 ffff83007dd7f000 ffff83007dbae000
ffff83107cd38000
(XEN) 00000000ffffffff ffff83107fb0d000 ffff82d080562500
0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000
ffff880736083fd8
(XEN) ffff880736083fd4 2e6b180200000001 0000000000000179
0000000000000000
(XEN) 00000000000000ff 0000000000000179 000000fa00000000
ffffffff81030f66
(XEN) 0000000000000000 0000000000000046 ffff880736083fc0
0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08022fe40>]
sched_credit2.c#csched2_schedule+0xf60/0x1330
(XEN) [<ffff82d0802344fa>] get_cpu_idle_time+0x3a/0x50
(XEN) [<ffff82d080233eed>] schedule.c#schedule+0xdd/0x5d0
(XEN) [<ffff82d080267cf5>] reprogram_timer+0x75/0xe0
(XEN) [<ffff82d0802bffe0>] cpuidle_menu.c#menu_reflect+0/0x80
(XEN) [<ffff82d0802c4505>] mwait-idle.c#mwait_idle+0x275/0x3d0
(XEN) [<ffff82d08023ae48>] timer.c#timer_softirq_action+0x138/0x210
(XEN) [<ffff82d0802375b2>] softirq.c#__do_softirq+0x62/0x90
(XEN) [<ffff82d08026d2d5>] domain.c#idle_loop+0x45/0xb0
(XEN)
(XEN) Removing cpu 15 from runqueue 0
(XEN)
(XEN) ****************************************
(XEN) Adding cpu 16 to runqueue 0
(XEN) Panic on CPU 13:
(XEN) Removing cpu 16 from runqueue 0
(XEN) Xen BUG at sched_credit2.c:3452
(XEN) Adding cpu 17 to runqueue 0
(XEN) ****************************************
(XEN)
(XEN) Removing cpu 17 from runqueue 0
(XEN) Reboot in five seconds...
(XEN) Adding cpu 18 to runqueue 0
(XEN) Removing cpu 18 from runqueue 0
Only thing I could do at this point was to hit the reset button.
Full boot logs (if helpful):
(XEN) Xen version 4.10.2-pre (mockbuild@) (gcc (GCC) 4.8.5 20150623 (Red
Hat 4.8.5-28)) debug=n Fri Aug 24 15:13:31 AEST 2018
(XEN) Latest ChangeSet: Tue Aug 14 15:13:09 2018 -0700 git:bf1b7f5-dirty
(XEN) Bootloader: GRUB 2.02~beta2
(XEN) Command line: placeholder dom0_mem=2048M,max:2048M ucode=scan
dom0_max_vcpus=2 sched=credit2 com2=115200,8n1 console=com2 loglvl=all
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009ac00 (usable)
(XEN) 000000000009ac00 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 000000007df1e000 (usable)
(XEN) 000000007df1e000 - 000000007e0d4000 (reserved)
(XEN) 000000007e0d4000 - 000000007e2dc000 (ACPI NVS)
(XEN) 000000007e2dc000 - 000000007f358000 (reserved)
(XEN) 000000007f358000 - 000000007f800000 (ACPI NVS)
(XEN) 0000000080000000 - 0000000090000000 (reserved)
(XEN) 00000000fed1c000 - 00000000fed40000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000001080000000 (usable)
(XEN) New Xen image base address: 0x7d800000
(XEN) ACPI: RSDP 000F04A0, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT 7E1F4090, 009C (r1 ALASKA A M I 1072009 AMI
10013)
(XEN) ACPI: FACP 7E201048, 010C (r5 ALASKA A M I 1072009 AMI
10013)
(XEN) ACPI: DSDT 7E1F41C0, CE87 (r2 ALASKA A M I 15 INTL
20051117)
(XEN) ACPI: FACS 7E2D3080, 0040
(XEN) ACPI: APIC 7E201158, 01B4 (r3 ALASKA A M I 1072009 AMI
10013)
(XEN) ACPI: FPDT 7E201310, 0044 (r1 ALASKA A M I 1072009 AMI
10013)
(XEN) ACPI: MCFG 7E201358, 003C (r1 ALASKA OEMMCFG. 1072009 MSFT
97)
(XEN) ACPI: SRAT 7E201398, 0430 (r1 A M I AMI SRAT 1 AMI.
0)
(XEN) ACPI: SLIT 7E2017C8, 0030 (r1 A M I AMI SLIT 0 AMI.
0)
(XEN) ACPI: HPET 7E2017F8, 0038 (r1 ALASKA A M I 1072009 AMI.
5)
(XEN) ACPI: PRAD 7E201830, 00BE (r2 PRADID PRADTID 1 MSFT
3000001)
(XEN) ACPI: SPMI 7E2018F0, 0040 (r5 A M I OEMSPMI 0 AMI.
0)
(XEN) ACPI: SSDT 7E201930, D0CB0 (r2 INTEL CpuPm 4000 INTL
20051117)
(XEN) ACPI: EINJ 7E2D25E0, 0130 (r1 AMI AMI EINJ 0
0)
(XEN) ACPI: ERST 7E2D2710, 0230 (r1 AMIER AMI ERST 0
0)
(XEN) ACPI: HEST 7E2D2940, 00A8 (r1 AMI AMI HEST 0
0)
(XEN) ACPI: BERT 7E2D29E8, 0030 (r1 AMI AMI BERT 0
0)
(XEN) ACPI: DMAR 7E2D2A18, 0160 (r1 A M I OEMDMAR 1 INTL
1)
(XEN) System RAM: 65502MB (67074784kB)
(XEN) SRAT: PXM 0 -> APIC 00 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 01 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 02 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 03 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 04 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 05 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 06 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 07 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 08 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 09 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 0a -> Node 0
(XEN) SRAT: PXM 0 -> APIC 0b -> Node 0
(XEN) SRAT: PXM 1 -> APIC 20 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 21 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 22 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 23 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 24 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 25 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 26 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 27 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 28 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 29 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 2a -> Node 1
(XEN) SRAT: PXM 1 -> APIC 2b -> Node 1
(XEN) SRAT: Node 0 PXM 0 0-80000000
(XEN) SRAT: Node 0 PXM 0 100000000-1080000000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) SRAT: Node 1 has no memory. BIOS Bug or mis-configured hardware?
(XEN) Domain heap initialised DMA width 32 bits
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 62 (0x3e), Stepping 4
(raw 000306e4)
(XEN) found SMP MP-table at 000fdb60
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408 (32 bits)
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
7e2d3080/0000000000000000, using 32
(XEN) ACPI: wakeup_vec[7e2d308c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x20] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x22] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x24] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x28] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2a] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x21] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x23] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x25] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x27] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x29] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x2b] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x11] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x13] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x15] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x17] high edge lint[0x1])
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec01000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) ACPI: IOAPIC (id[0x03] address[0xfec40000] gsi_base[48])
(XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) Xen ERST support is initialized.
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 72 GSI, 4552 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) CPU0: Intel machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN) Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- SSBD-,
Other: IBPB L1D_FLUSH
(XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe
address 300000000000
(XEN) Support for VMs: PV: MSR_SPEC_CTRL RSB EAGER_FPU, HVM:
MSR_SPEC_CTRL RSB EAGER_FPU
(XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled
(XEN) PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 2600.019 MHz processor.
(XEN) Initing memory sharing.
(XEN) alt table ffff82d080419778 -> ffff82d08041b260
(XEN) PCI: MCFG configuration 0: base 80000000 segment 0000 buses 00 -
ff
(XEN) PCI: MCFG area at 80000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 2
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) Defaulting to alternative key handling; send 'A' to switch to
normal mode.
(XEN) Allocated console ring of 256 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4.1 model 0x3e
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) - APIC Register Virtualization
(XEN) - Virtual Interrupt Delivery
(XEN) - Posted Interrupt Processing
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) Adding cpu 4 to runqueue 0
(XEN) Adding cpu 5 to runqueue 0
(XEN) Adding cpu 6 to runqueue 0
(XEN) Adding cpu 7 to runqueue 0
(XEN) Adding cpu 8 to runqueue 0
(XEN) Adding cpu 9 to runqueue 0
(XEN) Adding cpu 10 to runqueue 0
(XEN) Adding cpu 11 to runqueue 0
(XEN) Adding cpu 12 to runqueue 1
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 13 to runqueue 1
(XEN) Adding cpu 14 to runqueue 1
(XEN) Adding cpu 15 to runqueue 1
(XEN) Adding cpu 16 to runqueue 1
(XEN) Adding cpu 17 to runqueue 1
(XEN) Adding cpu 18 to runqueue 1
(XEN) Adding cpu 19 to runqueue 1
(XEN) Adding cpu 20 to runqueue 1
(XEN) Adding cpu 21 to runqueue 1
(XEN) Adding cpu 22 to runqueue 1
(XEN) Adding cpu 23 to runqueue 1
(XEN) Brought up 24 CPUs
(XEN) build-id: 1e7674bde0383cfdea16d2d4f65364172565de0a
(XEN) ACPI sleep modes: S3
(XEN) VPMU: disabled
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 456 PIRQs
(XEN) NX (Execute Disable) protection active
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2768000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 000000104c000000->0000001050000000 (503266 pages
to be allocated)
(XEN) Init. ramdisk: 000000107ede2000->000000107ffff6d8
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff82768000
(XEN) Init. ramdisk: 0000000000000000->0000000000000000
(XEN) Phys-Mach map: 0000008000000000->0000008000400000
(XEN) Start info: ffffffff82768000->ffffffff827684b4
(XEN) Xenstore ring: 0000000000000000->0000000000000000
(XEN) Console ring: 0000000000000000->0000000000000000
(XEN) Page tables: ffffffff82769000->ffffffff82782000
(XEN) Boot stack: ffffffff82782000->ffffffff82783000
(XEN) TOTAL: ffffffff80000000->ffffffff82c00000
(XEN) ENTRY ADDRESS: ffffffff82452180
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM on 2 nodes using 6 CPUs
(XEN)
........................................................................................done.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) ***************************************************
(XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
(XEN) enabled. Please assess your configuration and choose an
(XEN) explicit 'smt=<bool>' setting. See XSA-273.
(XEN) 3... 2... 1...
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 472kB init memory
mapping kernel into physical memory
about to get started...
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) PCI add device 0000:00:03.2
(XEN) PCI add device 0000:00:04.0
(XEN) PCI add device 0000:00:04.1
(XEN) PCI add device 0000:00:04.2
(XEN) PCI add device 0000:00:04.3
(XEN) PCI add device 0000:00:04.4
(XEN) PCI add device 0000:00:04.5
(XEN) PCI add device 0000:00:04.6
(XEN) PCI add device 0000:00:04.7
(XEN) PCI add device 0000:00:05.0
(XEN) PCI add device 0000:00:05.2
(XEN) PCI add device 0000:00:05.4
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:16.1
(XEN) PCI add device 0000:00:1a.0
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1e.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.2
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:00:1f.6
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:02:00.1
(XEN) PCI add device 0000:07:00.0
(XEN) PCI add device 0000:08:03.0
(XEN) PCI add device 0000:7f:08.0
(XEN) PCI add device 0000:7f:09.0
(XEN) PCI add device 0000:7f:0a.0
(XEN) PCI add device 0000:7f:0a.1
(XEN) PCI add device 0000:7f:0a.2
(XEN) PCI add device 0000:7f:0a.3
(XEN) PCI add device 0000:7f:0b.0
(XEN) PCI add device 0000:7f:0b.3
(XEN) PCI add device 0000:7f:0c.0
(XEN) PCI add device 0000:7f:0c.1
(XEN) PCI add device 0000:7f:0c.2
(XEN) PCI add device 0000:7f:0d.0
(XEN) PCI add device 0000:7f:0d.1
(XEN) PCI add device 0000:7f:0d.2
(XEN) PCI add device 0000:7f:0e.0
(XEN) PCI add device 0000:7f:0e.1
(XEN) PCI add device 0000:7f:0f.0
(XEN) PCI add device 0000:7f:0f.1
(XEN) PCI add device 0000:7f:0f.2
(XEN) PCI add device 0000:7f:0f.3
(XEN) PCI add device 0000:7f:0f.4
(XEN) PCI add device 0000:7f:0f.5
(XEN) PCI add device 0000:7f:10.0
(XEN) PCI add device 0000:7f:10.1
(XEN) PCI add device 0000:7f:10.2
(XEN) PCI add device 0000:7f:10.3
(XEN) PCI add device 0000:7f:10.4
(XEN) PCI add device 0000:7f:10.5
(XEN) PCI add device 0000:7f:10.6
(XEN) PCI add device 0000:7f:10.7
(XEN) PCI add device 0000:7f:13.0
(XEN) PCI add device 0000:7f:13.1
(XEN) PCI add device 0000:7f:13.4
(XEN) PCI add device 0000:7f:13.5
(XEN) PCI add device 0000:7f:16.0
(XEN) PCI add device 0000:7f:16.1
(XEN) PCI add device 0000:7f:16.2
(XEN) PCI add device 0000:80:01.0
(XEN) PCI add device 0000:80:02.0
(XEN) PCI add device 0000:80:03.0
(XEN) PCI add device 0000:80:04.0
(XEN) PCI add device 0000:80:04.1
(XEN) PCI add device 0000:80:04.2
(XEN) PCI add device 0000:80:04.3
(XEN) PCI add device 0000:80:04.4
(XEN) PCI add device 0000:80:04.5
(XEN) PCI add device 0000:80:04.6
(XEN) PCI add device 0000:80:04.7
(XEN) PCI add device 0000:80:05.0
(XEN) PCI add device 0000:80:05.2
(XEN) PCI add device 0000:80:05.4
(XEN) PCI add device 0000:ff:08.0
(XEN) PCI add device 0000:ff:09.0
(XEN) PCI add device 0000:ff:0a.0
(XEN) PCI add device 0000:ff:0a.1
(XEN) PCI add device 0000:ff:0a.2
(XEN) PCI add device 0000:ff:0a.3
(XEN) PCI add device 0000:ff:0b.0
(XEN) PCI add device 0000:ff:0b.3
(XEN) PCI add device 0000:ff:0c.0
(XEN) PCI add device 0000:ff:0c.1
(XEN) PCI add device 0000:ff:0c.2
(XEN) PCI add device 0000:ff:0d.0
(XEN) PCI add device 0000:ff:0d.1
(XEN) PCI add device 0000:ff:0d.2
(XEN) PCI add device 0000:ff:0e.0
(XEN) PCI add device 0000:ff:0e.1
(XEN) PCI add device 0000:ff:0f.0
(XEN) PCI add device 0000:ff:0f.1
(XEN) PCI add device 0000:ff:0f.2
(XEN) PCI add device 0000:ff:0f.3
(XEN) PCI add device 0000:ff:0f.4
(XEN) PCI add device 0000:ff:0f.5
(XEN) PCI add device 0000:ff:10.0
(XEN) PCI add device 0000:ff:10.1
(XEN) PCI add device 0000:ff:10.2
(XEN) PCI add device 0000:ff:10.3
(XEN) PCI add device 0000:ff:10.4
(XEN) PCI add device 0000:ff:10.5
(XEN) PCI add device 0000:ff:10.6
(XEN) PCI add device 0000:ff:10.7
(XEN) PCI add device 0000:ff:13.0
(XEN) PCI add device 0000:ff:13.1
(XEN) PCI add device 0000:ff:13.4
(XEN) PCI add device 0000:ff:13.5
(XEN) PCI add device 0000:ff:16.0
(XEN) PCI add device 0000:ff:16.1
(XEN) PCI add device 0000:ff:16.2
Login prompt at this stage.
--
Steven Haigh
? netwiz@crc.id.au ? http://www.crc.id.au
? +61 (3) 9001 6090 ? 0412 935 897
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-30 4:01 ` Steven Haigh
@ 2018-08-30 7:13 ` Juergen Gross
2018-08-30 8:33 ` Jan Beulich
2018-09-12 15:13 ` RFE: Detect NUMA misconfigurations and prevent machine freezes Dario Faggioli
2 siblings, 0 replies; 11+ messages in thread
From: Juergen Gross @ 2018-08-30 7:13 UTC (permalink / raw)
To: Steven Haigh, Dario Faggioli; +Cc: xen-devel
Added Dario as the scheduler's maintainer (he is on vacation this and
the next week, so don't expect a fast answer).
Juergen
On 30/08/18 06:01, Steven Haigh wrote:
> On 2018-08-29 15:49, Juergen Gross wrote:
>> On 29/08/18 07:33, Steven Haigh wrote:
>>> When playing with NUMA support recently, I noticed a host would
>>> always hang
>>> when trying to create a cpupool for the second NUMA node in the system.
>>>
>>> I was using the following commands:
>>> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
>>> # xl cpupool-cpu-remove Pool-0 node:1
>>> # xl cpupool-cpu-add Pool-1 node:1
>>>
>>> After the last command, the system would hang - requiring a hard
>>> reset of the
>>> machine to fix.
>>>
>>> I tried a different variation with the same result:
>>> # xl cpupool-create name=\"Pool-1\" sched=\"credit2\
>>> # xl cpupool-cpu-remove Pool-0 node:1
>>> # xl cpupool-cpu-add Pool-1 12
>>>
>>> It turns out that the RAM was installed sub-optimally in this machine. A
>>> partial output from 'xl info -n' shows:
>>> numa_info :
>>> node: memsize memfree distances
>>> 0: 67584 62608 10,21
>>> 1: 0 0 21,10
>>>
>>> A machine where we could get this working every time shows:
>>> node: memsize memfree distances
>>> 0: 34816 30483 10,21
>>> 1: 32768 32125 21,10
>>>
>>> As we can deduce RAM misconfigurations in this scenario, I believe we
>>> should
>>> check to ensure that RAM configuration / layout is sane *before*
>>> attempting to
>>> split the system and print a warning.
>>>
>>> This would prevent a hard system freeze in this scenario.
>>
>> RAM placement should not matter here. As the name already suggests
>> cpupools do assignment of cpus. RAM allocated will be preferred taken
>> from a local node, but this shouldn't be mandatory for success.
>>
>> Would it be possible to use a debug hypervisor (e.g. 4.12-unstable) for
>> generating a verbose log (hypervisor boot parameter "loglvl=all") and
>> sending the complete hypervisor log?
>
> I don't have a package for 4.11 or 4.12 built at all - but I did this on
> 4.10.2-pre (built from staging-4.10).
>
> Managed to get the same crash log when adding CPUs to Pool-1 as follows:
>
> Create the pool:
> (XEN) Initializing Credit2 scheduler
> (XEN) load_precision_shift: 18
> (XEN) load_window_shift: 30
> (XEN) underload_balance_tolerance: 0
> (XEN) overload_balance_tolerance: -3
> (XEN) runqueues arrangement: socket
> (XEN) cap enforcement granularity: 10ms
> (XEN) load tracking window length 1073741824 ns
>
> Add the CPUs:
> (XEN) Adding cpu 12 to runqueue 0
> (XEN) First cpu on runqueue, activating
> (XEN) Removing cpu 12 from runqueue 0
> (XEN) Adding cpu 13 to runqueue 0
> (XEN) Removing cpu 13 from runqueue 0
> (XEN) Adding cpu 14 to runqueue 0
> (XEN) Removing cpu 14 from runqueue 0
> (XEN) Xen BUG at sched_credit2.c:3452
> (XEN) Adding cpu 15 to runqueue 0
> (XEN) ----[ Xen-4.10.2-pre x86_64 debug=n Not tainted ]----
> (XEN) CPU: 13
> (XEN) RIP: e008:[<ffff82d08022fe40>]
> sched_credit2.c#csched2_schedule+0xf60/0x1330
> (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor
> (XEN) rax: 0000000000000000 rbx: ffff82d080562500 rcx: fffffffffffffffd
> (XEN) rdx: ffff831040ae1f80 rsi: 000000000000000d rdi: ffff831050abfe38
> (XEN) rbp: ffff831044ba7f30 rsp: ffff831050abfd20 r8: ffff831050ac61a0
> (XEN) r9: ffff82d08022eee0 r10: 0000000000000001 r11: ffff83007dc11060
> (XEN) r12: 000000000000000d r13: ffff831050aefec0 r14: ffff831050ac6180
> (XEN) r15: ffff82d080562500 cr0: 000000008005003b cr4: 00000000001526e0
> (XEN) cr3: 000000007dc32000 cr2: 00007fff9cb47838
> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> (XEN) Xen code around <ffff82d08022fe40>
> (sched_credit2.c#csched2_schedule+0xf60/0x1330):
> (XEN) ff ff 0f 0b 0f 1f 40 00 <0f> 0b 0f 0b 4c 89 ef e8 d4 b6 ff ff e9
> 87 f1 ff
> (XEN) Xen stack trace from rsp=ffff831050abfd20:
> (XEN) ffff83107f5f6010 ffff831050abfe38 0000002dd914464a
> 000000000000000d
> (XEN) 0000000002f8d264 ffff831000000000 0000000000000000
> ffff8310419efdd0
> (XEN) ffff831050ac6400 ffff82d0802344fa 0000000000000000
> 0000001bc97d2ea7
> (XEN) 0000002d9f005422 ffff831050ac63c0 ffff82d080562500
> ffff82d080577280
> (XEN) 000000000000000d 0000000000000046 0000000000000282
> 0000000000000082
> (XEN) 000000000000000d ffff831050ac61c8 ffff82d080576fac
> 000000000000000d
> (XEN) ffff83007dd7f000 ffff831044ba8000 0000002dd914464a
> ffff831050ac6180
> (XEN) ffff82d080562500 ffff82d080233eed 0000002d00000000
> ffff831050ac61a0
> (XEN) ffff831050ab6010 ffff82d080267cf5 ffff82d0802bffe0
> ffff82d0802c4505
> (XEN) 0000002dd61b7fed ffff82d08023ae48 000038000000380b
> 00000000ffffffff
> (XEN) ffffffffffffffff ffff831050abffff ffff82d08054bc80
> ffff82d080562500
> (XEN) ffff831050abffff ffff82d0802375b2 ffff82d0805771f0
> ffff82d08054c300
> (XEN) ffff82d0805771f0 000000000000000d 000000000000000d
> ffff82d08026d2d5
> (XEN) ffff83007dd7f000 ffff83007dd7f000 ffff83007dbae000
> ffff83107cd38000
> (XEN) 00000000ffffffff ffff83107fb0d000 ffff82d080562500
> 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000
> ffff880736083fd8
> (XEN) ffff880736083fd4 2e6b180200000001 0000000000000179
> 0000000000000000
> (XEN) 00000000000000ff 0000000000000179 000000fa00000000
> ffffffff81030f66
> (XEN) 0000000000000000 0000000000000046 ffff880736083fc0
> 0000000000000000
> (XEN) Xen call trace:
> (XEN) [<ffff82d08022fe40>] sched_credit2.c#csched2_schedule+0xf60/0x1330
> (XEN) [<ffff82d0802344fa>] get_cpu_idle_time+0x3a/0x50
> (XEN) [<ffff82d080233eed>] schedule.c#schedule+0xdd/0x5d0
> (XEN) [<ffff82d080267cf5>] reprogram_timer+0x75/0xe0
> (XEN) [<ffff82d0802bffe0>] cpuidle_menu.c#menu_reflect+0/0x80
> (XEN) [<ffff82d0802c4505>] mwait-idle.c#mwait_idle+0x275/0x3d0
> (XEN) [<ffff82d08023ae48>] timer.c#timer_softirq_action+0x138/0x210
> (XEN) [<ffff82d0802375b2>] softirq.c#__do_softirq+0x62/0x90
> (XEN) [<ffff82d08026d2d5>] domain.c#idle_loop+0x45/0xb0
> (XEN)
> (XEN) Removing cpu 15 from runqueue 0
> (XEN)
> (XEN) ****************************************
> (XEN) Adding cpu 16 to runqueue 0
> (XEN) Panic on CPU 13:
> (XEN) Removing cpu 16 from runqueue 0
> (XEN) Xen BUG at sched_credit2.c:3452
> (XEN) Adding cpu 17 to runqueue 0
> (XEN) ****************************************
> (XEN)
> (XEN) Removing cpu 17 from runqueue 0
> (XEN) Reboot in five seconds...
> (XEN) Adding cpu 18 to runqueue 0
> (XEN) Removing cpu 18 from runqueue 0
>
> Only thing I could do at this point was to hit the reset button.
>
> Full boot logs (if helpful):
> (XEN) Xen version 4.10.2-pre (mockbuild@) (gcc (GCC) 4.8.5 20150623 (Red
> Hat 4.8.5-28)) debug=n Fri Aug 24 15:13:31 AEST 2018
> (XEN) Latest ChangeSet: Tue Aug 14 15:13:09 2018 -0700 git:bf1b7f5-dirty
> (XEN) Bootloader: GRUB 2.02~beta2
> (XEN) Command line: placeholder dom0_mem=2048M,max:2048M ucode=scan
> dom0_max_vcpus=2 sched=credit2 com2=115200,8n1 console=com2 loglvl=all
> (XEN) Xen image load base address: 0
> (XEN) Video information:
> (XEN) VGA is text mode 80x25, font 8x16
> (XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN) EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN) Found 2 MBR signatures
> (XEN) Found 2 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN) 0000000000000000 - 000000000009ac00 (usable)
> (XEN) 000000000009ac00 - 00000000000a0000 (reserved)
> (XEN) 00000000000e0000 - 0000000000100000 (reserved)
> (XEN) 0000000000100000 - 000000007df1e000 (usable)
> (XEN) 000000007df1e000 - 000000007e0d4000 (reserved)
> (XEN) 000000007e0d4000 - 000000007e2dc000 (ACPI NVS)
> (XEN) 000000007e2dc000 - 000000007f358000 (reserved)
> (XEN) 000000007f358000 - 000000007f800000 (ACPI NVS)
> (XEN) 0000000080000000 - 0000000090000000 (reserved)
> (XEN) 00000000fed1c000 - 00000000fed40000 (reserved)
> (XEN) 00000000ff000000 - 0000000100000000 (reserved)
> (XEN) 0000000100000000 - 0000001080000000 (usable)
> (XEN) New Xen image base address: 0x7d800000
> (XEN) ACPI: RSDP 000F04A0, 0024 (r2 ALASKA)
> (XEN) ACPI: XSDT 7E1F4090, 009C (r1 ALASKA A M I 1072009 AMI 10013)
> (XEN) ACPI: FACP 7E201048, 010C (r5 ALASKA A M I 1072009 AMI 10013)
> (XEN) ACPI: DSDT 7E1F41C0, CE87 (r2 ALASKA A M I 15 INTL 20051117)
> (XEN) ACPI: FACS 7E2D3080, 0040
> (XEN) ACPI: APIC 7E201158, 01B4 (r3 ALASKA A M I 1072009 AMI 10013)
> (XEN) ACPI: FPDT 7E201310, 0044 (r1 ALASKA A M I 1072009 AMI 10013)
> (XEN) ACPI: MCFG 7E201358, 003C (r1 ALASKA OEMMCFG. 1072009 MSFT 97)
> (XEN) ACPI: SRAT 7E201398, 0430 (r1 A M I AMI SRAT 1 AMI. 0)
> (XEN) ACPI: SLIT 7E2017C8, 0030 (r1 A M I AMI SLIT 0 AMI. 0)
> (XEN) ACPI: HPET 7E2017F8, 0038 (r1 ALASKA A M I 1072009 AMI. 5)
> (XEN) ACPI: PRAD 7E201830, 00BE (r2 PRADID PRADTID 1 MSFT 3000001)
> (XEN) ACPI: SPMI 7E2018F0, 0040 (r5 A M I OEMSPMI 0 AMI. 0)
> (XEN) ACPI: SSDT 7E201930, D0CB0 (r2 INTEL CpuPm 4000 INTL
> 20051117)
> (XEN) ACPI: EINJ 7E2D25E0, 0130 (r1 AMI AMI EINJ 0 0)
> (XEN) ACPI: ERST 7E2D2710, 0230 (r1 AMIER AMI ERST 0 0)
> (XEN) ACPI: HEST 7E2D2940, 00A8 (r1 AMI AMI HEST 0 0)
> (XEN) ACPI: BERT 7E2D29E8, 0030 (r1 AMI AMI BERT 0 0)
> (XEN) ACPI: DMAR 7E2D2A18, 0160 (r1 A M I OEMDMAR 1 INTL 1)
> (XEN) System RAM: 65502MB (67074784kB)
> (XEN) SRAT: PXM 0 -> APIC 00 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 01 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 02 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 03 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 04 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 05 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 06 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 07 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 08 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 09 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 0a -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 0b -> Node 0
> (XEN) SRAT: PXM 1 -> APIC 20 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 21 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 22 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 23 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 24 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 25 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 26 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 27 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 28 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 29 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 2a -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 2b -> Node 1
> (XEN) SRAT: Node 0 PXM 0 0-80000000
> (XEN) SRAT: Node 0 PXM 0 100000000-1080000000
> (XEN) NUMA: Using 20 for the hash shift.
> (XEN) SRAT: Node 1 has no memory. BIOS Bug or mis-configured hardware?
> (XEN) Domain heap initialised DMA width 32 bits
> (XEN) CPU Vendor: Intel, Family 6 (0x6), Model 62 (0x3e), Stepping 4
> (raw 000306e4)
> (XEN) found SMP MP-table at 000fdb60
> (XEN) DMI 2.7 present.
> (XEN) Using APIC driver default
> (XEN) ACPI: PM-Timer IO Port: 0x408 (32 bits)
> (XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
> (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
> (XEN) ACPI: 32/64X FACS address mismatch in FADT -
> 7e2d3080/0000000000000000, using 32
> (XEN) ACPI: wakeup_vec[7e2d308c], vec_size[20]
> (XEN) ACPI: Local APIC address 0xfee00000
> (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x20] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x22] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x24] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x28] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2a] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x21] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x23] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x25] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x27] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x29] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x2b] enabled)
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x11] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x13] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x15] high edge lint[0x1])
> (XEN) ACPI: LAPIC_NMI (acpi_id[0x17] high edge lint[0x1])
> (XEN) Overriding APIC driver with bigsmp
> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x02] address[0xfec01000] gsi_base[24])
> (XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
> (XEN) ACPI: IOAPIC (id[0x03] address[0xfec40000] gsi_base[48])
> (XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
> (XEN) Xen ERST support is initialized.
> (XEN) HEST: Table parsing has been initialized
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) SMP: Allowing 24 CPUs (0 hotplug CPUs)
> (XEN) IRQ limits: 72 GSI, 4552 MSI/MSI-X
> (XEN) Switched to APIC driver x2apic_cluster.
> (XEN) xstate: size: 0x340 and states: 0x7
> (XEN) CPU0: Intel machine check reporting enabled
> (XEN) Speculative mitigation facilities:
> (XEN) Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
> (XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
> (XEN) Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- SSBD-,
> Other: IBPB L1D_FLUSH
> (XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe
> address 300000000000
> (XEN) Support for VMs: PV: MSR_SPEC_CTRL RSB EAGER_FPU, HVM:
> MSR_SPEC_CTRL RSB EAGER_FPU
> (XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled
> (XEN) PV L1TF shadowing: Dom0 disabled, DomU enabled
> (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
> (XEN) Initializing Credit2 scheduler
> (XEN) load_precision_shift: 18
> (XEN) load_window_shift: 30
> (XEN) underload_balance_tolerance: 0
> (XEN) overload_balance_tolerance: -3
> (XEN) runqueues arrangement: socket
> (XEN) cap enforcement granularity: 10ms
> (XEN) load tracking window length 1073741824 ns
> (XEN) Adding cpu 0 to runqueue 0
> (XEN) First cpu on runqueue, activating
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Detected 2600.019 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) alt table ffff82d080419778 -> ffff82d08041b260
> (XEN) PCI: MCFG configuration 0: base 80000000 segment 0000 buses 00 - ff
> (XEN) PCI: MCFG area at 80000000 reserved in E820
> (XEN) PCI: Using MCFG for segment 0000 bus 00-ff
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d Snoop Control enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Posted Interrupt not enabled.
> (XEN) Intel VT-d Shared EPT tables enabled.
> (XEN) I/O virtualisation enabled
> (XEN) - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
> (XEN) nr_sockets: 2
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN) -> Using old ACK method
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) TSC deadline timer enabled
> (XEN) Defaulting to alternative key handling; send 'A' to switch to
> normal mode.
> (XEN) Allocated console ring of 256 KiB.
> (XEN) mwait-idle: MWAIT substates: 0x1120
> (XEN) mwait-idle: v0.4.1 model 0x3e
> (XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
> (XEN) VMX: Supported advanced features:
> (XEN) - APIC MMIO access virtualisation
> (XEN) - APIC TPR shadow
> (XEN) - Extended Page Tables (EPT)
> (XEN) - Virtual-Processor Identifiers (VPID)
> (XEN) - Virtual NMI
> (XEN) - MSR direct-access bitmap
> (XEN) - Unrestricted Guest
> (XEN) - APIC Register Virtualization
> (XEN) - Virtual Interrupt Delivery
> (XEN) - Posted Interrupt Processing
> (XEN) HVM: ASIDs enabled.
> (XEN) HVM: VMX enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
> (XEN) Adding cpu 1 to runqueue 0
> (XEN) Adding cpu 2 to runqueue 0
> (XEN) Adding cpu 3 to runqueue 0
> (XEN) Adding cpu 4 to runqueue 0
> (XEN) Adding cpu 5 to runqueue 0
> (XEN) Adding cpu 6 to runqueue 0
> (XEN) Adding cpu 7 to runqueue 0
> (XEN) Adding cpu 8 to runqueue 0
> (XEN) Adding cpu 9 to runqueue 0
> (XEN) Adding cpu 10 to runqueue 0
> (XEN) Adding cpu 11 to runqueue 0
> (XEN) Adding cpu 12 to runqueue 1
> (XEN) First cpu on runqueue, activating
> (XEN) Adding cpu 13 to runqueue 1
> (XEN) Adding cpu 14 to runqueue 1
> (XEN) Adding cpu 15 to runqueue 1
> (XEN) Adding cpu 16 to runqueue 1
> (XEN) Adding cpu 17 to runqueue 1
> (XEN) Adding cpu 18 to runqueue 1
> (XEN) Adding cpu 19 to runqueue 1
> (XEN) Adding cpu 20 to runqueue 1
> (XEN) Adding cpu 21 to runqueue 1
> (XEN) Adding cpu 22 to runqueue 1
> (XEN) Adding cpu 23 to runqueue 1
> (XEN) Brought up 24 CPUs
> (XEN) build-id: 1e7674bde0383cfdea16d2d4f65364172565de0a
> (XEN) ACPI sleep modes: S3
> (XEN) VPMU: disabled
> (XEN) mcheck_poll: Machine check polling timer started.
> (XEN) Dom0 has maximum 456 PIRQs
> (XEN) NX (Execute Disable) protection active
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Xen kernel: 64-bit, lsb, compat32
> (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2768000
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN) Dom0 alloc.: 000000104c000000->0000001050000000 (503266 pages
> to be allocated)
> (XEN) Init. ramdisk: 000000107ede2000->000000107ffff6d8
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN) Loaded kernel: ffffffff81000000->ffffffff82768000
> (XEN) Init. ramdisk: 0000000000000000->0000000000000000
> (XEN) Phys-Mach map: 0000008000000000->0000008000400000
> (XEN) Start info: ffffffff82768000->ffffffff827684b4
> (XEN) Xenstore ring: 0000000000000000->0000000000000000
> (XEN) Console ring: 0000000000000000->0000000000000000
> (XEN) Page tables: ffffffff82769000->ffffffff82782000
> (XEN) Boot stack: ffffffff82782000->ffffffff82783000
> (XEN) TOTAL: ffffffff80000000->ffffffff82c00000
> (XEN) ENTRY ADDRESS: ffffffff82452180
> (XEN) Dom0 has maximum 2 VCPUs
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Scrubbing Free RAM on 2 nodes using 6 CPUs
> (XEN)
> ........................................................................................done.
>
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) ***************************************************
> (XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
> (XEN) enabled. Please assess your configuration and choose an
> (XEN) explicit 'smt=<bool>' setting. See XSA-273.
> (XEN) 3... 2... 1...
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
> input to Xen)
> (XEN) Freed 472kB init memory
> mapping kernel into physical memory
> about to get started...
> (XEN) PCI add device 0000:00:00.0
> (XEN) PCI add device 0000:00:01.0
> (XEN) PCI add device 0000:00:01.1
> (XEN) PCI add device 0000:00:02.0
> (XEN) PCI add device 0000:00:03.0
> (XEN) PCI add device 0000:00:03.2
> (XEN) PCI add device 0000:00:04.0
> (XEN) PCI add device 0000:00:04.1
> (XEN) PCI add device 0000:00:04.2
> (XEN) PCI add device 0000:00:04.3
> (XEN) PCI add device 0000:00:04.4
> (XEN) PCI add device 0000:00:04.5
> (XEN) PCI add device 0000:00:04.6
> (XEN) PCI add device 0000:00:04.7
> (XEN) PCI add device 0000:00:05.0
> (XEN) PCI add device 0000:00:05.2
> (XEN) PCI add device 0000:00:05.4
> (XEN) PCI add device 0000:00:11.0
> (XEN) PCI add device 0000:00:16.0
> (XEN) PCI add device 0000:00:16.1
> (XEN) PCI add device 0000:00:1a.0
> (XEN) PCI add device 0000:00:1d.0
> (XEN) PCI add device 0000:00:1e.0
> (XEN) PCI add device 0000:00:1f.0
> (XEN) PCI add device 0000:00:1f.2
> (XEN) PCI add device 0000:00:1f.3
> (XEN) PCI add device 0000:00:1f.6
> (XEN) PCI add device 0000:02:00.0
> (XEN) PCI add device 0000:02:00.1
> (XEN) PCI add device 0000:07:00.0
> (XEN) PCI add device 0000:08:03.0
> (XEN) PCI add device 0000:7f:08.0
> (XEN) PCI add device 0000:7f:09.0
> (XEN) PCI add device 0000:7f:0a.0
> (XEN) PCI add device 0000:7f:0a.1
> (XEN) PCI add device 0000:7f:0a.2
> (XEN) PCI add device 0000:7f:0a.3
> (XEN) PCI add device 0000:7f:0b.0
> (XEN) PCI add device 0000:7f:0b.3
> (XEN) PCI add device 0000:7f:0c.0
> (XEN) PCI add device 0000:7f:0c.1
> (XEN) PCI add device 0000:7f:0c.2
> (XEN) PCI add device 0000:7f:0d.0
> (XEN) PCI add device 0000:7f:0d.1
> (XEN) PCI add device 0000:7f:0d.2
> (XEN) PCI add device 0000:7f:0e.0
> (XEN) PCI add device 0000:7f:0e.1
> (XEN) PCI add device 0000:7f:0f.0
> (XEN) PCI add device 0000:7f:0f.1
> (XEN) PCI add device 0000:7f:0f.2
> (XEN) PCI add device 0000:7f:0f.3
> (XEN) PCI add device 0000:7f:0f.4
> (XEN) PCI add device 0000:7f:0f.5
> (XEN) PCI add device 0000:7f:10.0
> (XEN) PCI add device 0000:7f:10.1
> (XEN) PCI add device 0000:7f:10.2
> (XEN) PCI add device 0000:7f:10.3
> (XEN) PCI add device 0000:7f:10.4
> (XEN) PCI add device 0000:7f:10.5
> (XEN) PCI add device 0000:7f:10.6
> (XEN) PCI add device 0000:7f:10.7
> (XEN) PCI add device 0000:7f:13.0
> (XEN) PCI add device 0000:7f:13.1
> (XEN) PCI add device 0000:7f:13.4
> (XEN) PCI add device 0000:7f:13.5
> (XEN) PCI add device 0000:7f:16.0
> (XEN) PCI add device 0000:7f:16.1
> (XEN) PCI add device 0000:7f:16.2
> (XEN) PCI add device 0000:80:01.0
> (XEN) PCI add device 0000:80:02.0
> (XEN) PCI add device 0000:80:03.0
> (XEN) PCI add device 0000:80:04.0
> (XEN) PCI add device 0000:80:04.1
> (XEN) PCI add device 0000:80:04.2
> (XEN) PCI add device 0000:80:04.3
> (XEN) PCI add device 0000:80:04.4
> (XEN) PCI add device 0000:80:04.5
> (XEN) PCI add device 0000:80:04.6
> (XEN) PCI add device 0000:80:04.7
> (XEN) PCI add device 0000:80:05.0
> (XEN) PCI add device 0000:80:05.2
> (XEN) PCI add device 0000:80:05.4
> (XEN) PCI add device 0000:ff:08.0
> (XEN) PCI add device 0000:ff:09.0
> (XEN) PCI add device 0000:ff:0a.0
> (XEN) PCI add device 0000:ff:0a.1
> (XEN) PCI add device 0000:ff:0a.2
> (XEN) PCI add device 0000:ff:0a.3
> (XEN) PCI add device 0000:ff:0b.0
> (XEN) PCI add device 0000:ff:0b.3
> (XEN) PCI add device 0000:ff:0c.0
> (XEN) PCI add device 0000:ff:0c.1
> (XEN) PCI add device 0000:ff:0c.2
> (XEN) PCI add device 0000:ff:0d.0
> (XEN) PCI add device 0000:ff:0d.1
> (XEN) PCI add device 0000:ff:0d.2
> (XEN) PCI add device 0000:ff:0e.0
> (XEN) PCI add device 0000:ff:0e.1
> (XEN) PCI add device 0000:ff:0f.0
> (XEN) PCI add device 0000:ff:0f.1
> (XEN) PCI add device 0000:ff:0f.2
> (XEN) PCI add device 0000:ff:0f.3
> (XEN) PCI add device 0000:ff:0f.4
> (XEN) PCI add device 0000:ff:0f.5
> (XEN) PCI add device 0000:ff:10.0
> (XEN) PCI add device 0000:ff:10.1
> (XEN) PCI add device 0000:ff:10.2
> (XEN) PCI add device 0000:ff:10.3
> (XEN) PCI add device 0000:ff:10.4
> (XEN) PCI add device 0000:ff:10.5
> (XEN) PCI add device 0000:ff:10.6
> (XEN) PCI add device 0000:ff:10.7
> (XEN) PCI add device 0000:ff:13.0
> (XEN) PCI add device 0000:ff:13.1
> (XEN) PCI add device 0000:ff:13.4
> (XEN) PCI add device 0000:ff:13.5
> (XEN) PCI add device 0000:ff:16.0
> (XEN) PCI add device 0000:ff:16.1
> (XEN) PCI add device 0000:ff:16.2
>
> Login prompt at this stage.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-30 4:01 ` Steven Haigh
2018-08-30 7:13 ` Juergen Gross
@ 2018-08-30 8:33 ` Jan Beulich
2018-08-30 8:49 ` BUG: sched=credit2 crashes system when using cpupools Steven Haigh
2018-09-12 15:13 ` RFE: Detect NUMA misconfigurations and prevent machine freezes Dario Faggioli
2 siblings, 1 reply; 11+ messages in thread
From: Jan Beulich @ 2018-08-30 8:33 UTC (permalink / raw)
To: Steven Haigh; +Cc: George Dunlap, xen-devel, Juergen Gross, Dario Faggioli
>>> On 30.08.18 at 06:01, <netwiz@crc.id.au> wrote:
> Managed to get the same crash log when adding CPUs to Pool-1 as follows:
>
> Create the pool:
> (XEN) Initializing Credit2 scheduler
> (XEN) load_precision_shift: 18
> (XEN) load_window_shift: 30
> (XEN) underload_balance_tolerance: 0
> (XEN) overload_balance_tolerance: -3
> (XEN) runqueues arrangement: socket
> (XEN) cap enforcement granularity: 10ms
> (XEN) load tracking window length 1073741824 ns
>
> Add the CPUs:
> (XEN) Adding cpu 12 to runqueue 0
> (XEN) First cpu on runqueue, activating
> (XEN) Removing cpu 12 from runqueue 0
> (XEN) Adding cpu 13 to runqueue 0
> (XEN) Removing cpu 13 from runqueue 0
> (XEN) Adding cpu 14 to runqueue 0
> (XEN) Removing cpu 14 from runqueue 0
> (XEN) Xen BUG at sched_credit2.c:3452
credit2 still not being the default - do things work if you don't override
the default (of using credit1)? I guess the problem is connected to the
"Removing cpu <N> from runqueue 0", considering this
BUG_ON(!cpumask_test_cpu(cpu, &rqd->active));
is what triggers. Anyway - as Jürgen says, something for the scheduler
maintainers to look into.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: sched=credit2 crashes system when using cpupools
2018-08-30 8:33 ` Jan Beulich
@ 2018-08-30 8:49 ` Steven Haigh
2018-09-12 15:11 ` Dario Faggioli
0 siblings, 1 reply; 11+ messages in thread
From: Steven Haigh @ 2018-08-30 8:49 UTC (permalink / raw)
To: Jan Beulich; +Cc: George Dunlap, xen-devel, Juergen Gross, Dario Faggioli
On 2018-08-30 18:33, Jan Beulich wrote:
>>>> On 30.08.18 at 06:01, <netwiz@crc.id.au> wrote:
>> Managed to get the same crash log when adding CPUs to Pool-1 as
>> follows:
>>
>> Create the pool:
>> (XEN) Initializing Credit2 scheduler
>> (XEN) load_precision_shift: 18
>> (XEN) load_window_shift: 30
>> (XEN) underload_balance_tolerance: 0
>> (XEN) overload_balance_tolerance: -3
>> (XEN) runqueues arrangement: socket
>> (XEN) cap enforcement granularity: 10ms
>> (XEN) load tracking window length 1073741824 ns
>>
>> Add the CPUs:
>> (XEN) Adding cpu 12 to runqueue 0
>> (XEN) First cpu on runqueue, activating
>> (XEN) Removing cpu 12 from runqueue 0
>> (XEN) Adding cpu 13 to runqueue 0
>> (XEN) Removing cpu 13 from runqueue 0
>> (XEN) Adding cpu 14 to runqueue 0
>> (XEN) Removing cpu 14 from runqueue 0
>> (XEN) Xen BUG at sched_credit2.c:3452
>
> credit2 still not being the default - do things work if you don't
> override
> the default (of using credit1)? I guess the problem is connected to the
> "Removing cpu <N> from runqueue 0", considering this
>
> BUG_ON(!cpumask_test_cpu(cpu, &rqd->active));
>
> is what triggers. Anyway - as Jürgen says, something for the scheduler
> maintainers to look into.
Yep - I just want to confirm that we tested this in BOTH NUMA
configurations - and credit2 crashed on both.
I switched back to sched=credit, and it seems to work as expected:
# xl cpupool-list
Name CPUs Sched Active Domain count
Pool-node0 12 credit y 3
Pool-node1 12 credit y 0
I've updated the subject - as this isn't a NUMA issue at all.
--
Steven Haigh
? netwiz@crc.id.au ? http://www.crc.id.au
? +61 (3) 9001 6090 ? 0412 935 897
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: sched=credit2 crashes system when using cpupools
2018-08-30 8:49 ` BUG: sched=credit2 crashes system when using cpupools Steven Haigh
@ 2018-09-12 15:11 ` Dario Faggioli
2018-09-12 15:27 ` Steven Haigh
0 siblings, 1 reply; 11+ messages in thread
From: Dario Faggioli @ 2018-09-12 15:11 UTC (permalink / raw)
To: Steven Haigh, Jan Beulich; +Cc: George Dunlap, xen-devel, Juergen Gross
[-- Attachment #1.1: Type: text/plain, Size: 1812 bytes --]
On Thu, 2018-08-30 at 18:49 +1000, Steven Haigh wrote:
> On 2018-08-30 18:33, Jan Beulich wrote:
> > > > >
> > Anyway - as Jürgen says, something for the scheduler
> > maintainers to look into.
>
Ok, I'm back.
> Yep - I just want to confirm that we tested this in BOTH NUMA
> configurations - and credit2 crashed on both.
>
> I switched back to sched=credit, and it seems to work as expected:
> # xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-node0 12 credit y 3
> Pool-node1 12 credit y 0
>
Wait, in a previous message, you said: "A machine where we could get
this working every time shows". Doesn't that mean creating a separate
pool for node 1 works with both Credit and Credit2, if the node has
memory?
I mean, trying to clarifying, my understanding is that you have to
systems:
system A: node 1 has *no* memory
system B: both node 0 and node 1 have memory
Creating a Credit pool with pcpus from node 1 always work on both
systems.
OTOH, when you try to create a Credit2 pool with pcpus from node 1,
does it always crash on both systems, or does it work on system B and
crashes on system A ?
I do have a NUMA box with RAM in both nodes (so similar to system B).
Last time I checked, what you're trying to do worked there, pretty much
with any scheduler combination, but I'll recheck.
I don't have a box similar to system A. I'll try to remove some of the
RAM from that NUMA box, and check what happens.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-08-30 4:01 ` Steven Haigh
2018-08-30 7:13 ` Juergen Gross
2018-08-30 8:33 ` Jan Beulich
@ 2018-09-12 15:13 ` Dario Faggioli
2018-09-13 11:49 ` Dario Faggioli
2 siblings, 1 reply; 11+ messages in thread
From: Dario Faggioli @ 2018-09-12 15:13 UTC (permalink / raw)
To: Steven Haigh, Juergen Gross; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1527 bytes --]
On Thu, 2018-08-30 at 14:01 +1000, Steven Haigh wrote:
> On 2018-08-29 15:49, Juergen Gross wrote:
> >
> > RAM placement should not matter here. As the name already suggests
> > cpupools do assignment of cpus. RAM allocated will be preferred
> > taken
> > from a local node, but this shouldn't be mandatory for success.
> >
> > Would it be possible to use a debug hypervisor (e.g. 4.12-unstable)
> > for
> > generating a verbose log (hypervisor boot parameter "loglvl=all")
> > and
> > sending the complete hypervisor log?
>
> I don't have a package for 4.11 or 4.12 built at all - but I did this
> on
> 4.10.2-pre (built from staging-4.10).
>
> [..]
>
> Add the CPUs:
> (XEN) Adding cpu 12 to runqueue 0
> (XEN) First cpu on runqueue, activating
> (XEN) Removing cpu 12 from runqueue 0
> (XEN) Adding cpu 13 to runqueue 0
> (XEN) Removing cpu 13 from runqueue 0
> (XEN) Adding cpu 14 to runqueue 0
> (XEN) Removing cpu 14 from runqueue 0
> (XEN) Xen BUG at sched_credit2.c:3452
> (XEN) Adding cpu 15 to runqueue 0
> (XEN) ----[ Xen-4.10.2-pre x86_64 debug=n Not tainted ]----
>
That's not a debug build, though.
It would be useful to test one such build (as suggested by Juergen), to
see if any of the existing ASSERTs triggers.
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: sched=credit2 crashes system when using cpupools
2018-09-12 15:11 ` Dario Faggioli
@ 2018-09-12 15:27 ` Steven Haigh
0 siblings, 0 replies; 11+ messages in thread
From: Steven Haigh @ 2018-09-12 15:27 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Juergen Gross, Jan Beulich, Dario Faggioli
[-- Attachment #1.1: Type: text/plain, Size: 2426 bytes --]
On Thursday, 13 September 2018 1:11:20 AM AEST Dario Faggioli wrote:
> On Thu, 2018-08-30 at 18:49 +1000, Steven Haigh wrote:
> > On 2018-08-30 18:33, Jan Beulich wrote:
> > > Anyway - as Jürgen says, something for the scheduler
> > > maintainers to look into.
>
> Ok, I'm back.
>
> > Yep - I just want to confirm that we tested this in BOTH NUMA
> > configurations - and credit2 crashed on both.
> >
> > I switched back to sched=credit, and it seems to work as expected:
> > # xl cpupool-list
> > Name CPUs Sched Active Domain count
> > Pool-node0 12 credit y 3
> > Pool-node1 12 credit y 0
Hi Dario,
I'll try to clarify below.
> Wait, in a previous message, you said: "A machine where we could get
> this working every time shows". Doesn't that mean creating a separate
> pool for node 1 works with both Credit and Credit2, if the node has
> memory?
>
> I mean, trying to clarifying, my understanding is that you have to
> systems:
>
> system A: node 1 has *no* memory
> system B: both node 0 and node 1 have memory
>
> Creating a Credit pool with pcpus from node 1 always work on both
> systems.
Correct. With the credit scheduler, the pool split worked correctly on both
systems.
> OTOH, when you try to create a Credit2 pool with pcpus from node 1,
> does it always crash on both systems, or does it work on system B and
> crashes on system A ?
Both systems crashed when using credit2. We originally thought this was due to
the different memory layout between the two systems. This bit turned out to
not matter as both systems crashed in the same way.
> I do have a NUMA box with RAM in both nodes (so similar to system B).
> Last time I checked, what you're trying to do worked there, pretty much
> with any scheduler combination, but I'll recheck.
>
> I don't have a box similar to system A. I'll try to remove some of the
> RAM from that NUMA box, and check what happens.
In theory, if you reproduce what we did, it should crash anyway. The RAM
layout shouldn't matter.
We changed the scheduler in the grub line as 'sched=credit2'. Then did the
split. I didn't try changing the Dom0 to boot with credit, but making the
pools credit2.
--
Steven Haigh
📧 netwiz@crc.id.au 💻 https://www.crc.id.au
📞 +61 (3) 9001 6090 📱 0412 935 897
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RFE: Detect NUMA misconfigurations and prevent machine freezes
2018-09-12 15:13 ` RFE: Detect NUMA misconfigurations and prevent machine freezes Dario Faggioli
@ 2018-09-13 11:49 ` Dario Faggioli
0 siblings, 0 replies; 11+ messages in thread
From: Dario Faggioli @ 2018-09-13 11:49 UTC (permalink / raw)
To: Steven Haigh, Juergen Gross; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1140 bytes --]
On Wed, 2018-09-12 at 17:13 +0200, Dario Faggioli wrote:
> On Thu, 2018-08-30 at 14:01 +1000, Steven Haigh wrote:
> >
> > Add the CPUs:
> > (XEN) Adding cpu 12 to runqueue 0
> > (XEN) First cpu on runqueue, activating
> > (XEN) Removing cpu 12 from runqueue 0
> > (XEN) Adding cpu 13 to runqueue 0
> > (XEN) Removing cpu 13 from runqueue 0
> > (XEN) Adding cpu 14 to runqueue 0
> > (XEN) Removing cpu 14 from runqueue 0
> > (XEN) Xen BUG at sched_credit2.c:3452
> > (XEN) Adding cpu 15 to runqueue 0
> > (XEN) ----[ Xen-4.10.2-pre x86_64 debug=n Not tainted ]----
> >
> That's not a debug build, though.
>
> It would be useful to test one such build (as suggested by Juergen),
> to
> see if any of the existing ASSERTs triggers.
>
Ok, ignore this.
I have identified the problem, and I'm in the process of sending a
patch (to which I'll Cc you).
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #2: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-09-13 11:49 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-29 5:33 RFE: Detect NUMA misconfigurations and prevent machine freezes Steven Haigh
2018-08-29 5:49 ` Juergen Gross
2018-08-30 4:01 ` Steven Haigh
2018-08-30 7:13 ` Juergen Gross
2018-08-30 8:33 ` Jan Beulich
2018-08-30 8:49 ` BUG: sched=credit2 crashes system when using cpupools Steven Haigh
2018-09-12 15:11 ` Dario Faggioli
2018-09-12 15:27 ` Steven Haigh
2018-09-12 15:13 ` RFE: Detect NUMA misconfigurations and prevent machine freezes Dario Faggioli
2018-09-13 11:49 ` Dario Faggioli
2018-08-29 6:39 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.