Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

Boot Hangs at: HVM: HAP page sizes: 4kB, 2MB or Adding cpu [INSERT 1-7] 
to runqueue 0

Log (preserved for 6 months at https://pastebin.com/BDPP7Pzk)

fs0:\efi\gentoo> man_xen.efiip startup.nsh, any other key to continue.
Xen 4.12.0-rc (c/s Mon Feb 25 13:06:22 2019 +0000 git:f393b82fe5-dirty) 
EFI loader
Using configuration file 'man_xen.cfg'
xen-4.12.0-rc.gz: 0x000000005ad2b000-0x000000005ae49640
0x0000:0x02:0x00.0x0: ROM: 0x8000 bytes at 0x7c8bc028
  Xen 4.12.0-rc
(XEN) [00000024693d1c40] Xen version 4.12.0-rc (root@[unknown]) (gcc 
(Gentoo 8.2.0-r6 p1.7) 8.2.0) debug=y  Wed Mar 20 22:35:59 PDT 2019
(XEN) [000000246afc47a8] Latest ChangeSet: Mon Feb 25 13:06:22 2019 
+0000 git:f393b82fe5-dirty
(XEN) [000000246c2cdd38] Console output is synchronous.
(XEN) [000000246ce239e0] Bootloader: EFI
(XEN) [000000246d6f0cc8] Command line: console=vga,com1 com1=115200,8n1 
console_timestamps console_to_ring conring_size=64 log_buf_len=16M 
loglvl=all guest_loglvl=all sync_console=true sched_debug iommu=verbose 
apic_verbosity=debug efi=no-rs reboot=kbd
(XEN) [00000024709d96d0] Xen image load base address: 0x5b000000
(XEN) [00000024716f7420] Video information:
(XEN) [000000247205f6d0]  VGA is graphics mode 800x600, 32 bpp
(XEN) [0000002472d0f8d8] Disc information:
(XEN) [00000024735d5510]  Found 0 MBR signatures
(XEN) [0000002474042280]  Found 1 EDD information structures
(XEN) [0000002474c89b48] EFI RAM map:
(XEN) [0000002475448878]  0000000000000000 - 00000000000a0000 (usable)
(XEN) [0000002476427298]  0000000000100000 - 000000007e16b000 (usable)
(XEN) [0000002477273790]  000000007e16b000 - 000000007eba2000 (reserved)
(XEN) [000000247811aa98]  000000007eba2000 - 000000007ed0d000 (usable)
(XEN) [0000002478fd4af0]  000000007ed0d000 - 000000007f28d000 (ACPI NVS)
(XEN) [0000002479e80a18]  000000007f28d000 - 000000007f648000 (reserved)
(XEN) [000000247adbe038]  000000007f648000 - 000000007f800000 (usable)
(XEN) [000000247bc044b8]  00000000e0000000 - 00000000e4000000 (reserved)
(XEN) [000000247cab3db8]  00000000fed01000 - 00000000fed04000 (reserved)
(XEN) [000000247d9ebeb8]  00000000fed08000 - 00000000fed09000 (reserved)
(XEN) [000000247e893070]  00000000fed0c000 - 00000000fed10000 (reserved)
(XEN) [000000247f7c1f60]  00000000fed1c000 - 00000000fed1d000 (reserved)
(XEN) [000000248066f790]  00000000fef00000 - 00000000ff000000 (reserved)
(XEN) [000000248151b790]  00000000ff800000 - 0000000100000000 (reserved)
(XEN) [0000002482453308]  0000000100000000 - 0000000ff0000000 (usable)
(XEN) [000000248366b500] ACPI: RSDP 7ED55000, 0024 (r2 ALASKA)
(XEN) [00000024843b69e8] ACPI: XSDT 7ED55098, 00B4 (r1 ALASKA   A M I   
1072009 AMI     10013)
(XEN) [00000024856c0830] ACPI: FACP 7ED58690, 010C (r5 ALASKA   A M I   
1072009 AMI     10013)
(XEN) [0000002486a2b500] ACPI: DSDT 7ED551E8, 34A1 (r2 ALASKA   A M I   
1072009 INTL 20061109)
(XEN) [0000002487db8898] ACPI: FACS 7F28AF80, 0040
(XEN) [0000002488871280] ACPI: FPDT 7ED587A0, 0044 (r1 ALASKA   A M I   
1072009 AMI     10013)
(XEN) [0000002489b7b248] ACPI: FIDT 7ED587E8, 009C (r1 ALASKA   A M I   
1072009 AMI     10013)
(XEN) [000000248af0ab18] ACPI: SPMI 7ED58888, 0040 (r5 ALASKA   A M 
I         0 AMI.        0)
(XEN) [000000248c214dc8] ACPI: MCFG 7ED588C8, 003C (r1 ALASKA    A M I  
1072009 MSFT       97)
(XEN) [000000248d5a7ce0] ACPI: WDAT 7ED58908, 01AC (r1 ALASKA    A M I  
1072009 MSFT    10013)
(XEN) [000000248e8b0190] ACPI: UEFI 7ED58AB8, 0042 
(r1                        0             0)
(XEN) [000000248fc428e0] ACPI: APIC 7ED58B00, 0098 (r3 INTEL 
TIANO           1 MSFT        0)
(XEN) [0000002490f49dd0] ACPI: BDAT 7ED58B98, 0030 
(r1                        0             0)
(XEN) [00000024922e04e0] ACPI: HPET 7ED58BC8, 0038 (r1 
INTEL                  1 MSFT  1000013)
(XEN) [00000024935e57b0] ACPI: SSDT 7ED58C00, 09F1 (r1  PmRef CpuPm     
3000 INTL 20061109)
(XEN) [0000002494a8cdd0] ACPI: SSDT 7ED595F8, 0259 (r1  PmRef 
Cpu0Tst     3000 INTL 20061109)
(XEN) [0000002495d95070] ACPI: SSDT 7ED59858, 0357 (r1  PmRef ApTst     
3000 INTL 20061109)
(XEN) [000000249711b280] ACPI: SPCR 7ED59BB0, 0050 (r1  A M I APTIO V  
1072009 AMI.        5)
(XEN) [000000249841f650] ACPI: HEST 7ED59C00, 00A8 (r1 INTEL AVOTON 
B        1 INTL        1)
(XEN) [00000024997b69a8] ACPI: BERT 7ED59CA8, 0030 (r1 INTEL AVOTON 
B        1 INTL        1)
(XEN) [000000249aabafd0] ACPI: ERST 7ED59CD8, 0230 (r1 INTEL AVOTON 
B        1 INTL        1)
(XEN) [000000249be4f688] ACPI: EINJ 7ED59F08, 0150 (r1 INTEL AVOTON 
B        1 INTL        1)
(XEN) [000000249da28708] System RAM: 63204MB (64721080kB)
(XEN) [00000024b155f680] No NUMA configuration found
(XEN) [00000024b21f32e0] Faking a node at 0000000000000000-0000000ff0000000
(XEN) [00000027a6d2aa00] Domain heap initialised
(XEN) [00000027a77e1e40] Allocated console ring of 64 KiB.
(XEN) [00000027dde7d060] vesafb: framebuffer at 0xde000000, mapped to 
0xffff82c000201000, using 1920k, total 1920k
(XEN) [00000027df602ed0] vesafb: mode is 800x600x32, linelength=3200, 
font 8x8
(XEN) [00000027e05e2670] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
(XEN) [00000027e1560f58] CPU Vendor: Intel, Family 6 (0x6), Model 77 
(0x4d), Stepping 8 (raw 000406d8)
(XEN) [00000027f9e790d8] SMBIOS 2.8 present.
(XEN) [00000027fa894e90] DMI 2.7 present.
(XEN) [00000027fb23d3a0] APIC boot state is 'xapic'
(XEN) [00000027fbdfa3b0] Using APIC driver default
(XEN) [00000027fca7c5e0] ACPI: PM-Timer IO Port: 0x408 (32 bits)
(XEN) [00000027fd92f510] ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) [00000027fe9f4720] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], 
pm1x_evt[1:400,1:0]
(XEN) [00000027ffd14f90] ACPI: 32/64X FACS address mismatch in FADT - 
7f28af80/0000000000000000, using 32
(XEN) [00000028014468c0] ACPI:             wakeup_vec[7f28af8c], 
vec_size[20]
(XEN) [0000002802603888] ACPI: Local APIC address 0xfee00000
(XEN) [0000002803474140] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [00000028045a86c8] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [000000280576d568] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [00000028068a4d18] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [0000002807a64b90] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x08] enabled)
(XEN) [0000002808bab340] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
(XEN) [0000002809ddc660] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled)
(XEN) [000000280af14098] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x0e] enabled)
(XEN) [000000280c16f028] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) [000000280d2eccf0] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) [000000280e685f98] ACPI: IOAPIC (id[0x02] address[0xfec00000] 
gsi_base[0])
(XEN) [000000280f8e7c48] IOAPIC[0]: apic_id 2, version 32, address 
0xfec00000, GSI 0-23
(XEN) [0000002810d8df48] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 
dfl dfl)
(XEN) [000000281202a4b8] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 
high level)
(XEN) [0000002813501280] ACPI: IRQ0 used by override.
(XEN) [000000281413eab0] ACPI: IRQ2 used by override.
(XEN) [0000002814d7c6a0] ACPI: IRQ9 used by override.
(XEN) [0000002815a8a748] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [0000002816aaf428] ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [0000002817a95510] PCI: MCFG configuration 0: base e0000000 
segment 0000 buses 00 - ff
(XEN) [0000002818fc7eb0] PCI: MCFG area at e0000000 reserved in E820
(XEN) [0000002819fe6848] PCI: updated MCFG configuration 0: base 
e0000000 segment 0 buses 0 - 63
(XEN) [000000281b6072f8] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [000000281c62ea80] Xen ERST support is initialized.
(XEN) [000000281d364ad0] HEST: Table parsing has been initialized
(XEN) [000000281e25e9c8] Using ACPI (MADT) for SMP configuration information
(XEN) [000000281f45cca0] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [00000028202c5590] mapped APIC to ffff82cfffffb000 (fee00000)
(XEN) [000000282129d038] mapped IOAPIC to ffff82cfffffa000 (fec00000)
(XEN) [0000002822279b38] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [0000002823142ec0] mce_intel.c:780: MCA Capability: firstbank 0, 
extended MCE MSR 0, BCAST
(XEN) [0000002824749450] CPU0: Intel machine check reporting enabled
(XEN) [00000028257a81f8] Unrecognised CPU model 0x4d - assuming not 
reptpoline safe
(XEN) [0000002826ac5fa8] Speculative mitigation facilities:
(XEN) [0000002827944db0]   Hardware features:
(XEN) [0000002828399640]   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) [0000002829518be0]   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: 
No, Other:
(XEN) [000000282a8c0528]   Support for HVM VMs: RSB
(XEN) [000000282b560278]   Support for PV VMs: RSB
(XEN) [000000282c1cdb08]   XPTI (64-bit PV only): Dom0 enabled, DomU 
enabled (without PCID)
(XEN) [000000282d6bfc48]   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) [000000282e866c38] Using scheduler: SMP Credit Scheduler rev2 
(credit2)
(XEN) [000000282fa0da00] Initializing Credit2 scheduler
(XEN) [00000028306bd4d0]  load_precision_shift: 18
(XEN) [00000028312c5ed0]  load_window_shift: 30
(XEN) [0000002831e42758]  underload_balance_tolerance: 0
(XEN) [0000002832b28468]  overload_balance_tolerance: -3
(XEN) [0000002833886058]  runqueues arrangement: socket
(XEN) [0000002834533890]  cap enforcement granularity: 10ms
(XEN) [00000028353bb4a0] load tracking window length 1073741824 ns
(XEN) [00000028362f7500] Adding cpu 0 to runqueue 0
(XEN) [0000002836ec43e0]  First cpu on runqueue, activating
(XEN) [000000283ef8c6b8] Platform timer is 14.318MHz HPET
(XEN) [   72.057296] Detected 2399.047 MHz processor.
(XEN) [   72.071612] EFI memory map:
(XEN) [   72.074956]  0000000000000-0000000007fff type=3 
attr=000000000000000f
(XEN) [   72.082744]  0000000008000-000000003dfff type=7 
attr=000000000000000f
(XEN) [   72.090269]  000000003e000-000000003ffff type=2 
attr=000000000000000f
(XEN) [   72.097941]  0000000040000-000000009ffff type=3 
attr=000000000000000f
(XEN) [   72.105466]  0000000100000-00000001fffff type=7 
attr=000000000000000f
(XEN) [   72.113175]  0000000200000-000000023ffff type=4 
attr=000000000000000f
(XEN) [   72.120917]  0000000240000-000005ad2afff type=7 
attr=000000000000000f
(XEN) [   72.163315]  000005ad2b000-000005ae53fff type=2 
attr=000000000000000f
(XEN) [   72.205563]  000005ae54000-000005c110fff type=1 
attr=000000000000000f
(XEN) [   72.248048]  000005c111000-000007ad5afff type=2 
attr=000000000000000f
(XEN) [   72.290932]  000007ad5b000-000007ad5efff type=7 
attr=000000000000000f
(XEN) [   72.333997]  000007ad5f000-000007ad64fff type=2 
attr=000000000000000f
(XEN) [   72.377007]  000007ad65000-000007ad65fff type=7 
attr=000000000000000f
(XEN) [   72.420576]  000007ad66000-000007ad66fff type=2 
attr=000000000000000f
(XEN) [   72.463962]  000007ad67000-000007ad67fff type=7 
attr=000000000000000f
(XEN) [   72.506928]  000007ad68000-000007ad6afff type=2 
attr=000000000000000f
(XEN) [   72.550292]  000007ad6b000-000007ad73fff type=7 
attr=000000000000000f
(XEN) [   72.593815]  000007ad74000-000007ad7efff type=2 
attr=000000000000000f
(XEN) [   72.636525]  000007ad7f000-000007db6afff type=4 
attr=000000000000000f
(XEN) [   72.679425]  000007db6b000-000007df0ffff type=7 
attr=000000000000000f
(XEN) [   72.722532]  000007df10000-000007e16afff type=3 
attr=000000000000000f
(XEN) [   72.765658]  000007e16b000-000007eba1fff type=0 
attr=000000000000000f
(XEN) [   72.809009]  000007eba2000-000007ed0cfff type=7 
attr=000000000000000f
(XEN) [   72.852010]  000007ed0d000-000007f28cfff type=10 
attr=000000000000000f
(XEN) [   72.895302]  000007f28d000-000007f5f2fff type=6 
attr=800000000000000f
(XEN) [   72.937987]  000007f5f3000-000007f647fff type=5 
attr=800000000000000f
(XEN) [   72.981389]  000007f648000-000007f7fffff type=4 
attr=000000000000000f
(XEN) [   73.025007]  0000100000000-0000fefffffff type=7 
attr=000000000000000f
(XEN) [   73.067909]  00000e0000000-00000e3ffffff type=11 
attr=8000000000000001
(XEN) [   73.111120]  00000fed01000-00000fed03fff type=11 
attr=8000000000000001
(XEN) [   73.154487]  00000fed08000-00000fed08fff type=11 
attr=8000000000000001
(XEN) [   73.197629]  00000fed0c000-00000fed0ffff type=11 
attr=8000000000000001
(XEN) [   73.241260]  00000fed1c000-00000fed1cfff type=11 
attr=8000000000000001
(XEN) [   73.284256]  00000fef00000-00000feffffff type=11 
attr=8000000000000001
(XEN) [   73.327462]  00000ff800000-00000ffffffff type=11 
attr=8000000000000001
(XEN) [   73.370746] Initing memory sharing.
(XEN) [   73.410811] alt table ffff82d080674c70 -> ffff82d080676a68
(XEN) [   73.454321] I/O virtualisation disabled
(XEN) [   73.495438] nr_sockets: 1
(XEN) [   73.534743] Getting VERSION: 50014
(XEN) [   73.575240] Getting VERSION: 50014
(XEN) [   73.614700] Getting ID: 0
(XEN) [   73.653432] Getting LVT0: 700
(XEN) [   73.692536] Getting LVT1: 400
(XEN) [   73.731789] enabled ExtINT on CPU#0
(XEN) [   73.770190] ENABLING IO-APIC IRQs
(XEN) [   73.808477]  -> Using new ACK method
(XEN) [   73.846109] init IO_APIC IRQs
(XEN) [   73.883940]  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 
2-20, 2-21, 2-22, 2-23 not connected.
(XEN) [   73.965229] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [   74.106548] number of MP IRQ sources: 15.
(XEN) [   74.145195] number of IO-APIC #2 registers: 24.
(XEN) [   74.184159] testing the IO APIC.......................
(XEN) [   74.224688] IO APIC #2......
(XEN) [   74.262138] .... register #00: 02000000
(XEN) [   74.299887] .......    : physical APIC id: 02
(XEN) [   74.337690] .......    : Delivery Type: 0
(XEN) [   74.375168] .......    : LTS          : 0
(XEN) [   74.413463] .... register #01: 00170020
(XEN) [   74.451529] .......     : max redirection entries: 0017
(XEN) [   74.489582] .......     : PRQ implemented: 0
(XEN) [   74.526914] .......     : IO APIC version: 0020
(XEN) [   74.564517] .... IRQ redirection table:
(XEN) [   74.602092]  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) [   74.641885]  00 000 00  1    0    0   0   0    0    0 00
(XEN) [   74.681877]  01 001 01  0    0    0   0   0    1    1 28
(XEN) [   74.722577]  02 001 01  0    0    0   0   0    1    1 F0
(XEN) [   74.762500]  03 001 01  0    0    0   0   0    1    1 30
(XEN) [   74.802607]  04 001 01  1    0    0   0   0    1    1 F1
(XEN) [   74.843045]  05 001 01  0    0    0   0   0    1    1 38
(XEN) [   74.883117]  06 001 01  0    0    0   0   0    1    1 40
(XEN) [   74.923458]  07 001 01  0    0    0   0   0    1    1 48
(XEN) [   74.963574]  08 001 01  0    0    0   0   0    1    1 50
(XEN) [   75.003555]  09 001 01  1    1    0   0   0    1    1 58
(XEN) [   75.044071]  0a 001 01  0    0    0   0   0    1    1 60
(XEN) [   75.083822]  0b 001 01  0    0    0   0   0    1    1 68
(XEN) [   75.123683]  0c 001 01  0    0    0   0   0    1    1 70
(XEN) [   75.163876]  0d 001 01  0    0    0   0   0    1    1 78
(XEN) [   75.203588]  0e 001 01  0    0    0   0   0    1    1 88
(XEN) [   75.243635]  0f 001 01  0    0    0   0   0    1    1 90
(XEN) [   75.283365]  10 08D 0D  1    0    0   0   0    0    2 F7
(XEN) [   75.322818]  11 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.362882]  12 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.402330]  13 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.442553]  14 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.481903]  15 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.521633]  16 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.561389]  17 000 00  1    0    0   0   0    0    0 00
(XEN) [   75.601074] Using vector-based indexing
(XEN) [   75.637043] IRQ to pin mappings:
(XEN) [   75.673259] IRQ240 -> 0:2
(XEN) [   75.709316] IRQ40 -> 0:1
(XEN) [   75.743641] IRQ48 -> 0:3
(XEN) [   75.779468] IRQ241 -> 0:4
(XEN) [   75.813390] IRQ56 -> 0:5
(XEN) [   75.848219] IRQ64 -> 0:6
(XEN) [   75.882180] IRQ72 -> 0:7
(XEN) [   75.916590] IRQ80 -> 0:8
(XEN) [   75.949802] IRQ88 -> 0:9
(XEN) [   75.985189] IRQ96 -> 0:10
(XEN) [   76.018278] IRQ104 -> 0:11
(XEN) [   76.050553] IRQ112 -> 0:12
(XEN) [   76.085008] IRQ120 -> 0:13
(XEN) [   76.117439] IRQ136 -> 0:14
(XEN) [   76.149053] IRQ144 -> 0:15
(XEN) [   76.180044] .................................... done.
(XEN) [   76.213471] Using local APIC timer interrupts.
(XEN) [   76.247000] calibrating APIC timer ...
(XEN) [   76.383322] ..... CPU clock speed is 2400.0482 MHz.
(XEN) [   76.415850] ..... host bus clock speed is 100.0019 MHz.
(XEN) [   76.448293] ..... bus_scale = 0x6669
(XEN) [   76.478840] TSC deadline timer enabled
(XEN) [2019-03-21 16:15:08] mwait-idle: MWAIT substates: 0x3000020
(XEN) [2019-03-21 16:15:08] mwait-idle: v0.4.1 model 0x4d
(XEN) [2019-03-21 16:15:08] mwait-idle: lapic_timer_reliable_states 
0xffffffff
(XEN) [2019-03-21 16:15:08] VMX: Supported advanced features:
(XEN) [2019-03-21 16:15:08]  - APIC MMIO access virtualisation
(XEN) [2019-03-21 16:15:08]  - APIC TPR shadow
(XEN) [2019-03-21 16:15:08]  - Extended Page Tables (EPT)
(XEN) [2019-03-21 16:15:08]  - Virtual-Processor Identifiers (VPID)
(XEN) [2019-03-21 16:15:08]  - Virtual NMI
(XEN) [2019-03-21 16:15:08]  - MSR direct-access bitmap
(XEN) [2019-03-21 16:15:08]  - Unrestricted Guest
(XEN) [2019-03-21 16:15:08]  - VM Functions
(XEN) [2019-03-21 16:15:09] HVM: ASIDs enabled.
(XEN) [2019-03-21 16:15:09] HVM: VMX enabled
(XEN) [2019-03-21 16:15:09] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2019-03-21 16:15:09] HVM: HAP page sizes: 4kB, 2MB

I previously contacted XEN-DEV on March 15th at

https://lists.xenproject.org/archives/html/xen-devel/2019-03/msg01281.html

The environment then was a Gentoo-based distribution with patches.  Andrew
Cooper concluded "root of your problem is that Xen can't find the ACPI 
tables,
which is either going to be a grub or a Xen build misconfiguration."

I learned the Gentoo build contained patches maintained by a Gentoo 
developer,
so I decided to try an apples-to-apples approach: I cloned the Xen 
repository
set my python environment to 2.7 and
".configure", "make","make test", & "make install"
and used the EFI loader and kernel created from the Xen source code w/o 
patches,
log of which is above.  My previous Gentoo-based log had:

(XEN) ACPI Error (tbxfroot-0217): A valid RSDP was not found [20070126]

The Xen source build above does not have that error.

Yet, I'm still getting hung sessions either right after the report out of
"HVM: HAP page sizes: 4kB, 2MB" or during the masking such as:

(XEN) [2019-03-21 16:11:43] masked ExtINT on CPU#2
(XEN) [2019-03-21 16:13:40] Adding cpu 2 to runqueue 0

at https://pastebin.com/3EkQdRYA

and

(XEN) [2019-03-21 16:08:54] masked ExtINT on CPU#5
(XEN) [2019-03-21 16:10:05] Adding cpu 5 to runqueue 0

at https://pastebin.com/gUqCVa66

I am not seeing any error messages such the ACPI error message encountered
within the Gentoo-built Xen kernel, yet I am still facing the problem of 
inconsistent
hangs.  Note: unlike with the Gentoo built Kernel, I have not been able to
successfully get past the "masked ExtINT" phase.

Is there something more I can include in the options or provide to help 
identify
what is causing this problem.  The only negative thing I can discern is 
the post of:

(XEN) [00000028257a81f8] Unrecognised CPU model 0x4d - assuming not 
reptpoline safe

And, of course, the Gentoo kernel for a regular Linux session continues 
to boot reliably.
See https://pastebin.com/VDMP0xGD

-- John


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Thu, Mar 21, 2019 at 10:06:38AM -0700, John L. Poole wrote:
> Boot Hangs at: HVM: HAP page sizes: 4kB, 2MB or Adding cpu [INSERT 1-7] to
> runqueue 0
> 
> Log (preserved for 6 months at https://pastebin.com/BDPP7Pzk)
> 
> fs0:\efi\gentoo> man_xen.efiip startup.nsh, any other key to continue.
> Xen 4.12.0-rc (c/s Mon Feb 25 13:06:22 2019 +0000 git:f393b82fe5-dirty) EFI
> loader
> Using configuration file 'man_xen.cfg'
> xen-4.12.0-rc.gz: 0x000000005ad2b000-0x000000005ae49640
> 0x0000:0x02:0x00.0x0: ROM: 0x8000 bytes at 0x7c8bc028
>  Xen 4.12.0-rc
> (XEN) [00000024693d1c40] Xen version 4.12.0-rc (root@[unknown]) (gcc (Gentoo
> 8.2.0-r6 p1.7) 8.2.0) debug=y  Wed Mar 20 22:35:59 PDT 2019
> (XEN) [000000246afc47a8] Latest ChangeSet: Mon Feb 25 13:06:22 2019 +0000
> git:f393b82fe5-dirty
> (XEN) [000000246c2cdd38] Console output is synchronous.
> (XEN) [000000246ce239e0] Bootloader: EFI
> (XEN) [000000246d6f0cc8] Command line: console=vga,com1 com1=115200,8n1
> console_timestamps console_to_ring conring_size=64 log_buf_len=16M
> loglvl=all guest_loglvl=all sync_console=true sched_debug iommu=verbose
> apic_verbosity=debug efi=no-rs reboot=kbd

Could you try to add 'cpuinfo' to the Xen command line?

This should print some extra information when bringing up the APs.

Have you tried to see if the system boots reliably when no APs are
brought up by setting 'maxcpus=1' on the command line?

Have you tested if Xen can boot reliably on the box when using legacy
BIOS instead of UEFI?

Lastly you could also try to add the 'watchdog' option to the command
line? AFAICT the hang seems to happen before the watchdog is
enabled, but it's worth a try just in case.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/22/2019 12:59 AM, Roger Pau Monné wrote:
> On Thu, Mar 21, 2019 at 10:06:38AM -0700, John L. Poole wrote:
>> Boot Hangs at: HVM: HAP page sizes: 4kB, 2MB or Adding cpu [INSERT 1-7] to
>> runqueue 0
>>
>> Log (preserved for 6 months at https://pastebin.com/BDPP7Pzk)
>>
>> fs0:\efi\gentoo> man_xen.efiip startup.nsh, any other key to continue.
>> Xen 4.12.0-rc (c/s Mon Feb 25 13:06:22 2019 +0000 git:f393b82fe5-dirty) EFI
>> loader
>> Using configuration file 'man_xen.cfg'
>> xen-4.12.0-rc.gz: 0x000000005ad2b000-0x000000005ae49640
>> 0x0000:0x02:0x00.0x0: ROM: 0x8000 bytes at 0x7c8bc028
>>   Xen 4.12.0-rc
>> (XEN) [00000024693d1c40] Xen version 4.12.0-rc (root@[unknown]) (gcc (Gentoo
>> 8.2.0-r6 p1.7) 8.2.0) debug=y  Wed Mar 20 22:35:59 PDT 2019
>> (XEN) [000000246afc47a8] Latest ChangeSet: Mon Feb 25 13:06:22 2019 +0000
>> git:f393b82fe5-dirty
>> (XEN) [000000246c2cdd38] Console output is synchronous.
>> (XEN) [000000246ce239e0] Bootloader: EFI
>> (XEN) [000000246d6f0cc8] Command line: console=vga,com1 com1=115200,8n1
>> console_timestamps console_to_ring conring_size=64 log_buf_len=16M
>> loglvl=all guest_loglvl=all sync_console=true sched_debug iommu=verbose
>> apic_verbosity=debug efi=no-rs reboot=kbd
> Could you try to add 'cpuinfo' to the Xen command line?
>
> This should print some extra information when bringing up the APs.
>
> Have you tried to see if the system boots reliably when no APs are
> brought up by setting 'maxcpus=1' on the command line?
>
> Have you tested if Xen can boot reliably on the box when using legacy
> BIOS instead of UEFI?
>
> Lastly you could also try to add the 'watchdog' option to the command
> line? AFAICT the hang seems to happen before the watchdog is
> enabled, but it's worth a try just in case.
>
> Thanks, Roger.
Hi Roger,

Here are my responses and logs to the 4 points you made:

1) Xen Source - here is the log of an attempt adding "cpuinfo"
as an option in my man_xen.cfg:
https://pastebin.com/nBDPV5v7 (6 months)

The last two lines are:

(XEN) [2019-03-22 09:24:41] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2019-03-22 09:24:41] Booting processor 1/2 eip 3e000

Result: hangs around the same place

2) Xen Source - here is the log of an attempt adding "cpuinfor maxcpus=1"
as an option in myman_xen.cfg:
https://pastebin.com/ifHZqCuX (6months)

The last 20+ lines are:
(XEN) [2019-03-22 09:28:28] CPU: Physical Processor ID: 0
(XEN) [2019-03-22 09:28:28] CPU: Processor Core ID: 1
(XEN) [2019-03-22 09:28:28] CPU: L1 I cache: 32K, L1 D cache: 24K
(XEN) [2019-03-22 09:28:28] CPU: L2 cache: 1024K
(XEN) [2019-03-22 09:28:28] CMCI: CPU1 has no CMCI support
(XEN) [2019-03-22 09:28:28] CPU1: Thermal monitoring enabled (TM1)
(XEN) [2019-03-22 09:30:47] CPU1: Intel(R) Atom(TM) CPU C2750  @ 2.40GHz 
stepping 08
(XEN) [2019-03-22 09:30:47] Adding cpu 1 to runqueue 0
(XEN) [2019-03-22 09:30:47] Removing cpu 1 from runqueue 0
(XEN) [2019-03-22 09:30:47] Booting processor 2/4 eip 3e000
(XEN) [2019-03-22 09:28:28] Initializing CPU#2
(XEN) [2019-03-22 09:28:28] masked ExtINT on CPU#2
(XEN) [2019-03-22 09:28:28] CPU: Physical Processor ID: 0
(XEN) [2019-03-22 09:28:28] CPU: Processor Core ID: 2
(XEN) [2019-03-22 09:28:28] CPU: L1 I cache: 32K, L1 D cache: 24K
(XEN) [2019-03-22 09:28:28] CPU: L2 cache: 1024K
(XEN) [2019-03-22 09:28:28] CMCI: CPU2 has no CMCI support
(XEN) [2019-03-22 09:28:28] CPU2: Thermal monitoring enabled (TM1)
(XEN) [2019-03-22 09:30:48] CPU2: Intel(R) Atom(TM) CPU C2750  @ 2.40GHz 
stepping 08
(XEN) [2019-03-22 09:30:48] Adding cpu 2 to runqueue 0
(XEN) [2019-03-22 09:30:48] Removing cpu 2 from runqueue 0
(XEN) [2019-03-22 09:30:48] Booting processor 3/6 eip 3e000

Result: hangs around the same place

3)Xen Source - here is the log of an attempt adding
"cpuinfor maxcpus=1 watchdog"
as an option in myman_xen.cfg:
https://pastebin.com/b682FWmC (6 months)

The last 12 lines:
(XEN) [2019-03-22 09:37:49] Booting processor 2/4 eip 3e000
(XEN) [2019-03-22 09:35:28] Initializing CPU#2
(XEN) [2019-03-22 09:35:28] masked ExtINT on CPU#2
(XEN) [2019-03-22 09:35:28] CPU: Physical Processor ID: 0
(XEN) [2019-03-22 09:35:28] CPU: Processor Core ID: 2
(XEN) [2019-03-22 09:35:28] CPU: L1 I cache: 32K, L1 D cache: 24K
(XEN) [2019-03-22 09:35:28] CPU: L2 cache: 1024K
(XEN) [2019-03-22 09:35:28] CMCI: CPU2 has no CMCI support
(XEN) [2019-03-22 09:35:28] CPU2: Thermal monitoring enabled (TM1)
(XEN) [2019-03-22 09:37:49] CPU2: Intel(R) Atom(TM) CPU  C2750 @ 2.40GHz 
stepping 08
(XEN) [2019-03-22 09:37:49] Adding cpu 2 to runqueue 0
(XEN) [2019-03-22 09:37:49] Removing cpu 2 from runqueue 0
(XEN) [2019-03-22 09:37:49] Booting processor 3/6 eip 3e000

Result: hangs around the same place

4) My Supermicro Intel Atom based unit does not offer legacy BIOS, at
least as far as I can tell. I'm stuck with their UEFI.
I have screen shots of most of the BIOS windows
if you want.  Here's a link to my model:
https://www.supermicro.com/products/system/1u/5018/SYS-5018A-TN4.cfm

The site has this BIOS specification:

BIOS Type

     64Mb SPI Flash EEPROM with AMI UEFI BIOS


-John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Andrew Cooper]

On 21/03/2019 17:06, John L. Poole wrote:

> Is there something more I can include in the options or provide to
> help identify
> what is causing this problem.  The only negative thing I can discern
> is the post of:
>
> (XEN) [00000028257a81f8] Unrecognised CPU model 0x4d - assuming not
> reptpoline safe

I've subsequently fixed this.

http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=17f74242ccf0ce6e51c03a5860947865c0ef0dc2

It is safe to ignore the warning, but it will also disappear if you
update to the the latest master.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Andrew Cooper]

On 22/03/2019 09:53, John L. Poole wrote:
> 3)Xen Source - here is the log of an attempt adding
> "cpuinfor maxcpus=1 watchdog"
> as an option in myman_xen.cfg:
> https://pastebin.com/b682FWmC (6 months)
>
> The last 12 lines:
> (XEN) [2019-03-22 09:37:49] Booting processor 2/4 eip 3e000
> (XEN) [2019-03-22 09:35:28] Initializing CPU#2
> (XEN) [2019-03-22 09:35:28] masked ExtINT on CPU#2
> (XEN) [2019-03-22 09:35:28] CPU: Physical Processor ID: 0
> (XEN) [2019-03-22 09:35:28] CPU: Processor Core ID: 2
> (XEN) [2019-03-22 09:35:28] CPU: L1 I cache: 32K, L1 D cache: 24K
> (XEN) [2019-03-22 09:35:28] CPU: L2 cache: 1024K
> (XEN) [2019-03-22 09:35:28] CMCI: CPU2 has no CMCI support
> (XEN) [2019-03-22 09:35:28] CPU2: Thermal monitoring enabled (TM1)
> (XEN) [2019-03-22 09:37:49] CPU2: Intel(R) Atom(TM) CPU  C2750 @
> 2.40GHz stepping 08
> (XEN) [2019-03-22 09:37:49] Adding cpu 2 to runqueue 0
> (XEN) [2019-03-22 09:37:49] Removing cpu 2 from runqueue 0
> (XEN) [2019-03-22 09:37:49] Booting processor 3/6 eip 3e000
>
> Result: hangs around the same place

Ok.  Something is clearly stalling while we are trying to start
secondary processors.

Can you apply this patch and rebuild please?

andrewcoop@andrewcoop:/local/xen.git$ git d
diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
index 9d7ec93..14ac0b1 100644
--- a/xen/include/asm-x86/apic.h
+++ b/xen/include/asm-x86/apic.h
@@ -5,7 +5,7 @@
 #include <asm/fixmap.h>
 #include <asm/msr.h>
 
-#define Dprintk(x...) do {} while (0)
+#define Dprintk printk
 
 /*
  * Debugging macros

which should give us some better diagnostics of the INIT-SIPI-SIPI
mechanism.

Do you have any options such as TXT or SMX enabled in firmware?  They
can interfere with AP bringup, so it would be useful to disable them for
now.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/22/2019 7:40 AM, Andrew Cooper wrote:
> On 22/03/2019 09:53, John L. Poole wrote:
>> 3)Xen Source - here is the log of an attempt adding
>> "cpuinfor maxcpus=1 watchdog"
>> as an option in myman_xen.cfg:
>> https://pastebin.com/b682FWmC (6 months)
>>
>> The last 12 lines:
>> (XEN) [2019-03-22 09:37:49] Booting processor 2/4 eip 3e000
>> (XEN) [2019-03-22 09:35:28] Initializing CPU#2
>> (XEN) [2019-03-22 09:35:28] masked ExtINT on CPU#2
>> (XEN) [2019-03-22 09:35:28] CPU: Physical Processor ID: 0
>> (XEN) [2019-03-22 09:35:28] CPU: Processor Core ID: 2
>> (XEN) [2019-03-22 09:35:28] CPU: L1 I cache: 32K, L1 D cache: 24K
>> (XEN) [2019-03-22 09:35:28] CPU: L2 cache: 1024K
>> (XEN) [2019-03-22 09:35:28] CMCI: CPU2 has no CMCI support
>> (XEN) [2019-03-22 09:35:28] CPU2: Thermal monitoring enabled (TM1)
>> (XEN) [2019-03-22 09:37:49] CPU2: Intel(R) Atom(TM) CPU  C2750 @
>> 2.40GHz stepping 08
>> (XEN) [2019-03-22 09:37:49] Adding cpu 2 to runqueue 0
>> (XEN) [2019-03-22 09:37:49] Removing cpu 2 from runqueue 0
>> (XEN) [2019-03-22 09:37:49] Booting processor 3/6 eip 3e000
>>
>> Result: hangs around the same place
> Ok.  Something is clearly stalling while we are trying to start
> secondary processors.
>
> Can you apply this patch and rebuild please?
>
> andrewcoop@andrewcoop:/local/xen.git$ git d
> diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
> index 9d7ec93..14ac0b1 100644
> --- a/xen/include/asm-x86/apic.h
> +++ b/xen/include/asm-x86/apic.h
> @@ -5,7 +5,7 @@
>   #include <asm/fixmap.h>
>   #include <asm/msr.h>
>   
> -#define Dprintk(x...) do {} while (0)
> +#define Dprintk printk
>   
>   /*
>    * Debugging macros
>
> which should give us some better diagnostics of the INIT-SIPI-SIPI
> mechanism.
>
> Do you have any options such as TXT or SMX enabled in firmware?  They
> can interfere with AP bringup, so it would be useful to disable them for
> now.
>
> ~Andrew

done.

I tried patching and then make, but ran into an error.  So I performed:

git pull
make clean

then verified the patch was still in effect, and then:

make

There was some problem in the install so I hand moved:
...
-rw-r--r-- 1 root root2991647 Mar 22 11:01 xen-4.13-unstable.efi
...
under /usr/local/src/xen/dist/install/usr/lib64/efi/
to /boot/efi/gentoo and renamed it man_xen.efi.

Likewise, if found a xen kernel under
/usr/local/src/xen/xen/dist/install/boot/
...
-rw-r--r-- 1 root root 1181850 Mar 22 11:01 xen-4.13-unstable.gz
...
and moved it to /boot/efi/gentoo -- not renaming it and
making sure /boot/efi/gentoo/man_xen.cfg defines the kernel as
"xen-4.13-unstable.gz"

Result: same failure, but with more debugging information.

Here are the last ten lines (starting at line 287):

(XEN) [2019-03-23 00:36:06] HVM: ASIDs enabled.
(XEN) [2019-03-23 00:36:06] HVM: VMX enabled
(XEN) [2019-03-23 00:36:06] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2019-03-23 00:36:06] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2019-03-23 00:36:06] Booting processor 1/2 eip 3e000
(XEN) [2019-03-23 00:36:06] Setting warm reset code and vector.
(XEN) [2019-03-23 00:36:06] 1.
(XEN) [2019-03-23 00:36:06] 2.
(XEN) [2019-03-23 00:36:06] 3.
(XEN) [2019-03-23 00:36:06] Asserting INIT.
(XEN) [2019-03-23 00:36:06] Waiting for send to finish...

Here is the full boot log:
https://pastebin.com/0LgrJH25

I am not familiar with TXT or SMX.  I looked around in the BIOS setup
and could not find anything that matched those acronyms. Consequently,
my BIOS is and remains unaltered from the above attempt and prior attempts.
I also searched Supermicros User Manaul for the "B1SA4-2750F/B1SA4-2550F"
and neither "TXT" or "SMX" was found.

My Google search of "BIOS TXT" gave me:
https://en.wikipedia.org/wiki/Trusted_Execution_Technology

Intel Trusted Execution Technology (Intel TXT, formerly known as 
LaGrande Technology)

My Google search of "BIOS SMX" gave me:
https://www.thomas-krenn.com/en/wiki/Activating_the_Intel_VT_Virtualization_Feature
which states:
The BIOS setting can be changed from
Advanced --> Processor Configuration --> Intel(R) Virtualization 
Technology.

I did not find either technology referenced within my BIOS when visiting 
various
configuration settings.

I can post on a server screen shots of the BIOS pages if that would help.

Thank you.

-John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Fri, Mar 22, 2019 at 05:46:26PM -0700, John L. Poole wrote:
> 
> On 3/22/2019 7:40 AM, Andrew Cooper wrote:
> > On 22/03/2019 09:53, John L. Poole wrote:
> > > 3)Xen Source - here is the log of an attempt adding
> > > "cpuinfor maxcpus=1 watchdog"
> > > as an option in myman_xen.cfg:
> > > https://pastebin.com/b682FWmC (6 months)
> > > 
> > > The last 12 lines:
> > > (XEN) [2019-03-22 09:37:49] Booting processor 2/4 eip 3e000
> > > (XEN) [2019-03-22 09:35:28] Initializing CPU#2
> > > (XEN) [2019-03-22 09:35:28] masked ExtINT on CPU#2
> > > (XEN) [2019-03-22 09:35:28] CPU: Physical Processor ID: 0
> > > (XEN) [2019-03-22 09:35:28] CPU: Processor Core ID: 2
> > > (XEN) [2019-03-22 09:35:28] CPU: L1 I cache: 32K, L1 D cache: 24K
> > > (XEN) [2019-03-22 09:35:28] CPU: L2 cache: 1024K
> > > (XEN) [2019-03-22 09:35:28] CMCI: CPU2 has no CMCI support
> > > (XEN) [2019-03-22 09:35:28] CPU2: Thermal monitoring enabled (TM1)
> > > (XEN) [2019-03-22 09:37:49] CPU2: Intel(R) Atom(TM) CPU  C2750 @
> > > 2.40GHz stepping 08
> > > (XEN) [2019-03-22 09:37:49] Adding cpu 2 to runqueue 0
> > > (XEN) [2019-03-22 09:37:49] Removing cpu 2 from runqueue 0
> > > (XEN) [2019-03-22 09:37:49] Booting processor 3/6 eip 3e000
> > > 
> > > Result: hangs around the same place
> > Ok.  Something is clearly stalling while we are trying to start
> > secondary processors.
> > 
> > Can you apply this patch and rebuild please?
> > 
> > andrewcoop@andrewcoop:/local/xen.git$ git d
> > diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
> > index 9d7ec93..14ac0b1 100644
> > --- a/xen/include/asm-x86/apic.h
> > +++ b/xen/include/asm-x86/apic.h
> > @@ -5,7 +5,7 @@
> >   #include <asm/fixmap.h>
> >   #include <asm/msr.h>
> > -#define Dprintk(x...) do {} while (0)
> > +#define Dprintk printk
> >   /*
> >    * Debugging macros
> > 
> > which should give us some better diagnostics of the INIT-SIPI-SIPI
> > mechanism.
> > 
> > Do you have any options such as TXT or SMX enabled in firmware?  They
> > can interfere with AP bringup, so it would be useful to disable them for
> > now.
> > 
> > ~Andrew
> 
> done.
> 
> I tried patching and then make, but ran into an error.  So I performed:
> 
> git pull
> make clean
> 
> then verified the patch was still in effect, and then:
> 
> make
> 
> There was some problem in the install so I hand moved:
> ...
> -rw-r--r-- 1 root root2991647 Mar 22 11:01 xen-4.13-unstable.efi
> ...
> under /usr/local/src/xen/dist/install/usr/lib64/efi/
> to /boot/efi/gentoo and renamed it man_xen.efi.
> 
> Likewise, if found a xen kernel under
> /usr/local/src/xen/xen/dist/install/boot/
> ...
> -rw-r--r-- 1 root root 1181850 Mar 22 11:01 xen-4.13-unstable.gz
> ...
> and moved it to /boot/efi/gentoo -- not renaming it and
> making sure /boot/efi/gentoo/man_xen.cfg defines the kernel as
> "xen-4.13-unstable.gz"
> 
> Result: same failure, but with more debugging information.
> 
> Here are the last ten lines (starting at line 287):
> 
> (XEN) [2019-03-23 00:36:06] HVM: ASIDs enabled.
> (XEN) [2019-03-23 00:36:06] HVM: VMX enabled
> (XEN) [2019-03-23 00:36:06] HVM: Hardware Assisted Paging (HAP) detected
> (XEN) [2019-03-23 00:36:06] HVM: HAP page sizes: 4kB, 2MB
> (XEN) [2019-03-23 00:36:06] Booting processor 1/2 eip 3e000
> (XEN) [2019-03-23 00:36:06] Setting warm reset code and vector.
> (XEN) [2019-03-23 00:36:06] 1.
> (XEN) [2019-03-23 00:36:06] 2.
> (XEN) [2019-03-23 00:36:06] 3.
> (XEN) [2019-03-23 00:36:06] Asserting INIT.
> (XEN) [2019-03-23 00:36:06] Waiting for send to finish...
> 
> Here is the full boot log:
> https://pastebin.com/0LgrJH25

I'm currently away from home, and cannot really help much ATM, also I
don't have access to a system with a CPU that exhibits such behavior,
much makes debugging it harder.

I've taken a look at the difference in AP startup code between Linux
and Xen at or before the point you get the hang, and I'm not able to
spot anything obvious that could make Linux work and not Xen.

I've realized however that Linux disables interrupts when writing to
the local APIC ICR register for other reasons, but maybe this somehow
affects bring up in this CPU, hence the patch below. Could you please
give it a spin together with the patch provided by Andrew?

There are other minor differences between Linux and Xen AP bring up,
so I guess there are further changes to test if the patch below
doesn't make things better.

Thanks, Roger.
---8<---
diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
index 9d7ec93042..f28e922e2e 100644
--- a/xen/include/asm-x86/apic.h
+++ b/xen/include/asm-x86/apic.h
@@ -138,8 +138,12 @@ static __inline void apic_icr_write(u32 low, u32 dest)
         apic_wrmsr(APIC_ICR, low | ((uint64_t)dest << 32));
     else
     {
+        unsigned long flags;
+
+        local_irq_save(flags);
         apic_mem_write(APIC_ICR2, dest << 24);
         apic_mem_write(APIC_ICR, low);
+        local_irq_restore(flags);
     }
 }
 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/23/2019 11:50 PM, Roger Pau Monné wrote:
> On Fri, Mar 22, 2019 at 05:46:26PM -0700, John L. Poole wrote:
>> On 3/22/2019 7:40 AM, Andrew Cooper wrote:
>>> On 22/03/2019 09:53, John L. Poole wrote:
>>>> 3)Xen Source - here is the log of an attempt adding
>>>> "cpuinfor maxcpus=1 watchdog"
>>>> as an option in myman_xen.cfg:
>>>> https://pastebin.com/b682FWmC (6 months)
>>>>
>>>> The last 12 lines:
>>>> (XEN) [2019-03-22 09:37:49] Booting processor 2/4 eip 3e000
>>>> (XEN) [2019-03-22 09:35:28] Initializing CPU#2
>>>> (XEN) [2019-03-22 09:35:28] masked ExtINT on CPU#2
>>>> (XEN) [2019-03-22 09:35:28] CPU: Physical Processor ID: 0
>>>> (XEN) [2019-03-22 09:35:28] CPU: Processor Core ID: 2
>>>> (XEN) [2019-03-22 09:35:28] CPU: L1 I cache: 32K, L1 D cache: 24K
>>>> (XEN) [2019-03-22 09:35:28] CPU: L2 cache: 1024K
>>>> (XEN) [2019-03-22 09:35:28] CMCI: CPU2 has no CMCI support
>>>> (XEN) [2019-03-22 09:35:28] CPU2: Thermal monitoring enabled (TM1)
>>>> (XEN) [2019-03-22 09:37:49] CPU2: Intel(R) Atom(TM) CPU  C2750 @
>>>> 2.40GHz stepping 08
>>>> (XEN) [2019-03-22 09:37:49] Adding cpu 2 to runqueue 0
>>>> (XEN) [2019-03-22 09:37:49] Removing cpu 2 from runqueue 0
>>>> (XEN) [2019-03-22 09:37:49] Booting processor 3/6 eip 3e000
>>>>
>>>> Result: hangs around the same place
>>> Ok.  Something is clearly stalling while we are trying to start
>>> secondary processors.
>>>
>>> Can you apply this patch and rebuild please?
>>>
>>> andrewcoop@andrewcoop:/local/xen.git$ git d
>>> diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
>>> index 9d7ec93..14ac0b1 100644
>>> --- a/xen/include/asm-x86/apic.h
>>> +++ b/xen/include/asm-x86/apic.h
>>> @@ -5,7 +5,7 @@
>>>    #include <asm/fixmap.h>
>>>    #include <asm/msr.h>
>>> -#define Dprintk(x...) do {} while (0)
>>> +#define Dprintk printk
>>>    /*
>>>     * Debugging macros
>>>
>>> which should give us some better diagnostics of the INIT-SIPI-SIPI
>>> mechanism.
>>>
>>> Do you have any options such as TXT or SMX enabled in firmware?  They
>>> can interfere with AP bringup, so it would be useful to disable them for
>>> now.
>>>
>>> ~Andrew
>> done.
>>
>> I tried patching and then make, but ran into an error.  So I performed:
>>
>> git pull
>> make clean
>>
>> then verified the patch was still in effect, and then:
>>
>> make
>>
>> There was some problem in the install so I hand moved:
>> ...
>> -rw-r--r-- 1 root root2991647 Mar 22 11:01 xen-4.13-unstable.efi
>> ...
>> under /usr/local/src/xen/dist/install/usr/lib64/efi/
>> to /boot/efi/gentoo and renamed it man_xen.efi.
>>
>> Likewise, if found a xen kernel under
>> /usr/local/src/xen/xen/dist/install/boot/
>> ...
>> -rw-r--r-- 1 root root 1181850 Mar 22 11:01 xen-4.13-unstable.gz
>> ...
>> and moved it to /boot/efi/gentoo -- not renaming it and
>> making sure /boot/efi/gentoo/man_xen.cfg defines the kernel as
>> "xen-4.13-unstable.gz"
>>
>> Result: same failure, but with more debugging information.
>>
>> Here are the last ten lines (starting at line 287):
>>
>> (XEN) [2019-03-23 00:36:06] HVM: ASIDs enabled.
>> (XEN) [2019-03-23 00:36:06] HVM: VMX enabled
>> (XEN) [2019-03-23 00:36:06] HVM: Hardware Assisted Paging (HAP) detected
>> (XEN) [2019-03-23 00:36:06] HVM: HAP page sizes: 4kB, 2MB
>> (XEN) [2019-03-23 00:36:06] Booting processor 1/2 eip 3e000
>> (XEN) [2019-03-23 00:36:06] Setting warm reset code and vector.
>> (XEN) [2019-03-23 00:36:06] 1.
>> (XEN) [2019-03-23 00:36:06] 2.
>> (XEN) [2019-03-23 00:36:06] 3.
>> (XEN) [2019-03-23 00:36:06] Asserting INIT.
>> (XEN) [2019-03-23 00:36:06] Waiting for send to finish...
>>
>> Here is the full boot log:
>> https://pastebin.com/0LgrJH25
> I'm currently away from home, and cannot really help much ATM, also I
> don't have access to a system with a CPU that exhibits such behavior,
> much makes debugging it harder.
>
> I've taken a look at the difference in AP startup code between Linux
> and Xen at or before the point you get the hang, and I'm not able to
> spot anything obvious that could make Linux work and not Xen.
>
> I've realized however that Linux disables interrupts when writing to
> the local APIC ICR register for other reasons, but maybe this somehow
> affects bring up in this CPU, hence the patch below. Could you please
> give it a spin together with the patch provided by Andrew?
>
> There are other minor differences between Linux and Xen AP bring up,
> so I guess there are further changes to test if the patch below
> doesn't make things better.
>
> Thanks, Roger.
> ---8<---
> diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
> index 9d7ec93042..f28e922e2e 100644
> --- a/xen/include/asm-x86/apic.h
> +++ b/xen/include/asm-x86/apic.h
> @@ -138,8 +138,12 @@ static __inline void apic_icr_write(u32 low, u32 dest)
>           apic_wrmsr(APIC_ICR, low | ((uint64_t)dest << 32));
>       else
>       {
> +        unsigned long flags;
> +
> +        local_irq_save(flags);
>           apic_mem_write(APIC_ICR2, dest << 24);
>           apic_mem_write(APIC_ICR, low);
> +        local_irq_restore(flags);
>       }
>   }
>   
The patch made a line of progress (got to Deasserting INIT):

(XEN) [2019-03-24 16:17:26] HVM: VMX enabled
(XEN) [2019-03-24 16:17:26] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2019-03-24 16:17:27] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2019-03-24 16:17:27] Booting processor 1/2 eip 3e000
(XEN) [2019-03-24 16:17:27] Setting warm reset code and vector.
(XEN) [2019-03-24 16:17:27] 1.
(XEN) [2019-03-24 16:17:27] 2.
(XEN) [2019-03-24 16:17:27] 3.
(XEN) [2019-03-24 16:17:27] Asserting INIT.
(XEN) [2019-03-24 16:17:27] Waiting for send to finish...
(XEN) [2019-03-24 16:17:27] +Deasserting INIT.

Full log at:
Xen Source 4.13 unstable w/201903231150_pau.patch
at: https://pastebin.com/eewfy91P

For posterity, here's my patch log:

    zeta /usr/local/src/xen # patch <201903231150_pau.patch
    ...
    zeta /usr/local/src/xen # cat xen/include/asm-x86/apic.h |grep -n 
"restore(flags)"
    146:        local_irq_restore(flags);
    zeta /usr/local/src/xen # cat xen/include/asm-x86/apic.h |grep -n 
"Dprintk printk"
    8:#define Dprintk printk
    zeta /usr/local/src/xen #

    make
    cp dist/install/usr/lib64/efi/xen-4.13-unstable.efi 
/boot/efi/gentoo/man_xen.efi
    cp dist/install/boot/xen-4.13-unstable.gz /boot/efi/gentoo
    reboot

I performed a boot a 2nd time, the ending result was with these two lines

(no "+"s and no Deasserting):

(XEN) [2019-03-24 16:23:51] Asserting INIT.
(XEN) [2019-03-24 16:23:51] Waiting for send to finish...

I performed a boot a 3rd time, the ending result on my serial console was:

(XEN) [2019-03-24 16:25:53] Waiting for send to finish...
(XEN) [2019-03-24 16:25:53] +Deassertin

Note: the console attached to the server ("server console")
only had the "+"
and no "Deassertin" [sic - missing "g"], so there seems
to be an inconsistency between the server console's output
and the serial port console.  Probably not relevant to this
inquiry, but I note it so that in the future I will always check
the last entries in server's console vs. the serial PuTTy port.

For what it is worth, I've posted
UEFI vars, dmidecode, & hwinfo
at:
https://pastebin.com/d6zjv7x0

I am very impressed with the dedication you two have demonstrated.

Thank you.

-John


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Jan Beulich]

>>> On 22.03.19 at 10:53, <jlpoole56@gmail.com> wrote:
> 2) Xen Source - here is the log of an attempt adding "cpuinfor maxcpus=1"
> as an option in myman_xen.cfg:
> https://pastebin.com/ifHZqCuX (6months)

Well, the "maxcpus=1" doesn't really hide the issue (anymore) because
we now bring up all secondary processors despite this option (and then
park them because of the option). But other than for 1) (yet like for 3))
two of the secondary processors get brought up fine here. Could you
check whether this is random, i.e. whether with the same set of options
you observe varying or consistent mileage? In the latter case slowing
things further down might by another worthwhile experiment, all to get
a feel for whether we're dealing with some systematic issue in our code
or some timing issue.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/25/2019 8:53 AM, Jan Beulich wrote:
>>>> On 22.03.19 at 10:53, <jlpoole56@gmail.com> wrote:
>> 2) Xen Source - here is the log of an attempt adding "cpuinfor maxcpus=1"
>> as an option in myman_xen.cfg:
>> https://pastebin.com/ifHZqCuX (6months)
> Well, the "maxcpus=1" doesn't really hide the issue (anymore) because
> we now bring up all secondary processors despite this option (and then
> park them because of the option). But other than for 1) (yet like for 3))
> two of the secondary processors get brought up fine here. Could you
> check whether this is random, i.e. whether with the same set of options
> you observe varying or consistent mileage? In the latter case slowing
> things further down might by another worthwhile experiment, all to get
> a feel for whether we're dealing with some systematic issue in our code
> or some timing issue.
>
> Jan
>
>
I removed the "maxcpus=1" and attempted seven (7) boots.
There were minor variations, they do not seem to get
past the first additional processor.  Below is a summary.

Hang points at https://pastebin.com/9gab3fZm

293:
(XEN) [2019-03-25 16:31:47] 3.
(XEN) [2019-03-25 16:31:47] Asserting INIT.
(XEN) [2019-03-25 16:31:47] Waiting for send to finish...

604
(XEN) [2019-03-25 16:33:35] Asserting INIT.
(XEN) [2019-03-25 16:33:35] Waiting for send to finish...
(XEN) [2019-03-25 16:33:35] +Deasserting INIT.
(XEN) [2019-03-25 16:33:35] Waiting for send to finish...

915:
(XEN) [2019-03-25 16:35:12] 3.
(XEN) [2019-03-25 16:35:12] Asserting INIT.
(XEN) [2019-03-25 16:35:12] Waiting for send to finish...
(XEN) [2019-03-25 16:35:12] +Deasserting INIT.

1225:
(XEN) [2019-03-25 16:36:52] 3.
(XEN) [2019-03-25 16:36:52] Asserting INIT.
(XEN) [2019-03-25 16:36:52] Waiting for send to finish...

1538:
(XEN) [2019-03-25 16:38:23] 3.
(XEN) [2019-03-25 16:38:23] Asserting INIT.
(XEN) [2019-03-25 16:38:23] Waiting for send to finish...
(XEN) [2019-03-25 16:38:24] +Deasserting INIT.
(XEN) [2019-03-25 16:38:24] Waiting for send to finish...

1848:
(XEN) [2019-03-25 16:39:52] 3.
(XEN) [2019-03-25 16:39:52] Asserting INIT.
(XEN) [2019-03-25 16:39:52] Waiting for send to finish...

2159:
(XEN) [2019-03-25 16:41:19] 1.
(XEN) [2019-03-25 16:41:19] 2.
(XEN) [2019-03-25 16:41:19] 3.
(XEN) [2019-03-25 16:41:19] Asserting INIT.
(XEN) [2019-03-25 16:41:19] Waiting for send to finish...

-- John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Jan Beulich]

>>> On 25.03.19 at 18:00, <jlpoole56@gmail.com> wrote:
> I removed the "maxcpus=1" and attempted seven (7) boots.
> There were minor variations, they do not seem to get
> past the first additional processor.  Below is a summary.

And with "maxcpus=1" in place, is it as stable hanging when trying
to bring up the 3rd AP? I find it rather surprising that, with you
already having sync_console on the command line, there's such a
variation. I wonder whether SMM is somehow getting in the way.

The main difference to bare metal Linux is that we enable VMX
while they don't (afaict). Since for whatever reason we don't
(yet) have a command line option to suppress this, could you
re-build Xen with CONFIG_HVM turned off in xen/.config?

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

[-- Attachment #1.1: Type: text/plain, Size: 2025 bytes --]


On 3/26/2019 1:04 AM, Jan Beulich wrote:
>>>> On 25.03.19 at 18:00, <jlpoole56@gmail.com> wrote:
>> I removed the "maxcpus=1" and attempted seven (7) boots.
>> There were minor variations, they do not seem to get
>> past the first additional processor.  Below is a summary.
> And with "maxcpus=1" in place, is it as stable hanging when trying
> to bring up the 3rd AP? I find it rather surprising that, with you
> already having sync_console on the command line, there's such a
> variation. I wonder whether SMM is somehow getting in the way.
>
> The main difference to bare metal Linux is that we enable VMX
> while they don't (afaict). Since for whatever reason we don't
> (yet) have a command line option to suppress this, could you
> re-build Xen with CONFIG_HVM turned off in xen/.config?
>
> Jan
>
>
# rem'd out CONFIG_HVM=y
make clean
make

7:46 AM 3/26/2019 build of xen w/o HVM
prompted with:
HVM support (HVM) [Y/n/?]
replied: n

cp /boot/efi/gentoo/man_xen.efi /boot/efi/gentoo/man_xen.efi_HVM_yes
cp /usr/local/src/xen/dist/install/usr/lib64/efi/xen-4.13-unstable.efi 
/boot/efi/gentoo/man_xen.efi

cp /boot/efi/gentoo/xen-4.13-unstable.gz 
/boot/efi/gentoo/xen-4.13-unstable.gz_HVM_yes
cp /usr/local/src/xen/dist/install/boot/xen-4.13-unstable.gz 
/boot/efi/gentoo/xen-4.13-unstable.gz

zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
# CONFIG_HVM is not set
zeta /usr/local/src/xen #

# tried 2 boot attempts
log at: https://pastebin.com/nL4BWJ6Y

Hang points at lines:
298:
(XEN) [2019-03-26 17:14:01] Setting warm reset code and vector.
(XEN) [2019-03-26 17:14:01] 1.
(XEN) [2019-03-26 17:14:01] 2.
(XEN) [2019-03-26 17:14:01] 3.
(XEN) [2019-03-26 17:14:01] Asserting INIT.
(XEN) [2019-03-26 17:14:01] Waiting for send to finish...

599:
(XEN) [2019-03-26 17:16:30] Setting warm reset code and vector.
(XEN) [2019-03-26 17:16:30] 1.
(XEN) [2019-03-26 17:16:30] 2.
(XEN) [2019-03-26 17:16:30] 3.
(XEN) [2019-03-26 17:16:30] Asserting INIT.
(XEN) [2019-03-26 17:16:30] Waiting for send to finish...

[-- Attachment #1.2: Type: text/html, Size: 3496 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Jan Beulich]

>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
> # CONFIG_HVM is not set
> zeta /usr/local/src/xen #
> 
> # tried 2 boot attempts
> log at: https://pastebin.com/nL4BWJ6Y 
> 
> Hang points at lines:

Thanks for trying anyway; one further possibility eliminated. Looking
at the logs I've had another thought (wild guess again, so not really
much hope): Could you try "mwait-idle=no"?

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/27/2019 1:14 AM, Jan Beulich wrote:
>>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
>> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
>> # CONFIG_HVM is not set
>> zeta /usr/local/src/xen #
>>
>> # tried 2 boot attempts
>> log at: https://pastebin.com/nL4BWJ6Y
>>
>> Hang points at lines:
> Thanks for trying anyway; one further possibility eliminated. Looking
> at the logs I've had another thought (wild guess again, so not really
> much hope): Could you try "mwait-idle=no"?
>
> Jan
>
>
I modified man_xen.cfg by adding at the end the kernel parameter:

mwait-idle=no

Rebooted.
Result: hung:

...
(XEN) [ 200.939353] TSC deadline timer enabled
(XEN) [2019-03-27 13:21:21] Platform timer appears to have unexpectedly 
wrapped 1 times.
(XEN) [2019-03-27 13:21:21] mwait-idle: disabled
(XEN) [2019-03-27 13:21:21] Booting processor 1/2 eip 3e000
(XEN) [2019-03-27 13:21:21] Setting warm reset code and vector.
(XEN) [2019-03-27 13:21:21] 1.
(XEN) [2019-03-27 13:21:21] 2.
(XEN) [2019-03-27 13:21:21] 3.
(XEN) [2019-03-27 13:21:21] Asserting INIT.
(XEN) [2019-03-27 13:21:22] Waiting for send to finish...

Log at: https://pastebin.com/zdyhCtGv

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Jan Beulich]

>>> On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
> On 3/27/2019 1:14 AM, Jan Beulich wrote:
>>>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
>>> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
>>> # CONFIG_HVM is not set
>>> zeta /usr/local/src/xen #
>>>
>>> # tried 2 boot attempts
>>> log at: https://pastebin.com/nL4BWJ6Y 
>>>
>>> Hang points at lines:
>> Thanks for trying anyway; one further possibility eliminated. Looking
>> at the logs I've had another thought (wild guess again, so not really
>> much hope): Could you try "mwait-idle=no"?
>>
> I modified man_xen.cfg by adding at the end the kernel parameter:
> 
> mwait-idle=no
> 
> Rebooted.
> Result: hung:

Thanks. I'm afraid I'm out of ideas for the moment.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 3/27/2019 7:21 AM, Jan Beulich wrote:
>>>> On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
>> On 3/27/2019 1:14 AM, Jan Beulich wrote:
>>>>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
>>>> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
>>>> # CONFIG_HVM is not set
>>>> zeta /usr/local/src/xen #
>>>>
>>>> # tried 2 boot attempts
>>>> log at: https://pastebin.com/nL4BWJ6Y
>>>>
>>>> Hang points at lines:
>>> Thanks for trying anyway; one further possibility eliminated. Looking
>>> at the logs I've had another thought (wild guess again, so not really
>>> much hope): Could you try "mwait-idle=no"?
>>>
>> I modified man_xen.cfg by adding at the end the kernel parameter:
>>
>> mwait-idle=no
>>
>> Rebooted.
>> Result: hung:
> Thanks. I'm afraid I'm out of ideas for the moment.
>
> Jan
>
>
Jan,

Recall, the Xen kernel successfully launched in 2017 when I first built
Xen in Gentoo, that was about version 4.7.1.  I had to launch it
from an EFI console.  I've tried to revert back to 4.7.1 and
build a kernel and I have found it too difficult as certain
dependencies have since been removed from Gentoo.

I've been studying apic.c and the differences between 4.7.1
and HEAD.  Here's a DIFF:

http://quickdiff.net/?unique_id=948598C4-31A2-D028-CE95-F04632C1871A
Create a new one? <https://quickdiff.net/>

I see that currently there is a structure:

static const struct x86_cpu_id __initconstrel deadline_match[] = {

which identifies the microarchitecture, e.g. Haswell, Skylake.

https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/apic.c;h=2a2432619e3edce2cdbc275abbd4e80ffcdcd9f0;hb=HEAD#l1146


Line 1176 has a return if there is a failure to match, yet further down 
if there
is a version mismatch, there is a XENLOG_WARNING:

TSC_DEADLINE disabled due to Errata;
please update microcode to version %#x (or later)


My serveris Atom based, a Supermicro A1SAi-2750F
https://www.supermicro.com/products/motherboard/Atom/X10/A1SAi-2750F.cfm
which has an Intel® Atom™ Processor C2750.

https://ark.intel.com/content/www/us/en/ark/products/77987/intel-atom-processor-c2750-4m-cache-2-40-ghz.html

I believe my CPU chip is from a 22 nanometer fabrication process and
Wikipedia tells me that accordingly, the microarchitecture is Silvermont.
https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchitectures#Atom_Lines

Moreover, the Intel documentation confirms this:

"2.1.15 The Intel® Atom™ Processor Family Based on Silvermont 
Microarchitecture (2013)
Intel Atom Processor C2xxx, E3xxx, S1xxx series are based on the
Silvermont microarchitecture. Processors based on the Silvermont
microarchitecture supports instruction set extensions up to and
including SSE4.2, AESNI, and PCLMULQDQ."  from page 2-5 of the
Software Developer’s Manual (below).

I'm wondering if the fact that I was able to boot a kernel under Xen 2.4.7
and the unexplained hanging at boot for 4.7.12+ is related to the fact 
that the
Silvermont architecture is not accounted from in the deadline structure.

sheet 784-5 states that bit 24 "TSC-Deadline" with this description:
"A value of 1 indicates that the processor’s local APIC timer supports
one-shot operation using a TSC deadline value."

from the Intel® 64 and IA-32 Architectures
Software Developer’s Manual
Combined Volumes:
1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4

at:
https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf

Is any of the above bird-dogging helpful or cause you to have an
"Ahah!" moment?

John




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Jan Beulich]

>>> On 16.04.19 at 18:23, <jlpoole56@gmail.com> wrote:

> On 3/27/2019 7:21 AM, Jan Beulich wrote:
>>>>> On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
>>> On 3/27/2019 1:14 AM, Jan Beulich wrote:
>>>>>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
>>>>> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
>>>>> # CONFIG_HVM is not set
>>>>> zeta /usr/local/src/xen #
>>>>>
>>>>> # tried 2 boot attempts
>>>>> log at: https://pastebin.com/nL4BWJ6Y 
>>>>>
>>>>> Hang points at lines:
>>>> Thanks for trying anyway; one further possibility eliminated. Looking
>>>> at the logs I've had another thought (wild guess again, so not really
>>>> much hope): Could you try "mwait-idle=no"?
>>>>
>>> I modified man_xen.cfg by adding at the end the kernel parameter:
>>>
>>> mwait-idle=no
>>>
>>> Rebooted.
>>> Result: hung:
>> Thanks. I'm afraid I'm out of ideas for the moment.
>>
>> Jan
>>
>>
> Jan,
> 
> Recall, the Xen kernel successfully launched in 2017 when I first built
> Xen in Gentoo, that was about version 4.7.1.  I had to launch it
> from an EFI console.  I've tried to revert back to 4.7.1 and
> build a kernel and I have found it too difficult as certain
> dependencies have since been removed from Gentoo.
> 
> I've been studying apic.c and the differences between 4.7.1
> and HEAD.  Here's a DIFF:
> 
> http://quickdiff.net/?unique_id=948598C4-31A2-D028-CE95-F04632C1871A 
> Create a new one? <https://quickdiff.net/>
> 
> I see that currently there is a structure:
> 
> static const struct x86_cpu_id __initconstrel deadline_match[] = {
> 
> which identifies the microarchitecture, e.g. Haswell, Skylake.
> 
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/apic.c;h=2a2 
> 432619e3edce2cdbc275abbd4e80ffcdcd9f0;hb=HEAD#l1146
> 
> 
> Line 1176 has a return if there is a failure to match, yet further down 
> if there
> is a version mismatch, there is a XENLOG_WARNING:
> 
> TSC_DEADLINE disabled due to Errata;
> please update microcode to version %#x (or later)
> 
> 
> My serveris Atom based, a Supermicro A1SAi-2750F
> https://www.supermicro.com/products/motherboard/Atom/X10/A1SAi-2750F.cfm 
> which has an Intel® Atom™ Processor C2750.
> 
> https://ark.intel.com/content/www/us/en/ark/products/77987/intel-atom-proces 
> sor-c2750-4m-cache-2-40-ghz.html
> 
> I believe my CPU chip is from a 22 nanometer fabrication process and
> Wikipedia tells me that accordingly, the microarchitecture is Silvermont.
> https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchitectures#Atom_Line 
> s
> 
> Moreover, the Intel documentation confirms this:
> 
> "2.1.15 The Intel® Atom™ Processor Family Based on Silvermont 
> Microarchitecture (2013)
> Intel Atom Processor C2xxx, E3xxx, S1xxx series are based on the
> Silvermont microarchitecture. Processors based on the Silvermont
> microarchitecture supports instruction set extensions up to and
> including SSE4.2, AESNI, and PCLMULQDQ."  from page 2-5 of the
> Software Developer’s Manual (below).
> 
> I'm wondering if the fact that I was able to boot a kernel under Xen 2.4.7
> and the unexplained hanging at boot for 4.7.12+ is related to the fact 
> that the
> Silvermont architecture is not accounted from in the deadline structure.
> 
> sheet 784-5 states that bit 24 "TSC-Deadline" with this description:
> "A value of 1 indicates that the processor’s local APIC timer supports
> one-shot operation using a TSC deadline value."
> 
> from the Intel® 64 and IA-32 Architectures
> Software Developer’s Manual
> Combined Volumes:
> 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4
> 
> at:
> https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol- 
> 1-2abcd-3abcd.pdf
> 
> Is any of the above bird-dogging helpful or cause you to have an
> "Ahah!" moment?

Not really, no. It's not clear to me where the TSC deadline
connection comes from, but if you suspect something then it
would be helpful if you pointed out the respective erratum
for the specific CPU model you use, or if you simply
suppressed use of the deadline timer by using the respective
command line option ("tdt=0").

Beyond that I'm afraid it'll take someone else to have an idea,
or someone to be able to actually debug the issue on a system
where the issue surfaces.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Tue, Apr 16, 2019 at 09:23:11AM -0700, John L. Poole wrote:
> 
> On 3/27/2019 7:21 AM, Jan Beulich wrote:
> > > > > On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
> > > On 3/27/2019 1:14 AM, Jan Beulich wrote:
> > > > > > > On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
> > > > > zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
> > > > > # CONFIG_HVM is not set
> > > > > zeta /usr/local/src/xen #
> > > > > 
> > > > > # tried 2 boot attempts
> > > > > log at: https://pastebin.com/nL4BWJ6Y
> > > > > 
> > > > > Hang points at lines:
> > > > Thanks for trying anyway; one further possibility eliminated. Looking
> > > > at the logs I've had another thought (wild guess again, so not really
> > > > much hope): Could you try "mwait-idle=no"?
> > > > 
> > > I modified man_xen.cfg by adding at the end the kernel parameter:
> > > 
> > > mwait-idle=no
> > > 
> > > Rebooted.
> > > Result: hung:
> > Thanks. I'm afraid I'm out of ideas for the moment.
> > 
> > Jan
> > 
> > 
> Jan,
> 
> Recall, the Xen kernel successfully launched in 2017 when I first built
> Xen in Gentoo, that was about version 4.7.1.  I had to launch it
> from an EFI console.  I've tried to revert back to 4.7.1 and
> build a kernel and I have found it too difficult as certain
> dependencies have since been removed from Gentoo.

Unless some of us can get our hands on one of such systems I think
your best bet would be to try to bisect the changes between 4.7 and
4.8 if 4.7 was indeed working on your system.

Note that you only need to build the Xen hypervisor (make xen) in
order to bisect this issue, there's no need to build the tools at
all, which is where almost all of the dependencies come from. The
hypervisor is mostly standalone and only have dependencies on a
couple of tools.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

[-- Attachment #1.1: Type: text/plain, Size: 2962 bytes --]


On 4/29/2019 5:02 AM, Roger Pau Monné wrote:
> On Tue, Apr 16, 2019 at 09:23:11AM -0700, John L. Poole wrote:
>> On 3/27/2019 7:21 AM, Jan Beulich wrote:
>>>>>> On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
>>>> On 3/27/2019 1:14 AM, Jan Beulich wrote:
>>>>>>>> On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
>>>>>> zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
>>>>>> # CONFIG_HVM is not set
>>>>>> zeta /usr/local/src/xen #
>>>>>>
>>>>>> # tried 2 boot attempts
>>>>>> log at: https://pastebin.com/nL4BWJ6Y
>>>>>>
>>>>>> Hang points at lines:
>>>>> Thanks for trying anyway; one further possibility eliminated. Looking
>>>>> at the logs I've had another thought (wild guess again, so not really
>>>>> much hope): Could you try "mwait-idle=no"?
>>>>>
>>>> I modified man_xen.cfg by adding at the end the kernel parameter:
>>>>
>>>> mwait-idle=no
>>>>
>>>> Rebooted.
>>>> Result: hung:
>>> Thanks. I'm afraid I'm out of ideas for the moment.
>>>
>>> Jan
>>>
>>>
>> Jan,
>>
>> Recall, the Xen kernel successfully launched in 2017 when I first built
>> Xen in Gentoo, that was about version 4.7.1.  I had to launch it
>> from an EFI console.  I've tried to revert back to 4.7.1 and
>> build a kernel and I have found it too difficult as certain
>> dependencies have since been removed from Gentoo.
> Unless some of us can get our hands on one of such systems I think
> your best bet would be to try to bisect the changes between 4.7 and
> 4.8 if 4.7 was indeed working on your system.
>
> Note that you only need to build the Xen hypervisor (make xen) in
> order to bisect this issue, there's no need to build the tools at
> all, which is where almost all of the dependencies come from. The
> hypervisor is mostly standalone and only have dependencies on a
> couple of tools.
>
> Roger.
Thank you, Roger.

In Gentoo, there are two packages of xen and xen-tools which
tend to progress with the same release number. When
I try to go back an install a 4.7.x set, my compiler errors out
in the xen-tools  with:
"directive writing up to 39 bytes into a region of size between 17 and 37 "
so I suspect the advancement of my compiler is finding problems
my compiler of 2 years ago could not.

I'll try to if I can modify a xen 4.7.x package pointing to a recent 
xen-tools,
e.g. 4.12.0.  If that doesn't work, I'll try using a xen source from your
repository and hand compile.

Yes, be assured I did, at one time, build xen on this machine as I have
a couple of VMs on there that I depend upon and I have to confess
that during my upgrade I did not take the step to back-up the
working xen kernel.  So now I am effectively locked out from my
day-to-day resources on my atom VMs which has compounded the
exasperation I experience.  have been without my working data
set locked up in a Gentoo kernel VM because I cannot launch Dom0.
A very painful lesson learned: always back up the working kernel
and EFI files before undertaking an upgrade.

[-- Attachment #1.2: Type: text/html, Size: 6834 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Mon, Apr 29, 2019 at 08:08:33AM -0700, John L. Poole wrote:
> 
> On 4/29/2019 5:02 AM, Roger Pau Monné wrote:
> > On Tue, Apr 16, 2019 at 09:23:11AM -0700, John L. Poole wrote:
> > > On 3/27/2019 7:21 AM, Jan Beulich wrote:
> > > > > > > On 27.03.19 at 14:25, <jlpoole56@gmail.com> wrote:
> > > > > On 3/27/2019 1:14 AM, Jan Beulich wrote:
> > > > > > > > > On 26.03.19 at 18:21, <jlpoole56@gmail.com> wrote:
> > > > > > > zeta /usr/local/src/xen # cat xen/.config |grep CONFIG_HVM
> > > > > > > # CONFIG_HVM is not set
> > > > > > > zeta /usr/local/src/xen #
> > > > > > > 
> > > > > > > # tried 2 boot attempts
> > > > > > > log at: https://pastebin.com/nL4BWJ6Y
> > > > > > > 
> > > > > > > Hang points at lines:
> > > > > > Thanks for trying anyway; one further possibility eliminated. Looking
> > > > > > at the logs I've had another thought (wild guess again, so not really
> > > > > > much hope): Could you try "mwait-idle=no"?
> > > > > > 
> > > > > I modified man_xen.cfg by adding at the end the kernel parameter:
> > > > > 
> > > > > mwait-idle=no
> > > > > 
> > > > > Rebooted.
> > > > > Result: hung:
> > > > Thanks. I'm afraid I'm out of ideas for the moment.
> > > > 
> > > > Jan
> > > > 
> > > > 
> > > Jan,
> > > 
> > > Recall, the Xen kernel successfully launched in 2017 when I first built
> > > Xen in Gentoo, that was about version 4.7.1.  I had to launch it
> > > from an EFI console.  I've tried to revert back to 4.7.1 and
> > > build a kernel and I have found it too difficult as certain
> > > dependencies have since been removed from Gentoo.
> > Unless some of us can get our hands on one of such systems I think
> > your best bet would be to try to bisect the changes between 4.7 and
> > 4.8 if 4.7 was indeed working on your system.
> > 
> > Note that you only need to build the Xen hypervisor (make xen) in
> > order to bisect this issue, there's no need to build the tools at
> > all, which is where almost all of the dependencies come from. The
> > hypervisor is mostly standalone and only have dependencies on a
> > couple of tools.
> > 
> > Roger.
> Thank you, Roger.
> 
> In Gentoo, there are two packages of xen and xen-tools which
> tend to progress with the same release number. When
> I try to go back an install a 4.7.x set, my compiler errors out
> in the xen-tools  with:
> "directive writing up to 39 bytes into a region of size between 17 and 37 "
> so I suspect the advancement of my compiler is finding problems
> my compiler of 2 years ago could not.
> 
> I'll try to if I can modify a xen 4.7.x package pointing to a recent
> xen-tools,
> e.g. 4.12.0.  If that doesn't work, I'll try using a xen source from your
> repository and hand compile.

IMO it would be better if you can build directly from the upstream git
repository [0], that way you could use git-bisect(1) in order to figure
out which commit broke your system. For example:

# git clone git://xenbits.xen.org/xen.git
# cd xen
# git checkout RELEASE-4.7.0
# make xen -j8

That should give you a set of Xen binaries in the xen/ directory, IIRC
you are booting from EFI so you likely need xen/xen.efi.

If that works, then you can test RELEASE-4.8.0 and if that fails to
boot you should have a range of commits that you can bisect in order
to find the culprit.

Thanks, Roger.

[0] http://xenbits.xen.org/gitweb/?p=xen.git;a=summary

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
> IMO it would be better if you can build directly from the upstream git
> repository [0], that way you could use git-bisect(1) in order to figure
> out which commit broke your system. For example:
> 
> # git clone git://xenbits.xen.org/xen.git
> # cd xen
> # git checkout RELEASE-4.7.0
> # make xen -j8
> 
> That should give you a set of Xen binaries in the xen/ directory, IIRC
> you are booting from EFI so you likely need xen/xen.efi.
> 
> If that works, then you can test RELEASE-4.8.0 and if that fails to
> boot you should have a range of commits that you can bisect in order
> to find the culprit.

FWIW, I've been unable to find a box with the same CPU model (C2750)
that you are using. I've found a couple of old Atom boxes using
different CPUs but they all seem to boot fine using latest
xen-unstable. I've looked on eBay for that CPU but everything
containing it is server-grade and >200$ which I'm sadly not going to
pay.

Unless you are able to bisect the tree and give us the bad commit
that's causing your issues I'm afraid at least myself I won't be able
to progress this any further, sorry.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

[-- Attachment #1.1: Type: text/plain, Size: 1910 bytes --]


On 5/27/2019 9:18 AM, Roger Pau Monné wrote:
> On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
>> IMO it would be better if you can build directly from the upstream git
>> repository [0], that way you could use git-bisect(1) in order to figure
>> out which commit broke your system. For example:
>>
>> # git clone git://xenbits.xen.org/xen.git
>> # cd xen
>> # git checkout RELEASE-4.7.0
>> # make xen -j8
>>
>> That should give you a set of Xen binaries in the xen/ directory, IIRC
>> you are booting from EFI so you likely need xen/xen.efi.
>>
>> If that works, then you can test RELEASE-4.8.0 and if that fails to
>> boot you should have a range of commits that you can bisect in order
>> to find the culprit.
> FWIW, I've been unable to find a box with the same CPU model (C2750)
> that you are using. I've found a couple of old Atom boxes using
> different CPUs but they all seem to boot fine using latest
> xen-unstable. I've looked on eBay for that CPU but everything
> containing it is server-grade and >200$ which I'm sadly not going to
> pay.
>
> Unless you are able to bisect the tree and give us the bad commit
> that's causing your issues I'm afraid at least myself I won't be able
> to progress this any further, sorry.
>
> Roger.
I attempted to work backwards and ran into a nightmare with Gentoo.   I kept
getting compiler errors which I suspect was a result of having a newer 
version
of GCC and other things.  It's not an easy thing to travel
back in time in Gentoo because everything keeps getting upgraded.  I just
cannot make the time now to unravel this as I have some demands on my time
and will be engaged for the next four to six weeks.

How much would it cost for you to obtain the machine you need? I may
consider paying for it. I bought this Atom server just to economically run
Xen so the machine has marginal value to me if I cannot run Xen on it.

John



[-- Attachment #1.2: Type: text/html, Size: 2964 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[Roger Pau Monné]

On Mon, May 27, 2019 at 03:35:21PM -0700, John L. Poole wrote:
> 
> On 5/27/2019 9:18 AM, Roger Pau Monné wrote:
> > On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
> > > IMO it would be better if you can build directly from the upstream git
> > > repository [0], that way you could use git-bisect(1) in order to figure
> > > out which commit broke your system. For example:
> > > 
> > > # git clone git://xenbits.xen.org/xen.git
> > > # cd xen
> > > # git checkout RELEASE-4.7.0
> > > # make xen -j8
> > > 
> > > That should give you a set of Xen binaries in the xen/ directory, IIRC
> > > you are booting from EFI so you likely need xen/xen.efi.
> > > 
> > > If that works, then you can test RELEASE-4.8.0 and if that fails to
> > > boot you should have a range of commits that you can bisect in order
> > > to find the culprit.
> > FWIW, I've been unable to find a box with the same CPU model (C2750)
> > that you are using. I've found a couple of old Atom boxes using
> > different CPUs but they all seem to boot fine using latest
> > xen-unstable. I've looked on eBay for that CPU but everything
> > containing it is server-grade and >200$ which I'm sadly not going to
> > pay.
> > 
> > Unless you are able to bisect the tree and give us the bad commit
> > that's causing your issues I'm afraid at least myself I won't be able
> > to progress this any further, sorry.
> > 
> > Roger.
> I attempted to work backwards and ran into a nightmare with Gentoo.   I kept
> getting compiler errors which I suspect was a result of having a newer
> version
> of GCC and other things.  It's not an easy thing to travel
> back in time in Gentoo because everything keeps getting upgraded.  I just
> cannot make the time now to unravel this as I have some demands on my time
> and will be engaged for the next four to six weeks.

IMO your best bet is to build Xen using Debian stretch, that's used by
the Xen test system, and is likely to be able to build the different
Xen versions, stable-* branches tested by osstest should build on
stretch.

What I've done in the past if that also triggers compiler errors is to
build a chroot with an older version of Debian and then build Xen
inside of it. You can do this in a box different from the one you are
testing, ie: you could create a Debian VM and build Xen from there.

Note that in order to bisect this issue you only need to build the Xen
kernel (make xen, no need to run ./configure), there's no need to
build the tools, hence you need almost no dependencies installed on
the builder.

I've done a build of the stable-4.7 branch myself and uploaded the
hypervisor binaries to:

http://xenbits.xen.org/people/royger/stable-4.7/

Could you give those a try (I wasn't sure whether you need xen.gz or
xen.efi so I've uploaded both) and see if you still have issues
booting?

Testing those binaries should be as simple as placing them in /boot/
and fixing your bootloader configuration to boot from those. Please
send the serial log when booting from the provided binaries.

> How much would it cost for you to obtain the machine you need? I may
> consider paying for it. I bought this Atom server just to economically run
> Xen so the machine has marginal value to me if I cannot run Xen on it.

Even if we go that route, there's no guarantee that I would be able to
fix the issue, and there's also the possibility that the hardware you
have is somehow broken, and that the new one won't exhibit this issue.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

[-- Attachment #1.1: Type: text/plain, Size: 19822 bytes --]


On 5/28/2019 12:41 AM, Roger Pau Monné wrote:
> On Mon, May 27, 2019 at 03:35:21PM -0700, John L. Poole wrote:
>> On 5/27/2019 9:18 AM, Roger Pau Monné wrote:
>>> On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
>>>> IMO it would be better if you can build directly from the upstream git
>>>> repository [0], that way you could use git-bisect(1) in order to figure
>>>> out which commit broke your system. For example:
>>>>
>>>> # git clone git://xenbits.xen.org/xen.git
>>>> # cd xen
>>>> # git checkout RELEASE-4.7.0
>>>> # make xen -j8
>>>>
>>>> That should give you a set of Xen binaries in the xen/ directory, IIRC
>>>> you are booting from EFI so you likely need xen/xen.efi.
>>>>
>>>> If that works, then you can test RELEASE-4.8.0 and if that fails to
>>>> boot you should have a range of commits that you can bisect in order
>>>> to find the culprit.
>>> FWIW, I've been unable to find a box with the same CPU model (C2750)
>>> that you are using. I've found a couple of old Atom boxes using
>>> different CPUs but they all seem to boot fine using latest
>>> xen-unstable. I've looked on eBay for that CPU but everything
>>> containing it is server-grade and >200$ which I'm sadly not going to
>>> pay.
>>>
>>> Unless you are able to bisect the tree and give us the bad commit
>>> that's causing your issues I'm afraid at least myself I won't be able
>>> to progress this any further, sorry.
>>>
>>> Roger.
>> I attempted to work backwards and ran into a nightmare with Gentoo.   I kept
>> getting compiler errors which I suspect was a result of having a newer
>> version
>> of GCC and other things.  It's not an easy thing to travel
>> back in time in Gentoo because everything keeps getting upgraded.  I just
>> cannot make the time now to unravel this as I have some demands on my time
>> and will be engaged for the next four to six weeks.
> IMO your best bet is to build Xen using Debian stretch, that's used by
> the Xen test system, and is likely to be able to build the different
> Xen versions, stable-* branches tested by osstest should build on
> stretch.
>
> What I've done in the past if that also triggers compiler errors is to
> build a chroot with an older version of Debian and then build Xen
> inside of it. You can do this in a box different from the one you are
> testing, ie: you could create a Debian VM and build Xen from there.
>
> Note that in order to bisect this issue you only need to build the Xen
> kernel (make xen, no need to run ./configure), there's no need to
> build the tools, hence you need almost no dependencies installed on
> the builder.
>
> I've done a build of the stable-4.7 branch myself and uploaded the
> hypervisor binaries to:
>
> http://xenbits.xen.org/people/royger/stable-4.7/
>
> Could you give those a try (I wasn't sure whether you need xen.gz or
> xen.efi so I've uploaded both) and see if you still have issues
> booting?
>
> Testing those binaries should be as simple as placing them in /boot/
> and fixing your bootloader configuration to boot from those. Please
> send the serial log when booting from the provided binaries.
>
>> How much would it cost for you to obtain the machine you need? I may
>> consider paying for it. I bought this Atom server just to economically run
>> Xen so the machine has marginal value to me if I cannot run Xen on it.
> Even if we go that route, there's no guarantee that I would be able to
> fix the issue, and there's also the possibility that the hardware you
> have is somehow broken, and that the new one won't exhibit this issue.
>
> Roger.
I downloaded these files and placed accordingly:

xen.efi => /boot/efi/gentoo/roger0528.efi
xen.gz => /boot/efit/gentoo/roger0528_xen.gz

I created /boot/efi/gentoo/roger0528.cfg which has:

============== BEGIN CFG =================
zeta /home/jlpoole # cat /boot/efi/gentoo/roger0528.cfg
[global]
default=abc

[abc]
options=console=vga,com1 com1=115200,8n1 console_timestamps 
console_to_ring conring_size=64 log_buf_len=16M loglvl=all 
guest_loglvl=all sync_console=true sched_debug iommu=verbose 
apic_verbosity=debug efi=no-rs reboot=kbd cpuinfo watchdog mwait-idle=no

#kernel=xen-4.13-unstable.gz root=/dev/sda4 vga=gfx-1024x768x16  
com1=115200,8n1 console=com1 console_timestamps=date console_to_ring 
conring_size=16k loglvl=all guest_loglvl=all sync_console=true 
iommu=debug apic_verbosity=debug

#
# 4/4/19 jlpoole: trying Gentoo 4.12.0 per request in
# https://bugs.gentoo.org/show_bug.cgi?id=679824
#
kernel=roger0528_xen.gz root=/dev/sda4  vga=gfx-1024x768x16 
com1=115200,8n1 console=com1 console_timestamps=date console_to_ring 
conring_size=16k  loglvl=all guest_loglvl=all sync_console=true 
iommu=debug apic_verbosity=debug

initramfs=initramfs-genkernel-x86_64-4.19.23-gentoo


zeta /home/jlpoole
============== END CFG =================

Here's is the log from my serial console:


============== BEGIN BOOT LOG =================
fs0:\efi\gentoo> roger0528.efi2,655,559  man_xen_gentoo-4.12.0.efi
          23 File(s)  39,839,728 bytes
           3 Dir(s)
Xen 4.7.6 (c/s Wed Feb 27 10:33:42 2019 +0000 git:206d3f65f7) EFI loader
Using configuration file 'roger0528.cfg'
roger0528_xen.gz: 0x000000005ad63000-0x000000005ae49f80
0x0000:0x02:0x00.0x0: ROM: 0x8000 bytes at 0x7c8bc028
  Xen 4.7.6
(XEN) Xen version 4.7.6 (root@) (gcc (Debian 6.3.0-18+deb9u1) 6.3.0 
20170516) debug=n Tue May 28 09:25:50 CEST 2019
(XEN) Latest ChangeSet: Wed Feb 27 10:33:42 2019 +0000 git:206d3f65f7
(XEN) Console output is synchronous.
(XEN) Bootloader: EFI
(XEN) Command line: console=vga,com1 com1=115200,8n1 console_timestamps 
console_to_ring conring_size=64 log_buf_len=16M loglvl=all 
guest_loglvl=all sync_console=true sched_debug iommu=verbose 
apic_verbosity=debug efi=no-rs reboot=kbd cpuinfo watchdog mwait-idle=no
(XEN) Video information:
(XEN)  VGA is graphics mode 800x600, 32 bpp
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) EFI RAM map:
(XEN)  0000000000000000 - 00000000000a0000 (usable)
(XEN)  0000000000100000 - 000000007e16b000 (usable)
(XEN)  000000007e16b000 - 000000007eba2000 (reserved)
(XEN)  000000007eba2000 - 000000007ed0d000 (usable)
(XEN)  000000007ed0d000 - 000000007f28d000 (ACPI NVS)
(XEN)  000000007f28d000 - 000000007f648000 (reserved)
(XEN)  000000007f648000 - 000000007f800000 (usable)
(XEN)  00000000e0000000 - 00000000e4000000 (reserved)
(XEN)  00000000fed01000 - 00000000fed04000 (reserved)
(XEN)  00000000fed08000 - 00000000fed09000 (reserved)
(XEN)  00000000fed0c000 - 00000000fed10000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed1d000 (reserved)
(XEN)  00000000fef00000 - 00000000ff000000 (reserved)
(XEN)  00000000ff800000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000ff0000000 (usable)
(XEN) ACPI: RSDP 7ED55000, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT 7ED55098, 00B4 (r1 ALASKA   A M I   1072009 AMI     10013)
(XEN) ACPI: FACP 7ED58690, 010C (r5 ALASKA   A M I   1072009 AMI     10013)
(XEN) ACPI: DSDT 7ED551E8, 34A1 (r2 ALASKA   A M I   1072009 INTL 20061109)
(XEN) ACPI: FACS 7F28AF80, 0040
(XEN) ACPI: FPDT 7ED587A0, 0044 (r1 ALASKA   A M I   1072009 AMI     10013)
(XEN) ACPI: FIDT 7ED587E8, 009C (r1 ALASKA   A M I   1072009 AMI     10013)
(XEN) ACPI: SPMI 7ED58888, 0040 (r5 ALASKA   A M I         0 AMI.        0)
(XEN) ACPI: MCFG 7ED588C8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: WDAT 7ED58908, 01AC (r1 ALASKA    A M I  1072009 MSFT    10013)
(XEN) ACPI: UEFI 7ED58AB8, 0042 (r1 0             0)
(XEN) ACPI: APIC 7ED58B00, 0098 (r3 INTEL  TIANO           1 MSFT        0)
(XEN) ACPI: BDAT 7ED58B98, 0030 (r1 0             0)
(XEN) ACPI: HPET 7ED58BC8, 0038 (r1 INTEL                  1 MSFT  1000013)
(XEN) ACPI: SSDT 7ED58C00, 09F1 (r1  PmRef    CpuPm     3000 INTL 20061109)
(XEN) ACPI: SSDT 7ED595F8, 0259 (r1  PmRef  Cpu0Tst     3000 INTL 20061109)
(XEN) ACPI: SSDT 7ED59858, 0357 (r1  PmRef    ApTst     3000 INTL 20061109)
(XEN) ACPI: SPCR 7ED59BB0, 0050 (r1  A M I  APTIO V  1072009 AMI.        5)
(XEN) ACPI: HEST 7ED59C00, 00A8 (r1 INTEL  AVOTON B        1 INTL        1)
(XEN) ACPI: BERT 7ED59CA8, 0030 (r1 INTEL  AVOTON B        1 INTL        1)
(XEN) ACPI: ERST 7ED59CD8, 0230 (r1 INTEL  AVOTON B        1 INTL        1)
(XEN) ACPI: EINJ 7ED59F08, 0150 (r1 INTEL  AVOTON B        1 INTL        1)
(XEN) System RAM: 63204MB (64721080kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000ff0000000
(XEN) Domain heap initialised
(XEN) Allocated console ring of 64 KiB.
(XEN) vesafb: framebuffer at 0xde000000, mapped to 0xffff82c000201000, 
using 1920k, total 1920k
(XEN) vesafb: mode is 800x600x32, linelength=3200, font 8x8
(XEN) vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
(XEN) SMBIOS 2.8 present.
(XEN) DMI 2.7 present.
(XEN) APIC boot state is 'xapic'
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 
7f28af80/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[7f28af8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x08] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x0e] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) Xen ERST support is initialized.
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) mapped APIC to ffff82cfffffb000 (fee00000)
(XEN) mapped IOAPIC to ffff82cfffffa000 (fec00000)
(XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) Levelling caps: 0x1
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) CPU: L1 I cache: 32K, L1 D cache: 24K
(XEN) CPU: L2 cache: 1024K
(XEN) CMCI: CPU0 has no CMCI support
(XEN) CPU0: Thermal monitoring enabled (TM1)
(XEN) Intel machine check reporting enabled
(XEN) Unrecognised CPU model 0x4d - assuming not reptpoline safe
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other:
(XEN)   Support for VMs: PV: RSB, HVM: RSB
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2400.528 MHz processor.
(XEN) EFI memory map:
(XEN)  0000000000000-0000000007fff type=3 attr=000000000000000f
(XEN)  0000000008000-000000003cfff type=7 attr=000000000000000f
(XEN)  000000003d000-000000003ffff type=2 attr=000000000000000f
(XEN)  0000000040000-000000009ffff type=3 attr=000000000000000f
(XEN)  0000000100000-00000001fffff type=7 attr=000000000000000f
(XEN)  0000000200000-000000023ffff type=4 attr=000000000000000f
(XEN)  0000000240000-000005ad62fff type=7 attr=000000000000000f
(XEN)  000005ad63000-000005ae53fff type=2 attr=000000000000000f
(XEN)  000005ae54000-000005c110fff type=1 attr=000000000000000f
(XEN)  000005c111000-000007ad5afff type=2 attr=000000000000000f
(XEN)  000007ad5b000-000007ad5dfff type=7 attr=000000000000000f
(XEN)  000007ad5e000-000007ad63fff type=2 attr=000000000000000f
(XEN)  000007ad64000-000007ad65fff type=7 attr=000000000000000f
(XEN)  000007ad66000-000007ad66fff type=2 attr=000000000000000f
(XEN)  000007ad67000-000007ad67fff type=7 attr=000000000000000f
(XEN)  000007ad68000-000007ad6afff type=2 attr=000000000000000f
(XEN)  000007ad6b000-000007ad73fff type=7 attr=000000000000000f
(XEN)  000007ad74000-000007ad7efff type=2 attr=000000000000000f
(XEN)  000007ad7f000-000007db6afff type=4 attr=000000000000000f
(XEN)  000007db6b000-000007df0ffff type=7 attr=000000000000000f
(XEN)  000007df10000-000007e16afff type=3 attr=000000000000000f
(XEN)  000007e16b000-000007eba1fff type=0 attr=000000000000000f
(XEN)  000007eba2000-000007ed0cfff type=7 attr=000000000000000f
(XEN)  000007ed0d000-000007f28cfff type=10 attr=000000000000000f
(XEN)  000007f28d000-000007f5f2fff type=6 attr=800000000000000f
(XEN)  000007f5f3000-000007f647fff type=5 attr=800000000000000f
(XEN)  000007f648000-000007f7fffff type=4 attr=000000000000000f
(XEN)  0000100000000-0000fefffffff type=7 attr=000000000000000f
(XEN)  00000e0000000-00000e3ffffff type=11 attr=8000000000000001
(XEN)  00000fed01000-00000fed03fff type=11 attr=8000000000000001
(XEN)  00000fed08000-00000fed08fff type=11 attr=8000000000000001
(XEN)  00000fed0c000-00000fed0ffff type=11 attr=8000000000000001
(XEN)  00000fed1c000-00000fed1cfff type=11 attr=8000000000000001
(XEN)  00000fef00000-00000feffffff type=11 attr=8000000000000001
(XEN)  00000ff800000-00000ffffffff type=11 attr=8000000000000001
(XEN) Initing memory sharing.
(XEN) alt table ffff82d080652710 -> ffff82d080654060
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) PCI: updated MCFG configuration 0: base e0000000 segment 0 buses 0 
- 63
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) I/O virtualisation disabled
(XEN) CPU0: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz stepping 08
(XEN) nr_sockets: 1
(XEN) Getting VERSION: 50014
(XEN) Getting VERSION: 50014
(XEN) Getting ID: 0
(XEN) Getting LVT0: 700
(XEN) Getting LVT1: 400
(XEN) enabled ExtINT on CPU#0
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 
2-22, 2-23 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #2 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #2......
(XEN) .... register #00: 02000000
(XEN) .......    : physical APIC id: 02
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 001 01  0    0    0   0   0    1    1    28
(XEN)  02 001 01  0    0    0   0   0    1    1    F0
(XEN)  03 001 01  0    0    0   0   0    1    1    30
(XEN)  04 001 01  0    0    0   0   0    1    1    F1
(XEN)  05 001 01  0    0    0   0   0    1    1    38
(XEN)  06 001 01  0    0    0   0   0    1    1    40
(XEN)  07 001 01  0    0    0   0   0    1    1    48
(XEN)  08 001 01  0    0    0   0   0    1    1    50
(XEN)  09 001 01  1    1    0   0   0    1    1    58
(XEN)  0a 001 01  0    0    0   0   0    1    1    60
(XEN)  0b 001 01  0    0    0   0   0    1    1    68
(XEN)  0c 001 01  0    0    0   0   0    1    1    70
(XEN)  0d 001 01  0    0    0   0   0    1    1    78
(XEN)  0e 001 01  0    0    0   0   0    1    1    88
(XEN)  0f 001 01  0    0    0   0   0    1    1    90
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ40 -> 0:1
(XEN) IRQ48 -> 0:3
(XEN) IRQ241 -> 0:4
(XEN) IRQ56 -> 0:5
(XEN) IRQ64 -> 0:6
(XEN) IRQ72 -> 0:7
(XEN) IRQ80 -> 0:8
(XEN) IRQ88 -> 0:9
(XEN) IRQ96 -> 0:10
(XEN) IRQ104 -> 0:11
(XEN) IRQ112 -> 0:12
(XEN) IRQ120 -> 0:13
(XEN) IRQ136 -> 0:14
(XEN) IRQ144 -> 0:15
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) calibrating APIC timer ...
(XEN) ..... CPU clock speed is 2400.0808 MHz.
(XEN) ..... host bus clock speed is 100.0036 MHz.
(XEN) ..... bus_scale = 0x6669
(XEN) TSC deadline timer enabled
(XEN) [2019-05-28 14:24:11] Platform timer appears to have unexpectedly 
wrapped 1 times.
(XEN) [2019-05-28 14:24:11] Platform timer is 14.318MHz HPET
(XEN) [2019-05-28 14:24:11] mwait-idle: disabled
(XEN) [2019-05-28 14:24:11] VMX: Supported advanced features:
(XEN) [2019-05-28 14:24:11]  - APIC MMIO access virtualisation
(XEN) [2019-05-28 14:24:11]  - APIC TPR shadow
(XEN) [2019-05-28 14:24:11]  - Extended Page Tables (EPT)
(XEN) [2019-05-28 14:24:11]  - Virtual-Processor Identifiers (VPID)
(XEN) [2019-05-28 14:24:11]  - Virtual NMI
(XEN) [2019-05-28 14:24:11]  - MSR direct-access bitmap
(XEN) [2019-05-28 14:24:11]  - Unrestricted Guest
(XEN) [2019-05-28 14:24:11]  - VM Functions
(XEN) [2019-05-28 14:24:11] HVM: ASIDs enabled.
(XEN) [2019-05-28 14:24:11] HVM: VMX enabled
(XEN) [2019-05-28 14:24:11] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2019-05-28 14:24:11] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2019-05-28 14:24:11] Booting processor 1/2 eip 3d000
(XEN) [2019-05-28 14:24:11] Initializing CPU#1
(XEN) [2019-05-28 14:24:11] masked ExtINT on CPU#1
(XEN) [2019-05-28 14:24:11] CPU: Physical Processor ID: 0
(XEN) [2019-05-28 14:24:12] CPU: Processor Core ID: 1
(XEN) [2019-05-28 14:24:12] CPU: L1 I cache: 32K, L1 D cache: 24K
(XEN) [2019-05-28 14:24:12] CPU: L2 cache: 1024K
(XEN) [2019-05-28 14:24:12] CMCI: CPU1 has no CMCI support
(XEN) [2019-05-28 14:24:12] CPU1: Thermal monitoring enabled (TM1)
(XEN) [2019-05-28 14:24:12] CPU1: Intel(R) Atom(TM) CPU C2750  @ 2.40GHz 
stepping 08
(XEN) [2019-05-28 14:24:12] Booting processor 2/4 eip 3d000

[hangs]
============== END BOOT LOG =================

Note:  My procedure is to start the machine, then drop into an EFI shell,
navigate to the /boot/efi/gentoo directory and then launch the efi with:

      roger0528.efi

The above log has some distortion of character overlap at the top due
to a misalignment of the serial console (PuTTY) vs. output.

Thank you,
John

[-- Attachment #1.2: Type: text/html, Size: 29287 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

[-- Attachment #1.1: Type: text/plain, Size: 4780 bytes --]


On 5/28/2019 12:41 AM, Roger Pau Monné wrote:
> On Mon, May 27, 2019 at 03:35:21PM -0700, John L. Poole wrote:
>> On 5/27/2019 9:18 AM, Roger Pau Monné wrote:
>>> On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
>>>> IMO it would be better if you can build directly from the upstream git
>>>> repository [0], that way you could use git-bisect(1) in order to figure
>>>> out which commit broke your system. For example:
>>>>
>>>> # git clone git://xenbits.xen.org/xen.git
>>>> # cd xen
>>>> # git checkout RELEASE-4.7.0
>>>> # make xen -j8
>>>>
>>>> That should give you a set of Xen binaries in the xen/ directory, IIRC
>>>> you are booting from EFI so you likely need xen/xen.efi.
>>>>
>>>> If that works, then you can test RELEASE-4.8.0 and if that fails to
>>>> boot you should have a range of commits that you can bisect in order
>>>> to find the culprit.
>>> FWIW, I've been unable to find a box with the same CPU model (C2750)
>>> that you are using. I've found a couple of old Atom boxes using
>>> different CPUs but they all seem to boot fine using latest
>>> xen-unstable. I've looked on eBay for that CPU but everything
>>> containing it is server-grade and >200$ which I'm sadly not going to
>>> pay.
>>>
>>> Unless you are able to bisect the tree and give us the bad commit
>>> that's causing your issues I'm afraid at least myself I won't be able
>>> to progress this any further, sorry.
>>>
>>> Roger.
>> I attempted to work backwards and ran into a nightmare with Gentoo.   I kept
>> getting compiler errors which I suspect was a result of having a newer
>> version
>> of GCC and other things.  It's not an easy thing to travel
>> back in time in Gentoo because everything keeps getting upgraded.  I just
>> cannot make the time now to unravel this as I have some demands on my time
>> and will be engaged for the next four to six weeks.
> IMO your best bet is to build Xen using Debian stretch, that's used by
> the Xen test system, and is likely to be able to build the different
> Xen versions, stable-* branches tested by osstest should build on
> stretch.
>
> What I've done in the past if that also triggers compiler errors is to
> build a chroot with an older version of Debian and then build Xen
> inside of it. You can do this in a box different from the one you are
> testing, ie: you could create a Debian VM and build Xen from there.
>
> Note that in order to bisect this issue you only need to build the Xen
> kernel (make xen, no need to run ./configure), there's no need to
> build the tools, hence you need almost no dependencies installed on
> the builder.
>
> I've done a build of the stable-4.7 branch myself and uploaded the
> hypervisor binaries to:
>
> http://xenbits.xen.org/people/royger/stable-4.7/
>
> Could you give those a try (I wasn't sure whether you need xen.gz or
> xen.efi so I've uploaded both) and see if you still have issues
> booting?
>
> Testing those binaries should be as simple as placing them in /boot/
> and fixing your bootloader configuration to boot from those. Please
> send the serial log when booting from the provided binaries.
>
>> How much would it cost for you to obtain the machine you need? I may
>> consider paying for it. I bought this Atom server just to economically run
>> Xen so the machine has marginal value to me if I cannot run Xen on it.
> Even if we go that route, there's no guarantee that I would be able to
> fix the issue, and there's also the possibility that the hardware you
> have is somehow broken, and that the new one won't exhibit this issue.
>
> Roger.
Roger,

You have given me an idea.  I have several VMs on my hard disk that are not
backed up.  So, I think what I'll do is remove the current hard disk and 
place
a fresh hard disk in and then try to install a Debian based Xen anew so I
do not risk altering my Gentoo-based hard disk.  This approach should free
me from the entanglement of a bleeding edge distribution, e.g. Gentoo.

I was looking back at my notes.  I acquired this Atom-based server in 
November
of 2016 and installed the Debian Xen to test and it worked.  So I then 
installed
Gentoo and ran into problems with GRUB.  I learned that GRUB was not yet 
ready
to support EFI and Xen, so I used the manual method to drop into an EFI 
shell
and launch my DOM0 instance.  I later tried to upgrade the kernel and 
ran into
problems and aborted an upgrade, I just kept what I had working since I had
already created some Gentoo-based VMs.  During my build process, I had
run into an issue "coff-x86-64 pe-x86-64" which Jan Beulich had assisted 
on and
determined was something worth of the attention of the "binutils folks."

I'll attempt the hard disk swap in a few days after I receive a shipment 
of the new disk.

Thank you,
John



[-- Attachment #1.2: Type: text/html, Size: 8074 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen 4.12.0-rc Hangs Around masked ExtINT on CPU#

[John L. Poole]

On 5/28/2019 8:02 AM, John L. Poole wrote:
>
>
> On 5/28/2019 12:41 AM, Roger Pau Monné wrote:
>> On Mon, May 27, 2019 at 03:35:21PM -0700, John L. Poole wrote:
>>> On 5/27/2019 9:18 AM, Roger Pau Monné wrote:
>>>> On Mon, Apr 29, 2019 at 05:27:34PM +0200, Roger Pau Monné wrote:
>>>>> IMO it would be better if you can build directly from the upstream git
>>>>> repository [0], that way you could use git-bisect(1) in order to figure
>>>>> out which commit broke your system. For example:
>>>>>
>>>>> # git clone git://xenbits.xen.org/xen.git
>>>>> # cd xen
>>>>> # git checkout RELEASE-4.7.0
>>>>> # make xen -j8
>>>>>
>>>>> That should give you a set of Xen binaries in the xen/ directory, IIRC
>>>>> you are booting from EFI so you likely need xen/xen.efi.
>>>>>
>>>>> If that works, then you can test RELEASE-4.8.0 and if that fails to
>>>>> boot you should have a range of commits that you can bisect in order
>>>>> to find the culprit.
>>>> FWIW, I've been unable to find a box with the same CPU model (C2750)
>>>> that you are using. I've found a couple of old Atom boxes using
>>>> different CPUs but they all seem to boot fine using latest
>>>> xen-unstable. I've looked on eBay for that CPU but everything
>>>> containing it is server-grade and >200$ which I'm sadly not going to
>>>> pay.
>>>>
>>>> Unless you are able to bisect the tree and give us the bad commit
>>>> that's causing your issues I'm afraid at least myself I won't be able
>>>> to progress this any further, sorry.
>>>>
>>>> Roger.
>>> I attempted to work backwards and ran into a nightmare with Gentoo.   I kept
>>> getting compiler errors which I suspect was a result of having a newer
>>> version
>>> of GCC and other things.  It's not an easy thing to travel
>>> back in time in Gentoo because everything keeps getting upgraded.  I just
>>> cannot make the time now to unravel this as I have some demands on my time
>>> and will be engaged for the next four to six weeks.
>> IMO your best bet is to build Xen using Debian stretch, that's used by
>> the Xen test system, and is likely to be able to build the different
>> Xen versions, stable-* branches tested by osstest should build on
>> stretch.
>>
>> What I've done in the past if that also triggers compiler errors is to
>> build a chroot with an older version of Debian and then build Xen
>> inside of it. You can do this in a box different from the one you are
>> testing, ie: you could create a Debian VM and build Xen from there.
>>
>> Note that in order to bisect this issue you only need to build the Xen
>> kernel (make xen, no need to run ./configure), there's no need to
>> build the tools, hence you need almost no dependencies installed on
>> the builder.
>>
>> I've done a build of the stable-4.7 branch myself and uploaded the
>> hypervisor binaries to:
>>
>> http://xenbits.xen.org/people/royger/stable-4.7/
>>
>> Could you give those a try (I wasn't sure whether you need xen.gz or
>> xen.efi so I've uploaded both) and see if you still have issues
>> booting?
>>
>> Testing those binaries should be as simple as placing them in /boot/
>> and fixing your bootloader configuration to boot from those. Please
>> send the serial log when booting from the provided binaries.
>>
>>> How much would it cost for you to obtain the machine you need? I may
>>> consider paying for it. I bought this Atom server just to economically run
>>> Xen so the machine has marginal value to me if I cannot run Xen on it.
>> Even if we go that route, there's no guarantee that I would be able to
>> fix the issue, and there's also the possibility that the hardware you
>> have is somehow broken, and that the new one won't exhibit this issue.
>>
>> Roger.
> Roger,
>
> You have given me an idea.  I have several VMs on my hard disk that 
> are not
> backed up.  So, I think what I'll do is remove the current hard disk 
> and place
> a fresh hard disk in and then try to install a Debian based Xen anew so I
> do not risk altering my Gentoo-based hard disk.  This approach should free
> me from the entanglement of a bleeding edge distribution, e.g. Gentoo.
>
> I was looking back at my notes.  I acquired this Atom-based server in 
> November
> of 2016 and installed the Debian Xen to test and it worked.  So I then 
> installed
> Gentoo and ran into problems with GRUB.  I learned that GRUB was not 
> yet ready
> to support EFI and Xen, so I used the manual method to drop into an 
> EFI shell
> and launch my DOM0 instance.  I later tried to upgrade the kernel and 
> ran into
> problems and aborted an upgrade, I just kept what I had working since 
> I had
> already created some Gentoo-based VMs.  During my build process, I had
> run into an issue "coff-x86-64 pe-x86-64" which Jan Beulich had 
> assisted on and
> determined was something worth of the attention of the "binutils folks."
>
> I'll attempt the hard disk swap in a few days after I receive a 
> shipment of the new disk.
>
> Thank you,
> John
>
>
Update (9/25/2019).

Short version: Windows wireless USB keyboard hardware incompatibility 
caused the problem.

The Take-away: a USB keyboard can affect the boot for the xen kernel

This was a hardware caused problem.

Long version:

I had several critical matters that I could not postpone so my work
on this was suspended since May.  I finally had time to resume work on 
this problem.  Recall,
I could successfully boot a Gentoo kernel, but when I tried a Xen 
kernel, the system would
hand early on at the masking of the CPUs.

By chance, I decided to swap out the USB keyboard "Microsoft Wireless 
Desktop Receiver 3.1" model: 1028,
because I had to keep replacing batteries and the range was very 
limited, e.g. 15",
and characters were dropping out.  I replaced it with a generic Amazon 
USB keyboard.
Suddenly the boot problems went away: no more hanging at the CPU masking 
point.

I sailed throught and successfully booted.  Moreover, I had placed in a 
new hard disk
in the server, disengaged the exsting one, and installed the
Debian version, 8.6.0 of 11/8/2016, I first used to test this server so 
I had an apples-to-apples
test case before I returned this for service under warranty, and the 
installation while occurring,
had video artifacts the prohibited the graphic install and dropped me 
into a console install
with colorations that caused invisible selections. After I installed the 
Debian 8.6.0, I had the
same problem -- I could not get past the "masked ExtINT on CPU#..."

Since this discovery several days ago, I have booted my various xen 
kernels (in EFI) and have
not encountered any of the problems I previously suffered. While I do 
have some other issues
that relate to Gentoo specific tweaks, I am not concerned and I wanted 
to close this issue
by reporting this discovery.  Of course, I can make available the USB 
unit to qualified persons if
they want to test or I can affix it to the server to test a debugging 
version.

Thank you Roger and Jan and others for all your help.

Related bug: https://bugs.gentoo.org/679826


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel