Re: [Xen-devel] Latest development (master) Xen fails to boot on HP ProLiant DL20 GEN10

[Roman Shaposhnik <roman@zededa.com>]

[-- Attachment #1: Type: text/plain, Size: 3822 bytes --]

Btw, forgot to attach the patch with maxcpus=2 -- interestingly enough
Xen seems to hang much further down than before (basically after
attempting to build out Dom0)

On Mon, Sep 30, 2019 at 3:16 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Mon, Sep 30, 2019 at 2:56 AM Jan Beulich <jbeulich@suse.com> wrote:
> >
> > On 28.09.2019 05:07, Roman Shaposhnik wrote:
> > > On Thu, Sep 26, 2019 at 12:44 AM Jan Beulich <jbeulich@suse.com> wrote:
> > >>
> > >> On 26.09.2019 00:31, Roman Shaposhnik wrote:
> > >>> Jan, Roger, thank you so much for the initial ideas. I tried a few of
> > >>> those and here's where I am.
> > >>>
> > >>> First of all, it is definitely related to CPU bring up. Adding
> > >>> cpuidle=0 to xen command line made Xen boot.
> > >>>
> > >>> Then, a good friend of mine (who you may know from ancient Xen days
> > >>> ;-)) suggested that this could be related to this:
> > >>>      https://wiki.xenproject.org/wiki/Xen_power_management
> > >>> so I went to the BIOS settings and quite to my surprise all of them
> > >>> were grayed out (not tweakable).
> > >>>
> > >>> The only one that wasn't was 2xAPIC support. So just for kicks -- I
> > >>> disabled that.
> > >>>
> > >>> That, in turn, made Xen boot even without cpuidle=0. I'm attaching that log.
> > >>
> > >> Interesting, but unfortunately this particular log is of no real use
> > >> for investigation of the issue (other than knowing the CPU model). I
> > >> also notice it's a 4.12.0 log, when your original report was against
> > >> latest master.
> > >>
> > >>> So I guess at this point, you could say that I have a functional
> > >>> system, but I'm curious whether you guys would be interested to look
> > >>> into 2xAPIC situation.
> > >>
> > >> Of course we do. As a next step I'd suggest reverting the BIOS settings
> > >> change you did, and instead using the "x2apic=0" Xen command line option.
> > >
> > > Interestingly enough, this doesn't really solve the problem completely.
> > > Specifying x2apic=0 certainly makes Xen go much further to a point
> > > where it tries to load Dom0 and then the console VGA screen goes
> > > blank (this is where that serial debug output would be very useful :-().
> >
> > Now that's again unexpected. In any event you could try "vga=keep".
> >
> > >> And then we of course need a complete boot log (as requested earlier) of
> > >> a problem case.
> > >>
> > >> Further I'd suggest moving away from the black-and-white "cpuidle="
> > >> option, and instead limiting use of deep C states ("max_cstate="). I
> > >> wouldn't be surprised if this was the issue; we'd then have to first
> > >> of all go through errata for the part your system is using.
> > >
> > > Yup. max_cstate=1 makes it boot fine. max_cstate=2 though hangs
> > > the system *exactly* in the same way as specifying x2apic=0
> > > (which is different from the original problem as I've described above).
> >
> > "max_cstate=2" is much less of a "deep" C state than I had expected,
> > but well, so be it then. As to the hang - did you meanwhile figure
> > whether _any_ number of CPUs above 1 would result in a hang, or
> > whether instead there's a certain amount of them that would allow
> > boot to progress fine.
>
> Do you mean by tweaking maxcpus setting?
>
> Anything above 1 doesn't work
>
> maxcpus=1 boots fine tho
>
> > > Can you please elaborate on "we'd then have to first of all go through
> > > errata for the part your system is using"
> >
> > Well, it wouldn't be the first time that hardware had issues with C
> > state handling. Therefore we'd need to (a) be sure you use up-to-date
> > microcode and (b) there are no errata documented for your CPU model
> > workarounds for which basically suggest to avoid use of deep C states.
>
> It seems you've done that already -- thank you!
>
> Thanks,
> Roman.

[-- Attachment #2: xen3.log --]
[-- Type: application/octet-stream, Size: 14307 bytes --]

0x0000:0x00:0x17.0x0: ROM: 0x5400 bytes at 0x73c86018
0x0000:0x01:0x00.0x1: ROM: 0x8000 bytes at 0x73c74018
 Xen 4.12.0
(XEN) Xen version 4.12.0 (@) (gcc (Alpine 6.4.0) 6.4.0) debug=n  Sat Jul 27 03:04:16 UTC 2019
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.03
(XEN) Command line: loglvl=all com1=115200,8n1 console=com1,vga vga=keep maxcpus=2 dom0_mem=1024M,max:1024M dom0_max_vcpus=1 dom0_vcpus_pin smt=false dom0_mem=1024M,max:1024M dom0_max_vcpus=1 dom0_vcpus_pin smt=false
(XEN) Xen image load base address: 0x71800000
(XEN) Video information:
(XEN)  VGA is graphics mode 1280x1024, 32 bpp
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) EFI RAM map:
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000065439000 (usable)
(XEN)  0000000065439000 - 0000000065448000 (reserved)
(XEN)  0000000065448000 - 0000000065484000 (usable)
(XEN)  0000000065484000 - 0000000065488000 (reserved)
(XEN)  0000000065488000 - 000000006e2b7000 (usable)
(XEN)  000000006e2b7000 - 000000006e2c9000 (reserved)
(XEN)  000000006e2c9000 - 000000006e2ca000 (usable)
(XEN)  000000006e2ca000 - 000000006f4ec000 (reserved)
(XEN)  000000006f4ec000 - 000000006f4ed000 (ACPI NVS)
(XEN)  000000006f4ed000 - 000000006f4ff000 (reserved)
(XEN)  000000006f4ff000 - 0000000078cff000 (usable)
(XEN)  0000000078cff000 - 00000000799ff000 (reserved)
(XEN)  00000000799ff000 - 000000007bdff000 (ACPI NVS)
(XEN)  000000007bdff000 - 000000007bfff000 (ACPI data)
(XEN)  000000007bfff000 - 000000007c000000 (usable)
(XEN)  000000007c000000 - 0000000080000000 (reserved)
(XEN)  00000000fe010000 - 00000000fe011000 (reserved)
(XEN)  0000000100000000 - 000000027f000000 (usable)
(XEN) ACPI: RSDP 7BFFE014, 0024 (r2 HPE   )
(XEN) ACPI: XSDT 7BFE7188, 00FC (r1 HPE    Server          1 INTL 20160422)
(XEN) ACPI: FACP 7BFF4000, 010C (r6 HPE    Server          1 1590        1)
(XEN) ACPI: DSDT 7BFE4000, 1533 (r2 HPE    Server          2 HPE     20000)
(XEN) ACPI: FACS 7BDD0000, 0040
(XEN) ACPI: SSDT 7BFFC000, 0197 (r1 HPE     CpuSsdt        1 INTL 20130328)
(XEN) ACPI: BOOT 7BFFB000, 0028 (r1 HPE    Server          2 MSFT  1000013)
(XEN) ACPI: SSDT 7BFFA000, 003A (r1 HPE    PLATOPRG        1 INTL 20130328)
(XEN) ACPI: HEST 7BFF9000, 00A8 (r1 HPE    Server          1 INTL        1)
(XEN) ACPI: BERT 7BFF8000, 0030 (r1 HPE    Server          1 INTL        1)
(XEN) ACPI: ERST 7BFF7000, 0250 (r1 HPE    Server          1 INTL        1)
(XEN) ACPI: FPDT 7BFF6000, 0034 (r1 HPE    Server   20170001 INTL 20160422)
(XEN) ACPI: BGRT 7BFF5000, 0038 (r1 HPE    Server   20170001 1590 20160422)
(XEN) ACPI: HPET 7BFF3000, 0038 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: WDDT 7BFF2000, 0040 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: APIC 7BFF1000, 00BC (r3 HPE    Server          1 1590        1)
(XEN) ACPI: MCFG 7BFF0000, 003C (r1 HPE    Server          1 1590        1)
(XEN) ACPI: SLIT 7BFEF000, 002D (r1 HPE    Server          1 1590        1)
(XEN) ACPI: SRAT 7BFEE000, 0158 (r3 HPE    Server          1 1590        1)
(XEN) ACPI: SPMI 7BFED000, 0041 (r5 HPE    Server          1 1590        1)
(XEN) ACPI: RASF 7BFEC000, 0030 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: SPCR 7BFEB000, 0050 (r2 HPE    Server          1 1590        1)
(XEN) ACPI: MSCT 7BFEA000, 004E (r2 HPE    Server          1 1590        1)
(XEN) ACPI: BDAT 7BFE9000, 0030 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: WSMT 7BFE8000, 0028 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: PCCT 7BFFD000, 0030 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: DMAR 7BFE6000, 00B0 (r1 HPE    Server          1 1590        1)
(XEN) ACPI: SSDT 7BFE1000, 2AA2 (r2 HPE    PCISSDT         2 HPE     20000)
(XEN) ACPI: SSDT 7BFE0000, 01C4 (r2 HPE    TIMESSDT        2 HPE     20000)
(XEN) ACPI: SSDT 7BFDF000, 02F2 (r1 HPE        pmab        1 INTL 20130328)
(XEN) ACPI: OEM1 7BFDE000, 00DF (r2 HPE    CPU  CPC        2 HPE     20000)
(XEN) System RAM: 8042MB (8235284kB)
(XEN) SRAT: PXM 0 -> APIC 00000000 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 00000004 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 00000002 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 00000006 -> Node 0
(XEN) SRAT: Node 0 PXM 0 3f500000-7c000000
(XEN) SRAT: Node 0 PXM 0 0-a0000
(XEN) SRAT: Node 0 PXM 0 100000-3f500000
(XEN) SRAT: Node 0 PXM 0 100000000-200000000
(XEN) SRAT: Node 0 PXM 0 200000000-27f000000
(XEN) NUMA: Allocated memnodemap from 278b58000 - 278b5b000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised
(XEN) vesafb: framebuffer at 0x84000000, mapped to 0xffff82c000201000, using 5120k, total 5120k
(XEN) vesafb: mode is 1280x1024x32, linelength=5120, font 8x16
(XEN) vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 158 (0x9e), Stepping 10 (raw 000906ea)
(XEN) SMBIOS 3.1 present.
(XEN) x2APIC mode is already enabled by BIOS.
(XEN) Using APIC driver x2apic_cluster
(XEN) ACPI: PM-Timer IO Port: 0x508 (32 bits)
(XEN) ACPI: v5 SLEEP INFO: control[1:0], status[1:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:504,1:0], pm1x_evt[1:500,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 7bdd0000/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[7bdd000c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: X2APIC (apic_id[0x00] uid[0x00] enabled)
(XEN) ACPI: X2APIC (apic_id[0x04] uid[0x04] enabled)
(XEN) ACPI: X2APIC (apic_id[0x02] uid[0x02] enabled)
(XEN) ACPI: X2APIC (apic_id[0x06] uid[0x06] enabled)
(XEN) ACPI: X2APIC_NMI (uid[0x00] high level lint[0x1])
(XEN) ACPI: X2APIC_NMI (uid[0x02] high level lint[0x1])
(XEN) ACPI: X2APIC_NMI (uid[0x04] high level lint[0x1])
(XEN) ACPI: X2APIC_NMI (uid[0x06] high level lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
(XEN) Xen ERST support is initialized.
(XEN) HEST: Table parsing has been initialized
(XEN) ACPI: BGRT: invalidating v1 image at 0x72ca8000
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 2 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 120 GSI, 840 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) xstate: size: 0x440 and states: 0x1f
(XEN) CPU0: Intel machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN)   Compiled-in support: SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 39, Safe address 8000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (with PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Platform timer is 23.999MHz HPET
(XEN) Detected 3312.165 MHz processor.
(XEN) EFI memory map:
(XEN)  0000000000000-0000000003fff type=2 attr=000000000000000f
(XEN)  0000000004000-000000008efff type=7 attr=000000000000000f
(XEN)  000000008f000-000000009efff type=2 attr=000000000000000f
(XEN)  000000009f000-000000009ffff type=0 attr=000000000000000f
(XEN)  0000000100000-00000008c5fff type=2 attr=000000000000000f
(XEN)  00000008c6000-000001452dfff type=7 attr=000000000000000f
(XEN)  000001452e000-000003f4fffff type=1 attr=000000000000000f
(XEN)  000003f500000-000003f57ffff type=4 attr=000000000000000f
(XEN)  000003f580000-000003fcfffff type=7 attr=000000000000000f
(XEN)  000003fd00000-000003fd00fff type=3 attr=000000000000000f
(XEN)  000003fd01000-00000454a4fff type=4 attr=000000000000000f
(XEN)  00000454a5000-0000065438fff type=7 attr=000000000000000f
(XEN)  0000065439000-0000065447fff type=6 attr=800000000000000f
(XEN)  0000065448000-0000065483fff type=3 attr=000000000000000f
(XEN)  0000065484000-0000065487fff type=6 attr=800000000000000f
(XEN)  0000065488000-00000660b5fff type=3 attr=000000000000000f
(XEN)  00000660b6000-000006e2b6fff type=4 attr=000000000000000f
(XEN)  000006e2b7000-000006e2c8fff type=6 attr=800000000000000f
(XEN)  000006e2c9000-000006e2c9fff type=7 attr=000000000000000f
(XEN)  000006e2ca000-000006f4ebfff type=6 attr=800000000000000f
(XEN)  000006f4ec000-000006f4ecfff type=10 attr=000000000000000f
(XEN)  000006f4ed000-000006f4fefff type=6 attr=800000000000000f
(XEN)  000006f4ff000-000006f8f7fff type=7 attr=000000000000000f
(XEN)  000006f8f8000-000006f8f8fff type=2 attr=000000000000000f
(XEN)  000006f8f9000-000006fa61fff type=7 attr=000000000000000f
(XEN)  000006fa62000-000006fcfefff type=1 attr=000000000000000f
(XEN)  000006fcff000-00000719fffff type=7 attr=000000000000000f
(XEN)  0000071a00000-0000071d88fff type=2 attr=000000000000000f
(XEN)  0000071d89000-0000071d95fff type=7 attr=000000000000000f
(XEN)  0000071d96000-0000072a18fff type=4 attr=000000000000000f
(XEN)  0000072a19000-0000072be2fff type=7 attr=000000000000000f
(XEN)  0000072be3000-0000073b76fff type=4 attr=000000000000000f
(XEN)  0000073b77000-0000073b7efff type=7 attr=000000000000000f
(XEN)  0000073b7f000-0000073b7ffff type=4 attr=000000000000000f
(XEN)  0000073b80000-0000073b82fff type=7 attr=000000000000000f
(XEN)  0000073b83000-0000077cfefff type=4 attr=000000000000000f
(XEN)  0000077cff000-0000077cfffff type=7 attr=000000000000000f
(XEN)  0000077d00000-0000078cfefff type=3 attr=000000000000000f
(XEN)  0000078cff000-00000794fefff type=5 attr=800000000000000f
(XEN)  00000794ff000-00000798fefff type=6 attr=800000000000000f
(XEN)  00000798ff000-00000799fefff type=0 attr=000000000000000f
(XEN)  00000799ff000-000007bdfefff type=10 attr=000000000000000f
(XEN)  000007bdff000-000007bffefff type=9 attr=000000000000000f
(XEN)  000007bfff000-000007bffffff type=4 attr=000000000000000f
(XEN)  0000100000000-000027effffff type=7 attr=000000000000000f
(XEN)  00000000a0000-00000000fffff type=0 attr=0000000000000000
(XEN)  000007c000000-000007fffffff type=0 attr=0000000000000000
(XEN)  00000fe010000-00000fe010fff type=11 attr=8000000000000001
(XEN) Initing memory sharing.
(XEN) alt table ffff82d08042ac70 -> ffff82d08042c97a
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC_DEADLINE disabled due to Errata; please update microcode to version 0x52 (or later)
(XEN) Allocated console ring of 32 KiB.
(XEN) mwait-idle: MWAIT substates: 0x11142120
(XEN) mwait-idle: v0.4.1 model 0x9e
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) mwait-idle: max C-state count of 8 reached
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN)  - VMCS shadowing
(XEN)  - VM Functions
(XEN)  - Virtualisation Exceptions
(XEN)  - Page Modification Logging
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Adding cpu 1 to runqueue 0
(XEN) mwait-idle: max C-state count of 8 reached
(XEN) Adding cpu 2 to runqueue 0
(XEN) mwait-idle: max C-state count of 8 reached
(XEN) Removing cpu 2 from runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) mwait-idle: max C-state count of 8 reached
(XEN) Removing cpu 3 from runqueue 0
(XEN) Brought up 2 CPUs
(XEN) Parked 2 CPUs
(XEN) build-id: 97eb63b27af38eba068e90a79c3cf36ad6a4211b
(XEN) ACPI sleep modes: S3
(XEN) VPMU: disabled
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 312 PIRQs
(XEN) NX (Execute Disable) protection active
(XEN) *** Building a PV Dom0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2a2c000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000270000000->0000000274000000 (245760 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff82a2c000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->0000008000200000
(XEN)  Start info:    ffffffff82a2c000->ffffffff82a2c4b4
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff82a2d000->ffffffff82a46000
(XEN)  Boot stack:    ffffffff82a46000->ffffffff82a47000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82c00000
(XEN)  ENTRY ADDRESS: ffffffff826af180
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is keeping VGA console.
(XEN) Boot video device 01:00.1
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel