Re: [PATCH v2] x86/smpboot: Make logical package management more robust

[Max Vozeler <max@hinterhof.net>]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

On Tue, Jun 06, 2017 at 09:48:37PM -0400, Boris Ostrovsky wrote:
> On 06/06/2017 09:39 AM, Max Vozeler wrote:
> >there is a problem booting recent kernels on some Xen domUs hosted by
> >provider JiffyBox.
> >
> >The kernel seems to crash just after logging
> >[    0.038700] SMP alternatives: switching to SMP code
> 
> Do you have the crash splat? Stack trace and such.
> 
> In fact, full boot log might be useful.

Unfortunately, we don't have much more information.

Just after "switching to SMP code" the console connection is lost and we
get a notification that the VM has crashed. I'm attaching the boot log up
to that point.. just in case.

I have asked the hosting provider if they can provide XEN hypervisor logs.

> >We started seeing this with 4.9.2 and bisecting the 4.9 stable kernels
> >determined that this commit introduced the problem. Reverting it from 4.9.2
> >makes the kernel boot again.
> >
> >Older kernels (starting from 3.16 up to and including 4.9.1) were running
> >fine in this setup. But recent mainline (tested 4.12-rc3) and 4.9.x both
> >fail to boot there.
> >
> >Unfortunately we have no detailed information about the hypervisor or
> >setup and the provider is not very forthcoming with details. I'm attaching
> >dmesg of a successful boot (4.9.2 with this commit reverted).
> >
> >It shows a fairly old XEN version:
> >
> >[    0.000000] Xen version: 3.1.2-416.el5 (preserve-AD)
> 
> This is a 10 year old hypervisor so it's not especially surprising that
> newer kernels don't work. (If anything, I am surprised that you actually
> booted 4.9 at all).
> 
> There have been a bunch of problems in this area (topology) on PV guests.

Thanks and kind regards,

	Max


[-- Attachment #2: dmesg-crash.txt --]
[-- Type: text/plain, Size: 5991 bytes --]

 Filesystem type is ext2fs, using whole disk
kernel  /boot/vmlinuz-4.9.2-bisect-00033-g2b95c93 root=/dev/xvda ro cgroup_enab
le=memory apparmor=1 security=apparmor
initrd  /boot/initrd.img-4.9.2-bisect-00033-g2b95c93

============= Init TPM Front ================
Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
Tpmfront:Info Shutting down tpmfront
close blk: backend=/local/domain/0/backend/vbd/226/51712 node=device/vbd/51712
close blk: backend=/local/domain/0/backend/vbd/226/51728 node=device/vbd/51728
[    0.000000] Linux version 4.9.2-bisect-00033-g2b95c93 (aaa@example.com) (gcc version 4.9.2 (Debian 4.9.2-10) ) #5 SMP Wed May 31 10:54:25 UTC 2017
[    0.000000] Command line: root=/dev/xvda ro cgroup_enable=memory apparmor=1 security=apparmor
[    0.000000] x86/fpu: Legacy x87 FPU detected.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Released 0 page(s)
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000000807fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] Hypervisor detected: Xen
[    0.000000] e820: last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] MTRR: Disabled
[    0.000000] x86/PAT: MTRRs disabled, skipping PAT initialization too.
[    0.000000] x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC
[    0.000000] RAMDISK: [mem 0x01fe1000-0x0264afff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000000807fffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x7fc17000-0x7fc1bfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000807fffff]
[    0.000000]   Normal   empty
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000807fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000000807fffff]
[    0.000000] p2m virtual area at ffffc90000000000, size is 40000000
[    0.000000] Remapped 0 page(s)
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] smpboot: Allowing 3 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.000000] e820: [mem 0x80800000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 3.1.2-416.el5 (preserve-AD)
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:3 nr_node_ids:1
[    0.000000] percpu: Embedded 35 pages/cpu @ffff88007d000000 s105240 r8192 d29928 u524288
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 517994
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/xvda ro cgroup_enable=memory apparmor=1 security=apparmor
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory: 2030340K/2104956K available (7412K kernel code, 1426K rwdata, 3120K rodata, 1480K init, 848K bss, 74616K reserved, 0K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=3.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=3
[    0.000000] Using NULL legacy PIC
[    0.000000] NR_IRQS:33024 nr_irqs:64 0
[    0.000000] xen:events: Using 2-level ABI
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2133.431 MHz processor
[    0.008000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4266.81 BogoMIPS (lpj=8533632)
[    0.008000] pid_max: default: 32768 minimum: 301
[    0.008000] Security Framework initialized
[    0.008000] Yama: becoming mindful.
[    0.008000] AppArmor: AppArmor initialized
[    0.008000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.008000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.008000] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.008000] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.008000] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[    0.008000] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0
[    0.037256] ftrace: allocating 28843 entries in 113 pages
[    0.044095] cpu 0 spinlock event irq 1
[    0.044108] smpboot: Max logical packages: 1
[    0.044114] smpboot: CPU 0 Converting physical 33 to logical package 0
[    0.044123] VPMU disabled by hypervisor.
[    0.044144] Performance Events: unsupported p6 CPU model 44 no PMU driver, software events only.
[    0.044911] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.044922] NMI watchdog: Shutting down hard lockup detector on all cpus
[    0.045096] installing Xen timer for CPU 1
[    0.045135] SMP alternatives: switching to SMP code
Verbindung mit Console getrennt