* Regression, host crash with 4.5rc1
@ 2014-11-03 21:46 Steve Freitas
2014-11-03 21:52 ` Steve Freitas
` (2 more replies)
0 siblings, 3 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-03 21:46 UTC (permalink / raw)
To: Don Slutz, xen-devel
Hi all,
I've got a Windows 7 x64 VM that is stable on 4.4.1 but crashes the host
after a few hours on 4.5rc1. The machine is a ThinkStation D20, an X5660
with 5500-series chipset with an Nvidia Quadro FX4800 (genuine) passed
through. Distro is Debian Jessie running stock distro kernel and
seabios. Both 4.4.1 and 4.5rc1 were built from source. Apologies if this
issue has already been spotted but I can't keep up with the traffic on
this list! :-)
It looks as if something is stepping on PCI devices when the crash
happens. In the log included below, it's the SATA system that's
complaining but I've seen it hit the ethernet chip first, then SATA
after that. I'm happy to troubleshoot, apply patches, give more
information, etc.
I've seen the crash when running Windows Update, when running the
Unigine graphics benchmark, when running an Avast anti-virus scan.
Haven't found a common thread.
Don, one curious thing I noted which may be of no value whatsoever:
Under 4.4.1, I can't give the VM > 3.5 gigs of RAM without breaking VGA
passthrough. Under 4.5rc1 I can give the VM more than 3.5 gigs, yet I
*don't* need to use the "mmio_hole" settings in the VM config. Not sure
why or what that might signify, if anything.
Possibly useful information follows.
Thanks,
Steve
===================
windows_vm.cfg:
builder='hvm'
memory = 3584
vcpus = 6
shadow_memory = 64
name = "clippy2"
vif = [ 'bridge=br0, mac=00:16:3f:7a:d0:85' ]
acpi = 1
apic = 1
disk = [ 'phy:/dev/vg_g2/wintest,hda,w',
'file:/home/steve/X17-59465.iso,hdc:cdrom,r' ]
gfx_passthru=0
pci = ['02:00.0']
viridian = 1
localtime=1
boot="cd"
usb=1
usbdevice=['tablet', 'host:17ef:6009']
vnc=1
vnclisten="10.0.0.21"
vncconsole=0
vncpasswd=''
serial='pty'
root@g2:~# xl info
host : g2
release : 3.16-3-amd64
version : #1 SMP Debian 3.16.5-1 (2014-10-10)
machine : x86_64
nr_cpus : 6
max_cpu_id : 5
nr_nodes : 1
cores_per_socket : 6
threads_per_core : 1
cpu_mhz : 2800
hw_caps :
bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 49143
free_memory : 754
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 5
xen_extra : -rc1
xen_version : 4.5-rc1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Fri Oct 24 10:22:40 2014 -0400 git:1b068a5
xen_commandline : placeholder cpufreq=xen:ondemand cpuidle
clocksource=hpet log_lvl=all guest_loglvl=all xen-pciback.hide=(02:00.0)
iommu=no-intremap
cc_compiler : gcc (Debian 4.9.1-16) 4.9.1
cc_compile_by : steve
cc_compile_domain :
cc_compile_date : Thu Oct 30 15:43:54 PDT 2014
xend_config_format : 4
(Note that the hide command in the xen_commandline is a no-op because
xen is compiled in this kernel as modules.)
root@g2:~# xl dmesg
Xen 4.5-rc1
(XEN) Xen version 4.5-rc1 (steve@) (gcc (Debian 4.9.1-16) 4.9.1) debug=y
Thu Oct 30 15:43:54 PDT 2014
(XEN) Latest ChangeSet: Fri Oct 24 10:22:40 2014 -0400 git:1b068a5
(XEN) Bootloader: GRUB 2.02~beta2-15
(XEN) Command line: placeholder cpufreq=xen:ondemand cpuidle
clocksource=hpet log_lvl=all guest_loglvl=all xen-pciback.hide=(02:00.0)
iommu=no-intremap
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009a800 (usable)
(XEN) 000000000009a800 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf7a0000 (usable)
(XEN) 00000000bf7a0000 - 00000000bf7ca000 (ACPI data)
(XEN) 00000000bf7ca000 - 00000000bf7cc000 (ACPI NVS)
(XEN) 00000000bf7cc000 - 00000000c0000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000c40000000 (usable)
(XEN) ACPI: RSDP 000F6A40, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT BF7BF38F, 0084 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: FACP BF7C98D1, 00F4 (r3 LENOVO TC-61 60400D0 PTL 2)
(XEN) ACPI: DSDT BF7BF413, A43A (r1 LENOVO TC-61 60400D0 MSFT 100000E)
(XEN) ACPI: FACS BF7CBFC0, 0040
(XEN) ACPI: SSDT BF7AF33B, 2694 (r1 INTEL PPM RCM 80000001 INTL 20061109)
(XEN) ACPI: SLIT BF7C99E9, 0030 (r1 Intel TYLERBRG 60400D0 LOHR 5A)
(XEN) ACPI: TCPA BF7C9A19, 0032 (r1 LENOVO TC-61 60400D0 PTL 0)
(XEN) ACPI: SLIC BF7C9A4B, 0176 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: SRAT BF7C9BC1, 00E0 (r1 Intel Tylerbrg 60400D0 PHX. 1)
(XEN) ACPI: DMAR BF7C9CA1, 01C0 (r1 Intel OEMDMAR 60400D0 LOHR 1)
(XEN) ACPI: APIC BF7C9E61, 00A0 (r1 PTLTD APIC 60400D0
LTP 0)
(XEN) ACPI: MCFG BF7C9F01, 003C (r1 PTLTD MCFG 60400D0 LTP 0)
(XEN) ACPI: HPET BF7C9F3D, 0038 (r1 PTLTD HPETTBL 60400D0 LTP 1)
(XEN) ACPI: BOOT BF7C9F75, 0028 (r1 PTLTD $SBFTBL$ 60400D0 LTP 1)
(XEN) ACPI: ASF! BF7C9F9D, 0063 (r32 LENOVO TC-61 60400D0 PTL 1)
(XEN) System RAM: 49143MB (50322664kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-c0000000
(XEN) SRAT: Node 0 PXM 0 100000000-c40000000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f6a70
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1004,1:0], pm1x_evt[1:1000,1:0]
(XEN) ACPI: wakeup_vec[bf7cbfcc], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
(XEN) Processor #2 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
(XEN) Processor #4 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x10] enabled)
(XEN) Processor #16 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
(XEN) Processor #18 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
(XEN) Processor #20 6:12 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1144 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2800.195 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) alt table ffff82d0802d5e90 -> ffff82d0802d6eb0
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 09
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-09
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping disabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2c
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 6 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x7c8000
(XEN) elf_parse_binary: phdr: paddr=0x1800000 memsz=0xed000
(XEN) elf_parse_binary: phdr: paddr=0x18ed000 memsz=0x14f40
(XEN) elf_parse_binary: phdr: paddr=0x1902000 memsz=0x614000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1f16000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff819021f0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0x0
(XEN) virt_offset = 0xffffffff80000000
(XEN) virt_kstart = 0xffffffff81000000
(XEN) virt_kend = 0xffffffff81f16000
(XEN) virt_entry = 0xffffffff819021f0
(XEN) p2m_base = 0xffffffffffffffff
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f16000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000c00000000->0000000c10000000 (12316671 pages
to be allocated)
(XEN) Init. ramdisk: 0000000c3f0da000->0000000c3ffff86e
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff81f16000
(XEN) Init. ramdisk: ffffffff81f16000->ffffffff82e3b86e
(XEN) Phys-Mach map: ffffffff82e3c000->ffffffff88cbb928
(XEN) Start info: ffffffff88cbc000->ffffffff88cbc4b4
(XEN) Page tables: ffffffff88cbd000->ffffffff88d08000
(XEN) Boot stack: ffffffff88d08000->ffffffff88d09000
(XEN) TOTAL: ffffffff80000000->ffffffff89000000
(XEN) ENTRY ADDRESS: ffffffff819021f0
(XEN) Dom0 has maximum 6 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff817c8000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81800000 -> 0xffffffff818ed000
(XEN) elf_load_binary: phdr 2 at 0xffffffff818ed000 -> 0xffffffff81901f40
(XEN) elf_load_binary: phdr 3 at 0xffffffff81902000 -> 0xffffffff81a1f000
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) Scrubbing Free RAM on 1 nodes using 6 CPUs
(XEN)
..................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 304kB init memory.
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) PCI add device 0000:00:00.0
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) PCI add device 0000:00:01.0
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) PCI add device 0000:00:03.0
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) PCI add device 0000:00:07.0
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.1
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:16.1
(XEN) PCI add device 0000:00:16.2
(XEN) PCI add device 0000:00:16.3
(XEN) PCI add device 0000:00:16.4
(XEN) PCI add device 0000:00:16.5
(XEN) PCI add device 0000:00:16.6
(XEN) PCI add device 0000:00:16.7
(XEN) PCI add device 0000:00:1a.0
(XEN) PCI add device 0000:00:1a.1
(XEN) PCI add device 0000:00:1a.7
(XEN) PCI add device 0000:00:1b.0
(XEN) PCI add device 0000:00:1c.0
(XEN) PCI add device 0000:00:1c.4
(XEN) PCI add device 0000:00:1c.5
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1d.1
(XEN) PCI add device 0000:00:1d.2
(XEN) PCI add device 0000:00:1d.3
(XEN) PCI add device 0000:00:1d.7
(XEN) PCI add device 0000:00:1e.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.2
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:05:00.0
(XEN) PCI add device 0000:06:00.0
(XEN) PCI add device 0000:07:02.0
(XEN) PCI add device 0000:07:03.0
(XEN) PCI add device 0000:09:00.0
(XEN) PCI add device 0000:0a:0e.0
(d1) HVM Loader
(d1) Detected Xen v4.5-rc1
(d1) Xenbus rings @0xfeffc000, event channel 1
(d1) System requested SeaBIOS
(d1) CPU speed is 2800 MHz
(d1) Relocating guest memory for lowmem MMIO space disabled
(XEN) irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(d1) PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(d1) PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(d1) PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(d1) PCI-ISA link 3 routed to IRQ5
(d1) pci dev 01:2 INTD->IRQ5
(d1) pci dev 01:3 INTA->IRQ10
(d1) pci dev 02:0 INTA->IRQ11
(d1) pci dev 04:0 INTA->IRQ5
(d1) pci dev 05:0 INTA->IRQ10
(d1) No RAM in high memory; setting high_mem resource base to 100000000
(d1) pci dev 05:0 bar 14 size 010000000: 0e000000c
(XEN) memory_map:add: dom1 gfn=e0000 mfn=d0000 nr=10000
(d1) pci dev 03:0 bar 10 size 002000000: 0f0000008
(XEN) memory_map:add: dom1 gfn=f2000 mfn=e8000 nr=2000
(d1) pci dev 05:0 bar 1c size 002000000: 0f2000004
(d1) pci dev 02:0 bar 14 size 001000000: 0f4000008
(XEN) memory_map:add: dom1 gfn=f5000 mfn=ea000 nr=1000
(d1) pci dev 05:0 bar 10 size 001000000: 0f5000000
(d1) pci dev 05:0 bar 30 size 000080000: 0f6000000
(d1) pci dev 04:0 bar 30 size 000040000: 0f6080000
(d1) pci dev 03:0 bar 30 size 000010000: 0f60c0000
(d1) pci dev 03:0 bar 14 size 000001000: 0f60d0000
(d1) pci dev 02:0 bar 10 size 000000100: 00000c001
(d1) pci dev 04:0 bar 10 size 000000100: 00000c101
(d1) pci dev 04:0 bar 14 size 000000100: 0f60d1000
(d1) pci dev 05:0 bar 24 size 000000080: 00000c201
(XEN) ioport_map:add: dom1 gport=c200 mport=3000 nr=80
(d1) pci dev 01:2 bar 20 size 000000020: 00000c281
(d1) pci dev 01:1 bar 20 size 000000010: 00000c2a1
(d1) Multiprocessor initialisation:
(d1) - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) - CPU1 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) - CPU2 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) - CPU3 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) - CPU4 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) - CPU5 ... 40-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) Testing HVM environment:
(d1) - REP INSB across page boundaries ... passed
(d1) - GS base MSRs and SWAPGS ... passed
(d1) Passed 2 of 2 tests
(d1) Writing SMBIOS tables ...
(d1) Loading SeaBIOS ...
(d1) Creating MP tables ...
(d1) Loading ACPI ...
(d1) vm86 TSS at fc00a280
(d1) BIOS map:
(d1) 10000-100d3: Scratch space
(d1) c0000-fffff: Main BIOS
(d1) E820 table:
(d1) [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d1) HOLE: 00000000:000a0000 - 00000000:000c0000
(d1) [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
(d1) [02]: 00000000:00100000 - 00000000:df800000: RAM
(d1) HOLE: 00000000:df800000 - 00000000:fc000000
(d1) [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d1) Invoking SeaBIOS ...
(d1) SeaBIOS (version rel-1.7.5-0-ge51488c-20141030_133131-g2)
(d1)
(d1) Found Xen hypervisor signature at 40000100
(d1) Running on QEMU (i440fx)
(d1) xen: copy e820...
(d1) Relocating init from 0x000e05e9 to 0xdf7af370 (size 68548)
(d1) CPU Mhz=2801
(d1) Found 9 PCI devices (max PCI bus is 00)
(d1) Allocated Xen hypercall page at df7ff000
(d1) Detected Xen v4.5-rc1
(d1) xen: copy BIOS tables...
(d1) Copying SMBIOS entry point from 0x00010010 to 0x000f1190
(d1) Copying MPTABLE from 0xfc001200/fc001210 to 0x000f1040
(d1) Copying PIR from 0x00010030 to 0x000f0fc0
(d1) Copying ACPI RSDP from 0x000100b0 to 0x000f0f90
(d1) Using pmtimer, ioport 0xb008
(d1) Scan for VGA option rom
(d1) Running option rom at c000:0003
(XEN) stdvga.c:147:d1v0 entering stdvga and caching modes
(d1) pmm call arg1=0
(d1) Turning on vga text mode console
(d1) SeaBIOS (version rel-1.7.5-0-ge51488c-20141030_133131-g2)
(d1) Machine UUID aa695d3e-9592-4139-929c-5d2857702fa9
(d1) UHCI init on dev 00:01.2 (io=c280)
(d1) Found 0 lpt ports
(d1) Found 1 serial ports
(d1) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
(d1) ATA controller 2 at 170/374/0 (irq 15 dev 9)
(d1) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (61440 MiBytes)
(d1) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(d1) DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(d1) Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(d1) PS2 keyboard initialized
(d1) USB keyboard initialized
(d1) Initialized USB HUB (1 ports used)
(d1) All threads complete.
(d1) Scan for option roms
(d1) Running option rom at c980:0003
(d1) pmm call arg1=1
(d1) pmm call arg1=0
(d1) pmm call arg1=1
(d1) pmm call arg1=0
(d1) Searching bootorder for: /pci@i0cf8/*@4
(d1)
(d1) Press F12 for boot menu.
(d1)
(d1) Searching bootorder for: HALT
(d1) drive 0x000f0f40: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
s=125829120
(d1)
(d1) Space available for UMB: ca800-ee800, f0000-f0ee0
(d1) Returned 249856 bytes of ZoneHigh
(d1) e820 map has 6 items:
(d1) 0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d1) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d1) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d1) 3: 0000000000100000 - 00000000df7fd000 = 1 RAM
(d1) 4: 00000000df7fd000 - 00000000df800000 = 2 RESERVED
(d1) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(d1) enter handle_19:
(d1) NULL
(d1) Booting from Hard Disk...
(d1) Booting from 0000:7c00
(XEN) stdvga.c:151:d1v0 leaving stdvga
(XEN) d1: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 1
build: 1db1
(XEN) d1: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
(XEN) d1v0: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fffe
(XEN) d1v1: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fffd
(XEN) d1v2: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fffc
(XEN) d1v3: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fffb
(XEN) d1v4: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fffa
(XEN) d1v5: VIRIDIAN APIC_ASSIST: enabled: 1 pfn: 3fff9
(XEN) irq.c:270: Dom1 PCI link 0 changed 5 -> 0
(XEN) irq.c:270: Dom1 PCI link 1 changed 10 -> 0
(XEN) irq.c:270: Dom1 PCI link 2 changed 11 -> 0
(XEN) irq.c:270: Dom1 PCI link 3 changed 5 -> 0
(XEN) memory_map:remove: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:remove: dom1 gfn=f2000 mfn=e8000 nr=2000
(XEN) memory_map:remove: dom1 gfn=f5000 mfn=ea000 nr=1000
(XEN) ioport_map:remove: dom1 gport=c200 mport=3000 nr=80
(XEN) memory_map:add: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:add: dom1 gfn=f2000 mfn=e8000 nr=2000
(XEN) memory_map:add: dom1 gfn=f5000 mfn=ea000 nr=1000
(XEN) ioport_map:add: dom1 gport=c200 mport=3000 nr=80
(XEN) grant_table.c:1272:d1v2 Expanding dom (1) grant table from (4) to
(32) frames.
(XEN) irq.c:380: Dom1 callback via changed to GSI 24
(XEN) memory_map:remove: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:remove: dom1 gfn=f2000 mfn=e8000 nr=2000
(XEN) memory_map:remove: dom1 gfn=f5000 mfn=ea000 nr=1000
(XEN) ioport_map:remove: dom1 gport=c200 mport=3000 nr=80
(XEN) memory_map:add: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:add: dom1 gfn=f2000 mfn=e8000 nr=2000
(XEN) memory_map:add: dom1 gfn=f5000 mfn=ea000 nr=1000
(XEN) ioport_map:add: dom1 gport=c200 mport=3000 nr=80
(XEN) memory_map:remove: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:remove: dom1 gfn=f2000 mfn=e8000 nr=2000
(XEN) memory_map:add: dom1 gfn=e0000 mfn=d0000 nr=10000
(XEN) memory_map:add: dom1 gfn=f2000 mfn=e8000 nr=2000
root@g2:~# tail -f /var/log/syslog
Nov 2 21:48:38 g2 kernel: [ 203.011898] ip_tables: (C) 2000-2006
Netfilter Core Team
Nov 2 21:48:38 g2 root: /etc/xen/scripts/vif-bridge: Successful
vif-bridge online for vif1.0, bridge br0.
Nov 2 21:48:38 g2 root: /etc/xen/scripts/vif-bridge: Writing
backend/vif/1/0/hotplug-status connected to xenstore.
Nov 2 21:48:38 g2 root: /etc/xen/scripts/vif-bridge: add type_if=tap
XENBUS_PATH=backend/vif/1/0
Nov 2 21:48:38 g2 kernel: [ 203.233482] device vif1.0-emu entered
promiscuous mode
Nov 2 21:48:38 g2 kernel: [ 203.236671] br0: port 3(vif1.0-emu)
entered forwarding state
Nov 2 21:48:38 g2 kernel: [ 203.236677] br0: port 3(vif1.0-emu)
entered forwarding state
Nov 2 21:48:38 g2 root: /etc/xen/scripts/vif-bridge: Successful
vif-bridge add for vif1.0-emu, bridge br0.
Nov 2 21:48:39 g2 kernel: [ 204.651250] xen_pciback: vpci:
0000:02:00.0: assign to virtual slot 0
Nov 2 21:48:40 g2 kernel: [ 205.533377] usb 7-1: reset low-speed USB
device number 2 using uhci_hcd
Nov 2 21:48:53 g2 kernel: [ 218.261360] br0: port 3(vif1.0-emu)
entered forwarding state
Nov 2 21:49:20 g2 kernel: [ 245.379901] xen-blkback:ring-ref 16383,
event-channel 11, protocol 1 (x86_64-abi)
Nov 2 21:49:26 g2 kernel: [ 250.836687] IPv6: ADDRCONF(NETDEV_CHANGE):
vif1.0: link becomes ready
Nov 2 21:49:26 g2 kernel: [ 250.836726] br0: port 2(vif1.0) entered
forwarding state
Nov 2 21:49:26 g2 kernel: [ 250.836732] br0: port 2(vif1.0) entered
forwarding state
Nov 2 21:49:28 g2 kernel: [ 253.485368] usb 7-1: reset low-speed USB
device number 2 using uhci_hcd
Nov 2 21:49:29 g2 kernel: [ 254.101376] usb 7-1: reset low-speed USB
device number 2 using uhci_hcd
Nov 2 21:49:41 g2 kernel: [ 265.845376] br0: port 2(vif1.0) entered
forwarding state
Nov 2 22:17:01 g2 /USR/SBIN/CRON[2361]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Nov 2 23:17:01 g2 /USR/SBIN/CRON[2373]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Nov 3 00:00:01 g2 /USR/SBIN/CRON[2387]: (root) CMD (invoke-rc.d atop _cron)
Nov 3 00:17:01 g2 /USR/SBIN/CRON[2423]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Nov 3 00:52:02 g2 kernel: [11207.677448] ata3.00: exception Emask 0x0
SAct 0x1ffc000 SErr 0x0 action 0x6 frozen
Nov 3 00:52:02 g2 kernel: [11207.677512] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.677568] ata3.00: cmd
61/58:70:90:57:23/00:00:04:00:00/40 tag 14 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.677568] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.677637] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.677686] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.677740] ata3.00: cmd
61/58:78:e8:57:23/00:00:04:00:00/40 tag 15 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.677740] res
40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.677809] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.677857] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.677912] ata3.00: cmd
61/58:80:40:58:23/00:00:04:00:00/40 tag 16 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.677912] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.677980] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.678029] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.678099] ata3.00: cmd
61/58:88:98:58:23/00:00:04:00:00/40 tag 17 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.678099] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.678231] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.678295] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.678365] ata3.00: cmd
61/58:90:f0:58:23/00:00:04:00:00/40 tag 18 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.678365] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.678496] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.678560] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.678631] ata3.00: cmd
61/58:98:48:59:23/00:00:04:00:00/40 tag 19 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.678631] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.678761] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.678826] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.678896] ata3.00: cmd
61/58:a0:a0:59:23/00:00:04:00:00/40 tag 20 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.678896] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.679026] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.679090] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.679160] ata3.00: cmd
61/58:a8:f8:59:23/00:00:04:00:00/40 tag 21 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.679160] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.679291] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.679355] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.679425] ata3.00: cmd
61/58:b0:50:5a:23/00:00:04:00:00/40 tag 22 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.679425] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.679556] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.679620] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.679690] ata3.00: cmd
61/58:b8:a8:5a:23/00:00:04:00:00/40 tag 23 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.679690] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.679821] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.679886] ata3.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.679955] ata3.00: cmd
61/18:c0:00:5b:23/00:00:04:00:00/40 tag 24 ncq 12288 out
Nov 3 00:52:02 g2 kernel: [11207.679955] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.680090] ata3.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.680157] ata3: hard resetting link
Nov 3 00:52:02 g2 kernel: [11207.680170] ata1.00: exception Emask 0x0
SAct 0x3ff800 SErr 0x0 action 0x6 frozen
Nov 3 00:52:02 g2 kernel: [11207.680262] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.680333] ata1.00: cmd
61/58:58:90:57:23/00:00:04:00:00/40 tag 11 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.680333] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.680465] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.680530] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.680600] ata1.00: cmd
61/58:60:e8:57:23/00:00:04:00:00/40 tag 12 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.680600] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.680732] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.680796] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.680866] ata1.00: cmd
61/58:68:40:58:23/00:00:04:00:00/40 tag 13 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.680866] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.680998] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.681062] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.681132] ata1.00: cmd
61/58:70:98:58:23/00:00:04:00:00/40 tag 14 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.681132] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.681264] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.681353] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.681428] ata1.00: cmd
61/58:78:f0:58:23/00:00:04:00:00/40 tag 15 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.681428] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.681562] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.681627] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.681698] ata1.00: cmd
61/58:80:48:59:23/00:00:04:00:00/40 tag 16 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.681698] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.681830] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.681895] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.681965] ata1.00: cmd
61/58:88:a0:59:23/00:00:04:00:00/40 tag 17 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.681965] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.682096] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.682161] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.682232] ata1.00: cmd
61/58:90:f8:59:23/00:00:04:00:00/40 tag 18 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.682232] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.682363] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.682428] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.682498] ata1.00: cmd
61/58:98:50:5a:23/00:00:04:00:00/40 tag 19 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.682498] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.682630] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.682695] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.682765] ata1.00: cmd
61/58:a0:a8:5a:23/00:00:04:00:00/40 tag 20 ncq 45056 out
Nov 3 00:52:02 g2 kernel: [11207.682765] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.682897] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.682961] ata1.00: failed command: WRITE
FPDMA QUEUED
Nov 3 00:52:02 g2 kernel: [11207.683032] ata1.00: cmd
61/18:a8:00:5b:23/00:00:04:00:00/40 tag 21 ncq 12288 out
Nov 3 00:52:02 g2 kernel: [11207.683032] res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 3 00:52:02 g2 kernel: [11207.683163] ata1.00: status: { DRDY }
Nov 3 00:52:02 g2 kernel: [11207.683229] ata1: hard resetting link
Write failed: Broken pipe
(I was monitoring syslog over a network connection since nothing of
value can be saved to disk under the circumstances.)
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-03 21:46 Regression, host crash with 4.5rc1 Steve Freitas
@ 2014-11-03 21:52 ` Steve Freitas
2014-11-04 14:01 ` Don Slutz
2014-11-04 8:20 ` Pasi Kärkkäinen
2014-11-04 14:39 ` Don Slutz
2 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-03 21:52 UTC (permalink / raw)
To: Don Slutz, xen-devel
On 11/3/2014 13:46, Steve Freitas wrote:
> Hi all,
>
> I've got a Windows 7 x64 VM that is stable on 4.4.1 but crashes the
> host after a few hours on 4.5rc1.
Whoops, neglected to include the following log which, I expect, is useless:
root@g2:/var/log/xen# cat qemu-dm-clippy2.log
char device redirected to /dev/pts/2 (label serial0)
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-03 21:46 Regression, host crash with 4.5rc1 Steve Freitas
2014-11-03 21:52 ` Steve Freitas
@ 2014-11-04 8:20 ` Pasi Kärkkäinen
2014-11-04 10:15 ` Jan Beulich
2014-11-04 18:35 ` Steve Freitas
2014-11-04 14:39 ` Don Slutz
2 siblings, 2 replies; 39+ messages in thread
From: Pasi Kärkkäinen @ 2014-11-04 8:20 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
On Mon, Nov 03, 2014 at 01:46:03PM -0800, Steve Freitas wrote:
> Hi all,
>
Hello,
> I've got a Windows 7 x64 VM that is stable on 4.4.1 but crashes the
> host after a few hours on 4.5rc1. The machine is a ThinkStation D20,
> an X5660 with 5500-series chipset with an Nvidia Quadro FX4800
> (genuine) passed through. Distro is Debian Jessie running stock
> distro kernel and seabios. Both 4.4.1 and 4.5rc1 were built from
> source. Apologies if this issue has already been spotted but I can't
> keep up with the traffic on this list! :-)
>
> It looks as if something is stepping on PCI devices when the crash
> happens. In the log included below, it's the SATA system that's
> complaining but I've seen it hit the ethernet chip first, then SATA
> after that. I'm happy to troubleshoot, apply patches, give more
> information, etc.
>
> I've seen the crash when running Windows Update, when running the
> Unigine graphics benchmark, when running an Avast anti-virus scan.
> Haven't found a common thread.
>
> Don, one curious thing I noted which may be of no value whatsoever:
> Under 4.4.1, I can't give the VM > 3.5 gigs of RAM without breaking
> VGA passthrough. Under 4.5rc1 I can give the VM more than 3.5 gigs,
> yet I *don't* need to use the "mmio_hole" settings in the VM config.
> Not sure why or what that might signify, if anything.
>
Does it run stable with Xen 4.5 without PCI/GPU passthru?
> Possibly useful information follows.
>
> Thanks,
>
> Steve
>
>
> (I was monitoring syslog over a network connection since nothing of
> value can be saved to disk under the circumstances.)
>
It'd be very helpful if you could set up a serial console to log all the Xen and dom0 Linux kernel messages,
so we could see where it actually crashes..
-- Pasi
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-04 8:20 ` Pasi Kärkkäinen
@ 2014-11-04 10:15 ` Jan Beulich
2014-11-10 8:03 ` Steve Freitas
2014-11-04 18:35 ` Steve Freitas
1 sibling, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-04 10:15 UTC (permalink / raw)
To: Steve Freitas, Pasi Kärkkäinen; +Cc: Don Slutz, xen-devel
>>> On 04.11.14 at 09:20, <pasik@iki.fi> wrote:
> On Mon, Nov 03, 2014 at 01:46:03PM -0800, Steve Freitas wrote:
>> (I was monitoring syslog over a network connection since nothing of
>> value can be saved to disk under the circumstances.)
>>
>
> It'd be very helpful if you could set up a serial console to log all the Xen
> and dom0 Linux kernel messages,
> so we could see where it actually crashes..
In fact this would not just be nice, but is strictly needed, and (as
with any pass-through related problems) additionally requires
running with "iommu=debug" alongside the usual need of setting
the log levels to the maximum.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-03 21:52 ` Steve Freitas
@ 2014-11-04 14:01 ` Don Slutz
0 siblings, 0 replies; 39+ messages in thread
From: Don Slutz @ 2014-11-04 14:01 UTC (permalink / raw)
To: Steve Freitas, Don Slutz, xen-devel
On 11/03/14 16:52, Steve Freitas wrote:
> On 11/3/2014 13:46, Steve Freitas wrote:
>> Hi all,
>>
>> I've got a Windows 7 x64 VM that is stable on 4.4.1 but crashes the
>> host after a few hours on 4.5rc1.
>
> Whoops, neglected to include the following log which, I expect, is
> useless:
>
> root@g2:/var/log/xen# cat qemu-dm-clippy2.log
> char device redirected to /dev/pts/2 (label serial0)
>
Yup, However it would be good to know if QEMU is still running.
-Don Slutz
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-03 21:46 Regression, host crash with 4.5rc1 Steve Freitas
2014-11-03 21:52 ` Steve Freitas
2014-11-04 8:20 ` Pasi Kärkkäinen
@ 2014-11-04 14:39 ` Don Slutz
2014-11-06 23:20 ` Steve Freitas
2 siblings, 1 reply; 39+ messages in thread
From: Don Slutz @ 2014-11-04 14:39 UTC (permalink / raw)
To: Steve Freitas, Don Slutz, xen-devel
On 11/03/14 16:46, Steve Freitas wrote:
> Hi all,
>
> I've got a Windows 7 x64 VM that is stable on 4.4.1 but crashes the
> host after a few hours on 4.5rc1. The machine is a ThinkStation D20,
> an X5660 with 5500-series chipset with an Nvidia Quadro FX4800
> (genuine) passed through. Distro is Debian Jessie running stock distro
> kernel and seabios. Both 4.4.1 and 4.5rc1 were built from source.
> Apologies if this issue has already been spotted but I can't keep up
> with the traffic on this list! :-)
>
> It looks as if something is stepping on PCI devices when the crash
> happens. In the log included below, it's the SATA system that's
> complaining but I've seen it hit the ethernet chip first, then SATA
> after that. I'm happy to troubleshoot, apply patches, give more
> information, etc.
>
> I've seen the crash when running Windows Update, when running the
> Unigine graphics benchmark, when running an Avast anti-virus scan.
> Haven't found a common thread.
>
> Don, one curious thing I noted which may be of no value whatsoever:
> Under 4.4.1, I can't give the VM > 3.5 gigs of RAM without breaking
> VGA passthrough. Under 4.5rc1 I can give the VM more than 3.5 gigs,
> yet I *don't* need to use the "mmio_hole" settings in the VM config.
> Not sure why or what that might signify, if anything.
>
First off, I do not know much about pci-passthru. The fact that it now
boots with
more then 3.5 GiB just tells me that things have changed (like what
physical addresses
are used get changed). Without a clear way to cause this issue, getting
to the root
cause is much harder. For example, there is no clear way to say that
4.4.1 will never
have this issue (it just has not yet done so).
There are 3 mmio_size values that make sense to try.
1) mmio_hole = 1033
This is what Konrad Wilk suggested (hole based on bare metal bios).
This has been
rounded up to the nearest MB.
2) mmio_hole = 1024
This might be a good size (512 may be just a little too small).
3) mmio_size = 512
This is the size that you think has worked in the past.
I would also see if less guest ram (like 3.0 GiB) can still reproduce
the issue.
-Don Slutz
> Possibly useful information follows.
>
> Thanks,
>
> Steve
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-04 8:20 ` Pasi Kärkkäinen
2014-11-04 10:15 ` Jan Beulich
@ 2014-11-04 18:35 ` Steve Freitas
1 sibling, 0 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-04 18:35 UTC (permalink / raw)
To: Pasi Kärkkäinen; +Cc: JBeulich, Don Slutz, xen-devel
On 11/4/2014 0:20, Pasi Kärkkäinen wrote:
> Does it run stable with Xen 4.5 without PCI/GPU passthru?
So far yes, I need to work it a bit harder to be sure, though.
> It'd be very helpful if you could set up a serial console to log all the Xen and dom0 Linux kernel messages,
> so we could see where it actually crashes..
Okay, I will do that with verbose logging and iommu=intremap,debug.
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-04 14:39 ` Don Slutz
@ 2014-11-06 23:20 ` Steve Freitas
0 siblings, 0 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-06 23:20 UTC (permalink / raw)
To: Don Slutz, xen-devel, Pasi Kärkkäinen, Jan Beulich
Hi Don,
On 11/4/2014 6:39, Don Slutz wrote:
>
> I would also see if less guest ram (like 3.0 GiB) can still reproduce
> the issue.
Still working on getting serial logging set up, but I was just able to
verify that 3.0 GiB of guest RAM reproduces the issue.
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-04 10:15 ` Jan Beulich
@ 2014-11-10 8:03 ` Steve Freitas
2014-11-10 8:51 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-10 8:03 UTC (permalink / raw)
To: Jan Beulich, Pasi Kärkkäinen; +Cc: Don Slutz, xen-devel
On 11/04/2014 02:15 AM, Jan Beulich wrote:
> In fact this would not just be nice, but is strictly needed, and (as
> with any pass-through related problems) additionally requires
> running with "iommu=debug" alongside the usual need of setting
> the log levels to the maximum.
>
Hi all,
Sorry for the delay, took some debugging on another computer to get
serial logging working. Due to its size, I've posted the entire log of a
crashed session here: http://pastebin.com/AiPHUZRH In this case I used a
3.0 gig memory size for the Windows domU.
As I mentioned before, sometimes it's the SATA that goes first, other
times the tg3 ethernet driver. Also note that between 4.4.1 and 4.5rc1,
the kernel I'm using (stock Debian Jessie) has not changed.
Please let me know if you need any other information. Thanks!
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-10 8:03 ` Steve Freitas
@ 2014-11-10 8:51 ` Jan Beulich
2014-11-10 20:05 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-10 8:51 UTC (permalink / raw)
To: Steve Freitas, pasik; +Cc: Don Slutz, xen-devel
>>> On 10.11.14 at 09:03, <sflist@ihonk.com> wrote:
> Sorry for the delay, took some debugging on another computer to get
> serial logging working. Due to its size, I've posted the entire log of a
> crashed session here: http://pastebin.com/AiPHUZRH In this case I used a
> 3.0 gig memory size for the Windows domU.
>
> As I mentioned before, sometimes it's the SATA that goes first, other
> times the tg3 ethernet driver. Also note that between 4.4.1 and 4.5rc1,
> the kernel I'm using (stock Debian Jessie) has not changed.
>
> Please let me know if you need any other information. Thanks!
Raising the kernel log level to maximum too would have helped.
Regardless of that, the first device showing anomalies here appears
to be the UHCI controller:
[ 147.415713] usb 7-1: reset low-speed USB device number 2 using uhci_hcd
while booting the guest.
And these
[ 199.775209] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
[ 199.775238] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Transmitter ID)
[ 199.775251] pcieport 0000:00:03.0: device [8086:340a] error status/mask=00001100/00002000
[ 199.775255] pcieport 0000:00:03.0: [ 8] RELAY_NUM Rollover
[ 199.775258] pcieport 0000:00:03.0: [12] Replay Timer Timeout
hint at a problem in the system's design. 00:03.0 is the parent bridge
of 02:00.0 (and from what I can tell that's the only device behind that
bridge), and hence the above messages can only reasonably have
their origin at the passed through VGA device. IOW it may well be that
you were just lucky that things worked earlier on.
And btw - the title saying "host crash" seems to not match the provided
log, as there's no sign of a crash anywhere (the host may be hung from
what is visible). Was that just badly worded, or have you actually seen
crashes too?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-10 8:51 ` Jan Beulich
@ 2014-11-10 20:05 ` Steve Freitas
2014-11-11 8:05 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-10 20:05 UTC (permalink / raw)
To: Jan Beulich, pasik; +Cc: Don Slutz, xen-devel
On 11/10/2014 0:51, Jan Beulich wrote:
>>>> On 10.11.14 at 09:03, <sflist@ihonk.com> wrote:
>> Sorry for the delay, took some debugging on another computer to get
>> serial logging working. Due to its size, I've posted the entire log of a
>> crashed session here: http://pastebin.com/AiPHUZRH In this case I used a
>> 3.0 gig memory size for the Windows domU.
>>
>> As I mentioned before, sometimes it's the SATA that goes first, other
>> times the tg3 ethernet driver. Also note that between 4.4.1 and 4.5rc1,
>> the kernel I'm using (stock Debian Jessie) has not changed.
>>
>> Please let me know if you need any other information. Thanks!
> Raising the kernel log level to maximum too would have helped.
Okay, I've done that and the output is here, let me know if you have any
preferred logging flags instead:
http://pastebin.com/M3yvWNTT
> Regardless of that, the first device showing anomalies here appears
> to be the UHCI controller:
>
> [ 147.415713] usb 7-1: reset low-speed USB device number 2 using uhci_hcd
>
> while booting the guest.
I assume this is related to the USB device (a keyboard) I'm passing
through to the domU.
> And these
>
> [ 199.775209] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
> [ 199.775238] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Transmitter ID)
> [ 199.775251] pcieport 0000:00:03.0: device [8086:340a] error status/mask=00001100/00002000
> [ 199.775255] pcieport 0000:00:03.0: [ 8] RELAY_NUM Rollover
> [ 199.775258] pcieport 0000:00:03.0: [12] Replay Timer Timeout
>
> hint at a problem in the system's design. 00:03.0 is the parent bridge
> of 02:00.0 (and from what I can tell that's the only device behind that
> bridge), and hence the above messages can only reasonably have
> their origin at the passed through VGA device.
You are correct that the VGA card is the only device on 03.0:
root@g2:~# lspci -tv
-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port
+-01.0-[01]----00.0 Marvell Technology Group Ltd.
MV64460/64461/64462 System Controller, Revision B
+-03.0-[02]----00.0 NVIDIA Corporation GT200GL [Quadro FX 4800]
+-07.0-[03]--
+-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System
Management Registers
+-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO
and Scratch Pad Registers
+-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control
Status and RAS Registers
+-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device
+-1a.0 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #4
+-1a.1 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #5
+-1a.7 Intel Corporation 82801JI (ICH10 Family) USB2 EHCI
Controller #2
+-1b.0 Intel Corporation 82801JI (ICH10 Family) HD Audio
Controller
+-1c.0-[04]--
+-1c.4-[05]----00.0 Broadcom Corporation NetXtreme BCM5755
Gigabit Ethernet PCI Express
+-1c.5-[06-09]----00.0-[07-09]--+-02.0-[08]--
| \-03.0-[09]----00.0 Broadcom
Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express
+-1d.0 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #1
+-1d.1 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #2
+-1d.2 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #3
+-1d.3 Intel Corporation 82801JI (ICH10 Family) USB UHCI
Controller #6
+-1d.7 Intel Corporation 82801JI (ICH10 Family) USB2 EHCI
Controller #1
+-1e.0-[0a]----0e.0 Advanced Micro Devices, Inc. [AMD/ATI]
RV100 [Radeon 7000 / Radeon VE]
+-1f.0 Intel Corporation 82801JIB (ICH10) LPC Interface
Controller
+-1f.2 Intel Corporation 82801JI (ICH10 Family) SATA AHCI
Controller
\-1f.3 Intel Corporation 82801JI (ICH10 Family) SMBus
Controller
What problem in the system's design does this hint at?
> IOW it may well be that
> you were just lucky that things worked earlier on.
Certainly possible but this is a very common machine in the corporate
world -- a Lenovo ThinkStation D20 running the X58 chipset. If it's an
inherent defect in the machine and somebody else hasn't already tripped
over it I would be very surprised.
> And btw - the title saying "host crash" seems to not match the provided
> log, as there's no sign of a crash anywhere (the host may be hung from
> what is visible). Was that just badly worded, or have you actually seen
> crashes too?
>
Only seen hanging. Sorry for the lack of technical rigor on the title,
but from the other end of the ethernet cable, it might as well have crashed.
If the expanded logging doesn't tell you anything useful, I'll see if I
can bisect the problem.
Thanks very much for your time.
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-10 20:05 ` Steve Freitas
@ 2014-11-11 8:05 ` Jan Beulich
2014-11-17 19:21 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-11 8:05 UTC (permalink / raw)
To: Steve Freitas, pasik; +Cc: Don Slutz, xen-devel
>>> On 10.11.14 at 21:05, <sflist@ihonk.com> wrote:
> On 11/10/2014 0:51, Jan Beulich wrote:
>> Raising the kernel log level to maximum too would have helped.
>
> Okay, I've done that and the output is here, let me know if you have any
> preferred logging flags instead:
>
> http://pastebin.com/M3yvWNTT
Hmm, I can't spot any further useful messages there, which may be
due to the log having got partly garbled.
>> Regardless of that, the first device showing anomalies here appears
>> to be the UHCI controller:
>>
>> [ 147.415713] usb 7-1: reset low-speed USB device number 2 using
> uhci_hcd
>>
>> while booting the guest.
>
> I assume this is related to the USB device (a keyboard) I'm passing
> through to the domU.
But not by passing through the HCD I assume, since the log only
shows the VGA card being consumed by pciback?
>> And these
>>
>> [ 199.775209] pcieport 0000:00:03.0: AER: Multiple Corrected error
> received: id=0018
>> [ 199.775238] pcieport 0000:00:03.0: PCIe Bus Error:
> severity=Corrected, type=Data Link Layer, id=0018(Transmitter ID)
>> [ 199.775251] pcieport 0000:00:03.0: device [8086:340a] error
> status/mask=00001100/00002000
>> [ 199.775255] pcieport 0000:00:03.0: [ 8] RELAY_NUM Rollover
>> [ 199.775258] pcieport 0000:00:03.0: [12] Replay Timer Timeout
>>
>> hint at a problem in the system's design. 00:03.0 is the parent bridge
>> of 02:00.0 (and from what I can tell that's the only device behind that
>> bridge), and hence the above messages can only reasonably have
>> their origin at the passed through VGA device.
>
> You are correct that the VGA card is the only device on 03.0:
> [...]
> What problem in the system's design does this hint at?
It's not the topology that I referred to, but last events reported in
the quoted log lines above. Such should not be happening repeatedly
on a properly functioning system.
>> IOW it may well be that
>> you were just lucky that things worked earlier on.
>
> Certainly possible but this is a very common machine in the corporate
> world -- a Lenovo ThinkStation D20 running the X58 chipset. If it's an
> inherent defect in the machine and somebody else hasn't already tripped
> over it I would be very surprised.
Except that pass-through, and especially VGA pass-through, aren't
being used by that many people.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-11 8:05 ` Jan Beulich
@ 2014-11-17 19:21 ` Steve Freitas
2014-11-18 7:54 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-17 19:21 UTC (permalink / raw)
To: Jan Beulich, pasik; +Cc: Don Slutz, xen-devel
Hi Jan,
On 11/11/2014 0:05, Jan Beulich wrote:
>
>>> And these
>>>
>>> [ 199.775209] pcieport 0000:00:03.0: AER: Multiple Corrected error
>> received: id=0018
>>> [ 199.775238] pcieport 0000:00:03.0: PCIe Bus Error:
>> severity=Corrected, type=Data Link Layer, id=0018(Transmitter ID)
>>> [ 199.775251] pcieport 0000:00:03.0: device [8086:340a] error
>> status/mask=00001100/00002000
>>> [ 199.775255] pcieport 0000:00:03.0: [ 8] RELAY_NUM Rollover
>>> [ 199.775258] pcieport 0000:00:03.0: [12] Replay Timer Timeout
>>>
>>> hint at a problem in the system's design. 00:03.0 is the parent bridge
>>> of 02:00.0 (and from what I can tell that's the only device behind that
>>> bridge), and hence the above messages can only reasonably have
>>> their origin at the passed through VGA device.
>>
Okay, I did a bisection and was not able to correlate the above error
message with the problem I'm seeing. Not saying it's not related, but I
had plenty of successful test runs in the presence of that error.
Took me about a week (sometimes it takes as much as 6 hours to produce
the error), but bisect narrowed it down to this commit:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9a727a813e9b25003e433b3dc3fa47e621f9e238
What do you think?
Thanks!
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-17 19:21 ` Steve Freitas
@ 2014-11-18 7:54 ` Jan Beulich
2014-11-20 1:23 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-18 7:54 UTC (permalink / raw)
To: Steve Freitas, pasik; +Cc: Don Slutz, xen-devel
>>> On 17.11.14 at 20:21, <sflist@ihonk.com> wrote:
> Okay, I did a bisection and was not able to correlate the above error
> message with the problem I'm seeing. Not saying it's not related, but I
> had plenty of successful test runs in the presence of that error.
>
> Took me about a week (sometimes it takes as much as 6 hours to produce
> the error), but bisect narrowed it down to this commit:
>
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9a727a813e9b25003e433b3d
> c3fa47e621f9e238
>
> What do you think?
Thanks for narrowing this, even if this change didn't show any other
bad effects so far (and it's been widely tested by now), and even if
problems here would generally be expected to surface independent
of the use of PCI pass-through. But a hang (rather than a crash)
would indeed be the most natural result of something being wrong
here. To double check the result, could you, in an up-to-date tree,
simply make x86's arch_skip_send_event_check() return 0
unconditionally? Plus, without said adjustment, first just disable the
MWAIT CPU idle driver ("mwait-idle=0") and then, if that didn't make
a difference, use of C states altogether ("cpuidle=0"). If any of this
does make a difference, limiting use of C states without fully
excluding their use may need to be the next step.
Another thing - now that serial logging appears to be working for
you, did you try whether the host, once hung, still reacts to serial
input (perhaps force input to go to Xen right at boot via the
"conswitch=" option)? If so, 'd' debug-key output would likely be
the piece of most interest. If not, try adding "watchdog" on the Xen
command line (this doesn't always work, but should on the CPU you
originally reported the issue for).
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-18 7:54 ` Jan Beulich
@ 2014-11-20 1:23 ` Steve Freitas
2014-11-20 7:59 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-20 1:23 UTC (permalink / raw)
To: Jan Beulich, pasik; +Cc: Don Slutz, xen-devel
On 11/17/2014 23:54, Jan Beulich wrote:
>>>> On 17.11.14 at 20:21, <sflist@ihonk.com> wrote:
>> Okay, I did a bisection and was not able to correlate the above error
>> message with the problem I'm seeing. Not saying it's not related, but I
>> had plenty of successful test runs in the presence of that error.
>>
>> Took me about a week (sometimes it takes as much as 6 hours to produce
>> the error), but bisect narrowed it down to this commit:
>>
>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9a727a813e9b25003e433b3d
>> c3fa47e621f9e238
>>
>> What do you think?
> Thanks for narrowing this, even if this change didn't show any other
> bad effects so far (and it's been widely tested by now), and even if
> problems here would generally be expected to surface independent
> of the use of PCI pass-through. But a hang (rather than a crash)
> would indeed be the most natural result of something being wrong
> here. To double check the result, could you, in an up-to-date tree,
> simply make x86's arch_skip_send_event_check() return 0
> unconditionally?
Made this change and the host was happy.
> Plus, without said adjustment, first just disable the
> MWAIT CPU idle driver ("mwait-idle=0") and then, if that didn't make
> a difference, use of C states altogether ("cpuidle=0"). If any of this
> does make a difference, limiting use of C states without fully
> excluding their use may need to be the next step.
Will do this next.
> Another thing - now that serial logging appears to be working for
> you, did you try whether the host, once hung, still reacts to serial
> input (perhaps force input to go to Xen right at boot via the
> "conswitch=" option)? If so, 'd' debug-key output would likely be
> the piece of most interest.
Here you go. Performed with a checkout of 9a727a81 (because it was
handy), let me know if you'd rather see the results from 4.5-rc2 or any
other Xen debugging info:
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000281e2c1>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 00003acd4939f3e7 rbx: 00003acd493a0cce rcx: 000000000000ffff
(XEN) rdx: 00003acd00000000 rsi: 0000000000000000 rdi: 0000000000000057
(XEN) rbp: 000000000000645c rsp: fffff880033edf90 r8: fffff880033edff0
(XEN) r9: 0000000000000000 r10: fffff880033ee040 r11: 0000000342934690
(XEN) r12: fffff880033ee3c8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000066aca000 cr2: fffff98002680000
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a943e000 rcx: 0000000000000001
(XEN) rdx: ffff830c3dc70000 rsi: 0000000000000004 rdi: ffff830c3dc7a088
(XEN) rbp: ffff830c3dc77ec8 rsp: ffff830c3dc77e40 r8: ffff830c3dc7a0a0
(XEN) r9: 0000000000000000 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: 0000151cc8b48756 r13: ffff8300a943e000 r14: ffff830c3dc7a088
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c18962000 cr2: 00000000ff331aa0
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc77e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dc7a0a0 0000000100c77e78
(XEN) ffff830c3dc7a080 ffff82d0801b5277 ffff8300a943e000 fffff88002fe2d70
(XEN) ffff8300a943e000 0000000001c9c380 ffff82d0801e0f00 ffff830c3dc77f08
(XEN) ffff82d0802f8080 ffff82d0802f8000 ffffffffffffffff ffff830c3dc70000
(XEN) 0000000000000001 ffff830c3dc77ef8 ffff82d08012a1b3 ffff8300a943e000
(XEN) fffff88002fe2d70 000036d08fbeebe8 000000000000000f ffff830c3dc77f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 000036d08fbeebe8 fffff88002fe2d70 000000000000000f
(XEN) fffff88002fd8180 fffff88002fe2d70 fffff88002fd82a0 000034711df61755
(XEN) fffff88002fd82a0 0000000000000002 fffff88002fd81c0 0000000000000400
(XEN) 0000000000000000 fffff88002fe2eb0 0000beef0000beef fffff8000298520c
(XEN) 000000bf0000beef 0000000000000046 fffff88002fe2c20 000000000000beef
(XEN) c2c2c2c2c2c2beef c2c2c2c2c2c2beef c2c2c2c2c2c2beef c2c2c2c2c2c2beef
(XEN) c2c2c2c200000001 ffff8300a943e000 0000003bbd958e00 c2c2c2c2c2c2c2c2
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000298520c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000034711df61755 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 000036d08fbeebe8 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000ff331aa0
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000298520e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: 000034edd4ec417c r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 000036d08fe55d56 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000776ebfb8
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d0801d4bb4>] enable_intr_window+0xe4/0xed
(XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor
(XEN) rax: 00000000b6a065fe rbx: 000000000000d202 rcx: 0000000000000000
(XEN) rdx: 0000000000000004 rsi: 000000000000d202 rdi: ffff8300a942f000
(XEN) rbp: ffff830c3dca7e98 rsp: ffff830c3dca7e68 r8: ffff8300a942f000
(XEN) r9: 00000000ffffffff r10: ffff830c3f68d650 r11: fffff80000ba6d30
(XEN) r12: ffff8300a942f000 r13: 000000000000d202 r14: 00000000000000d2
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1b519000 cr2: 0000000004af6354
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7e68:
(XEN) ffff830c3dca7e98 ffff82d0801bdff8 ffff830c3dca7e98 ffff8300a942f000
(XEN) ffff8300a942f000 000000000000d202 ffff830c3dca7f08 ffff82d0801d4ea4
(XEN) ffff82d0802f8000 000000d2ffffffff ffff830c3dca0000 0000000000000001
(XEN) ffff830c3dca7ef8 ffff82d08012a1b3 ffff8300a942f000 ffff8300a942f000
(XEN) 0000151cdcad2c52 ffff8300a942f000 ffff830c3dcaa088 0000000001c9c380
(XEN) ffff830c3dca7e28 ffff82d0801e3c86 0000000000000001 000000000000000f
(XEN) 000036d08fbabfb5 fffff80000ba6d30 000000000000000f fffff80002a47e80
(XEN) fffff80000ba6d30 fffff80002a47fa0 0000362abb330cfb fffff80002a47fa0
(XEN) 0000000000000002 fffff80002a47ec0 0000000000000400 0000000000000000
(XEN) fffff80000ba6e70 0000beef0000beef fffff8000298520c 000000bf0000beef
(XEN) 0000000000000046 fffff80000ba6be0 000000000000beef 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 0000000000000003
(XEN) ffff8300a942f000 0000003bbd988e00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801d4bb4>] enable_intr_window+0xe4/0xed
(XEN) [<ffff82d0801d4ea4>] vmx_intr_assist+0x28c/0x51c
(XEN) [<ffff82d0801e3c86>] vmx_asm_vmexit_handler+0x46/0xc0
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298520c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a47e80 rcx: fffff80002a47ec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80000ba6e70
(XEN) rbp: 000000000000000f rsp: fffff80000ba6be0 r8: fffff80002a47fa0
(XEN) r9: 0000362abb330cfb r10: fffff80002a47fa0 r11: fffff80000ba6d30
(XEN) r12: fffff80000ba6d30 r13: 000036d08fbabfb5 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000004af6354
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff82d080321280 rcx: 0000000000000004
(XEN) rdx: ffff830c3dc90000 rsi: ffff8300a942e000 rdi: ffff830c3dc9c088
(XEN) rbp: ffff830c3dc97d58 rsp: ffff830c3dc97d10 r8: fffff88002e402a0
(XEN) r9: 000032242fda2dba r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: ffff8300a942e000 r13: ffff830c3dc9c088 r14: ffff82d080321280
(XEN) r15: ffff830c3dc9c088 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1b505000 cr2: fffff8a00033e03e
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc97d10:
(XEN) ffff82d080128f45 0000000000000006 ffff830c3dc97d38 ffff82d08012aa17
(XEN) 0000000000000000 ffff8300a942e000 0000000000000028 0000000000000000
(XEN) 0000000000000000 ffff830c3dc97d88 ffff82d080129079 0000000000000246
(XEN) ffff830c3dc97d88 ffff82d08012a80f ffff830c3dc97f18 ffff830c3dc97f08
(XEN) ffff82d0801dda32 ffff8300a942e000 ffff8300a942e000 ffff8300a942e000
(XEN) ffff830c3dc9c088 ffff830c3dc97e08 0000000000000000 ffff830c3dc97de8
(XEN) ffff830c3dc90000 ffff830c3dc9c0a0 ffff8300a942e000 0000151cebb9d23b
(XEN) ffff8300a942e000 ffff830c3dc9c088 0000000001c9c380 0000000000000292
(XEN) ffff830c3dc97e28 ffff82d08012a80f ffff8300a942e000 ffff830c3dc97e98
(XEN) ffff82d0801caea5 ffff830c3dc97ec8 ffff8300a942e508 ffff830c3dc97e58
(XEN) ffff82d0801c89a3 ffff830c3dc97e78 ffff830c3dc97e88 ffff82d0801b5277
(XEN) ffff8300a942e000 0000151cebb9d23b ffff8300a942e000 ffff830c3dc97f08
(XEN) ffff82d0801e0f69 ffff830c3dc97f18 ffff8300a942e000 ffff830c3dc97f08
(XEN) ffff82d0801ddba6 ffff830c3dc90000 0000000000000001 ffff830c3dc97ef8
(XEN) ffff82d08012a1b3 ffff8300a942e000 ffff8300a942e000 fffff88002e4ad70
(XEN) 000036d08fbab094 000000000000000f 0000000000000001 000000000000000f
(XEN) ffff82d0801e3c81 0000000000000001 000000000000000f 000036d08fbab094
(XEN) fffff88002e4ad70 000000000000000f fffff88002e40180 fffff88002e4ad70
(XEN) fffff88002e402a0 000032242fda2dba fffff88002e402a0 0000000000000002
(XEN) fffff88002e401c0 0000000000000400 0000000000000000 fffff88002e4aeb0
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d080129079>] do_sched_op_compat+0x26/0xa1
(XEN) [<ffff82d0801dda32>] vmx_vmexit_handler+0x1845/0x195e
(XEN) [<ffff82d0801e3c81>] vmx_asm_vmexit_handler+0x41/0xc0
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000298520c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 000032242fda2dba r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 000036d08fbab094 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: fffff8a00033e03e
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a942c000 rcx: 0000000000000005
(XEN) rdx: ffff830c3dc80000 rsi: 0000000000000005 rdi: ffff830c3dc8e088
(XEN) rbp: ffff830c3dc87ec8 rsp: ffff830c3dc87e40 r8: ffff830c3dc8e0a0
(XEN) r9: 0000000000000000 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: 0000151cfdff3536 r13: ffff8300a942c000 r14: ffff830c3dc8e088
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c19cb4000 cr2: 0000000001550320
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc87e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dc8e0a0 0000000500c87e78
(XEN) ffff830c3dc8e080 ffff82d0801b5277 ffff8300a942c000 fffff88002f36d70
(XEN) ffff8300a942c000 0000000001c9c380 ffff82d0801e0f00 ffff830c3dc87f08
(XEN) ffff82d0802f8280 ffff82d0802f8000 ffffffffffffffff ffff830c3dc80000
(XEN) 0000000000000001 ffff830c3dc87ef8 ffff82d08012a1b3 ffff8300a942c000
(XEN) fffff88002f36d70 000036d08fbb1ba4 000000000000000f ffff830c3dc87f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 000036d08fbb1ba4 fffff88002f36d70 000000000000000f
(XEN) fffff88002f2c180 fffff88002f36d70 fffff88002f2c2a0 000035b9461016e5
(XEN) fffff88002f2c2a0 0000000000000002 fffff88002f2c1c0 0000000000000400
(XEN) 0000000000000000 fffff88002f36eb0 0000beef0000beef fffff8000298520c
(XEN) 000000bf0000beef 0000000000000046 fffff88002f36c20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000005 ffff8300a942c000 0000003bbd96ce00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000298520c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 000035b9461016e5 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000036d08fbb1ba4 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000001550320
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
Thanks!
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-20 1:23 ` Steve Freitas
@ 2014-11-20 7:59 ` Jan Beulich
2014-11-20 20:07 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-20 7:59 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 20.11.14 at 02:23, <sflist@ihonk.com> wrote:
> On 11/17/2014 23:54, Jan Beulich wrote:
>> Another thing - now that serial logging appears to be working for
>> you, did you try whether the host, once hung, still reacts to serial
>> input (perhaps force input to go to Xen right at boot via the
>> "conswitch=" option)? If so, 'd' debug-key output would likely be
>> the piece of most interest.
>
> Here you go. Performed with a checkout of 9a727a81 (because it was
> handy), let me know if you'd rather see the results from 4.5-rc2 or any
> other Xen debugging info:
Interesting - all CPUs are executing stuff from Dom1, which be itself
is not indicating a problem, but may (considering the host hang) hint
at guest vCPU-s not getting de-scheduled anymore on one or more of
the CPUs. The 'a' debug key output would hopefully give us enough
information to know whether that's the case. Ideally do 'd' and 'a'
a couple of times each (alternating, and with a few seconds in
between).
Also, for double checking purposes, could you make the xen-syms
of a build you observed the problem with available somewhere?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-20 7:59 ` Jan Beulich
@ 2014-11-20 20:07 ` Steve Freitas
2014-11-21 8:42 ` Jan Beulich
2014-11-21 9:31 ` Jan Beulich
0 siblings, 2 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-20 20:07 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, xen-devel
Hi Jan,
Thanks for all your help so far! Here's my latest update.
On 11/17/2014 23:54, Jan Beulich wrote:
> Plus, without said adjustment, first just disable the
> MWAIT CPU idle driver ("mwait-idle=0") and then, if that didn't make
> a difference, use of C states altogether ("cpuidle=0"). If any of this
> does make a difference, limiting use of C states without fully
> excluding their use may need to be the next step.
Running with mwait-idle=0 solves (hides?) the problem. Next step is to
fiddle with the C states?
On 11/19/2014 23:59, Jan Beulich wrote:
> Also, for double checking purposes, could you make the xen-syms of a
> build you observed the problem with available somewhere?
The xen-syms (from my build of 9a727a81) can be found here:
http://steve.freitas.org/xen-syms-4.5-unstable.gz
> Interesting - all CPUs are executing stuff from Dom1, which be itself
> is not indicating a problem, but may (considering the host hang) hint
> at guest vCPU-s not getting de-scheduled anymore on one or more of the
> CPUs. The 'a' debug key output would hopefully give us enough
> information to know whether that's the case. Ideally do 'd' and 'a' a
> couple of times each (alternating, and with a few seconds in between).
Here ya go (as before, from 9a727a81). I had to pick and choose what
parts to pull from the log because it was getting spammed with kernel
complaints about the disk subsystem being locked up, so the hypervisor
debugging info was hard to read amidst the kernel errors. As ever, let
me know if you need more.
Thanks again!
Steve
(XEN) CPU00:
(XEN) ex= 1445us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 9918us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 8390us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 70409499us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 12265483us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 94364us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 18390us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) CPU01:
(XEN) ex= 390us timer=ffff830c17ceb460
cb=pt_timer_fn(ffff830c17ceb420)
(XEN) ex= 14101194us timer=ffff830c17ceb4e0
cb=pt_timer_fn(ffff830c17ceb4a0)
(XEN) ex= 153445us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 44171681527us timer=ffff830c17ceb290
cb=rtc_alarm_cb(ffff830c17ceb1f0)
(XEN) CPU02:
(XEN) ex= 1445us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 8390us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 18390us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 29570us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 25445us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) CPU04:
(XEN) ex= 634us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) CPU05:
(XEN) ex= 1445us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 388096702us timer=ffff830c17ceb5d0
cb=pmt_timer_callback(ffff830c17ceb5a8)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: 000003295cdc3c57 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 000003940027203c r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000001cb8300
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000002f7d38
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff800028232c1>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000039c01e4b276 rbx: 0000039c01e4ddc3 rcx: 000000000000ffff
(XEN) rdx: 0000039c00000000 rsi: 0000000000000000 rdi: 0000000000000038
(XEN) rbp: 00000000000024b3 rsp: fffff880048d04c0 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 00000003428f7600
(XEN) r12: fffff880048d08f8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000148f940
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 00000353a8bde5bd r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000039400271946 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef88138c8
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000003071d1e7b55 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 00000394002e01ea r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000043fd28
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a9784000 rcx: 0000000000000005
(XEN) rdx: ffff830c3dc80000 rsi: 0000000000000003 rdi: ffff830c3dc8e088
(XEN) rbp: ffff830c3dc87ec8 rsp: ffff830c3dc87e40 r8: ffff830c3dc8e0a0
(XEN) r9: 000001de682f4faa r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: 000001de682108bd r13: ffff8300a9784000 r14: ffff830c3dc8e088
(XEN) r15: 00000000000f4240 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c185e7000 cr2: 000000000031f268
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc87e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dc8e0a0 0000000500c87e78
(XEN) ffff830c3dc8e080 ffff82d0801b5277 ffff8300a9784000 fffff88002f36d70
(XEN) ffff8300a9784000 00000000000f4240 ffff82d0801e0f00 ffff830c3dc87f08
(XEN) ffff82d0802f8280 ffff82d0802f8000 ffffffffffffffff ffff830c3dc80000
(XEN) 0000000000000001 ffff830c3dc87ef8 ffff82d08012a1b3 ffff8300a9784000
(XEN) fffff88002f36d70 000003940051e450 000000000000000f ffff830c3dc87f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 000003940051e450 fffff88002f36d70 000000000000000f
(XEN) fffff88002f2c180 fffff88002f36d70 fffff88002f2c2a0 0000034bbd959ead
(XEN) fffff88002f2c2a0 0000000000000002 fffff88002f2c1c0 0000000000000400
(XEN) 0000000000000000 fffff88002f36eb0 0000beef0000beef fffff8000298a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002f36c20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000005 ffff8300a9784000 0000003bbd96ce00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000003940051e450 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000031f268
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 282us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 17360us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 1690us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 2536293us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 44362799us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 957308us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 11690us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) CPU01:
(XEN) ex= 659us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 54745us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 1025us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 150745us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 128408us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 26745us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 11690us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) CPU02:
(XEN) ex= 1943us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 11690us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 30041us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 1937us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 11690us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 44145634827us timer=ffff830c17ceb290
cb=rtc_alarm_cb(ffff830c17ceb1f0)
(XEN) ex= 30048us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 1827us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 11690us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 806745us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 30058us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 1928us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 11690us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 362050002us timer=ffff830c17ceb5d0
cb=pmt_timer_callback(ffff830c17ceb5a8)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000003940051e450 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000031f268
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 00000353a8bde5bd r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000039400271946 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef88138c8
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: 000003295cdc3c57 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 000003940027203c r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000001cb8300
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d08012a132>] __do_softirq+0/0x8c
(XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor
(XEN) rax: 0000000000000246 rbx: ffff8300a9786000 rcx: 0000000000000000
(XEN) rdx: ffff82d0802f8000 rsi: ffff8300a9786000 rdi: 0000000000000000
(XEN) rbp: ffff830c3dca7f08 rsp: ffff830c3dca7f00 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c17ec0000 cr2: 00000000002f7d38
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7f00:
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 00000394002722bb fffff88002e4ad70 000000000000000f
(XEN) fffff88002e40180 fffff88002e4ad70 fffff88002e402a0 00000256dc1c8f33
(XEN) fffff88002e402a0 0000000000000002 fffff88002e401c0 0000000000000400
(XEN) 0000000000000000 fffff88002e4aeb0 0000beef0000beef fffff8000298a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002e4ac20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000003 ffff8300a9786000 0000003bbd988e00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a132>] __do_softirq+0/0x8c
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000002f7d38
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff800028232c3>]
(XEN) RFLAGS: 0000000000000006 CONTEXT: hvm guest
(XEN) rax: 00000000336e527f rbx: 000003a26d8bfe95 rcx: 000000000000ffff
(XEN) rdx: 00000000000003a1 rsi: 0000000000000000 rdi: 0000000002c0c9a9
(XEN) rbp: 0000000000000000 rsp: fffff880048d0490 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 0027cb813b6e0000
(XEN) r12: fffff880048d08f8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000148f940
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000003071d1e7b55 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 00000394002e01ea r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000043fd28
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 119us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 11733us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) ex= 1733us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 28524us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 77612us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 35482842us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 9656339us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 44136754870us timer=ffff830c17ceb290
cb=rtc_alarm_cb(ffff830c17ceb1f0)
(XEN) CPU01:
(XEN) ex= 2406us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 11733us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 74788us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 270788us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 30027us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 1952us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 78788us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 11733us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 1066788us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 118766us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 30038us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 1926us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 11733us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) CPU04:
(XEN) ex= 1963us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 11733us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 30059us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 1978us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 11733us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 353170045us timer=ffff830c17ceb5d0
cb=pmt_timer_callback(ffff830c17ceb5a8)
(XEN) ex= 30065us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000298a217>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000003071d1e7b55 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 00000394002e01ea r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000043fd28
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000002f7d38
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff800028232c3>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000010984f60 rbx: 000003a710988443 rcx: 000000000000ffff
(XEN) rdx: 00000000000003a7 rsi: 0000000000000000 rdi: 0000000000000007
(XEN) rbp: 000000000000af00 rsp: fffff880048d04c0 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 00000003428f7600
(XEN) r12: fffff880048d08f8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000148f940
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d08011cc9d>] ASSERT_NOT_IN_ATOMIC+0x37/0x52
(XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor
(XEN) rax: 0000000000000180 rbx: ffff8300a9784000 rcx: 0000000000000001
(XEN) rdx: 0000003bbd988e00 rsi: ffff830c3dcaad20 rdi: 000000005c723445
(XEN) rbp: ffff830c3dca7e38 rsp: ffff830c3dca7e38 r8: 000001e23194cc09
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: 000001e25298787d r13: ffff830c3dcaa130 r14: ffff830c3dca0000
(XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c185e7000 cr2: 000000000031f268
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7e38:
(XEN) ffff830c3dca7ec8 ffff82d080126d91 ffff830c3dca7e58 ffff82d0801c89a3
(XEN) 000000033dca7e78 ffff830c3dca7e88 ffff82d0801b5277 ffff8300a9784000
(XEN) fffff88002f36d70 000003940051e450 ffff830c3dca7f08 ffff82d0801e0f69
(XEN) ffff830c3dca7f08 ffff82d0802f8180 ffff82d0802f8000 ffffffffffffffff
(XEN) ffff830c3dca0000 0000000000000001 ffff830c3dca7ef8 ffff82d08012a1b3
(XEN) ffff8300a9784000 fffff88002f36d70 000003940051e450 000000000000000f
(XEN) ffff830c3dca7f08 ffff82d08012a20b 000000000000000f ffff82d0801e3d2a
(XEN) 0000000000000001 000000000000000f 000003940051e450 fffff88002f36d70
(XEN) 000000000000000f fffff88002f2c180 fffff88002f36d70 fffff88002f2c2a0
(XEN) 0000034bbd959ead fffff88002f2c2a0 0000000000000002 fffff88002f2c1c0
(XEN) 0000000000000400 0000000000000000 fffff88002f36eb0 0000beef0000beef
(XEN) fffff8000298a20c 000000bf0000beef 0000000000000046 fffff88002f36c20
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 000000000000beef 0000000000000003 ffff8300a9784000 0000003bbd988e00
(XEN) 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08011cc9d>] ASSERT_NOT_IN_ATOMIC+0x37/0x52
(XEN) [<ffff82d080126d91>] schedule+0x47/0x612
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000003940051e450 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000031f268
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: e008:[<ffff82d0801a8461>] mwait_idle+0x2ab/0x2ff
(XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: 000001e262c8819c rcx: 20c49ba5e353f7cf
(XEN) rdx: ffff830c3dc90000 rsi: 000001e262c8819c rdi: ffff830c3dcacba8
(XEN) rbp: ffff830c3dc97ef0 rsp: ffff830c3dc97e80 r8: 000001e23194cc09
(XEN) r9: 0000000000000000 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: ffff830c3dcacb80 r13: 0000000000000003 r14: 0000000000000004
(XEN) r15: ffff830c3dcacc10 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c18947000 cr2: 0000000001cb8300
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc97e80:
(XEN) 00000010bf79b000 000001e25ccbd093 ffff82d0801e0f00 ffff830c3dc97f08
(XEN) ffff82d0802f8200 0000000000000000 0000000000000000 000002e800000115
(XEN) 0000000000000001 ffff830c3dc90000 ffff8300a978f000 00000000ffffff01
(XEN) ffff830c3dc9c088 0000000000000003 ffff830c3dc97f10 ffff82d08015ee06
(XEN) ffff82d08012a20b ffff8300bf79b000 ffff830c3dc97de8 0000000000000001
(XEN) 000000000000000f 000003940027203c fffff88002facd70 000000000000000f
(XEN) fffff88002fa2180 fffff88002facd70 fffff88002fa22a0 000003295cdc3c57
(XEN) fffff88002fa22a0 0000000000000002 fffff88002fa21c0 0000000000000400
(XEN) 0000000000000000 fffff88002faceb0 0000beef0000beef fffff8000298a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002facc20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000004 ffff8300bf79b000 0000003bbd97ae00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801a8461>] mwait_idle+0x2ab/0x2ff
(XEN) [<ffff82d08015ee06>] idle_loop+0x51/0x6b
(XEN)
(XEN) *** Dumping CPU5 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82d0801a8461>] mwait_idle+0x2ab/0x2ff
(XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: 000001e26cdd6ba0 rcx: 20c49ba5e353f7cf
(XEN) rdx: ffff830c3dc80000 rsi: 000001e26cdd6ba0 rdi: ffff830c3dc8d448
(XEN) rbp: ffff830c3dc87ef0 rsp: ffff830c3dc87e80 r8: 000001e23194cc09
(XEN) r9: ffff830c17ceb5d0 r10: 0000000000000000 r11: 000000000000000a
(XEN) r12: ffff830c3dc8d420 r13: 0000000000000004 r14: 0000000000000008
(XEN) r15: ffff830c3dc8d4d0 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 00000000bf48b000 cr2: 000007fef88138c8
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc87e80:
(XEN) 00000020bf79a000 000001e263681669 ffff82d0802f8000 ffffffffffffffff
(XEN) ffff830c3dc80000 0000000000000000 0000000000000000 00001f3e0000130a
(XEN) ffff830c3dc80000 ffff830c3dc80000 000001e263680d45 ffff8300bf79a000
(XEN) ffff830c3dc8e088 ffffffffffffffff ffff830c3dc87f10 ffff82d08015ee06
(XEN) ffff82d08012a20b ffff8300bf79a000 ffff830c3dc87e10 0000000000000001
(XEN) 000000000000000f 00000394002722bb fffff88002e4ad70 000000000000000f
(XEN) fffff88002e40180 fffff88002e4ad70 fffff88002e402a0 00000256dc1c8f33
(XEN) fffff88002e402a0 0000000000000002 fffff88002e401c0 0000000000000400
(XEN) 0000000000000000 fffff88002e4aeb0 0000beef0000beef fffff8000298a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002e4ac20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000005 ffff8300bf79a000 0000003bbd96ce00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801a8461>] mwait_idle+0x2ab/0x2ff
(XEN) [<ffff82d08015ee06>] idle_loop+0x51/0x6b
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 597us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 597us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) ex= 15145207us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 10541us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 155653us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 24971706us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 44126243734us timer=ffff830c17ceb290
cb=rtc_alarm_cb(ffff830c17ceb1f0)
(XEN) ex= 730445us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 107653us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) CPU01:
(XEN) ex= 597us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 1023us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 259653us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 30029us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 597us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 1074us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 30039us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 597us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 947us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 563653us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 30047us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 597us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 996us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 30055us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 155us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 981us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 597us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 342658909us timer=ffff830c17ceb5d0
cb=pmt_timer_callback(ffff830c17ceb5a8)
(XEN) ex= 91653us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 1062us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000003940051e450 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000031f268
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: 000003295cdc3c57 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 000003940027203c r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000001cb8300
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 00000353a8bde5bd r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000039400271946 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef88138c8
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000003071d1e7b55 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 00000394002e01ea r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000043fd28
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: e008:[<ffff82d0801dc4fe>] vmx_vmexit_handler+0x311/0x195e
(XEN) RFLAGS: 0000000000000297 CONTEXT: hypervisor
(XEN) rax: 0000000000187000 rbx: ffff830c3dc97f18 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: ffff830c3dc97f18
(XEN) rbp: ffff830c3dc97f08 rsp: ffff830c3dc97d98 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: ffff8300a9786000 r13: 0000000000000028 r14: 0000000000000000
(XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c17ec0000 cr2: 00000000002f7d38
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc97d98:
(XEN) ffff8300a9786000 ffff8300a9786000 ffff8300a9786000 ffff830c3dc9c088
(XEN) ffff830c3dc97e08 0000000000000000 ffff830c3dc97de8 ffff830c3dc90000
(XEN) ffff830c3dc9c0a0 ffff8300a9786000 000001e512e27b8b ffff8300a9786000
(XEN) ffff830c3dc9c088 0000000001c9c380 0000000000000292 ffff830c3dc97e28
(XEN) ffff82d08012a80f ffff8300a9786000 ffff830c3dc97e98 ffff82d0801caea5
(XEN) ffff830c3dc97ec8 ffff8300a9786508 ffff82d080321280 ffff830c3dc9c0a0
(XEN) ffff830c3dc97e78 ffff830c3dc97e88 ffff82d0801b5277 ffff8300a9786000
(XEN) 000001e512e27b8b ffff8300a9786000 ffff830c3dc97f08 ffff82d0801e0f69
(XEN) ffff830c3dc97f18 ffff8300a9786000 ffff830c3dc97f08 ffff82d0801ddba6
(XEN) ffff830c3dc90000 0000000000000001 ffff830c3dc97ef8 ffff82d08012a1b3
(XEN) ffff8300a9786000 ffff8300a9786000 fffff88002e4ad70 00000394002722bb
(XEN) 000000000000000f 0000000000000001 000000000000000f ffff82d0801e3c81
(XEN) 0000000000000001 000000000000000f 00000394002722bb fffff88002e4ad70
(XEN) 000000000000000f fffff88002e40180 fffff88002e4ad70 fffff88002e402a0
(XEN) 00000256dc1c8f33 fffff88002e402a0 0000000000000002 fffff88002e401c0
(XEN) 0000000000000400 0000000000000000 fffff88002e4aeb0 0000beef0000beef
(XEN) fffff8000298a20c 000000bf0000beef 0000000000000046 fffff88002e4ac20
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 000000000000beef 0000000000000004 ffff8300a9786000 0000003bbd97ae00
(XEN) 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801dc4fe>] vmx_vmexit_handler+0x311/0x195e
(XEN) [<ffff82d0801e3c81>] vmx_asm_vmexit_handler+0x41/0xc0
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000002f7d38
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82d08012a825>] _spin_lock+0x27/0x54
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000001 rbx: ffff830c17ceb628 rcx: 0000000000000000
(XEN) rdx: 0000000000000000 rsi: 0000000000000008 rdi: ffff830c17ceb62c
(XEN) rbp: ffff830c3dc87df8 rsp: ffff830c3dc87df0 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 00000003428f7600
(XEN) r12: 0000000000000008 r13: 0000000000000008 r14: ffff830c17ceb628
(XEN) r15: ffff8300a9787000 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c180ec000 cr2: 000000000148f940
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc87df0:
(XEN) ffff830c17ceb000 ffff830c3dc87e28 ffff82d0801c08e6 0000000000000008
(XEN) 0000000000000008 0000000000000001 ffff8300a9787000 ffff830c3dc87e98
(XEN) ffff82d0801caef7 ffff830c3dc87ec8 ffff8300a9787508 000000023dc87e58
(XEN) ffff830c17ceb4a0 00000143f866c070 ffff8300a9787510 ffff82d0801b5277
(XEN) ffff8300a9787000 fffff880048d08f8 0000000000001000 0000000000000000
(XEN) 0000000000000058 ffff830c3dc87f08 ffff82d0801d4c75 ffff830c3dc87f08
(XEN) ffffffff801ddba6 ffff830c3dc80000 0000000000000058 ffff830c3dc87ef8
(XEN) ffff82d08012a1b3 ffff8300a9787000 ffff8300a9787000 fffff880048d08f8
(XEN) 0000000000001000 0000000000000000 0000000000000058 000000000000897d
(XEN) ffff82d0801e3c86 0000000000000058 0000000000000000 0000000000001000
(XEN) fffff880048d08f8 000000000000897d 000003af014f24a8 00000003428f7600
(XEN) fffff880048d0570 0000000000000000 fffff880048d0520 000003af014efe7d
(XEN) 000000000000ffff 000003af00000000 0000000000000000 0000000000000052
(XEN) 0000beef0000beef fffff800028232bf 000000bf0000beef 0000000000000002
(XEN) fffff880048d04c0 000000000000beef 000000000000beef 000000000000beef
(XEN) 000000000000beef 000000000000beef 0000000000000005 ffff8300a9787000
(XEN) 0000003bbd96ce00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a825>] _spin_lock+0x27/0x54
(XEN) [<ffff82d0801c08e6>] hvm_isa_irq_deassert+0x34/0x77
(XEN) [<ffff82d0801caef7>] pt_update_irq+0x1b8/0x230
(XEN) [<ffff82d0801d4c75>] vmx_intr_assist+0x5d/0x51c
(XEN) [<ffff82d0801e3c86>] vmx_asm_vmexit_handler+0x46/0xc0
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff800028232bf>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 000003af014efe7d rbx: 000003af014f24a8 rcx: 000000000000ffff
(XEN) rdx: 000003af00000000 rsi: 0000000000000000 rdi: 0000000000000052
(XEN) rbp: 000000000000897d rsp: fffff880048d04c0 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 00000003428f7600
(XEN) r12: fffff880048d08f8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000148f940
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 1796us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 6341us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 16341us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) ex= 4520950us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 891686us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 14347450us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) CPU01:
(XEN) ex= 6500us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 16341us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 939396us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 67396us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 30022us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 6493us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 16341us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 135396us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 30035us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 753us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 1042us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 44115619478us timer=ffff830c17ceb290
cb=rtc_alarm_cb(ffff830c17ceb1f0)
(XEN) ex= 16341us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 6535us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) CPU04:
(XEN) ex= 4039us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 6506us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 16341us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 30055us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 6536us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 16341us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 755722us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 332034653us timer=ffff830c17ceb5d0
cb=pmt_timer_callback(ffff830c17ceb5a8)
(XEN) ex= 30066us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 00000353a8bde5bd r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000039400271946 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef88138c8
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff800028232c1>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 000003b6a07ae098 rbx: 000003b6a07b06ed rcx: 000000000000ffff
(XEN) rdx: 000003b600000000 rsi: 0000000000000000 rdi: 0000000000000051
(XEN) rbp: 0000000000006ed0 rsp: fffff880048d04c0 r8: fffff880048d0520
(XEN) r9: 0000000000000000 r10: fffff880048d0570 r11: 00000003428f7600
(XEN) r12: fffff880048d08f8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000148f940
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000034bbd959ead r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000003940051e450 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000031f268
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000298a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: 000003295cdc3c57 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 000003940027203c r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 0000000001cb8300
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: 00000256dc1c8f33 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 00000394002722bb r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 00000000002f7d38
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000298a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000003071d1e7b55 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 00000394002e01ea r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000000000043fd28
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) 'R' pressed -> rebooting machine
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-20 20:07 ` Steve Freitas
@ 2014-11-21 8:42 ` Jan Beulich
2014-11-23 1:28 ` Steve Freitas
2014-11-21 9:31 ` Jan Beulich
1 sibling, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-21 8:42 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 20.11.14 at 21:07, <sflist@ihonk.com> wrote:
> Running with mwait-idle=0 solves (hides?) the problem. Next step is to
> fiddle with the C states?
For that I'd first of all like to know how much use of C states the
system still makes with that option in place. For that I'd need the
output of "xenpm get-cpuidle-states" (actually for comparison
purposes seeing it once with and once without that option in
place would be best). Or alternatively 'c' debug key output.
>> Interesting - all CPUs are executing stuff from Dom1, which be itself
>> is not indicating a problem, but may (considering the host hang) hint
>> at guest vCPU-s not getting de-scheduled anymore on one or more of the
>> CPUs. The 'a' debug key output would hopefully give us enough
>> information to know whether that's the case. Ideally do 'd' and 'a' a
>> couple of times each (alternating, and with a few seconds in between).
>
> Here ya go (as before, from 9a727a81). I had to pick and choose what
> parts to pull from the log because it was getting spammed with kernel
> complaints about the disk subsystem being locked up, so the hypervisor
> debugging info was hard to read amidst the kernel errors. As ever, let
> me know if you need more.
Pretty strange: While the first 'a' output suggests that several CPUs
are idle, the first 'd' output contradicts. While this isn't wrong by itself,
it may provide a hint I'm simply unable to interpret yet. In any event
I didn't find what I expected - timers the expiry of which was in the
past.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-20 20:07 ` Steve Freitas
2014-11-21 8:42 ` Jan Beulich
@ 2014-11-21 9:31 ` Jan Beulich
1 sibling, 0 replies; 39+ messages in thread
From: Jan Beulich @ 2014-11-21 9:31 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 20.11.14 at 21:07, <sflist@ihonk.com> wrote:
> Running with mwait-idle=0 solves (hides?) the problem. Next step is to
> fiddle with the C states?
So this also prompted me to go over the list of errata. Just to confirm
- your CPU is family 6 model 44? What stepping? And what nominal
frequency?
There are a couple potentially relevant errata (BC36, BC38, BC54,
BC77, BC110).
To exclude BC36, a boot log with "apic-verbosity=debug" and debug
key 'i' output would be necessary.
BC38 should not affect us since we don't enter C states from ISRs.
BC54 is probably irrelevant since we meanwhile know that your
system doesn't really hang hard.
For BC77 it would be worth trying to disable turbo mode instead of
disabling the mwait-idle driver ("xenpm disable-turbo-mode" right
after boot).
And BC110 would be relevant only if without the mwait-idle driver
there would be no use of C3. Plus anyway this would more likely end
up in a hard hang too.
And then, considering that my system with a family 6 model 44 CPU
has never shown anything similar (albeit that doesn't mean all that
much since our workloads are likely very different), you're not
over-clocking? And did you disable hyper-threading on purpose (if
so could you check whether enabling it makes a difference)?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-21 8:42 ` Jan Beulich
@ 2014-11-23 1:28 ` Steve Freitas
2014-11-24 8:45 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-23 1:28 UTC (permalink / raw)
To: Jan Beulich, xen-devel, Pasi Kärkkäinen, Don Slutz
On 11/21/2014 0:42, Jan Beulich wrote:
>>>> On 20.11.14 at 21:07, <sflist@ihonk.com> wrote:
>> Running with mwait-idle=0 solves (hides?) the problem. Next step is to
>> fiddle with the C states?
> For that I'd first of all like to know how much use of C states the
> system still makes with that option in place. For that I'd need the
> output of "xenpm get-cpuidle-states" (actually for comparison
> purposes seeing it once with and once without that option in
> place would be best). Or alternatively 'c' debug key output.
With mwait-idle=0:
(XEN) 'c' pressed -> printing ACPI Cx structures
(XEN) ==cpu0==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930624784]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
(XEN) ==cpu1==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930681815]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
(XEN) ==cpu2==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930766930]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
(XEN) ==cpu3==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930824772]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
(XEN) ==cpu4==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930882528]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
(XEN) ==cpu5==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
duration[0]
(XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
duration[0]
(XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
duration[0]
(XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
duration[0]
(XEN) *C0: usage[00000000] duration[46930940315]
(XEN) PC2[0] PC3[0] PC6[0] PC7[0]
(XEN) CC3[0] CC6[0] CC7[0]
Without mwait-idle=0:
(XEN) 'c' pressed -> printing ACPI Cx structures
(XEN) ==cpu0==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00004503] method[ FFH]
duration[1668990535]
(XEN) C2: type[C1] latency[010] usage[00009956] method[ FFH]
duration[4639219546]
(XEN) C3: type[C2] latency[020] usage[00654270] method[ FFH]
duration[882706893390]
(XEN) C4: type[C3] latency[200] usage[01483059] method[ FFH]
duration[35921809585234]
(XEN) *C0: usage[02151788] duration[62576274678]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[870647607639] CC6[35852001619202] CC7[0]
(XEN) ==cpu1==
(XEN) active state: C4
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00297689] method[ FFH]
duration[1941100848]
(XEN) C2: type[C1] latency[010] usage[00001168] method[ FFH]
duration[1442327615]
(XEN) C3: type[C2] latency[020] usage[00006546] method[ FFH]
duration[8936496716]
(XEN) *C4: type[C3] latency[200] usage[01558115] method[ FFH]
duration[36835128713631]
(XEN) C0: usage[01863518] duration[25952414199]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[8737041745] CC6[36744372677161] CC7[0]
(XEN) ==cpu2==
(XEN) active state: C4
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00289127] method[ FFH]
duration[1652020841]
(XEN) C2: type[C1] latency[010] usage[00000522] method[ FFH]
duration[1133871888]
(XEN) C3: type[C2] latency[020] usage[00004190] method[ FFH]
duration[5497931261]
(XEN) *C4: type[C3] latency[200] usage[01466569] method[ FFH]
duration[36844578249201]
(XEN) C0: usage[01760408] duration[20539080318]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[5364573735] CC6[36756863793181] CC7[0]
(XEN) ==cpu3==
(XEN) active state: C4
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00323774] method[ FFH]
duration[21886547916]
(XEN) C2: type[C1] latency[010] usage[00000609] method[ FFH]
duration[45620838149]
(XEN) C3: type[C2] latency[020] usage[00006012] method[ FFH]
duration[1209202160481]
(XEN) *C4: type[C3] latency[200] usage[00081570] method[ FFH]
duration[35586514444690]
(XEN) C0: usage[00411965] duration[10177272890]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[1208967409452] CC6[35580116939846] CC7[0]
(XEN) ==cpu4==
(XEN) active state: C4
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00305724] method[ FFH]
duration[2083310858]
(XEN) C2: type[C1] latency[010] usage[00000597] method[ FFH]
duration[2346319460]
(XEN) C3: type[C2] latency[020] usage[00004138] method[ FFH]
duration[67793456157]
(XEN) *C4: type[C3] latency[200] usage[00274791] method[ FFH]
duration[36779940944067]
(XEN) C0: usage[00585250] duration[21237335856]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[67665029339] CC6[36762696627875] CC7[0]
(XEN) ==cpu5==
(XEN) active state: C4
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[00299747] method[ FFH]
duration[20553360971]
(XEN) C2: type[C1] latency[010] usage[00000609] method[ FFH]
duration[42279548469]
(XEN) C3: type[C2] latency[020] usage[00005673] method[ FFH]
duration[1169803847553]
(XEN) *C4: type[C3] latency[200] usage[00095108] method[ FFH]
duration[35630372847429]
(XEN) C0: usage[00401137] duration[10391860839]
(XEN) PC2[0] PC3[1932546211563] PC6[32621026287476] PC7[0]
(XEN) CC3[1169586367967] CC6[35622884210881] CC7[0]
>>> Interesting - all CPUs are executing stuff from Dom1, which be itself
>>> is not indicating a problem, but may (considering the host hang) hint
>>> at guest vCPU-s not getting de-scheduled anymore on one or more of the
>>> CPUs. The 'a' debug key output would hopefully give us enough
>>> information to know whether that's the case. Ideally do 'd' and 'a' a
>>> couple of times each (alternating, and with a few seconds in between).
>> Here ya go (as before, from 9a727a81). I had to pick and choose what
>> parts to pull from the log because it was getting spammed with kernel
>> complaints about the disk subsystem being locked up, so the hypervisor
>> debugging info was hard to read amidst the kernel errors. As ever, let
>> me know if you need more.
> Pretty strange: While the first 'a' output suggests that several CPUs
> are idle, the first 'd' output contradicts. While this isn't wrong by itself,
> it may provide a hint I'm simply unable to interpret yet. In any event
> I didn't find what I expected - timers the expiry of which was in the
> past.
Would this strangeness be explained if the first invocations of 'a' and
'd' weren't actually contiguous? Because of the log spamming I had to
discard a number of invocations, which I think I did between the first
two invocations I passed on to you. Here's another attempt, these are
contiguous, this is from an attempt with turbo mode disabled:
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 2680us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 7344us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) ex= 7344us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 989016us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 14334142us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 143474414us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 500179us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 2272179us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) CPU01:
(XEN) ex= 4179us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 7914us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 7344us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) CPU02:
(XEN) ex= 7344us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 7408us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 236179us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 26738118512us timer=ffff830c18437290
cb=rtc_alarm_cb(ffff830c184371f0)
(XEN) ex= 30039us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 7344us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 7405us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 30053us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 7344us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 7405us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 2272179us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 30061us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 179us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 1074us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 501037117us timer=ffff830c184375d0
cb=pmt_timer_callback(ffff830c184375a8)
(XEN) ex= 7408us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 7344us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: fffffffffffffff7 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 000037d3e7b7ffcf r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffdb478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000000000000001 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000037d3e7b7f98e r14: 000000000000000f
(XEN) r15: fffffa8000bcf3d0 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fefb8684ee
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a0ce80 rcx: fffff80002a0cec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80000ba6e70
(XEN) rbp: 000000000000000f rsp: fffff80000ba6be0 r8: fffff80002a0cfa0
(XEN) r9: 000034da77fd3a29 r10: fffff80002a0cfa0 r11: fffff80000ba6d30
(XEN) r12: fffff80000ba6d30 r13: 000037d3e7c82ee7 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: fffff88001333898
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d0801dc4fe>] vmx_vmexit_handler+0x311/0x195e
(XEN) RFLAGS: 0000000000000297 CONTEXT: hypervisor
(XEN) rax: 0000000000187000 rbx: ffff830c3dca7f18 rcx: 000000000000ffff
(XEN) rdx: 00003a4400000000 rsi: 0000000000000000 rdi: ffff830c3dca7f18
(XEN) rbp: ffff830c3dca7f08 rsp: ffff830c3dca7d98 r8: fffff880031ea140
(XEN) r9: 0000000000000000 r10: fffff880031ea190 r11: 00000003429653d0
(XEN) r12: ffff8300a99a5000 r13: 0000000000000028 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1b5d6000 cr2: 000007fef93521dc
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7d98:
(XEN) ffff8300a99a5000 ffff8300a99a5000 ffff8300a99a5000 ffff830c3dcaa088
(XEN) ffff830c3dca7e08 0000000000000000 ffff830c3dca7de8 ffff830c3dca0000
(XEN) ffff830c3dcaa0a0 ffff8300a99a5000 000014e14d5273bf ffff8300a99a5000
(XEN) ffff830c3dcaa088 0000000001c9c380 0000000000000292 ffff830c3dca7e28
(XEN) ffff82d08012a80f ffff8300a99a5000 ffff830c3dca7e98 ffff82d0801caea5
(XEN) ffff830c3dca7ec8 ffff8300a99a5508 ffff82d080321280 ffff830c3dcaa0a0
(XEN) ffff830c3dca7e78 ffff830c3dca7e88 ffff82d0801b5277 ffff8300a99a5000
(XEN) 000014e14d5273bf ffff8300a99a5000 ffff830c3dca7f08 ffff82d0801e0f69
(XEN) ffff830c3dca7f18 ffff8300a99a5000 ffff830c3dca7f08 ffff82d0801ddba6
(XEN) ffff830c3dca0000 0000000000000058 ffff830c3dca7ef8 ffff82d08012a1b3
(XEN) ffff8300a99a5000 ffff8300a99a5000 fffff880031ea518 0000000000001000
(XEN) 0000000000000000 0000000000000058 000000000000432f ffff82d0801e3c81
(XEN) 0000000000000058 0000000000000000 0000000000001000 fffff880031ea518
(XEN) 000000000000432f 00003a44732bdba0 00000003429653d0 fffff880031ea190
(XEN) 0000000000000000 fffff880031ea140 00003a44732bda2e 000000000000ffff
(XEN) 00003a4400000000 0000000000000000 0000000000000097 0000beef0000beef
(XEN) fffff80002e112bf 000000bf0000beef 0000000000000002 fffff880031ea0e0
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 000000000000beef 0000000000000003 ffff8300a99a5000 0000003bbd988e00
(XEN) 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801dc4fe>] vmx_vmexit_handler+0x311/0x195e
(XEN) [<ffff82d0801e3c81>] vmx_asm_vmexit_handler+0x41/0xc0
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff80002e112bf>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 00003a44732bda2e rbx: 00003a44732bdba0 rcx: 000000000000ffff
(XEN) rdx: 00003a4400000000 rsi: 0000000000000000 rdi: 0000000000000097
(XEN) rbp: 000000000000432f rsp: fffff880031ea0e0 r8: fffff880031ea140
(XEN) r9: 0000000000000000 r10: fffff880031ea190 r11: 00000003429653d0
(XEN) r12: fffff880031ea518 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: ffffffffffffffdf r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 000037d3e7b7f0a9 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a99a7000 rcx: 0000000000000005
(XEN) rdx: ffff830c3dc80000 rsi: 0000000000000004 rdi: ffff830c3dc8e088
(XEN) rbp: ffff830c3dc87ec8 rsp: ffff830c3dc87e40 r8: ffff830c3dc8e0a0
(XEN) r9: 0000000000000000 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: 000014e162f97027 r13: ffff8300a99a7000 r14: ffff830c3dc8e088
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c182ef000 cr2: 000007fffffa9478
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dc87e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dc8e0a0 0000000500c87e78
(XEN) ffff830c3dc8e080 ffff82d0801b5277 ffff8300a99a7000 fffff88002ec0d70
(XEN) ffff8300a99a7000 0000000001c9c380 ffff82d0801e0f00 ffff830c3dc87f08
(XEN) ffff82d0802f8280 ffff82d0802f8000 ffffffffffffffff ffff830c3dc80000
(XEN) fffff6fc4003b208 ffff830c3dc87ef8 ffff82d08012a1b3 ffff8300a99a7000
(XEN) fffff88002ec0d70 000037d3e7b7f896 000000000000000f ffff830c3dc87f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a fffff6fc4003b208
(XEN) 000000000000000f 000037d3e7b7f896 fffff88002ec0d70 000000000000000f
(XEN) fffff88002eb6180 fffff88002ec0d70 fffff88002eb62a0 0000000000000001
(XEN) fffff88002eb62a0 0000000000000002 fffff88002eb61c0 0000000000000400
(XEN) 0000000000000000 fffff88002ec0eb0 0000beef0000beef fffff8000294a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002ec0c20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000005 ffff8300a99a7000 0000003bbd96ce00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 0000000000000001 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 000037d3e7b7f896 r14: 000000000000000f
(XEN) r15: fffff6fc4003b208 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffa9478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 3021us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 7725us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 7725us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) ex= 134454796us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 652872us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 5314524us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) CPU01:
(XEN) ex= 7725us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 7789us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 26729098894us timer=ffff830c18437290
cb=rtc_alarm_cb(ffff830c184371f0)
(XEN) ex= 30023us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 7725us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 7791us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 24560us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) CPU03:
(XEN) ex= 249us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 1041us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 216560us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 8273us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 7725us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) CPU04:
(XEN) ex= 7725us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 7790us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 1016560us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 1252560us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 30054us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 7725us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 7789us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 40560us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 492017498us timer=ffff830c184375d0
cb=pmt_timer_callback(ffff830c184375a8)
(XEN) ex= 30068us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: fffffffffffffff7 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 000037d3e7b7ffcf r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffdb478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000294a214>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a0ce80 rcx: fffff80002a0cec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80000ba6e70
(XEN) rbp: 000000000000000f rsp: fffff80000ba6be0 r8: fffff80002a0cfa0
(XEN) r9: 000034da77fd3a29 r10: fffff80002a0cfa0 r11: fffff80000ba6d30
(XEN) r12: fffff80000ba6d30 r13: 000037d3e7c82ee7 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: fffff88001333898
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a99a2000 rcx: 0000000000000002
(XEN) rdx: ffff830c3dcb0000 rsi: 0000000000000002 rdi: ffff830c3dcb8088
(XEN) rbp: ffff830c3dcb7ec8 rsp: ffff830c3dcb7e40 r8: ffff830c3dcb80a0
(XEN) r9: 000014e3652e1b9a r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: 000014e36521c02b r13: ffff8300a99a2000 r14: ffff830c3dcb8088
(XEN) r15: 00000000000f4240 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1a237000 cr2: 000007fef93521dc
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dcb7e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dcb80a0 0000000200cb7e78
(XEN) ffff830c3dcb8080 ffff82d0801b5277 ffff8300a99a2000 fffff88002e4ad70
(XEN) ffff8300a99a2000 00000000000f4240 ffff82d0801e0f00 ffff830c3dcb7f08
(XEN) ffff82d0802f8100 ffff82d0802f8000 ffffffffffffffff ffff830c3dcb0000
(XEN) 0000000000000001 ffff830c3dcb7ef8 ffff82d08012a1b3 ffff8300a99a2000
(XEN) fffff88002e4ad70 000037d3e7b7f0a9 000000000000000f ffff830c3dcb7f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 000037d3e7b7f0a9 fffff88002e4ad70 000000000000000f
(XEN) fffff88002e40180 fffff88002e4ad70 fffff88002e402a0 ffffffffffffffdf
(XEN) fffff88002e402a0 0000000000000002 fffff88002e401c0 0000000000000400
(XEN) 0000000000000000 fffff88002e4aeb0 0000beef0000beef fffff8000294a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002e4ac20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000002 ffff8300a99a2000 0000003bbd996e00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: ffffffffffffffdf r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 000037d3e7b7f0a9 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 0000000000000001 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 000037d3e7b7f896 r14: 000000000000000f
(XEN) r15: fffff6fc4003b208 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffa9478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff80002e112c3>]
(XEN) RFLAGS: 0000000000000006 CONTEXT: hvm guest
(XEN) rax: 00000000876ba63c rbx: 00003a4a876bd272 rcx: 000000000000ffff
(XEN) rdx: 0000000000003a4a rsi: 0000000000000000 rdi: 000000000000002e
(XEN) rbp: 000000000000943d rsp: fffff880031ea0e0 r8: fffff880031ea140
(XEN) r9: 0000000000000000 r10: fffff880031ea190 r11: 00000003429653d0
(XEN) r12: fffff880031ea518 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000000000000001 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000037d3e7b7f98e r14: 000000000000000f
(XEN) r15: fffffa8000bcf3d0 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fefb8684ee
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 3787us timer=ffff8300bf527060
cb=vcpu_singleshot_timer_fn(ffff8300bf527000)
(XEN) ex= 780451us timer=ffff82d08031d280
cb=time_calibration(0000000000000000)
(XEN) ex= 4953us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 126582023us timer=ffff82d08031d1e0
cb=plt_overflow(0000000000000000)
(XEN) ex= 13441754us timer=ffff82d08031f4e0
cb=mce_work_fn(0000000000000000)
(XEN) ex= 9226us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 14953us timer=ffff82d080321560
cb=do_dbs_timer(ffff82d0803215a0)
(XEN) CPU01:
(XEN) ex= 223us timer=ffff8300bf525060
cb=vcpu_singleshot_timer_fn(ffff8300bf525000)
(XEN) ex= 1028us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 14953us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 5271us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) CPU02:
(XEN) ex= 1037us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 6999us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 5324us timer=ffff830c3dc797c8
cb=csched_tick(0000000000000002)
(XEN) ex= 14953us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) CPU03:
(XEN) ex= 5392us timer=ffff830c3dcac088
cb=csched_tick(0000000000000003)
(XEN) ex= 14953us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 26721226121us timer=ffff830c18437290
cb=rtc_alarm_cb(ffff830c184371f0)
(XEN) ex= 340334us timer=ffff8300bf526060
cb=vcpu_singleshot_timer_fn(ffff8300bf526000)
(XEN) ex= 30050us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 5381us timer=ffff830c3dcac8d8
cb=csched_tick(0000000000000004)
(XEN) ex= 14953us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 1379787us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 51787us timer=ffff8300bf524060
cb=vcpu_singleshot_timer_fn(ffff8300bf524000)
(XEN) ex= 30063us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 5323us timer=ffff830c3dc8d178
cb=csched_tick(0000000000000005)
(XEN) ex= 14953us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 484144725us timer=ffff830c184375d0
cb=pmt_timer_callback(ffff830c184375a8)
(XEN) ex= 30075us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v0): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a0ce80 rcx: fffff80002a0cec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80000ba6e70
(XEN) rbp: 000000000000000f rsp: fffff80000ba6be0 r8: fffff80002a0cfa0
(XEN) r9: 000034da77fd3a29 r10: fffff80002a0cfa0 r11: fffff80000ba6d30
(XEN) r12: fffff80000ba6d30 r13: 000037d3e7c82ee7 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: fffff88001333898
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v4): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff80002e112c1>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 00003a4fa26b020b rbx: 00003a4fa26b0c79 rcx: 000000000000ffff
(XEN) rdx: 00003a4f00000000 rsi: 0000000000000000 rdi: 000000000000005d
(XEN) rbp: 000000000000054a rsp: fffff880031ea0e0 r8: fffff880031ea140
(XEN) r9: 0000000000000000 r10: fffff880031ea190 r11: 00000003429653d0
(XEN) r12: fffff880031ea518 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v3): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: 0000000000000001 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 000037d3e7b7f98e r14: 000000000000000f
(XEN) r15: fffffa8000bcf3d0 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fefb8684ee
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a99a2000 rcx: 0000000000000003
(XEN) rdx: ffff830c3dca0000 rsi: 0000000000000004 rdi: ffff830c3dcaa088
(XEN) rbp: ffff830c3dca7ec8 rsp: ffff830c3dca7e40 r8: ffff830c3dcaa0a0
(XEN) r9: 0000000000000000 r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: 000014e55456c519 r13: ffff8300a99a2000 r14: ffff830c3dcaa088
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1a237000 cr2: 000007fef93521dc
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7e40:
(XEN) ffff82d080126ec5 ffff82d080321280 ffff830c3dcaa0a0 0000000300ca7e78
(XEN) ffff830c3dcaa080 ffff82d0801b5277 ffff8300a99a2000 fffff88002e4ad70
(XEN) ffff8300a99a2000 0000000001c9c380 ffff82d0801e0f00 ffff830c3dca7f08
(XEN) ffff82d0802f8180 ffff82d0802f8000 ffffffffffffffff ffff830c3dca0000
(XEN) 0000000000000001 ffff830c3dca7ef8 ffff82d08012a1b3 ffff8300a99a2000
(XEN) fffff88002e4ad70 000037d3e7b7f0a9 000000000000000f ffff830c3dca7f08
(XEN) ffff82d08012a20b 000000000000000f ffff82d0801e3d2a 0000000000000001
(XEN) 000000000000000f 000037d3e7b7f0a9 fffff88002e4ad70 000000000000000f
(XEN) fffff88002e40180 fffff88002e4ad70 fffff88002e402a0 ffffffffffffffdf
(XEN) fffff88002e402a0 0000000000000002 fffff88002e401c0 0000000000000400
(XEN) 0000000000000000 fffff88002e4aeb0 0000beef0000beef fffff8000294a20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002e4ac20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000003 ffff8300a99a2000 0000003bbd988e00 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012a9a1>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012a1b3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012a20b>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e3d2a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v1): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002e40180 rcx: fffff88002e401c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002e4aeb0
(XEN) rbp: 000000000000000f rsp: fffff88002e4ac20 r8: fffff88002e402a0
(XEN) r9: ffffffffffffffdf r10: fffff88002e402a0 r11: fffff88002e4ad70
(XEN) r12: fffff88002e4ad70 r13: 000037d3e7b7f0a9 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fef93521dc
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v2): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000294a20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: 0000000000000001 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 000037d3e7b7f896 r14: 000000000000000f
(XEN) r15: fffff6fc4003b208 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffa9478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v5): ***
(XEN) ----[ Xen-4.5-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000294a20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: fffffffffffffff7 r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 000037d3e7b7ffcf r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffdb478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
Going to continue replying to your next email here, as I'd like to
minimize my impact on the list.
On 11/21/2014 1:31, Jan Beulich wrote:
>>>> On 20.11.14 at 21:07, <sflist@ihonk.com> wrote:
>> Running with mwait-idle=0 solves (hides?) the problem. Next step is to
>> fiddle with the C states?
> So this also prompted me to go over the list of errata. Just to confirm
> - your CPU is family 6 model 44? What stepping? And what nominal
> frequency?
CPU information for one of the cores, 2.8 GHz is nominal, stepping is 2.
Not sure how to translate that stepping number into Intel's format:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5660 @ 2.80GHz
stepping : 2
microcode : 0x1a
cpu MHz : 2800.184
cache size : 12288 KB
physical id : 0
siblings : 6
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good
nopl nonstop_tsc pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2
popcnt aes hypervisor lahf_lm ida arat epb dtherm
bogomips : 5600.36
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
> There are a couple potentially relevant errata (BC36, BC38, BC54,
> BC77, BC110).
>
> To exclude BC36, a boot log with "apic-verbosity=debug" and debug
> key 'i' output would be necessary.
Done, see the very end of the email.
> BC38 should not affect us since we don't enter C states from ISRs.
>
> BC54 is probably irrelevant since we meanwhile know that your
> system doesn't really hang hard.
>
> For BC77 it would be worth trying to disable turbo mode instead of
> disabling the mwait-idle driver ("xenpm disable-turbo-mode" right
> after boot).
I looked up BC77 but as a result found this document[1], which seems to
relate to the i7. Would this[2] not be the relevant document?
[1]
http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/core-i7-900-ee-and-desktop-processor-series-32nm-spec-update.pdf
[2]
http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf
In any case, I ran a test with turbo mode disabled via "xenpm
disable-turbo-mode" and the host hung.
> And BC110 would be relevant only if without the mwait-idle driver
> there would be no use of C3. Plus anyway this would more likely end
> up in a hard hang too.
>
> And then, considering that my system with a family 6 model 44 CPU
> has never shown anything similar (albeit that doesn't mean all that
> much since our workloads are likely very different), you're not
> over-clocking?
No, no overclocking at all.
> And did you disable hyper-threading on purpose (if
> so could you check whether enabling it makes a difference)?
Yes, I disabled HT intentionally. I've tested with it enabled and with 6
vCPUs assigned to the domU and the host hung. Took quite a bit longer
than usual to hang, though, 11 hours.
As promised, below is the apic-verbosity=debug log, with 'i'. Thanks!
Steve
Xen 4.5-unstable
(XEN) Xen version 4.5-unstable (steve@) (gcc (Debian 4.9.1-19) 4.9.1)
debug=y Thu Nov 20 11:00:27 PST 2014
(XEN) Latest ChangeSet: Thu Sep 18 14:43:49 2014 +0200 git:9a727a8-dirty
(XEN) Bootloader: GRUB 2.02~beta2-15
(XEN) Command line: placeholder cpufreq=xen:ondemand cpuidle
clocksource=hpet loglvl=all guest_loglvl=all iommu=no-intremap,debug
com1=115200,8n1 console=com1,vga apic-verbosity=debug
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009d000 (usable)
(XEN) 000000000009d000 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf7a0000 (usable)
(XEN) 00000000bf7a0000 - 00000000bf7ca000 (ACPI data)
(XEN) 00000000bf7ca000 - 00000000bf7cc000 (ACPI NVS)
(XEN) 00000000bf7cc000 - 00000000c0000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000c40000000 (usable)
(XEN) ACPI: RSDP 000F6A40, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT BF7BF38F, 0084 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: FACP BF7C98D1, 00F4 (r3 LENOVO TC-61 60400D0 PTL 2)
(XEN) ACPI: DSDT BF7BF413, A43A (r1 LENOVO TC-61 60400D0 MSFT 100000E)
(XEN) ACPI: FACS BF7CBFC0, 0040
(XEN) ACPI: SSDT BF7AF33B, 2694 (r1 INTEL PPM RCM 80000001 INTL 20061109)
(XEN) ACPI: SLIT BF7C99E9, 0030 (r1 Intel TYLERBRG 60400D0 LOHR 5A)
(XEN) ACPI: TCPA BF7C9A19, 0032 (r1 LENOVO TC-61 60400D0 PTL 0)
(XEN) ACPI: SLIC BF7C9A4B, 0176 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: SRAT BF7C9BC1, 00E0 (r1 Intel Tylerbrg 60400D0 PHX. 1)
(XEN) ACPI: DMAR BF7C9CA1, 01C0 (r1 Intel OEMDMAR 60400D0 LOHR 1)
(XEN) ACPI: APIC BF7C9E61, 00A0 (r1 PTLTD APIC 60400D0
LTP 0)
(XEN) ACPI: MCFG BF7C9F01, 003C (r1 PTLTD MCFG 60400D0 LTP 0)
(XEN) ACPI: HPET BF7C9F3D, 0038 (r1 PTLTD HPETTBL 60400D0 LTP 1)
(XEN) ACPI: BOOT BF7C9F75, 0028 (r1 PTLTD $SBFTBL$ 60400D0 LTP 1)
(XEN) ACPI: ASF! BF7C9F9D, 0063 (r32 LENOVO TC-61 60400D0 PTL 1)
(XEN) System RAM: 49143MB (50322676kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-c0000000
(XEN) SRAT: Node 0 PXM 0 100000000-c40000000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f6a70
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1004,1:0], pm1x_evt[1:1000,1:0]
(XEN) ACPI: wakeup_vec[bf7cbfcc], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
(XEN) Processor #2 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
(XEN) Processor #4 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x10] enabled)
(XEN) Processor #16 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
(XEN) Processor #18 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
(XEN) Processor #20 6:12 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) [VT-D]dmar.c:788: Host address width 40
(XEN) [VT-D]dmar.c:802: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:472: dmaru->address = fe710000
(XEN) [VT-D]iommu.c:1145: drhd->address = fe710000 iommu->reg =
ffff82c000201000
(XEN) [VT-D]iommu.c:1147: cap = c90780106f0462 ecap = f0207f
(XEN) [VT-D]dmar.c:397: IOAPIC: 0000:f0:1f.7
(XEN) [VT-D]dmar.c:486: flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.3
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7cd000 end_address
bf7dbfff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e4000 end_address
bf7e4fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e5000 end_address
bf7e5fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7ea000 end_address
bf7eafff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e6000 end_address
bf7e6fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e7000 end_address
bf7e7fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e8000 end_address
bf7e8fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.3
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e9000 end_address
bf7e9fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7eb000 end_address
bf7ebfff
(XEN) [VT-D]dmar.c:812: found ACPI_DMAR_ATSR:
(XEN) [VT-D]dmar.c:705: atsru->all_ports: 0
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:01.0 start=0 sec=1 sub=1
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:03.0 start=0 sec=2 sub=2
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:07.0 start=0 sec=3 sub=3
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1144 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2800.166 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) alt table ffff82d0802d1cd0 -> ffff82d0802d2cf0
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 09
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-09
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping disabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2c
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 6 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x7c8000
(XEN) elf_parse_binary: phdr: paddr=0x1800000 memsz=0xed000
(XEN) elf_parse_binary: phdr: paddr=0x18ed000 memsz=0x14f40
(XEN) elf_parse_binary: phdr: paddr=0x1902000 memsz=0x614000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1f16000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff819021f0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0x0
(XEN) virt_offset = 0xffffffff80000000
(XEN) virt_kstart = 0xffffffff81000000
(XEN) virt_kend = 0xffffffff81f16000
(XEN) virt_entry = 0xffffffff819021f0
(XEN) p2m_base = 0xffffffffffffffff
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f16000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000c00000000->0000000c10000000 (12316675 pages
to be allocated)
(XEN) Init. ramdisk: 0000000c3f0da000->0000000c3ffff86e
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff81f16000
(XEN) Init. ramdisk: ffffffff81f16000->ffffffff82e3b86e
(XEN) Phys-Mach map: ffffffff82e3c000->ffffffff88cbb948
(XEN) Start info: ffffffff88cbc000->ffffffff88cbc4b4
(XEN) Page tables: ffffffff88cbd000->ffffffff88d08000
(XEN) Boot stack: ffffffff88d08000->ffffffff88d09000
(XEN) TOTAL: ffffffff80000000->ffffffff89000000
(XEN) ENTRY ADDRESS: ffffffff819021f0
(XEN) Dom0 has maximum 6 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff817c8000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81800000 -> 0xffffffff818ed000
(XEN) elf_load_binary: phdr 2 at 0xffffffff818ed000 -> 0xffffffff81901f40
(XEN) elf_load_binary: phdr 3 at 0xffffffff81902000 -> 0xffffffff81a1f000
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:00:00.0 map
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:14.0
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:14.1
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:14.2
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.1
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.2
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.3
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.4
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.5
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.6
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:16.7
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1a.0
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1a.1
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1a.7
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:00:1b.0
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1d.0
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1d.1
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1d.2
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1d.3
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1d.7
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:00:1f.3
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:01:00.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:05:00.0
(XEN) [VT-D]iommu.c:1440: d0:PCIe: map 0000:09:00.0
(XEN) [VT-D]iommu.c:1452: d0:PCI: map 0000:0a:0e.0
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:00.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:00.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.2 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.3 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.4 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:02.5 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:03.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:03.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:03.2 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:03.4 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:04.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:04.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:04.2 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:04.3 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:05.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:05.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:05.2 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:05.3 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:06.0 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:06.1 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:06.2 map
(XEN) [VT-D]iommu.c:1426: d0:Hostbridge: skip 0000:3f:06.3 map
(XEN) [VT-D]iommu.c:738: iommu_enable_translation: iommu->reg =
ffff82c000201000
(XEN) Scrubbing Free RAM on 1 nodes using 6 CPUs
(XEN)
..................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 288kB init memory.
mapping kernel into physical memory
about to get started...
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.16-3-amd64
(debian-kernel@lists.debian.org) (gcc version 4.8.3 (Debian 4.8.3-12) )
#1 SMP Debian 3.16.5-1 (2014-10-10)
[ 0.000000] Command line: placeholder
root=/dev/mapper/vg_g2-lv_g2_root ro console=hvc0 earlyprintk=serial
xencons=hvc debug ignore_loglevel log_buf_len=10M print_fatal_signals=1
LOGLEVEL=8 sched_debug
[ 0.000000] Freeing 9d-100 pfn range: 99 pages freed
[ 0.000000] Freeing bf7a0-100000 pfn range: 264288 pages freed
[ 0.000000] Released 264387 pages of unused memory
[ 0.000000] Set 264387 page(s) to 1-1 mapping
[ 0.000000] Populating bcff29-c107ec pfn range: 264387 pages added
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
[ 0.000000] Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x00000000bf79ffff] usable
[ 0.000000] Xen: [mem 0x00000000bf7a0000-0x00000000bf7c9fff] ACPI data
[ 0.000000] Xen: [mem 0x00000000bf7ca000-0x00000000bf7cbfff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000bf7cc000-0x00000000bfffffff] reserved
[ 0.000000] Xen: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000c3fffffff] usable
[ 0.000000] bootconsole [earlyser0] enabled
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.6 present.
[ 0.000000] DMI: LENOVO 4158WRG/LENOVO, BIOS 61KT50AUS 01/14/2014
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] AGP: No AGP bridge found
[ 0.000000] e820: last_pfn = 0xc40000 max_arch_pfn = 0x400000000
[ 0.000000] e820: last_pfn = 0xbf7a0 max_arch_pfn = 0x400000000
[ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[ 0.000000] [mem 0x00000000-0x000fffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0xc10400000-0xc105fffff]
[ 0.000000] [mem 0xc10400000-0xc105fffff] page 4k
[ 0.000000] BRK [0x01b61000, 0x01b61fff] PGTABLE
[ 0.000000] BRK [0x01b62000, 0x01b62fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0xc10000000-0xc103fffff]
[ 0.000000] [mem 0xc10000000-0xc103fffff] page 4k
[ 0.000000] BRK [0x01b63000, 0x01b63fff] PGTABLE
[ 0.000000] BRK [0x01b64000, 0x01b64fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0xc00000000-0xc0fffffff]
[ 0.000000] [mem 0xc00000000-0xc0fffffff] page 4k
[ 0.000000] BRK [0x01b65000, 0x01b65fff] PGTABLE
[ 0.000000] BRK [0x01b66000, 0x01b66fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x00100000-0xbf79ffff]
[ 0.000000] [mem 0x00100000-0xbf79ffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0x100000000-0xbffffffff]
[ 0.000000] [mem 0x100000000-0xbffffffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0xc10600000-0xc3fffffff]
[ 0.000000] [mem 0xc10600000-0xc3fffffff] page 4k
[ 0.000000] log_buf_len: 16777216
[ 0.000000] early log buf free: 127372(97%)
[ 0.000000] RAMDISK: [mem 0x01f16000-0x02e3bfff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F6A40 000024 (v02 PTLTD )
[ 0.000000] ACPI: XSDT 0x00000000BF7BF38F 000084 (v01 LENOVO TC-61
060400D0 LTP 00000000)
[ 0.000000] ACPI: FACP 0x00000000BF7C98D1 0000F4 (v03 LENOVO TC-61
060400D0 PTL 00000002)
[ 0.000000] ACPI: DSDT 0x00000000BF7BF413 00A43A (v01 LENOVO TC-61
060400D0 MSFT 0100000E)
[ 0.000000] ACPI: FACS 0x00000000BF7CBFC0 000040
[ 0.000000] ACPI: SSDT 0x00000000BF7AF33B 002694 (v01 INTEL PPM RCM
80000001 INTL 20061109)
[ 0.000000] ACPI: SLIT 0x00000000BF7C99E9 000030 (v01 Intel TYLERBRG
060400D0 LOHR 0000005A)
[ 0.000000] ACPI: TCPA 0x00000000BF7C9A19 000032 (v01 LENOVO TC-61
060400D0 PTL 00000000)
[ 0.000000] ACPI: SLIC 0x00000000BF7C9A4B 000176 (v01 LENOVO TC-61
060400D0 LTP 00000000)
[ 0.000000] ACPI: SRAT 0x00000000BF7C9BC1 0000E0 (v01 Intel Tylerbrg
060400D0 PHX. 00000001)
[ 0.000000] ACPI: XMAR 0x00000000BF7C9CA1 0001C0 (v01 Intel OEMDMAR
060400D0 LOHR 00000001)
[ 0.000000] ACPI: APIC 0x00000000BF7C9E61 0000A0 (v01 PTLTD ? APIC
060400D0 LTP 00000000)
[ 0.000000] ACPI: MCFG 0x00000000BF7C9F01 00003C (v01 PTLTD MCFG
060400D0 LTP 00000000)
[ 0.000000] ACPI: HPET 0x00000000BF7C9F3D 000038 (v01 PTLTD HPETTBL
060400D0 LTP 00000001)
[ 0.000000] ACPI: BOOT 0x00000000BF7C9F75 000028 (v01 PTLTD $SBFTBL$
060400D0 LTP 00000001)
[ 0.000000] ACPI: ASF! 0x00000000BF7C9F9D 000063 (v32 LENOVO TC-61
060400D0 PTL 00000001)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000c3fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x00000000-0xc3fffffff]
[ 0.000000] NODE_DATA [mem 0xc107e7000-0xc107ebfff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal [mem 0x100000000-0xc3fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009cfff]
[ 0.000000] node 0: [mem 0x00100000-0xbf79ffff]
[ 0.000000] node 0: [mem 0x100000000-0xc3fffffff]
[ 0.000000] On node 0 totalpages: 12580668
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3996 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 10667 pages used for memmap
[ 0.000000] DMA32 zone: 780192 pages, LIFO batch:31
[ 0.000000] Normal zone: 161280 pages used for memmap
[ 0.000000] Normal zone: 11796480 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x1008
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x10] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI
0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ2 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
[ 0.000000] smpboot: Allowing 6 CPUs, 0 hotplug CPUs
[ 0.000000] nr_irqs_gsi: 40
[ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x000fffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7a0000-0xbf7c9fff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7ca000-0xbf7cbfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7cc000-0xbfffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xf7ffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfebfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec0ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfec10000-0xfedfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfeefffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfef00000-0xfeffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[ 0.000000] e820: [mem 0xc0000000-0xf7ffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.5-unstable (preserve-AD)
[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512
nr_cpu_ids:6 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff880c03400000 s85824
r8192 d20672 u262144
[ 0.000000] pcpu-alloc: s85824 r8192 d20672 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 - -
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
Total pages: 12408644
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: placeholder
root=/dev/mapper/vg_g2-lv_g2_root ro console=hvc0 earlyprintk=serial
xencons=hvc debug ignore_loglevel log_buf_len=10M print_fatal_signals=1
LOGLEVEL=8 sched_debug
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] software IO TLB [mem 0xbd4e00000-0xbd8e00000] (64MB)
mapped at [ffff880bd4e00000-ffff880bd8dfffff]
[ 0.000000] Memory: 48548972K/50322672K available (5189K kernel code,
942K rwdata, 1824K rodata, 1200K init, 840K bss, 1773700K reserved)
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=6.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
[ 0.000000] NR_IRQS:33024 nr_irqs:728 16
[ 0.000000] xen:events: Using FIFO-based ABI
[ 0.000000] xen: sci override: global_irq=9 trigger=0 polarity=0
[ 0.000000] xen: registering gsi 9 triggering 0 polarity 0
[ 0.000000] xen: --> pirq=9 -> irq=9 (gsi=9)
[ 0.000000] xen: acpi sci 9
[ 0.000000] xen: --> pirq=1 -> irq=1 (gsi=1)
[ 0.000000] xen: --> pirq=2 -> irq=2 (gsi=2)
[ 0.000000] xen: --> pirq=3 -> irq=3 (gsi=3)
[ 0.000000] xen: --> pirq=4 -> irq=4 (gsi=4)
[ 0.000000] xen: --> pirq=5 -> irq=5 (gsi=5)
[ 0.000000] xen: --> pirq=6 -> irq=6 (gsi=6)
[ 0.000000] xen: --> pirq=7 -> irq=7 (gsi=7)
[ 0.000000] xen: --> pirq=8 -> irq=8 (gsi=8)
[ 0.000000] xen: --> pirq=10 -> irq=10 (gsi=10)
[ 0.000000] xen: --> pirq=11 -> irq=11 (gsi=11)
[ 0.000000] xen: --> pirq=12 -> irq=12 (gsi=12)
[ 0.000000] xen: --> pirq=13 -> irq=13 (gsi=13)
[ 0.000000] xen: --> pirq=14 -> irq=14 (gsi=14)
[ 0.000000] xen: --> pirq=15 -> irq=15 (gsi=15)
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [hvc0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] bootconsole [earlyser0] disabled
[ 0.000000] bootconsole [earlyser0] disabled
[ 0.000000] bootconsole [xenboot0] disabled
[ 0.000000] bootconsole [xenboot0] disabled
[ 0.000000] Xen: using vcpuop timer interface
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] tsc: Detected 2800.166 MHz processor
[ 10.913942] Calibrating delay loop (skipped), value calculated using
timer frequency.. 5600.33 BogoMIPS (lpj=11200664)
[ 10.913948] pid_max: default: 32768 minimum: 301
[ 10.913957] ACPI: Core revision 20140424
[ 10.918311] ACPI: All ACPI Tables successfully acquired
[ 10.925345] Security Framework initialized
[ 10.925354] AppArmor: AppArmor disabled by boot time parameter
[ 10.925357] Yama: disabled by default; enable with sysctl kernel.yama.*
[ 10.933807] Dentry cache hash table entries: 8388608 (order: 14,
67108864 bytes)
[ 11.051238] Inode-cache hash table entries: 4194304 (order: 13,
33554432 bytes)
[ 11.055771] Mount-cache hash table entries: 131072 (order: 8, 1048576
bytes)
[ 11.055917] Mountpoint-cache hash table entries: 131072 (order: 8,
1048576 bytes)
[ 11.056359] Initializing cgroup subsys memory
[ 11.056365] Initializing cgroup subsys devices
[ 11.056372] Initializing cgroup subsys freezer
[ 11.056376] Initializing cgroup subsys net_cls
[ 11.056380] Initializing cgroup subsys blkio
[ 11.056385] Initializing cgroup subsys perf_event
[ 11.056388] Initializing cgroup subsys net_prio
[ 11.056437] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 11.056437] ENERGY_PERF_BIAS: View and update with
x86_energy_perf_policy(8)
[ 11.056443] CPU: Physical Processor ID: 0
[ 11.056445] CPU: Processor Core ID: 0
[ 11.056448] mce: CPU supports 2 MCE banks
[ 11.056462] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[ 11.056462] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0
[ 11.056462] tlb_flushall_shift: 6
[ 11.056527] Freeing SMP alternatives memory: 20K (ffffffff81a19000 -
ffffffff81a1e000)
[ 11.057351] ftrace: allocating 21546 entries in 85 pages
[ 11.064078] Performance Events: unsupported p6 CPU model 44 no PMU
driver, software events only.
[ 11.065170] NMI watchdog: disabled (cpu0): hardware events not enabled
[ 11.065246] installing Xen timer for CPU 1
[ 11.065590] installing Xen timer for CPU 2
[ 11.065965] installing Xen timer for CPU 3
[ 11.066389] installing Xen timer for CPU 4
[ 11.066751] installing Xen timer for CPU 5
[ 11.066998] x86: Booted up 1 node, 6 CPUs
[ 11.067017] CPU0 attaching sched-domain:
[ 11.067020] domain 0: span 0-5 level MC
[ 11.067022] groups: 0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023)
2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023)
5 (cpu_capacity = 1023)
[ 11.067033] CPU1 attaching sched-domain:
[ 11.067035] domain 0: span 0-5 level MC
[ 11.067037] groups: 1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023)
3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023)
0 (cpu_capacity = 1023)
[ 11.067047] CPU2 attaching sched-domain:
[ 11.067049] domain 0: span 0-5 level MC
[ 11.067051] groups: 2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023)
4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023)
1 (cpu_capacity = 1023)
[ 11.067061] CPU3 attaching sched-domain:
[ 11.067063] domain 0: span 0-5 level MC
[ 11.067065] groups: 3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023)
5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023)
2 (cpu_capacity = 1023)
[ 11.067075] CPU4 attaching sched-domain:
[ 11.067077] domain 0: span 0-5 level MC
[ 11.067079] groups: 4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023)
0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023)
3 (cpu_capacity = 1023)
[ 11.067089] CPU5 attaching sched-domain:
[ 11.067091] domain 0: span 0-5 level MC
[ 11.067093] groups: 5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023)
1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023)
4 (cpu_capacity = 1023)
[ 11.067472] devtmpfs: initialized
[ 11.073663] PM: Registering ACPI NVS region [mem
0xbf7ca000-0xbf7cbfff] (8192 bytes)
[ 11.074540] pinctrl core: initialized pinctrl subsystem
[ 11.074698] NET: Registered protocol family 16
[ 11.074711] xen:grant_table: Grant tables using version 1 layout
[ 11.074725] Grant table initialized
[ 11.075185] ACPI FADT declares the system doesn't support PCIe ASPM,
so disable it
[ 11.075189] ACPI: bus type PCI registered
[ 11.075192] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 11.075445] PCI: MMCONFIG for domain 0000 [bus 00-09] at [mem
0xf8000000-0xf89fffff] (base 0xf8000000)
[ 11.075450] PCI: MMCONFIG at [mem 0xf8000000-0xf89fffff] reserved in E820
[ 11.076571] PCI: Using configuration type 1 for base access
[ 11.089430] ACPI: Added _OSI(Module Device)
[ 11.089434] ACPI: Added _OSI(Processor Device)
[ 11.089436] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 11.089438] ACPI: Added _OSI(Processor Aggregator Device)
[ 11.092694] ACPI: Interpreter enabled
[ 11.092700] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S1_] (20140424/hwxface-580)
[ 11.092706] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S2_] (20140424/hwxface-580)
[ 11.092717] ACPI: (supports S0 S3 S4 S5)
[ 11.092719] ACPI: Using IOAPIC for interrupt routing
[ 11.092741] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[ 11.108183] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 11.108191] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI]
[ 11.108401] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug PME
AER PCIeCapability]
[ 11.109031] acpi PNP0A03:00: [Firmware Info]: MMCONFIG for domain
0000 [bus 00-09] only partially covers this bridge
[ 11.109200] PCI host bridge to bus 0000:00
[ 11.109204] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 11.109207] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
[ 11.109210] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff]
[ 11.109212] pci_bus 0000:00: root bus resource [mem
0x000a0000-0x000bffff]
[ 11.109215] pci_bus 0000:00: root bus resource [mem
0x000d0000-0x000d3fff]
[ 11.109218] pci_bus 0000:00: root bus resource [mem
0x000d4000-0x000d7fff]
[ 11.109221] pci_bus 0000:00: root bus resource [mem
0x000d8000-0x000dbfff]
[ 11.109224] pci_bus 0000:00: root bus resource [mem
0x000dc000-0x000dffff]
[ 11.109227] pci_bus 0000:00: root bus resource [mem
0xc0000000-0xfdffffff]
[ 11.109245] pci 0000:00:00.0: [8086:3406] type 00 class 0x060000
[ 11.109341] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) PCI add device 0000:00:00.0
[ 11.109451] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[ 11.109556] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[ 11.109603] pci 0000:00:01.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) PCI add device 0000:00:01.0
[ 11.109668] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[ 11.109781] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[ 11.109830] pci 0000:00:03.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) PCI add device 0000:00:03.0
[ 11.109898] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[ 11.109995] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[ 11.110045] pci 0000:00:07.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) PCI add device 0000:00:07.0
[ 11.110117] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) PCI add device 0000:00:14.0
[ 11.110272] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
(XEN) PCI add device 0000:00:14.1
[ 11.110443] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
(XEN) PCI add device 0000:00:14.2
[ 11.110605] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[ 11.110626] pci 0000:00:16.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.0
[ 11.110786] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[ 11.110806] pci 0000:00:16.1: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.1
[ 11.110964] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[ 11.110985] pci 0000:00:16.2: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.2
[ 11.111143] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[ 11.111164] pci 0000:00:16.3: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.3
[ 11.111324] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[ 11.111344] pci 0000:00:16.4: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.4
[ 11.111503] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[ 11.111523] pci 0000:00:16.5: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.5
[ 11.111682] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[ 11.111703] pci 0000:00:16.6: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.6
[ 11.111881] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[ 11.111902] pci 0000:00:16.7: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.7
[ 11.112063] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[ 11.112131] pci 0000:00:1a.0: reg 0x20: [io 0x1800-0x181f]
[ 11.112236] pci 0000:00:1a.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.0
[ 11.112294] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[ 11.112387] pci 0000:00:1a.1: reg 0x20: [io 0x1820-0x183f]
[ 11.112515] pci 0000:00:1a.1: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.1
[ 11.112585] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[ 11.112617] pci 0000:00:1a.7: reg 0x10: [mem 0xec604000-0xec6043ff]
[ 11.112755] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[ 11.112800] pci 0000:00:1a.7: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.7
[ 11.112858] pci 0000:00:1b.0: [8086:3a3e] type 00 class 0x040300
[ 11.112884] pci 0000:00:1b.0: reg 0x10: [mem 0xec600000-0xec603fff 64bit]
[ 11.113000] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
(XEN) PCI add device 0000:00:1b.0
[ 11.113091] pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
[ 11.113225] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[ 11.113273] pci 0000:00:1c.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.0
[ 11.113330] pci 0000:00:1c.4: [8086:3a48] type 01 class 0x060400
[ 11.113449] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[ 11.113497] pci 0000:00:1c.4: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.4
[ 11.113549] pci 0000:00:1c.5: [8086:3a4a] type 01 class 0x060400
[ 11.113668] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[ 11.113717] pci 0000:00:1c.5: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.5
[ 11.113780] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[ 11.113848] pci 0000:00:1d.0: reg 0x20: [io 0x1840-0x185f]
[ 11.113955] pci 0000:00:1d.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.0
[ 11.114008] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[ 11.114076] pci 0000:00:1d.1: reg 0x20: [io 0x1860-0x187f]
[ 11.114182] pci 0000:00:1d.1: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.1
[ 11.114240] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[ 11.114331] pci 0000:00:1d.2: reg 0x20: [io 0x1880-0x189f]
[ 11.114461] pci 0000:00:1d.2: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.2
[ 11.114538] pci 0000:00:1d.3: [8086:3a39] type 00 class 0x0c0300
[ 11.114622] pci 0000:00:1d.3: reg 0x20: [io 0x18a0-0x18bf]
[ 11.114726] pci 0000:00:1d.3: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.3
[ 11.114790] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[ 11.114823] pci 0000:00:1d.7: reg 0x10: [mem 0xec605000-0xec6053ff]
[ 11.114960] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[ 11.115005] pci 0000:00:1d.7: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.7
[ 11.115056] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[ 11.115166] pci 0000:00:1e.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1e.0
[ 11.115218] pci 0000:00:1f.0: [8086:3a18] type 00 class 0x060100
(XEN) PCI add device 0000:00:1f.0
[ 11.115446] pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
[ 11.115477] pci 0000:00:1f.2: reg 0x10: [io 0x18f0-0x18f7]
[ 11.115490] pci 0000:00:1f.2: reg 0x14: [io 0x18e4-0x18e7]
[ 11.115503] pci 0000:00:1f.2: reg 0x18: [io 0x18e8-0x18ef]
[ 11.115516] pci 0000:00:1f.2: reg 0x1c: [io 0x18e0-0x18e3]
[ 11.115528] pci 0000:00:1f.2: reg 0x20: [io 0x18c0-0x18df]
[ 11.115541] pci 0000:00:1f.2: reg 0x24: [mem 0xec606000-0xec6067ff]
[ 11.115617] pci 0000:00:1f.2: PME# supported from D3hot
(XEN) PCI add device 0000:00:1f.2
[ 11.115703] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[ 11.115728] pci 0000:00:1f.3: reg 0x10: [mem 0xec607000-0xec6070ff 64bit]
[ 11.115762] pci 0000:00:1f.3: reg 0x20: [io 0x1100-0x111f]
(XEN) PCI add device 0000:00:1f.3
[ 11.115968] pci 0000:01:00.0: [11ab:6485] type 00 class 0x010400
[ 11.116011] pci 0000:01:00.0: reg 0x18: [io 0x2000-0x207f]
[ 11.116041] pci 0000:01:00.0: reg 0x20: [mem 0xec000000-0xec00ffff 64bit]
[ 11.116055] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[ 11.116117] pci 0000:01:00.0: supports D1
[ 11.116119] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[ 11.116150] pci 0000:01:00.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:01:00.0
[ 11.121894] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 11.121909] pci 0000:00:01.0: bridge window [io 0x2000-0x2fff]
[ 11.121920] pci 0000:00:01.0: bridge window [mem 0xec000000-0xec0fffff]
[ 11.122054] pci 0000:02:00.0: [10de:05fe] type 00 class 0x030000
[ 11.122071] pci 0000:02:00.0: reg 0x10: [mem 0xea000000-0xeaffffff]
[ 11.122087] pci 0000:02:00.0: reg 0x14: [mem 0xd0000000-0xdfffffff
64bit pref]
[ 11.1[ 11.886737] tg3 0000:05:00.0: no hotplug settings from platform
[ 11.886873] pci_bus 0000:07: Allocating resources
[ 11.886956] pcieport 0000:06:00.0: bridge window [io 0x1000-0x0fff]
to [bus 07-09] add_size 1000
[ 11.886961] pcieport 0000:06:00.0: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 07-09] add_size 200000
[ 11.886966] pcieport 0000:06:00.0: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[ 11.886970] pcieport 0000:06:00.0: res[13]=[io 0x1000-0x0fff]
get_res_add_size add_size 1000
[ 11.886976] pcieport 0000:06:00.0: BAR 15: assigned [mem
0xc0600000-0xc07fffff 64bit pref]
[ 11.886979] pcieport 0000:06:00.0: BAR 13: assigned [io 0x7000-0x7fff]
[ 11.887002] pcieport 0000:06:00.0: no hotplug settings from platform
[ 11.887024] pcieport 0000:07:02.0: no hotplug settings from platform
[ 11.887045] pcieport 0000:07:03.0: no hotplug settings from platform
[ 11.887059] tg3 0000:09:00.0: no hotplug settings from platform
[ 11.994139] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 11.994195] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 11.994218] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 11.997794] ata2.00: ATAPI: HL-DT-STDVD-RAM GH60N, NY02, max UDMA/100
[ 12.000368] ata1.00: ATA-8: WDC WD3000BLFS-08YBU0, 04.04V04, max UDMA/100
[ 12.000376] ata1.00: 586072368 sectors, multi 16: LBA48 NCQ (depth
31/32), AA
[ 12.000605] ata3.00: ATA-8: WDC WD3000BLFS-08YBU0, 04.04V04, max UDMA/100
[ 12.000633] ata3.00: 586072368 sectors, multi 16: LBA48 NCQ (depth
31/32), AA
[ 12.002074] ata2.00: configured for UDMA/100
[ 12.006362] ata1.00: configured for UDMA/100
[ 12.006685] scsi 1:0:0:0: Direct-Access ATA WDC WD3000BLFS-0
4V04 PQ: 0 ANSI: 5
[ 12.007576] ata3.00: configured for UDMA/100
[ 12.010795] scsi 2:0:0:0: CD-ROM HL-DT-ST DVD-RAM GH60N
NY02 PQ: 0 ANSI: 5
[ 12.020253] scsi 3:0:0:0: Direct-Access ATA WDC WD3000BLFS-0
4V04 PQ: 0 ANSI: 5
[ 12.023987] sd 1:0:0:0: [sda] 586072368 512-byte logical blocks: (300
GB/279 GiB)
[ 12.024095] sd 1:0:0:0: [sda] Write Protect is off
[ 12.024099] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 12.024137] sd 1:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 12.026873] sr0: scsi3-mmc drive: 40x/40x writer dvd-ram cd/rw
xa/form2 cdda tray
[ 12.026877] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 12.027026] sr 2:0:0:0: Attached scsi CD-ROM sr0
[ 12.027143] sd 3:0:0:0: [sdb] 586072368 512-byte logical blocks: (300
GB/279 GiB)
[ 12.027173] sd 3:0:0:0: [sdb] Write Protect is off
[ 12.027177] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 12.027190] sd 3:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 12.027657] sd 1:0:0:0: Attached scsi generic sg0 type 0
[ 12.027714] sr 2:0:0:0: Attached scsi generic sg1 type 5
[ 12.027760] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 12.033518] sdb: sdb1 sdb2
[ 12.033848] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 12.035876] sda: sda1 sda2
[ 12.036699] sd 1:0:0:0: [sda] Attached SCSI disk
[ 12.115971] md: bind<sdb1>
[ 12.116953] md: bind<sdb2>
[ 12.120452] md: bind<sda1>
[ 12.132123] md: raid1 personality registered for level 1
[ 12.132627] md/raid1:md0: active with 2 out of 2 mirrors
[ 12.132652] md0: detected capacity change from 0 to 536805376
[ 12.132993] md0: unknown partition table
[ 12.138313] md: bind<sda2>
[ 12.140816] md/raid1:md1: active with 2 out of 2 mirrors
[ 12.140884] created bitmap (3 pages) for device md1
[ 12.141017] md1: bitmap initialized from disk: read 1 pages, set 0 of
4462 bits
[ 12.149347] md1: detected capacity change from 0 to 299396562944
[ 12.165820] md1: unknown partition table
[ 12.207392] usb 1-4: New USB device found, idVendor=0bda, idProduct=0181
[ 12.207402] usb 1-4: New USB device strings: Mfr=1, Product=2,
SerialNumber=3
[ 12.207408] usb 1-4: Product: USB2.0-CRW
[ 12.207413] usb 1-4: Manufacturer: Generic
[ 12.207417] usb 1-4: SerialNumber: 20060413092100000
[ 12.215524] usb-storage 1-4:1.0: USB Mass Storage device detected
[ 12.215864] scsi7 : usb-storage 1-4:1.0
[ 12.215924] usbcore: registered new interface driver usb-storage
[ 12.506248] usb 6-2: new low-speed USB device number 2 using uhci_hcd
[ 12.693319] usb 6-2: New USB device found, idVendor=17ef, idProduct=6009
[ 12.693329] usb 6-2: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[ 12.693336] usb 6-2: Product: ThinkPad USB Keyboard with TrackPoint
[ 12.693342] usb 6-2: Manufacturer: Lite-On Technology Corp.
[ 12.698056] hidraw: raw HID events driver (C) Jiri Kosina
[ 12.743346] usbcore: registered new interface driver usbhid
[ 12.743353] usbhid: USB HID core driver
[ 12.744735] input: Lite-On Technology Corp. ThinkPad USB Keyboard
with TrackPoint as
/devices/pci0000:00/0000:00:1d.2/usb6/6-2/6-2:1.0/0003:17EF:6009.0001/input/input2
[ 12.744927] lenovo_tpkbd 0003:17EF:6009.0001: input,hidraw0: USB HID
v1.10 Keyboard [Lite-On Technology Corp. ThinkPad USB Keyboard with
TrackPoint] on usb-0000:00:1d.2-2/input0
[ 12.761360] input: Lite-On Technology Corp. ThinkPad USB Keyboard
with TrackPoint as
/devices/pci0000:00/0000:00:1d.2/usb6/6-2/6-2:1.1/0003:17EF:6009.0002/input/input3
[ 12.762013] lenovo_tpkbd 0003:17EF:6009.0002: input,hiddev0,hidraw1:
USB HID v1.10 Mouse [Lite-On Technology Corp. ThinkPad USB Keyboard with
TrackPoint] on usb-0000:00:1d.2-2/input1
[ 13.217937] scsi 7:0:0:0: Direct-Access Generic- Compact Flash
1.00 PQ: 0 ANSI: 0 CCS
[ 13.221024] scsi 7:0:0:1: Direct-Access Generic- SM/xD-Picture
1.00 PQ: 0 ANSI: 0 CCS
[ 13.224150] scsi 7:0:0:2: Direct-Access Generic- SD/MMC
1.00 PQ: 0 ANSI: 0 CCS
[ 13.227396] scsi 7:0:0:3: Direct-Access Generic- MS/MS-Pro/HG
1.00 PQ: 0 ANSI: 0 CCS
[ 13.228034] sd 7:0:0:0: Attached scsi generic sg3 type 0
[ 13.228345] sd 7:0:0:1: Attached scsi generic sg4 type 0
[ 13.228574] sd 7:0:0:2: Attached scsi generic sg5 type 0
[ 13.228785] sd 7:0:0:3: Attached scsi generic sg6 type 0
[ 13.346389] sd 7:0:0:1: [sdd] Attached SCSI removable disk
[ 13.349011] sd 7:0:0:2: [sde] Attached SCSI removable disk
[ 13.351524] sd 7:0:0:3: [sdf] Attached SCSI removable disk
[ 13.354008] sd 7:0:0:0: [sdc] Attached SCSI removable disk
[ 14.394199] random: nonblocking pool is initialized
[ 16.542191] scsi0 : mvsas
[ 20.145393] device-mapper: uevent: version 1.0.3
[ 20.145537] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30)
initialised: dm-devel@redhat.com
[ 20.402023] PM: Starting manual resume from disk
[ 20.402038] PM: Hibernation image partition 253:1 present
[ 20.402040] PM: Looking for hibernation image.
[ 20.402260] PM: Image not found (code -22)
[ 20.402268] PM: Hibernation image not present or could not be loaded.
[ 20.428292] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Opts: (null)
INIT: version 2.88 booting
[^[[36minfo^[[39;49m] Using makefile-style concurrent boot in runlevel S.
calling: info
[....] Starting the hotplug events dispatcher: udevd[ 21.400512]
systemd-udevd[385]: starting version 215
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Synthesizing the initial hotplug events...calling: trigger
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Waiting for /dev to be fully populated...calling: settle
[ 21.753344] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 21.765849] dca service started, version 1.12.1
[ 21.778668] EDAC MC: Ver: 3.0.0
[ 21.783168] Monitor-Mwait will be used to enter C-1 state
[ 21.783245] Monitor-Mwait will be used to enter C-3 state
[ 21.783295] Monitor-Mwait will be used to enter C-3 state
[ 21.786434] Warning: Processor Platform Limit not supported.
[ 21.797492] ioatdma: Intel(R) QuickData Technology Driver 4.00
[ 21.797594] ioatdma 0000:00:16.0: enabling device (0000 -> 0002)
[ 21.797772] ioatdma 0000:00:16.0: can't derive routing for PCI INT A
[ 21.797776] ioatdma 0000:00:16.0: PCI INT A: no GSI
[ 21.797986] ioatdma 0000:00:16.1: enabling device (0000 -> 0002)
[ 21.798177] ioatdma 0000:00:16.1: can't derive routing for PCI INT B
[ 21.798181] ioatdma 0000:00:16.1: PCI INT B: no GSI
[ 21.798368] ioatdma 0000:00:16.2: enabling device (0000 -> 0002)
[ 21.798545] ioatdma 0000:00:16.2: can't derive routing for PCI INT C
[ 21.798548] ioatdma 0000:00:16.2: PCI INT C: no GSI
[ 21.798739] ioatdma 0000:00:16.3: enabling device (0000 -> 0002)
[ 21.798914] ioatdma 0000:00:16.3: can't derive routing for PCI INT D
[ 21.798918] ioatdma 0000:00:16.3: PCI INT D: no GSI
[ 21.799112] ioatdma 0000:00:16.4: enabling device (0000 -> 0002)
[ 21.799287] ioatdma 0000:00:16.4: can't derive routing for PCI INT A
[ 21.799291] ioatdma 0000:00:16.4: PCI INT A: no GSI
[ 21.799471] ioatdma 0000:00:16.5: enabling device (0000 -> 0002)
[ 21.799646] ioatdma 0000:00:16.5: can't derive routing for PCI INT B
[ 21.799650] ioatdma 0000:00:16.5: PCI INT B: no GSI
[ 21.799841] ioatdma 0000:00:16.6: enabling device (0000 -> 0002)
[ 21.800039] ioatdma 0000:00:16.6: can't derive routing for PCI INT C
[ 21.800042] ioatdma 0000:00:16.6: PCI INT C: no GSI
[ 21.800235] ioatdma 0000:00:16.7: enabling device (0000 -> 0002)
[ 21.800410] ioatdma 0000:00:16.7: can't derive routing for PCI INT D
[ 21.800414] ioatdma 0000:00:16.7: PCI INT D: no GSI
[ 21.823210] input: Power Button as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/PNP0C0C:00/input/input4
[ 21.823218] ACPI: Power Button [PWRB]
[ 21.823257] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input5
[ 21.823261] ACPI: Power Button [PWRF]
[ 21.826900] wmi: Mapper loaded
[ 21.868557] tpm_tis 00:00: 1.2 TPM (device-id 0x1002, rev-id 2)
[ 21.886793] ACPI Warning: SystemIO range
0x0000000000001028-0x000000000000102f conflicts with OpRegion
0x0000000000001000-0x000000000000102f (\_SB_.PCI0.LPC0.PMIO)
(20140424/utaddress-258)
[ 21.886803] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 21.886809] ACPI Warning: SystemIO range
0x00000000000011b0-0x00000000000011bf conflicts with OpRegion
0x0000000000001180-0x00000000000011bf (\_SB_.PCI0.LPC0.GPOX)
(20140424/utaddress-258)
[ 21.886816] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 21.886819] ACPI Warning: SystemIO range
0x0000000000001180-0x00000000000011af conflicts with OpRegion
0x0000000000001180-0x00000000000011bf (\_SB_.PCI0.LPC0.GPOX)
(20140424/utaddress-258)
[ 21.886826] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 21.886829] lpc_ich: Resource conflict(s) found affecting gpio_ich
[ 21.947151] xen: registering gsi 16 triggering 0 polarity 1
[ 21.947158] Already setup the GSI :16
[ 21.983732] xen: registering gsi 17 triggering 0 polarity 1
[ 21.983738] Already setup the GSI :17
[ 21.983758] i801_smbus 0000:00:1f.3: SMBus using PCI Interrupt
[ 22.136624] SSE version of gcm_enc/dec engaged.
[ 22.140249] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[ 22.145847] alg: No test for crc32 (crc32-pclmul)
[ 22.206627] iTCO_vendor_support: vendor-support=0
[ 22.207326] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[ 22.207358] iTCO_wdt: Found a ICH10 TCO device (Version=2,
TCOBASE=0x1060)
[ 22.207508] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[ 22.353888] sound hdaudioC0D2: autoconfig: line_outs=4
(0x12/0x16/0x24/0x25/0x0) type:line
[ 22.353902] sound hdaudioC0D2: speaker_outs=1 (0x13/0x0/0x0/0x0/0x0)
[ 22.353905] sound hdaudioC0D2: hp_outs=1 (0x11/0x0/0x0/0x0/0x0)
[ 22.353908] sound hdaudioC0D2: mono: mono_out=0x0
[ 22.353911] sound hdaudioC0D2: dig-out=0x1b/0x0
[ 22.353913] sound hdaudioC0D2: inputs:
[ 22.353916] sound hdaudioC0D2: Front Mic=0x14
[ 22.353919] sound hdaudioC0D2: Rear Mic=0x17
[ 22.353921] sound hdaudioC0D2: Line=0x15
[ 22.353923] sound hdaudioC0D2: dig-in=0x1c
[ 22.366866] input: HDA Digital PCBeep as
/devices/pci0000:00/0000:00:1b.0/sound/card0/hdaudioC0D2/input7
[ 22.367371] input: HDA Intel Front Mic as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[ 22.367459] input: HDA Intel Rear Mic as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[ 22.367977] input: HDA Intel Line as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[ 22.368546] input: HDA Intel Line Out Front as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[ 22.368616] input: HDA Intel Line Out Surround as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[ 22.368680] input: HDA Intel Line Out CLFE as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[ 22.368745] input: HDA Intel Line Out Side as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input14
[ 22.368819] input: HDA Intel Front Headphone as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input15
[ 23.874188] tpm_tis 00:00: tpm_transmit: tpm_send: error -62
[ 23.874205] tpm_tis 00:00: A TPM error (-62) occurred attempting to
determine the timeouts
[ 23.886254] tpm_tis 00:00: Adjusting TPM timeout parameters.
[ 23.934213] tpm_tis 00:00: TPM is disabled/deactivated (0x6)
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Setting parameters of disc: (none)^[[?25l^[[?1c^[7^[[1G[^[[32m
ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting preliminary keymap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[ 24.563713] EXT4-fs (dm-0): re-mounted. Opts: (null)
[....] Checking root file system...fsck from util-linux 2.25.1
/dev/mapper/vg_g2-lv_g2_root: clean, 276615/1048576 files,
3663404/4194304 blocks
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 24.723128] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
[ 25.036256] xen_acpi_processor: Uploading Xen processor PM info
[ 25.055115] xen_pciback: backend is vpci
[ 25.081652] fuse init (API version 7.23)
[^[[36minfo^[[39;49m] Loading kernel module xen-acpi-processor.
[^[[36minfo^[[39;49m] Loading kernel module xen-pciback.
[^[[36minfo^[[39;49m] Loading kernel module fuse.
[....] Cleaning up temporary files...
/tmp^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Generating udev events for MD
arrays...^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Setting up LVM Volume Groups...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Activating lvm and md swap...[ 25.748811] Adding 2097148k swap
on /dev/mapper/vg_g2-lv_g2_swap. Priority:-1 extents:1 across:2097148k FS
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Checking file systems...fsck from util-linux 2.25.1
/dev/md0: clean, 334/131072 files, 74476/524224 blocks
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 26.072686] EXT4-fs (md0): mounting ext3 file system using the ext4
subsystem
[ 26.082564] EXT4-fs (md0): mounted filesystem with ordered data mode.
Opts: (null)
[....] Mounting local filesystems...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Activating swapfile swap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Cleaning up temporary files...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting kernel variables ...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[ 26.766049] Bridge firewalling registered
[ 26.773155] device eth0 entered promiscuous mode
[ 26.822319] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 26.825755] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[....] Configuring network interfaces...
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
[ 29.990761] tg3 0000:05:00.0 eth0: Link is up at 1000 Mbps, full duplex
[ 29.990778] tg3 0000:05:00.0 eth0: Flow control is on for TX and on
for RX
[ 29.990798] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 29.990821] br0: port 1(eth0) entered forwarding state
[ 29.990829] br0: port 1(eth0) entered forwarding state
[ 29.990849] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Cleaning up temporary files...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[^[[36minfo^[[39;49m] Setting console screen modes.
setterm: cannot (un)set powersave mode: Inappropriate ioctl for device
^[[9;30]^[[14;30][....] Setting up console font and
keymap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 30.864229] ttyS0: LSR safety check engaged!
[ 30.866679] ttyS0: LSR safety check engaged!
Loading the saved-state of the serial devices...
/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A
[....] Setting up X socket directories... /tmp/.X11-unix
/tmp/.ICE-unix^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting sensors limits^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
INIT: Entering runlevel: 2
[^[[36minfo^[[39;49m] Using makefile-style concurrent boot in runlevel 2.
[....] Starting enhanced syslogd:
rsyslogd^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting MD monitoring service: mdadm
--monitor^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting ACPI services...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting mouse interface server:
gpm^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting periodic command scheduler:
cron^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Loading cpufreq kernel modules...modprobe: ERROR: could not
insert 'cpufreq_userspace': No such device
modprobe: ERROR: could not insert 'cpufreq_stats': Invalid argument
modprobe: ERROR: could not insert 'cpufreq_powersave': No such device
modprobe: ERROR: could not insert 'cpufreq_conservative': No such device
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone (none).
[....] Starting system message bus: dbus^[[?25l^[[?1c^[7^[[1G[^[[32m
ok ^[[39;49m^[8^[[?25h^[[?0c.
[ 31.825880] xen:xen_evtchn: Event-channel device installed
[....] CPUFreq Utilities: Setting ondemand CPUFreq governor...disabled,
governor not available...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Starting OpenBSD Secure Shell server:
sshd^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
Starting /usr/local/sbin/oxenstored...
Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenconsoled...
Starting QEMU as disk backend for dom0
^[[9;0]^[[14;0]
[ 35.632309] pciback 0000:02:00.0: seizing device
[ 35.632389] pciback 0000:02:00.0: enabling device (0004 -> 0007)
[ 35.632475] xen: registering gsi 16 triggering 0 polarity 1
[ 35.632487] Already setup the GSI :16
libxl: warning: libxl_pci.c:706:libxl__device_pci_assignable_add:
0000:02:00.0 not bound to a driver, will not be rebound.
Debian GNU/Linux jessie/sid g2 hvc0
g2 login: [ 45.034166] br0: port 1(eth0) entered forwarding state
(XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch input
to DOM0)
(XEN) IRQ information:
(XEN) IRQ: 0 affinity:01 vec:f0 type=IO-APIC-edge status=00000000
timer_interrupt()
(XEN) IRQ: 1 affinity:01 vec:30 type=IO-APIC-edge status=00000014
in-flight=0 domain-list=0: 1(---),
(XEN) IRQ: 3 affinity:01 vec:38 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 4 affinity:01 vec:f1 type=IO-APIC-edge status=00000000
ns16550_interrupt()
(XEN) IRQ: 5 affinity:01 vec:40 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 6 affinity:01 vec:48 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 6(---),
(XEN) IRQ: 7 affinity:01 vec:50 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 8 affinity:01 vec:58 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 8(---),
(XEN) IRQ: 9 affinity:01 vec:60 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 9(---),
(XEN) IRQ: 10 affinity:01 vec:68 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 11 affinity:01 vec:70 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 12 affinity:01 vec:78 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 12(---),
(XEN) IRQ: 13 affinity:3f vec:88 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 14 affinity:01 vec:90 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 15 affinity:01 vec:98 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 16 affinity:01 vec:a0 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 16(---),
(XEN) IRQ: 17 affinity:01 vec:a8 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 17(---),
(XEN) IRQ: 18 affinity:01 vec:b0 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 18(---),
(XEN) IRQ: 19 affinity:01 vec:b8 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 19(---),
(XEN) IRQ: 24 affinity:3f vec:28 type=DMA_MSI status=00000000
iommu_page_fault()
(XEN) IRQ: 25 affinity:01 vec:c0 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:279(---),
(XEN) IRQ: 26 affinity:01 vec:c8 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:278(---),
(XEN) IRQ: 27 affinity:01 vec:d0 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:277(---),
(XEN) IRQ: 28 affinity:01 vec:d8 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:276(---),
(XEN) IRQ: 29 affinity:01 vec:21 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:275(---),
(XEN) IRQ: 30 affinity:01 vec:29 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:274(---),
(XEN) IRQ: 31 affinity:3f vec:31 type=PCI-MSI status=00000002
mapped, unbound
(XEN) IRQ: 32 affinity:3f vec:39 type=PCI-MSI status=00000002
mapped, unbound
(XEN) IRQ: 33 affinity:01 vec:49 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:271(---),
(XEN) IRQ: 34 affinity:01 vec:51 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:270(---),
(XEN) IRQ: 35 affinity:01 vec:59 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:269(---),
(XEN) IRQ: 36 affinity:01 vec:61 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:268(---),
(XEN) IRQ: 37 affinity:01 vec:69 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:267(---),
(XEN) IRQ: 38 affinity:01 vec:71 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:266(---),
(XEN) IRQ: 39 affinity:01 vec:79 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:265(---),
(XEN) IRQ: 40 affinity:01 vec:81 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:264(---),
(XEN) IRQ: 41 affinity:01 vec:89 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:263(---),
(XEN) IRQ: 42 affinity:01 vec:91 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:262(---),
(XEN) IRQ: 43 affinity:01 vec:99 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:261(---),
(XEN) Direct vector information:
(XEN) 0x20 -> irq_move_cleanup_interrupt()
(XEN) 0xf2 -> cmci_interrupt()
(XEN) 0xf3 -> intel_thermal_interrupt()
(XEN) 0xf9 -> pmu_apic_interrupt()
(XEN) 0xfa -> apic_timer_interrupt()
(XEN) 0xfb -> call_function_interrupt()
(XEN) 0xfc -> event_check_interrupt()
(XEN) 0xfd -> invalidate_interrupt()
(XEN) 0xfe -> error_interrupt()
(XEN) 0xff -> spurious_interrupt()
(XEN) IO-APIC interrupt information:
(XEN) IRQ 0 Vec240:
(XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 1 Vec 48:
(XEN) Apic 0x00, Pin 1: vec=30 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 3 Vec 56:
(XEN) Apic 0x00, Pin 3: vec=38 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 4 Vec241:
(XEN) Apic 0x00, Pin 4: vec=f1 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 5 Vec 64:
(XEN) Apic 0x00, Pin 5: vec=40 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 6 Vec 72:
(XEN) Apic 0x00, Pin 6: vec=48 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 7 Vec 80:
(XEN) Apic 0x00, Pin 7: vec=50 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 8 Vec 88:
(XEN) Apic 0x00, Pin 8: vec=58 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 9 Vec 96:
(XEN) Apic 0x00, Pin 9: vec=60 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 10 Vec104:
(XEN) Apic 0x00, Pin 10: vec=68 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 11 Vec112:
(XEN) Apic 0x00, Pin 11: vec=70 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 12 Vec120:
(XEN) Apic 0x00, Pin 12: vec=78 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 13 Vec136:
(XEN) Apic 0x00, Pin 13: vec=88 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=1 dest_id:63
(XEN) IRQ 14 Vec144:
(XEN) Apic 0x00, Pin 14: vec=90 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 15 Vec152:
(XEN) Apic 0x00, Pin 15: vec=98 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 16 Vec160:
(XEN) Apic 0x00, Pin 16: vec=a0 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 17 Vec168:
(XEN) Apic 0x00, Pin 17: vec=a8 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 18 Vec176:
(XEN) Apic 0x00, Pin 18: vec=b0 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 19 Vec184:
(XEN) Apic 0x00, Pin 19: vec=b8 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-23 1:28 ` Steve Freitas
@ 2014-11-24 8:45 ` Jan Beulich
2014-11-24 9:08 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-24 8:45 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 23.11.14 at 02:28, <sflist@ihonk.com> wrote:
> With mwait-idle=0:
>
> (XEN) 'c' pressed -> printing ACPI Cx structures
> (XEN) ==cpu0==
> (XEN) active state: C0
> (XEN) max_cstate: C7
> (XEN) states:
> (XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
> duration[0]
> (XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
> duration[0]
> (XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
> duration[0]
> (XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
> duration[0]
> (XEN) *C0: usage[00000000] duration[46930624784]
> (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
> (XEN) CC3[0] CC6[0] CC7[0]
>[...]
Very interesting - the hypervisor has C-state information, but never
entered any of them. That certainly explains the difference between
using/not using the ,wait-idle driver, but puts us back to there being
a more general issue with C-state use on this CPU model. Possibly
related to C2 having entry method "NONE", but then again I can't
see how such a state could get entered into the table the first place:
set_cx() bails upon check_cx() returning an error, and hence its
switch()'s default statement should never be reached. Plus even if an
array entry was set to "NONE", it should simply be ignored when
looking for a state to enter. I'll probably need to put together a
debugging patch to figure out what's going on here.
In any event C2 being set to "NONE" and that information presumably
coming from firmware is an indication that there's a problem with C2
(note that the numbering doesn't really match up with what the
document says, this likely really is C1E) on that CPU. Which gets us
back to ...
> CPU information for one of the cores, 2.8 GHz is nominal, stepping is 2.
> Not sure how to translate that stepping number into Intel's format:
>
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 44
> model name : Intel(R) Xeon(R) CPU X5660 @ 2.80GHz
> stepping : 2
>[...]
>> There are a couple potentially relevant errata (BC36, BC38, BC54,
>> BC77, BC110).
>>
>> To exclude BC36, a boot log with "apic-verbosity=debug" and debug
>> key 'i' output would be necessary.
>
> Done, see the very end of the email.
>
>> BC38 should not affect us since we don't enter C states from ISRs.
>>
>> BC54 is probably irrelevant since we meanwhile know that your
>> system doesn't really hang hard.
>>
>> For BC77 it would be worth trying to disable turbo mode instead of
>> disabling the mwait-idle driver ("xenpm disable-turbo-mode" right
>> after boot).
>
> I looked up BC77 but as a result found this document[1], which seems to
> relate to the i7. Would this[2] not be the relevant document?
>
> [1]
> http://www.intel.com/content/dam/www/public/us/en/documents/specification-upd
> ates/core-i7-900-ee-and-desktop-processor-series-32nm-spec-update.pdf
>
> [2]
> http://www.intel.com/content/dam/www/public/us/en/documents/specification-upd
> ates/xeon-5600-specification-update.pdf
Indeed. I wasn't aware that there are family/model/stepping tuples
that can be both Xeon and desktop CPUs.
> As promised, below is the apic-verbosity=debug log, with 'i'. Thanks!
I'm sorry, I misspelled the option, it's really "apic_verbosity=debug".
The 'i' output at least already confirms that there are no ExtINT
entries among the IO-APIC RTEs.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-24 8:45 ` Jan Beulich
@ 2014-11-24 9:08 ` Steve Freitas
2014-11-24 9:15 ` Jan Beulich
2014-11-24 11:41 ` Jan Beulich
0 siblings, 2 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-24 9:08 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, xen-devel
On Nov 24, 2014, at 00:45, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 23.11.14 at 02:28, <sflist@ihonk.com> wrote:
>> With mwait-idle=0:
>>
>> (XEN) 'c' pressed -> printing ACPI Cx structures
>> (XEN) ==cpu0==
>> (XEN) active state: C0
>> (XEN) max_cstate: C7
>> (XEN) states:
>> (XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
>> duration[0]
>> (XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
>> duration[0]
>> (XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
>> duration[0]
>> (XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
>> duration[0]
>> (XEN) *C0: usage[00000000] duration[46930624784]
>> (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
>> (XEN) CC3[0] CC6[0] CC7[0]
>> [...]
>
> Very interesting - the hypervisor has C-state information, but never
> entered any of them. That certainly explains the difference between
> using/not using the ,wait-idle driver, but puts us back to there being
> a more general issue with C-state use on this CPU model. Possibly
> related to C2 having entry method "NONE", but then again I can't
> see how such a state could get entered into the table the first place:
> set_cx() bails upon check_cx() returning an error, and hence its
> switch()'s default statement should never be reached. Plus even if an
> array entry was set to "NONE", it should simply be ignored when
> looking for a state to enter. I'll probably need to put together a
> debugging patch to figure out what's going on here.
>
Okay, happy to give it a go whenever you have the time to put something together.
>
>>
>> As promised, below is the apic-verbosity=debug log, with 'i'. Thanks!
>
> I'm sorry, I misspelled the option, it's really "apic_verbosity=debug".
> The 'i' output at least already confirms that there are no ExtINT
> entries among the IO-APIC RTEs.
>
>
No sweat. Do you need me to run it again with the corrected option?
Thanks!
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-24 9:08 ` Steve Freitas
@ 2014-11-24 9:15 ` Jan Beulich
2014-11-24 11:41 ` Jan Beulich
1 sibling, 0 replies; 39+ messages in thread
From: Jan Beulich @ 2014-11-24 9:15 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 24.11.14 at 10:08, <sflist@ihonk.com> wrote:
> On Nov 24, 2014, at 00:45, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 23.11.14 at 02:28, <sflist@ihonk.com> wrote:
>>> As promised, below is the apic-verbosity=debug log, with 'i'. Thanks!
>>
>> I'm sorry, I misspelled the option, it's really "apic_verbosity=debug".
>> The 'i' output at least already confirms that there are no ExtINT
>> entries among the IO-APIC RTEs.
>
> No sweat. Do you need me to run it again with the corrected option?
Yes please, just to be certain we can discount that one.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-24 9:08 ` Steve Freitas
2014-11-24 9:15 ` Jan Beulich
@ 2014-11-24 11:41 ` Jan Beulich
2014-11-24 22:17 ` Steve Freitas
1 sibling, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-24 11:41 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]
>>> On 24.11.14 at 10:08, <sflist@ihonk.com> wrote:
> On Nov 24, 2014, at 00:45, Jan Beulich <JBeulich@suse.com> wrote:
>
>>>>> On 23.11.14 at 02:28, <sflist@ihonk.com> wrote:
>>> With mwait-idle=0:
>>>
>>> (XEN) 'c' pressed -> printing ACPI Cx structures
>>> (XEN) ==cpu0==
>>> (XEN) active state: C0
>>> (XEN) max_cstate: C7
>>> (XEN) states:
>>> (XEN) C1: type[C1] latency[001] usage[00000000] method[ FFH]
>>> duration[0]
>>> (XEN) C2: type[C0] latency[000] usage[00000000] method[ NONE]
>>> duration[0]
>>> (XEN) C3: type[C3] latency[064] usage[00000000] method[ FFH]
>>> duration[0]
>>> (XEN) C4: type[C3] latency[096] usage[00000000] method[ FFH]
>>> duration[0]
>>> (XEN) *C0: usage[00000000] duration[46930624784]
>>> (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
>>> (XEN) CC3[0] CC6[0] CC7[0]
>>> [...]
>>
>> Very interesting - the hypervisor has C-state information, but never
>> entered any of them. That certainly explains the difference between
>> using/not using the ,wait-idle driver, but puts us back to there being
>> a more general issue with C-state use on this CPU model. Possibly
>> related to C2 having entry method "NONE", but then again I can't
>> see how such a state could get entered into the table the first place:
>> set_cx() bails upon check_cx() returning an error, and hence its
>> switch()'s default statement should never be reached. Plus even if an
>> array entry was set to "NONE", it should simply be ignored when
>> looking for a state to enter. I'll probably need to put together a
>> debugging patch to figure out what's going on here.
>
> Okay, happy to give it a go whenever you have the time to put something
> together.
While putting this together I found the reason for the strange
C2: line, and the attached debugging patch already has the fix
for it (which I'll also submit separately, and hence you may need
to drop that specific hunk should you end up applying it on a tree
which already has that fix). You'll need to again run with
"mwait-idle=0", and it's the boot messages along with the 'c'
debug key output that's of interest.
Thanks, Jan
[-- Attachment #2: Freitas-Cx.patch --]
[-- Type: text/plain, Size: 2613 bytes --]
--- unstable.orig/xen/arch/x86/acpi/cpu_idle.c
+++ unstable/xen/arch/x86/acpi/cpu_idle.c
@@ -58,7 +58,7 @@
#include <xen/notifier.h>
#include <xen/cpu.h>
-/*#define DEBUG_PM_CX*/
+#define DEBUG_PM_CX
#define GET_HW_RES_IN_NS(msr, val) \
do { rdmsrl(msr, val); val = tsc_ticks2ns(val); } while( 0 )
@@ -238,6 +238,9 @@ static char* acpi_cstate_method_name[] =
"HALT"
};
+struct reasons { unsigned long max, pwr, urg, nxt; };//temp
+static DEFINE_PER_CPU(struct reasons, reasons);//temp
+
static void print_acpi_power(uint32_t cpu, struct acpi_processor_power *power)
{
uint32_t i, idle_usage = 0;
@@ -273,6 +276,8 @@ static void print_acpi_power(uint32_t cp
printk((last_state_idx == 0) ? " *" : " ");
printk("C0:\tusage[%08d] duration[%"PRId64"]\n",
idle_usage, NOW() - idle_res);
+printk("max=%lx pwr=%lx urg=%lx nxt=%lx\n",//temp
+ per_cpu(reasons.max, cpu), per_cpu(reasons.pwr, cpu), per_cpu(reasons.urg, cpu), per_cpu(reasons.nxt, cpu));
print_hw_residencies(cpu);
}
@@ -501,6 +506,7 @@ static void acpi_processor_idle(void)
u32 exp = 0, pred = 0;
u32 irq_traced[4] = { 0 };
+next_state = 1;//temp
if ( max_cstate > 0 && power && !sched_has_urgent_vcpu() &&
(next_state = cpuidle_current_governor->select(power)) > 0 )
{
@@ -519,6 +525,10 @@ static void acpi_processor_idle(void)
}
if ( !cx )
{
+this_cpu(reasons.max) += max_cstate <= 0;//temp
+this_cpu(reasons.pwr) += !power;//temp
+this_cpu(reasons.urg) += !!sched_has_urgent_vcpu();//temp
+this_cpu(reasons.nxt) += next_state <= 0;//temp
if ( pm_idle_save )
pm_idle_save();
else
@@ -1007,6 +1017,7 @@ static void set_cx(
cx->entry_method = ACPI_CSTATE_EM_SYSIO;
break;
default:
+printk("CPU%u: C%u space %x?\n", acpi_power->cpu, xen_cx->type, xen_cx->reg.space_id);//temp
cx->entry_method = ACPI_CSTATE_EM_NONE;
break;
}
@@ -1015,7 +1026,7 @@ static void set_cx(
cx->target_residency = cx->latency * latency_factor;
smp_wmb();
- acpi_power->count++;
+ acpi_power->count += (cx->type != ACPI_STATE_C1);
if ( cx->type == ACPI_STATE_C1 || cx->type == ACPI_STATE_C2 )
acpi_power->safe_state = cx;
}
@@ -1141,6 +1152,7 @@ long set_cx_pminfo(uint32_t cpu, struct
/* FIXME: C-state dependency is not supported by far */
+printk("CPU%u: %pS, %pS\n", cpu, pm_idle_save, pm_idle);//temp
if ( cpu_id == 0 )
{
if ( pm_idle_save == NULL )
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-24 11:41 ` Jan Beulich
@ 2014-11-24 22:17 ` Steve Freitas
2014-11-25 8:16 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-24 22:17 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, xen-devel
> On 24.11.14 at 10:08,<sflist@ihonk.com> wrote:
> On Nov 24, 2014, at 00:45, Jan Beulich<JBeulich@suse.com> wrote:
>>>>> On 23.11.14 at 02:28,<sflist@ihonk.com> wrote:
>>> As promised, below is the apic-verbosity=debug log, with 'i'. Thanks!
>> I'm sorry, I misspelled the option, it's really "apic_verbosity=debug".
>> The 'i' output at least already confirms that there are no ExtINT
>> entries among the IO-APIC RTEs.
> No sweat. Do you need me to run it again with the corrected option?
> Yes please, just to be certain we can discount that one.
>
I'm combining this action with your patch, see below. Please let me know
if this was incorrect.
On 11/24/2014 3:41, Jan Beulich wrote:
> While putting this together I found the reason for the strange C2:
> line, and the attached debugging patch already has the fix for it
> (which I'll also submit separately, and hence you may need to drop
> that specific hunk should you end up applying it on a tree which
> already has that fix). You'll need to again run with "mwait-idle=0",
> and it's the boot messages along with the 'c' debug key output that's
> of interest. Thanks, Jan
I applied the patch to 4.5-rc2, this was the patch output:
steve@g2:~/git/xen$ patch -p1 < ../Freitas-Cx.patch
patching file xen/arch/x86/acpi/cpu_idle.c
Hunk #2 succeeded at 229 (offset -9 lines).
Hunk #3 succeeded at 263 (offset -10 lines).
Hunk #4 succeeded at 491 (offset -10 lines).
Hunk #5 succeeded at 504 (offset -15 lines).
Hunk #6 succeeded at 992 (offset -15 lines).
Hunk #7 succeeded at 1000 (offset -15 lines).
patch unexpectedly ends in middle of line
Hunk #8 succeeded at 1126 with fuzz 1 (offset -15 lines).
And here's the boot log, with 'c' and 'i' output from Xen:
^[[H^[[J^[[1;1H^[[?25l^[[m^[[H^[[J^[[1;1H^[[2;25HGNU GRUB version 2.02~beta2-15
^[[m^[[4;2H+----------------------------------------------------------------------------+^[[5;2H|^[[5;79H|^[[6;2H|^[[6;79H|^[[7;2H|^[[7;79H|^[[8;2H|^[[8;79H|^[[9;2H|^[[9;79H|^[[10;2H|^[[10;79H|^[[11;2H|^[[11;79H|^[[12;2H|^[[12;79H|^[[13;2H|^[[13;79H|^[[14;2H|^[[14;79H|^[[15;2H|^[[15;79H|^[[16;2H|^[[16;79H|^[[17;2H+----------------------------------------------------------------------------+^[[m^[[18;2H^[[19;2H^[[m
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a
command-line. ^[[5;80H ^[[7m^[[5;3H*Debian
GNU/Linux, with Xen hypervisor ^[[m^[[5;78H^[[m^[[m^[[6;3H Advanced options
for Debian GNU/Linux (with Xen hypervisor)
^[[m^[[6;78H^[[m^[[m^[[7;3H Debian GNU/Linux ^[[m^[[7;78H^[[m^[[m^[[8;3H Advanced
options for Debian GNU/Linux ^[[m^[[8;78H^[[m^[[m^[[9;3H
^[[m^[[9;78H^[[m^[[m^[[10;3H ^[[m^[[10;78H^[[m^[[m^[[11;3H
^[[m^[[11;78H^[[m^[[m^[[12;3H ^[[m^[[12;78H^[[m^[[m^[[13;3H
^[[m^[[13;78H^[[m^[[m^[[14;3H ^[[m^[[14;78H^[[m^[[m^[[15;3H
^[[m^[[15;78H^[[m^[[m^[[16;3H ^[[m^[[16;78H^[[m^[[16;80H ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in
10s. ^[[5;78H^[[22;1H The highlighted entry will be
executed automatically in 9s. ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in
8s. ^[[5;78H^[[22;1H The highlighted entry will be
executed automatically in 7s. ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in
6s. ^[[5;78H^[[22;1H The highlighted entry will be
executed automatically in 5s. ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in
4s. ^[[5;78H^[[22;1H The highlighted entry will be
executed automatically in 3s. ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in
2s. ^[[5;78H^[[22;1H The highlighted entry will be
executed automatically in 1s. ^[[5;78H^[[22;1H The
highlighted entry will be executed automatically in 0s.
^[[5;78H^[[?25h^[[H^[[J^[[1;1H^[[H^[[J^[[1;1HLoading Xen xen ...
Loading Linux 3.16-3-amd64 ...
Loading initial ramdisk ...
Xen 4.5.0-rc
(XEN) Xen version 4.5.0-rc (steve@) (gcc (Debian 4.9.1-19) 4.9.1)
debug=y Mon Nov 24 13:47:48 PST 2014
(XEN) Latest ChangeSet: Tue Nov 11 09:40:20 2014 -0500 git:cacfcc5-dirty
(XEN) Bootloader: GRUB 2.02~beta2-15
(XEN) Command line: placeholder cpufreq=xen:ondemand cpuidle
clocksource=hpet loglvl=all guest_loglvl=all iommu=no-intremap,debug
com1=115200,8n1 console=com1,vga apic_verbosity=debug mwait-idle=0
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009d000 (usable)
(XEN) 000000000009d000 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf7a0000 (usable)
(XEN) 00000000bf7a0000 - 00000000bf7ca000 (ACPI data)
(XEN) 00000000bf7ca000 - 00000000bf7cc000 (ACPI NVS)
(XEN) 00000000bf7cc000 - 00000000c0000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000c40000000 (usable)
(XEN) ACPI: RSDP 000F6A40, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT BF7BF38F, 0084 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: FACP BF7C98D1, 00F4 (r3 LENOVO TC-61 60400D0 PTL 2)
(XEN) ACPI: DSDT BF7BF413, A43A (r1 LENOVO TC-61 60400D0 MSFT 100000E)
(XEN) ACPI: FACS BF7CBFC0, 0040
(XEN) ACPI: SSDT BF7AF33B, 2694 (r1 INTEL PPM RCM 80000001 INTL 20061109)
(XEN) ACPI: SLIT BF7C99E9, 0030 (r1 Intel TYLERBRG 60400D0 LOHR 5A)
(XEN) ACPI: TCPA BF7C9A19, 0032 (r1 LENOVO TC-61 60400D0 PTL 0)
(XEN) ACPI: SLIC BF7C9A4B, 0176 (r1 LENOVO TC-61 60400D0 LTP 0)
(XEN) ACPI: SRAT BF7C9BC1, 00E0 (r1 Intel Tylerbrg 60400D0 PHX. 1)
(XEN) ACPI: DMAR BF7C9CA1, 01C0 (r1 Intel OEMDMAR 60400D0 LOHR 1)
(XEN) ACPI: APIC BF7C9E61, 00A0 (r1 PTLTD APIC 60400D0
LTP 0)
(XEN) ACPI: MCFG BF7C9F01, 003C (r1 PTLTD MCFG 60400D0 LTP 0)
(XEN) ACPI: HPET BF7C9F3D, 0038 (r1 PTLTD HPETTBL 60400D0 LTP 1)
(XEN) ACPI: BOOT BF7C9F75, 0028 (r1 PTLTD $SBFTBL$ 60400D0 LTP 1)
(XEN) ACPI: ASF! BF7C9F9D, 0063 (r32 LENOVO TC-61 60400D0 PTL 1)
(XEN) System RAM: 49143MB (50322676kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-c0000000
(XEN) SRAT: Node 0 PXM 0 100000000-c40000000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f6a70
(XEN) DMI present.
(XEN) APIC boot state is 'xapic'
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1004,1:0], pm1x_evt[1:1000,1:0]
(XEN) ACPI: wakeup_vec[bf7cbfcc], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
(XEN) Processor #2 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
(XEN) Processor #4 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x10] enabled)
(XEN) Processor #16 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
(XEN) Processor #18 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
(XEN) Processor #20 6:12 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) [VT-D]dmar.c:788: Host address width 40
(XEN) [VT-D]dmar.c:802: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:472: dmaru->address = fe710000
(XEN) [VT-D]iommu.c:1146: drhd->address = fe710000 iommu->reg =
ffff82c000201000
(XEN) [VT-D]iommu.c:1148: cap = c90780106f0462 ecap = f0207f
(XEN) [VT-D]dmar.c:397: IOAPIC: 0000:f0:1f.7
(XEN) [VT-D]dmar.c:486: flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.3
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7cd000 end_address
bf7dbfff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e4000 end_address
bf7e4fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e5000 end_address
bf7e5fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7ea000 end_address
bf7eafff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e6000 end_address
bf7e6fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e7000 end_address
bf7e7fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e8000 end_address
bf7e8fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.3
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7e9000 end_address
bf7e9fff
(XEN) [VT-D]dmar.c:807: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383: endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:676: RMRR region: base_addr bf7eb000 end_address
bf7ebfff
(XEN) [VT-D]dmar.c:812: found ACPI_DMAR_ATSR:
(XEN) [VT-D]dmar.c:705: atsru->all_ports: 0
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:01.0 start=0 sec=1 sub=1
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:03.0 start=0 sec=2 sub=2
(XEN) [VT-D]dmar.c:353: bridge: 0000:00:07.0 start=0 sec=3 sub=3
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs)
(XEN) mapped APIC to ffff82cfffdfb000 (fee00000)
(XEN) mapped IOAPIC to ffff82cfffdfa000 (fec00000)
(XEN) IRQ limits: 24 GSI, 1144 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2800.164 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) alt table ffff82d0802d7fd0 -> ffff82d0802d8ff0
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 09
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-09
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping disabled
(XEN) Getting VERSION: 1060015
(XEN) Getting VERSION: 1060015
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Getting ID: 0
(XEN) Getting LVT0: 700
(XEN) Getting LVT1: 400
(XEN) Suppress EOI broadcast on CPU#0
(XEN) enabled ExtINT on CPU#0
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) init IO_APIC IRQs
(XEN) IO-APIC (apicid-pin) 1-0, 1-16, 1-17, 1-18, 1-19, 1-20, 1-21,
1-22, 1-23 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #1 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #1......
(XEN) .... register #00: 01000000
(XEN) ....... : physical APIC id: 01
(XEN) ....... : Delivery Type: 0
(XEN) ....... : LTS : 0
(XEN) .... register #01: 00170020
(XEN) ....... : max redirection entries: 0017
(XEN) ....... : PRQ implemented: 0
(XEN) ....... : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) 00 000 00 1 0 0 0 0 0 0 00
(XEN) 01 001 01 0 0 0 0 0 1 1 30
(XEN) 02 001 01 0 0 0 0 0 1 1 F0
(XEN) 03 001 01 0 0 0 0 0 1 1 38
(XEN) 04 001 01 0 0 0 0 0 1 1 F1
(XEN) 05 001 01 0 0 0 0 0 1 1 40
(XEN) 06 001 01 0 0 0 0 0 1 1 48
(XEN) 07 001 01 0 0 0 0 0 1 1 50
(XEN) 08 001 01 0 0 0 0 0 1 1 58
(XEN) 09 001 01 1 1 0 0 0 1 1 60
(XEN) 0a 001 01 0 0 0 0 0 1 1 68
(XEN) 0b 001 01 0 0 0 0 0 1 1 70
(XEN) 0c 001 01 0 0 0 0 0 1 1 78
(XEN) 0d 001 01 0 0 0 0 0 1 1 88
(XEN) 0e 001 01 0 0 0 0 0 1 1 90
(XEN) 0f 001 01 0 0 0 0 0 1 1 98
(XEN) 10 000 00 1 0 0 0 0 0 0 00
(XEN) 11 000 00 1 0 0 0 0 0 0 00
(XEN) 12 000 00 1 0 0 0 0 0 0 00
(XEN) 13 000 00 1 0 0 0 0 0 0 00
(XEN) 14 000 00 1 0 0 0 0 0 0 00
(XEN) 15 000 00 1 0 0 0 0 0 0 00
(XEN) 16 000 00 1 0 0 0 0 0 0 00
(XEN) 17 000 00 1 0 0 0 0 0 0 00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ48 -> 0:1
(XEN) IRQ56 -> 0:3
(XEN) IRQ241 -> 0:4
(XEN) IRQ64 -> 0:5
(XEN) IRQ72 -> 0:6
(XEN) IRQ80 -> 0:7
(XEN) IRQ88 -> 0:8
(XEN) IRQ96 -> 0:9
(XEN) IRQ104 -> 0:10
(XEN) IRQ112 -> 0:11
(XEN) IRQ120 -> 0:12
(XEN) IRQ136 -> 0:13
(XEN) IRQ144 -> 0:14
(XEN) IRQ152 -> 0:15
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) calibrating APIC timer ...
(XEN) ..... CPU clock speed is 2800.1136 MHz.
(XEN) ..... host bus clock speed is 133.3387 MHz.
(XEN) ..... bus_scale = 0x888d
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: disabled
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Suppress EOI broadcast on CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) Suppress EOI broadcast on CPU#2
(XEN) masked ExtINT on CPU#2
(XEN) Suppress EOI broadcast on CPU#3
(XEN) masked ExtINT on CPU#3
(XEN) Suppress EOI broadcast on CPU#4
(XEN) masked ExtINT on CPU#4
(XEN) Suppress EOI broadcast on CPU#5
(XEN) masked ExtINT on CPU#5
(XEN) Brought up 6 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x7c8000
(XEN) elf_parse_binary: phdr: paddr=0x1800000 memsz=0xed000
(XEN) elf_parse_binary: phdr: paddr=0x18ed000 memsz=0x14f40
(XEN) elf_parse_binary: phdr: paddr=0x1902000 memsz=0x614000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1f16000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff819021f0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0x0
(XEN) virt_offset = 0xffffffff80000000
(XEN) virt_kstart = 0xffffffff81000000
(XEN) virt_kend = 0xffffffff81f16000
(XEN) virt_entry = 0xffffffff819021f0
(XEN) p2m_base = 0xffffffffffffffff
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f16000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000c00000000->0000000c10000000 (12316667 pages
to be allocated)
(XEN) Init. ramdisk: 0000000c3f0da000->0000000c3ffff86e
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff81f16000
(XEN) Init. ramdisk: ffffffff81f16000->ffffffff82e3b86e
(XEN) Phys-Mach map: ffffffff82e3c000->ffffffff88cbb908
(XEN) Start info: ffffffff88cbc000->ffffffff88cbc4b4
(XEN) Page tables: ffffffff88cbd000->ffffffff88d08000
(XEN) Boot stack: ffffffff88d08000->ffffffff88d09000
(XEN) TOTAL: ffffffff80000000->ffffffff89000000
(XEN) ENTRY ADDRESS: ffffffff819021f0
(XEN) Dom0 has maximum 6 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff817c8000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81800000 -> 0xffffffff818ed000
(XEN) elf_load_binary: phdr 2 at 0xffffffff818ed000 -> 0xffffffff81901f40
(XEN) elf_load_binary: phdr 3 at 0xffffffff81902000 -> 0xffffffff81a1f000
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:00:00.0 map
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:14.0
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:14.1
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:14.2
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.1
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.2
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.3
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.4
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.5
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.6
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:16.7
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1a.0
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1a.1
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1a.7
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:00:1b.0
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1d.0
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1d.1
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1d.2
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1d.3
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1d.7
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:00:1f.3
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:01:00.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:05:00.0
(XEN) [VT-D]iommu.c:1444: d0:PCIe: map 0000:09:00.0
(XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:0a:0e.0
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:00.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:00.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.2 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.3 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.4 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:02.5 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:03.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:03.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:03.2 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:03.4 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:04.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:04.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:04.2 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:04.3 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:05.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:05.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:05.2 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:05.3 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:06.0 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:06.1 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:06.2 map
(XEN) [VT-D]iommu.c:1430: d0:Hostbridge: skip 0000:3f:06.3 map
(XEN) [VT-D]iommu.c:739: iommu_enable_translation: iommu->reg =
ffff82c000201000
(XEN) Scrubbing Free RAM on 1 nodes using 6 CPUs
(XEN)
..................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 296kB init memory.
mapping kernel into physical memory
about to get started...
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.16-3-amd64
(debian-kernel@lists.debian.org) (gcc version 4.8.3 (Debian 4.8.3-12) )
#1 SMP Debian 3.16.5-1 (2014-10-10)
[ 0.000000] Command line: placeholder
root=/dev/mapper/vg_g2-lv_g2_root ro console=hvc0 earlyprintk=serial
xencons=hvc debug ignore_loglevel log_buf_len=10M print_fatal_signals=1
LOGLEVEL=8 sched_debug
[ 0.000000] Freeing 9d-100 pfn range: 99 pages freed
[ 0.000000] Freeing bf7a0-100000 pfn range: 264288 pages freed
[ 0.000000] Released 264387 pages of unused memory
[ 0.000000] Set 264387 page(s) to 1-1 mapping
[ 0.000000] Populating bcff21-c107e4 pfn range: 264387 pages added
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
[ 0.000000] Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x00000000bf79ffff] usable
[ 0.000000] Xen: [mem 0x00000000bf7a0000-0x00000000bf7c9fff] ACPI data
[ 0.000000] Xen: [mem 0x00000000bf7ca000-0x00000000bf7cbfff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000bf7cc000-0x00000000bfffffff] reserved
[ 0.000000] Xen: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[ 0.000000] Xen: [mem 0x00000000fe710000-0x00000000fe710fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000c3fffffff] usable
[ 0.000000] bootconsole [earlyser0] enabled
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.6 present.
[ 0.000000] DMI: LENOVO 4158WRG/LENOVO, BIOS 61KT50AUS 01/14/2014
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] AGP: No AGP bridge found
[ 0.000000] e820: last_pfn = 0xc40000 max_arch_pfn = 0x400000000
[ 0.000000] e820: last_pfn = 0xbf7a0 max_arch_pfn = 0x400000000
[ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[ 0.000000] [mem 0x00000000-0x000fffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0xc10400000-0xc105fffff]
[ 0.000000] [mem 0xc10400000-0xc105fffff] page 4k
[ 0.000000] BRK [0x01b61000, 0x01b61fff] PGTABLE
[ 0.000000] BRK [0x01b62000, 0x01b62fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0xc10000000-0xc103fffff]
[ 0.000000] [mem 0xc10000000-0xc103fffff] page 4k
[ 0.000000] BRK [0x01b63000, 0x01b63fff] PGTABLE
[ 0.000000] BRK [0x01b64000, 0x01b64fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0xc00000000-0xc0fffffff]
[ 0.000000] [mem 0xc00000000-0xc0fffffff] page 4k
[ 0.000000] BRK [0x01b65000, 0x01b65fff] PGTABLE
[ 0.000000] BRK [0x01b66000, 0x01b66fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x00100000-0xbf79ffff]
[ 0.000000] [mem 0x00100000-0xbf79ffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0x100000000-0xbffffffff]
[ 0.000000] [mem 0x100000000-0xbffffffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0xc10600000-0xc3fffffff]
[ 0.000000] [mem 0xc10600000-0xc3fffffff] page 4k
[ 0.000000] log_buf_len: 16777216
[ 0.000000] early log buf free: 127296(97%)
[ 0.000000] RAMDISK: [mem 0x01f16000-0x02e3bfff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F6A40 000024 (v02 PTLTD )
[ 0.000000] ACPI: XSDT 0x00000000BF7BF38F 000084 (v01 LENOVO TC-61
060400D0 LTP 00000000)
[ 0.000000] ACPI: FACP 0x00000000BF7C98D1 0000F4 (v03 LENOVO TC-61
060400D0 PTL 00000002)
[ 0.000000] ACPI: DSDT 0x00000000BF7BF413 00A43A (v01 LENOVO TC-61
060400D0 MSFT 0100000E)
[ 0.000000] ACPI: FACS 0x00000000BF7CBFC0 000040
[ 0.000000] ACPI: SSDT 0x00000000BF7AF33B 002694 (v01 INTEL PPM RCM
80000001 INTL 20061109)
[ 0.000000] ACPI: SLIT 0x00000000BF7C99E9 000030 (v01 Intel TYLERBRG
060400D0 LOHR 0000005A)
[ 0.000000] ACPI: TCPA 0x00000000BF7C9A19 000032 (v01 LENOVO TC-61
060400D0 PTL 00000000)
[ 0.000000] ACPI: SLIC 0x00000000BF7C9A4B 000176 (v01 LENOVO TC-61
060400D0 LTP 00000000)
[ 0.000000] ACPI: SRAT 0x00000000BF7C9BC1 0000E0 (v01 Intel Tylerbrg
060400D0 PHX. 00000001)
[ 0.000000] ACPI: XMAR 0x00000000BF7C9CA1 0001C0 (v01 Intel OEMDMAR
060400D0 LOHR 00000001)
[ 0.000000] ACPI: APIC 0x00000000BF7C9E61 0000A0 (v01 PTLTD ? APIC
060400D0 LTP 00000000)
[ 0.000000] ACPI: MCFG 0x00000000BF7C9F01 00003C (v01 PTLTD MCFG
060400D0 LTP 00000000)
[ 0.000000] ACPI: HPET 0x00000000BF7C9F3D 000038 (v01 PTLTD HPETTBL
060400D0 LTP 00000001)
[ 0.000000] ACPI: BOOT 0x00000000BF7C9F75 000028 (v01 PTLTD $SBFTBL$
060400D0 LTP 00000001)
[ 0.000000] ACPI: ASF! 0x00000000BF7C9F9D 000063 (v32 LENOVO TC-61
060400D0 PTL 00000001)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000c3fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x00000000-0xc3fffffff]
[ 0.000000] NODE_DATA [mem 0xc107df000-0xc107e3fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal [mem 0x100000000-0xc3fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009cfff]
[ 0.000000] node 0: [mem 0x00100000-0xbf79ffff]
[ 0.000000] node 0: [mem 0x100000000-0xc3fffffff]
[ 0.000000] On node 0 totalpages: 12580668
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3996 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 10667 pages used for memmap
[ 0.000000] DMA32 zone: 780192 pages, LIFO batch:31
[ 0.000000] Normal zone: 161280 pages used for memmap
[ 0.000000] Normal zone: 11796480 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x1008
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x10] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI
0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ2 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
[ 0.000000] smpboot: Allowing 6 CPUs, 0 hotplug CPUs
[ 0.000000] nr_irqs_gsi: 40
[ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x000fffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7a0000-0xbf7c9fff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7ca000-0xbf7cbfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7cc000-0xbfffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xf7ffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfe70ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfe710000-0xfe710fff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfe711000-0xfebfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec0ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfec10000-0xfedfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfeefffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfef00000-0xfeffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[ 0.000000] e820: [mem 0xc0000000-0xf7ffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.5.0-rc (preserve-AD)
[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512
nr_cpu_ids:6 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff880c03400000 s85824
r8192 d20672 u262144
[ 0.000000] pcpu-alloc: s85824 r8192 d20672 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 - -
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
Total pages: 12408644
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: placeholder
root=/dev/mapper/vg_g2-lv_g2_root ro console=hvc0 earlyprintk=serial
xencons=hvc debug ignore_loglevel log_buf_len=10M print_fatal_signals=1
LOGLEVEL=8 sched_debug
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] software IO TLB [mem 0xbd4e00000-0xbd8e00000] (64MB)
mapped at [ffff880bd4e00000-ffff880bd8dfffff]
[ 0.000000] Memory: 48548940K/50322672K available (5189K kernel code,
942K rwdata, 1824K rodata, 1200K init, 840K bss, 1773732K reserved)
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=6.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
[ 0.000000] NR_IRQS:33024 nr_irqs:728 16
[ 0.000000] xen:events: Using FIFO-based ABI
[ 0.000000] xen: sci override: global_irq=9 trigger=0 polarity=0
[ 0.000000] xen: registering gsi 9 triggering 0 polarity 0
[ 0.000000] xen: --> pirq=9 -> irq=9 (gsi=9)
(XEN) IOAPIC[0]: Set PCI routing entry (1-9 -> 0x60 -> IRQ 9 Mode:1
Active:0)
[ 0.000000] xen: acpi sci 9
[ 0.000000] xen: --> pirq=1 -> irq=1 (gsi=1)
[ 0.000000] xen: --> pirq=2 -> irq=2 (gsi=2)
[ 0.000000] xen: --> pirq=3 -> irq=3 (gsi=3)
[ 0.000000] xen: --> pirq=4 -> irq=4 (gsi=4)
[ 0.000000] xen: --> pirq=5 -> irq=5 (gsi=5)
[ 0.000000] xen: --> pirq=6 -> irq=6 (gsi=6)
[ 0.000000] xen: --> pirq=7 -> irq=7 (gsi=7)
[ 0.000000] xen: --> pirq=8 -> irq=8 (gsi=8)
[ 0.000000] xen: --> pirq=10 -> irq=10 (gsi=10)
[ 0.000000] xen: --> pirq=11 -> irq=11 (gsi=11)
[ 0.000000] xen: --> pirq=12 -> irq=12 (gsi=12)
[ 0.000000] xen: --> pirq=13 -> irq=13 (gsi=13)
[ 0.000000] xen: --> pirq=14 -> irq=14 (gsi=14)
[ 0.000000] xen: --> pirq=15 -> irq=15 (gsi=15)
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [hvc0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] bootconsole [earlyser0] disabled
[ 0.000000] bootconsole [earlyser0] disabled
[ 0.000000] bootconsole [xenboot0] disabled
[ 0.000000] bootconsole [xenboot0] disabled
[ 0.000000] Xen: using vcpuop timer interface
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] tsc: Detected 2800.164 MHz processor
[ 11.929093] Calibrating delay loop (skipped), value calculated using
timer frequency.. 5600.32 BogoMIPS (lpj=11200656)
[ 11.929098] pid_max: default: 32768 minimum: 301
[ 11.929107] ACPI: Core revision 20140424
[ 11.933681] ACPI: All ACPI Tables successfully acquired
[ 11.940307] Security Framework initialized
[ 11.940317] AppArmor: AppArmor disabled by boot time parameter
[ 11.940320] Yama: disabled by default; enable with sysctl kernel.yama.*
[ 11.948791] Dentry cache hash table entries: 8388608 (order: 14,
67108864 bytes)
[ 11.962761] Inode-cache hash table entries: 4194304 (order: 13,
33554432 bytes)
[ 11.967450] Mount-cache hash table entries: 131072 (order: 8, 1048576
bytes)
[ 11.967602] Mountpoint-cache hash table entries: 131072 (order: 8,
1048576 bytes)
[ 11.968060] Initializing cgroup subsys memory
[ 11.968066] Initializing cgroup subsys devices
[ 11.968073] Initializing cgroup subsys freezer
[ 11.968077] Initializing cgroup subsys net_cls
[ 11.968082] Initializing cgroup subsys blkio
[ 11.968086] Initializing cgroup subsys perf_event
[ 11.968089] Initializing cgroup subsys net_prio
[ 11.968142] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 11.968142] ENERGY_PERF_BIAS: View and update with
x86_energy_perf_policy(8)
[ 11.968171] CPU: Physical Processor ID: 0
[ 11.968173] CPU: Processor Core ID: 0
[ 11.968176] mce: CPU supports 2 MCE banks
[ 11.968191] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[ 11.968191] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0
[ 11.968191] tlb_flushall_shift: 6
[ 11.968257] Freeing SMP alternatives memory: 20K (ffffffff81a19000 -
ffffffff81a1e000)
[ 11.969139] ftrace: allocating 21546 entries in 85 pages
[ 11.976086] Performance Events: unsupported p6 CPU model 44 no PMU
driver, software events only.
[ 11.977204] NMI watchdog: disabled (cpu0): hardware events not enabled
[ 11.977282] installing Xen timer for CPU 1
[ 11.977528] installing Xen timer for CPU 2
[ 11.977726] installing Xen timer for CPU 3
[ 11.977921] installing Xen timer for CPU 4
[ 11.978123] installing Xen timer for CPU 5
[ 11.978244] x86: Booted up 1 node, 6 CPUs
[ 11.978264] CPU0 attaching sched-domain:
[ 11.978267] domain 0: span 0-5 level MC
[ 11.978269] groups: 0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023)
2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023)
5 (cpu_capacity = 1023)
[ 11.978281] CPU1 attaching sched-domain:
[ 11.978283] domain 0: span 0-5 level MC
[ 11.978285] groups: 1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023)
3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023)
0 (cpu_capacity = 1023)
[ 11.978296] CPU2 attaching sched-domain:
[ 11.978298] domain 0: span 0-5 level MC
[ 11.978300] groups: 2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023)
4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023)
1 (cpu_capacity = 1023)
[ 11.978310] CPU3 attaching sched-domain:
[ 11.978312] domain 0: span 0-5 level MC
[ 11.978314] groups: 3 (cpu_capacity = 1023) 4 (cpu_capacity = 1023)
5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023)
2 (cpu_capacity = 1023)
[ 11.978324] CPU4 attaching sched-domain:
[ 11.978326] domain 0: span 0-5 level MC
[ 11.978328] groups: 4 (cpu_capacity = 1023) 5 (cpu_capacity = 1023)
0 (cpu_capacity = 1023) 1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023)
3 (cpu_capacity = 1023)
[ 11.978339] CPU5 attaching sched-domain:
[ 11.978341] domain 0: span 0-5 level MC
[ 11.978343] groups: 5 (cpu_capacity = 1023) 0 (cpu_capacity = 1023)
1 (cpu_capacity = 1023) 2 (cpu_capacity = 1023) 3 (cpu_capacity = 1023)
4 (cpu_capacity = 1023)
[ 11.978524] devtmpfs: initialized
[ 11.984790] PM: Registering ACPI NVS region [mem
0xbf7ca000-0xbf7cbfff] (8192 bytes)
[ 11.985592] pinctrl core: initialized pinctrl subsystem
[ 11.985667] NET: Registered protocol family 16
[ 11.985679] xen:grant_table: Grant tables using version 1 layout
[ 11.985690] Grant table initialized
[ 11.985944] ACPI FADT declares the system doesn't support PCIe ASPM,
so disable it
[ 11.985949] ACPI: bus type PCI registered
[ 11.985951] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 11.986130] PCI: MMCONFIG for domain 0000 [bus 00-09] at [mem
0xf8000000-0xf89fffff] (base 0xf8000000)
[ 11.986135] PCI: MMCONFIG at [mem 0xf8000000-0xf89fffff] reserved in E820
[ 11.987060] PCI: Using configuration type 1 for base access
[ 11.996829] ACPI: Added _OSI(Module Device)
[ 11.996852] ACPI: Added _OSI(Processor Device)
[ 11.996854] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 11.996856] ACPI: Added _OSI(Processor Aggregator Device)
[ 11.999982] ACPI: Interpreter enabled
[ 11.999989] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S1_] (20140424/hwxface-580)
[ 11.999994] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S2_] (20140424/hwxface-580)
[ 12.000006] ACPI: (supports S0 S3 S4 S5)
[ 12.000008] ACPI: Using IOAPIC for interrupt routing
[ 12.000032] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[ 12.013771] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 12.013779] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI]
[ 12.013998] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug PME
AER PCIeCapability]
[ 12.014671] acpi PNP0A03:00: [Firmware Info]: MMCONFIG for domain
0000 [bus 00-09] only partially covers this bridge
[ 12.014847] PCI host bridge to bus 0000:00
[ 12.014851] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 12.014878] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
[ 12.014881] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff]
[ 12.014884] pci_bus 0000:00: root bus resource [mem
0x000a0000-0x000bffff]
[ 12.014887] pci_bus 0000:00: root bus resource [mem
0x000d0000-0x000d3fff]
[ 12.014890] pci_bus 0000:00: root bus resource [mem
0x000d4000-0x000d7fff]
[ 12.014893] pci_bus 0000:00: root bus resource [mem
0x000d8000-0x000dbfff]
[ 12.014896] pci_bus 0000:00: root bus resource [mem
0x000dc000-0x000dffff]
[ 12.014899] pci_bus 0000:00: root bus resource [mem
0xc0000000-0xfdffffff]
[ 12.014918] pci 0000:00:00.0: [8086:3406] type 00 class 0x060000
[ 12.015018] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
(XEN) Found masked UR signaling on 0000:00:00.0
(XEN) PCI add device 0000:00:00.0
[ 12.015131] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[ 12.015239] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[ 12.015288] pci 0000:00:01.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:01.0
(XEN) PCI add device 0000:00:01.0
[ 12.015356] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[ 12.015464] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[ 12.015513] pci 0000:00:03.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:03.0
(XEN) PCI add device 0000:00:03.0
[ 12.015583] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[ 12.015682] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[ 12.015733] pci 0000:00:07.0: System wakeup disabled by ACPI
(XEN) Found masked UR signaling on 0000:00:07.0
(XEN) PCI add device 0000:00:07.0
[ 12.015807] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
(XEN) Masked VT-d error signaling on 0000:00:14.0
(XEN) PCI add device 0000:00:14.0
[ 12.015968] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
(XEN) PCI add device 0000:00:14.1
[ 12.016122] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
(XEN) PCI add device 0000:00:14.2
[ 12.016294] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[ 12.016316] pci 0000:00:16.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.0
[ 12.016480] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[ 12.016502] pci 0000:00:16.1: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.1
[ 12.016667] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[ 12.016688] pci 0000:00:16.2: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.2
[ 12.016853] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[ 12.016874] pci 0000:00:16.3: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.3
[ 12.017039] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[ 12.017060] pci 0000:00:16.4: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.4
[ 12.017226] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[ 12.017247] pci 0000:00:16.5: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.5
[ 12.017411] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[ 12.017433] pci 0000:00:16.6: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.6
[ 12.017597] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[ 12.017618] pci 0000:00:16.7: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
(XEN) PCI add device 0000:00:16.7
[ 12.017784] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[ 12.017854] pci 0000:00:1a.0: reg 0x20: [io 0x1800-0x181f]
[ 12.017963] pci 0000:00:1a.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.0
[ 12.018018] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[ 12.018088] pci 0000:00:1a.1: reg 0x20: [io 0x1820-0x183f]
[ 12.018207] pci 0000:00:1a.1: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.1
[ 12.018277] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[ 12.018311] pci 0000:00:1a.7: reg 0x10: [mem 0xec604000-0xec6043ff]
[ 12.018451] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[ 12.018498] pci 0000:00:1a.7: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1a.7
[ 12.018558] pci 0000:00:1b.0: [8086:3a3e] type 00 class 0x040300
[ 12.018585] pci 0000:00:1b.0: reg 0x10: [mem 0xec600000-0xec603fff 64bit]
[ 12.018704] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
(XEN) PCI add device 0000:00:1b.0
[ 12.018798] pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
[ 12.018912] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[ 12.018962] pci 0000:00:1c.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.0
[ 12.019022] pci 0000:00:1c.4: [8086:3a48] type 01 class 0x060400
[ 12.019158] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[ 12.019209] pci 0000:00:1c.4: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.4
[ 12.019263] pci 0000:00:1c.5: [8086:3a4a] type 01 class 0x060400
[ 12.019385] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[ 12.019436] pci 0000:00:1c.5: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1c.5
[ 12.019493] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[ 12.019562] pci 0000:00:1d.0: reg 0x20: [io 0x1840-0x185f]
[ 12.019670] pci 0000:00:1d.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.0
[ 12.019724] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[ 12.019794] pci 0000:00:1d.1: reg 0x20: [io 0x1860-0x187f]
[ 12.019902] pci 0000:00:1d.1: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.1
[ 12.019960] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[ 12.020055] pci 0000:00:1d.2: reg 0x20: [io 0x1880-0x189f]
[ 12.020185] pci 0000:00:1d.2: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.2
[ 12.020243] pci 0000:00:1d.3: [8086:3a39] type 00 class 0x0c0300
[ 12.020313] pci 0000:00:1d.3: reg 0x20: [io 0x18a0-0x18bf]
[ 12.020422] pci 0000:00:1d.3: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.3
[ 12.020490] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[ 12.020524] pci 0000:00:1d.7: reg 0x10: [mem 0xec605000-0xec6053ff]
[ 12.020664] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[ 12.020711] pci 0000:00:1d.7: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1d.7
[ 12.020764] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[ 12.020878] pci 0000:00:1e.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:00:1e.0
[ 12.020932] pci 0000:00:1f.0: [8086:3a18] type 00 class 0x060100
(XEN) PCI add device 0000:00:1f.0
[ 12.021167] pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
[ 12.021199] pci 0000:00:1f.2: reg 0x10: [io 0x18f0-0x18f7]
[ 12.021212] pci 0000:00:1f.2: reg 0x14: [io 0x18e4-0x18e7]
[ 12.021225] pci 0000:00:1f.2: reg 0x18: [io 0x18e8-0x18ef]
[ 12.021238] pci 0000:00:1f.2: reg 0x1c: [io 0x18e0-0x18e3]
[ 12.021252] pci 0000:00:1f.2: reg 0x20: [io 0x18c0-0x18df]
[ 12.021265] pci 0000:00:1f.2: reg 0x24: [mem 0xec606000-0xec6067ff]
[ 12.021343] pci 0000:00:1f.2: PME# supported from D3hot
(XEN) PCI add device 0000:00:1f.2
[ 12.021433] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[ 12.021459] pci 0000:00:1f.3: reg 0x10: [mem 0xec607000-0xec6070ff 64bit]
[ 12.021494] pci 0000:00:1f.3: reg 0x20: [io 0x1100-0x111f]
(XEN) PCI add device 0000:00:1f.3
[ 12.021686] pci 0000:01:00.0: [11ab:6485] type 00 class 0x010400
[ 12.021730] pci 0000:01:00.0: reg 0x18: [io 0x2000-0x207f]
[ 12.021761] pci 0000:01:00.0: reg 0x20: [mem 0xec000000-0xec00ffff 64bit]
[ 12.021776] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[ 12.021839] pci 0000:01:00.0: supports D1
[ 12.021842] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[ 12.021874] pci 0000:01:00.0: System wakeup disabled by ACPI
(XEN) PCI add device 0000:01:00.0
[ 12.132501] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 12.132513] pci 0000:00:01.0: bridge window [io 0x2000-0x2fff]
[ 12.132523] pci 0000:00:01.0: bridge window [mem 0xec000000-0xec0fffff]
[ 12.132640] pci 0000:02:00.0: [10de:05fe] type 00 class 0x030000
[ 12[ 12.916581] tg3 0000:05:00.0: no hotplug settings from platform
[ 12.916719] pci_bus 0000:07: Allocating resources
[ 12.916778] pcieport 0000:06:00.0: bridge window [io 0x1000-0x0fff]
to [bus 07-09] add_size 1000
[ 12.916783] pcieport 0000:06:00.0: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 07-09] add_size 200000
[ 12.916788] pcieport 0000:06:00.0: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[ 12.916793] pcieport 0000:06:00.0: res[13]=[io 0x1000-0x0fff]
get_res_add_size add_size 1000
[ 12.916799] pcieport 0000:06:00.0: BAR 15: assigned [mem
0xc0600000-0xc07fffff 64bit pref]
[ 12.916802] pcieport 0000:06:00.0: BAR 13: assigned [io 0x7000-0x7fff]
[ 12.916825] pcieport 0000:06:00.0: no hotplug settings from platform
[ 12.916848] pcieport 0000:07:02.0: no hotplug settings from platform
[ 12.916870] pcieport 0000:07:03.0: no hotplug settings from platform
[ 12.916885] tg3 0000:09:00.0: no hotplug settings from platform
[ 13.000128] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 13.000151] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 13.000172] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 13.003705] ata2.00: ATAPI: HL-DT-STDVD-RAM GH60N, NY02, max UDMA/100
[ 13.006112] ata1.00: ATA-8: WDC WD3000BLFS-08YBU0, 04.04V04, max UDMA/100
[ 13.006119] ata1.00: 586072368 sectors, multi 16: LBA48 NCQ (depth
31/32), AA
[ 13.006464] ata3.00: ATA-8: WDC WD3000BLFS-08YBU0, 04.04V04, max UDMA/100
[ 13.006471] ata3.00: 586072368 sectors, multi 16: LBA48 NCQ (depth
31/32), AA
[ 13.008025] ata2.00: configured for UDMA/100
[ 13.012115] ata1.00: configured for UDMA/100
[ 13.012249] scsi 1:0:0:0: Direct-Access ATA WDC WD3000BLFS-0
4V04 PQ: 0 ANSI: 5
[ 13.013518] ata3.00: configured for UDMA/100
[ 13.015809] scsi 2:0:0:0: CD-ROM HL-DT-ST DVD-RAM GH60N
NY02 PQ: 0 ANSI: 5
[ 13.025883] scsi 3:0:0:0: Direct-Access ATA WDC WD3000BLFS-0
4V04 PQ: 0 ANSI: 5
[ 13.028500] sd 1:0:0:0: [sda] 586072368 512-byte logical blocks: (300
GB/279 GiB)
[ 13.028604] sd 1:0:0:0: [sda] Write Protect is off
[ 13.028609] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 13.028651] sd 1:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 13.031377] sr0: scsi3-mmc drive: 40x/40x writer dvd-ram cd/rw
xa/form2 cdda tray
[ 13.031386] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 13.031517] sr 2:0:0:0: Attached scsi CD-ROM sr0
[ 13.031631] sd 3:0:0:0: [sdb] 586072368 512-byte logical blocks: (300
GB/279 GiB)
[ 13.031724] sd 3:0:0:0: [sdb] Write Protect is off
[ 13.031728] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 13.031768] sd 3:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 13.032021] sd 1:0:0:0: Attached scsi generic sg0 type 0
[ 13.032079] sr 2:0:0:0: Attached scsi generic sg1 type 5
[ 13.032124] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 13.037844] sda: sda1 sda2
[ 13.038211] sd 1:0:0:0: [sda] Attached SCSI disk
[ 13.043673] sdb: sdb1 sdb2
[ 13.043983] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 13.121238] md: bind<sdb2>
[ 13.122241] md: bind<sda1>
[ 13.132958] md: bind<sda2>
[ 13.135006] md: raid1 personality registered for level 1
[ 13.135363] md/raid1:md1: active with 2 out of 2 mirrors
[ 13.135437] created bitmap (3 pages) for device md1
[ 13.135600] md1: bitmap initialized from disk: read 1 pages, set 0 of
4462 bits
[ 13.140431] md1: detected capacity change from 0 to 299396562944
[ 13.143731] md1: unknown partition table
[ 13.152229] md: bind<sdb1>
[ 13.154307] md/raid1:md0: active with 2 out of 2 mirrors
[ 13.154327] md0: detected capacity change from 0 to 536805376
[ 13.160470] md0: unknown partition table
[ 13.232470] usb 1-4: New USB device found, idVendor=0bda, idProduct=0181
[ 13.232480] usb 1-4: New USB device strings: Mfr=1, Product=2,
SerialNumber=3
[ 13.232486] usb 1-4: Product: USB2.0-CRW
[ 13.232491] usb 1-4: Manufacturer: Generic
[ 13.232495] usb 1-4: SerialNumber: 20060413092100000
[ 13.239350] usb-storage 1-4:1.0: USB Mass Storage device detected
[ 13.239478] scsi7 : usb-storage 1-4:1.0
[ 13.239535] usbcore: registered new interface driver usb-storage
[ 13.532137] usb 7-2: new low-speed USB device number 2 using uhci_hcd
[ 13.923165] usb 7-2: New USB device found, idVendor=17ef, idProduct=6009
[ 13.923174] usb 7-2: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[ 13.923181] usb 7-2: Product: ThinkPad USB Keyboard with TrackPoint
[ 13.923186] usb 7-2: Manufacturer: Lite-On Technology Corp.
[ 13.927205] hidraw: raw HID events driver (C) Jiri Kosina
[ 13.972212] usbcore: registered new interface driver usbhid
[ 13.972215] usbhid: USB HID core driver
[ 13.972906] input: Lite-On Technology Corp. ThinkPad USB Keyboard
with TrackPoint as
/devices/pci0000:00/0000:00:1d.2/usb7/7-2/7-2:1.0/0003:17EF:6009.0001/input/input2
[ 13.972969] lenovo_tpkbd 0003:17EF:6009.0001: input,hidraw0: USB HID
v1.10 Keyboard [Lite-On Technology Corp. ThinkPad USB Keyboard with
TrackPoint] on usb-0000:00:1d.2-2/input0
[ 13.990235] input: Lite-On Technology Corp. ThinkPad USB Keyboard
with TrackPoint as
/devices/pci0000:00/0000:00:1d.2/usb7/7-2/7-2:1.1/0003:17EF:6009.0002/input/input3
[ 13.990392] lenovo_tpkbd 0003:17EF:6009.0002: input,hiddev0,hidraw1:
USB HID v1.10 Mouse [Lite-On Technology Corp. ThinkPad USB Keyboard with
TrackPoint] on usb-0000:00:1d.2-2/input1
[ 14.239376] scsi 7:0:0:0: Direct-Access Generic- Compact Flash
1.00 PQ: 0 ANSI: 0 CCS
[ 14.242371] scsi 7:0:0:1: Direct-Access Generic- SM/xD-Picture
1.00 PQ: 0 ANSI: 0 CCS
[ 14.245371] scsi 7:0:0:2: Direct-Access Generic- SD/MMC
1.00 PQ: 0 ANSI: 0 CCS
[ 14.248495] scsi 7:0:0:3: Direct-Access Generic- MS/MS-Pro/HG
1.00 PQ: 0 ANSI: 0 CCS
[ 14.248736] sd 7:0:0:0: Attached scsi generic sg3 type 0
[ 14.248893] sd 7:0:0:1: Attached scsi generic sg4 type 0
[ 14.249071] sd 7:0:0:2: Attached scsi generic sg5 type 0
[ 14.249291] sd 7:0:0:3: Attached scsi generic sg6 type 0
[ 14.340337] random: nonblocking pool is initialized
[ 14.362979] sd 7:0:0:0: [sdc] Attached SCSI removable disk
[ 14.365478] sd 7:0:0:1: [sdd] Attached SCSI removable disk
[ 14.368123] sd 7:0:0:2: [sde] Attached SCSI removable disk
[ 14.370608] sd 7:0:0:3: [sdf] Attached SCSI removable disk
[ 17.556123] scsi0 : mvsas
[ 21.183838] device-mapper: uevent: version 1.0.3
[ 21.183918] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30)
initialised: dm-devel@redhat.com
[ 21.448437] PM: Starting manual resume from disk
[ 21.448452] PM: Hibernation image partition 253:1 present
[ 21.448455] PM: Looking for hibernation image.
[ 21.448623] PM: Image not found (code -22)
[ 21.448634] PM: Hibernation image not present or could not be loaded.
[ 21.467950] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Opts: (null)
INIT: version 2.88 booting
[^[[36minfo^[[39;49m] Using makefile-style concurrent boot in runlevel S.
calling: info
[....] Starting the hotplug events dispatcher: udevd[ 22.384236]
systemd-udevd[387]: starting version 215
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Synthesizing the initial hotplug events...calling: trigger
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Waiting for /dev to be fully populated...calling: settle
[ 22.771824] dca service started, version 1.12.1
[ 22.787412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 22.793212] Monitor-Mwait will be used to enter C-1 state
[ 22.793243] Monitor-Mwait will be used to enter C-3 state
[ 22.793262] Monitor-Mwait will be used to enter C-3 state
[ 22.796020] Warning: Processor Platform Limit not supported.
[ 22.798559] EDAC MC: Ver: 3.0.0
[ 22.819933] ioatdma: Intel(R) QuickData Technology Driver 4.00
[ 22.819978] ioatdma 0000:00:16.0: enabling device (0000 -> 0002)
[ 22.820180] ioatdma 0000:00:16.0: can't derive routing for PCI INT A
[ 22.820184] ioatdma 0000:00:16.0: PCI INT A: no GSI
[ 22.820398] ioatdma 0000:00:16.1: enabling device (0000 -> 0002)
[ 22.820569] ioatdma 0000:00:16.1: can't derive routing for PCI INT B
[ 22.820572] ioatdma 0000:00:16.1: PCI INT B: no GSI
[ 22.820767] ioatdma 0000:00:16.2: enabling device (0000 -> 0002)
[ 22.820938] ioatdma 0000:00:16.2: can't derive routing for PCI INT C
[ 22.820941] ioatdma 0000:00:16.2: PCI INT C: no GSI
[ 22.821127] ioatdma 0000:00:16.3: enabling device (0000 -> 0002)
[ 22.821320] ioatdma 0000:00:16.3: can't derive routing for PCI INT D
[ 22.821324] ioatdma 0000:00:16.3: PCI INT D: no GSI
[ 22.821529] ioatdma 0000:00:16.4: enabling device (0000 -> 0002)
[ 22.821700] ioatdma 0000:00:16.4: can't derive routing for PCI INT A
[ 22.821704] ioatdma 0000:00:16.4: PCI INT A: no GSI
[ 22.821891] ioatdma 0000:00:16.5: enabling device (0000 -> 0002)
[ 22.822061] ioatdma 0000:00:16.5: can't derive routing for PCI INT B
[ 22.822065] ioatdma 0000:00:16.5: PCI INT B: no GSI
[ 22.822261] ioatdma 0000:00:16.6: enabling device (0000 -> 0002)
[ 22.822432] ioatdma 0000:00:16.6: can't derive routing for PCI INT C
[ 22.822435] ioatdma 0000:00:16.6: PCI INT C: no GSI
[ 22.822621] ioatdma 0000:00:16.7: enabling device (0000 -> 0002)
[ 22.822792] ioatdma 0000:00:16.7: can't derive routing for PCI INT D
[ 22.822796] ioatdma 0000:00:16.7: PCI INT D: no GSI
[ 22.829482] xen: registering gsi 16 triggering 0 polarity 1
[ 22.829489] Already setup the GSI :16
[ 22.830810] input: Power Button as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/PNP0C0C:00/input/input4
[ 22.830818] ACPI: Power Button [PWRB]
[ 22.830857] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input5
[ 22.830861] ACPI: Power Button [PWRF]
[ 22.840696] wmi: Mapper loaded
[ 22.844439] tpm_tis 00:00: 1.2 TPM (device-id 0x1002, rev-id 2)
[ 22.908910] ACPI Warning: SystemIO range
0x0000000000001028-0x000000000000102f conflicts with OpRegion
0x0000000000001000-0x000000000000102f (\_SB_.PCI0.LPC0.PMIO)
(20140424/utaddress-258)
[ 22.908920] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 22.908927] ACPI Warning: SystemIO range
0x00000000000011b0-0x00000000000011bf conflicts with OpRegion
0x0000000000001180-0x00000000000011bf (\_SB_.PCI0.LPC0.GPOX)
(20140424/utaddress-258)
[ 22.908934] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 22.908937] ACPI Warning: SystemIO range
0x0000000000001180-0x00000000000011af conflicts with OpRegion
0x0000000000001180-0x00000000000011bf (\_SB_.PCI0.LPC0.GPOX)
(20140424/utaddress-258)
[ 22.908944] ACPI: If an ACPI driver is available for this device, you
should use it instead of the native driver
[ 22.908947] lpc_ich: Resource conflict(s) found affecting gpio_ich
[ 22.925327] xen: registering gsi 17 triggering 0 polarity 1
[ 22.925334] Already setup the GSI :17
[ 22.925356] i801_smbus 0000:00:1f.3: SMBus using PCI Interrupt
[ 23.054556] SSE version of gcm_enc/dec engaged.
[ 23.057659] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[ 23.142657] alg: No test for crc32 (crc32-pclmul)
[ 23.188538] iTCO_vendor_support: vendor-support=0
[ 23.189004] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[ 23.189039] iTCO_wdt: Found a ICH10 TCO device (Version=2,
TCOBASE=0x1060)
[ 23.189153] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[ 23.253301] sound hdaudioC0D2: autoconfig: line_outs=4
(0x12/0x16/0x24/0x25/0x0) type:line
[ 23.253315] sound hdaudioC0D2: speaker_outs=1 (0x13/0x0/0x0/0x0/0x0)
[ 23.253318] sound hdaudioC0D2: hp_outs=1 (0x11/0x0/0x0/0x0/0x0)
[ 23.253321] sound hdaudioC0D2: mono: mono_out=0x0
[ 23.253324] sound hdaudioC0D2: dig-out=0x1b/0x0
[ 23.253326] sound hdaudioC0D2: inputs:
[ 23.253329] sound hdaudioC0D2: Front Mic=0x14
[ 23.253332] sound hdaudioC0D2: Rear Mic=0x17
[ 23.253334] sound hdaudioC0D2: Line=0x15
[ 23.253336] sound hdaudioC0D2: dig-in=0x1c
[ 23.265840] input: HDA Digital PCBeep as
/devices/pci0000:00/0000:00:1b.0/sound/card0/hdaudioC0D2/input7
[ 23.266180] input: HDA Intel Front Mic as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[ 23.266275] input: HDA Intel Rear Mic as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[ 23.266339] input: HDA Intel Line as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[ 23.266414] input: HDA Intel Line Out Front as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[ 23.266493] input: HDA Intel Line Out Surround as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[ 23.267432] input: HDA Intel Line Out CLFE as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[ 23.267659] input: HDA Intel Line Out Side as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input14
[ 23.267728] input: HDA Intel Front Headphone as
/devices/pci0000:00/0000:00:1b.0/sound/card0/input15
[ 24.852132] tpm_tis 00:00: tpm_transmit: tpm_send: error -62
[ 24.852148] tpm_tis 00:00: A TPM error (-62) occurred attempting to
determine the timeouts
[ 24.864212] tpm_tis 00:00: Adjusting TPM timeout parameters.
[ 24.912148] tpm_tis 00:00: TPM is disabled/deactivated (0x6)
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Setting parameters of disc: (none)^[[?25l^[[?1c^[7^[[1G[^[[32m
ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting preliminary keymap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[ 25.517579] EXT4-fs (dm-0): re-mounted. Opts: (null)
[....] Checking root file system...fsck from util-linux 2.25.1
/dev/mapper/vg_g2-lv_g2_root: clean, 275985/1048576 files,
3651430/4194304 blocks
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 25.654371] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
[ 25.937101] xen_acpi_processor: Uploading Xen processor PM info
(XEN) cpu0 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) Monitor-Mwait will be used to enter C1 state
(XEN) Monitor-Mwait will be used to enter C3 state
(XEN) CPU0: 0000000000000000, default_idle+0/0x7b
(XEN) cpu1 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) CPU1: default_idle+0/0x7b, acpi_processor_idle+0/0x531
(XEN) cpu2 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) CPU2: default_idle+0/0x7b, acpi_processor_idle+0/0x531
(XEN) cpu3 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) CPU3: default_idle+0/0x7b, acpi_processor_idle+0/0x531
(XEN) cpu4 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) CPU4: default_idle+0/0x7b, acpi_processor_idle+0/0x531
(XEN) cpu5 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) CPU5: default_idle+0/0x7b, acpi_processor_idle+0/0x531
(XEN) cpu6 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) No CPU ID for APIC ID 0x6
(XEN) cpu7 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) cpu8 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) cpu9 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) cpu10 cx acpi info:
(XEN) count = 3
(XEN) flags: bm_cntl[0], bm_chk[1], has_cst[1],
(XEN) pwr_setup_done[1], bm_rld_set[0]
(XEN) states[0]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0
(XEN) type = 1
(XEN) latency = 1
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[1]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x10
(XEN) type = 3
(XEN) latency = 64
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
(XEN) states[2]:
(XEN) reg.space_id = 0x7f
(XEN) reg.bit_width = 0x1
(XEN) reg.bit_offset = 0x2
(XEN) reg.access_size = 0
(XEN) reg.address = 0x20
(XEN) type = 3
(XEN) latency = 96
(XEN) power = 0
(XEN) dp(@0x0000000000000000)
[ 25.967008] xen_pciback: backend is vpci
[ 25.990298] fuse init (API version 7.23)
[^[[36minfo^[[39;49m] Loading kernel module xen-acpi-processor.
[^[[36minfo^[[39;49m] Loading kernel module xen-pciback.
[^[[36minfo^[[39;49m] Loading kernel module fuse.
[....] Cleaning up temporary files...
/tmp^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Generating udev events for MD
arrays...^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Setting up LVM Volume Groups...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Activating lvm and md swap...[ 26.642119] Adding 2097148k swap
on /dev/mapper/vg_g2-lv_g2_swap. Priority:-1 extents:1 across:2097148k FS
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Checking file systems...fsck from util-linux 2.25.1
/dev/md0: clean, 334/131072 files, 74860/524224 blocks
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 26.948583] EXT4-fs (md0): mounting ext3 file system using the ext4
subsystem
[ 26.960457] EXT4-fs (md0): mounted filesystem with ordered data mode.
Opts: (null)
[....] Mounting local filesystems...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Activating swapfile swap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Cleaning up temporary files...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting kernel variables ...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[ 27.618140] Bridge firewalling registered
[ 27.625153] device eth0 entered promiscuous mode
[ 27.676402] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 27.678737] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[....] Configuring network interfaces...
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
[ 30.819293] tg3 0000:05:00.0 eth0: Link is up at 1000 Mbps, full duplex
[ 30.819310] tg3 0000:05:00.0 eth0: Flow control is on for TX and on
for RX
[ 30.819327] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 30.819350] br0: port 1(eth0) entered forwarding state
[ 30.819358] br0: port 1(eth0) entered forwarding state
[ 30.819377] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Cleaning up temporary files...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[^[[36minfo^[[39;49m] Setting console screen modes.
setterm: cannot (un)set powersave mode: Inappropriate ioctl for device
^[[9;30]^[[14;30][....] Setting up console font and
keymap...^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone.
[ 31.553893] ttyS0: LSR safety check engaged!
[ 31.556055] ttyS0: LSR safety check engaged!
Loading the saved-state of the serial devices...
/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A
[....] Setting up X socket directories... /tmp/.X11-unix
/tmp/.ICE-unix^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Setting sensors limits^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
INIT: Entering runlevel: 2
[^[[36minfo^[[39;49m] Using makefile-style concurrent boot in runlevel 2.
[....] Starting enhanced syslogd:
rsyslogd^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting MD monitoring service: mdadm
--monitor^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting ACPI services...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0c.
[ 32.114950] xen:xen_evtchn: Event-channel device installed
[....] Starting mouse interface server:
gpm^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting periodic command scheduler:
cron^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Loading cpufreq kernel modules...modprobe: ERROR: could not
insert 'cpufreq_userspace': No such device
modprobe: ERROR: could not insert 'cpufreq_stats': Invalid argument
modprobe: ERROR: could not insert 'cpufreq_powersave': No such device
modprobe: ERROR: could not insert 'cpufreq_conservative': No such device
^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0cdone (none).
[....] CPUFreq Utilities: Setting ondemand CPUFreq governor...disabled,
governor not available...^[[?25l^[[?1c^[7^[[1G[^[[32m ok
^[[39;49m^[8^[[?25h^[[?0cdone.
[....] Starting system message bus: dbus^[[?25l^[[?1c^[7^[[1G[^[[32m
ok ^[[39;49m^[8^[[?25h^[[?0c.
[....] Starting OpenBSD Secure Shell server:
sshd^[[?25l^[[?1c^[7^[[1G[^[[32m ok ^[[39;49m^[8^[[?25h^[[?0c.
Starting /usr/local/sbin/oxenstored...
Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenconsoled...
Starting QEMU as disk backend for dom0
^[[9;0]^[[14;0]
[ 36.119927] pciback 0000:02:00.0: seizing device
[ 36.119991] pciback 0000:02:00.0: enabling device (0004 -> 0007)
[ 36.120076] xen: registering gsi 16 triggering 0 polarity 1
[ 36.120085] Already setup the GSI :16
libxl: warning: libxl_pci.c:706:libxl__device_pci_assignable_add:
0000:02:00.0 not bound to a driver, will not be rebound.
Debian GNU/Linux jessie/sid g2 hvc0
g2 login: [ 45.856185] br0: port 1(eth0) entered forwarding state
(XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch input
to DOM0)
(XEN) IRQ information:
(XEN) IRQ: 0 affinity:01 vec:f0 type=IO-APIC-edge status=00000000
timer_interrupt()
(XEN) IRQ: 1 affinity:01 vec:30 type=IO-APIC-edge status=00000014
in-flight=0 domain-list=0: 1(---),
(XEN) IRQ: 3 affinity:01 vec:38 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 4 affinity:01 vec:f1 type=IO-APIC-edge status=00000000
ns16550_interrupt()
(XEN) IRQ: 5 affinity:01 vec:40 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 6 affinity:01 vec:48 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 6(---),
(XEN) IRQ: 7 affinity:01 vec:50 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 8 affinity:01 vec:58 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 8(---),
(XEN) IRQ: 9 affinity:01 vec:60 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 9(---),
(XEN) IRQ: 10 affinity:01 vec:68 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 11 affinity:01 vec:70 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 12 affinity:01 vec:78 type=IO-APIC-edge status=00000010
in-flight=0 domain-list=0: 12(---),
(XEN) IRQ: 13 affinity:3f vec:88 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 14 affinity:01 vec:90 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 15 affinity:01 vec:98 type=IO-APIC-edge status=00000002
mapped, unbound
(XEN) IRQ: 16 affinity:01 vec:a0 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 16(---),
(XEN) IRQ: 17 affinity:01 vec:a8 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 17(---),
(XEN) IRQ: 18 affinity:01 vec:b0 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 18(---),
(XEN) IRQ: 19 affinity:01 vec:b8 type=IO-APIC-level status=00000010
in-flight=0 domain-list=0: 19(---),
(XEN) IRQ: 24 affinity:3f vec:28 type=DMA_MSI status=00000000
iommu_page_fault()
(XEN) IRQ: 25 affinity:01 vec:c0 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:279(---),
(XEN) IRQ: 26 affinity:01 vec:c8 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:278(---),
(XEN) IRQ: 27 affinity:01 vec:d0 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:277(---),
(XEN) IRQ: 28 affinity:01 vec:d8 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:276(---),
(XEN) IRQ: 29 affinity:01 vec:21 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:275(---),
(XEN) IRQ: 30 affinity:01 vec:29 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:274(---),
(XEN) IRQ: 31 affinity:3f vec:31 type=PCI-MSI status=00000002
mapped, unbound
(XEN) IRQ: 32 affinity:3f vec:39 type=PCI-MSI status=00000002
mapped, unbound
(XEN) IRQ: 33 affinity:01 vec:49 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:271(---),
(XEN) IRQ: 34 affinity:01 vec:51 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:270(---),
(XEN) IRQ: 35 affinity:01 vec:59 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:269(---),
(XEN) IRQ: 36 affinity:01 vec:61 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:268(---),
(XEN) IRQ: 37 affinity:01 vec:69 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:267(---),
(XEN) IRQ: 38 affinity:01 vec:71 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:266(---),
(XEN) IRQ: 39 affinity:01 vec:79 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:265(---),
(XEN) IRQ: 40 affinity:01 vec:81 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:264(---),
(XEN) IRQ: 41 affinity:01 vec:89 type=PCI-MSI/-X status=00000010
in-flight=0 domain-list=0:263(---),
(XEN) IRQ: 42 affinity:01 vec:91 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:262(---),
(XEN) IRQ: 43 affinity:01 vec:99 type=PCI-MSI status=00000010
in-flight=0 domain-list=0:261(---),
(XEN) Direct vector information:
(XEN) 0x20 -> irq_move_cleanup_interrupt()
(XEN) 0xf2 -> cmci_interrupt()
(XEN) 0xf3 -> intel_thermal_interrupt()
(XEN) 0xf9 -> pmu_apic_interrupt()
(XEN) 0xfa -> apic_timer_interrupt()
(XEN) 0xfb -> call_function_interrupt()
(XEN) 0xfc -> event_check_interrupt()
(XEN) 0xfd -> invalidate_interrupt()
(XEN) 0xfe -> error_interrupt()
(XEN) 0xff -> spurious_interrupt()
(XEN) IO-APIC interrupt information:
(XEN) IRQ 0 Vec240:
(XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 1 Vec 48:
(XEN) Apic 0x00, Pin 1: vec=30 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 3 Vec 56:
(XEN) Apic 0x00, Pin 3: vec=38 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 4 Vec241:
(XEN) Apic 0x00, Pin 4: vec=f1 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 5 Vec 64:
(XEN) Apic 0x00, Pin 5: vec=40 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 6 Vec 72:
(XEN) Apic 0x00, Pin 6: vec=48 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 7 Vec 80:
(XEN) Apic 0x00, Pin 7: vec=50 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 8 Vec 88:
(XEN) Apic 0x00, Pin 8: vec=58 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 9 Vec 96:
(XEN) Apic 0x00, Pin 9: vec=60 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 10 Vec104:
(XEN) Apic 0x00, Pin 10: vec=68 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 11 Vec112:
(XEN) Apic 0x00, Pin 11: vec=70 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 12 Vec120:
(XEN) Apic 0x00, Pin 12: vec=78 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 13 Vec136:
(XEN) Apic 0x00, Pin 13: vec=88 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=1 dest_id:63
(XEN) IRQ 14 Vec144:
(XEN) Apic 0x00, Pin 14: vec=90 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 15 Vec152:
(XEN) Apic 0x00, Pin 15: vec=98 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:1
(XEN) IRQ 16 Vec160:
(XEN) Apic 0x00, Pin 16: vec=a0 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 17 Vec168:
(XEN) Apic 0x00, Pin 17: vec=a8 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 18 Vec176:
(XEN) Apic 0x00, Pin 18: vec=b0 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) IRQ 19 Vec184:
(XEN) Apic 0x00, Pin 19: vec=b8 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:1
(XEN) 'c' pressed -> printing ACPI Cx structures
(XEN) ==cpu0==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00005664] method[ FFH]
duration[4042540627]
(XEN) C2: type[C3] latency[064] usage[00000414] method[ FFH]
duration[447258888]
(XEN) C3: type[C3] latency[096] usage[00002366] method[ FFH]
duration[28183588810]
(XEN) *C0: usage[00008444] duration[26752178344]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[484210884] CC6[27943480555] CC7[0]
(XEN) ==cpu1==
(XEN) active state: C3
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00002007] method[ FFH]
duration[4316094103]
(XEN) C2: type[C3] latency[064] usage[00000179] method[ FFH]
duration[430291017]
(XEN) *C3: type[C3] latency[096] usage[00000785] method[ FFH]
duration[27543087914]
(XEN) C0: usage[00002971] duration[27136150937]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[498924899] CC6[26514562346] CC7[0]
(XEN) ==cpu2==
(XEN) active state: C3
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00001925] method[ FFH]
duration[3925569185]
(XEN) C2: type[C3] latency[064] usage[00000199] method[ FFH]
duration[341001922]
(XEN) *C3: type[C3] latency[096] usage[00000761] method[ FFH]
duration[27805737869]
(XEN) C0: usage[00002885] duration[27353411294]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[377373007] CC6[27202250545] CC7[0]
(XEN) ==cpu3==
(XEN) active state: C3
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00001880] method[ FFH]
duration[2679771477]
(XEN) C2: type[C3] latency[064] usage[00000229] method[ FFH]
duration[188146428]
(XEN) *C3: type[C3] latency[096] usage[00001031] method[ FFH]
duration[29498117979]
(XEN) C0: usage[00003140] duration[27059782098]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[254623106] CC6[28394085736] CC7[0]
(XEN) ==cpu4==
(XEN) active state: C3
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00001756] method[ FFH]
duration[2769254927]
(XEN) C2: type[C3] latency[064] usage[00000229] method[ FFH]
duration[301480682]
(XEN) *C3: type[C3] latency[096] usage[00000859] method[ FFH]
duration[28897863294]
(XEN) C0: usage[00002844] duration[27457318503]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[383731673] CC6[28101648533] CC7[0]
(XEN) ==cpu5==
(XEN) active state: C3
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00001570] method[ FFH]
duration[3273775119]
(XEN) C2: type[C3] latency[064] usage[00000265] method[ FFH]
duration[367342058]
(XEN) *C3: type[C3] latency[096] usage[00000719] method[ FFH]
duration[28591526276]
(XEN) C0: usage[00002554] duration[27193372517]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
(XEN) CC3[470602725] CC6[29075968196] CC7[0]
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-24 22:17 ` Steve Freitas
@ 2014-11-25 8:16 ` Jan Beulich
2014-11-25 9:38 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-25 8:16 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 24.11.14 at 23:17, <sflist@ihonk.com> wrote:
> I'm combining this action with your patch, see below. Please let me know
> if this was incorrect.
Thanks, that's perfectly fine.
> (XEN) 'c' pressed -> printing ACPI Cx structures
> (XEN) ==cpu0==
> (XEN) active state: C0
> (XEN) max_cstate: C7
> (XEN) states:
> (XEN) C1: type[C1] latency[001] usage[00005664] method[ FFH] duration[4042540627]
> (XEN) C2: type[C3] latency[064] usage[00000414] method[ FFH] duration[447258888]
> (XEN) C3: type[C3] latency[096] usage[00002366] method[ FFH] duration[28183588810]
> (XEN) *C0: usage[00008444] duration[26752178344]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
> (XEN) CC3[484210884] CC6[27943480555] CC7[0]
Interesting, so other than for me (perhaps due to other patches
I have in my tree) the change resulted in C states now being used
again despite mwait-idle=0, which is good. Question now is - with
this being the case, did the hangs re-occur?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-25 8:16 ` Jan Beulich
@ 2014-11-25 9:38 ` Steve Freitas
2014-11-25 11:00 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-25 9:38 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, xen-devel
On 11/25/2014 12:16 AM, Jan Beulich wrote:
>> (XEN) 'c' pressed -> printing ACPI Cx structures
>> (XEN) ==cpu0==
>> (XEN) active state: C0
>> (XEN) max_cstate: C7
>> (XEN) states:
>> (XEN) C1: type[C1] latency[001] usage[00005664] method[ FFH] duration[4042540627]
>> (XEN) C2: type[C3] latency[064] usage[00000414] method[ FFH] duration[447258888]
>> (XEN) C3: type[C3] latency[096] usage[00002366] method[ FFH] duration[28183588810]
>> (XEN) *C0: usage[00008444] duration[26752178344]
>> (XEN) max=0 pwr=0 urg=0 nxt=0
>> (XEN) PC2[0] PC3[112428588] PC6[21869019218] PC7[0]
>> (XEN) CC3[484210884] CC6[27943480555] CC7[0]
> Interesting, so other than for me (perhaps due to other patches
> I have in my tree) the change resulted in C states now being used
> again despite mwait-idle=0, which is good. Question now is - with
> this being the case, did the hangs re-occur?
Unfortunately they did. (Happened unusually quick this time, though I
doubt the statistical significance.) Not sure what the desirable output
is, so I did a couple of 'a' and 'd' requests, capped off by a 'c':
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82d08012c9a2>] _spin_unlock_irq+0x30/0x31
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300a943d000 rcx: 0000000000000000
(XEN) rdx: ffff82d0802e0000 rsi: 0000000000000008 rdi: ffff82d080329308
(XEN) rbp: ffff82d0802e7ec8 rsp: ffff82d0802e7e40 r8: ffff82d080329320
(XEN) r9: 0000000000000000 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: 000001227e5280e4 r13: ffff8300a943d000 r14: ffff82d080329308
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1b57c000 cr2: 000007fefca62000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff82d0802e7e40:
(XEN) ffff82d080128cb5 ffff82d080329300 ffff82d080329320 00000000002e7e78
(XEN) ffff82d080329300 ffff82d0801b977e ffff8300a943d000 fffff88002f36d70
(XEN) ffff8300a943d000 0000000001c9c380 ffff82d0801e5600 ffff82d0802e7f08
(XEN) ffff82d080300080 ffff82d080300080 ffffffffffffffff ffff82d0802e0000
(XEN) fffffa8004fdff80 ffff82d0802e7ef8 ffff82d08012bfa3 ffff8300a943d000
(XEN) fffff88002f36d70 0000018507f7ef25 000000000000000f ffff82d0802e7f08
(XEN) ffff82d08012bffb 000000000000000f ffff82d0801e849a fffffa8004fdff80
(XEN) 000000000000000f 0000018507f7ef25 fffff88002f36d70 000000000000000f
(XEN) fffff88002f2c180 fffff88002f36d70 fffff88002f2c2a0 fffff880043cb960
(XEN) fffff88002f2c2a0 0000000000000002 fffff88002f2c1c0 0000000000000400
(XEN) 0000000000000000 fffff88002f36eb0 0000beef0000beef fffff8000293e20c
(XEN) 000000bf0000beef 0000000000000046 fffff88002f36c20 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) 0000000000000000 ffff8300a943d000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08012c9a2>] _spin_unlock_irq+0x30/0x31
(XEN) [<ffff82d08012bfa3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012bffb>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e849a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v3): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000293e20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: fffff880043cb960 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 0000018507f7ef25 r14: 000000000000000f
(XEN) r15: fffffa8004fdff80 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 00000000536d9000 cr2: 000007fefca62000
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v4): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000293e20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fa2180 rcx: fffff88002fa21c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002faceb0
(XEN) rbp: 000000000000000f rsp: fffff88002facc20 r8: fffff88002fa22a0
(XEN) r9: fffff88002fcaca0 r10: fffff88002fa22a0 r11: fffff88002facd70
(XEN) r12: fffff88002facd70 r13: 0000000000000000 r14: 000000000000000f
(XEN) r15: fffff88002fa6fc0 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fffffaf478
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v5): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000293e20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000001526da65c1e r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 0000018507f90b28 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fefabb2654
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 guest state (d1v1): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: 0010:[<fffff80002e052bf>]
(XEN) RFLAGS: 0000000000000006 CONTEXT: hvm guest
(XEN) rax: 000002be8e13da4d rbx: 000002be8e13e51b rcx: 000000000000ffff
(XEN) rdx: 000002be00000000 rsi: 0000000000000000 rdi: 00000000000000a5
(XEN) rbp: 000000000000bad2 rsp: fffff88007bd8e80 r8: fffff88007bd8ee0
(XEN) r9: 0000000000000000 r10: fffff88007bd8f30 r11: 000000034291bff0
(XEN) r12: fffff88007bd92b8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 000000005c4ec000 cr2: 000000007730853f
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v0): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff8000293e20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a00e80 rcx: fffff80002a00ec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80003cfce70
(XEN) rbp: 000000000000000f rsp: fffff80003cfcbe0 r8: fffff80002a00fa0
(XEN) r9: 0000000000000000 r10: fffff80002a00fa0 r11: fffff80003cfcd30
(XEN) r12: fffff80003cfcd30 r13: 0000018507f832b0 r14: 000000000000000f
(XEN) r15: 0000000000020181 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 00000000b3b80000 cr2: 000007fefcf91444
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v2): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000293e20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: fffff88006775960 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000018507fa203d r14: 000000000000000f
(XEN) r15: fffffa800451a130 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 000000009b3e8000 cr2: fffff98006d0b008
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 9736us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 99892us timer=ffff8300bf52c060
cb=vcpu_singleshot_timer_fn(ffff8300bf52c000)
(XEN) ex= 10359us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 101576185us timer=ffff82d080325260
cb=plt_overflow(0000000000000000)
(XEN) ex= 7904630us timer=ffff82d080327560
cb=mce_work_fn(0000000000000000)
(XEN) ex= 100351us timer=ffff82d080325300
cb=time_calibration(0000000000000000)
(XEN) ex= 19736us timer=ffff82d0803295e0
cb=do_dbs_timer(ffff82d080329620)
(XEN) CPU01:
(XEN) ex= 9996us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 19736us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 30028us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 177us timer=ffff830c3dc79858
cb=csched_tick(0000000000000002)
(XEN) ex= 103892us timer=ffff8300bf52e060
cb=vcpu_singleshot_timer_fn(ffff8300bf52e000)
(XEN) ex= 565us timer=ffff8300bf52d060
cb=vcpu_singleshot_timer_fn(ffff8300bf52d000)
(XEN) ex= 103892us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 80621762913us timer=ffff830c17cc2290
cb=rtc_alarm_cb(ffff830c17cc21f0)
(XEN) ex= 19736us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 1036us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 9518us timer=ffff830c3dcac0e8
cb=csched_tick(0000000000000003)
(XEN) ex= 19736us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 30053us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 10042us timer=ffff830c3dcac968
cb=csched_tick(0000000000000004)
(XEN) ex= 19736us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) ex= 103892us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 30059us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU05:
(XEN) ex= 10044us timer=ffff830c3dc8d208
cb=csched_tick(0000000000000005)
(XEN) ex= 19736us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 74855186us timer=ffff830c17cc25d0
cb=pmt_timer_callback(ffff830c17cc25a8)
(XEN) ex= 23892us timer=ffff8300bf52f060
cb=vcpu_singleshot_timer_fn(ffff8300bf52f000)
(XEN) ex= 30068us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU0 guest state (d1v0): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: 0010:[<fffff8000293e20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff80002a00e80 rcx: fffff80002a00ec0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff80003cfce70
(XEN) rbp: 000000000000000f rsp: fffff80003cfcbe0 r8: fffff80002a00fa0
(XEN) r9: 0000000000000000 r10: fffff80002a00fa0 r11: fffff80003cfcd30
(XEN) r12: fffff80003cfcd30 r13: 0000018507f832b0 r14: 000000000000000f
(XEN) r15: 0000000000020181 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 00000000b3b80000 cr2: 000007fefcf91444
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU1 guest state (d1v5): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000293e20e>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002fd8180 rcx: fffff88002fd81c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002fe2eb0
(XEN) rbp: 000000000000000f rsp: fffff88002fe2c20 r8: fffff88002fd82a0
(XEN) r9: 000001526da65c1e r10: fffff88002fd82a0 r11: fffff88002fe2d70
(XEN) r12: fffff88002fe2d70 r13: 0000018507f90b28 r14: 000000000000000f
(XEN) r15: 0000000000000001 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 0000000000187000 cr2: 000007fefabb2654
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU2 host state: ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: e008:[<ffff82d0801dc4ca>] vmx_do_resume+0x1/0x154
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: ffff8300a943d000 rbx: ffff8300a943d000 rcx: 0000000000000002
(XEN) rdx: ffff830c3dcb0000 rsi: 0000000000000004 rdi: ffff8300a943d000
(XEN) rbp: ffff830c3dcb7e38 rsp: ffff830c3dcb7e28 r8: ffff830c3dcb80a0
(XEN) r9: 0000000000000000 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: 00000123d6398bd9 r13: ffff8300a943d000 r14: ffff830c3dcb8088
(XEN) r15: 0000000001c9c380 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1b57c000 cr2: 000007fefca62000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dcb7e28:
(XEN) ffff830c3dcb7e38 ffff82d080165e89 ffff830c3dcb7ec8 ffff82d080128cef
(XEN) ffff82d080329300 ffff830c3dcb80a0 0000000200cb7e78 ffff830c3dcb8080
(XEN) ffff82d0801b977e ffff8300a943d000 fffff88002f36d70 ffff8300a943d000
(XEN) 0000000001c9c380 ffff82d0801e5600 ffff830c3dcb7f08 ffff82d080300180
(XEN) ffff82d080300080 ffffffffffffffff ffff830c3dcb0000 fffffa8004fdff80
(XEN) ffff830c3dcb7ef8 ffff82d08012bfa3 ffff8300a943d000 fffff88002f36d70
(XEN) 0000018507f7ef25 000000000000000f ffff830c3dcb7f08 ffff82d08012bffb
(XEN) 000000000000000f ffff82d0801e849a fffffa8004fdff80 000000000000000f
(XEN) 0000018507f7ef25 fffff88002f36d70 000000000000000f fffff88002f2c180
(XEN) fffff88002f36d70 fffff88002f2c2a0 fffff880043cb960 fffff88002f2c2a0
(XEN) 0000000000000002 fffff88002f2c1c0 0000000000000400 0000000000000000
(XEN) fffff88002f36eb0 0000beef0000beef fffff8000293e20c 000000bf0000beef
(XEN) 0000000000000046 fffff88002f36c20 000000000000beef 000000000000beef
(XEN) 000000000000beef 000000000000beef 000000000000beef 0000000000000002
(XEN) ffff8300a943d000 0000003bbd98ed80 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801dc4ca>] vmx_do_resume+0x1/0x154
(XEN) [<ffff82d080128cef>] schedule+0x1b5/0x612
(XEN) [<ffff82d08012bfa3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012bffb>] do_softirq+0x13/0x15
(XEN) [<ffff82d0801e849a>] vmx_asm_do_vmentry+0x2a/0x45
(XEN)
(XEN) *** Dumping CPU2 guest state (d1v3): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: 0010:[<fffff8000293e20c>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002f2c180 rcx: fffff88002f2c1c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002f36eb0
(XEN) rbp: 000000000000000f rsp: fffff88002f36c20 r8: fffff88002f2c2a0
(XEN) r9: fffff880043cb960 r10: fffff88002f2c2a0 r11: fffff88002f36d70
(XEN) r12: fffff88002f36d70 r13: 0000018507f7ef25 r14: 000000000000000f
(XEN) r15: fffffa8004fdff80 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 00000000536d9000 cr2: 000007fefca62000
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e008:[<ffff82d08010009f>] __bitmap_empty+0x8/0x96
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff82d080313e00 rcx: 0000000000000006
(XEN) rdx: 0000000000000004 rsi: 0000000000000006 rdi: ffff82d080313e00
(XEN) rbp: ffff830c3dca7e58 rsp: ffff830c3dca7e58 r8: 0000000000000045
(XEN) r9: 000000000000006d r10: 0000000000000000 r11: fffff88002facd70
(XEN) r12: 0000000000000100 r13: 0000000000000000 r14: ffff830c3dca0000
(XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1815b000 cr2: 000007fffffaf478
(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830c3dca7e58:
(XEN) ffff830c3dca7e88 ffff82d080186e22 ffff82d0801b977e ffff830c3dca7e98
(XEN) ffff82d080300080 ffffffffffffffff ffff830c3dca7ec8 ffff82d080186ea2
(XEN) 0000000000000037 0000000000000000 0000000000000000 0000000000000000
(XEN) ffff830c3dca0000 ffff82d080300200 ffff830c3dca7ef8 ffff82d08012bfa3
(XEN) ffff8300bf52c000 ffff8300a943c000 0000000000000001 ffff830c3dcaa088
(XEN) ffff830c3dca7f08 ffff82d08012bffb ffff830c3dca7de8 ffff82d08022fd51
(XEN) 0000000000000000 00000000ffffffed 0000000000000000 ffff880bce434000
(XEN) 0000000000000004 ffffffff818e1800 0000000000000246 ffff880aef7e7aa8
(XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810013aa
(XEN) 0000000000000001 00000000deadbeef 00000000deadbeef 0002010000000000
(XEN) ffffffff810013aa 000000000000e033 0000000000000246 ffff880bce437ec8
(XEN) 000000000000e02b 000000000000beef 000000000000beef 000000000000beef
(XEN) 000000000000beef 0000000000000003 ffff8300bf52c000 0000003bbd980d80
(XEN) 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08010009f>] __bitmap_empty+0x8/0x96
(XEN) [<ffff82d080186e22>] flush_area_mask+0x119/0x134
(XEN) [<ffff82d080186ea2>] new_tlbflush_clock_period+0x65/0x81
(XEN) [<ffff82d08012bfa3>] __do_softirq+0x81/0x8c
(XEN) [<ffff82d08012bffb>] do_softirq+0x13/0x15
(XEN) [<ffff82d08022fd51>] process_softirqs+0x21/0x30
(XEN)
(XEN) *** Dumping CPU3 guest state (d0v4): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff810013aa>]
(XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest
(XEN) rax: 0000000000000000 rbx: ffffffff818e1800 rcx: ffffffff810013aa
(XEN) rdx: 0000000000000001 rsi: 00000000deadbeef rdi: 00000000deadbeef
(XEN) rbp: 0000000000000004 rsp: ffff880bce437ec8 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: ffff880aef7e7aa8 r11: 0000000000000246
(XEN) r12: ffff880bce434000 r13: 0000000000000000 r14: 00000000ffffffed
(XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000c1815b000 cr2: 00007fd8239b7000
(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffff880bce437ec8:
(XEN) 0000000000000000 ffffffff81855060 ffffffff81009e0c ffffffff8101c849
(XEN) ffffffff818e1800 0000000000000000 ffffffff810a5d80 ffff880bce434000
(XEN) 0000000000000004 ffff880bce434000 ffff880bce437fd8 c789d0ce26296124
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffffffffffffffff 0000000000000000 0000000000000010
(XEN) 0000000000000202 ffff880bce437f58 0000000000000018
(XEN)
(XEN) *** Dumping CPU4 guest state (d1v1): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0010:[<fffff80002e052c1>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 000002c273235b8e rbx: 000002c273236218 rcx: 000000000000ffff
(XEN) rdx: 000002c200000000 rsi: 0000000000000000 rdi: 0000000000000089
(XEN) rbp: 000000000000a191 rsp: fffff88007bd8e80 r8: fffff88007bd8ee0
(XEN) r9: 0000000000000000 r10: fffff88007bd8f30 r11: 000000034291bff0
(XEN) r12: fffff88007bd92b8 r13: 0000000000001000 r14: 0000000000000000
(XEN) r15: 0000000000000058 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 000000005c4ec000 cr2: 000000007730853f
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
(XEN)
(XEN) *** Dumping CPU5 guest state (d1v2): ***
(XEN) ----[ Xen-4.5.0-rc x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0010:[<fffff8000293e214>]
(XEN) RFLAGS: 0000000000000046 CONTEXT: hvm guest
(XEN) rax: 0000000000000002 rbx: fffff88002eb6180 rcx: fffff88002eb61c0
(XEN) rdx: 0000000000000400 rsi: 0000000000000000 rdi: fffff88002ec0eb0
(XEN) rbp: 000000000000000f rsp: fffff88002ec0c20 r8: fffff88002eb62a0
(XEN) r9: fffff88006775960 r10: fffff88002eb62a0 r11: fffff88002ec0d70
(XEN) r12: fffff88002ec0d70 r13: 0000018507fa203d r14: 000000000000000f
(XEN) r15: fffffa800451a130 cr0: 0000000080050031 cr4: 00000000000006f8
(XEN) cr3: 000000009b3e8000 cr2: fffff98006d0b008
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0000 cs: 0010
(XEN)
(XEN) Dumping timer queues:
(XEN) CPU00:
(XEN) ex= 3238us timer=ffff830c3dcc2d08
cb=csched_tick(0000000000000000)
(XEN) ex= 3238us timer=ffff82d0803295e0
cb=do_dbs_timer(ffff82d080329620)
(XEN) ex= 23022us timer=ffff8300bf52e060
cb=vcpu_singleshot_timer_fn(ffff8300bf52e000)
(XEN) ex= 984001us timer=ffff82d080325300
cb=time_calibration(0000000000000000)
(XEN) ex= 97459687us timer=ffff82d080325260
cb=plt_overflow(0000000000000000)
(XEN) ex= 3788132us timer=ffff82d080327560
cb=mce_work_fn(0000000000000000)
(XEN) ex= 27212us timer=ffff830c3dc4b1e0
cb=csched_acct(ffff830c3dc4b1c0)
(XEN) ex= 80617646415us timer=ffff830c17cc2290
cb=rtc_alarm_cb(ffff830c17cc21f0)
(XEN) CPU01:
(XEN) ex= 3129us timer=ffff830c3dc55f28
cb=csched_tick(0000000000000001)
(XEN) ex= 3238us timer=ffff830c3dc7a360
cb=do_dbs_timer(ffff830c3dc7a3a0)
(XEN) ex= 30029us timer=ffff830c3dc7a0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU02:
(XEN) ex= 3238us timer=ffff830c3dcb8360
cb=do_dbs_timer(ffff830c3dcb83a0)
(XEN) ex= 3359us timer=ffff830c3dc79858
cb=csched_tick(0000000000000002)
(XEN) ex= 945174us timer=ffff8300bf52c060
cb=vcpu_singleshot_timer_fn(ffff8300bf52c000)
(XEN) ex= 771394us timer=ffff8300bf798060
cb=vcpu_singleshot_timer_fn(ffff8300bf798000)
(XEN) ex= 30036us timer=ffff830c3dcb80a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU03:
(XEN) ex= 3238us timer=ffff830c3dcaa360
cb=do_dbs_timer(ffff830c3dcaa3a0)
(XEN) ex= 6099us timer=ffff830c3dcac0e8
cb=csched_tick(0000000000000003)
(XEN) ex= 3987394us timer=ffff8300bf52d060
cb=vcpu_singleshot_timer_fn(ffff8300bf52d000)
(XEN) ex= 15394us timer=ffff8300bf52f060
cb=vcpu_singleshot_timer_fn(ffff8300bf52f000)
(XEN) ex= 30047us timer=ffff830c3dcaa0a0
cb=s_timer_fn(0000000000000000)
(XEN) CPU04:
(XEN) ex= 296us timer=ffff8300bf2fb060
cb=vcpu_singleshot_timer_fn(ffff8300bf2fb000)
(XEN) ex= 1060us timer=ffff830c3dc9c0a0
cb=s_timer_fn(0000000000000000)
(XEN) ex= 6764us timer=ffff830c3dcac968
cb=csched_tick(0000000000000004)
(XEN) ex= 3238us timer=ffff830c3dc9c360
cb=do_dbs_timer(ffff830c3dc9c3a0)
(XEN) CPU05:
(XEN) ex= 3238us timer=ffff830c3dc8e360
cb=do_dbs_timer(ffff830c3dc8e3a0)
(XEN) ex= 7336us timer=ffff830c3dc8d208
cb=csched_tick(0000000000000005)
(XEN) ex= 70738688us timer=ffff830c17cc25d0
cb=pmt_timer_callback(ffff830c17cc25a8)
(XEN) ex= 30070us timer=ffff830c3dc8e0a0
cb=s_timer_fn(0000000000000000)
(XEN) 'c' pressed -> printing ACPI Cx structures
(XEN) ==cpu0==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00879961] method[ FFH]
duration[258917692906]
(XEN) C2: type[C3] latency[064] usage[00219972] method[ FFH]
duration[185157317681]
(XEN) C3: type[C3] latency[096] usage[00021194] method[ FFH]
duration[130347582819]
(XEN) *C0: usage[01121127] duration[742037344879]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[178118468514] CC6[120838993791] CC7[0]
(XEN) ==cpu1==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00732768] method[ FFH]
duration[244601938735]
(XEN) C2: type[C3] latency[064] usage[00241040] method[ FFH]
duration[205647145982]
(XEN) C3: type[C3] latency[096] usage[00017258] method[ FFH]
duration[164619828246]
(XEN) *C0: usage[00991066] duration[701591066797]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[197708830338] CC6[153278778335] CC7[0]
(XEN) ==cpu2==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00736573] method[ FFH]
duration[239398385639]
(XEN) C2: type[C3] latency[064] usage[00243735] method[ FFH]
duration[208117053326]
(XEN) C3: type[C3] latency[096] usage[00018811] method[ FFH]
duration[163124076095]
(XEN) *C0: usage[00999119] duration[705820535124]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[201124993103] CC6[147956371559] CC7[0]
(XEN) ==cpu3==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00737249] method[ FFH]
duration[241046283679]
(XEN) C2: type[C3] latency[064] usage[00244020] method[ FFH]
duration[210189165328]
(XEN) C3: type[C3] latency[096] usage[00016474] method[ FFH]
duration[164857248289]
(XEN) *C0: usage[00997743] duration[700367394408]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[202741688061] CC6[152613177986] CC7[0]
(XEN) ==cpu4==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00784949] method[ FFH]
duration[238150508853]
(XEN) C2: type[C3] latency[064] usage[00242440] method[ FFH]
duration[208233744834]
(XEN) C3: type[C3] latency[096] usage[00017019] method[ FFH]
duration[166844569919]
(XEN) *C0: usage[01044408] duration[703231307727]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[200748358163] CC6[153896919179] CC7[0]
(XEN) ==cpu5==
(XEN) active state: C0
(XEN) max_cstate: C7
(XEN) states:
(XEN) C1: type[C1] latency[001] usage[00881857] method[ FFH]
duration[277043571093]
(XEN) C2: type[C3] latency[064] usage[00201963] method[ FFH]
duration[172166602574]
(XEN) C3: type[C3] latency[096] usage[00018587] method[ FFH]
duration[161929908741]
(XEN) *C0: usage[01102407] duration[705320088716]
(XEN) max=0 pwr=0 urg=2d nxt=0
(XEN) PC2[0] PC3[36377080311] PC6[84924866353] PC7[0]
(XEN) CC3[166777643669] CC6[150006253526] CC7[0]
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-25 9:38 ` Steve Freitas
@ 2014-11-25 11:00 ` Jan Beulich
2014-11-27 5:29 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-25 11:00 UTC (permalink / raw)
To: Steve Freitas; +Cc: Don Slutz, xen-devel
>>> On 25.11.14 at 10:38, <sflist@ihonk.com> wrote:
> On 11/25/2014 12:16 AM, Jan Beulich wrote:
>> Interesting, so other than for me (perhaps due to other patches
>> I have in my tree) the change resulted in C states now being used
>> again despite mwait-idle=0, which is good. Question now is - with
>> this being the case, did the hangs re-occur?
>
> Unfortunately they did. (Happened unusually quick this time, though I
> doubt the statistical significance.) Not sure what the desirable output
> is, so I did a couple of 'a' and 'd' requests, capped off by a 'c':
Okay, so it's not really the mwait-idle driver causing the regression,
but it is C-state related. Hence we're now down to seeing whether all
or just the deeper C states are affected, i.e. I now need to ask you
to play with "max_cstate=". For that you'll have to remember that the
option's effect differs between the ACPI and the MWAIT idle drivers.
In the spirit of bisection I'd suggest using "max_cstate=2" first no
matter which of the two scenarios you pick. If that still hangs,
"max_cstate=1" obviously is the only other thing to try. Should that
not hang (and you left out "mwait-idle=0"), trying "max_cstate=3"
in that same scenario would be the other case to check.
No need for 'd' and 'a' output for the time being, but 'c' output would
be much appreciated for all cases where you observe hangs.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-25 11:00 ` Jan Beulich
@ 2014-11-27 5:29 ` Steve Freitas
2014-11-27 9:27 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-27 5:29 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, xen-devel
On 11/25/2014 03:00 AM, Jan Beulich wrote:
> Okay, so it's not really the mwait-idle driver causing the regression,
> but it is C-state related. Hence we're now down to seeing whether all
> or just the deeper C states are affected, i.e. I now need to ask you
> to play with "max_cstate=". For that you'll have to remember that the
> option's effect differs between the ACPI and the MWAIT idle drivers.
> In the spirit of bisection I'd suggest using "max_cstate=2" first no
> matter which of the two scenarios you pick. If that still hangs,
> "max_cstate=1" obviously is the only other thing to try. Should that
> not hang (and you left out "mwait-idle=0"), trying "max_cstate=3"
> in that same scenario would be the other case to check.
>
> No need for 'd' and 'a' output for the time being, but 'c' output would
> be much appreciated for all cases where you observe hangs.
>
Okay, working through that now. I tried max_cstate=2 and got no hangs,
whether with or without mwait-idle=0. However, I was puzzled by this:
(XEN) 'c' pressed -> printing ACPI Cx structures
(XEN) ==cpu0==
(XEN) active state: C0
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
duration[1190961948551]
(XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
duration[2015393965907]
(XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
duration[30527997858148]
(XEN) *C0: usage[73351700] duration[9974627547595]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[28794734145697] CC6[0] CC7[0]
(XEN) ==cpu1==
(XEN) active state: C3
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[10699950] method[ FFH]
duration[1141422044112]
(XEN) C2: type[C1] latency[010] usage[06382904] method[ FFH]
duration[1329739264322]
(XEN) *C3: type[C2] latency[020] usage[44630764] method[ FFH]
duration[31676618425954]
(XEN) C0: usage[61713618] duration[9561201640320]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[30066495105056] CC6[0] CC7[0]
(XEN) ==cpu2==
(XEN) active state: C3
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[10829791] method[ FFH]
duration[1145244102917]
(XEN) C2: type[C1] latency[010] usage[06392468] method[ FFH]
duration[1330830147023]
(XEN) *C3: type[C2] latency[020] usage[44705668] method[ FFH]
duration[31741190985486]
(XEN) C0: usage[61927927] duration[9491716216846]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[30117696095715] CC6[0] CC7[0]
(XEN) ==cpu3==
(XEN) active state: C3
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[10692336] method[ FFH]
duration[1144876437514]
(XEN) C2: type[C1] latency[010] usage[06394051] method[ FFH]
duration[1333961503379]
(XEN) *C3: type[C2] latency[020] usage[44744178] method[ FFH]
duration[31803488799434]
(XEN) C0: usage[61830565] duration[9426654792908]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[30191557548300] CC6[0] CC7[0]
(XEN) ==cpu4==
(XEN) active state: C3
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[10746634] method[ FFH]
duration[1144044534459]
(XEN) C2: type[C1] latency[010] usage[06444054] method[ FFH]
duration[1340637424913]
(XEN) *C3: type[C2] latency[020] usage[44690901] method[ FFH]
duration[31663207165902]
(XEN) C0: usage[61881589] duration[9561092487876]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[30049235012919] CC6[0] CC7[0]
(XEN) ==cpu5==
(XEN) active state: C3
(XEN) max_cstate: C2
(XEN) states:
(XEN) C1: type[C1] latency[003] usage[10694684] method[ FFH]
duration[1140625901110]
(XEN) C2: type[C1] latency[010] usage[06461563] method[ FFH]
duration[1342115502967]
(XEN) *C3: type[C2] latency[020] usage[44834522] method[ FFH]
duration[31719560664023]
(XEN) C0: usage[61990769] duration[9506679619986]
(XEN) max=0 pwr=0 urg=0 nxt=0
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
Why would some of the cores be in C3 even though they list max_cstate as C2?
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-27 5:29 ` Steve Freitas
@ 2014-11-27 9:27 ` Jan Beulich
2014-11-28 8:24 ` Steve Freitas
` (2 more replies)
0 siblings, 3 replies; 39+ messages in thread
From: Jan Beulich @ 2014-11-27 9:27 UTC (permalink / raw)
To: Steve Freitas, Donald D Dugger, Jun Nakajima; +Cc: Don Slutz, xen-devel
>>> On 27.11.14 at 06:29, <sflist@ihonk.com> wrote:
> On 11/25/2014 03:00 AM, Jan Beulich wrote:
>> Okay, so it's not really the mwait-idle driver causing the regression,
>> but it is C-state related. Hence we're now down to seeing whether all
>> or just the deeper C states are affected, i.e. I now need to ask you
>> to play with "max_cstate=". For that you'll have to remember that the
>> option's effect differs between the ACPI and the MWAIT idle drivers.
>> In the spirit of bisection I'd suggest using "max_cstate=2" first no
>> matter which of the two scenarios you pick. If that still hangs,
>> "max_cstate=1" obviously is the only other thing to try. Should that
>> not hang (and you left out "mwait-idle=0"), trying "max_cstate=3"
>> in that same scenario would be the other case to check.
>>
>> No need for 'd' and 'a' output for the time being, but 'c' output would
>> be much appreciated for all cases where you observe hangs.
>>
>
> Okay, working through that now. I tried max_cstate=2 and got no hangs,
> whether with or without mwait-idle=0. However, I was puzzled by this:
>
> (XEN) 'c' pressed -> printing ACPI Cx structures
> (XEN) ==cpu0==
> (XEN) active state: C0
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
> duration[1190961948551]
> (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
> duration[2015393965907]
> (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
> duration[30527997858148]
> (XEN) *C0: usage[73351700] duration[9974627547595]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[28794734145697] CC6[0] CC7[0]
> (XEN) ==cpu1==
> (XEN) active state: C3
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[10699950] method[ FFH]
> duration[1141422044112]
> (XEN) C2: type[C1] latency[010] usage[06382904] method[ FFH]
> duration[1329739264322]
> (XEN) *C3: type[C2] latency[020] usage[44630764] method[ FFH]
> duration[31676618425954]
> (XEN) C0: usage[61713618] duration[9561201640320]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[30066495105056] CC6[0] CC7[0]
>[...]
>
> Why would some of the cores be in C3 even though they list max_cstate as C2?
This was precisely the reason why I told you that the numbering
differs (and is confusing and has nothing to do with actual C state
numbers): What max_cstate refers to in the mwait-idle driver is
what above is listed as type[Cx], i.e. the state at index 1 is C1, at
2 we've got C1E, and at 3 we've got C2. And those still aren't in
line with the numbering the CPU documentation uses, it's rather
kind of meant to refer to the ACPI numbering (but probably also
not fully matching up).
So max_cstate=2 working suggests a problem with what the CPU
calls C6, which presumably isn't all that surprising considering the
many errata (BD35, BD38, BD40, BD59, BD87, and BD104). Not
sure how to proceed from here - I suppose you already made
sure you run with the latest available BIOS. And with 6 errata
documented it's not all that unlikely that there's a 7th one with
MONITOR/MWAIT behavior. The commit you bisected to (and
which you had verified to be the culprit by just forcing
arch_skip_send_event_check() to always return false) could be
reasonably assumed to be broken only when MWAIT use for all
C states didn't work.
Don, Jun - is there anything known but not yet publicly
documented for Family 6 Model 44 Xeons?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-27 9:27 ` Jan Beulich
@ 2014-11-28 8:24 ` Steve Freitas
2014-11-28 8:50 ` Jan Beulich
2014-12-03 17:14 ` Dugger, Donald D
2015-02-27 17:25 ` Dugger, Donald D
2 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2014-11-28 8:24 UTC (permalink / raw)
To: Jan Beulich, Donald D Dugger, Jun Nakajima; +Cc: Don Slutz, xen-devel
On 11/27/2014 01:27 AM, Jan Beulich wrote:
> This was precisely the reason why I told you that the numbering
> differs (and is confusing and has nothing to do with actual C state
> numbers): What max_cstate refers to in the mwait-idle driver is
> what above is listed as type[Cx], i.e. the state at index 1 is C1, at
> 2 we've got C1E, and at 3 we've got C2. And those still aren't in
> line with the numbering the CPU documentation uses, it's rather
> kind of meant to refer to the ACPI numbering (but probably also
> not fully matching up).
Ah, thanks for the explanation.
> So max_cstate=2 working suggests a problem with what the CPU
> calls C6, which presumably isn't all that surprising considering the
> many errata (BD35, BD38, BD40, BD59, BD87, and BD104). Not
> sure how to proceed from here - I suppose you already made
> sure you run with the latest available BIOS.
Yes, latest available BIOS.
> And with 6 errata
> documented it's not all that unlikely that there's a 7th one with
> MONITOR/MWAIT behavior. The commit you bisected to (and
> which you had verified to be the culprit by just forcing
> arch_skip_send_event_check() to always return false) could be
> reasonably assumed to be broken only when MWAIT use for all
> C states didn't work.
Now I did get a hang with max_cstate=3 and mwait-idle=0. May I assume
that mwait-idle=0 means that ACPI is responsible for the throttling?
Thanks again for all your help!
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-28 8:24 ` Steve Freitas
@ 2014-11-28 8:50 ` Jan Beulich
2014-11-28 9:44 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2014-11-28 8:50 UTC (permalink / raw)
To: Jun Nakajima; +Cc: Steve Freitas, Don Slutz, Donald D Dugger, xen-devel
>>> On 28.11.14 at 09:24, <sflist@ihonk.com> wrote:
>> And with 6 errata
>> documented it's not all that unlikely that there's a 7th one with
>> MONITOR/MWAIT behavior. The commit you bisected to (and
>> which you had verified to be the culprit by just forcing
>> arch_skip_send_event_check() to always return false) could be
>> reasonably assumed to be broken only when MWAIT use for all
>> C states didn't work.
>
> Now I did get a hang with max_cstate=3 and mwait-idle=0.
According to the data you provided earlier, max_cstate=3 is
identical to not using that option at all when you also use
mwait-idle=0. It would make a difference only when not using
that latter option (and I specifically pointed this out in earlier
replies).
> May I assume
> that mwait-idle=0 means that ACPI is responsible for the throttling?
ACPI is providing the data to do C state management in that case,
but it's still Xen doing things.
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-28 8:50 ` Jan Beulich
@ 2014-11-28 9:44 ` Steve Freitas
0 siblings, 0 replies; 39+ messages in thread
From: Steve Freitas @ 2014-11-28 9:44 UTC (permalink / raw)
To: Jan Beulich; +Cc: Don Slutz, Donald D Dugger, Jun Nakajima, xen-devel
On Nov 28, 2014, at 00:50, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 28.11.14 at 09:24, <sflist@ihonk.com> wrote:
>>> And with 6 errata
>>> documented it's not all that unlikely that there's a 7th one with
>>> MONITOR/MWAIT behavior. The commit you bisected to (and
>>> which you had verified to be the culprit by just forcing
>>> arch_skip_send_event_check() to always return false) could be
>>> reasonably assumed to be broken only when MWAIT use for all
>>> C states didn't work.
>>
>> Now I did get a hang with max_cstate=3 and mwait-idle=0.
>
> According to the data you provided earlier, max_cstate=3 is
> identical to not using that option at all when you also use
> mwait-idle=0. It would make a difference only when not using
> that latter option (and I specifically pointed this out in earlier
> replies).
>
Apologies for asking you to repeat yourself. Most of this stuff is over my head -- the only time I was this far down the rabbit hole was on an 8051. Thanks for your patience. :-)
Steve
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-27 9:27 ` Jan Beulich
2014-11-28 8:24 ` Steve Freitas
@ 2014-12-03 17:14 ` Dugger, Donald D
2015-02-27 17:25 ` Dugger, Donald D
2 siblings, 0 replies; 39+ messages in thread
From: Dugger, Donald D @ 2014-12-03 17:14 UTC (permalink / raw)
To: Jan Beulich, Steve Freitas, Nakajima, Jun; +Cc: Don Slutz, xen-devel
Jan-
No, I have no knowledge of an unpublished errata related to C State issues.
--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786
-----Original Message-----
From: Jan Beulich [mailto:JBeulich@suse.com]
Sent: Thursday, November 27, 2014 2:28 AM
To: Steve Freitas; Dugger, Donald D; Nakajima, Jun
Cc: xen-devel@lists.xen.org; Don Slutz
Subject: Re: [Xen-devel] Regression, host crash with 4.5rc1
>>> On 27.11.14 at 06:29, <sflist@ihonk.com> wrote:
> On 11/25/2014 03:00 AM, Jan Beulich wrote:
>> Okay, so it's not really the mwait-idle driver causing the
>> regression, but it is C-state related. Hence we're now down to seeing
>> whether all or just the deeper C states are affected, i.e. I now need
>> to ask you to play with "max_cstate=". For that you'll have to
>> remember that the option's effect differs between the ACPI and the MWAIT idle drivers.
>> In the spirit of bisection I'd suggest using "max_cstate=2" first no
>> matter which of the two scenarios you pick. If that still hangs,
>> "max_cstate=1" obviously is the only other thing to try. Should that
>> not hang (and you left out "mwait-idle=0"), trying "max_cstate=3"
>> in that same scenario would be the other case to check.
>>
>> No need for 'd' and 'a' output for the time being, but 'c' output
>> would be much appreciated for all cases where you observe hangs.
>>
>
> Okay, working through that now. I tried max_cstate=2 and got no hangs,
> whether with or without mwait-idle=0. However, I was puzzled by this:
>
> (XEN) 'c' pressed -> printing ACPI Cx structures
> (XEN) ==cpu0==
> (XEN) active state: C0
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
> duration[1190961948551]
> (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
> duration[2015393965907]
> (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
> duration[30527997858148]
> (XEN) *C0: usage[73351700] duration[9974627547595]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[28794734145697] CC6[0] CC7[0]
> (XEN) ==cpu1==
> (XEN) active state: C3
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[10699950] method[ FFH]
> duration[1141422044112]
> (XEN) C2: type[C1] latency[010] usage[06382904] method[ FFH]
> duration[1329739264322]
> (XEN) *C3: type[C2] latency[020] usage[44630764] method[ FFH]
> duration[31676618425954]
> (XEN) C0: usage[61713618] duration[9561201640320]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[30066495105056] CC6[0] CC7[0] [...]
>
> Why would some of the cores be in C3 even though they list max_cstate as C2?
This was precisely the reason why I told you that the numbering differs (and is confusing and has nothing to do with actual C state
numbers): What max_cstate refers to in the mwait-idle driver is what above is listed as type[Cx], i.e. the state at index 1 is C1, at
2 we've got C1E, and at 3 we've got C2. And those still aren't in line with the numbering the CPU documentation uses, it's rather kind of meant to refer to the ACPI numbering (but probably also not fully matching up).
So max_cstate=2 working suggests a problem with what the CPU calls C6, which presumably isn't all that surprising considering the many errata (BD35, BD38, BD40, BD59, BD87, and BD104). Not sure how to proceed from here - I suppose you already made sure you run with the latest available BIOS. And with 6 errata documented it's not all that unlikely that there's a 7th one with MONITOR/MWAIT behavior. The commit you bisected to (and which you had verified to be the culprit by just forcing
arch_skip_send_event_check() to always return false) could be reasonably assumed to be broken only when MWAIT use for all C states didn't work.
Don, Jun - is there anything known but not yet publicly documented for Family 6 Model 44 Xeons?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2014-11-27 9:27 ` Jan Beulich
2014-11-28 8:24 ` Steve Freitas
2014-12-03 17:14 ` Dugger, Donald D
@ 2015-02-27 17:25 ` Dugger, Donald D
2015-02-27 17:50 ` Brown, Len
2 siblings, 1 reply; 39+ messages in thread
From: Dugger, Donald D @ 2015-02-27 17:25 UTC (permalink / raw)
To: Jan Beulich, Steve Freitas, Nakajima, Jun, Brown, Len
Cc: Don Slutz, xen-devel
Len (CC'd on this email) is our power expert who has some ideas on this issue, I'll let him explain further.
--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786
-----Original Message-----
From: Jan Beulich [mailto:JBeulich@suse.com]
Sent: Thursday, November 27, 2014 2:28 AM
To: Steve Freitas; Dugger, Donald D; Nakajima, Jun
Cc: xen-devel@lists.xen.org; Don Slutz
Subject: Re: [Xen-devel] Regression, host crash with 4.5rc1
>>> On 27.11.14 at 06:29, <sflist@ihonk.com> wrote:
> On 11/25/2014 03:00 AM, Jan Beulich wrote:
>> Okay, so it's not really the mwait-idle driver causing the
>> regression, but it is C-state related. Hence we're now down to seeing
>> whether all or just the deeper C states are affected, i.e. I now need
>> to ask you to play with "max_cstate=". For that you'll have to
>> remember that the option's effect differs between the ACPI and the MWAIT idle drivers.
>> In the spirit of bisection I'd suggest using "max_cstate=2" first no
>> matter which of the two scenarios you pick. If that still hangs,
>> "max_cstate=1" obviously is the only other thing to try. Should that
>> not hang (and you left out "mwait-idle=0"), trying "max_cstate=3"
>> in that same scenario would be the other case to check.
>>
>> No need for 'd' and 'a' output for the time being, but 'c' output
>> would be much appreciated for all cases where you observe hangs.
>>
>
> Okay, working through that now. I tried max_cstate=2 and got no hangs,
> whether with or without mwait-idle=0. However, I was puzzled by this:
>
> (XEN) 'c' pressed -> printing ACPI Cx structures
> (XEN) ==cpu0==
> (XEN) active state: C0
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
> duration[1190961948551]
> (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
> duration[2015393965907]
> (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
> duration[30527997858148]
> (XEN) *C0: usage[73351700] duration[9974627547595]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[28794734145697] CC6[0] CC7[0]
> (XEN) ==cpu1==
> (XEN) active state: C3
> (XEN) max_cstate: C2
> (XEN) states:
> (XEN) C1: type[C1] latency[003] usage[10699950] method[ FFH]
> duration[1141422044112]
> (XEN) C2: type[C1] latency[010] usage[06382904] method[ FFH]
> duration[1329739264322]
> (XEN) *C3: type[C2] latency[020] usage[44630764] method[ FFH]
> duration[31676618425954]
> (XEN) C0: usage[61713618] duration[9561201640320]
> (XEN) max=0 pwr=0 urg=0 nxt=0
> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> (XEN) CC3[30066495105056] CC6[0] CC7[0] [...]
>
> Why would some of the cores be in C3 even though they list max_cstate as C2?
This was precisely the reason why I told you that the numbering differs (and is confusing and has nothing to do with actual C state
numbers): What max_cstate refers to in the mwait-idle driver is what above is listed as type[Cx], i.e. the state at index 1 is C1, at
2 we've got C1E, and at 3 we've got C2. And those still aren't in line with the numbering the CPU documentation uses, it's rather kind of meant to refer to the ACPI numbering (but probably also not fully matching up).
So max_cstate=2 working suggests a problem with what the CPU calls C6, which presumably isn't all that surprising considering the many errata (BD35, BD38, BD40, BD59, BD87, and BD104). Not sure how to proceed from here - I suppose you already made sure you run with the latest available BIOS. And with 6 errata documented it's not all that unlikely that there's a 7th one with MONITOR/MWAIT behavior. The commit you bisected to (and which you had verified to be the culprit by just forcing
arch_skip_send_event_check() to always return false) could be reasonably assumed to be broken only when MWAIT use for all C states didn't work.
Don, Jun - is there anything known but not yet publicly documented for Family 6 Model 44 Xeons?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2015-02-27 17:25 ` Dugger, Donald D
@ 2015-02-27 17:50 ` Brown, Len
2015-03-02 15:24 ` Jan Beulich
0 siblings, 1 reply; 39+ messages in thread
From: Brown, Len @ 2015-02-27 17:50 UTC (permalink / raw)
To: Dugger, Donald D, Jan Beulich, Steve Freitas, Nakajima, Jun
Cc: Don Slutz, xen-devel
[-- Attachment #1: Type: text/plain, Size: 2644 bytes --]
(Please forgive my lack of Xen-fu knowledge in advance)
If this issue were to happen on Linux/bare-metal, this is how I'd debug it.
Hopefully some of this will translate to Xen in one way or another.
dmesg | grep idle
will tell us what idle driver is running (on Dom0 kernel)
and if it is intel_idle, it will also tell us the supported sub-states (CPUID.MWAIT.EDX value)
grep . /sys/devices/system/cpu/cpu0/cpuidle/*/*
will tell us what states the OS is requesting,
It will expand on the "FFH" bit here:
> > (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
> > duration[1190961948551]
> > (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
> > duration[2015393965907]
> > (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
> > duration[30527997858148]
I'm hopeful that this information comes from the hardware's BIOS
and not some hypervisor tricking out Dom0 with a fake BIOS, yes?
If Xen doesn't have cpuidle, or its sysfs, then acpidump for the platform
should be able to tell us what the platform is exporting.
Next, hopefully the attached turbostat utility can be invoked on Dom0
and it can read the MSRs on at least 1 processor via the /dev/cpu interface.
This will tell you what the hardware supports, and what HW states are actually
being invoked. (which may be different from what the OS asks for...)
It may tell us just the same thing I think we learned here:
> > (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
> > (XEN) CC3[28794734145697] CC6[0] CC7[0]
which I'm assuming are a dump of the MSR residency counters.
If yes, it appears to be that this platform is not invoking c6 and pc6 at all,
and that the deepest state being used is actually cc3 and pc3.
I don't know if that is because you've booted the kernel with max_cstate=N
of some kind, or if this is default.
attached is turbostat, source and binary, run it this way
and send the ts.out file:
# ./turbostat --debug sleep 5 > ts.out 2>&1
Guessing...
If no surprises in the debug stuff requested above, and
If the XEN debug stuff above is with c6 explicitly disabled...
Note that here are two kinds of c6 -- CC6 (core) and PC6 (package).
If this box supports both, the next thing to try will be to keep CC6
enabled, but to just disable PC6. This is done via an MSR that turbostat
dumps out (MSR_NHM_SNB_PKG_CST_CFG_CTL) via the wrmsr(8) utility.
Though if that MSR is locked by the BIOS, then BIOS SETUP option
may be the only way to disable the package C-state limit without
also disabling the associated core C-state.
cheers,
-Len
ps.
[-- Attachment #2: turbostat-test.tar.gz --]
[-- Type: application/x-gzip, Size: 49844 bytes --]
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2015-02-27 17:50 ` Brown, Len
@ 2015-03-02 15:24 ` Jan Beulich
2015-03-09 0:45 ` Steve Freitas
0 siblings, 1 reply; 39+ messages in thread
From: Jan Beulich @ 2015-03-02 15:24 UTC (permalink / raw)
To: Steve Freitas, Len Brown
Cc: Don Slutz, Jun Nakajima, Donald D Dugger, xen-devel
>>> On 27.02.15 at 18:50, <len.brown@intel.com> wrote:
> If this issue were to happen on Linux/bare-metal, this is how I'd debug it.
> Hopefully some of this will translate to Xen in one way or another.
Sadly not really - the kernel plays only a minor role (forwarding ACPI
data to the hypervisor) in C-state handling under Xen.
> dmesg | grep idle
> will tell us what idle driver is running (on Dom0 kernel)
> and if it is intel_idle, it will also tell us the supported sub-states
> (CPUID.MWAIT.EDX value)
Yeah, we call the driver mwait-idle in the hypervisor, and the log
would be accssible via "xl dmesg", but yes, that information is
available there too.
>> > (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
>> > duration[1190961948551]
>> > (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
>> > duration[2015393965907]
>> > (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
>> > duration[30527997858148]
>
> I'm hopeful that this information comes from the hardware's BIOS
> and not some hypervisor tricking out Dom0 with a fake BIOS, yes?
In the case of mwait-idle (intel_idle on Linux) it would be built-in
knowledge of the driver. For acpi-cpuidle it would come from
actual firmware, not anything fake/virtual.
> Next, hopefully the attached turbostat utility can be invoked on Dom0
> and it can read the MSRs on at least 1 processor via the /dev/cpu interface.
Yes, that would be possible, provided it's not important what specific
CPU it gets executed on.
> It may tell us just the same thing I think we learned here:
>
>> > (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
>> > (XEN) CC3[28794734145697] CC6[0] CC7[0]
>
> which I'm assuming are a dump of the MSR residency counters.
> If yes, it appears to be that this platform is not invoking c6 and pc6 at
> all,
> and that the deepest state being used is actually cc3 and pc3.
> I don't know if that is because you've booted the kernel with max_cstate=N
> of some kind, or if this is default.
Sadly I haven't been able to tell which original mail the quotes
above are from, and since I had Steve experiment with disabling
the deepest C-state permitted to be used, it may well be that
this output was from one of those experiments. Remember, we
already know that with use of C6 alone disabled things work for
him (Steve - please correct me if I'm misremembering).
> Guessing...
> If no surprises in the debug stuff requested above, and
> If the XEN debug stuff above is with c6 explicitly disabled...
> Note that here are two kinds of c6 -- CC6 (core) and PC6 (package).
> If this box supports both, the next thing to try will be to keep CC6
> enabled, but to just disable PC6. This is done via an MSR that turbostat
> dumps out (MSR_NHM_SNB_PKG_CST_CFG_CTL) via the wrmsr(8) utility.
I don't think the wrmsr tool can be used (unmodified) to reliably do
this on all CPUs in the system - we'd likely have to cook up a patch
to the hypervisor instead, or I'd have to hand my patch to msr-tools
to Steve so he could use the tool under Xen (albeit that would also
require him to use one of our forward ported kernels, as the
upstream one doesn't have a pCPU sysfs interface yet afaik).
> Though if that MSR is locked by the BIOS, then BIOS SETUP option
> may be the only way to disable the package C-state limit without
> also disabling the associated core C-state.
Steve, could you check whether any such option exists (it's been
a while, so apologies if we had asked already)?
Jan
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2015-03-02 15:24 ` Jan Beulich
@ 2015-03-09 0:45 ` Steve Freitas
2015-03-26 20:49 ` Brown, Len
0 siblings, 1 reply; 39+ messages in thread
From: Steve Freitas @ 2015-03-09 0:45 UTC (permalink / raw)
To: Jan Beulich, Len Brown
Cc: Don Slutz, Jun Nakajima, Donald D Dugger, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 8063 bytes --]
Hi Len, thanks for chiming in. I am a Xen noob and generally clueless to
the inner workings of this power management stuff, so apologies in
advance if I don't understand what is asked. I am, however, happy to try
whatever you'd like me to in pursuing this issue.
On 03/02/2015 07:24 AM, Jan Beulich wrote:
>>>> On 27.02.15 at 18:50, <len.brown@intel.com> wrote:
>> If this issue were to happen on Linux/bare-metal, this is how I'd debug it.
>> Hopefully some of this will translate to Xen in one way or another.
> Sadly not really - the kernel plays only a minor role (forwarding ACPI
> data to the hypervisor) in C-state handling under Xen.
>
>> dmesg | grep idle
>> will tell us what idle driver is running (on Dom0 kernel)
>> and if it is intel_idle, it will also tell us the supported sub-states
>> (CPUID.MWAIT.EDX value)
root@g2:~# dmesg | grep idle
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 11.391708] intel_idle: MWAIT substates: 0x1120
[ 11.391711] intel_idle: v0.4 model 0x2C
[ 11.391712] intel_idle: lapic_timer_reliable_states 0xffffffff
[ 11.391780] intel_idle: intel_idle yielding to none
(This output is the same whether I've got max_cstate=2 set or not.)
> Yeah, we call the driver mwait-idle in the hypervisor, and the log
> would be accssible via "xl dmesg", but yes, that information is
> available there too.
>
>>>> (XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
>>>> duration[1190961948551]
>>>> (XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
>>>> duration[2015393965907]
>>>> (XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
>>>> duration[30527997858148]
>> I'm hopeful that this information comes from the hardware's BIOS
>> and not some hypervisor tricking out Dom0 with a fake BIOS, yes?
> In the case of mwait-idle (intel_idle on Linux) it would be built-in
> knowledge of the driver. For acpi-cpuidle it would come from
> actual firmware, not anything fake/virtual.
>
>> Next, hopefully the attached turbostat utility can be invoked on Dom0
>> and it can read the MSRs on at least 1 processor via the /dev/cpu interface.
> Yes, that would be possible, provided it's not important what specific
> CPU it gets executed on.
I've run it (with the "max_cstate=2" intact from Xen's boot line) and
the output is as follows, while running the problematic graphics
benchmark on my Win 7 VM:
root@g2:~/turbostat-test# ./turbostat
./turbostat: APERF or MPERF went backwards *
* Frequency results do not cover entire interval *
* fix this by running Linux-2.6.30 or later *
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 36804******** 2736 2800
0 64323******** 2560 2800
1 8244******** 3398 2800
2 125758******** 2760 2800
3 17811******** 3032 2800
4 735******** 2977 2800
5 3954******** 2656 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 47728******** 2804 2800
0 18007******** 3025 2800
1 69086******** 2634 2800
2 522******** 2713 2800
3 77486******** 2680 2800
4 58487******** 2932 2800
5 62777******** 3006 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 49031******** 2728 2800
0 78178******** 2681 2800
1 62045******** 2561 2800
2 9060******** 3110 2800
3 16619******** 3255 2800
4 720******** 2661 2800
5 127565******** 2763 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 65471******** 2700 2800
0 70582******** 2638 2800
1 2173******** 1954 2800
2 49981******** 2899 2800
3 78668******** 2682 2800
4 128293******** 2762 2800
5 63131******** 2566 2800
Not sure why the warning about the kernel version, this box is running
Debian's Linux 3.16 kernel.
With "max_cstate=2" removed from Xen's boot line, this is the result:
root@g2:~/turbostat-test# ./turbostat
./turbostat: APERF or MPERF went backwards *
* Frequency results do not cover entire interval *
* fix this by running Linux-2.6.30 or later *
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 23507******** 2621 2800
0 27631******** 2552 2800
1 35945******** 2978 2800
2 24417******** 2472 2800
3 1001******** 2948 2800
4 24417******** 2472 2800
5 27631******** 2552 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 14114******** 2687 2800
0 529******** 2738 2800
1 60363******** 2750 2800
2 21290******** 2497 2800
3 1028******** 2934 2800
4 629******** 2943 2800
5 842******** 2937 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 15048******** 2714 2800
0 25703******** 2489 2800
1 36024******** 2975 2800
2 5248******** 2454 2800
3 5248******** 2454 2800
4 9138******** 2755 2800
5 8925******** 2751 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 32859******** 2598 2800
0 23089******** 2526 2800
1 61730******** 2751 2800
2 26138******** 2492 2800
3 26138******** 2492 2800
4 30029******** 2574 2800
5 30029******** 2574 2800
>> It may tell us just the same thing I think we learned here:
>>
>>>> (XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
>>>> (XEN) CC3[28794734145697] CC6[0] CC7[0]
>> which I'm assuming are a dump of the MSR residency counters.
>> If yes, it appears to be that this platform is not invoking c6 and pc6 at
>> all,
>> and that the deepest state being used is actually cc3 and pc3.
>> I don't know if that is because you've booted the kernel with max_cstate=N
>> of some kind, or if this is default.
> Sadly I haven't been able to tell which original mail the quotes
> above are from, and since I had Steve experiment with disabling
> the deepest C-state permitted to be used, it may well be that
> this output was from one of those experiments. Remember, we
> already know that with use of C6 alone disabled things work for
> him (Steve - please correct me if I'm misremembering).
AIUI, that is correct. My Xen boot line (which eliminates the dom0/U
hangs) includes "mwait-idle=1 max_cstate=2".
>> Guessing...
>> If no surprises in the debug stuff requested above, and
>> If the XEN debug stuff above is with c6 explicitly disabled...
>> Note that here are two kinds of c6 -- CC6 (core) and PC6 (package).
>> If this box supports both, the next thing to try will be to keep CC6
>> enabled, but to just disable PC6. This is done via an MSR that turbostat
>> dumps out (MSR_NHM_SNB_PKG_CST_CFG_CTL) via the wrmsr(8) utility.
> I don't think the wrmsr tool can be used (unmodified) to reliably do
> this on all CPUs in the system - we'd likely have to cook up a patch
> to the hypervisor instead, or I'd have to hand my patch to msr-tools
> to Steve so he could use the tool under Xen (albeit that would also
> require him to use one of our forward ported kernels, as the
> upstream one doesn't have a pCPU sysfs interface yet afaik).
I'm game for whatever.
>> Though if that MSR is locked by the BIOS, then BIOS SETUP option
>> may be the only way to disable the package C-state limit without
>> also disabling the associated core C-state.
> Steve, could you check whether any such option exists (it's been
> a while, so apologies if we had asked already)?
No problem. I've cruised through the BIOS options and this is what I see
that may apply:
If you'd like me to make any changes to those settings, please let me
know. For reference this is a Lenovo ThinkStation D20 running a Xeon X5660.
Thanks!
Steve
[-- Attachment #1.2.1: Type: text/html, Size: 11470 bytes --]
[-- Attachment #1.2.2: Type: image/jpeg, Size: 55018 bytes --]
[-- Attachment #1.2.3: Type: image/jpeg, Size: 60102 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: Regression, host crash with 4.5rc1
2015-03-09 0:45 ` Steve Freitas
@ 2015-03-26 20:49 ` Brown, Len
0 siblings, 0 replies; 39+ messages in thread
From: Brown, Len @ 2015-03-26 20:49 UTC (permalink / raw)
To: Steve Freitas, Jan Beulich
Cc: Don Slutz, Nakajima, Jun, Dugger, Donald D, xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 9235 bytes --]
Steve,
Yes, you've found the right BIOS SETUP option.
Package C-state limit is currently set to C6.
Set it to C3. Re-run w/ all OS C-states enabled
and see if joy results. IF it doesn't, set to C1
and try again.
I'm not sure what the NHM C7S option does.
It may have no effect for native MWAIT idle;
but it if above doesn't work, it is the next
thing to investigate.
cheers,
-Len
ps. You want to invoke turbostat with the "-debug" option
that will dump the MSRs - or try to. maybe an exercise
in futility on top of Xen, we'll see. But you found
the Package C-state limit in the BIOS, so running
turbostat is somewhat academic now.
pps. sorry for the top-post, i'm using a primitive mailer.
From: Steve Freitas [mailto:sflist@ihonk.com]
Sent: Sunday, March 08, 2015 8:45 PM
To: Jan Beulich; Brown, Len
Cc: Dugger, Donald D; Nakajima, Jun; xen-devel@lists.xen.org; Don Slutz
Subject: Re: [Xen-devel] Regression, host crash with 4.5rc1
Hi Len, thanks for chiming in. I am a Xen noob and generally clueless to the inner workings of this power management stuff, so apologies in advance if I don't understand what is asked. I am, however, happy to try whatever you'd like me to in pursuing this issue.
On 03/02/2015 07:24 AM, Jan Beulich wrote:
On 27.02.15 at 18:50, <len.brown@intel.com><mailto:len.brown@intel.com> wrote:
If this issue were to happen on Linux/bare-metal, this is how I'd debug it.
Hopefully some of this will translate to Xen in one way or another.
Sadly not really - the kernel plays only a minor role (forwarding ACPI
data to the hypervisor) in C-state handling under Xen.
dmesg | grep idle
will tell us what idle driver is running (on Dom0 kernel)
and if it is intel_idle, it will also tell us the supported sub-states
(CPUID.MWAIT.EDX value)
root@g2:~# dmesg | grep idle
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 11.391708] intel_idle: MWAIT substates: 0x1120
[ 11.391711] intel_idle: v0.4 model 0x2C
[ 11.391712] intel_idle: lapic_timer_reliable_states 0xffffffff
[ 11.391780] intel_idle: intel_idle yielding to none
(This output is the same whether I've got max_cstate=2 set or not.)
Yeah, we call the driver mwait-idle in the hypervisor, and the log
would be accssible via "xl dmesg", but yes, that information is
available there too.
(XEN) C1: type[C1] latency[003] usage[12219860] method[ FFH]
duration[1190961948551]
(XEN) C2: type[C1] latency[010] usage[10205554] method[ FFH]
duration[2015393965907]
(XEN) C3: type[C2] latency[020] usage[50926286] method[ FFH]
duration[30527997858148]
I'm hopeful that this information comes from the hardware's BIOS
and not some hypervisor tricking out Dom0 with a fake BIOS, yes?
In the case of mwait-idle (intel_idle on Linux) it would be built-in
knowledge of the driver. For acpi-cpuidle it would come from
actual firmware, not anything fake/virtual.
Next, hopefully the attached turbostat utility can be invoked on Dom0
and it can read the MSRs on at least 1 processor via the /dev/cpu interface.
Yes, that would be possible, provided it's not important what specific
CPU it gets executed on.
I've run it (with the "max_cstate=2" intact from Xen's boot line) and the output is as follows, while running the problematic graphics benchmark on my Win 7 VM:
root@g2:~/turbostat-test# ./turbostat
./turbostat: APERF or MPERF went backwards *
* Frequency results do not cover entire interval *
* fix this by running Linux-2.6.30 or later *
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 36804******** 2736 2800
0 64323******** 2560 2800
1 8244******** 3398 2800
2 125758******** 2760 2800
3 17811******** 3032 2800
4 735******** 2977 2800
5 3954******** 2656 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 47728******** 2804 2800
0 18007******** 3025 2800
1 69086******** 2634 2800
2 522******** 2713 2800
3 77486******** 2680 2800
4 58487******** 2932 2800
5 62777******** 3006 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 49031******** 2728 2800
0 78178******** 2681 2800
1 62045******** 2561 2800
2 9060******** 3110 2800
3 16619******** 3255 2800
4 720******** 2661 2800
5 127565******** 2763 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 65471******** 2700 2800
0 70582******** 2638 2800
1 2173******** 1954 2800
2 49981******** 2899 2800
3 78668******** 2682 2800
4 128293******** 2762 2800
5 63131******** 2566 2800
Not sure why the warning about the kernel version, this box is running Debian's Linux 3.16 kernel.
With "max_cstate=2" removed from Xen's boot line, this is the result:
root@g2:~/turbostat-test# ./turbostat
./turbostat: APERF or MPERF went backwards *
* Frequency results do not cover entire interval *
* fix this by running Linux-2.6.30 or later *
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 23507******** 2621 2800
0 27631******** 2552 2800
1 35945******** 2978 2800
2 24417******** 2472 2800
3 1001******** 2948 2800
4 24417******** 2472 2800
5 27631******** 2552 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 14114******** 2687 2800
0 529******** 2738 2800
1 60363******** 2750 2800
2 21290******** 2497 2800
3 1028******** 2934 2800
4 629******** 2943 2800
5 842******** 2937 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 15048******** 2714 2800
0 25703******** 2489 2800
1 36024******** 2975 2800
2 5248******** 2454 2800
3 5248******** 2454 2800
4 9138******** 2755 2800
5 8925******** 2751 2800
CPU Avg_MHz %Busy Bzy_MHz TSC_MHz
- 32859******** 2598 2800
0 23089******** 2526 2800
1 61730******** 2751 2800
2 26138******** 2492 2800
3 26138******** 2492 2800
4 30029******** 2574 2800
5 30029******** 2574 2800
It may tell us just the same thing I think we learned here:
(XEN) PC2[0] PC3[8589642315848] PC6[0] PC7[0]
(XEN) CC3[28794734145697] CC6[0] CC7[0]
which I'm assuming are a dump of the MSR residency counters.
If yes, it appears to be that this platform is not invoking c6 and pc6 at
all,
and that the deepest state being used is actually cc3 and pc3.
I don't know if that is because you've booted the kernel with max_cstate=N
of some kind, or if this is default.
Sadly I haven't been able to tell which original mail the quotes
above are from, and since I had Steve experiment with disabling
the deepest C-state permitted to be used, it may well be that
this output was from one of those experiments. Remember, we
already know that with use of C6 alone disabled things work for
him (Steve - please correct me if I'm misremembering).
AIUI, that is correct. My Xen boot line (which eliminates the dom0/U hangs) includes "mwait-idle=1 max_cstate=2".
Guessing...
If no surprises in the debug stuff requested above, and
If the XEN debug stuff above is with c6 explicitly disabled...
Note that here are two kinds of c6 -- CC6 (core) and PC6 (package).
If this box supports both, the next thing to try will be to keep CC6
enabled, but to just disable PC6. This is done via an MSR that turbostat
dumps out (MSR_NHM_SNB_PKG_CST_CFG_CTL) via the wrmsr(8) utility.
I don't think the wrmsr tool can be used (unmodified) to reliably do
this on all CPUs in the system - we'd likely have to cook up a patch
to the hypervisor instead, or I'd have to hand my patch to msr-tools
to Steve so he could use the tool under Xen (albeit that would also
require him to use one of our forward ported kernels, as the
upstream one doesn't have a pCPU sysfs interface yet afaik).
I'm game for whatever.
Though if that MSR is locked by the BIOS, then BIOS SETUP option
may be the only way to disable the package C-state limit without
also disabling the associated core C-state.
Steve, could you check whether any such option exists (it's been
a while, so apologies if we had asked already)?
No problem. I've cruised through the BIOS options and this is what I see that may apply:
[cid:image001.jpg@01D067E4.4AD40E00]
[cid:image002.jpg@01D067E4.4AD40E00]
If you'd like me to make any changes to those settings, please let me know. For reference this is a Lenovo ThinkStation D20 running a Xeon X5660.
Thanks!
Steve
[-- Attachment #1.1.2: Type: text/html, Size: 23408 bytes --]
[-- Attachment #1.2: image001.jpg --]
[-- Type: image/jpeg, Size: 55018 bytes --]
[-- Attachment #1.3: image002.jpg --]
[-- Type: image/jpeg, Size: 60102 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 39+ messages in thread
end of thread, other threads:[~2015-03-26 20:49 UTC | newest]
Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-11-03 21:46 Regression, host crash with 4.5rc1 Steve Freitas
2014-11-03 21:52 ` Steve Freitas
2014-11-04 14:01 ` Don Slutz
2014-11-04 8:20 ` Pasi Kärkkäinen
2014-11-04 10:15 ` Jan Beulich
2014-11-10 8:03 ` Steve Freitas
2014-11-10 8:51 ` Jan Beulich
2014-11-10 20:05 ` Steve Freitas
2014-11-11 8:05 ` Jan Beulich
2014-11-17 19:21 ` Steve Freitas
2014-11-18 7:54 ` Jan Beulich
2014-11-20 1:23 ` Steve Freitas
2014-11-20 7:59 ` Jan Beulich
2014-11-20 20:07 ` Steve Freitas
2014-11-21 8:42 ` Jan Beulich
2014-11-23 1:28 ` Steve Freitas
2014-11-24 8:45 ` Jan Beulich
2014-11-24 9:08 ` Steve Freitas
2014-11-24 9:15 ` Jan Beulich
2014-11-24 11:41 ` Jan Beulich
2014-11-24 22:17 ` Steve Freitas
2014-11-25 8:16 ` Jan Beulich
2014-11-25 9:38 ` Steve Freitas
2014-11-25 11:00 ` Jan Beulich
2014-11-27 5:29 ` Steve Freitas
2014-11-27 9:27 ` Jan Beulich
2014-11-28 8:24 ` Steve Freitas
2014-11-28 8:50 ` Jan Beulich
2014-11-28 9:44 ` Steve Freitas
2014-12-03 17:14 ` Dugger, Donald D
2015-02-27 17:25 ` Dugger, Donald D
2015-02-27 17:50 ` Brown, Len
2015-03-02 15:24 ` Jan Beulich
2015-03-09 0:45 ` Steve Freitas
2015-03-26 20:49 ` Brown, Len
2014-11-21 9:31 ` Jan Beulich
2014-11-04 18:35 ` Steve Freitas
2014-11-04 14:39 ` Don Slutz
2014-11-06 23:20 ` Steve Freitas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.