HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Hi guys,
today I have upgraded from XEN 4.5.1 to XEN 4.5.2 and also upgraded the 
dom0 kernel from 3.18.9 to 4.1.7

After the upgrade HVM domUs appear to no longer work - regardless of the 
dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV 
domUs, however, work just fine as before on both dom0 kernels.

xl dmesg shows the following information after the first crashed HVM 
domU which is started as part of the machine booting up:

===== start xl dmesg output =====
  Xen 4.5.2
(XEN) Xen version 4.5.2 (@myhomeismycastle.com) (x86_64-pc-linux-gnu-gcc 
(Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Wed Nov 11 
13:00:00 CET 2015
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder ucode=-1 loglvl=warning 
guest_loglvl=none/warning dom0_mem=4G,max:4G tmem=1 tmem_compress=1 
tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle 
clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040000000 (usable)
(XEN)  0000000040000000 - 0000000040200000 (reserved)
(XEN)  0000000040200000 - 00000000db9f0000 (usable)
(XEN)  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN)  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN)  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN)  00000000dc651000 - 00000000dc652000 (usable)
(XEN)  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN)  00000000dc695000 - 00000000dcdba000 (usable)
(XEN)  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN)  00000000dcff2000 - 00000000dd000000 (usable)
(XEN)  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000081e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) ACPI: FACS DC1F7F80, 0040
(XEN) ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) System RAM: 32674MB (33458948kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 
dc1f7f80/0000000000000000, using 32
(XEN) Processor #0 6:10 APIC version 21
(XEN) Processor #2 6:10 APIC version 21
(XEN) Processor #4 6:10 APIC version 21
(XEN) Processor #6 6:10 APIC version 21
(XEN) Processor #1 6:10 APIC version 21
(XEN) Processor #3 6:10 APIC version 21
(XEN) Processor #5 6:10 APIC version 21
(XEN) Processor #7 6:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2394.643 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 8 CPUs
(XEN) tmem: initialized comp=1 dedup=1 tze=0
(XEN) Dom0 has maximum 792 PIRQs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages 
to be allocated)
(XEN)  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN)  Start info:    ffffffff82400000->ffffffff824004b4
(XEN)  Page tables:   ffffffff82401000->ffffffff82418000
(XEN)  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff818e31f0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen)
(XEN) Freed 316kB init memory.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 
0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 
0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 
0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 
0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 
0x0000000000074700 to 0x0000000000047700.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 
0xffff83080ca73000 to 0xffffffff81646d10.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 
0xffff83080ca73080 to 0xffffffff81648e70.
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
state (0).
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011, 
gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, 
gh_mask=ffffffffffffffff
(XEN) CR3: actual=0x0000000000800000, target_count=0
(XEN)      target0=0000000000000000, target1=0000000000000000
(XEN)      target2=0000000000000000, target3=0000000000000000
(XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 
0x0000000100000000 (0x0000000100000000)
(XEN) RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 
0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, 
base=0x0000000000000000
(XEN) DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, 
base=0x0000000000000000
(XEN) SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, 
base=0x0000000000000000
(XEN) ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, 
base=0x0000000000000000
(XEN) FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GDTR:                           limit=0x00000037, 
base=0x00000000000f6d80
(XEN) LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, 
base=0x0000000000000000
(XEN) IDTR:                           limit=0x00000000, 
base=0x00000000000f6dbe
(XEN) TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, 
base=0x0000000000000000
(XEN) Guest PAT = 0x0007040600070406
(XEN) TSC Offset = ffffffe7c228bb48
(XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) Interruptibility=0000 ActivityState=0000
(XEN) *** Host State ***
(XEN) RSP = 0xffff8308063fff90  RIP = 0xffff82d0801f46a0
(XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 
TRBase=ffff83080ca88c80
(XEN) GDTBase=ffff83080ca7c000 IDTBase=ffff83080ca8d000
(XEN) CR0=000000008005003b CR3=000000070bf21000 CR4=00000000000426f0
(XEN) Sysenter RSP=ffff8308063fffc0 CS:RIP=e008:ffff82d080235550
(XEN) Host PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) EntryControls=000051ff ExitControls=000fefff
(XEN) ExceptionBitmap=000400c0
(XEN) VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN)         reason=80000021 qualification=00000000
(XEN) IDTVectoring: info=00000000 errcode=00000000
(XEN) TPR Threshold = 0x00
(XEN) EPT pointer = 0x000000070bf1f01e
(XEN) Virtual processor ID = 0x0002
(XEN) **************************************
(XEN) domain_crash called from vmx.c:2511
(XEN) Domain 1 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    0008:[<0000000100000000>]
(XEN) RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) cr3: 0000000000800000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
===== end xl dmesg output =====

It is probably also worth pointing out again that this crash happens 
with both kernels, 3.18.9 and 4.1.7; the only obvious differenc (other 
than slight address differences) between 3.18.9 and 4.1.7 are the 
following lines in xl dmesg above which only show up on 4.1.7:

(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 
0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 
0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 
0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 
0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 
0x0000000000074700 to 0x0000000000047700.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 
0xffff83080ca73000 to 0xffffffff81646d10.
(XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 
0xffff83080ca73080 to 0xffffffff81648e70.


The configuration file for the crashing HVM domain contains nothing 
fancy and looks as follows:

===== start configuration file =====
builder                 = 'hvm'
cpus                    = '2-7'
vcpus                   = 2
cpu_weight              = 512
memory                  = 640
name                    = 'pfsense'
disk                    = [ 
'vdev=xvda,format=raw,access=rw,target=/etc/xen/guests/disk.d/pfsense.disk' 
]
vif                     = [ 
'mac=00:16:3e:a1:64:01,bridge=xenbr0,type=vif,vifname=pfsense.0,script=vif-bridge.noTXoffload' 
]
on_poweroff             = 'destroy'
on_reboot               = 'restart'
on_crash                = 'destroy'
localtime               = 0
boot                    = 'c'
vnc                     = 1
vnclisten               = '127.0.0.1'
vncpasswd               = ''
keymap                  = 'de'
nographic               = 1
serial                  = 'pty'
nx                      = 1
pci                     = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
===== end configuration file =====

The crash seems to not be related to this domU; there's another HVM domU 
which also crashes with a similar message.

Prior to the upgrade this has all worked flawlessly for a number of 
months on 4.5.1. Any insight on what's going on and how to return to 
proper operation would greatly be appreciated.

Many thanks in advance, Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
> After the upgrade HVM domUs appear to no longer work - regardless of the 
> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV 
> domUs, however, work just fine as before on both dom0 kernels.
> 
> xl dmesg shows the following information after the first crashed HVM 
> domU which is started as part of the machine booting up:
>[...]
> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
> state (0).
> (XEN) ************* VMCS Area **************
> (XEN) *** Guest State ***
> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011, 
> gh_mask=ffffffffffffffff
> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, 
> gh_mask=ffffffffffffffff
> (XEN) CR3: actual=0x0000000000800000, target_count=0
> (XEN)      target0=0000000000000000, target1=0000000000000000
> (XEN)      target2=0000000000000000, target3=0000000000000000
> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 
> 0x0000000100000000 (0x0000000100000000)

Other than RIP looking odd for a guest still in non-paged protected
mode I can't seem to spot anything wrong with guest state.
Considering that there was just a single HVM-related commit
between the two releases (which looks completely unrelated) I
wonder whether you're observing a problem that's a side effect
of something else, e.g. a build system change (compiler update or
alike). If that can be ruled out, I guess the only chance would be
for you to bisect for the offending commit.

Are you observing this on more than one kind of system?

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 12/11/15 12:52, Jan Beulich wrote:
>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>> After the upgrade HVM domUs appear to no longer work - regardless of the 
>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV 
>> domUs, however, work just fine as before on both dom0 kernels.
>>
>> xl dmesg shows the following information after the first crashed HVM 
>> domU which is started as part of the machine booting up:
>> [...]
>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
>> state (0).
>> (XEN) ************* VMCS Area **************
>> (XEN) *** Guest State ***
>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011, 
>> gh_mask=ffffffffffffffff
>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, 
>> gh_mask=ffffffffffffffff
>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>> (XEN)      target0=0000000000000000, target1=0000000000000000
>> (XEN)      target2=0000000000000000, target3=0000000000000000
>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 
>> 0x0000000100000000 (0x0000000100000000)
> Other than RIP looking odd for a guest still in non-paged protected
> mode I can't seem to spot anything wrong with guest state.

odd? That will be the source of the failure.

Out of long mode, the upper 32bit of %rip should all be zero, and it
should not be possible to set any of them.

I suspect that the guest has exited for emulation, and there has been a
bad update to %rip.  The alternative (which I hope is not the case) is
that there is a hardware errata which allows the guest to accidentally
get it self into this condition.

Are you able to rerun with a debug build of the hypervisor?

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Hi Andrew,
thanks for your reply. Answers are inline further down.

Am 12.11.15 um 14:01 schrieb Andrew Cooper:
> On 12/11/15 12:52, Jan Beulich wrote:
>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>> After the upgrade HVM domUs appear to no longer work - regardless of the
>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>> domUs, however, work just fine as before on both dom0 kernels.
>>>
>>> xl dmesg shows the following information after the first crashed HVM
>>> domU which is started as part of the machine booting up:
>>> [...]
>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>> state (0).
>>> (XEN) ************* VMCS Area **************
>>> (XEN) *** Guest State ***
>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>> gh_mask=ffffffffffffffff
>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>> gh_mask=ffffffffffffffff
>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>> 0x0000000100000000 (0x0000000100000000)
>> Other than RIP looking odd for a guest still in non-paged protected
>> mode I can't seem to spot anything wrong with guest state.
> odd? That will be the source of the failure.
>
> Out of long mode, the upper 32bit of %rip should all be zero, and it
> should not be possible to set any of them.
>
> I suspect that the guest has exited for emulation, and there has been a
> bad update to %rip.  The alternative (which I hope is not the case) is
> that there is a hardware errata which allows the guest to accidentally
> get it self into this condition.
>
> Are you able to rerun with a debug build of the hypervisor?
Given that I am compiling from source under gentoo and provided you lend 
me a helping hand in case I get stuck, I am confident that this is possible.

gentoo has three xen packages (they call those ebuilds) as follows
     app-emulation/xen
     app-emulation/xen-tools
     app-emulation/pvgrub
all of which are installed on my system. The former two offer a debug 
USE-flag and I assume that debug code for the latter is not required as 
this is for (the still working) PV domUs only. Furthermore as you are 
talking about the hypervisor, I guess it is safe to assume that it is 
app-emulation/xen and not xen-tools. Right?

BTW: The description of the debug USE flag reads as follows:
Enable extra debug codepaths, like asserts and extra output. If you want 
to get meaningful backtraces see 
https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
I assume that backtraces are probably not required to get things moving.

Another question is whether prior to enabling the debug USE flag it 
might make sense to re-compile with gcc-4.8.5 (please see my previous 
list reply) to rule out any compiler related issues. Jan, Andrew - what 
are your thoughts?

Many thanks Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 12.11.15 at 15:29, <ariel.atom2@web2web.at> wrote:
> Another question is whether prior to enabling the debug USE flag it 
> might make sense to re-compile with gcc-4.8.5 (please see my previous 
> list reply) to rule out any compiler related issues. Jan, Andrew - what 
> are your thoughts?

The order of things you try is entirely up to you.

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 12/11/15 14:29, Atom2 wrote:
> Hi Andrew,
> thanks for your reply. Answers are inline further down.
>
> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>> of the
>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>
>>>> xl dmesg shows the following information after the first crashed HVM
>>>> domU which is started as part of the machine booting up:
>>>> [...]
>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>> state (0).
>>>> (XEN) ************* VMCS Area **************
>>>> (XEN) *** Guest State ***
>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>> gh_mask=ffffffffffffffff
>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>> gh_mask=ffffffffffffffff
>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>> 0x0000000100000000 (0x0000000100000000)
>>> Other than RIP looking odd for a guest still in non-paged protected
>>> mode I can't seem to spot anything wrong with guest state.
>> odd? That will be the source of the failure.
>>
>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>> should not be possible to set any of them.
>>
>> I suspect that the guest has exited for emulation, and there has been a
>> bad update to %rip.  The alternative (which I hope is not the case) is
>> that there is a hardware errata which allows the guest to accidentally
>> get it self into this condition.
>>
>> Are you able to rerun with a debug build of the hypervisor?
> Given that I am compiling from source under gentoo and provided you
> lend me a helping hand in case I get stuck, I am confident that this
> is possible.
>
> gentoo has three xen packages (they call those ebuilds) as follows
>     app-emulation/xen
>     app-emulation/xen-tools
>     app-emulation/pvgrub
> all of which are installed on my system. The former two offer a debug
> USE-flag and I assume that debug code for the latter is not required
> as this is for (the still working) PV domUs only. Furthermore as you
> are talking about the hypervisor, I guess it is safe to assume that it
> is app-emulation/xen and not xen-tools. Right?

I would guess so.

>
> BTW: The description of the debug USE flag reads as follows:
> Enable extra debug codepaths, like asserts and extra output. If you
> want to get meaningful backtraces see
> https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
> I assume that backtraces are probably not required to get things moving.

By the sounds of it, the debug USE flag is what you want.

>
> Another question is whether prior to enabling the debug USE flag it
> might make sense to re-compile with gcc-4.8.5 (please see my previous
> list reply) to rule out any compiler related issues. Jan, Andrew -
> what are your thoughts?

First of all, check whether the compiler makes a difference on 4.5.2

If both compiles result in a guest crashing in that manner, test a debug
Xen to see if any assertions/errors are encountered just before the
guest crashes.

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 3759 bytes --]

Am 12.11.15 um 17:43 schrieb Andrew Cooper:
> On 12/11/15 14:29, Atom2 wrote:
>> Hi Andrew,
>> thanks for your reply. Answers are inline further down.
>>
>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>> of the
>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>
>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>> domU which is started as part of the machine booting up:
>>>>> [...]
>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>> state (0).
>>>>> (XEN) ************* VMCS Area **************
>>>>> (XEN) *** Guest State ***
>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>> gh_mask=ffffffffffffffff
>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>> gh_mask=ffffffffffffffff
>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>> 0x0000000100000000 (0x0000000100000000)
>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>> mode I can't seem to spot anything wrong with guest state.
>>> odd? That will be the source of the failure.
>>>
>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>> should not be possible to set any of them.
>>>
>>> I suspect that the guest has exited for emulation, and there has been a
>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>> that there is a hardware errata which allows the guest to accidentally
>>> get it self into this condition.
>>>
>>> Are you able to rerun with a debug build of the hypervisor?
>> [snip]
>> Another question is whether prior to enabling the debug USE flag it
>> might make sense to re-compile with gcc-4.8.5 (please see my previous
>> list reply) to rule out any compiler related issues. Jan, Andrew -
>> what are your thoughts?
> First of all, check whether the compiler makes a difference on 4.5.2
Hi Andrew,
I changed the compiler and there was no change to the better: 
Unfortunately the HVM domU is still crashing with a similar error 
message as soon as it is being started.
> If both compiles result in a guest crashing in that manner, test a debug
> Xen to see if any assertions/errors are encountered just before the
> guest crashes.
>
As the compiler did not make any difference, I enabled the debug USE 
flag, re-compiled (using gcc-4.9.3), and rebooted using a serial console 
to capture output. Unfortunately I did not get very far and things 
become even stranger: This time the system did not even finnish the boot 
process, but rather hard-stopped pretty early with a message reading 
"Panic on CPU 3: DOUBLE FAULT -- system shutdown". The captured logfile 
is attached as "serial log.txt".

As this happened immediately after the CPU microcode update, I thought 
there might be a connection and disabled the microcode update. After the 
next reboot it seemed as if the boot process got a bit further as 
evidenced by a few more lines in the log file (those between lines 136 
and 197 in the second log file named "serial log no ucode.txt"), but in 
the end it finnished off with an identical error message (only the CPU # 
was different this time, but that number seems to change between boots 
anyways).

I hope that makes some sense to you.

Many thanks for your support and best regards Atom2

[-- Attachment #2: serial log.txt --]
[-- Type: text/plain, Size: 8257 bytes --]

 Xen 4.5.2
(XEN) Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=y Thu Nov 12 21:30:05 CET 2015
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040000000 (usable)
(XEN)  0000000040000000 - 0000000040200000 (reserved)
(XEN)  0000000040200000 - 00000000db9f0000 (usable)
(XEN)  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN)  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN)  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN)  00000000dc651000 - 00000000dc652000 (usable)
(XEN)  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN)  00000000dc695000 - 00000000dcdba000 (usable)
(XEN)  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN)  00000000dcff2000 - 00000000dd000000 (usable)
(XEN)  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000081e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) ACPI: FACS DC1F7F80, 0040
(XEN) ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) System RAM: 32674MB (33458948kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000081e600000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fd8d0
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) Processor #1 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) Processor #3 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) Processor #5 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) Processor #7 6:10 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2397.844 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) alt table ffff82d0802e8810 -> ffff82d0802e9830
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2a
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) *** DOUBLE FAULT ***
(XEN) ----[ Xen-4.5.2  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    e008:[<ffff82d08017658c>] do_IRQ+0x15/0x64c
(XEN) RFLAGS: 0000000000010083   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000000028   rcx: 0000000000000000
(XEN) rdx: 0000000000000002   rsi: 0000000000000028   rdi: ffff8308063e6c68
(XEN) rbp: ffff8308063e6c58   rsp: ffff8308063e5bd8   r8:  0000000000000000
(XEN) r9:  0000000000000039   r10: ffff82d08033a2f8   r11: ffff82d08033a2e8
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000002
(XEN) r15: ffff83080ca74000   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) cr3: 00000000db69f000   cr2: ffff8308063e5bc8
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Valid stack range: ffff8308063e6000-ffff8308063e8000, sp=ffff8308063e5bd8, tss.esp0=ffff8308063e7fc0
(XEN) No stack overflow detected. Skipping stack trace.
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 6:
(XEN) DOUBLE FAULT -- system shutdown
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

[-- Attachment #3: serial log no ucode.txt --]
[-- Type: text/plain, Size: 11432 bytes --]

 Xen 4.5.2
(XEN) Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=y Thu Nov 12 21:30:05 CET 2015
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040000000 (usable)
(XEN)  0000000040000000 - 0000000040200000 (reserved)
(XEN)  0000000040200000 - 00000000db9f0000 (usable)
(XEN)  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN)  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN)  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN)  00000000dc651000 - 00000000dc652000 (usable)
(XEN)  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN)  00000000dc695000 - 00000000dcdba000 (usable)
(XEN)  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN)  00000000dcff2000 - 00000000dd000000 (usable)
(XEN)  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000081e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) ACPI: FACS DC1F7F80, 0040
(XEN) ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) System RAM: 32674MB (33458948kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000081e600000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fd8d0
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) Processor #1 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) Processor #3 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) Processor #5 6:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) Processor #7 6:10 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2394.622 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) alt table ffff82d0802e8810 -> ffff82d0802e9830
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2a
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 8 CPUs
(XEN) tmem: initialized comp=1 dedup=1 tze=0
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 792 PIRQs
(XEN) Multiple initrd candidates, picking module #1
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x65033b
(XEN) elf_parse_binary: phdr: paddr=0x1651000 memsz=0x1e5000
(XEN) elf_parse_binary: phdr: paddr=0x1836000 memsz=0x98000
(XEN) elf_parse_binary: phdr: paddr=0x18ce000 memsz=0x1000
(XEN) elf_parse_binary: phdr: paddr=0x18cf000 memsz=0x13c98
(XEN) elf_parse_binary: phdr: paddr=0x18e3000 memsz=0x3d000
(XEN) elf_parse_binary: phdr: paddr=0x1920000 memsz=0x1120
(XEN) elf_parse_binary: phdr: paddr=0x1922000 memsz=0x2de000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1c00000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff818e31f0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff81c00000
(XEN)     virt_entry       = 0xffffffff818e31f0
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN)  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN)  Start info:    ffffffff82400000->ffffffff824004b4
(XEN)  Page tables:   ffffffff82401000->ffffffff82418000
(XEN)  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff818e31f0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff8165033b
(XEN) elf_load_binary: phdr 1 at 0xffffffff81651000 -> 0xffffffff81836000
(XEN) elf_load_binary: phdr 2 at 0xffffffff81836000 -> 0xffffffff818ce000
(XEN) elf_load_binary: phdr 3 at 0xffffffff818ce000 -> 0xffffffff818cf000
(XEN) elf_load_binary: phdr 4 at 0xffffffff818cf000 -> 0xffffffff818e2c98
(XEN) elf_load_binary: phdr 5 at 0xffffffff818e3000 -> 0xffffffff81920000
(XEN) elf_load_binary: phdr 6 at 0xffffffff81920000 -> 0xffffffff81921120
(XEN) elf_load_binary: phdr 7 at 0xffffffff81922000 -> 0xffffffff819dd000
(XEN) *** DOUBLE FAULT ***
(XEN) ----[ Xen-4.5.2  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e008:[<ffff82d08017658c>] do_IRQ+0x15/0x64c
(XEN) RFLAGS: 0000000000010083   CONTEXT: hypervisor
(XEN) rax: 000000378c762d00   rbx: 0000000000000003   rcx: 0000000000196bd0
(XEN) rdx: 000000000016c015   rsi: 00000000a0789100   rdi: ffff83080ca96d38
(XEN) rbp: ffff83080ca96d28   rsp: ffff83080ca95ca8   r8:  ffff82d080339380
(XEN) r9:  0000000000000000   r10: 00000000000000d8   r11: 00000000000000d8
(XEN) r12: ffff82d08028a3c0   r13: 0000000000000001   r14: 0000000000000002
(XEN) r15: ffff83080caa8830   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) cr3: 00000000db69f000   cr2: ffff83080ca95c98
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Valid stack range: ffff83080ca96000-ffff83080ca98000, sp=ffff83080ca95ca8, tss.esp0=ffff83080ca97fc0
(XEN) No stack overflow detected. Skipping stack trace.
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 3:
(XEN) DOUBLE FAULT -- system shutdown
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>> On 12/11/15 14:29, Atom2 wrote:
>>> Hi Andrew,
>>> thanks for your reply. Answers are inline further down.
>>>
>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>> of the
>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>
>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>> domU which is started as part of the machine booting up:
>>>>>> [...]
>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>> state (0).
>>>>>> (XEN) ************* VMCS Area **************
>>>>>> (XEN) *** Guest State ***
>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>> gh_mask=ffffffffffffffff
>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>> gh_mask=ffffffffffffffff
>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>> mode I can't seem to spot anything wrong with guest state.
>>>> odd? That will be the source of the failure.
>>>>
>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>> should not be possible to set any of them.
>>>>
>>>> I suspect that the guest has exited for emulation, and there has been a
>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>> that there is a hardware errata which allows the guest to accidentally
>>>> get it self into this condition.
>>>>
>>>> Are you able to rerun with a debug build of the hypervisor?
>>> [snip]
>>> Another question is whether prior to enabling the debug USE flag it
>>> might make sense to re-compile with gcc-4.8.5 (please see my previous
>>> list reply) to rule out any compiler related issues. Jan, Andrew -
>>> what are your thoughts?
>> First of all, check whether the compiler makes a difference on 4.5.2
> Hi Andrew,
> I changed the compiler and there was no change to the better: 
> Unfortunately the HVM domU is still crashing with a similar error 
> message as soon as it is being started.
>> If both compiles result in a guest crashing in that manner, test a debug
>> Xen to see if any assertions/errors are encountered just before the
>> guest crashes.
>>
> As the compiler did not make any difference, I enabled the debug USE 
> flag, re-compiled (using gcc-4.9.3), and rebooted using a serial console 
> to capture output. Unfortunately I did not get very far and things 
> become even stranger: This time the system did not even finnish the boot 
> process, but rather hard-stopped pretty early with a message reading 
> "Panic on CPU 3: DOUBLE FAULT -- system shutdown". The captured logfile 
> is attached as "serial log.txt".
> 
> As this happened immediately after the CPU microcode update, I thought 
> there might be a connection and disabled the microcode update. After the 
> next reboot it seemed as if the boot process got a bit further as 
> evidenced by a few more lines in the log file (those between lines 136 
> and 197 in the second log file named "serial log no ucode.txt"), but in 
> the end it finnished off with an identical error message (only the CPU # 
> was different this time, but that number seems to change between boots 
> anyways).
> 
> I hope that makes some sense to you.

Not really, other than now even more suspecting bad hardware or
something fundamentally wrong with your build. Did you retry with
a freshly built 4.5.1? Could you alternatively try with a known good
build of 4.5.2 (e.g. from osstest)?

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 5135 bytes --]

On 13/11/15 07:25, Jan Beulich wrote:
>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>> On 12/11/15 14:29, Atom2 wrote:
>>>> Hi Andrew,
>>>> thanks for your reply. Answers are inline further down.
>>>>
>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>> of the
>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>
>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>> domU which is started as part of the machine booting up:
>>>>>>> [...]
>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>> state (0).
>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>> (XEN) *** Guest State ***
>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>> gh_mask=ffffffffffffffff
>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>> gh_mask=ffffffffffffffff
>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>> odd? That will be the source of the failure.
>>>>>
>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>> should not be possible to set any of them.
>>>>>
>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>> get it self into this condition.
>>>>>
>>>>> Are you able to rerun with a debug build of the hypervisor?
>>>> [snip]
>>>> Another question is whether prior to enabling the debug USE flag it
>>>> might make sense to re-compile with gcc-4.8.5 (please see my previous
>>>> list reply) to rule out any compiler related issues. Jan, Andrew -
>>>> what are your thoughts?
>>> First of all, check whether the compiler makes a difference on 4.5.2
>> Hi Andrew,
>> I changed the compiler and there was no change to the better: 
>> Unfortunately the HVM domU is still crashing with a similar error 
>> message as soon as it is being started.
>>> If both compiles result in a guest crashing in that manner, test a debug
>>> Xen to see if any assertions/errors are encountered just before the
>>> guest crashes.
>>>
>> As the compiler did not make any difference, I enabled the debug USE 
>> flag, re-compiled (using gcc-4.9.3), and rebooted using a serial console 
>> to capture output. Unfortunately I did not get very far and things 
>> become even stranger: This time the system did not even finnish the boot 
>> process, but rather hard-stopped pretty early with a message reading 
>> "Panic on CPU 3: DOUBLE FAULT -- system shutdown". The captured logfile 
>> is attached as "serial log.txt".
>>
>> As this happened immediately after the CPU microcode update, I thought 
>> there might be a connection and disabled the microcode update. After the 
>> next reboot it seemed as if the boot process got a bit further as 
>> evidenced by a few more lines in the log file (those between lines 136 
>> and 197 in the second log file named "serial log no ucode.txt"), but in 
>> the end it finnished off with an identical error message (only the CPU # 
>> was different this time, but that number seems to change between boots 
>> anyways).
>>
>> I hope that makes some sense to you.
> Not really, other than now even more suspecting bad hardware or
> something fundamentally wrong with your build. Did you retry with
> a freshly built 4.5.1? Could you alternatively try with a known good
> build of 4.5.2 (e.g. from osstest)?

Agreed.  Double faults indicate that the exception handing entry points
are not set up in an appropriate state.  Something is definitely wrong
with either the compiled binary or the hardware.

Several questions and lines of investigation:

Is this straight Xen 4.5.1 and 2, or do Gentoo have their own patches on
top?

On repeated attempts, are the details of the double fault identical
(other than the cpu), or does it move around (i.e. always do_IRQ+0x15)

Can you boot with console_timestamps=boot on the command line in the
future.  This will put Linux-sytle timestamps on log messages.

Can you also compile in the attached patch? I haven't quite got it
suitable for inclusion upstream yet, but it will also dump the
instruction stream under the fault.

Finally, can you disassemble the xen-syms which results from the debug
build and paste the start of do_IRQ.  (i.e. `gdb xen-syms` and "disass
do_IRQ")

~Andrew

[-- Attachment #2: 0001-x86-traps-Dump-instruction-stream-in-show_execution_.patch --]
[-- Type: text/x-diff, Size: 3819 bytes --]

>From a0afc573ca47abe7f900a404d366599e5da93391 Mon Sep 17 00:00:00 2001
From: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Tue, 14 Jul 2015 16:58:49 +0100
Subject: [PATCH] x86/traps: Dump instruction stream in show_execution_state()

For first pass triage of crashes, it is useful to have the instruction
stream present, especially now that Xen binary patches itself.

A sample output now looks like:

(XEN) ----[ Xen-4.6-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d0801607e4>] default_idle+0x76/0x7b
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: ffff82d080331030   rbx: ffff83007fce8000   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: ffff82d080331b98   rdi: 0000000000000000
(XEN) rbp: ffff83007fcefef0   rsp: ffff83007fcefef0   r8:  ffff83007faf8118
(XEN) r9:  00000009983e89fd   r10: 00000009983e89fd   r11: 0000000000000246
(XEN) r12: ffff83007fd61000   r13: 00000000ffffffff   r14: ffff83007fad9000
(XEN) r15: ffff83007fae3000   cr0: 000000008005003b   cr4: 00000000000026e0
(XEN) cr3: 000000007fc9b000   cr2: 00007f70976b3fed
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen code around e008:ffff82d0801607e4 (default_idle+0x76/0x7b):
(XEN)  83 3c 10 00 75 04 fb f4 <eb> 01 fb 5d c3 55 48 89 e5 3b 3d 0d 50 12 00 72
(XEN) Xen stack trace from rsp=ffff83007fcefef0:
(XEN)    ffff83007fceff10 ffff82d080160e08 ffff82d08012c40a ffff83007faf9000
(XEN)    ffff83007fcefdd8 ffffffff81a01fd8 ffff88002f07d4c0 ffffffff81a01fd8
(XEN)    0000000000000000 ffffffff81a01e58 ffffffff81a01fd8 0000000000000246
(XEN)    00000000ffff0052 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffffffff810013aa 0000000000000001 00000000deadbeef 00000000deadbeef
(XEN)    0000010000000000 ffffffff810013aa 000000000000e033 0000000000000246
(XEN)    ffffffff81a01e40 000000000000e02b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff83007faf9000
(XEN)    0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d0801607e4>] default_idle+0x76/0x7b
(XEN)    [<ffff82d080160e08>] idle_loop+0x51/0x6e
(XEN)

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>

---
Currently limited to just hypervisor context, but it could be extended
to vcpus as well.
---
 xen/arch/x86/traps.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index e21fb78..658e0d7 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -114,6 +114,31 @@ boolean_param("ler", opt_ler);
 #define stack_words_per_line 4
 #define ESP_BEFORE_EXCEPTION(regs) ((unsigned long *)regs->rsp)
 
+static void show_code(const struct cpu_user_regs *regs)
+{
+    char insns[24];
+    unsigned int i, not_copied;
+    void *__user start_ip = (void *)regs->rip - 8;
+
+    if ( guest_mode(regs) )
+        return;
+
+    not_copied = __copy_from_user(insns, start_ip, ARRAY_SIZE(insns));
+
+    printk("Xen code around %04x:%p (%ps)%s:\n",
+           regs->cs, _p(regs->rip), _p(regs->rip),
+           !!not_copied ? " [fault on access]" : "");
+
+    for ( i = 0; i < ARRAY_SIZE(insns) - not_copied; ++i )
+    {
+        if ( (unsigned long)(start_ip + i) == regs->rip )
+            printk(" <%02x>", (unsigned char)insns[i]);
+        else
+            printk(" %02x", (unsigned char)insns[i]);
+    }
+    printk("\n");
+}
+
 static void show_guest_stack(struct vcpu *v, const struct cpu_user_regs *regs)
 {
     int i;
@@ -417,6 +442,7 @@ void show_stack_overflow(unsigned int cpu, const struct cpu_user_regs *regs)
 void show_execution_state(const struct cpu_user_regs *regs)
 {
     show_registers(regs);
+    show_code(regs);
     show_stack(regs);
 }
 
-- 
2.1.4


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1.1: Type: text/plain, Size: 8466 bytes --]

Am 13.11.15 um 11:09 schrieb Andrew Cooper:
> On 13/11/15 07:25, Jan Beulich wrote:
>>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>> Hi Andrew,
>>>>> thanks for your reply. Answers are inline further down.
>>>>>
>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>>> of the
>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>
>>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>> [...]
>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>>> state (0).
>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>> (XEN) *** Guest State ***
>>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>> odd? That will be the source of the failure.
>>>>>>
>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>>> should not be possible to set any of them.
>>>>>>
>>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>>> get it self into this condition.
>>>>>>
>>>>>> Are you able to rerun with a debug build of the hypervisor?
>>>>> [snip]
>>>>> Another question is whether prior to enabling the debug USE flag it
>>>>> might make sense to re-compile with gcc-4.8.5 (please see my previous
>>>>> list reply) to rule out any compiler related issues. Jan, Andrew -
>>>>> what are your thoughts?
>>>> First of all, check whether the compiler makes a difference on 4.5.2
>>> Hi Andrew,
>>> I changed the compiler and there was no change to the better:
>>> Unfortunately the HVM domU is still crashing with a similar error
>>> message as soon as it is being started.
>>>> If both compiles result in a guest crashing in that manner, test a debug
>>>> Xen to see if any assertions/errors are encountered just before the
>>>> guest crashes.
>>>>
>>> As the compiler did not make any difference, I enabled the debug USE
>>> flag, re-compiled (using gcc-4.9.3), and rebooted using a serial console
>>> to capture output. Unfortunately I did not get very far and things
>>> become even stranger: This time the system did not even finnish the boot
>>> process, but rather hard-stopped pretty early with a message reading
>>> "Panic on CPU 3: DOUBLE FAULT -- system shutdown". The captured logfile
>>> is attached as "serial log.txt".
>>>
>>> As this happened immediately after the CPU microcode update, I thought
>>> there might be a connection and disabled the microcode update. After the
>>> next reboot it seemed as if the boot process got a bit further as
>>> evidenced by a few more lines in the log file (those between lines 136
>>> and 197 in the second log file named "serial log no ucode.txt"), but in
>>> the end it finnished off with an identical error message (only the CPU #
>>> was different this time, but that number seems to change between boots
>>> anyways).
>>>
>>> I hope that makes some sense to you.
>> Not really, other than now even more suspecting bad hardware or
>> something fundamentally wrong with your build. Did you retry with
>> a freshly built 4.5.1? Could you alternatively try with a known good
>> build of 4.5.2 (e.g. from osstest)?
Andrew,
many thanks again for your help.
> Agreed.  Double faults indicate that the exception handing entry points
> are not set up in an appropriate state.  Something is definitely wrong
> with either the compiled binary or the hardware.
The hardware (it's a SandyBridge XEON processor with ECC RAM and 
Enterprise SATA disks) has worked for almost two years together with XEN 
and other than this issue there's also currently nothing strange (i.e. 
if I boot with a standard linux kernel, the system boots and works 
without any issues and is very stable and there are also no strange 
messages in /var/log/messages).
> Several questions and lines of investigation:
>
> Is this straight Xen 4.5.1 and 2, or do Gentoo have their own patches on
> top?
By the looks of it there are only security patches, but no gentoo 
specific patches.
> On repeated attempts, are the details of the double fault identical
> (other than the cpu), or does it move around (i.e. always do_IRQ+0x15)
It always seems to be do_IRQ+0x15 (I have made a number of boots), and 
more specifically it was always do_IRQ+0x15/0x64c. Timings varied, the 
CPU differed between boots and also the rax, rbx, rcx, rdx, rsi, rdi, 
rbp, rsp, and r8 values were diffent as were those for r15 and cr2 and 
the values next to valid stack range. I don't know whether that's of 
relevance, but I thought I'd mention it after a quick analysis of two 
serial logs.
> Can you boot with console_timestamps=boot on the command line in the
> future.  This will put Linux-sytle timestamps on log messages.
Done - see two the attached serial log files mentioned above.
> Can you also compile in the attached patch? I haven't quite got it
> suitable for inclusion upstream yet, but it will also dump the
> instruction stream under the fault.
I have added the patch, but it seems not to trigger - at least the text 
that is in the patch does not show in the serial console output; it is 
however in the (uncompressed) xen.gz file as a quick grep for texts "Xen 
code around " and " [fault on access]" confirmed (grep said: binary file 
matches).
> Finally, can you disassemble the xen-syms which results from the debug
> build and paste the start of do_IRQ.  (i.e. `gdb xen-syms` and "disass
> do_IRQ")
Please see the third file named "do_IRQ".

Furthermore I have managed to get the system to again boot up (a HVM 
domU, however, still crashes): If I turn off the debug USE flag and just 
add symbols to the XEN binary. It seems that the debug USE flag is 
probably not what I was expecting. On 
https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces it 
describes the purpose of the debug USE flag as follows (emphasis by me):

==== start excerpt from web page ====
Some ebuilds provide a *debug* USE flag. While some mistakenly use it to 
provide debug information and play with compiler flags when it is 
enabled, that is not its purpose.

_If you're trying to debug a reproduceable crash, you want to leave this 
USE flag alone, as it'll be building a different source than what you 
had before._ It is more efficient to get first a backtrace without 
changing the code, by simply emitting symbol information, and just 
afterward enable debug features to track the issue further down.

Debug features that are enabled by the USE flag include assertions, 
debug logs on screen, debug files, leak detection and extra-safe 
operations (such as scrubbing memory before use). Some of them might be 
taxing, especially for complex software or software where performance is 
an important issue.

For these reasons, please exercise caution when enabling the *debug* USE 
flag, and only consider it a last-chance card.
==== end excerpt from web page ====

Now _without_ the debug USE flag, but with debug information in the 
binary (I used splitdebug), all is back to where the problem started off 
(i.e. the system boots without issues until such time it starts a HVM 
domU which then crashes; PV domUs are working). I have attached the 
latest "xl dmesg" output with the timing information included.

I hope any of this makes sense to you.

Again many thanks and best regards

Atom2


[-- Attachment #1.2: Type: text/html, Size: 10872 bytes --]

[-- Attachment #2: serial log.1 --]
[-- Type: text/plain, Size: 15103 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=y Fri Nov 13 23:16:55 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.505109] Detected 2394.658 MHz processor.
(XEN) [    1.512873] Initing memory sharing.
(XEN) [    1.517204] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.524119] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.534311] Intel machine check reporting enabled
(XEN) [    1.539858] alt table ffff82d0802e8810 -> ffff82d0802e9830
(XEN) [    1.546192] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.554776] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.560847] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.567340] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.573578] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.579812] Intel VT-d Snoop Control not enabled.
(XEN) [    1.585450] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.591597] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.597316] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.603117] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.617545] I/O virtualisation enabled
(XEN) [    1.622124]  - Dom0 mode: Relaxed
(XEN) [    1.626281] Interrupt remapping enabled
(XEN) [    1.631100] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.637548] ENABLING IO-APIC IRQs
(XEN) [    1.641695]  -> Using old ACK method
(XEN) [    1.646648] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.855100] TSC deadline timer enabled
(XEN) [    2.242768] Platform timer is 14.318MHz HPET
(XEN) [    2.247566] 'A' pressed -> using alternative key handling
(XEN) [    2.248263] Allocated console ring of 64 KiB.
(XEN) [    2.249448] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.250472] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.251025] mwait-idle: v0.4 model 0x2a
(XEN) [    2.251543] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.252078] VMX: Supported advanced features:
(XEN) [    2.252630]  - APIC MMIO access virtualisation
(XEN) [    2.253152]  - APIC TPR shadow
(XEN) [    2.253665]  - Extended Page Tables (EPT)
(XEN) [    2.254214]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.254958]  - Virtual NMI
(XEN) [    2.255469]  - MSR direct-access bitmap
(XEN) [    2.256018]  - Unrestricted Guest
(XEN) [    2.256526] HVM: ASIDs enabled.
(XEN) [    2.257032] HVM: VMX enabled
(XEN) [    2.257571] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.258088] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.259248] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.260758] microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.262399] microcode: CPU6 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.263420] Brought up 8 CPUs
(XEN) [    2.269882] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.290536] ACPI sleep modes: S3
(XEN) [    2.291042] mcheck_poll: Machine check polling timer started.
(XEN) [    2.291574] Dom0 has maximum 792 PIRQs
(XEN) [    2.292172] *** LOADING DOMAIN 0 ***
(XEN) [    2.475265] elf_parse_binary: phdr: paddr=0x1000000 memsz=0x65033b
(XEN) [    2.475826] elf_parse_binary: phdr: paddr=0x1651000 memsz=0x1e5000
(XEN) [    2.476350] elf_parse_binary: phdr: paddr=0x1836000 memsz=0x98000
(XEN) [    2.476874] elf_parse_binary: phdr: paddr=0x18ce000 memsz=0x1000
(XEN) [    2.477431] elf_parse_binary: phdr: paddr=0x18cf000 memsz=0x13c98
(XEN) [    2.477954] elf_parse_binary: phdr: paddr=0x18e3000 memsz=0x3d000
(XEN) [    2.478477] elf_parse_binary: phdr: paddr=0x1920000 memsz=0x1120
(XEN) [    2.479254] elf_parse_binary: phdr: paddr=0x1922000 memsz=0x2de000
(XEN) [    2.479778] elf_parse_binary: memory: 0x1000000 -> 0x1c00000
(XEN) [    2.480304] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [    2.480856] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [    2.481374] elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) [    2.481891] elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) [    2.482447] elf_xen_parse_note: ENTRY = 0xffffffff818e31f0
(XEN) [    2.482967] elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) [    2.483494] elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) [    2.485067] elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) [    2.485587] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [    2.486102] elf_xen_parse_note: LOADER = "generic"
(XEN) [    2.486652] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [    2.487172] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [    2.487688] elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) [    2.488239] elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) [    2.488761] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [    2.489277] elf_xen_addr_calc_check: addresses:
(XEN) [    2.489825]     virt_base        = 0xffffffff80000000
(XEN) [    2.490342]     elf_paddr_offset = 0x0
(XEN) [    2.490849]     virt_offset      = 0xffffffff80000000
(XEN) [    2.491401]     virt_kstart      = 0xffffffff81000000
(XEN) [    2.491917]     virt_kend        = 0xffffffff81c00000
(XEN) [    2.492433]     virt_entry       = 0xffffffff818e31f0
(XEN) [    2.492983]     p2m_base         = 0xffffffffffffffff
(XEN) [    2.493498]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    2.494011]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    2.495485] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    2.495994]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    2.497017]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    2.497576] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    2.498084]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    2.498605]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    2.499125]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    2.499647]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    2.500168]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    2.500689]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    2.501210]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    2.501767]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    2.503127] Dom0 has maximum 8 VCPUs
(XEN) [    2.503677] elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff8165033b
(XEN) [    2.506063] elf_load_binary: phdr 1 at 0xffffffff81651000 -> 0xffffffff81836000
(XEN) [    2.507497] elf_load_binary: phdr 2 at 0xffffffff81836000 -> 0xffffffff818ce000
(XEN) [    2.508631] elf_load_binary: phdr 3 at 0xffffffff818ce000 -> 0xffffffff818cf000
(XEN) [    2.509679] elf_load_binary: phdr 4 at 0xffffffff818cf000 -> 0xffffffff818e2c98
(XEN) [    2.510709] elf_load_binary: phdr 5 at 0xffffffff818e3000 -> 0xffffffff81920000
(XEN) [    2.512031] elf_load_binary: phdr 6 at 0xffffffff81920000 -> 0xffffffff81921120
(XEN) [    2.513044] elf_load_binary: phdr 7 at 0xffffffff81922000 -> 0xffffffff819dd000
(XEN) [    2.837080] *** DOUBLE FAULT ***
(XEN) [    2.841145] ----[ Xen-4.5.2  x86_64  debug=y  Not tainted ]----
(XEN) [    2.848113] CPU:    2
(XEN) [    2.851223] RIP:    e008:[<ffff82d08017658c>] do_IRQ+0x15/0x64c
(XEN) [    2.857979] RFLAGS: 0000000000010083   CONTEXT: hypervisor
(XEN) [    2.864508] rax: 000000378c73ed00   rbx: 0000000000000002   rcx: 0000000000196bd0
(XEN) [    2.873177] rdx: 000000000006139c   rsi: 00000000960b58b0   rdi: ffff83080caa6d38
(XEN) [    2.881945] rbp: ffff83080caa6d28   rsp: ffff83080caa5ca8   r8:  ffff82d080339380
(XEN) [    2.890615] r9:  00000000000001c8   r10: 0000000000000001   r11: 0080000000000000
(XEN) [    2.899355] r12: ffff82d08028a3c0   r13: 0000000000000001   r14: 0000000000000002
(XEN) [    2.908026] r15: ffff83080ca9d060   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) [    2.916765] cr3: 00000000db69f000   cr2: ffff83080caa5c98
(XEN) [    2.923002] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [    2.931229] Valid stack range: ffff83080caa6000-ffff83080caa8000, sp=ffff83080caa5ca8, tss.esp0=ffff83080caa7fc0
(XEN) [    2.942600] No stack overflow detected. Skipping stack trace.
(XEN) [    2.949239] 
(XEN) [    2.951569] ****************************************
(XEN) [    2.957368] Panic on CPU 2:
(XEN) [    2.961094] DOUBLE FAULT -- system shutdown
(XEN) [    2.966116] ****************************************
(XEN) [    2.971914] 
(XEN) [    2.974240] Reboot in five seconds...

[-- Attachment #3: serial log.2 --]
[-- Type: text/plain, Size: 15103 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=y Fri Nov 13 23:16:55 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.510160] Detected 2394.628 MHz processor.
(XEN) [    1.518149] Initing memory sharing.
(XEN) [    1.522479] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.529324] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.539399] Intel machine check reporting enabled
(XEN) [    1.544940] alt table ffff82d0802e8810 -> ffff82d0802e9830
(XEN) [    1.551341] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.559830] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.565961] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.572376] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.578611] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.584910] Intel VT-d Snoop Control not enabled.
(XEN) [    1.590453] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.596818] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.602537] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.608250] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.622561] I/O virtualisation enabled
(XEN) [    1.627142]  - Dom0 mode: Relaxed
(XEN) [    1.631406] Interrupt remapping enabled
(XEN) [    1.636080] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.642525] ENABLING IO-APIC IRQs
(XEN) [    1.646783]  -> Using old ACK method
(XEN) [    1.651524] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.860153] TSC deadline timer enabled
(XEN) [    2.776868] Platform timer is 14.318MHz HPET
(XEN) [    2.781738] 'A' pressed -> using alternative key handling
(XEN) [    2.782434] Allocated console ring of 64 KiB.
(XEN) [    2.783614] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.784639] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.785192] mwait-idle: v0.4 model 0x2a
(XEN) [    2.785711] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.786249] VMX: Supported advanced features:
(XEN) [    2.786800]  - APIC MMIO access virtualisation
(XEN) [    2.787323]  - APIC TPR shadow
(XEN) [    2.787835]  - Extended Page Tables (EPT)
(XEN) [    2.788383]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.788906]  - Virtual NMI
(XEN) [    2.789418]  - MSR direct-access bitmap
(XEN) [    2.789966]  - Unrestricted Guest
(XEN) [    2.790473] HVM: ASIDs enabled.
(XEN) [    2.790979] HVM: VMX enabled
(XEN) [    2.791518] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.792036] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.793194] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.794705] microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.796347] microcode: CPU6 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.797562] Brought up 8 CPUs
(XEN) [    2.804020] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.824673] ACPI sleep modes: S3
(XEN) [    2.825181] mcheck_poll: Machine check polling timer started.
(XEN) [    2.825713] Dom0 has maximum 792 PIRQs
(XEN) [    2.826310] *** LOADING DOMAIN 0 ***
(XEN) [    3.009539] elf_parse_binary: phdr: paddr=0x1000000 memsz=0x65033b
(XEN) [    3.010100] elf_parse_binary: phdr: paddr=0x1651000 memsz=0x1e5000
(XEN) [    3.010625] elf_parse_binary: phdr: paddr=0x1836000 memsz=0x98000
(XEN) [    3.011148] elf_parse_binary: phdr: paddr=0x18ce000 memsz=0x1000
(XEN) [    3.011706] elf_parse_binary: phdr: paddr=0x18cf000 memsz=0x13c98
(XEN) [    3.012228] elf_parse_binary: phdr: paddr=0x18e3000 memsz=0x3d000
(XEN) [    3.012751] elf_parse_binary: phdr: paddr=0x1920000 memsz=0x1120
(XEN) [    3.013310] elf_parse_binary: phdr: paddr=0x1922000 memsz=0x2de000
(XEN) [    3.013832] elf_parse_binary: memory: 0x1000000 -> 0x1c00000
(XEN) [    3.014358] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [    3.014910] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [    3.015425] elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) [    3.015941] elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) [    3.016500] elf_xen_parse_note: ENTRY = 0xffffffff818e31f0
(XEN) [    3.017021] elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) [    3.017548] elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) [    3.019122] elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) [    3.019642] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [    3.020155] elf_xen_parse_note: LOADER = "generic"
(XEN) [    3.020706] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [    3.021443] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [    3.021960] elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) [    3.022509] elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) [    3.023032] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [    3.023548] elf_xen_addr_calc_check: addresses:
(XEN) [    3.024095]     virt_base        = 0xffffffff80000000
(XEN) [    3.024611]     elf_paddr_offset = 0x0
(XEN) [    3.025119]     virt_offset      = 0xffffffff80000000
(XEN) [    3.025671]     virt_kstart      = 0xffffffff81000000
(XEN) [    3.026187]     virt_kend        = 0xffffffff81c00000
(XEN) [    3.026702]     virt_entry       = 0xffffffff818e31f0
(XEN) [    3.027254]     p2m_base         = 0xffffffffffffffff
(XEN) [    3.027771]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    3.028285]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    3.029543] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    3.030052]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    3.031073]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    3.031634] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    3.032143]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    3.032664]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    3.033185]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    3.033706]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    3.034227]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    3.034748]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    3.035269]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    3.035826]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    3.037405] Dom0 has maximum 8 VCPUs
(XEN) [    3.037953] elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff8165033b
(XEN) [    3.040345] elf_load_binary: phdr 1 at 0xffffffff81651000 -> 0xffffffff81836000
(XEN) [    3.041778] elf_load_binary: phdr 2 at 0xffffffff81836000 -> 0xffffffff818ce000
(XEN) [    3.042915] elf_load_binary: phdr 3 at 0xffffffff818ce000 -> 0xffffffff818cf000
(XEN) [    3.043966] elf_load_binary: phdr 4 at 0xffffffff818cf000 -> 0xffffffff818e2c98
(XEN) [    3.044995] elf_load_binary: phdr 5 at 0xffffffff818e3000 -> 0xffffffff81920000
(XEN) [    3.046091] elf_load_binary: phdr 6 at 0xffffffff81920000 -> 0xffffffff81921120
(XEN) [    3.047106] elf_load_binary: phdr 7 at 0xffffffff81922000 -> 0xffffffff819dd000
(XEN) [    3.371151] *** DOUBLE FAULT ***
(XEN) [    3.375352] ----[ Xen-4.5.2  x86_64  debug=y  Not tainted ]----
(XEN) [    3.382116] CPU:    4
(XEN) [    3.385231] RIP:    e008:[<ffff82d08017658c>] do_IRQ+0x15/0x64c
(XEN) [    3.392055] RFLAGS: 0000000000010083   CONTEXT: hypervisor
(XEN) [    3.398387] rax: ffff82d08011f917   rbx: 0000000000000004   rcx: 0000000000000004
(XEN) [    3.407116] rdx: 000000378c756d00   rsi: 0000000000000004   rdi: ffff83080ca86d58
(XEN) [    3.415790] rbp: ffff83080ca86d48   rsp: ffff83080ca85cc8   r8:  0000000000000004
(XEN) [    3.424587] r9:  00000000000001c8   r10: 0000000000000001   r11: 0080000000000000
(XEN) [    3.433259] r12: ffff82d08028a3c0   r13: 0000000000000001   r14: 0000000000000002
(XEN) [    3.442047] r15: ffff83080ca9a060   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) [    3.450727] cr3: 00000000db69f000   cr2: ffff83080ca85cb8
(XEN) [    3.457090] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [    3.465244] Valid stack range: ffff83080ca86000-ffff83080ca88000, sp=ffff83080ca85cc8, tss.esp0=ffff83080ca87fc0
(XEN) [    3.476670] No stack overflow detected. Skipping stack trace.
(XEN) [    3.483255] 
(XEN) [    3.485792] ****************************************
(XEN) [    3.491595] Panic on CPU 4:
(XEN) [    3.495225] DOUBLE FAULT -- system shutdown
(XEN) [    3.500249] ****************************************
(XEN) [    3.506149] 
(XEN) [    3.508477] Reboot in five seconds...

[-- Attachment #4: do_IRQ --]
[-- Type: text/plain, Size: 40367 bytes --]

Dump of assembler code for function do_IRQ:
   0xffff82d080176577 <+0>:     add    %al,(%rax)
   0xffff82d080176579 <+2>:     add    %al,(%rax)
   0xffff82d08017657b <+4>:     add    %al,(%rax)
   0xffff82d08017657d <+6>:     add    %al,(%rax)
   0xffff82d08017657f <+8>:     add    %al,(%rax)
   0xffff82d080176581 <+10>:    add    %al,(%rax)
   0xffff82d080176583 <+12>:    add    %al,(%rax)
   0xffff82d080176585 <+14>:    add    %al,(%rax)
   0xffff82d080176587 <+16>:    add    %al,(%rax)
   0xffff82d080176589 <+18>:    add    %al,(%rax)
   0xffff82d08017658b <+20>:    add    %al,(%rax)
   0xffff82d08017658d <+22>:    add    %al,(%rax)
   0xffff82d08017658f <+24>:    add    %al,(%rax)
   0xffff82d080176591 <+26>:    add    %al,(%rax)
   0xffff82d080176593 <+28>:    add    %al,(%rax)
   0xffff82d080176595 <+30>:    add    %al,(%rax)
   0xffff82d080176597 <+32>:    add    %al,(%rax)
   0xffff82d080176599 <+34>:    add    %al,(%rax)
   0xffff82d08017659b <+36>:    add    %al,(%rax)
   0xffff82d08017659d <+38>:    add    %al,(%rax)
   0xffff82d08017659f <+40>:    add    %al,(%rax)
   0xffff82d0801765a1 <+42>:    add    %al,(%rax)
   0xffff82d0801765a3 <+44>:    add    %al,(%rax)
   0xffff82d0801765a5 <+46>:    add    %al,(%rax)
   0xffff82d0801765a7 <+48>:    add    %al,(%rax)
   0xffff82d0801765a9 <+50>:    add    %al,(%rax)
   0xffff82d0801765ab <+52>:    add    %al,(%rax)
   0xffff82d0801765ad <+54>:    add    %al,(%rax)
   0xffff82d0801765af <+56>:    add    %al,(%rax)
   0xffff82d0801765b1 <+58>:    add    %al,(%rax)
   0xffff82d0801765b3 <+60>:    add    %al,(%rax)
   0xffff82d0801765b5 <+62>:    add    %al,(%rax)
   0xffff82d0801765b7 <+64>:    add    %al,(%rax)
   0xffff82d0801765b9 <+66>:    add    %al,(%rax)
   0xffff82d0801765bb <+68>:    add    %al,(%rax)
   0xffff82d0801765bd <+70>:    add    %al,(%rax)
   0xffff82d0801765bf <+72>:    add    %al,(%rax)
   0xffff82d0801765c1 <+74>:    add    %al,(%rax)
   0xffff82d0801765c3 <+76>:    add    %al,(%rax)
   0xffff82d0801765c5 <+78>:    add    %al,(%rax)
   0xffff82d0801765c7 <+80>:    add    %al,(%rax)
   0xffff82d0801765c9 <+82>:    add    %al,(%rax)
   0xffff82d0801765cb <+84>:    add    %al,(%rax)
   0xffff82d0801765cd <+86>:    add    %al,(%rax)
   0xffff82d0801765cf <+88>:    add    %al,(%rax)
   0xffff82d0801765d1 <+90>:    add    %al,(%rax)
   0xffff82d0801765d3 <+92>:    add    %al,(%rax)
   0xffff82d0801765d5 <+94>:    add    %al,(%rax)
   0xffff82d0801765d7 <+96>:    add    %al,(%rax)
   0xffff82d0801765d9 <+98>:    add    %al,(%rax)
   0xffff82d0801765db <+100>:   add    %al,(%rax)
   0xffff82d0801765dd <+102>:   add    %al,(%rax)
   0xffff82d0801765df <+104>:   add    %al,(%rax)
   0xffff82d0801765e1 <+106>:   add    %al,(%rax)
   0xffff82d0801765e3 <+108>:   add    %al,(%rax)
   0xffff82d0801765e5 <+110>:   add    %al,(%rax)
   0xffff82d0801765e7 <+112>:   add    %al,(%rax)
   0xffff82d0801765e9 <+114>:   add    %al,(%rax)
   0xffff82d0801765eb <+116>:   add    %al,(%rax)
   0xffff82d0801765ed <+118>:   add    %al,(%rax)
   0xffff82d0801765ef <+120>:   add    %al,(%rax)
   0xffff82d0801765f1 <+122>:   add    %al,(%rax)
   0xffff82d0801765f3 <+124>:   add    %al,(%rax)
   0xffff82d0801765f5 <+126>:   add    %al,(%rax)
   0xffff82d0801765f7 <+128>:   add    %al,(%rax)
   0xffff82d0801765f9 <+130>:   add    %al,(%rax)
   0xffff82d0801765fb <+132>:   add    %al,(%rax)
   0xffff82d0801765fd <+134>:   add    %al,(%rax)
   0xffff82d0801765ff <+136>:   add    %al,(%rax)
   0xffff82d080176601 <+138>:   add    %al,(%rax)
   0xffff82d080176603 <+140>:   add    %al,(%rax)
   0xffff82d080176605 <+142>:   add    %al,(%rax)
   0xffff82d080176607 <+144>:   add    %al,(%rax)
   0xffff82d080176609 <+146>:   add    %al,(%rax)
   0xffff82d08017660b <+148>:   add    %al,(%rax)
   0xffff82d08017660d <+150>:   add    %al,(%rax)
   0xffff82d08017660f <+152>:   add    %al,(%rax)
   0xffff82d080176611 <+154>:   add    %al,(%rax)
   0xffff82d080176613 <+156>:   add    %al,(%rax)
   0xffff82d080176615 <+158>:   add    %al,(%rax)
   0xffff82d080176617 <+160>:   add    %al,(%rax)
   0xffff82d080176619 <+162>:   add    %al,(%rax)
   0xffff82d08017661b <+164>:   add    %al,(%rax)
   0xffff82d08017661d <+166>:   add    %al,(%rax)
   0xffff82d08017661f <+168>:   add    %al,(%rax)
   0xffff82d080176621 <+170>:   add    %al,(%rax)
   0xffff82d080176623 <+172>:   add    %al,(%rax)
   0xffff82d080176625 <+174>:   add    %al,(%rax)
   0xffff82d080176627 <+176>:   add    %al,(%rax)
   0xffff82d080176629 <+178>:   add    %al,(%rax)
   0xffff82d08017662b <+180>:   add    %al,(%rax)
   0xffff82d08017662d <+182>:   add    %al,(%rax)
   0xffff82d08017662f <+184>:   add    %al,(%rax)
   0xffff82d080176631 <+186>:   add    %al,(%rax)
   0xffff82d080176633 <+188>:   add    %al,(%rax)
   0xffff82d080176635 <+190>:   add    %al,(%rax)
   0xffff82d080176637 <+192>:   add    %al,(%rax)
   0xffff82d080176639 <+194>:   add    %al,(%rax)
   0xffff82d08017663b <+196>:   add    %al,(%rax)
   0xffff82d08017663d <+198>:   add    %al,(%rax)
   0xffff82d08017663f <+200>:   add    %al,(%rax)
   0xffff82d080176641 <+202>:   add    %al,(%rax)
   0xffff82d080176643 <+204>:   add    %al,(%rax)
   0xffff82d080176645 <+206>:   add    %al,(%rax)
   0xffff82d080176647 <+208>:   add    %al,(%rax)
   0xffff82d080176649 <+210>:   add    %al,(%rax)
   0xffff82d08017664b <+212>:   add    %al,(%rax)
   0xffff82d08017664d <+214>:   add    %al,(%rax)
   0xffff82d08017664f <+216>:   add    %al,(%rax)
   0xffff82d080176651 <+218>:   add    %al,(%rax)
   0xffff82d080176653 <+220>:   add    %al,(%rax)
   0xffff82d080176655 <+222>:   add    %al,(%rax)
   0xffff82d080176657 <+224>:   add    %al,(%rax)
   0xffff82d080176659 <+226>:   add    %al,(%rax)
   0xffff82d08017665b <+228>:   add    %al,(%rax)
   0xffff82d08017665d <+230>:   add    %al,(%rax)
   0xffff82d08017665f <+232>:   add    %al,(%rax)
   0xffff82d080176661 <+234>:   add    %al,(%rax)
   0xffff82d080176663 <+236>:   add    %al,(%rax)
   0xffff82d080176665 <+238>:   add    %al,(%rax)
   0xffff82d080176667 <+240>:   add    %al,(%rax)
   0xffff82d080176669 <+242>:   add    %al,(%rax)
   0xffff82d08017666b <+244>:   add    %al,(%rax)
   0xffff82d08017666d <+246>:   add    %al,(%rax)
   0xffff82d08017666f <+248>:   add    %al,(%rax)
   0xffff82d080176671 <+250>:   add    %al,(%rax)
   0xffff82d080176673 <+252>:   add    %al,(%rax)
   0xffff82d080176675 <+254>:   add    %al,(%rax)
   0xffff82d080176677 <+256>:   add    %al,(%rax)
   0xffff82d080176679 <+258>:   add    %al,(%rax)
   0xffff82d08017667b <+260>:   add    %al,(%rax)
   0xffff82d08017667d <+262>:   add    %al,(%rax)
   0xffff82d08017667f <+264>:   add    %al,(%rax)
   0xffff82d080176681 <+266>:   add    %al,(%rax)
   0xffff82d080176683 <+268>:   add    %al,(%rax)
   0xffff82d080176685 <+270>:   add    %al,(%rax)
   0xffff82d080176687 <+272>:   add    %al,(%rax)
   0xffff82d080176689 <+274>:   add    %al,(%rax)
   0xffff82d08017668b <+276>:   add    %al,(%rax)
   0xffff82d08017668d <+278>:   add    %al,(%rax)
   0xffff82d08017668f <+280>:   add    %al,(%rax)
   0xffff82d080176691 <+282>:   add    %al,(%rax)
   0xffff82d080176693 <+284>:   add    %al,(%rax)
   0xffff82d080176695 <+286>:   add    %al,(%rax)
   0xffff82d080176697 <+288>:   add    %al,(%rax)
   0xffff82d080176699 <+290>:   add    %al,(%rax)
   0xffff82d08017669b <+292>:   add    %al,(%rax)
   0xffff82d08017669d <+294>:   add    %al,(%rax)
   0xffff82d08017669f <+296>:   add    %al,(%rax)
   0xffff82d0801766a1 <+298>:   add    %al,(%rax)
   0xffff82d0801766a3 <+300>:   add    %al,(%rax)
   0xffff82d0801766a5 <+302>:   add    %al,(%rax)
   0xffff82d0801766a7 <+304>:   add    %al,(%rax)
   0xffff82d0801766a9 <+306>:   add    %al,(%rax)
   0xffff82d0801766ab <+308>:   add    %al,(%rax)
   0xffff82d0801766ad <+310>:   add    %al,(%rax)
   0xffff82d0801766af <+312>:   add    %al,(%rax)
   0xffff82d0801766b1 <+314>:   add    %al,(%rax)
   0xffff82d0801766b3 <+316>:   add    %al,(%rax)
   0xffff82d0801766b5 <+318>:   add    %al,(%rax)
   0xffff82d0801766b7 <+320>:   add    %al,(%rax)
   0xffff82d0801766b9 <+322>:   add    %al,(%rax)
   0xffff82d0801766bb <+324>:   add    %al,(%rax)
   0xffff82d0801766bd <+326>:   add    %al,(%rax)
   0xffff82d0801766bf <+328>:   add    %al,(%rax)
   0xffff82d0801766c1 <+330>:   add    %al,(%rax)
   0xffff82d0801766c3 <+332>:   add    %al,(%rax)
   0xffff82d0801766c5 <+334>:   add    %al,(%rax)
   0xffff82d0801766c7 <+336>:   add    %al,(%rax)
   0xffff82d0801766c9 <+338>:   add    %al,(%rax)
   0xffff82d0801766cb <+340>:   add    %al,(%rax)
   0xffff82d0801766cd <+342>:   add    %al,(%rax)
   0xffff82d0801766cf <+344>:   add    %al,(%rax)
   0xffff82d0801766d1 <+346>:   add    %al,(%rax)
   0xffff82d0801766d3 <+348>:   add    %al,(%rax)
   0xffff82d0801766d5 <+350>:   add    %al,(%rax)
   0xffff82d0801766d7 <+352>:   add    %al,(%rax)
   0xffff82d0801766d9 <+354>:   add    %al,(%rax)
   0xffff82d0801766db <+356>:   add    %al,(%rax)
   0xffff82d0801766dd <+358>:   add    %al,(%rax)
   0xffff82d0801766df <+360>:   add    %al,(%rax)
   0xffff82d0801766e1 <+362>:   add    %al,(%rax)
   0xffff82d0801766e3 <+364>:   add    %al,(%rax)
   0xffff82d0801766e5 <+366>:   add    %al,(%rax)
   0xffff82d0801766e7 <+368>:   add    %al,(%rax)
   0xffff82d0801766e9 <+370>:   add    %al,(%rax)
   0xffff82d0801766eb <+372>:   add    %al,(%rax)
   0xffff82d0801766ed <+374>:   add    %al,(%rax)
   0xffff82d0801766ef <+376>:   add    %al,(%rax)
   0xffff82d0801766f1 <+378>:   add    %al,(%rax)
   0xffff82d0801766f3 <+380>:   add    %al,(%rax)
   0xffff82d0801766f5 <+382>:   add    %al,(%rax)
   0xffff82d0801766f7 <+384>:   add    %al,(%rax)
   0xffff82d0801766f9 <+386>:   add    %al,(%rax)
   0xffff82d0801766fb <+388>:   add    %al,(%rax)
   0xffff82d0801766fd <+390>:   add    %al,(%rax)
   0xffff82d0801766ff <+392>:   add    %al,(%rax)
   0xffff82d080176701 <+394>:   add    %al,(%rax)
   0xffff82d080176703 <+396>:   add    %al,(%rax)
   0xffff82d080176705 <+398>:   add    %al,(%rax)
   0xffff82d080176707 <+400>:   add    %al,(%rax)
   0xffff82d080176709 <+402>:   add    %al,(%rax)
   0xffff82d08017670b <+404>:   add    %al,(%rax)
   0xffff82d08017670d <+406>:   add    %al,(%rax)
   0xffff82d08017670f <+408>:   add    %al,(%rax)
   0xffff82d080176711 <+410>:   add    %al,(%rax)
   0xffff82d080176713 <+412>:   add    %al,(%rax)
   0xffff82d080176715 <+414>:   add    %al,(%rax)
   0xffff82d080176717 <+416>:   add    %al,(%rax)
   0xffff82d080176719 <+418>:   add    %al,(%rax)
   0xffff82d08017671b <+420>:   add    %al,(%rax)
   0xffff82d08017671d <+422>:   add    %al,(%rax)
   0xffff82d08017671f <+424>:   add    %al,(%rax)
   0xffff82d080176721 <+426>:   add    %al,(%rax)
   0xffff82d080176723 <+428>:   add    %al,(%rax)
   0xffff82d080176725 <+430>:   add    %al,(%rax)
   0xffff82d080176727 <+432>:   add    %al,(%rax)
   0xffff82d080176729 <+434>:   add    %al,(%rax)
   0xffff82d08017672b <+436>:   add    %al,(%rax)
   0xffff82d08017672d <+438>:   add    %al,(%rax)
   0xffff82d08017672f <+440>:   add    %al,(%rax)
   0xffff82d080176731 <+442>:   add    %al,(%rax)
   0xffff82d080176733 <+444>:   add    %al,(%rax)
   0xffff82d080176735 <+446>:   add    %al,(%rax)
   0xffff82d080176737 <+448>:   add    %al,(%rax)
   0xffff82d080176739 <+450>:   add    %al,(%rax)
   0xffff82d08017673b <+452>:   add    %al,(%rax)
   0xffff82d08017673d <+454>:   add    %al,(%rax)
   0xffff82d08017673f <+456>:   add    %al,(%rax)
   0xffff82d080176741 <+458>:   add    %al,(%rax)
   0xffff82d080176743 <+460>:   add    %al,(%rax)
   0xffff82d080176745 <+462>:   add    %al,(%rax)
   0xffff82d080176747 <+464>:   add    %al,(%rax)
   0xffff82d080176749 <+466>:   add    %al,(%rax)
   0xffff82d08017674b <+468>:   add    %al,(%rax)
   0xffff82d08017674d <+470>:   add    %al,(%rax)
   0xffff82d08017674f <+472>:   add    %al,(%rax)
   0xffff82d080176751 <+474>:   add    %al,(%rax)
   0xffff82d080176753 <+476>:   add    %al,(%rax)
   0xffff82d080176755 <+478>:   add    %al,(%rax)
   0xffff82d080176757 <+480>:   add    %al,(%rax)
   0xffff82d080176759 <+482>:   add    %al,(%rax)
   0xffff82d08017675b <+484>:   add    %al,(%rax)
   0xffff82d08017675d <+486>:   add    %al,(%rax)
   0xffff82d08017675f <+488>:   add    %al,(%rax)
   0xffff82d080176761 <+490>:   add    %al,(%rax)
   0xffff82d080176763 <+492>:   add    %al,(%rax)
   0xffff82d080176765 <+494>:   add    %al,(%rax)
   0xffff82d080176767 <+496>:   add    %al,(%rax)
   0xffff82d080176769 <+498>:   add    %al,(%rax)
   0xffff82d08017676b <+500>:   add    %al,(%rax)
   0xffff82d08017676d <+502>:   add    %al,(%rax)
   0xffff82d08017676f <+504>:   add    %al,(%rax)
   0xffff82d080176771 <+506>:   add    %al,(%rax)
   0xffff82d080176773 <+508>:   add    %al,(%rax)
   0xffff82d080176775 <+510>:   add    %al,(%rax)
   0xffff82d080176777 <+512>:   add    %al,(%rax)
   0xffff82d080176779 <+514>:   add    %al,(%rax)
   0xffff82d08017677b <+516>:   add    %al,(%rax)
   0xffff82d08017677d <+518>:   add    %al,(%rax)
   0xffff82d08017677f <+520>:   add    %al,(%rax)
   0xffff82d080176781 <+522>:   add    %al,(%rax)
   0xffff82d080176783 <+524>:   add    %al,(%rax)
   0xffff82d080176785 <+526>:   add    %al,(%rax)
   0xffff82d080176787 <+528>:   add    %al,(%rax)
   0xffff82d080176789 <+530>:   add    %al,(%rax)
   0xffff82d08017678b <+532>:   add    %al,(%rax)
   0xffff82d08017678d <+534>:   add    %al,(%rax)
   0xffff82d08017678f <+536>:   add    %al,(%rax)
   0xffff82d080176791 <+538>:   add    %al,(%rax)
   0xffff82d080176793 <+540>:   add    %al,(%rax)
   0xffff82d080176795 <+542>:   add    %al,(%rax)
   0xffff82d080176797 <+544>:   add    %al,(%rax)
   0xffff82d080176799 <+546>:   add    %al,(%rax)
   0xffff82d08017679b <+548>:   add    %al,(%rax)
   0xffff82d08017679d <+550>:   add    %al,(%rax)
   0xffff82d08017679f <+552>:   add    %al,(%rax)
   0xffff82d0801767a1 <+554>:   add    %al,(%rax)
   0xffff82d0801767a3 <+556>:   add    %al,(%rax)
   0xffff82d0801767a5 <+558>:   add    %al,(%rax)
   0xffff82d0801767a7 <+560>:   add    %al,(%rax)
   0xffff82d0801767a9 <+562>:   add    %al,(%rax)
   0xffff82d0801767ab <+564>:   add    %al,(%rax)
   0xffff82d0801767ad <+566>:   add    %al,(%rax)
   0xffff82d0801767af <+568>:   add    %al,(%rax)
   0xffff82d0801767b1 <+570>:   add    %al,(%rax)
   0xffff82d0801767b3 <+572>:   add    %al,(%rax)
   0xffff82d0801767b5 <+574>:   add    %al,(%rax)
   0xffff82d0801767b7 <+576>:   add    %al,(%rax)
   0xffff82d0801767b9 <+578>:   add    %al,(%rax)
   0xffff82d0801767bb <+580>:   add    %al,(%rax)
   0xffff82d0801767bd <+582>:   add    %al,(%rax)
   0xffff82d0801767bf <+584>:   add    %al,(%rax)
   0xffff82d0801767c1 <+586>:   add    %al,(%rax)
   0xffff82d0801767c3 <+588>:   add    %al,(%rax)
   0xffff82d0801767c5 <+590>:   add    %al,(%rax)
   0xffff82d0801767c7 <+592>:   add    %al,(%rax)
   0xffff82d0801767c9 <+594>:   add    %al,(%rax)
   0xffff82d0801767cb <+596>:   add    %al,(%rax)
   0xffff82d0801767cd <+598>:   add    %al,(%rax)
   0xffff82d0801767cf <+600>:   add    %al,(%rax)
   0xffff82d0801767d1 <+602>:   add    %al,(%rax)
   0xffff82d0801767d3 <+604>:   add    %al,(%rax)
   0xffff82d0801767d5 <+606>:   add    %al,(%rax)
   0xffff82d0801767d7 <+608>:   add    %al,(%rax)
   0xffff82d0801767d9 <+610>:   add    %al,(%rax)
   0xffff82d0801767db <+612>:   add    %al,(%rax)
   0xffff82d0801767dd <+614>:   add    %al,(%rax)
   0xffff82d0801767df <+616>:   add    %al,(%rax)
   0xffff82d0801767e1 <+618>:   add    %al,(%rax)
   0xffff82d0801767e3 <+620>:   add    %al,(%rax)
   0xffff82d0801767e5 <+622>:   add    %al,(%rax)
   0xffff82d0801767e7 <+624>:   add    %al,(%rax)
   0xffff82d0801767e9 <+626>:   add    %al,(%rax)
   0xffff82d0801767eb <+628>:   add    %al,(%rax)
   0xffff82d0801767ed <+630>:   add    %al,(%rax)
   0xffff82d0801767ef <+632>:   add    %al,(%rax)
   0xffff82d0801767f1 <+634>:   add    %al,(%rax)
   0xffff82d0801767f3 <+636>:   add    %al,(%rax)
   0xffff82d0801767f5 <+638>:   add    %al,(%rax)
   0xffff82d0801767f7 <+640>:   add    %al,(%rax)
   0xffff82d0801767f9 <+642>:   add    %al,(%rax)
   0xffff82d0801767fb <+644>:   add    %al,(%rax)
   0xffff82d0801767fd <+646>:   add    %al,(%rax)
   0xffff82d0801767ff <+648>:   add    %al,(%rax)
   0xffff82d080176801 <+650>:   add    %al,(%rax)
   0xffff82d080176803 <+652>:   add    %al,(%rax)
   0xffff82d080176805 <+654>:   add    %al,(%rax)
   0xffff82d080176807 <+656>:   add    %al,(%rax)
   0xffff82d080176809 <+658>:   add    %al,(%rax)
   0xffff82d08017680b <+660>:   add    %al,(%rax)
   0xffff82d08017680d <+662>:   add    %al,(%rax)
   0xffff82d08017680f <+664>:   add    %al,(%rax)
   0xffff82d080176811 <+666>:   add    %al,(%rax)
   0xffff82d080176813 <+668>:   add    %al,(%rax)
   0xffff82d080176815 <+670>:   add    %al,(%rax)
   0xffff82d080176817 <+672>:   add    %al,(%rax)
   0xffff82d080176819 <+674>:   add    %al,(%rax)
   0xffff82d08017681b <+676>:   add    %al,(%rax)
   0xffff82d08017681d <+678>:   add    %al,(%rax)
   0xffff82d08017681f <+680>:   add    %al,(%rax)
   0xffff82d080176821 <+682>:   add    %al,(%rax)
   0xffff82d080176823 <+684>:   add    %al,(%rax)
   0xffff82d080176825 <+686>:   add    %al,(%rax)
   0xffff82d080176827 <+688>:   add    %al,(%rax)
   0xffff82d080176829 <+690>:   add    %al,(%rax)
   0xffff82d08017682b <+692>:   add    %al,(%rax)
   0xffff82d08017682d <+694>:   add    %al,(%rax)
   0xffff82d08017682f <+696>:   add    %al,(%rax)
   0xffff82d080176831 <+698>:   add    %al,(%rax)
   0xffff82d080176833 <+700>:   add    %al,(%rax)
   0xffff82d080176835 <+702>:   add    %al,(%rax)
   0xffff82d080176837 <+704>:   add    %al,(%rax)
   0xffff82d080176839 <+706>:   add    %al,(%rax)
   0xffff82d08017683b <+708>:   add    %al,(%rax)
   0xffff82d08017683d <+710>:   add    %al,(%rax)
   0xffff82d08017683f <+712>:   add    %al,(%rax)
   0xffff82d080176841 <+714>:   add    %al,(%rax)
   0xffff82d080176843 <+716>:   add    %al,(%rax)
   0xffff82d080176845 <+718>:   add    %al,(%rax)
   0xffff82d080176847 <+720>:   add    %al,(%rax)
   0xffff82d080176849 <+722>:   add    %al,(%rax)
   0xffff82d08017684b <+724>:   add    %al,(%rax)
   0xffff82d08017684d <+726>:   add    %al,(%rax)
   0xffff82d08017684f <+728>:   add    %al,(%rax)
   0xffff82d080176851 <+730>:   add    %al,(%rax)
   0xffff82d080176853 <+732>:   add    %al,(%rax)
   0xffff82d080176855 <+734>:   add    %al,(%rax)
   0xffff82d080176857 <+736>:   add    %al,(%rax)
   0xffff82d080176859 <+738>:   add    %al,(%rax)
   0xffff82d08017685b <+740>:   add    %al,(%rax)
   0xffff82d08017685d <+742>:   add    %al,(%rax)
   0xffff82d08017685f <+744>:   add    %al,(%rax)
   0xffff82d080176861 <+746>:   add    %al,(%rax)
   0xffff82d080176863 <+748>:   add    %al,(%rax)
   0xffff82d080176865 <+750>:   add    %al,(%rax)
   0xffff82d080176867 <+752>:   add    %al,(%rax)
   0xffff82d080176869 <+754>:   add    %al,(%rax)
   0xffff82d08017686b <+756>:   add    %al,(%rax)
   0xffff82d08017686d <+758>:   add    %al,(%rax)
   0xffff82d08017686f <+760>:   add    %al,(%rax)
   0xffff82d080176871 <+762>:   add    %al,(%rax)
   0xffff82d080176873 <+764>:   add    %al,(%rax)
   0xffff82d080176875 <+766>:   add    %al,(%rax)
   0xffff82d080176877 <+768>:   add    %al,(%rax)
   0xffff82d080176879 <+770>:   add    %al,(%rax)
   0xffff82d08017687b <+772>:   add    %al,(%rax)
   0xffff82d08017687d <+774>:   add    %al,(%rax)
   0xffff82d08017687f <+776>:   add    %al,(%rax)
   0xffff82d080176881 <+778>:   add    %al,(%rax)
   0xffff82d080176883 <+780>:   add    %al,(%rax)
   0xffff82d080176885 <+782>:   add    %al,(%rax)
   0xffff82d080176887 <+784>:   add    %al,(%rax)
   0xffff82d080176889 <+786>:   add    %al,(%rax)
   0xffff82d08017688b <+788>:   add    %al,(%rax)
   0xffff82d08017688d <+790>:   add    %al,(%rax)
   0xffff82d08017688f <+792>:   add    %al,(%rax)
   0xffff82d080176891 <+794>:   add    %al,(%rax)
   0xffff82d080176893 <+796>:   add    %al,(%rax)
   0xffff82d080176895 <+798>:   add    %al,(%rax)
   0xffff82d080176897 <+800>:   add    %al,(%rax)
   0xffff82d080176899 <+802>:   add    %al,(%rax)
   0xffff82d08017689b <+804>:   add    %al,(%rax)
   0xffff82d08017689d <+806>:   add    %al,(%rax)
   0xffff82d08017689f <+808>:   add    %al,(%rax)
   0xffff82d0801768a1 <+810>:   add    %al,(%rax)
   0xffff82d0801768a3 <+812>:   add    %al,(%rax)
   0xffff82d0801768a5 <+814>:   add    %al,(%rax)
   0xffff82d0801768a7 <+816>:   add    %al,(%rax)
   0xffff82d0801768a9 <+818>:   add    %al,(%rax)
   0xffff82d0801768ab <+820>:   add    %al,(%rax)
   0xffff82d0801768ad <+822>:   add    %al,(%rax)
   0xffff82d0801768af <+824>:   add    %al,(%rax)
   0xffff82d0801768b1 <+826>:   add    %al,(%rax)
   0xffff82d0801768b3 <+828>:   add    %al,(%rax)
   0xffff82d0801768b5 <+830>:   add    %al,(%rax)
   0xffff82d0801768b7 <+832>:   add    %al,(%rax)
   0xffff82d0801768b9 <+834>:   add    %al,(%rax)
   0xffff82d0801768bb <+836>:   add    %al,(%rax)
   0xffff82d0801768bd <+838>:   add    %al,(%rax)
   0xffff82d0801768bf <+840>:   add    %al,(%rax)
   0xffff82d0801768c1 <+842>:   add    %al,(%rax)
   0xffff82d0801768c3 <+844>:   add    %al,(%rax)
   0xffff82d0801768c5 <+846>:   add    %al,(%rax)
   0xffff82d0801768c7 <+848>:   add    %al,(%rax)
   0xffff82d0801768c9 <+850>:   add    %al,(%rax)
   0xffff82d0801768cb <+852>:   add    %al,(%rax)
   0xffff82d0801768cd <+854>:   add    %al,(%rax)
   0xffff82d0801768cf <+856>:   add    %al,(%rax)
   0xffff82d0801768d1 <+858>:   add    %al,(%rax)
   0xffff82d0801768d3 <+860>:   add    %al,(%rax)
   0xffff82d0801768d5 <+862>:   add    %al,(%rax)
   0xffff82d0801768d7 <+864>:   add    %al,(%rax)
   0xffff82d0801768d9 <+866>:   add    %al,(%rax)
   0xffff82d0801768db <+868>:   add    %al,(%rax)
   0xffff82d0801768dd <+870>:   add    %al,(%rax)
   0xffff82d0801768df <+872>:   add    %al,(%rax)
   0xffff82d0801768e1 <+874>:   add    %al,(%rax)
   0xffff82d0801768e3 <+876>:   add    %al,(%rax)
   0xffff82d0801768e5 <+878>:   add    %al,(%rax)
   0xffff82d0801768e7 <+880>:   add    %al,(%rax)
   0xffff82d0801768e9 <+882>:   add    %al,(%rax)
   0xffff82d0801768eb <+884>:   add    %al,(%rax)
   0xffff82d0801768ed <+886>:   add    %al,(%rax)
   0xffff82d0801768ef <+888>:   add    %al,(%rax)
   0xffff82d0801768f1 <+890>:   add    %al,(%rax)
   0xffff82d0801768f3 <+892>:   add    %al,(%rax)
   0xffff82d0801768f5 <+894>:   add    %al,(%rax)
   0xffff82d0801768f7 <+896>:   add    %al,(%rax)
   0xffff82d0801768f9 <+898>:   add    %al,(%rax)
   0xffff82d0801768fb <+900>:   add    %al,(%rax)
   0xffff82d0801768fd <+902>:   add    %al,(%rax)
   0xffff82d0801768ff <+904>:   add    %al,(%rax)
   0xffff82d080176901 <+906>:   add    %al,(%rax)
   0xffff82d080176903 <+908>:   add    %al,(%rax)
   0xffff82d080176905 <+910>:   add    %al,(%rax)
   0xffff82d080176907 <+912>:   add    %al,(%rax)
   0xffff82d080176909 <+914>:   add    %al,(%rax)
   0xffff82d08017690b <+916>:   add    %al,(%rax)
   0xffff82d08017690d <+918>:   add    %al,(%rax)
   0xffff82d08017690f <+920>:   add    %al,(%rax)
   0xffff82d080176911 <+922>:   add    %al,(%rax)
   0xffff82d080176913 <+924>:   add    %al,(%rax)
   0xffff82d080176915 <+926>:   add    %al,(%rax)
   0xffff82d080176917 <+928>:   add    %al,(%rax)
   0xffff82d080176919 <+930>:   add    %al,(%rax)
   0xffff82d08017691b <+932>:   add    %al,(%rax)
   0xffff82d08017691d <+934>:   add    %al,(%rax)
   0xffff82d08017691f <+936>:   add    %al,(%rax)
   0xffff82d080176921 <+938>:   add    %al,(%rax)
   0xffff82d080176923 <+940>:   add    %al,(%rax)
   0xffff82d080176925 <+942>:   add    %al,(%rax)
   0xffff82d080176927 <+944>:   add    %al,(%rax)
   0xffff82d080176929 <+946>:   add    %al,(%rax)
   0xffff82d08017692b <+948>:   add    %al,(%rax)
   0xffff82d08017692d <+950>:   add    %al,(%rax)
   0xffff82d08017692f <+952>:   add    %al,(%rax)
   0xffff82d080176931 <+954>:   add    %al,(%rax)
   0xffff82d080176933 <+956>:   add    %al,(%rax)
   0xffff82d080176935 <+958>:   add    %al,(%rax)
   0xffff82d080176937 <+960>:   add    %al,(%rax)
   0xffff82d080176939 <+962>:   add    %al,(%rax)
   0xffff82d08017693b <+964>:   add    %al,(%rax)
   0xffff82d08017693d <+966>:   add    %al,(%rax)
   0xffff82d08017693f <+968>:   add    %al,(%rax)
   0xffff82d080176941 <+970>:   add    %al,(%rax)
   0xffff82d080176943 <+972>:   add    %al,(%rax)
   0xffff82d080176945 <+974>:   add    %al,(%rax)
   0xffff82d080176947 <+976>:   add    %al,(%rax)
   0xffff82d080176949 <+978>:   add    %al,(%rax)
   0xffff82d08017694b <+980>:   add    %al,(%rax)
   0xffff82d08017694d <+982>:   add    %al,(%rax)
   0xffff82d08017694f <+984>:   add    %al,(%rax)
   0xffff82d080176951 <+986>:   add    %al,(%rax)
   0xffff82d080176953 <+988>:   add    %al,(%rax)
   0xffff82d080176955 <+990>:   add    %al,(%rax)
   0xffff82d080176957 <+992>:   add    %al,(%rax)
   0xffff82d080176959 <+994>:   add    %al,(%rax)
   0xffff82d08017695b <+996>:   add    %al,(%rax)
   0xffff82d08017695d <+998>:   add    %al,(%rax)
   0xffff82d08017695f <+1000>:  add    %al,(%rax)
   0xffff82d080176961 <+1002>:  add    %al,(%rax)
   0xffff82d080176963 <+1004>:  add    %al,(%rax)
   0xffff82d080176965 <+1006>:  add    %al,(%rax)
   0xffff82d080176967 <+1008>:  add    %al,(%rax)
   0xffff82d080176969 <+1010>:  add    %al,(%rax)
   0xffff82d08017696b <+1012>:  add    %al,(%rax)
   0xffff82d08017696d <+1014>:  add    %al,(%rax)
   0xffff82d08017696f <+1016>:  add    %al,(%rax)
   0xffff82d080176971 <+1018>:  add    %al,(%rax)
   0xffff82d080176973 <+1020>:  add    %al,(%rax)
   0xffff82d080176975 <+1022>:  add    %al,(%rax)
   0xffff82d080176977 <+1024>:  add    %al,(%rax)
   0xffff82d080176979 <+1026>:  add    %al,(%rax)
   0xffff82d08017697b <+1028>:  add    %al,(%rax)
   0xffff82d08017697d <+1030>:  add    %al,(%rax)
   0xffff82d08017697f <+1032>:  add    %al,(%rax)
   0xffff82d080176981 <+1034>:  add    %al,(%rax)
   0xffff82d080176983 <+1036>:  add    %al,(%rax)
   0xffff82d080176985 <+1038>:  add    %al,(%rax)
   0xffff82d080176987 <+1040>:  add    %al,(%rax)
   0xffff82d080176989 <+1042>:  add    %al,(%rax)
   0xffff82d08017698b <+1044>:  add    %al,(%rax)
   0xffff82d08017698d <+1046>:  add    %al,(%rax)
   0xffff82d08017698f <+1048>:  add    %al,(%rax)
   0xffff82d080176991 <+1050>:  add    %al,(%rax)
   0xffff82d080176993 <+1052>:  add    %al,(%rax)
   0xffff82d080176995 <+1054>:  add    %al,(%rax)
   0xffff82d080176997 <+1056>:  add    %al,(%rax)
   0xffff82d080176999 <+1058>:  add    %al,(%rax)
   0xffff82d08017699b <+1060>:  add    %al,(%rax)
   0xffff82d08017699d <+1062>:  add    %al,(%rax)
   0xffff82d08017699f <+1064>:  add    %al,(%rax)
   0xffff82d0801769a1 <+1066>:  add    %al,(%rax)
   0xffff82d0801769a3 <+1068>:  add    %al,(%rax)
   0xffff82d0801769a5 <+1070>:  add    %al,(%rax)
   0xffff82d0801769a7 <+1072>:  add    %al,(%rax)
   0xffff82d0801769a9 <+1074>:  add    %al,(%rax)
   0xffff82d0801769ab <+1076>:  add    %al,(%rax)
   0xffff82d0801769ad <+1078>:  add    %al,(%rax)
   0xffff82d0801769af <+1080>:  add    %al,(%rax)
   0xffff82d0801769b1 <+1082>:  add    %al,(%rax)
   0xffff82d0801769b3 <+1084>:  add    %al,(%rax)
   0xffff82d0801769b5 <+1086>:  add    %al,(%rax)
   0xffff82d0801769b7 <+1088>:  add    %al,(%rax)
   0xffff82d0801769b9 <+1090>:  add    %al,(%rax)
   0xffff82d0801769bb <+1092>:  add    %al,(%rax)
   0xffff82d0801769bd <+1094>:  add    %al,(%rax)
   0xffff82d0801769bf <+1096>:  add    %al,(%rax)
   0xffff82d0801769c1 <+1098>:  add    %al,(%rax)
   0xffff82d0801769c3 <+1100>:  add    %al,(%rax)
   0xffff82d0801769c5 <+1102>:  add    %al,(%rax)
   0xffff82d0801769c7 <+1104>:  add    %al,(%rax)
   0xffff82d0801769c9 <+1106>:  add    %al,(%rax)
   0xffff82d0801769cb <+1108>:  add    %al,(%rax)
   0xffff82d0801769cd <+1110>:  add    %al,(%rax)
   0xffff82d0801769cf <+1112>:  add    %al,(%rax)
   0xffff82d0801769d1 <+1114>:  add    %al,(%rax)
   0xffff82d0801769d3 <+1116>:  add    %al,(%rax)
   0xffff82d0801769d5 <+1118>:  add    %al,(%rax)
   0xffff82d0801769d7 <+1120>:  add    %al,(%rax)
   0xffff82d0801769d9 <+1122>:  add    %al,(%rax)
   0xffff82d0801769db <+1124>:  add    %al,(%rax)
   0xffff82d0801769dd <+1126>:  add    %al,(%rax)
   0xffff82d0801769df <+1128>:  add    %al,(%rax)
   0xffff82d0801769e1 <+1130>:  add    %al,(%rax)
   0xffff82d0801769e3 <+1132>:  add    %al,(%rax)
   0xffff82d0801769e5 <+1134>:  add    %al,(%rax)
   0xffff82d0801769e7 <+1136>:  add    %al,(%rax)
   0xffff82d0801769e9 <+1138>:  add    %al,(%rax)
   0xffff82d0801769eb <+1140>:  add    %al,(%rax)
   0xffff82d0801769ed <+1142>:  add    %al,(%rax)
   0xffff82d0801769ef <+1144>:  add    %al,(%rax)
   0xffff82d0801769f1 <+1146>:  add    %al,(%rax)
   0xffff82d0801769f3 <+1148>:  add    %al,(%rax)
   0xffff82d0801769f5 <+1150>:  add    %al,(%rax)
   0xffff82d0801769f7 <+1152>:  add    %al,(%rax)
   0xffff82d0801769f9 <+1154>:  add    %al,(%rax)
   0xffff82d0801769fb <+1156>:  add    %al,(%rax)
   0xffff82d0801769fd <+1158>:  add    %al,(%rax)
   0xffff82d0801769ff <+1160>:  add    %al,(%rax)
   0xffff82d080176a01 <+1162>:  add    %al,(%rax)
   0xffff82d080176a03 <+1164>:  add    %al,(%rax)
   0xffff82d080176a05 <+1166>:  add    %al,(%rax)
   0xffff82d080176a07 <+1168>:  add    %al,(%rax)
   0xffff82d080176a09 <+1170>:  add    %al,(%rax)
   0xffff82d080176a0b <+1172>:  add    %al,(%rax)
   0xffff82d080176a0d <+1174>:  add    %al,(%rax)
   0xffff82d080176a0f <+1176>:  add    %al,(%rax)
   0xffff82d080176a11 <+1178>:  add    %al,(%rax)
   0xffff82d080176a13 <+1180>:  add    %al,(%rax)
   0xffff82d080176a15 <+1182>:  add    %al,(%rax)
   0xffff82d080176a17 <+1184>:  add    %al,(%rax)
   0xffff82d080176a19 <+1186>:  add    %al,(%rax)
   0xffff82d080176a1b <+1188>:  add    %al,(%rax)
   0xffff82d080176a1d <+1190>:  add    %al,(%rax)
   0xffff82d080176a1f <+1192>:  add    %al,(%rax)
   0xffff82d080176a21 <+1194>:  add    %al,(%rax)
   0xffff82d080176a23 <+1196>:  add    %al,(%rax)
   0xffff82d080176a25 <+1198>:  add    %al,(%rax)
   0xffff82d080176a27 <+1200>:  add    %al,(%rax)
   0xffff82d080176a29 <+1202>:  add    %al,(%rax)
   0xffff82d080176a2b <+1204>:  add    %al,(%rax)
   0xffff82d080176a2d <+1206>:  add    %al,(%rax)
   0xffff82d080176a2f <+1208>:  add    %al,(%rax)
   0xffff82d080176a31 <+1210>:  add    %al,(%rax)
   0xffff82d080176a33 <+1212>:  add    %al,(%rax)
   0xffff82d080176a35 <+1214>:  add    %al,(%rax)
   0xffff82d080176a37 <+1216>:  add    %al,(%rax)
   0xffff82d080176a39 <+1218>:  add    %al,(%rax)
   0xffff82d080176a3b <+1220>:  add    %al,(%rax)
   0xffff82d080176a3d <+1222>:  add    %al,(%rax)
   0xffff82d080176a3f <+1224>:  add    %al,(%rax)
   0xffff82d080176a41 <+1226>:  add    %al,(%rax)
   0xffff82d080176a43 <+1228>:  add    %al,(%rax)
   0xffff82d080176a45 <+1230>:  add    %al,(%rax)
   0xffff82d080176a47 <+1232>:  add    %al,(%rax)
   0xffff82d080176a49 <+1234>:  add    %al,(%rax)
   0xffff82d080176a4b <+1236>:  add    %al,(%rax)
   0xffff82d080176a4d <+1238>:  add    %al,(%rax)
   0xffff82d080176a4f <+1240>:  add    %al,(%rax)
   0xffff82d080176a51 <+1242>:  add    %al,(%rax)
   0xffff82d080176a53 <+1244>:  add    %al,(%rax)
   0xffff82d080176a55 <+1246>:  add    %al,(%rax)
   0xffff82d080176a57 <+1248>:  add    %al,(%rax)
   0xffff82d080176a59 <+1250>:  add    %al,(%rax)
   0xffff82d080176a5b <+1252>:  add    %al,(%rax)
   0xffff82d080176a5d <+1254>:  add    %al,(%rax)
   0xffff82d080176a5f <+1256>:  add    %al,(%rax)
   0xffff82d080176a61 <+1258>:  add    %al,(%rax)
   0xffff82d080176a63 <+1260>:  add    %al,(%rax)
   0xffff82d080176a65 <+1262>:  add    %al,(%rax)
   0xffff82d080176a67 <+1264>:  add    %al,(%rax)
   0xffff82d080176a69 <+1266>:  add    %al,(%rax)
   0xffff82d080176a6b <+1268>:  add    %al,(%rax)
   0xffff82d080176a6d <+1270>:  add    %al,(%rax)
   0xffff82d080176a6f <+1272>:  add    %al,(%rax)
   0xffff82d080176a71 <+1274>:  add    %al,(%rax)
   0xffff82d080176a73 <+1276>:  add    %al,(%rax)
   0xffff82d080176a75 <+1278>:  add    %al,(%rax)
   0xffff82d080176a77 <+1280>:  add    %al,(%rax)
   0xffff82d080176a79 <+1282>:  add    %al,(%rax)
   0xffff82d080176a7b <+1284>:  add    %al,(%rax)
   0xffff82d080176a7d <+1286>:  add    %al,(%rax)
   0xffff82d080176a7f <+1288>:  add    %al,(%rax)
   0xffff82d080176a81 <+1290>:  add    %al,(%rax)
   0xffff82d080176a83 <+1292>:  add    %al,(%rax)
   0xffff82d080176a85 <+1294>:  add    %al,(%rax)
   0xffff82d080176a87 <+1296>:  add    %al,(%rax)
   0xffff82d080176a89 <+1298>:  add    %al,(%rax)
   0xffff82d080176a8b <+1300>:  add    %al,(%rax)
   0xffff82d080176a8d <+1302>:  add    %al,(%rax)
   0xffff82d080176a8f <+1304>:  add    %al,(%rax)
   0xffff82d080176a91 <+1306>:  add    %al,(%rax)
   0xffff82d080176a93 <+1308>:  add    %al,(%rax)
   0xffff82d080176a95 <+1310>:  add    %al,(%rax)
   0xffff82d080176a97 <+1312>:  add    %al,(%rax)
   0xffff82d080176a99 <+1314>:  add    %al,(%rax)
   0xffff82d080176a9b <+1316>:  add    %al,(%rax)
   0xffff82d080176a9d <+1318>:  add    %al,(%rax)
   0xffff82d080176a9f <+1320>:  add    %al,(%rax)
   0xffff82d080176aa1 <+1322>:  add    %al,(%rax)
   0xffff82d080176aa3 <+1324>:  add    %al,(%rax)
   0xffff82d080176aa5 <+1326>:  add    %al,(%rax)
   0xffff82d080176aa7 <+1328>:  add    %al,(%rax)
   0xffff82d080176aa9 <+1330>:  add    %al,(%rax)
   0xffff82d080176aab <+1332>:  add    %al,(%rax)
   0xffff82d080176aad <+1334>:  add    %al,(%rax)
   0xffff82d080176aaf <+1336>:  add    %al,(%rax)
   0xffff82d080176ab1 <+1338>:  add    %al,(%rax)
   0xffff82d080176ab3 <+1340>:  add    %al,(%rax)
   0xffff82d080176ab5 <+1342>:  add    %al,(%rax)
   0xffff82d080176ab7 <+1344>:  add    %al,(%rax)
   0xffff82d080176ab9 <+1346>:  add    %al,(%rax)
   0xffff82d080176abb <+1348>:  add    %al,(%rax)
   0xffff82d080176abd <+1350>:  add    %al,(%rax)
   0xffff82d080176abf <+1352>:  add    %al,(%rax)
   0xffff82d080176ac1 <+1354>:  add    %al,(%rax)
   0xffff82d080176ac3 <+1356>:  add    %al,(%rax)
   0xffff82d080176ac5 <+1358>:  add    %al,(%rax)
   0xffff82d080176ac7 <+1360>:  add    %al,(%rax)
   0xffff82d080176ac9 <+1362>:  add    %al,(%rax)
   0xffff82d080176acb <+1364>:  add    %al,(%rax)
   0xffff82d080176acd <+1366>:  add    %al,(%rax)
   0xffff82d080176acf <+1368>:  add    %al,(%rax)
   0xffff82d080176ad1 <+1370>:  add    %al,(%rax)
   0xffff82d080176ad3 <+1372>:  add    %al,(%rax)
   0xffff82d080176ad5 <+1374>:  add    %al,(%rax)
   0xffff82d080176ad7 <+1376>:  add    %al,(%rax)
   0xffff82d080176ad9 <+1378>:  add    %al,(%rax)
   0xffff82d080176adb <+1380>:  add    %al,(%rax)
   0xffff82d080176add <+1382>:  add    %al,(%rax)
   0xffff82d080176adf <+1384>:  add    %al,(%rax)
   0xffff82d080176ae1 <+1386>:  add    %al,(%rax)
   0xffff82d080176ae3 <+1388>:  add    %al,(%rax)
   0xffff82d080176ae5 <+1390>:  add    %al,(%rax)
   0xffff82d080176ae7 <+1392>:  add    %al,(%rax)
   0xffff82d080176ae9 <+1394>:  add    %al,(%rax)
   0xffff82d080176aeb <+1396>:  add    %al,(%rax)
   0xffff82d080176aed <+1398>:  add    %al,(%rax)
   0xffff82d080176aef <+1400>:  add    %al,(%rax)
   0xffff82d080176af1 <+1402>:  add    %al,(%rax)
   0xffff82d080176af3 <+1404>:  add    %al,(%rax)
   0xffff82d080176af5 <+1406>:  add    %al,(%rax)
   0xffff82d080176af7 <+1408>:  add    %al,(%rax)
   0xffff82d080176af9 <+1410>:  add    %al,(%rax)
   0xffff82d080176afb <+1412>:  add    %al,(%rax)
   0xffff82d080176afd <+1414>:  add    %al,(%rax)
   0xffff82d080176aff <+1416>:  add    %al,(%rax)
   0xffff82d080176b01 <+1418>:  add    %al,(%rax)
   0xffff82d080176b03 <+1420>:  add    %al,(%rax)
   0xffff82d080176b05 <+1422>:  add    %al,(%rax)
   0xffff82d080176b07 <+1424>:  add    %al,(%rax)
   0xffff82d080176b09 <+1426>:  add    %al,(%rax)
   0xffff82d080176b0b <+1428>:  add    %al,(%rax)
   0xffff82d080176b0d <+1430>:  add    %al,(%rax)
   0xffff82d080176b0f <+1432>:  add    %al,(%rax)
   0xffff82d080176b11 <+1434>:  add    %al,(%rax)
   0xffff82d080176b13 <+1436>:  add    %al,(%rax)
   0xffff82d080176b15 <+1438>:  add    %al,(%rax)
   0xffff82d080176b17 <+1440>:  add    %al,(%rax)
   0xffff82d080176b19 <+1442>:  add    %al,(%rax)
   0xffff82d080176b1b <+1444>:  add    %al,(%rax)
   0xffff82d080176b1d <+1446>:  add    %al,(%rax)
   0xffff82d080176b1f <+1448>:  add    %al,(%rax)
   0xffff82d080176b21 <+1450>:  add    %al,(%rax)
   0xffff82d080176b23 <+1452>:  add    %al,(%rax)
   0xffff82d080176b25 <+1454>:  add    %al,(%rax)
   0xffff82d080176b27 <+1456>:  add    %al,(%rax)
   0xffff82d080176b29 <+1458>:  add    %al,(%rax)
   0xffff82d080176b2b <+1460>:  add    %al,(%rax)
   0xffff82d080176b2d <+1462>:  add    %al,(%rax)
   0xffff82d080176b2f <+1464>:  add    %al,(%rax)
   0xffff82d080176b31 <+1466>:  add    %al,(%rax)
   0xffff82d080176b33 <+1468>:  add    %al,(%rax)
   0xffff82d080176b35 <+1470>:  add    %al,(%rax)
   0xffff82d080176b37 <+1472>:  add    %al,(%rax)
   0xffff82d080176b39 <+1474>:  add    %al,(%rax)
   0xffff82d080176b3b <+1476>:  add    %al,(%rax)
   0xffff82d080176b3d <+1478>:  add    %al,(%rax)
   0xffff82d080176b3f <+1480>:  add    %al,(%rax)
   0xffff82d080176b41 <+1482>:  add    %al,(%rax)
   0xffff82d080176b43 <+1484>:  add    %al,(%rax)
   0xffff82d080176b45 <+1486>:  add    %al,(%rax)
   0xffff82d080176b47 <+1488>:  add    %al,(%rax)
   0xffff82d080176b49 <+1490>:  add    %al,(%rax)
   0xffff82d080176b4b <+1492>:  add    %al,(%rax)
   0xffff82d080176b4d <+1494>:  add    %al,(%rax)
   0xffff82d080176b4f <+1496>:  add    %al,(%rax)
   0xffff82d080176b51 <+1498>:  add    %al,(%rax)
   0xffff82d080176b53 <+1500>:  add    %al,(%rax)
   0xffff82d080176b55 <+1502>:  add    %al,(%rax)
   0xffff82d080176b57 <+1504>:  add    %al,(%rax)
   0xffff82d080176b59 <+1506>:  add    %al,(%rax)
   0xffff82d080176b5b <+1508>:  add    %al,(%rax)
   0xffff82d080176b5d <+1510>:  add    %al,(%rax)
   0xffff82d080176b5f <+1512>:  add    %al,(%rax)
   0xffff82d080176b61 <+1514>:  add    %al,(%rax)
   0xffff82d080176b63 <+1516>:  add    %al,(%rax)
   0xffff82d080176b65 <+1518>:  add    %al,(%rax)
   0xffff82d080176b67 <+1520>:  add    %al,(%rax)
   0xffff82d080176b69 <+1522>:  add    %al,(%rax)
   0xffff82d080176b6b <+1524>:  add    %al,(%rax)
   0xffff82d080176b6d <+1526>:  add    %al,(%rax)
   0xffff82d080176b6f <+1528>:  add    %al,(%rax)
   0xffff82d080176b71 <+1530>:  add    %al,(%rax)
   0xffff82d080176b73 <+1532>:  add    %al,(%rax)
   0xffff82d080176b75 <+1534>:  add    %al,(%rax)
   0xffff82d080176b77 <+1536>:  add    %al,(%rax)
   0xffff82d080176b79 <+1538>:  add    %al,(%rax)
   0xffff82d080176b7b <+1540>:  add    %al,(%rax)
   0xffff82d080176b7d <+1542>:  add    %al,(%rax)
   0xffff82d080176b7f <+1544>:  add    %al,(%rax)
   0xffff82d080176b81 <+1546>:  add    %al,(%rax)
   0xffff82d080176b83 <+1548>:  add    %al,(%rax)
   0xffff82d080176b85 <+1550>:  add    %al,(%rax)
   0xffff82d080176b87 <+1552>:  add    %al,(%rax)
   0xffff82d080176b89 <+1554>:  add    %al,(%rax)
   0xffff82d080176b8b <+1556>:  add    %al,(%rax)
   0xffff82d080176b8d <+1558>:  add    %al,(%rax)
   0xffff82d080176b8f <+1560>:  add    %al,(%rax)
   0xffff82d080176b91 <+1562>:  add    %al,(%rax)
   0xffff82d080176b93 <+1564>:  add    %al,(%rax)
   0xffff82d080176b95 <+1566>:  add    %al,(%rax)
   0xffff82d080176b97 <+1568>:  add    %al,(%rax)
   0xffff82d080176b99 <+1570>:  add    %al,(%rax)
   0xffff82d080176b9b <+1572>:  add    %al,(%rax)
   0xffff82d080176b9d <+1574>:  add    %al,(%rax)
   0xffff82d080176b9f <+1576>:  add    %al,(%rax)
   0xffff82d080176ba1 <+1578>:  add    %al,(%rax)
   0xffff82d080176ba3 <+1580>:  add    %al,(%rax)
   0xffff82d080176ba5 <+1582>:  add    %al,(%rax)
   0xffff82d080176ba7 <+1584>:  add    %al,(%rax)
   0xffff82d080176ba9 <+1586>:  add    %al,(%rax)
   0xffff82d080176bab <+1588>:  add    %al,(%rax)
   0xffff82d080176bad <+1590>:  add    %al,(%rax)
   0xffff82d080176baf <+1592>:  add    %al,(%rax)
   0xffff82d080176bb1 <+1594>:  add    %al,(%rax)
   0xffff82d080176bb3 <+1596>:  add    %al,(%rax)
   0xffff82d080176bb5 <+1598>:  add    %al,(%rax)
   0xffff82d080176bb7 <+1600>:  add    %al,(%rax)
   0xffff82d080176bb9 <+1602>:  add    %al,(%rax)
   0xffff82d080176bbb <+1604>:  add    %al,(%rax)
   0xffff82d080176bbd <+1606>:  add    %al,(%rax)
   0xffff82d080176bbf <+1608>:  add    %al,(%rax)
   0xffff82d080176bc1 <+1610>:  add    %al,(%rax)
End of assembler dump.

[-- Attachment #5: dmesg --]
[-- Type: text/plain, Size: 28032 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Sat Nov 14 00:50:05 CET 2015
(XEN) [    0.000000] Latest ChangeSet:
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.435787] Detected 2394.592 MHz processor.
(XEN) [    1.443167] Initing memory sharing.
(XEN) [    1.447492] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.454435] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.464415] Intel machine check reporting enabled
(XEN) [    1.470025] alt table ffff82d0802cfa90 -> ffff82d0802d0bd0
(XEN) [    1.476364] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.484920] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.490988] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.497615] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.503856] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.510090] Intel VT-d Snoop Control not enabled.
(XEN) [    1.515754] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.521907] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.527629] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.533435] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.547704] I/O virtualisation enabled
(XEN) [    1.552284]  - Dom0 mode: Relaxed
(XEN) [    1.556440] Interrupt remapping enabled
(XEN) [    1.561331] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.567697] ENABLING IO-APIC IRQs
(XEN) [    1.571844]  -> Using old ACK method
(XEN) [    1.576568] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.785867] TSC deadline timer enabled
(XEN) [    2.016050] Platform timer is 14.318MHz HPET
(XEN) [    2.020833] Allocated console ring of 64 KiB.
(XEN) [    2.021967] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12
(XEN) [    2.023016] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.023529] mwait-idle: v0.4 model 0x2a
(XEN) [    2.024037] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.024598] VMX: Supported advanced features:
(XEN) [    2.025109]  - APIC MMIO access virtualisation
(XEN) [    2.025620]  - APIC TPR shadow
(XEN) [    2.026373]  - Extended Page Tables (EPT)
(XEN) [    2.026882]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.027396]  - Virtual NMI
(XEN) [    2.027931]  - MSR direct-access bitmap
(XEN) [    2.028440]  - Unrestricted Guest
(XEN) [    2.028945] HVM: ASIDs enabled.
(XEN) [    2.029486] HVM: VMX enabled
(XEN) [    2.029987] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.030504] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.031658] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12
(XEN) [    2.032693] Brought up 8 CPUs
(XEN) [    2.033214] microcode: CPU7 updated from revision 0x28 to 0x29, date = 2013-06-12
(XEN) [    2.039867] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.040383] microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12
(XEN) [    2.061561] ACPI sleep modes: S3
(XEN) [    2.062066] mcheck_poll: Machine check polling timer started.
(XEN) [    2.062597] Dom0 has maximum 792 PIRQs
(XEN) [    2.063184] *** LOADING DOMAIN 0 ***
(XEN) [    2.244646]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    2.245196]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    2.246430] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    2.246938]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    2.247995]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    2.248520] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    2.249028]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    2.249549]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    2.250068]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    2.250809]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    2.251328]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    2.251847]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    2.252366]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    2.252920]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    2.254260] Dom0 has maximum 8 VCPUs
(XEN) [    5.419481] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    5.427778] Std. Loglevel: All
(XEN) [    5.428320] Guest Loglevel: All
(XEN) [    5.428824] Xen is relinquishing VGA console.
(XEN) [    5.429901] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) [    5.429939] Freed 316kB init memory.
(XEN) [    5.686219] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.686224] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) [    5.686227] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) [    5.686231] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.686234] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) [    5.686238] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.686241] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.777389] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.777396] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080cab3000 to 0xffffffff81646d10.
(XEN) [    5.777402] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080cab3080 to 0xffffffff81648e70.
(XEN) [    5.777407] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.777412] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83080cab7fc0 to 0x0000000000000000.
(XEN) [    5.777417] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.777421] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.777710] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.777715] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff83080caa3000 to 0xffffffff81646d10.
(XEN) [    5.777719] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff83080caa3080 to 0xffffffff81648e70.
(XEN) [    5.777723] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.777726] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff83080caa7fc0 to 0x0000000000000000.
(XEN) [    5.777729] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.777733] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.777959] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.777963] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca93000 to 0xffffffff81646d10.
(XEN) [    5.777966] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca93080 to 0xffffffff81648e70.
(XEN) [    5.777969] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.777973] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca97fc0 to 0x0000000000000000.
(XEN) [    5.777976] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.777979] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.778177] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.778182] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca83000 to 0xffffffff81646d10.
(XEN) [    5.778186] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca83080 to 0xffffffff81648e70.
(XEN) [    5.778189] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.778193] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca87fc0 to 0x0000000000000000.
(XEN) [    5.778196] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.778199] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.778401] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.778405] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063f3000 to 0xffffffff81646d10.
(XEN) [    5.778409] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063f3080 to 0xffffffff81648e70.
(XEN) [    5.778412] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.778415] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308063f7fc0 to 0x0000000000000000.
(XEN) [    5.778418] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.778421] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.778618] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.778623] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063eb000 to 0xffffffff81646d10.
(XEN) [    5.778626] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063eb080 to 0xffffffff81648e70.
(XEN) [    5.778630] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.778634] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff8308063effc0 to 0x0000000000000000.
(XEN) [    5.778637] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.778640] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.778873] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.778877] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063db000 to 0xffffffff81646d10.
(XEN) [    5.778881] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063db080 to 0xffffffff81648e70.
(XEN) [    5.778884] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.778887] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000175 from 0xffff8308063dffc0 to 0x0000000000000000.
(XEN) [    5.778890] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.778894] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.926550] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    6.926553] PCI add device 0000:00:00.0
(XEN) [    6.926749] PCI add device 0000:00:01.0
(XEN) [    6.926931] PCI add device 0000:00:01.1
(XEN) [    6.927169] PCI add device 0000:00:02.0
(XEN) [    6.927357] PCI add device 0000:00:06.0
(XEN) [    6.927641] PCI add device 0000:00:16.0
(XEN) [    6.927937] PCI add device 0000:00:19.0
(XEN) [    6.928233] PCI add device 0000:00:1a.0
(XEN) [    6.928517] PCI add device 0000:00:1b.0
(XEN) [    6.928775] PCI add device 0000:00:1c.0
(XEN) [    6.929035] PCI add device 0000:00:1c.4
(XEN) [    6.929290] PCI add device 0000:00:1c.6
(XEN) [    6.929586] PCI add device 0000:00:1d.0
(XEN) [    6.929780] PCI add device 0000:00:1e.0
(XEN) [    6.930034] PCI add device 0000:00:1f.0
(XEN) [    6.930310] PCI add device 0000:00:1f.2
(XEN) [    6.930492] PCI add device 0000:00:1f.3
(XEN) [    6.930763] PCI add device 0000:01:00.0
(XEN) [    6.948211] PCI add device 0000:02:00.0
(XEN) [    6.968249] PCI add device 0000:03:00.0
(XEN) [    6.968685] PCI add device 0000:04:00.0
(XEN) [    6.988444] PCI add device 0000:05:00.0
(XEN) [    6.988869] PCI add device 0000:06:00.0
(XEN) [    6.989589] PCI add device 0000:07:00.0
(XEN) [    7.008479] PCI add device 0000:08:00.0
(XEN) [    7.008775] PCI add device 0000:09:00.0
(XEN) [    7.009266] PCI add device 0000:0a:08.0
(XEN) [    7.009552] PCI add device 0000:0a:09.0
(XEN) [    7.009835] PCI add device 0000:0a:0a.0
(XEN) [    7.010115] PCI add device 0000:0a:0b.0
(XEN) [    7.071573] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071594] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071639] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071658] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071675] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071684] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071729] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.071748] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(d1) [   34.982476] HVM Loader
(d1) [   34.982818] Detected Xen v4.5.2
(d1) [   34.983206] Xenbus rings @0xfeffc000, event channel 1
(d1) [   34.983622] System requested SeaBIOS
(d1) [   34.983826] CPU speed is 2395 MHz
(d1) [   34.984489] Relocating guest memory for lowmem MMIO space disabled
(XEN) [   34.985061] irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(d1) [   34.985401] PCI-ISA link 0 routed to IRQ5
(XEN) [   34.985649] irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(d1) [   34.985993] PCI-ISA link 1 routed to IRQ10
(XEN) [   34.986214] irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(d1) [   34.986566] PCI-ISA link 2 routed to IRQ11
(XEN) [   34.986766] irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(d1) [   34.987136] PCI-ISA link 3 routed to IRQ5
(d1) [   35.004032] pci dev 01:3 INTA->IRQ10
(d1) [   35.007014] pci dev 02:0 INTA->IRQ11
(d1) [   35.014288] pci dev 04:0 INTA->IRQ5
(d1) [   35.018608] pci dev 05:0 INTA->IRQ10
(d1) [   35.022666] pci dev 06:0 INTA->IRQ11
(d1) [   35.047933] No RAM in high memory; setting high_mem resource base to 100000000
(d1) [   35.048483] pci dev 03:0 bar 10 size 002000000: 0f0000008
(d1) [   35.050606] pci dev 02:0 bar 14 size 001000000: 0f2000008
(d1) [   35.052697] pci dev 04:0 bar 10 size 000020000: 0f3000004
(XEN) [   35.052893] memory_map:add: dom1 gfn=f3000 mfn=f7c00 nr=20
(d1) [   35.056201] pci dev 03:0 bar 30 size 000010000: 0f3020000
(d1) [   35.056814] pci dev 04:0 bar 30 size 000010000: 0f3030000
(d1) [   35.057420] pci dev 05:0 bar 30 size 000010000: 0f3040000
(d1) [   35.058021] pci dev 06:0 bar 10 size 000010000: 0f3050000
(XEN) [   35.058203] memory_map:add: dom1 gfn=f3050 mfn=f7800 nr=10
(d1) [   35.062960] pci dev 03:0 bar 14 size 000001000: 0f3060000
(XEN) [   35.063253] memory_map:add: dom1 gfn=f3061 mfn=f7842 nr=1
(d1) [   35.067114] pci dev 05:0 bar 14 size 000001000: 0f3061000
(d1) [   35.067645] pci dev 02:0 bar 10 size 000000100: 00000c001
(d1) [   35.069953] pci dev 05:0 bar 10 size 000000100: 00000c101
(XEN) [   35.070376] ioport_map:add: dom1 gport=c100 mport=c200 nr=100
(d1) [   35.073641] pci dev 01:1 bar 20 size 000000010: 00000c201
(d1) [   35.075752] Multiprocessor initialisation:
(d1) [   35.100556]  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   35.115647]  - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   35.115712] Testing HVM environment:
(d1) [   35.119040]  - REP INSB across page boundaries ... passed
(d1) [   35.120996]  - GS base MSRs and SWAPGS ... passed
(d1) [   35.121017] Passed 2 of 2 tests
(d1) [   35.121042] Writing SMBIOS tables ...
(d1) [   35.121771] Loading SeaBIOS ...
(d1) [   35.121810] Creating MP tables ...
(d1) [   35.121899] Loading ACPI ...
(d1) [   35.123225] vm86 TSS at fc00a380
(d1) [   35.123481] BIOS map:
(d1) [   35.123509]  10000-100d3: Scratch space
(d1) [   35.123532]  e0000-fffff: Main BIOS
(d1) [   35.123545] E820 table:
(d1) [   35.123592]  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d1) [   35.123635]  HOLE: 00000000:000a0000 - 00000000:000e0000
(d1) [   35.123687]  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d1) [   35.123735]  [02]: 00000000:00100000 - 00000000:27800000: RAM
(d1) [   35.123777]  HOLE: 00000000:27800000 - 00000000:fc000000
(d1) [   35.123830]  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d1) [   35.124046] Invoking SeaBIOS ...
(XEN) [   36.000827] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) [   36.000830] ************* VMCS Area **************
(XEN) [   36.000832] *** Guest State ***
(XEN) [   36.000833] CR0: actual=0x0000000000000039, shadow=0x0000000000000011, gh_mask=ffffffffffffffff
(XEN) [   36.000836] CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff
(XEN) [   36.000838] CR3: actual=0x0000000000800000, target_count=0
(XEN) [   36.000839]      target0=0000000000000000, target1=0000000000000000
(XEN) [   36.000840]      target2=0000000000000000, target3=0000000000000000
(XEN) [   36.000842] RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 0x0000000100000000 (0x0000000100000000)
(XEN) [   36.000844] RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 0x0000000000000400
(XEN) [   36.000846] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) [   36.000848] CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000850] DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000852] SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000854] ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000856] FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000858] GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.000860] GDTR:                           limit=0x00000037, base=0x00000000000f6d80
(XEN) [   36.000861] LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, base=0x0000000000000000
(XEN) [   36.000863] IDTR:                           limit=0x00000000, base=0x00000000000f6dbe
(XEN) [   36.000865] TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, base=0x0000000000000000
(XEN) [   36.000866] Guest PAT = 0x0007040600070406
(XEN) [   36.000868] TSC Offset = ffffffe7284b90c0
(XEN) [   36.000869] DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) [   36.000871] Interruptibility=0000 ActivityState=0000
(XEN) [   36.000872] *** Host State ***
(XEN) [   36.000874] RSP = 0xffff8308063eff90  RIP = 0xffff82d0801f4820
(XEN) [   36.000875] CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) [   36.000877] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff8308063f8c80
(XEN) [   36.000879] GDTBase=ffff8308063ec000 IDTBase=ffff8308063e6000
(XEN) [   36.000881] CR0=000000008005003b CR3=000000070bf71000 CR4=00000000000426f0
(XEN) [   36.000883] Sysenter RSP=ffff8308063effc0 CS:RIP=e008:ffff82d080235550
(XEN) [   36.000885] Host PAT = 0x0000050100070406
(XEN) [   36.000886] *** Control State ***
(XEN) [   36.000887] PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) [   36.000889] EntryControls=000051ff ExitControls=000fefff
(XEN) [   36.000890] ExceptionBitmap=000600c2
(XEN) [   36.000891] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.000893] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.000894]         reason=80000021 qualification=00000000
(XEN) [   36.000896] IDTVectoring: info=00000000 errcode=00000000
(XEN) [   36.000897] TPR Threshold = 0x00
(XEN) [   36.000898] EPT pointer = 0x000000070bf3f01e
(XEN) [   36.000899] Virtual processor ID = 0x000a
(XEN) [   36.000901] **************************************
(XEN) [   36.000902] domain_crash called from vmx.c:2505
(XEN) [   36.000904] Domain 1 (vcpu#0) crashed on cpu#6:
(XEN) [   36.000906] ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) [   36.000908] CPU:    6
(XEN) [   36.000909] RIP:    0008:[<0000000100000000>]
(XEN) [   36.000910] RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) [   36.000913] rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) [   36.000915] rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) [   36.000917] rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) [   36.000918] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [   36.000920] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [   36.000921] r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) [   36.000923] cr3: 0000000000800000   cr2: 0000000000000000
(XEN) [   36.000925] ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008


[-- Attachment #6: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 10075 bytes --]

On 14/11/2015 00:16, Atom2 wrote:
> Am 13.11.15 um 11:09 schrieb Andrew Cooper:
>> On 13/11/15 07:25, Jan Beulich wrote:
>>>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>>> Hi Andrew,
>>>>>> thanks for your reply. Answers are inline further down.
>>>>>>
>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>>>> of the
>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>>
>>>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>>> [...]
>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>>>> state (0).
>>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>>> (XEN) *** Guest State ***
>>>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>>> odd? That will be the source of the failure.
>>>>>>>
>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>>>> should not be possible to set any of them.
>>>>>>>
>>>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>>>> get it self into this condition.
>>>>>>>
>>>>>>> Are you able to rerun with a debug build of the hypervisor?
>>>>>> [snip]
>>>>>> Another question is whether prior to enabling the debug USE flag it
>>>>>> might make sense to re-compile with gcc-4.8.5 (please see my previous
>>>>>> list reply) to rule out any compiler related issues. Jan, Andrew -
>>>>>> what are your thoughts?
>>>>> First of all, check whether the compiler makes a difference on 4.5.2
>>>> Hi Andrew,
>>>> I changed the compiler and there was no change to the better: 
>>>> Unfortunately the HVM domU is still crashing with a similar error 
>>>> message as soon as it is being started.
>>>>> If both compiles result in a guest crashing in that manner, test a debug
>>>>> Xen to see if any assertions/errors are encountered just before the
>>>>> guest crashes.
>>>>>
>>>> As the compiler did not make any difference, I enabled the debug USE 
>>>> flag, re-compiled (using gcc-4.9.3), and rebooted using a serial console 
>>>> to capture output. Unfortunately I did not get very far and things 
>>>> become even stranger: This time the system did not even finnish the boot 
>>>> process, but rather hard-stopped pretty early with a message reading 
>>>> "Panic on CPU 3: DOUBLE FAULT -- system shutdown". The captured logfile 
>>>> is attached as "serial log.txt".
>>>>
>>>> As this happened immediately after the CPU microcode update, I thought 
>>>> there might be a connection and disabled the microcode update. After the 
>>>> next reboot it seemed as if the boot process got a bit further as 
>>>> evidenced by a few more lines in the log file (those between lines 136 
>>>> and 197 in the second log file named "serial log no ucode.txt"), but in 
>>>> the end it finnished off with an identical error message (only the CPU # 
>>>> was different this time, but that number seems to change between boots 
>>>> anyways).
>>>>
>>>> I hope that makes some sense to you.
>>> Not really, other than now even more suspecting bad hardware or
>>> something fundamentally wrong with your build. Did you retry with
>>> a freshly built 4.5.1? Could you alternatively try with a known good
>>> build of 4.5.2 (e.g. from osstest)?
> Andrew,
> many thanks again for your help.
>> Agreed.  Double faults indicate that the exception handing entry points
>> are not set up in an appropriate state.  Something is definitely wrong
>> with either the compiled binary or the hardware.
> The hardware (it's a SandyBridge XEON processor with ECC RAM and
> Enterprise SATA disks) has worked for almost two years together with
> XEN and other than this issue there's also currently nothing strange
> (i.e. if I boot with a standard linux kernel, the system boots and
> works without any issues and is very stable and there are also no
> strange messages in /var/log/messages).
>> Several questions and lines of investigation:
>>
>> Is this straight Xen 4.5.1 and 2, or do Gentoo have their own patches on
>> top?
> By the looks of it there are only security patches, but no gentoo
> specific patches.
>> On repeated attempts, are the details of the double fault identical
>> (other than the cpu), or does it move around (i.e. always do_IRQ+0x15)
> It always seems to be do_IRQ+0x15 (I have made a number of boots), and
> more specifically it was always do_IRQ+0x15/0x64c. Timings varied, the
> CPU differed between boots and also the rax, rbx, rcx, rdx, rsi, rdi,
> rbp, rsp, and r8 values were diffent as were those for r15 and cr2 and
> the values next to valid stack range. I don't know whether that's of
> relevance, but I thought I'd mention it after a quick analysis of two
> serial logs.
>> Can you boot with console_timestamps=boot on the command line in the
>> future.  This will put Linux-sytle timestamps on log messages.
> Done - see two the attached serial log files mentioned above.
>> Can you also compile in the attached patch? I haven't quite got it
>> suitable for inclusion upstream yet, but it will also dump the
>> instruction stream under the fault.
> I have added the patch, but it seems not to trigger - at least the
> text that is in the patch does not show in the serial console output;
> it is however in the (uncompressed) xen.gz file as a quick grep for
> texts "Xen code around " and " [fault on access]" confirmed (grep
> said: binary file matches).
>> Finally, can you disassemble the xen-syms which results from the debug
>> build and paste the start of do_IRQ.  (i.e. `gdb xen-syms` and "disass
>> do_IRQ")
> Please see the third file named "do_IRQ".
>
> Furthermore I have managed to get the system to again boot up (a HVM
> domU, however, still crashes): If I turn off the debug USE flag and
> just add symbols to the XEN binary. It seems that the debug USE flag
> is probably not what I was expecting. On
> https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces it
> describes the purpose of the debug USE flag as follows (emphasis by me):
>
> ==== start excerpt from web page ====
> Some ebuilds provide a *debug* USE flag. While some mistakenly use it
> to provide debug information and play with compiler flags when it is
> enabled, that is not its purpose.
>
> _If you're trying to debug a reproduceable crash, you want to leave
> this USE flag alone, as it'll be building a different source than what
> you had before._ It is more efficient to get first a backtrace without
> changing the code, by simply emitting symbol information, and just
> afterward enable debug features to track the issue further down.
>
> Debug features that are enabled by the USE flag include assertions,
> debug logs on screen, debug files, leak detection and extra-safe
> operations (such as scrubbing memory before use). Some of them might
> be taxing, especially for complex software or software where
> performance is an important issue.
>
> For these reasons, please exercise caution when enabling the *debug*
> USE flag, and only consider it a last-chance card.
> ==== end excerpt from web page ====
>
> Now _without_ the debug USE flag, but with debug information in the
> binary (I used splitdebug), all is back to where the problem started
> off (i.e. the system boots without issues until such time it starts a
> HVM domU which then crashes; PV domUs are working). I have attached
> the latest "xl dmesg" output with the timing information included.
>
> I hope any of this makes sense to you.
>
> Again many thanks and best regards
>

Right - it would appear that the USE flag is definitely not what you
wanted, and causes bad compilation for Xen.  The do_IRQ disassembly you
sent is a the result of disassembling a whole block of zeroes.  Sorry
for leading you on a goose chase - the double faults will be the product
of bad compilation, rather than anything to do with your specific problem.

However, the final log you sent (dmesg) is using a debug Xen, which is
what I was attempting to get you to do originally.

We still observe that the VM ends up in 32bit non-paged mode but with an
RIP with bit 32 set, which is an invalid state to be in.  However, there
was nothing particularly interesting in the extra log information.

Please can you rerun with "hvm_debug=0xc3f", which will cause far more
logging to occur to the console while the HVM guest is running.  That
might show some hints.

Also, the fact that this occurs just after starting SeaBIOS is
interesting.  As you have switched versions of Xen, you have also
switched hvmloader, which contains the SeaBIOS binary embedded in it. 
Would you be able to compile both 4.5.1 and 4.5.2 and switch the
hvmloader binaries in use.  It would be very interesting to see whether
the failure is caused by the hvmloader binary or the hypervisor.  (With
`xl`, you can use firmware_override="/full/path/to/firmware" to override
the default hvmloader).

Thanks,

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 13334 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1.1: Type: text/plain, Size: 6753 bytes --]

Am 14.11.15 um 21:32 schrieb Andrew Cooper:
> On 14/11/2015 00:16, Atom2 wrote:
>> Am 13.11.15 um 11:09 schrieb Andrew Cooper:
>>> On 13/11/15 07:25, Jan Beulich wrote:
>>>>>>> On 13.11.15 at 00:00,<ariel.atom2@web2web.at>  wrote:
>>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>>>> Hi Andrew,
>>>>>>> thanks for your reply. Answers are inline further down.
>>>>>>>
>>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>>>> On 12.11.15 at 02:08,<ariel.atom2@web2web.at>  wrote:
>>>>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>>>>> of the
>>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>>>
>>>>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>>>> [...]
>>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>>>>> state (0).
>>>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>>>> (XEN) *** Guest State ***
>>>>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>>>> odd? That will be the source of the failure.
>>>>>>>>
>>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>>>>> should not be possible to set any of them.
>>>>>>>>
>>>>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>>>>> get it self into this condition.
>>>>>>>>
>>>>>>>> Are you able to rerun with a debug build of the hypervisor?
[big snip]
>>>>>>> Now _without_ the debug USE flag, but with debug information in
>>>>>>>          the binary (I used splitdebug), all is back to where the problem
>>>>>>>          started off (i.e. the system boots without issues until such
>>>>>>>          time it starts a HVM domU which then crashes; PV domUs are
>>>>>>>          working). I have attached the latest "xl dmesg" output with the
>>>>>>>          timing information included.
>>>>>>>   
>>
>> I hope any of this makes sense to you.
>>
>> Again many thanks and best regards
>>
>
> Right - it would appear that the USE flag is definitely not what you 
> wanted, and causes bad compilation for Xen.  The do_IRQ disassembly 
> you sent is a the result of disassembling a whole block of zeroes.  
> Sorry for leading you on a goose chase - the double faults will be the 
> product of bad compilation, rather than anything to do with your 
> specific problem.
Hi Andrew,
there's absolutely no need to appologize as it is me who asked for help 
and you who generously stepped in and provided it. I really do 
appreciate your help and it is for me, as the one seeking help, to 
provide all the information you deem necessary and you ask for.
> However, the final log you sent (dmesg) is using a debug Xen, which is 
> what I was attempting to get you to do originally.
Next time I know better how to arrive at a debug XEN. It's all about 
learning.
> We still observe that the VM ends up in 32bit non-paged mode but with 
> an RIP with bit 32 set, which is an invalid state to be in. However, 
> there was nothing particularly interesting in the extra log information.
>
> Please can you rerun with "hvm_debug=0xc3f", which will cause far more 
> logging to occur to the console while the HVM guest is running.  That 
> might show some hints.
I haven't done that yet - but please see my next paragraph. If you are 
still interested in this, for whatever reason, I am clearly more than 
happy to rerun with your suggested option and provide that information 
as well.
> Also, the fact that this occurs just after starting SeaBIOS is 
> interesting.  As you have switched versions of Xen, you have also 
> switched hvmloader, which contains the SeaBIOS binary embedded in it.  
> Would you be able to compile both 4.5.1 and 4.5.2 and switch the 
> hvmloader binaries in use.  It would be very interesting to see 
> whether the failure is caused by the hvmloader binary or the 
> hypervisor.  (With `xl`, you can use 
> firmware_override="/full/path/to/firmware" to override the default 
> hvmloader).
Your analysis was absolutely spot on. After re-thinking this for a 
moment, I thought going down that route first would make a lot of sense 
as PV guests still do work and one of the differences to HVM domUs is 
that the former do _not_ require SeaBIOS. Looking at my log files of 
installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in 
the relevant timeframe which obviously had not made it to the hvmloader 
of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.

So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS 
1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed 
to strongly indicate that there indeed might be an issue with SeaBIOS as 
this probably was the only variable that had changed from the original 
install of xen-4.5.1.

My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile 
xen-4.5.1. Voila, the system was again up and running. While still 
having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you 
probably guessed it ... the problem was gone: The system boots up with 
no issues and everything is fine again.

So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2 
preventing HVM doamins from successfully starting up. I don't know what 
this is triggered from, if this is specific to my hardware or whether 
something else in my environment is to blame.

In any case, I am again more than happy to provide data / run a few 
tests should you wish to get to the grounds of this.

I do owe you a beer (or any other drink) should you ever be at my 
location (i.e. Vienna, Austria).

Many thanks again for your analysis and your first class support. Xen 
and their people absolutely rock!

Atom2

[-- Attachment #1.2: Type: text/html, Size: 9770 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 4507 bytes --]

On 15/11/15 00:14, Atom2 wrote:
>
>> Right - it would appear that the USE flag is definitely not what you
>> wanted, and causes bad compilation for Xen.  The do_IRQ disassembly
>> you sent is a the result of disassembling a whole block of zeroes. 
>> Sorry for leading you on a goose chase - the double faults will be
>> the product of bad compilation, rather than anything to do with your
>> specific problem.
> Hi Andrew,
> there's absolutely no need to appologize as it is me who asked for
> help and you who generously stepped in and provided it. I really do
> appreciate your help and it is for me, as the one seeking help, to
> provide all the information you deem necessary and you ask for.
>> However, the final log you sent (dmesg) is using a debug Xen, which
>> is what I was attempting to get you to do originally.
> Next time I know better how to arrive at a debug XEN. It's all about
> learning.
>> We still observe that the VM ends up in 32bit non-paged mode but with
>> an RIP with bit 32 set, which is an invalid state to be in.  However,
>> there was nothing particularly interesting in the extra log information.
>>
>> Please can you rerun with "hvm_debug=0xc3f", which will cause far
>> more logging to occur to the console while the HVM guest is running. 
>> That might show some hints.
> I haven't done that yet - but please see my next paragraph. If you are
> still interested in this, for whatever reason, I am clearly more than
> happy to rerun with your suggested option and provide that information
> as well.
>> Also, the fact that this occurs just after starting SeaBIOS is
>> interesting.  As you have switched versions of Xen, you have also
>> switched hvmloader, which contains the SeaBIOS binary embedded in
>> it.  Would you be able to compile both 4.5.1 and 4.5.2 and switch the
>> hvmloader binaries in use.  It would be very interesting to see
>> whether the failure is caused by the hvmloader binary or the
>> hypervisor.  (With `xl`, you can use
>> firmware_override="/full/path/to/firmware" to override the default
>> hvmloader).
> Your analysis was absolutely spot on. After re-thinking this for a
> moment, I thought going down that route first would make a lot of
> sense as PV guests still do work and one of the differences to HVM
> domUs is that the former do _not_ require SeaBIOS. Looking at my log
> files of installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
> 1.8.2 in the relevant timeframe which obviously had not made it to the
> hvmloader of xen-4.5.1 as I did not re-compile xen after the upgrade
> of SeaBIOS.
>
> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
> 1.8.2) and the same error as with xen-4.5.2 popped up - and that
> seemed to strongly indicate that there indeed might be an issue with
> SeaBIOS as this probably was the only variable that had changed from
> the original install of xen-4.5.1.
>
> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
> xen-4.5.1. Voila, the system was again up and running. While still
> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ...
> you probably guessed it ... the problem was gone: The system boots up
> with no issues and everything is fine again.
>
> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
> preventing HVM doamins from successfully starting up. I don't know
> what this is triggered from, if this is specific to my hardware or
> whether something else in my environment is to blame.
>
> In any case, I am again more than happy to provide data / run a few
> tests should you wish to get to the grounds of this.
>
> I do owe you a beer (or any other drink) should you ever be at my
> location (i.e. Vienna, Austria).
>
> Many thanks again for your analysis and your first class support. Xen
> and their people absolutely rock!

Great - so confirms the issue as a SeaBIOS interaction issue, rather
than a hypervisor regression.

As I said before, I am still certain that a guest should not be able to
get itself into the crashing state (short of a hardware errata), so I
still suspect that there is a latent hypervisor emulation bug which has
been tickled by the SeaBIOS update.

Would you please mind running the bad HVMLoader on Xen 4.5.2 with
hvm_debug=0xc3f ? I am still hoping that that will shed some light on
SeaBIOS actions just leading up to the crash.

Are you able to experiment with newer versions of Xen?  It would be
interesting to see whether the issue is still present in Xen 4.6

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 5893 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1.1: Type: text/plain, Size: 1671 bytes --]

Am 15.11.15 um 16:12 schrieb Andrew Cooper:
[big snip]
> Great - so confirms the issue as a SeaBIOS interaction issue, rather 
> than a hypervisor regression.
>
> As I said before, I am still certain that a guest should not be able 
> to get itself into the crashing state (short of a hardware errata), so 
> I still suspect that there is a latent hypervisor emulation bug which 
> has been tickled by the SeaBIOS update.
>
> Would you please mind running the bad HVMLoader on Xen 4.5.2 with 
> hvm_debug=0xc3f ? I am still hoping that that will shed some light on 
> SeaBIOS actions just leading up to the crash.
Hi Andrew,
Please see the attached two files. One is the dmesg from booting the 
system. This looks pretty normal in my view. The other is the output of 
"xl dmesg" which is most likely what you were after. It's probably worth 
noting that the "traps.c" output between lines 259 and 314 and again 
between lines 346 and 353 seem to be xen-4.5.2 specific and don't show 
up under xen-4.5.1, but that may not be of any relevance for the SeaBIOS 
issue we are experiencing. Though I'd still be interested to know 
whether that's anything for me to worry about ...
>
> Are you able to experiment with newer versions of Xen?  It would be 
> interesting to see whether the issue is still present in Xen 4.6
Currently xen-4.6 is not stable in gentoo and I try to stick to stable 
packages as much as possible. But in case the above does not help you 
any further, I am happy to give this a try as well. Would this just be a 
straightforward test to see whether it works at all or would you require 
debug symbols as well?

Many thanks again and best regards Atom2

[-- Attachment #1.2: Type: text/html, Size: 2323 bytes --]

[-- Attachment #2: dmesg --]
[-- Type: text/plain, Size: 61348 bytes --]

[    0.000000] PAT configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC  
[    0.000000] Linux version 4.1.7-hardened-r1 (root@vm-host) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) ) #1 SMP Wed Nov 11 11:47:43 CET 2015
[    0.000000] Command line: placeholder root=UUID=aeaa7943-af67-42eb-b6f9-e73ebf7296de ro rootflags=subvol=rootfs console=hvc0 earlyprintk=xen i915.modeset=1 xen-pciback.hide=(00:1b.0)(00:1d.0)(01:00.0)(02:00.0)(03:00.0)(04:00.0)(05:00.0)(06:00.0)(08:00.0)(0a:08.0)(0a:09.0)(0a:0a.0)(0a:0b.0) softlevel=xen.d
[    0.000000] Released 0 page(s)
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
[    0.000000] Xen: [mem 0x000000000009d800-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] Xen: [mem 0x0000000020000000-0x00000000201fffff] reserved
[    0.000000] Xen: [mem 0x0000000020200000-0x000000003fffffff] usable
[    0.000000] Xen: [mem 0x0000000040000000-0x00000000401fffff] reserved
[    0.000000] Xen: [mem 0x0000000040200000-0x00000000db9effff] usable
[    0.000000] Xen: [mem 0x00000000db9f0000-0x00000000dc0d9fff] reserved
[    0.000000] Xen: [mem 0x00000000dc0da000-0x00000000dc1f8fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000dc1f9000-0x00000000dc650fff] reserved
[    0.000000] Xen: [mem 0x00000000dc651000-0x00000000dc651fff] usable
[    0.000000] Xen: [mem 0x00000000dc652000-0x00000000dc694fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000dc695000-0x00000000dcdb9fff] usable
[    0.000000] Xen: [mem 0x00000000dcdba000-0x00000000dcff1fff] reserved
[    0.000000] Xen: [mem 0x00000000dcff2000-0x00000000dcffffff] usable
[    0.000000] Xen: [mem 0x00000000dd800000-0x00000000df9fffff] reserved
[    0.000000] Xen: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
[    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] Xen: [mem 0x00000000fed90000-0x00000000fed91fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x000000012433efff] usable
[    0.000000] Xen: [mem 0x000000012433f000-0x000000081e5fffff] unusable
[    0.000000] bootconsole [xenboot0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: iEi B199/B199, BIOS V38IAR10 07/16/2014
[    0.000000] Hypervisor detected: Xen
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x12433f max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0xdd000 max_arch_pfn = 0x400000000
[    0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
[    0.000000] reserving inaccessible SNB gfx pages
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0xdca00000-0xdcbfffff]
[    0.000000]  [mem 0xdca00000-0xdcbfffff] page 4k
[    0.000000] BRK [0x01a81000, 0x01a81fff] PGTABLE
[    0.000000] BRK [0x01a82000, 0x01a82fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0xc0000000-0xdb9effff]
[    0.000000]  [mem 0xc0000000-0xdb9effff] page 4k
[    0.000000] BRK [0x01a83000, 0x01a83fff] PGTABLE
[    0.000000] BRK [0x01a84000, 0x01a84fff] PGTABLE
[    0.000000] BRK [0x01a85000, 0x01a85fff] PGTABLE
[    0.000000] BRK [0x01a86000, 0x01a86fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0xdc651000-0xdc651fff]
[    0.000000]  [mem 0xdc651000-0xdc651fff] page 4k
[    0.000000] init_memory_mapping: [mem 0xdc695000-0xdc9fffff]
[    0.000000]  [mem 0xdc695000-0xdc9fffff] page 4k
[    0.000000] init_memory_mapping: [mem 0xa0000000-0xbfffffff]
[    0.000000]  [mem 0xa0000000-0xbfffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff]
[    0.000000]  [mem 0x00100000-0x1fffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x20200000-0x3fffffff]
[    0.000000]  [mem 0x20200000-0x3fffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x40200000-0x9fffffff]
[    0.000000]  [mem 0x40200000-0x9fffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0xdcc00000-0xdcdb9fff]
[    0.000000]  [mem 0xdcc00000-0xdcdb9fff] page 4k
[    0.000000] init_memory_mapping: [mem 0xdcff2000-0xdcffffff]
[    0.000000]  [mem 0xdcff2000-0xdcffffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x100000000-0x12433efff]
[    0.000000]  [mem 0x100000000-0x12433efff] page 4k
[    0.000000] RAMDISK: [mem 0x04000000-0x048fdfff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x00000000DC1E9078 000074 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x00000000DC1F3710 0000F4 (v04 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x00000000DC1E9188 00A587 (v02 ALASKA A M I    00000001 INTL 20051117)
[    0.000000] ACPI: FACS 0x00000000DC1F7F80 000040
[    0.000000] ACPI: APIC 0x00000000DC1F3808 000092 (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x00000000DC1F38A0 000044 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x00000000DC1F38E8 00003C (v01 ALASKA A M I    01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x00000000DC1F3928 000038 (v01 ALASKA A M I    01072009 AMI. 00000005)
[    0.000000] ACPI: SSDT 0x00000000DC1F3960 00036D (v01 SataRe SataTabl 00001000 INTL 20091112)
[    0.000000] ACPI: SSDT 0x00000000DC1F3CD0 00081E (v01 PmRef  Cpu0Ist  00003000 INTL 20051117)
[    0.000000] ACPI: SSDT 0x00000000DC1F44F0 000A92 (v01 PmRef  CpuPm    00003000 INTL 20051117)
[    0.000000] ACPI: XMAR 0x00000000DC1F4F88 0000B0 (v01 INTEL  SNB      00000001 INTL 00000001)
[    0.000000] ACPI: ASF! 0x00000000DC1F5038 0000A5 (v32 INTEL   HCG     00000001 TFSM 000F4240)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] Setting APIC routing to Xen PV.
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000012433efff]
[    0.000000] NODE_DATA(0) allocated [mem 0xdcca2000-0xdcca3fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000012433efff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009cfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000001fffffff]
[    0.000000]   node   0: [mem 0x0000000020200000-0x000000003fffffff]
[    0.000000]   node   0: [mem 0x0000000040200000-0x00000000db9effff]
[    0.000000]   node   0: [mem 0x00000000dc651000-0x00000000dc651fff]
[    0.000000]   node   0: [mem 0x00000000dc695000-0x00000000dcdb9fff]
[    0.000000]   node   0: [mem 0x00000000dcff2000-0x00000000dcffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000012433efff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000012433efff]
[    0.000000] On node 0 totalpages: 1048575
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 156 pages reserved
[    0.000000]   DMA zone: 3996 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14005 pages used for memmap
[    0.000000]   DMA32 zone: 896292 pages, LIFO batch:31
[    0.000000]   Normal zone: 2317 pages used for memmap
[    0.000000]   Normal zone: 148287 pages, LIFO batch:31
[    0.000000] p2m virtual area at ffffc90000000000, size is a00000
[    0.000000] Remapped 148287 page(s)
[    0.000000] Reserving Intel graphics stolen memory at 0xdda00000-0xdf9fffff
[    0.000000] ACPI: PM-Timer IO Port: 0x408
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] e820: [mem 0xdfa00000-0xf7ffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.5.2 (preserve-AD)
[    0.000000] clocksource refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] PERCPU: Embedded 29 pages/cpu @ffff880124000000 s81048 r8192 d29544 u262144
[    0.000000] pcpu-alloc: s81048 r8192 d29544 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] xen: PV spinlocks enabled
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1032033
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: placeholder root=UUID=aeaa7943-af67-42eb-b6f9-e73ebf7296de ro rootflags=subvol=rootfs console=hvc0 earlyprintk=xen i915.modeset=1 xen-pciback.hide=(00:1b.0)(00:1d.0)(01:00.0)(02:00.0)(03:00.0)(04:00.0)(05:00.0)(06:00.0)(08:00.0)(0a:08.0)(0a:09.0)(0a:0a.0)(0a:0b.0) softlevel=xen.d
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form
[    0.000000] software IO TLB [mem 0x120000000-0x124000000] (64MB) mapped at [ffff880120000000-ffff880123ffffff]
[    0.000000] Memory: 4022428K/4194300K available (6464K kernel code, 601K rwdata, 1916K rodata, 1048K init, 652K bss, 171872K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	CONFIG_RCU_FANOUT set to non-default value of 32
[    0.000000] NR_IRQS:4352 nr_irqs:488 16
[    0.000000] xen:events: Using FIFO-based ABI
[    0.000000] xen: --> pirq=1 -> irq=1 (gsi=1)
[    0.000000] xen: --> pirq=2 -> irq=2 (gsi=2)
[    0.000000] xen: --> pirq=3 -> irq=3 (gsi=3)
[    0.000000] xen: --> pirq=4 -> irq=4 (gsi=4)
[    0.000000] xen: --> pirq=5 -> irq=5 (gsi=5)
[    0.000000] xen: --> pirq=6 -> irq=6 (gsi=6)
[    0.000000] xen: --> pirq=7 -> irq=7 (gsi=7)
[    0.000000] xen: --> pirq=8 -> irq=8 (gsi=8)
[    0.000000] xen: --> pirq=9 -> irq=9 (gsi=9)
[    0.000000] xen: --> pirq=10 -> irq=10 (gsi=10)
[    0.000000] xen: --> pirq=11 -> irq=11 (gsi=11)
[    0.000000] xen: --> pirq=12 -> irq=12 (gsi=12)
[    0.000000] xen: --> pirq=13 -> irq=13 (gsi=13)
[    0.000000] xen: --> pirq=14 -> irq=14 (gsi=14)
[    0.000000] xen: --> pirq=15 -> irq=15 (gsi=15)
[    0.000000] NO_HZ: Clearing 0 from nohz_full range for timekeeping
[    0.000000] NO_HZ: Full dynticks CPUs: 1-7.
[    0.000000] 	Offload RCU callbacks from all CPUs
[    0.000000] 	Offload RCU callbacks from CPUs: 0-7.
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [hvc0] enabled
[    0.000000] bootconsole [xenboot0] disabled
[    0.000000] clocksource xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2394.640 MHz processor
[    5.857123] Calibrating delay loop (skipped), value calculated using timer frequency.. 4789.28 BogoMIPS (lpj=23946400)
[    5.857128] pid_max: default: 32768 minimum: 501
[    5.857137] ACPI: Core revision 20150410
[    5.882925] ACPI: All ACPI Tables successfully acquired
[    5.883910] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    5.884894] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    5.885262] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    5.885275] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    5.885525] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    5.885529] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    5.885533] CPU: Physical Processor ID: 0
[    5.885535] CPU: Processor Core ID: 0
[    5.885539] mce: CPU supports 2 MCE banks
[    5.885553] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8
[    5.885555] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0
[    5.885978] Freeing SMP alternatives memory: 32K (ffffffff819d5000 - ffffffff819dd000)
[    5.892800] cpu 0 spinlock event irq 25
[    5.893613] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
[    5.893862] installing Xen timer for CPU 1
[    5.893873] cpu 1 spinlock event irq 32
[    5.894185] installing Xen timer for CPU 2
[    5.894195] cpu 2 spinlock event irq 39
[    5.894487] installing Xen timer for CPU 3
[    5.894496] cpu 3 spinlock event irq 46
[    5.894771] installing Xen timer for CPU 4
[    5.894781] cpu 4 spinlock event irq 53
[    5.895042] installing Xen timer for CPU 5
[    5.895051] cpu 5 spinlock event irq 60
[    5.895290] installing Xen timer for CPU 6
[    5.895298] cpu 6 spinlock event irq 67
[    5.895531] installing Xen timer for CPU 7
[    5.895539] cpu 7 spinlock event irq 74
[    5.895687] x86: Booted up 1 node, 8 CPUs
[    5.895864] devtmpfs: initialized
[    5.896262] clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    5.896325] xor: automatically using best checksumming function:
[    5.987368]    avx       : 22624.400 MB/sec
[    5.987463] NET: Registered protocol family 16
[    5.987474] xen:grant_table: Grant tables using version 1 layout
[    5.987483] Grant table initialized
[    5.987765] ACPI: bus type PCI registered
[    5.987898] dca service started, version 1.12.1
[    5.987923] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
[    5.987928] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820
[    5.994529] PCI: Using configuration type 1 for base access
[    6.167436] raid6: sse2x1   gen()  6308 MB/s
[    6.337484] raid6: sse2x1   xor()  5051 MB/s
[    6.507550] raid6: sse2x2   gen()  7885 MB/s
[    6.677602] raid6: sse2x2   xor()  5666 MB/s
[    6.847651] raid6: sse2x4   gen()  9175 MB/s
[    7.017703] raid6: sse2x4   xor()  6816 MB/s
[    7.017706] raid6: using algorithm sse2x4 gen() 9175 MB/s
[    7.017709] raid6: .... xor() 6816 MB/s, rmw enabled
[    7.017711] raid6: using ssse3x2 recovery algorithm
[    7.017758] ACPI: Added _OSI(Module Device)
[    7.017764] ACPI: Added _OSI(Processor Device)
[    7.017769] ACPI: Added _OSI(3.0 _SCP Extensions)
[    7.017772] ACPI: Added _OSI(Processor Aggregator Device)
[    7.019223] xen: registering gsi 9 triggering 0 polarity 0
[    7.020992] ACPI: Executed 1 blocks of module-level executable AML code
[    7.024561] ACPI: Dynamic OEM Table Load:
[    7.024570] ACPI: SSDT 0xFFFF88011FC1D000 00083B (v01 PmRef  Cpu0Cst  00003001 INTL 20051117)
[    7.025236] ACPI: Dynamic OEM Table Load:
[    7.025242] ACPI: SSDT 0xFFFF88011F199000 000303 (v01 PmRef  ApIst    00003000 INTL 20051117)
[    7.025773] ACPI: Dynamic OEM Table Load:
[    7.025779] ACPI: SSDT 0xFFFF88011F123A00 000119 (v01 PmRef  ApCst    00003000 INTL 20051117)
[    7.027766] ACPI: Interpreter enabled
[    7.027773] ACPI: (supports S0 S5)
[    7.027776] ACPI: Using IOAPIC for interrupt routing
[    7.027806] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    7.035495] ACPI: Power Resource [FN00] (off)
[    7.035571] ACPI: Power Resource [FN01] (off)
[    7.035644] ACPI: Power Resource [FN02] (off)
[    7.035717] ACPI: Power Resource [FN03] (off)
[    7.035791] ACPI: Power Resource [FN04] (off)
[    7.036475] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e])
[    7.036482] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    7.036835] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME]
[    7.037043] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability]
[    7.037422] PCI host bridge to bus 0000:00
[    7.037427] pci_bus 0000:00: root bus resource [bus 00-3e]
[    7.037430] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    7.037433] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    7.037437] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    7.037440] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window]
[    7.037443] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window]
[    7.037447] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window]
[    7.037451] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window]
[    7.037454] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window]
[    7.037458] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window]
[    7.037461] pci_bus 0000:00: root bus resource [mem 0xdfa00000-0xfeafffff window]
[    7.037476] pci 0000:00:00.0: [8086:0108] type 00 class 0x060000
[    7.037634] pci 0000:00:01.0: [8086:0101] type 01 class 0x060400
[    7.037733] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    7.037780] pci 0000:00:01.0: System wakeup disabled by ACPI
[    7.037838] pci 0000:00:01.1: [8086:0105] type 01 class 0x060400
[    7.037923] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
[    7.037966] pci 0000:00:01.1: System wakeup disabled by ACPI
[    7.038062] pci 0000:00:02.0: [8086:010a] type 00 class 0x030000
[    7.038088] pci 0000:00:02.0: reg 0x10: [mem 0xf7400000-0xf77fffff 64bit]
[    7.038106] pci 0000:00:02.0: reg 0x18: [mem 0xe0000000-0xefffffff 64bit pref]
[    7.038117] pci 0000:00:02.0: reg 0x20: [io  0xf000-0xf03f]
[    7.038284] pci 0000:00:06.0: [8086:010d] type 01 class 0x060400
[    7.038369] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
[    7.038413] pci 0000:00:06.0: System wakeup disabled by ACPI
[    7.038492] pci 0000:00:16.0: [8086:1c3a] type 00 class 0x078000
[    7.038529] pci 0000:00:16.0: reg 0x10: [mem 0xf7f2c000-0xf7f2c00f 64bit]
[    7.038655] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
[    7.038766] pci 0000:00:19.0: [8086:1502] type 00 class 0x020000
[    7.038795] pci 0000:00:19.0: reg 0x10: [mem 0xf7f00000-0xf7f1ffff]
[    7.038809] pci 0000:00:19.0: reg 0x14: [mem 0xf7f29000-0xf7f29fff]
[    7.038824] pci 0000:00:19.0: reg 0x18: [io  0xf080-0xf09f]
[    7.038931] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold
[    7.038977] pci 0000:00:19.0: System wakeup disabled by ACPI
[    7.039033] pci 0000:00:1a.0: [8086:1c2d] type 00 class 0x0c0320
[    7.039064] pci 0000:00:1a.0: reg 0x10: [mem 0xf7f28000-0xf7f283ff]
[    7.039207] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
[    7.039269] pci 0000:00:1a.0: System wakeup disabled by ACPI
[    7.039330] pci 0000:00:1b.0: [8086:1c20] type 00 class 0x040300
[    7.039360] pci 0000:00:1b.0: reg 0x10: [mem 0xf7f20000-0xf7f23fff 64bit]
[    7.039523] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    7.039570] pci 0000:00:1b.0: System wakeup disabled by ACPI
[    7.039628] pci 0000:00:1c.0: [8086:1c10] type 01 class 0x060400
[    7.039755] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    7.039788] pci 0000:00:1c.0: Enabling MPC IRBNCE
[    7.039793] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[    7.039827] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    7.039890] pci 0000:00:1c.4: [8086:1c18] type 01 class 0x060400
[    7.040018] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[    7.040048] pci 0000:00:1c.4: Enabling MPC IRBNCE
[    7.040053] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled
[    7.040087] pci 0000:00:1c.4: System wakeup disabled by ACPI
[    7.040149] pci 0000:00:1c.6: [8086:1c1c] type 01 class 0x060400
[    7.040276] pci 0000:00:1c.6: PME# supported from D0 D3hot D3cold
[    7.040306] pci 0000:00:1c.6: Enabling MPC IRBNCE
[    7.040311] pci 0000:00:1c.6: Intel PCH root port ACS workaround enabled
[    7.040345] pci 0000:00:1c.6: System wakeup disabled by ACPI
[    7.040405] pci 0000:00:1d.0: [8086:1c26] type 00 class 0x0c0320
[    7.040436] pci 0000:00:1d.0: reg 0x10: [mem 0xf7f27000-0xf7f273ff]
[    7.040579] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    7.040642] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    7.040696] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    7.040855] pci 0000:00:1e.0: System wakeup disabled by ACPI
[    7.040909] pci 0000:00:1f.0: [8086:1c56] type 00 class 0x060100
[    7.041165] pci 0000:00:1f.2: [8086:1c02] type 00 class 0x010601
[    7.041195] pci 0000:00:1f.2: reg 0x10: [io  0xf0d0-0xf0d7]
[    7.041209] pci 0000:00:1f.2: reg 0x14: [io  0xf0c0-0xf0c3]
[    7.041223] pci 0000:00:1f.2: reg 0x18: [io  0xf0b0-0xf0b7]
[    7.041238] pci 0000:00:1f.2: reg 0x1c: [io  0xf0a0-0xf0a3]
[    7.041252] pci 0000:00:1f.2: reg 0x20: [io  0xf060-0xf07f]
[    7.041267] pci 0000:00:1f.2: reg 0x24: [mem 0xf7f26000-0xf7f267ff]
[    7.041339] pci 0000:00:1f.2: PME# supported from D3hot
[    7.041429] pci 0000:00:1f.3: [8086:1c22] type 00 class 0x0c0500
[    7.041457] pci 0000:00:1f.3: reg 0x10: [mem 0xf7f25000-0xf7f250ff 64bit]
[    7.041497] pci 0000:00:1f.3: reg 0x20: [io  0xf040-0xf05f]
[    7.041674] pci 0000:01:00.0: [1000:0072] type 00 class 0x010700
[    7.041693] pci 0000:01:00.0: reg 0x10: [io  0xe000-0xe0ff]
[    7.041712] pci 0000:01:00.0: reg 0x14: [mem 0xf7ec0000-0xf7ec3fff 64bit]
[    7.041731] pci 0000:01:00.0: reg 0x1c: [mem 0xf7e80000-0xf7ebffff 64bit]
[    7.041755] pci 0000:01:00.0: reg 0x30: [mem 0xf7e00000-0xf7e7ffff pref]
[    7.041820] pci 0000:01:00.0: supports D1 D2
[    7.057784] pci 0000:00:01.0: PCI bridge to [bus 01]
[    7.057793] pci 0000:00:01.0:   bridge window [io  0xe000-0xefff]
[    7.057798] pci 0000:00:01.0:   bridge window [mem 0xf7e00000-0xf7efffff]
[    7.057883] pci 0000:02:00.0: [1b21:1080] type 01 class 0x060400
[    7.058029] pci 0000:02:00.0: supports D1 D2
[    7.058030] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    7.077819] pci 0000:00:01.1: PCI bridge to [bus 02-03]
[    7.077831] pci 0000:00:01.1:   bridge window [mem 0xf7d00000-0xf7dfffff]
[    7.077924] pci 0000:03:00.0: [1b74:0100] type 00 class 0xff0000
[    7.077951] pci 0000:03:00.0: reg 0x10: [mem 0xf7d00000-0xf7d0ffff]
[    7.078199] pci 0000:02:00.0: PCI bridge to [bus 03]
[    7.078216] pci 0000:02:00.0:   bridge window [mem 0xf7d00000-0xf7dfffff]
[    7.078338] pci 0000:04:00.0: [168c:0030] type 00 class 0x028000
[    7.078369] pci 0000:04:00.0: reg 0x10: [mem 0xf7c00000-0xf7c1ffff 64bit]
[    7.078438] pci 0000:04:00.0: reg 0x30: [mem 0xf7c20000-0xf7c2ffff pref]
[    7.078505] pci 0000:04:00.0: supports D1
[    7.078507] pci 0000:04:00.0: PME# supported from D0 D1 D3hot
[    7.097851] pci 0000:00:06.0: PCI bridge to [bus 04]
[    7.097862] pci 0000:00:06.0:   bridge window [mem 0xf7c00000-0xf7cfffff]
[    7.097977] pci 0000:05:00.0: [10b5:8112] type 01 class 0x060400
[    7.098210] pci 0000:05:00.0: supports D1
[    7.098212] pci 0000:05:00.0: PME# supported from D0 D1 D3hot
[    7.098251] pci 0000:05:00.0: System wakeup disabled by ACPI
[    7.098309] pci 0000:05:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    7.098329] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
[    7.098340] pci 0000:00:1c.0:   bridge window [mem 0xf7b00000-0xf7bfffff]
[    7.098446] pci 0000:06:00.0: [1b74:0810] type 00 class 0x078000
[    7.098484] pci 0000:06:00.0: reg 0x10: [mem 0xf7b00000-0xf7b7ffff]
[    7.098864] pci 0000:05:00.0: PCI bridge to [bus 06]
[    7.098881] pci 0000:05:00.0:   bridge window [mem 0xf7b00000-0xf7bfffff]
[    7.099005] pci 0000:07:00.0: [8086:10d3] type 00 class 0x020000
[    7.099082] pci 0000:07:00.0: reg 0x10: [mem 0xf7a40000-0xf7a5ffff]
[    7.099133] pci 0000:07:00.0: reg 0x18: [io  0xd000-0xd01f]
[    7.099159] pci 0000:07:00.0: reg 0x1c: [mem 0xf7a60000-0xf7a63fff]
[    7.099234] pci 0000:07:00.0: reg 0x30: [mem 0xf7a00000-0xf7a3ffff pref]
[    7.099370] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
[    7.099411] pci 0000:07:00.0: System wakeup disabled by ACPI
[    7.117941] pci 0000:00:1c.4: PCI bridge to [bus 07]
[    7.117952] pci 0000:00:1c.4:   bridge window [io  0xd000-0xdfff]
[    7.117958] pci 0000:00:1c.4:   bridge window [mem 0xf7a00000-0xf7afffff]
[    7.118069] pci 0000:08:00.0: [1033:0194] type 00 class 0x0c0330
[    7.118113] pci 0000:08:00.0: reg 0x10: [mem 0xf7900000-0xf7901fff 64bit]
[    7.118325] pci 0000:08:00.0: PME# supported from D0 D3hot
[    7.118365] pci 0000:08:00.0: System wakeup disabled by ACPI
[    7.118498] pci 0000:00:1c.6: PCI bridge to [bus 08]
[    7.118509] pci 0000:00:1c.6:   bridge window [mem 0xf7900000-0xf79fffff]
[    7.118576] pci 0000:09:00.0: [12d8:8148] type 01 class 0x060400
[    7.118707] pci 0000:09:00.0: supports D1 D2
[    7.118826] pci 0000:00:1e.0: PCI bridge to [bus 09-0a] (subtractive decode)
[    7.118834] pci 0000:00:1e.0:   bridge window [io  0xc000-0xcfff]
[    7.118839] pci 0000:00:1e.0:   bridge window [mem 0xf7800000-0xf78fffff]
[    7.118849] pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
[    7.118850] pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff window] (subtractive decode)
[    7.118852] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode)
[    7.118854] pci 0000:00:1e.0:   bridge window [mem 0x000d0000-0x000d3fff window] (subtractive decode)
[    7.118855] pci 0000:00:1e.0:   bridge window [mem 0x000d4000-0x000d7fff window] (subtractive decode)
[    7.118857] pci 0000:00:1e.0:   bridge window [mem 0x000d8000-0x000dbfff window] (subtractive decode)
[    7.118859] pci 0000:00:1e.0:   bridge window [mem 0x000dc000-0x000dffff window] (subtractive decode)
[    7.118860] pci 0000:00:1e.0:   bridge window [mem 0x000e0000-0x000e3fff window] (subtractive decode)
[    7.118862] pci 0000:00:1e.0:   bridge window [mem 0x000e4000-0x000e7fff window] (subtractive decode)
[    7.118863] pci 0000:00:1e.0:   bridge window [mem 0xdfa00000-0xfeafffff window] (subtractive decode)
[    7.118957] pci 0000:0a:08.0: [10ec:8139] type 00 class 0x020000
[    7.118993] pci 0000:0a:08.0: reg 0x10: [io  0xc200-0xc2ff]
[    7.119014] pci 0000:0a:08.0: reg 0x14: [mem 0xf7842000-0xf78420ff]
[    7.119114] pci 0000:0a:08.0: reg 0x30: [mem 0xf7830000-0xf783ffff pref]
[    7.119169] pci 0000:0a:08.0: supports D1 D2
[    7.119171] pci 0000:0a:08.0: PME# supported from D1 D2 D3hot D3cold
[    7.119244] pci 0000:0a:09.0: [10ec:8139] type 00 class 0x020000
[    7.119279] pci 0000:0a:09.0: reg 0x10: [io  0xc100-0xc1ff]
[    7.119301] pci 0000:0a:09.0: reg 0x14: [mem 0xf7841000-0xf78410ff]
[    7.119400] pci 0000:0a:09.0: reg 0x30: [mem 0xf7820000-0xf782ffff pref]
[    7.119455] pci 0000:0a:09.0: supports D1 D2
[    7.119457] pci 0000:0a:09.0: PME# supported from D1 D2 D3hot D3cold
[    7.119528] pci 0000:0a:0a.0: [10ec:8139] type 00 class 0x020000
[    7.119564] pci 0000:0a:0a.0: reg 0x10: [io  0xc000-0xc0ff]
[    7.119585] pci 0000:0a:0a.0: reg 0x14: [mem 0xf7840000-0xf78400ff]
[    7.119685] pci 0000:0a:0a.0: reg 0x30: [mem 0xf7810000-0xf781ffff pref]
[    7.119740] pci 0000:0a:0a.0: supports D1 D2
[    7.119741] pci 0000:0a:0a.0: PME# supported from D1 D2 D3hot D3cold
[    7.119814] pci 0000:0a:0b.0: [168c:0029] type 00 class 0x028000
[    7.119885] pci 0000:0a:0b.0: reg 0x10: [mem 0xf7800000-0xf780ffff]
[    7.120058] pci 0000:0a:0b.0: PME# supported from D0 D3hot
[    7.120167] pci 0000:09:00.0: PCI bridge to [bus 0a]
[    7.120180] pci 0000:09:00.0:   bridge window [io  0xc000-0xcfff]
[    7.120186] pci 0000:09:00.0:   bridge window [mem 0xf7800000-0xf78fffff]
[    7.121200] xen: registering gsi 13 triggering 1 polarity 0
[    7.121416] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 11 12 *14 15)
[    7.121484] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 *15)
[    7.121549] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
[    7.121614] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12 14 15)
[    7.121679] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 *14 15)
[    7.121742] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
[    7.121808] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 *15)
[    7.121872] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 10 11 12 *14 15)
[    7.122093] ACPI: Enabled 4 GPEs in block 00 to 3F
[    7.122131] xen:balloon: Initialising balloon driver
[    7.122282] xen_balloon: Initialising balloon driver
[    7.122407] vgaarb: setting as boot device: PCI:0000:00:02.0
[    7.122411] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    7.122417] vgaarb: loaded
[    7.122419] vgaarb: bridge control possible 0000:00:02.0
[    7.122478] SCSI subsystem initialized
[    7.122490] libata version 3.00 loaded.
[    7.122493] ACPI: bus type USB registered
[    7.122511] usbcore: registered new interface driver usbfs
[    7.122521] usbcore: registered new interface driver hub
[    7.122546] usbcore: registered new device driver usb
[    7.122572] pps_core: LinuxPPS API ver. 1 registered
[    7.122574] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    7.122581] PTP clock support registered
[    7.122744] PCI: Using ACPI for IRQ routing
[    7.125846] PCI: pci_cache_line_size set to 64 bytes
[    7.125989] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff]
[    7.125991] e820: reserve RAM buffer [mem 0xdb9f0000-0xdbffffff]
[    7.125992] e820: reserve RAM buffer [mem 0xdc652000-0xdfffffff]
[    7.125994] e820: reserve RAM buffer [mem 0xdcdba000-0xdfffffff]
[    7.125996] e820: reserve RAM buffer [mem 0xdd000000-0xdfffffff]
[    7.125997] e820: reserve RAM buffer [mem 0x12433f000-0x127ffffff]
[    7.126196] Switched to clocksource xen
[    7.126334] pnp: PnP ACPI init
[    7.126412] system 00:00: [mem 0xfed40000-0xfed44fff] has been reserved
[    7.126417] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[    7.126491] system 00:01: [io  0x0680-0x069f] has been reserved
[    7.126495] system 00:01: [io  0x1000-0x100f] has been reserved
[    7.126498] system 00:01: [io  0xffff] has been reserved
[    7.126501] system 00:01: [io  0xffff] has been reserved
[    7.126504] system 00:01: [io  0x0400-0x0453] could not be reserved
[    7.126508] system 00:01: [io  0x0458-0x047f] has been reserved
[    7.126511] system 00:01: [io  0x0500-0x057f] has been reserved
[    7.126514] system 00:01: [io  0x164e-0x164f] has been reserved
[    7.126518] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    7.126525] xen: registering gsi 8 triggering 1 polarity 0
[    7.126554] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
[    7.126600] system 00:03: [io  0x0454-0x0457] has been reserved
[    7.126604] system 00:03: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    7.126768] system 00:04: [io  0x0a00-0x0a0f] has been reserved
[    7.126772] system 00:04: [io  0x0a10-0x0a1f] has been reserved
[    7.126805] system 00:04: [io  0x0a10-0x0a1f] has been reserved
[    7.126808] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active)
[    7.126829] xen: registering gsi 1 triggering 1 polarity 0
[    7.126863] pnp 00:05: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[    7.127155] ACPI: IRQ 4 override to edge, high
[    7.127158] xen: registering gsi 4 triggering 1 polarity 0
[    7.127160] Already setup the GSI :4
[    7.127207] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[    7.127808] system 00:07: [io  0x04d0-0x04d1] has been reserved
[    7.127813] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    7.128070] system 00:08: [mem 0xfed1c000-0xfed1ffff] has been reserved
[    7.128074] system 00:08: [mem 0xfed10000-0xfed17fff] has been reserved
[    7.128077] system 00:08: [mem 0xfed18000-0xfed18fff] has been reserved
[    7.128081] system 00:08: [mem 0xfed19000-0xfed19fff] has been reserved
[    7.128084] system 00:08: [mem 0xf8000000-0xfbffffff] has been reserved
[    7.128087] system 00:08: [mem 0xfed20000-0xfed3ffff] has been reserved
[    7.128091] system 00:08: [mem 0xfed90000-0xfed93fff] could not be reserved
[    7.128094] system 00:08: [mem 0xfed45000-0xfed8ffff] has been reserved
[    7.128098] system 00:08: [mem 0xff000000-0xffffffff] has been reserved
[    7.128101] system 00:08: [mem 0xfee00000-0xfeefffff] has been reserved
[    7.128104] system 00:08: [mem 0xdfa00000-0xdfa00fff] has been reserved
[    7.128108] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    7.128274] system 00:09: [mem 0x20000000-0x201fffff] has been reserved
[    7.128277] system 00:09: [mem 0x40000000-0x401fffff] has been reserved
[    7.128281] system 00:09: Plug and Play ACPI device, IDs PNP0c01 (active)
[    7.128297] pnp: PnP ACPI: found 10 devices
[    7.128333] pciback 0000:00:1b.0: seizing device
[    7.128345] pciback 0000:00:1d.0: seizing device
[    7.128360] pciback 0000:01:00.0: seizing device
[    7.128365] pciback 0000:02:00.0: seizing device
[    7.128370] pciback 0000:03:00.0: seizing device
[    7.128374] pciback 0000:04:00.0: seizing device
[    7.128379] pciback 0000:05:00.0: seizing device
[    7.128383] pciback 0000:06:00.0: seizing device
[    7.128390] pciback 0000:08:00.0: seizing device
[    7.128397] pciback 0000:0a:08.0: seizing device
[    7.128403] pciback 0000:0a:09.0: seizing device
[    7.128408] pciback 0000:0a:0a.0: seizing device
[    7.128412] pciback 0000:0a:0b.0: seizing device
[    7.135752] PM-Timer failed consistency check  (0xffffff) - aborting.
[    7.135875] pci 0000:00:01.0: PCI bridge to [bus 01]
[    7.135880] pci 0000:00:01.0:   bridge window [io  0xe000-0xefff]
[    7.135888] pci 0000:00:01.0:   bridge window [mem 0xf7e00000-0xf7efffff]
[    7.135899] pciback 0000:02:00.0: PCI bridge to [bus 03]
[    7.135909] pciback 0000:02:00.0:   bridge window [mem 0xf7d00000-0xf7dfffff]
[    7.135925] pci 0000:00:01.1: PCI bridge to [bus 02-03]
[    7.135931] pci 0000:00:01.1:   bridge window [mem 0xf7d00000-0xf7dfffff]
[    7.135943] pci 0000:00:06.0: PCI bridge to [bus 04]
[    7.135949] pci 0000:00:06.0:   bridge window [mem 0xf7c00000-0xf7cfffff]
[    7.135960] pciback 0000:05:00.0: PCI bridge to [bus 06]
[    7.135974] pciback 0000:05:00.0:   bridge window [mem 0xf7b00000-0xf7bfffff]
[    7.135999] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
[    7.136008] pci 0000:00:1c.0:   bridge window [mem 0xf7b00000-0xf7bfffff]
[    7.136023] pci 0000:00:1c.4: PCI bridge to [bus 07]
[    7.136027] pci 0000:00:1c.4:   bridge window [io  0xd000-0xdfff]
[    7.136036] pci 0000:00:1c.4:   bridge window [mem 0xf7a00000-0xf7afffff]
[    7.136051] pci 0000:00:1c.6: PCI bridge to [bus 08]
[    7.136060] pci 0000:00:1c.6:   bridge window [mem 0xf7900000-0xf79fffff]
[    7.136075] pci 0000:09:00.0: PCI bridge to [bus 0a]
[    7.136080] pci 0000:09:00.0:   bridge window [io  0xc000-0xcfff]
[    7.136090] pci 0000:09:00.0:   bridge window [mem 0xf7800000-0xf78fffff]
[    7.136106] pci 0000:00:1e.0: PCI bridge to [bus 09-0a]
[    7.136111] pci 0000:00:1e.0:   bridge window [io  0xc000-0xcfff]
[    7.136120] pci 0000:00:1e.0:   bridge window [mem 0xf7800000-0xf78fffff]
[    7.136134] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    7.136136] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    7.136138] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    7.136140] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff window]
[    7.136141] pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff window]
[    7.136143] pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff window]
[    7.136145] pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff window]
[    7.136146] pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff window]
[    7.136148] pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff window]
[    7.136150] pci_bus 0000:00: resource 13 [mem 0xdfa00000-0xfeafffff window]
[    7.136151] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    7.136153] pci_bus 0000:01: resource 1 [mem 0xf7e00000-0xf7efffff]
[    7.136155] pci_bus 0000:02: resource 1 [mem 0xf7d00000-0xf7dfffff]
[    7.136156] pci_bus 0000:03: resource 1 [mem 0xf7d00000-0xf7dfffff]
[    7.136158] pci_bus 0000:04: resource 1 [mem 0xf7c00000-0xf7cfffff]
[    7.136160] pci_bus 0000:05: resource 1 [mem 0xf7b00000-0xf7bfffff]
[    7.136161] pci_bus 0000:06: resource 1 [mem 0xf7b00000-0xf7bfffff]
[    7.136163] pci_bus 0000:07: resource 0 [io  0xd000-0xdfff]
[    7.136165] pci_bus 0000:07: resource 1 [mem 0xf7a00000-0xf7afffff]
[    7.136166] pci_bus 0000:08: resource 1 [mem 0xf7900000-0xf79fffff]
[    7.136168] pci_bus 0000:09: resource 0 [io  0xc000-0xcfff]
[    7.136169] pci_bus 0000:09: resource 1 [mem 0xf7800000-0xf78fffff]
[    7.136171] pci_bus 0000:09: resource 4 [io  0x0000-0x0cf7 window]
[    7.136173] pci_bus 0000:09: resource 5 [io  0x0d00-0xffff window]
[    7.136174] pci_bus 0000:09: resource 6 [mem 0x000a0000-0x000bffff window]
[    7.136176] pci_bus 0000:09: resource 7 [mem 0x000d0000-0x000d3fff window]
[    7.136178] pci_bus 0000:09: resource 8 [mem 0x000d4000-0x000d7fff window]
[    7.136179] pci_bus 0000:09: resource 9 [mem 0x000d8000-0x000dbfff window]
[    7.136181] pci_bus 0000:09: resource 10 [mem 0x000dc000-0x000dffff window]
[    7.136192] pci_bus 0000:09: resource 11 [mem 0x000e0000-0x000e3fff window]
[    7.136194] pci_bus 0000:09: resource 12 [mem 0x000e4000-0x000e7fff window]
[    7.136196] pci_bus 0000:09: resource 13 [mem 0xdfa00000-0xfeafffff window]
[    7.136198] pci_bus 0000:0a: resource 0 [io  0xc000-0xcfff]
[    7.136199] pci_bus 0000:0a: resource 1 [mem 0xf7800000-0xf78fffff]
[    7.136223] NET: Registered protocol family 2
[    7.136371] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
[    7.136487] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
[    7.136555] TCP: Hash tables configured (established 32768 bind 32768)
[    7.136584] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    7.136607] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    7.136649] NET: Registered protocol family 1
[    7.136750] RPC: Registered named UNIX socket transport module.
[    7.136753] RPC: Registered udp transport module.
[    7.136756] RPC: Registered tcp transport module.
[    7.136758] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    7.137220] pci 0000:00:02.0: BIOS left Intel GPU interrupts enabled; disabling
[    7.137976] pci 0000:00:02.0: Video device with shadowed ROM
[    7.138063] xen: registering gsi 16 triggering 0 polarity 1
[    7.138073] xen: --> pirq=16 -> irq=16 (gsi=16)
[    7.156353] xen: registering gsi 23 triggering 0 polarity 1
[    7.156361] xen: --> pirq=23 -> irq=23 (gsi=23)
[    7.176384] PCI: CLS mismatch (64 != 32), using 64 bytes
[    7.176455] xen: registering gsi 18 triggering 0 polarity 1
[    7.176464] xen: --> pirq=18 -> irq=18 (gsi=18)
[    7.176511] xen: registering gsi 18 triggering 0 polarity 1
[    7.176513] Already setup the GSI :18
[    7.176628] Unpacking initramfs...
[    7.181511] Freeing initrd memory: 9208K (ffff880004000000 - ffff8800048fe000)
[    7.181617] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer
[    7.181621] hw unit of domain pp0-core 2^-16 Joules
[    7.181623] hw unit of domain package 2^-16 Joules
[    7.181626] hw unit of domain pp1-gpu 2^-16 Joules
[    7.182865] AVX version of gcm_enc/dec engaged.
[    7.182870] AES CTR mode by8 optimization enabled
[    7.183733] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    7.183901] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    7.185768] zpool: loaded
[    7.185772] zbud: loaded
[    7.186507] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    7.186727] NFS: Registering the id_resolver key type
[    7.186737] Key type id_resolver registered
[    7.186740] Key type id_legacy registered
[    7.186744] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    7.187105] async_tx: api initialized (async)
[    7.187139] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    7.187144] io scheduler noop registered
[    7.187186] io scheduler cfq registered (default)
[    7.187346] xen: registering gsi 16 triggering 0 polarity 1
[    7.187350] Already setup the GSI :16
[    7.187532] xen: registering gsi 16 triggering 0 polarity 1
[    7.187534] Already setup the GSI :16
[    7.187690] xen: registering gsi 19 triggering 0 polarity 1
[    7.187699] xen: --> pirq=19 -> irq=19 (gsi=19)
[    7.187864] xen: registering gsi 16 triggering 0 polarity 1
[    7.187866] Already setup the GSI :16
[    7.187997] xen: registering gsi 16 triggering 0 polarity 1
[    7.187998] Already setup the GSI :16
[    7.188168] intel_idle: MWAIT substates: 0x1120
[    7.188170] intel_idle: v0.4 model 0x2A
[    7.188171] intel_idle: lapic_timer_reliable_states 0xffffffff
[    7.188222] intel_idle: intel_idle yielding to none
[    7.188228] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    7.188392] xen:xen_evtchn: Event-channel device installed
[    7.188658] xen: registering gsi 19 triggering 0 polarity 1
[    7.188660] Already setup the GSI :19
[    7.236394] xen: registering gsi 18 triggering 0 polarity 1
[    7.236398] Already setup the GSI :18
[    7.276402] xen: registering gsi 17 triggering 0 polarity 1
[    7.276413] xen: --> pirq=17 -> irq=17 (gsi=17)
[    7.316413] xen: registering gsi 16 triggering 0 polarity 1
[    7.316417] Already setup the GSI :16
[    7.356411] xen: registering gsi 18 triggering 0 polarity 1
[    7.356414] Already setup the GSI :18
[    8.176258] clocksource tsc: mask: 0xffffffffffffffff max_cycles: 0x22847025dca, max_idle_ns: 440795321856 ns
[    8.386463] xen: registering gsi 16 triggering 0 polarity 1
[    8.386467] Already setup the GSI :16
[    8.386506] xen: registering gsi 16 triggering 0 polarity 1
[    8.386508] Already setup the GSI :16
[    9.176558] Switched to clocksource tsc
[    9.456419] xen: registering gsi 19 triggering 0 polarity 1
[    9.456422] Already setup the GSI :19
[    9.496408] xen: registering gsi 17 triggering 0 polarity 1
[    9.496412] Already setup the GSI :17
[    9.496445] xen: registering gsi 17 triggering 0 polarity 1
[    9.496447] Already setup the GSI :17
[   10.526605] xen: registering gsi 16 triggering 0 polarity 1
[   10.526608] Already setup the GSI :16
[   10.636527] xen: registering gsi 23 triggering 0 polarity 1
[   10.636531] Already setup the GSI :23
[   10.746540] xen: registering gsi 22 triggering 0 polarity 1
[   10.746550] xen: --> pirq=22 -> irq=22 (gsi=22)
[   10.856528] xen_pciback: backend is vpci
[   10.861420] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[   10.861873] hpet_acpi_add: no address or irqs in _CRS
[   10.861902] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[   10.863363] loop: module loaded
[   10.863477] zram: Created 1 device(s)
[   10.863576] xen: registering gsi 16 triggering 0 polarity 1
[   10.863579] Already setup the GSI :16
[   10.869720] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042F conflicts with OpRegion 0x0000000000000400-0x000000000000047F (\PMIO) (20150410/utaddress-254)
[   10.869730] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.869736] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20150410/utaddress-254)
[   10.869742] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.869746] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20150410/utaddress-254)
[   10.869752] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.869755] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20150410/utaddress-254)
[   10.869762] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.869766] lpc_ich: Resource conflict(s) found affecting gpio_ich
[   10.869830] ahci 0000:00:1f.2: version 3.0
[   10.869903] xen: registering gsi 19 triggering 0 polarity 1
[   10.869906] Already setup the GSI :19
[   10.886312] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[   10.886317] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst 
[   10.986935] scsi host0: ahci
[   10.987191] scsi host1: ahci
[   10.987313] scsi host2: ahci
[   10.987424] scsi host3: ahci
[   10.987531] scsi host4: ahci
[   10.987635] scsi host5: ahci
[   10.987671] ata1: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26100 irq 88
[   10.987676] ata2: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26180 irq 88
[   10.987679] ata3: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26200 irq 88
[   10.987683] ata4: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26280 irq 88
[   10.987687] ata5: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26300 irq 88
[   10.987690] ata6: SATA max UDMA/133 abar m2048@0xf7f26000 port 0xf7f26380 irq 88
[   10.987724] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[   10.987728] e1000e: Copyright(c) 1999 - 2014 Intel Corporation.
[   10.987833] xen: registering gsi 20 triggering 0 polarity 1
[   10.987843] xen: --> pirq=20 -> irq=20 (gsi=20)
[   10.987961] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   11.192428] e1000e 0000:00:19.0 eth0: registered PHC clock
[   11.192433] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) 00:18:7d:1d:72:74
[   11.192437] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[   11.192480] e1000e 0000:00:19.0 eth0: MAC: 10, PHY: 11, PBA No: FFFFFF-0FF
[   11.192543] xen: registering gsi 16 triggering 0 polarity 1
[   11.192545] Already setup the GSI :16
[   11.192742] e1000e 0000:07:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   11.310330] e1000e 0000:07:00.0 eth1: registered PHC clock
[   11.310336] e1000e 0000:07:00.0 eth1: (PCI Express:2.5GT/s:Width x1) 00:18:7d:1d:71:de
[   11.310340] e1000e 0000:07:00.0 eth1: Intel(R) PRO/1000 Network Connection
[   11.310431] e1000e 0000:07:00.0 eth1: MAC: 3, PHY: 8, PBA No: FFFFFF-0FF
[   11.310463] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   11.310468] ehci-pci: EHCI PCI platform driver
[   11.310555] xen: registering gsi 16 triggering 0 polarity 1
[   11.310557] Already setup the GSI :16
[   11.310593] ehci-pci 0000:00:1a.0: EHCI Host Controller
[   11.310600] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1
[   11.310619] ehci-pci 0000:00:1a.0: debug port 2
[   11.314579] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported
[   11.314617] ehci-pci 0000:00:1a.0: irq 16, io mem 0xf7f28000
[   11.326295] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
[   11.326478] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   11.326484] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   11.326488] usb usb1: Product: EHCI Host Controller
[   11.326490] usb usb1: Manufacturer: Linux 4.1.7-hardened-r1 ehci_hcd
[   11.326493] usb usb1: SerialNumber: 0000:00:1a.0
[   11.326639] hub 1-0:1.0: USB hub found
[   11.326661] hub 1-0:1.0: 2 ports detected
[   11.326933] usbcore: registered new interface driver usbserial
[   11.326953] usbcore: registered new interface driver ftdi_sio
[   11.326963] usbserial: USB Serial support registered for FTDI USB Serial Device
[   11.326997] i8042: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[   11.327000] i8042: PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
[   11.327172] serio: i8042 KBD port at 0x60,0x64 irq 1
[   11.327346] mousedev: PS/2 mouse device common for all mice
[   11.327394] rtc_cmos 00:02: RTC can wake from S4
[   11.327628] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[   11.327689] rtc_cmos 00:02: alarms up to one month, y3k, 242 bytes nvram
[   11.327899] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   11.327922] iTCO_wdt: Found a Cougar Point TCO device (Version=2, TCOBASE=0x0460)
[   11.328058] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[   11.328074] md: raid1 personality registered for level 1
[   11.328079] md: raid10 personality registered for level 10
[   11.328234] md: raid6 personality registered for level 6
[   11.328238] md: raid5 personality registered for level 5
[   11.328241] md: raid4 personality registered for level 4
[   11.328331] device-mapper: uevent: version 1.0.3
[   11.328489] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12) initialised: dm-devel@redhat.com
[   11.328498] device-mapper: raid: Loading target version 1.6.0
[   11.328519] hidraw: raw HID events driver (C) Jiri Kosina
[   11.328655] NET: Registered protocol family 17
[   11.328677] Key type dns_resolver registered
[   11.328863] mce: Unable to init device /dev/mcelog (rc: -16)
[   11.329351] registered taskstats version 1
[   11.329910] Btrfs loaded
[   11.330651] rtc_cmos 00:02: setting system clock to 2015-11-15 23:53:36 UTC (1447631616)
[   11.336299] ata5: SATA link down (SStatus 0 SControl 300)
[   11.336332] ata3: SATA link down (SStatus 0 SControl 300)
[   11.336354] ata4: SATA link down (SStatus 0 SControl 300)
[   11.336375] ata6: SATA link down (SStatus 0 SControl 300)
[   11.346342] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.346365] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.346413] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[   11.347957] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[   11.347962] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[   11.347970] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[   11.348276] ata2.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[   11.348279] ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[   11.348285] ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[   11.349605] ata1.00: ATA-8: WDC WD1003FBYZ-010FB0, 01.01V03, max UDMA/133
[   11.349611] ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[   11.349912] ata2.00: ATA-8: WDC WD1003FBYZ-010FB0, 01.01V03, max UDMA/133
[   11.349915] ata2.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[   11.351161] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[   11.351164] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[   11.351171] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[   11.351736] ata2.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[   11.351740] ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[   11.351746] ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[   11.352530] ata1.00: configured for UDMA/133
[   11.352700] scsi 0:0:0:0: Direct-Access     ATA      WDC WD1003FBYZ-0 1V03 PQ: 0 ANSI: 5
[   11.352930] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[   11.353096] sd 0:0:0:0: [sda] Write Protect is off
[   11.353102] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   11.353147] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.354189] ata2.00: configured for UDMA/133
[   11.354304] scsi 1:0:0:0: Direct-Access     ATA      WDC WD1003FBYZ-0 1V03 PQ: 0 ANSI: 5
[   11.354541] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[   11.354655] sd 1:0:0:0: [sdb] Write Protect is off
[   11.354660] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[   11.354693] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.399691]  sda: sda1 sda2 sda3 sda4 sda5
[   11.400298] sd 0:0:0:0: [sda] Attached SCSI disk
[   11.402760]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5
[   11.403436] sd 1:0:0:0: [sdb] Attached SCSI disk
[   11.403838] Freeing unused kernel memory: 1048K (ffffffff818cf000 - ffffffff819d5000)
[   11.644555] BTRFS: device label root devid 1 transid 926982 /dev/sdb4
[   11.644694] BTRFS: device label root devid 3 transid 926982 /dev/sda4
[   11.646312] usb 1-1: new high-speed USB device number 2 using ehci-pci
[   11.647928] md: md1 stopped.
[   11.648542] md: bind<sda2>
[   11.648696] md: bind<sdb2>
[   11.649038] md/raid1:md1: active with 2 out of 2 mirrors
[   11.649058] md1: detected capacity change from 0 to 1073676288
[   11.652270] md: md2 stopped.
[   11.652850] md: bind<sda3>
[   11.652998] md: bind<sdb3>
[   11.653311] md/raid1:md2: active with 2 out of 2 mirrors
[   11.653328] md2: detected capacity change from 0 to 17179803648
[   11.656723] md: md3 stopped.
[   11.657316] md: bind<sda5>
[   11.657462] md: bind<sdb5>
[   11.657736] md/raid1:md3: active with 2 out of 2 mirrors
[   11.657753] md3: detected capacity change from 0 to 844374736896
[   11.797013] usb 1-1: New USB device found, idVendor=8087, idProduct=0024
[   11.797016] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   11.797416] hub 1-1:1.0: USB hub found
[   11.797593] hub 1-1:1.0: 6 ports detected
[   11.977886] BTRFS info (device sda4): enabling auto defrag
[   11.977889] BTRFS info (device sda4): disk space caching is enabled
[   12.002584] random: nonblocking pool is initialized
[   12.076342] usb 1-1.6: new full-speed USB device number 3 using ehci-pci
[   12.195822] usb 1-1.6: New USB device found, idVendor=0403, idProduct=fc0d
[   12.195825] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   12.195827] usb 1-1.6: Product: Crystalfontz CFA635-USB LCD
[   12.195829] usb 1-1.6: Manufacturer: Crystalfontz
[   12.195830] usb 1-1.6: SerialNumber: CF004445
[   12.199835] ftdi_sio 1-1.6:1.0: FTDI USB Serial Device converter detected
[   12.199855] usb 1-1.6: Detected FT232RL
[   12.200420] usb 1-1.6: FTDI USB Serial Device converter now attached to ttyUSB0
[   15.473634] udevd[322]: starting version 3.1.2
[   16.242876] Monitor-Mwait will be used to enter C-1 state
[   16.242905] Monitor-Mwait will be used to enter C-2 state
[   16.242927] Monitor-Mwait will be used to enter C-3 state
[   16.244799] Warning: Processor Platform Limit not supported.
[   16.263178] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[   16.263183] ACPI: Power Button [PWRB]
[   16.263226] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[   16.263228] ACPI: Power Button [PWRF]
[   16.286497] EDAC MC: Ver: 3.0.0
[   16.300701] Linux agpgart interface v0.103
[   16.303254] e1000e 0000:00:19.0 enp0s25: renamed from eth0
[   16.320396] xen: registering gsi 18 triggering 0 polarity 1
[   16.320402] Already setup the GSI :18
[   16.320410] ACPI Warning: SystemIO range 0x000000000000F040-0x000000000000F05F conflicts with OpRegion 0x000000000000F040-0x000000000000F04F (\_SB_.PCI0.SBUS.SMBI) (20150410/utaddress-254)
[   16.320416] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   16.329659] thermal LNXTHERM:00: registered as thermal_zone0
[   16.329662] ACPI: Thermal Zone [TZ00] (28 C)
[   16.329962] thermal LNXTHERM:01: registered as thermal_zone1
[   16.329964] ACPI: Thermal Zone [TZ01] (30 C)
[   16.376579] e1000e 0000:07:00.0 enp7s0: renamed from eth1
[   16.396998] EDAC MC0: Giving out device to module ie31200_edac controller IE31200: DEV 0000:00:00.0 (POLLED)
[   16.397477] [drm] Initialized drm 1.1.0 20060810
[   16.474564] 8139too: 8139too Fast Ethernet driver 0.9.28
[   17.064042] xen: registering gsi 16 triggering 0 polarity 1
[   17.064048] Already setup the GSI :16
[   17.064949] [drm] Memory usable by graphics device = 2048M
[   17.064951] [drm] Replacing VGA console driver
[   17.065495] Console: switching to colour dummy device 128x54
[   17.097700] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[   17.097702] [drm] Driver supports precise vblank timestamp query.
[   17.097854] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   17.101748] ACPI: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[   17.102011] acpi device:49: registered as cooling_device13
[   17.102116] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input3
[   17.102153] [drm] Initialized i915 1.6.0 20150327 for 0000:00:02.0 on minor 0
[   17.126051] fbcon: inteldrmfb (fb0) is primary device
[   17.172350] Console: switching to colour frame buffer device 128x48
[   17.174173] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[   17.174175] i915 0000:00:02.0: registered panic notifier
[   20.318348] device-mapper: thin: Data device (dm-25) discard unsupported: Disabling discard passdown.
[   21.699375] Adding 16777148k swap on /dev/md2.  Priority:-1 extents:1 across:16777148k FS
[   21.758189] EXT4-fs (dm-1): mounted filesystem without journal. Opts: (null)
[   24.396665] Ebtables v2.0 registered
[   24.592180] NET: Registered protocol family 10
[   24.604096] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   24.695803] ip_tables: (C) 2000-2006 Netfilter Core Team
[   24.784627] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[   24.792045] Bridge firewalling registered
[   25.187015] IPv6: ADDRCONF(NETDEV_UP): enp0s25: link is not ready
[   25.673933] IPv6: ADDRCONF(NETDEV_UP): enp7s0: link is not ready
[   26.129998] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[   26.141869] bond0: Setting MII monitoring interval to 100
[   26.282823] e1000e: enp0s25 NIC Link is Down
[   26.370699] e1000e: enp7s0 NIC Link is Down
[   26.377935] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
[   26.383409] bond0: Adding slave enp0s25
[   26.627066] IPv6: ADDRCONF(NETDEV_UP): enp0s25: link is not ready
[   26.627130] bond0: Enslaving enp0s25 as an active interface with a down link
[   26.632410] bond0: Adding slave enp7s0
[   26.714000] IPv6: ADDRCONF(NETDEV_UP): enp7s0: link is not ready
[   26.714057] bond0: Enslaving enp7s0 as an active interface with a down link
[   27.211121] device bond0 entered promiscuous mode
[   27.218759] device xenbr0 entered promiscuous mode
[   27.221107] IPv6: ADDRCONF(NETDEV_UP): xenbr0: link is not ready
[   29.117955] e1000e: enp7s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   29.157892] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   29.157904] NFSD: starting 90-second grace period (net ffffffff818b2140)
[   29.176893] bond0: link status definitely up for interface enp7s0, 1000 Mbps full duplex
[   29.176897] bond0: making interface enp7s0 the new active one
[   29.176899] device enp7s0 entered promiscuous mode
[   29.177230] bond0: first active interface up!
[   29.177239] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[   29.177265] xenbr0: port 1(bond0) entered forwarding state
[   29.177271] xenbr0: port 1(bond0) entered forwarding state
[   29.177384] IPv6: ADDRCONF(NETDEV_CHANGE): xenbr0: link becomes ready
[   29.522944] e1000e: enp0s25 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   29.576944] bond0: link status definitely up for interface enp0s25, 1000 Mbps full duplex
[   31.946335] xen_acpi_processor: Uploading Xen processor PM info
[   34.446872] vif vif-1-0 pfsense.0: renamed from vif1.0
[   34.531662] device pfsense.0 entered promiscuous mode
[   34.534200] IPv6: ADDRCONF(NETDEV_UP): pfsense.0: link is not ready
[   35.020524] xen_pciback: vpci: 0000:04:00.0: assign to virtual slot 0
[   35.048250] pciback 0000:04:00.0: registering for 1
[   35.052729] xen_pciback: vpci: 0000:0a:08.0: assign to virtual slot 1
[   35.054293] pciback 0000:0a:08.0: registering for 1
[   35.054987] xen_pciback: vpci: 0000:0a:0b.0: assign to virtual slot 2
[   35.056013] pciback 0000:0a:0b.0: registering for 1
[   36.378547] xenbr0: port 2(pfsense.0) entered disabled state
[   36.378820] device pfsense.0 left promiscuous mode
[   36.378824] xenbr0: port 2(pfsense.0) entered disabled state
[   44.217632] xenbr0: port 1(bond0) entered forwarding state

[-- Attachment #3: xl.dmesg --]
[-- Type: text/plain, Size: 35396 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@mydomain.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Mon Nov 16 00:28:43 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot hvm_debug=0xc3f dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.441382] Detected 2394.640 MHz processor.
(XEN) [    1.448760] Initing memory sharing.
(XEN) [    1.453220] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.460065] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.470189] Intel machine check reporting enabled
(XEN) [    1.475730] alt table ffff82d0802cfa90 -> ffff82d0802d0bd0
(XEN) [    1.482169] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.490663] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.496730] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.503207] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.509443] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.515753] Intel VT-d Snoop Control not enabled.
(XEN) [    1.521288] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.527439] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.533215] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.538926] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.553273] I/O virtualisation enabled
(XEN) [    1.557851]  - Dom0 mode: Relaxed
(XEN) [    1.562156] Interrupt remapping enabled
(XEN) [    1.566829] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.573274] ENABLING IO-APIC IRQs
(XEN) [    1.577503]  -> Using old ACK method
(XEN) [    1.582231] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.791374] TSC deadline timer enabled
(XEN) [    2.136059] Platform timer is 14.318MHz HPET
(XEN) [    2.140937] 'A' pressed -> using alternative key handling
(XEN) [    2.141635] Allocated console ring of 64 KiB.
(XEN) [    2.142750] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.143804] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.144326] mwait-idle: v0.4 model 0x2a
(XEN) [    2.144846] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.145411] VMX: Supported advanced features:
(XEN) [    2.145930]  - APIC MMIO access virtualisation
(XEN) [    2.146451]  - APIC TPR shadow
(XEN) [    2.146995]  - Extended Page Tables (EPT)
(XEN) [    2.147512]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.148037]  - Virtual NMI
(XEN) [    2.148578]  - MSR direct-access bitmap
(XEN) [    2.149094]  - Unrestricted Guest
(XEN) [    2.149601] HVM: ASIDs enabled.
(XEN) [    2.150142] HVM: VMX enabled
(XEN) [    2.150643] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.151160] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.152313] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.153348] Brought up 8 CPUs
(XEN) [    2.154167] microcode: CPU6 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.160830] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.161348] microcode: CPU5 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.182523] ACPI sleep modes: S3
(XEN) [    2.183029] mcheck_poll: Machine check polling timer started.
(XEN) [    2.183558] Dom0 has maximum 792 PIRQs
(XEN) [    2.184145] *** LOADING DOMAIN 0 ***
(XEN) [    2.366018]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    2.366569]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    2.367802] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    2.368310]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    2.369366]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    2.369890] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    2.370399]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    2.370919]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    2.371440]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    2.371960]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    2.372481]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    2.373003]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    2.373523]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    2.374081]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    2.375422] Dom0 has maximum 8 VCPUs
(XEN) [    5.541030] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    5.549328] Std. Loglevel: All
(XEN) [    5.549870] Guest Loglevel: All
(XEN) [    5.550375] Xen is relinquishing VGA console.
(XEN) [    5.551450] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) [    5.551490] Freed 316kB init memory.
(XEN) [    5.551497] '0' pressed -> dumping Dom0's registers
(XEN) [    5.551500] *** Dumping Dom0 vcpu#0 state: ***
(XEN) [    5.551503] RIP:    e033:[<ffffffff818e31f0>]
(XEN) [    5.551505] RFLAGS: 0000000000000200   EM: 1   CONTEXT: pv guest (d0v0)
(XEN) [    5.551508] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551510] rdx: 0000000000000000   rsi: ffffffff82400000   rdi: 0000000000000000
(XEN) [    5.551512] rbp: 0000000000000000   rsp: ffffffff82419000   r8:  0000000000000000
(XEN) [    5.551514] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551515] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551517] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551519] cr3: 0000000802401000   cr2: 0000000000000000
(XEN) [    5.551521] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) [    5.551523] Guest stack trace from rsp=ffffffff82419000:
(XEN) [    5.551524]   Stack empty.
(XEN) [    5.551532] *** Dumping Dom0 vcpu#1 state: ***
(XEN) [    5.551536] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551538] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v1)
(XEN) [    5.551541] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551544] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551546] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551548] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551550] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551553] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551555] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551558] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551589] *** Dumping Dom0 vcpu#2 state: ***
(XEN) [    5.551592] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551594] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v2)
(XEN) [    5.551597] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551599] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551601] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551603] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551604] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551606] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551608] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551609] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551616] *** Dumping Dom0 vcpu#3 state: ***
(XEN) [    5.551618] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551620] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v3)
(XEN) [    5.551623] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551624] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551626] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551628] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551629] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551631] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551632] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551634] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551663] *** Dumping Dom0 vcpu#4 state: ***
(XEN) [    5.551666] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551667] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v4)
(XEN) [    5.551671] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551672] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551674] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551676] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551678] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551679] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551681] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551683] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551689] *** Dumping Dom0 vcpu#5 state: ***
(XEN) [    5.551691] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551693] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v5)
(XEN) [    5.551695] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551697] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551699] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551700] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551702] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551704] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551705] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551707] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551735] *** Dumping Dom0 vcpu#6 state: ***
(XEN) [    5.551739] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551741] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v6)
(XEN) [    5.551744] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551746] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551747] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551749] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551751] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551752] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551754] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551756] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.551762] *** Dumping Dom0 vcpu#7 state: ***
(XEN) [    5.551764] RIP:    0000:[<0000000000000000>]
(XEN) [    5.551765] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v7)
(XEN) [    5.551768] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    5.551770] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    5.551771] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    5.551773] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    5.551775] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    5.551777] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    5.551778] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    5.551780] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    5.807770] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.807775] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) [    5.807779] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) [    5.807782] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.807785] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) [    5.807789] traps.c:2616:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.807792] traps.c:2616:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.893982] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.893989] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080cab3000 to 0xffffffff81646d10.
(XEN) [    5.893995] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080cab3080 to 0xffffffff81648e70.
(XEN) [    5.894000] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.894005] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83080cab7fc0 to 0x0000000000000000.
(XEN) [    5.894010] traps.c:2616:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.894015] traps.c:2616:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.894314] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.894319] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff83080caa3000 to 0xffffffff81646d10.
(XEN) [    5.894323] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff83080caa3080 to 0xffffffff81648e70.
(XEN) [    5.894327] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.894330] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff83080caa7fc0 to 0x0000000000000000.
(XEN) [    5.894333] traps.c:2616:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.894336] traps.c:2616:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.894614] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.894618] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca93000 to 0xffffffff81646d10.
(XEN) [    5.894621] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca93080 to 0xffffffff81648e70.
(XEN) [    5.894624] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.894627] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca97fc0 to 0x0000000000000000.
(XEN) [    5.894631] traps.c:2616:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.894634] traps.c:2616:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.894876] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.894881] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca83000 to 0xffffffff81646d10.
(XEN) [    5.894885] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca83080 to 0xffffffff81648e70.
(XEN) [    5.894888] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.894891] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca87fc0 to 0x0000000000000000.
(XEN) [    5.894895] traps.c:2616:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.894898] traps.c:2616:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.895132] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.895136] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063f3000 to 0xffffffff81646d10.
(XEN) [    5.895139] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063f3080 to 0xffffffff81648e70.
(XEN) [    5.895143] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.895146] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308063f7fc0 to 0x0000000000000000.
(XEN) [    5.895149] traps.c:2616:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.895152] traps.c:2616:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.895366] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.895371] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063eb000 to 0xffffffff81646d10.
(XEN) [    5.895374] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063eb080 to 0xffffffff81648e70.
(XEN) [    5.895378] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.895381] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff8308063effc0 to 0x0000000000000000.
(XEN) [    5.895384] traps.c:2616:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.895387] traps.c:2616:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.895587] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.895591] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063db000 to 0xffffffff81646d10.
(XEN) [    5.895595] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063db080 to 0xffffffff81648e70.
(XEN) [    5.895598] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.895601] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000175 from 0xffff8308063dffc0 to 0x0000000000000000.
(XEN) [    5.895604] traps.c:2616:d0v7 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    5.895607] traps.c:2616:d0v7 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    7.037608] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    7.037611] PCI add device 0000:00:00.0
(XEN) [    7.037815] PCI add device 0000:00:01.0
(XEN) [    7.037998] PCI add device 0000:00:01.1
(XEN) [    7.038256] PCI add device 0000:00:02.0
(XEN) [    7.038443] PCI add device 0000:00:06.0
(XEN) [    7.038728] PCI add device 0000:00:16.0
(XEN) [    7.039010] PCI add device 0000:00:19.0
(XEN) [    7.039304] PCI add device 0000:00:1a.0
(XEN) [    7.039605] PCI add device 0000:00:1b.0
(XEN) [    7.039862] PCI add device 0000:00:1c.0
(XEN) [    7.040124] PCI add device 0000:00:1c.4
(XEN) [    7.040379] PCI add device 0000:00:1c.6
(XEN) [    7.040677] PCI add device 0000:00:1d.0
(XEN) [    7.040888] PCI add device 0000:00:1e.0
(XEN) [    7.041140] PCI add device 0000:00:1f.0
(XEN) [    7.041414] PCI add device 0000:00:1f.2
(XEN) [    7.041597] PCI add device 0000:00:1f.3
(XEN) [    7.041870] PCI add device 0000:01:00.0
(XEN) [    7.058091] PCI add device 0000:02:00.0
(XEN) [    7.078114] PCI add device 0000:03:00.0
(XEN) [    7.078561] PCI add device 0000:04:00.0
(XEN) [    7.098298] PCI add device 0000:05:00.0
(XEN) [    7.098723] PCI add device 0000:06:00.0
(XEN) [    7.099456] PCI add device 0000:07:00.0
(XEN) [    7.118414] PCI add device 0000:08:00.0
(XEN) [    7.118748] PCI add device 0000:09:00.0
(XEN) [    7.119220] PCI add device 0000:0a:08.0
(XEN) [    7.119505] PCI add device 0000:0a:09.0
(XEN) [    7.119790] PCI add device 0000:0a:0a.0
(XEN) [    7.120106] PCI add device 0000:0a:0b.0
(XEN) [    7.181681] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181701] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181744] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181762] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181806] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181824] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181831] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(XEN) [    7.181849] traps.c:3188: GPF (0000): ffff82d08019920c -> ffff82d080239753
(d1) [   35.051817] HVM Loader
(d1) [   35.052114] Detected Xen v4.5.2
(d1) [   35.052502] Xenbus rings @0xfeffc000, event channel 1
(d1) [   35.052958] System requested SeaBIOS
(d1) [   35.053163] CPU speed is 2395 MHz
(d1) [   35.054070] Relocating guest memory for lowmem MMIO space disabled
(XEN) [   35.054615] irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(d1) [   35.054943] PCI-ISA link 0 routed to IRQ5
(XEN) [   35.055101] irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(d1) [   35.055444] PCI-ISA link 1 routed to IRQ10
(XEN) [   35.055550] irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(d1) [   35.055887] PCI-ISA link 2 routed to IRQ11
(XEN) [   35.056081] irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(d1) [   35.056482] PCI-ISA link 3 routed to IRQ5
(d1) [   35.075043] pci dev 01:3 INTA->IRQ10
(d1) [   35.079545] pci dev 02:0 INTA->IRQ11
(d1) [   35.086386] pci dev 04:0 INTA->IRQ5
(d1) [   35.090244] pci dev 05:0 INTA->IRQ10
(d1) [   35.093997] pci dev 06:0 INTA->IRQ11
(d1) [   35.118499] No RAM in high memory; setting high_mem resource base to 100000000
(d1) [   35.119051] pci dev 03:0 bar 10 size 002000000: 0f0000008
(d1) [   35.121185] pci dev 02:0 bar 14 size 001000000: 0f2000008
(d1) [   35.122902] pci dev 04:0 bar 10 size 000020000: 0f3000004
(XEN) [   35.123095] memory_map:add: dom1 gfn=f3000 mfn=f7c00 nr=20
(d1) [   35.126537] pci dev 03:0 bar 30 size 000010000: 0f3020000
(d1) [   35.127148] pci dev 04:0 bar 30 size 000010000: 0f3030000
(d1) [   35.127823] pci dev 05:0 bar 30 size 000010000: 0f3040000
(d1) [   35.128440] pci dev 06:0 bar 10 size 000010000: 0f3050000
(XEN) [   35.128622] memory_map:add: dom1 gfn=f3050 mfn=f7800 nr=10
(d1) [   35.133504] pci dev 03:0 bar 14 size 000001000: 0f3060000
(XEN) [   35.133835] memory_map:add: dom1 gfn=f3061 mfn=f7842 nr=1
(d1) [   35.137348] pci dev 05:0 bar 14 size 000001000: 0f3061000
(d1) [   35.137981] pci dev 02:0 bar 10 size 000000100: 00000c001
(d1) [   35.140362] pci dev 05:0 bar 10 size 000000100: 00000c101
(XEN) [   35.140756] ioport_map:add: dom1 gport=c100 mport=c200 nr=100
(d1) [   35.144081] pci dev 01:1 bar 20 size 000000010: 00000c201
(d1) [   35.146209] Multiprocessor initialisation:
(d1) [   35.170094]  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   35.194130]  - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   35.194228] Testing HVM environment:
(d1) [   35.199409]  - REP INSB across page boundaries ... passed
(d1) [   35.201608]  - GS base MSRs and SWAPGS ... passed
(d1) [   35.201628] Passed 2 of 2 tests
(d1) [   35.201654] Writing SMBIOS tables ...
(d1) [   35.202377] Loading SeaBIOS ...
(d1) [   35.202416] Creating MP tables ...
(d1) [   35.202506] Loading ACPI ...
(d1) [   35.203919] vm86 TSS at fc00a380
(d1) [   35.204227] BIOS map:
(d1) [   35.204254]  10000-100d3: Scratch space
(d1) [   35.204280]  e0000-fffff: Main BIOS
(d1) [   35.204293] E820 table:
(d1) [   35.204340]  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d1) [   35.204383]  HOLE: 00000000:000a0000 - 00000000:000e0000
(d1) [   35.204435]  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d1) [   35.204483]  [02]: 00000000:00100000 - 00000000:27800000: RAM
(d1) [   35.204525]  HOLE: 00000000:27800000 - 00000000:fc000000
(d1) [   35.204577]  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d1) [   35.204790] Invoking SeaBIOS ...
(XEN) [   36.063957] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) [   36.063959] ************* VMCS Area **************
(XEN) [   36.063960] *** Guest State ***
(XEN) [   36.063961] CR0: actual=0x0000000000000039, shadow=0x0000000000000011, gh_mask=ffffffffffffffff
(XEN) [   36.063963] CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff
(XEN) [   36.063964] CR3: actual=0x0000000000800000, target_count=0
(XEN) [   36.063966]      target0=0000000000000000, target1=0000000000000000
(XEN) [   36.063967]      target2=0000000000000000, target3=0000000000000000
(XEN) [   36.063968] RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 0x0000000100000000 (0x0000000100000000)
(XEN) [   36.063970] RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 0x0000000000000400
(XEN) [   36.063971] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) [   36.063973] CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063975] DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063976] SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063978] ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063979] FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063980] GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.063982] GDTR:                           limit=0x00000037, base=0x00000000000f6d80
(XEN) [   36.063983] LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, base=0x0000000000000000
(XEN) [   36.063985] IDTR:                           limit=0x00000000, base=0x00000000000f6dbe
(XEN) [   36.063986] TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, base=0x0000000000000000
(XEN) [   36.063987] Guest PAT = 0x0007040600070406
(XEN) [   36.063989] TSC Offset = ffffffe3da7c0a52
(XEN) [   36.063990] DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) [   36.063991] Interruptibility=0000 ActivityState=0000
(XEN) [   36.063992] *** Host State ***
(XEN) [   36.063993] RSP = 0xffff83080ca97f90  RIP = 0xffff82d0801f4820
(XEN) [   36.063994] CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) [   36.063996] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83080ca9ec80
(XEN) [   36.063997] GDTBase=ffff83080ca9a000 IDTBase=ffff83080ca9d000
(XEN) [   36.063999] CR0=000000008005003b CR3=000000070bf71000 CR4=00000000000426f0
(XEN) [   36.064000] Sysenter RSP=ffff83080ca97fc0 CS:RIP=e008:ffff82d080235550
(XEN) [   36.064002] Host PAT = 0x0000050100070406
(XEN) [   36.064003] *** Control State ***
(XEN) [   36.064004] PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) [   36.064005] EntryControls=000051ff ExitControls=000fefff
(XEN) [   36.064006] ExceptionBitmap=000600c2
(XEN) [   36.064007] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.064008] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.064010]         reason=80000021 qualification=00000000
(XEN) [   36.064011] IDTVectoring: info=00000000 errcode=00000000
(XEN) [   36.064012] TPR Threshold = 0x00
(XEN) [   36.064013] EPT pointer = 0x000000070bf3f01e
(XEN) [   36.064014] Virtual processor ID = 0x0011
(XEN) [   36.064015] **************************************
(XEN) [   36.064016] domain_crash called from vmx.c:2505
(XEN) [   36.064017] Domain 1 (vcpu#0) crashed on cpu#3:
(XEN) [   36.064019] ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) [   36.064021] CPU:    3
(XEN) [   36.064022] RIP:    0008:[<0000000100000000>]
(XEN) [   36.064023] RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) [   36.064025] rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) [   36.064027] rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) [   36.064028] rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) [   36.064029] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [   36.064030] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [   36.064032] r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) [   36.064033] cr3: 0000000000800000   cr2: 0000000000000000
(XEN) [   36.064034] ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 2035 bytes --]

On 16/11/15 00:39, Atom2 wrote:
> Am 15.11.15 um 16:12 schrieb Andrew Cooper:
> [big snip]
>> Great - so confirms the issue as a SeaBIOS interaction issue, rather
>> than a hypervisor regression.
>>
>> As I said before, I am still certain that a guest should not be able
>> to get itself into the crashing state (short of a hardware errata),
>> so I still suspect that there is a latent hypervisor emulation bug
>> which has been tickled by the SeaBIOS update.
>>
>> Would you please mind running the bad HVMLoader on Xen 4.5.2 with
>> hvm_debug=0xc3f ? I am still hoping that that will shed some light on
>> SeaBIOS actions just leading up to the crash.
> Hi Andrew,
> Please see the attached two files. One is the dmesg from booting the
> system. This looks pretty normal in my view. The other is the output
> of "xl dmesg" which is most likely what you were after. It's probably
> worth noting that the "traps.c" output between lines 259 and 314 and
> again between lines 346 and 353 seem to be xen-4.5.2 specific and
> don't show up under xen-4.5.1, but that may not be of any relevance
> for the SeaBIOS issue we are experiencing. Though I'd still be
> interested to know whether that's anything for me to worry about ...

Sorry, but you need to be using a debug build of Xen.  The internals of
hvm_debug are not compiled in a regular build.

I am also only interested in `xl dmesg`.  There will be lots of log
lines prefixed with [HVM:$DOMID.$VCPUID].

>>
>> Are you able to experiment with newer versions of Xen?  It would be
>> interesting to see whether the issue is still present in Xen 4.6
> Currently xen-4.6 is not stable in gentoo and I try to stick to stable
> packages as much as possible. But in case the above does not help you
> any further, I am happy to give this a try as well. Would this just be
> a straightforward test to see whether it works at all or would you
> require debug symbols as well?

That's ok.  It is unlikely that this issue has been fixed since, so I
suspect that it is still present.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 3003 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Doug Goldstein]

[-- Attachment #1.1: Type: text/plain, Size: 7649 bytes --]

On 11/14/15 6:14 PM, Atom2 wrote:
> Am 14.11.15 um 21:32 schrieb Andrew Cooper:
>> On 14/11/2015 00:16, Atom2 wrote:
>>> Am 13.11.15 um 11:09 schrieb Andrew Cooper:
>>>> On 13/11/15 07:25, Jan Beulich wrote:
>>>>>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>>>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>>>>> Hi Andrew,
>>>>>>>> thanks for your reply. Answers are inline further down.
>>>>>>>>
>>>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>>>>>> of the
>>>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>>>>
>>>>>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>>>>> [...]
>>>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>>>>>> state (0).
>>>>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>>>>> (XEN) *** Guest State ***
>>>>>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>>>>> odd? That will be the source of the failure.
>>>>>>>>>
>>>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>>>>>> should not be possible to set any of them.
>>>>>>>>>
>>>>>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>>>>>> get it self into this condition.
>>>>>>>>>
>>>>>>>>> Are you able to rerun with a debug build of the hypervisor?
> [big snip]
>>>>>>>> Now _without_ the debug USE flag, but with debug information in
>>>>>>>>         the binary (I used splitdebug), all is back to where the problem
>>>>>>>>         started off (i.e. the system boots without issues until such
>>>>>>>>         time it starts a HVM domU which then crashes; PV domUs are
>>>>>>>>         working). I have attached the latest "xl dmesg" output with the
>>>>>>>>         timing information included.
>>>>>>>>  
>>>
>>> I hope any of this makes sense to you.
>>>
>>> Again many thanks and best regards
>>>
>>
>> Right - it would appear that the USE flag is definitely not what you
>> wanted, and causes bad compilation for Xen.  The do_IRQ disassembly
>> you sent is a the result of disassembling a whole block of zeroes. 
>> Sorry for leading you on a goose chase - the double faults will be the
>> product of bad compilation, rather than anything to do with your
>> specific problem.
> Hi Andrew,
> there's absolutely no need to appologize as it is me who asked for help
> and you who generously stepped in and provided it. I really do
> appreciate your help and it is for me, as the one seeking help, to
> provide all the information you deem necessary and you ask for.
>> However, the final log you sent (dmesg) is using a debug Xen, which is
>> what I was attempting to get you to do originally.
> Next time I know better how to arrive at a debug XEN. It's all about
> learning.
>> We still observe that the VM ends up in 32bit non-paged mode but with
>> an RIP with bit 32 set, which is an invalid state to be in.  However,
>> there was nothing particularly interesting in the extra log information.
>>
>> Please can you rerun with "hvm_debug=0xc3f", which will cause far more
>> logging to occur to the console while the HVM guest is running.  That
>> might show some hints.
> I haven't done that yet - but please see my next paragraph. If you are
> still interested in this, for whatever reason, I am clearly more than
> happy to rerun with your suggested option and provide that information
> as well.
>> Also, the fact that this occurs just after starting SeaBIOS is
>> interesting.  As you have switched versions of Xen, you have also
>> switched hvmloader, which contains the SeaBIOS binary embedded in it. 
>> Would you be able to compile both 4.5.1 and 4.5.2 and switch the
>> hvmloader binaries in use.  It would be very interesting to see
>> whether the failure is caused by the hvmloader binary or the
>> hypervisor.  (With `xl`, you can use
>> firmware_override="/full/path/to/firmware" to override the default
>> hvmloader).
> Your analysis was absolutely spot on. After re-thinking this for a
> moment, I thought going down that route first would make a lot of sense
> as PV guests still do work and one of the differences to HVM domUs is
> that the former do _not_ require SeaBIOS. Looking at my log files of
> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
> the relevant timeframe which obviously had not made it to the hvmloader
> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
> 
> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
> to strongly indicate that there indeed might be an issue with SeaBIOS as
> this probably was the only variable that had changed from the original
> install of xen-4.5.1.
> 
> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
> xen-4.5.1. Voila, the system was again up and running. While still
> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
> probably guessed it ... the problem was gone: The system boots up with
> no issues and everything is fine again.
> 
> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
> preventing HVM doamins from successfully starting up. I don't know what
> this is triggered from, if this is specific to my hardware or whether
> something else in my environment is to blame.
> 
> In any case, I am again more than happy to provide data / run a few
> tests should you wish to get to the grounds of this.
> 
> I do owe you a beer (or any other drink) should you ever be at my
> location (i.e. Vienna, Austria).
> 
> Many thanks again for your analysis and your first class support. Xen
> and their people absolutely rock!
> 
> Atom2

I'm a little late to the thread but can you send me (you can do it
off-list if you'd like) the USE flags you used for xen, xen-tools and
seabios? Also emerge --info. You can kill two birds with one stone by
using emerge --info xen.

I'm not too familiar with the xen ebuilds but I was pretty sure that
xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
and builds it so that it remains consistent. But obviously your
experience shows otherwise.

I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
your info would be helpful.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 9115 bytes --]

Am 15.11.15 um 21:12 schrieb Doug Goldstein:
> On 11/14/15 6:14 PM, Atom2 wrote:
>> Am 14.11.15 um 21:32 schrieb Andrew Cooper:
>>> On 14/11/2015 00:16, Atom2 wrote:
>>>> Am 13.11.15 um 11:09 schrieb Andrew Cooper:
>>>>> On 13/11/15 07:25, Jan Beulich wrote:
>>>>>>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>>>>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>>>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>>>>>> Hi Andrew,
>>>>>>>>> thanks for your reply. Answers are inline further down.
>>>>>>>>>
>>>>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>>>>>>> After the upgrade HVM domUs appear to no longer work - regardless
>>>>>>>>>>>> of the
>>>>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>>>>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>>>>>
>>>>>>>>>>>> xl dmesg shows the following information after the first crashed HVM
>>>>>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>>>>>> [...]
>>>>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>>>>>>>>>>> state (0).
>>>>>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>>>>>> (XEN) *** Guest State ***
>>>>>>>>>>>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>>>>>> Other than RIP looking odd for a guest still in non-paged protected
>>>>>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>>>>>> odd? That will be the source of the failure.
>>>>>>>>>>
>>>>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, and it
>>>>>>>>>> should not be possible to set any of them.
>>>>>>>>>>
>>>>>>>>>> I suspect that the guest has exited for emulation, and there has been a
>>>>>>>>>> bad update to %rip.  The alternative (which I hope is not the case) is
>>>>>>>>>> that there is a hardware errata which allows the guest to accidentally
>>>>>>>>>> get it self into this condition.
>>>>>>>>>>
>>>>>>>>>> Are you able to rerun with a debug build of the hypervisor?
>> [big snip]
>>>>>>>>> Now _without_ the debug USE flag, but with debug information in
>>>>>>>>>          the binary (I used splitdebug), all is back to where the problem
>>>>>>>>>          started off (i.e. the system boots without issues until such
>>>>>>>>>          time it starts a HVM domU which then crashes; PV domUs are
>>>>>>>>>          working). I have attached the latest "xl dmesg" output with the
>>>>>>>>>          timing information included.
>>>>>>>>>   
>>>> I hope any of this makes sense to you.
>>>>
>>>> Again many thanks and best regards
>>>>
>>> Right - it would appear that the USE flag is definitely not what you
>>> wanted, and causes bad compilation for Xen.  The do_IRQ disassembly
>>> you sent is a the result of disassembling a whole block of zeroes.
>>> Sorry for leading you on a goose chase - the double faults will be the
>>> product of bad compilation, rather than anything to do with your
>>> specific problem.
>> Hi Andrew,
>> there's absolutely no need to appologize as it is me who asked for help
>> and you who generously stepped in and provided it. I really do
>> appreciate your help and it is for me, as the one seeking help, to
>> provide all the information you deem necessary and you ask for.
>>> However, the final log you sent (dmesg) is using a debug Xen, which is
>>> what I was attempting to get you to do originally.
>> Next time I know better how to arrive at a debug XEN. It's all about
>> learning.
>>> We still observe that the VM ends up in 32bit non-paged mode but with
>>> an RIP with bit 32 set, which is an invalid state to be in.  However,
>>> there was nothing particularly interesting in the extra log information.
>>>
>>> Please can you rerun with "hvm_debug=0xc3f", which will cause far more
>>> logging to occur to the console while the HVM guest is running.  That
>>> might show some hints.
>> I haven't done that yet - but please see my next paragraph. If you are
>> still interested in this, for whatever reason, I am clearly more than
>> happy to rerun with your suggested option and provide that information
>> as well.
>>> Also, the fact that this occurs just after starting SeaBIOS is
>>> interesting.  As you have switched versions of Xen, you have also
>>> switched hvmloader, which contains the SeaBIOS binary embedded in it.
>>> Would you be able to compile both 4.5.1 and 4.5.2 and switch the
>>> hvmloader binaries in use.  It would be very interesting to see
>>> whether the failure is caused by the hvmloader binary or the
>>> hypervisor.  (With `xl`, you can use
>>> firmware_override="/full/path/to/firmware" to override the default
>>> hvmloader).
>> Your analysis was absolutely spot on. After re-thinking this for a
>> moment, I thought going down that route first would make a lot of sense
>> as PV guests still do work and one of the differences to HVM domUs is
>> that the former do _not_ require SeaBIOS. Looking at my log files of
>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
>> the relevant timeframe which obviously had not made it to the hvmloader
>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
>>
>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
>> to strongly indicate that there indeed might be an issue with SeaBIOS as
>> this probably was the only variable that had changed from the original
>> install of xen-4.5.1.
>>
>> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
>> xen-4.5.1. Voila, the system was again up and running. While still
>> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
>> probably guessed it ... the problem was gone: The system boots up with
>> no issues and everything is fine again.
>>
>> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
>> preventing HVM doamins from successfully starting up. I don't know what
>> this is triggered from, if this is specific to my hardware or whether
>> something else in my environment is to blame.
>>
>> In any case, I am again more than happy to provide data / run a few
>> tests should you wish to get to the grounds of this.
>>
>> I do owe you a beer (or any other drink) should you ever be at my
>> location (i.e. Vienna, Austria).
>>
>> Many thanks again for your analysis and your first class support. Xen
>> and their people absolutely rock!
>>
>> Atom2
> I'm a little late to the thread but can you send me (you can do it
> off-list if you'd like) the USE flags you used for xen, xen-tools and
> seabios? Also emerge --info. You can kill two birds with one stone by
> using emerge --info xen.
Hi Doug,
here you go:
USE flags:
app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi 
-flask -xsm"
app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu 
screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml) 
-ovmf -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
sys-firmware/seabios-1.7.5::gentoo  USE="binary"
emerge --info: Please see the attached file
> I'm not too familiar with the xen ebuilds but I was pretty sure that
> xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
> and builds it so that it remains consistent. But obviously your
> experience shows otherwise.
You are right, it's xen-tools that builds hvmloader. If I remember 
correctly, the "system-seabios" USE flag (for xen-tools) specifies 
whether sys-firmware/seabios is used and the latter downloads SeaBIOS in 
it's binary form provided its "binary" USE flag is set. At least that's 
my understanding.
> I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
> your info would be helpful.
Great. Whatever makes gentoo and xen stronger will be awesome. What 
immediately springs to mind is to create a separate hvmloader package 
and slot that (that's just an idea and probably not fully thought 
through, but ss far as I understood Andrew, it would then be possible to 
specify the specific firmware version [i.e. hvmloader] to use on xl's 
command line by using firmware_override="full/path/to/firmware").

I also found out that an upgrade to sys-firmware/seabios obviously does 
not trigger an automatic re-emerge of xen-tools and thus hvmloader. 
Shouldn't this also happen automatically as xen-tools depends on seabios?

Thanks and best regards Atom2


P.S. If you prefer to take this off-list, just reply to my mail address.

[-- Attachment #2: info --]
[-- Type: text/plain, Size: 3773 bytes --]

Portage 2.2.20.1 (python 2.7.10-final-0, hardened/linux/amd64, gcc-4.9.3, glibc-2.21-r1, 4.1.7-hardened-r1 x86_64)
=================================================================
System uname: Linux-4.1.7-hardened-r1-x86_64-Intel-R-_Xeon-R-_CPU_E31260L_@_2.40GHz-with-gentoo-2.2
KiB Mem:     4032716 total,   3678784 free
KiB Swap:   16777148 total,  16777148 free
Timestamp of repository gentoo: Sun, 15 Nov 2015 00:45:01 +0000
sh bash 4.3_p39
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p39::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.10::gentoo, 3.4.3::gentoo
dev-util/cmake:           3.3.1-r1::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.17::gentoo
sys-apps/sandbox:         2.6-r1::gentoo
sys-devel/autoconf:       2.69::gentoo
sys-devel/automake:       1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            4.8.5::gentoo, 4.9.3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc:           2.21-r1::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage/
    priority: -1000

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build=y --buildpkg-exclude sys-kernel/hardened-sources"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gd.tuwien.ac.at/opsys/linux/gentoo/ ftp://gd.tuwien.ac.at/opsys/linux/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS=""
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--quiet --progress"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="acl aes amd64 avx bash-completion berkdb bzip2 cli cracklib crypt cxx gdbm hardened iconv justify lm_sensors mmx mmxext modules multilib ncurses nls nptl openmp pam pax_kernel pcre pie popcnt readline seccomp session sse sse2 sse3 sse4.1 sse4_1 ssl ssp ssse3 tcpd unicode urandom vim-syntax xattr xtpax zlib" ABI_X86="64" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4.1 ssse3" ELIBC="glibc" KERNEL="linux" LINGUAS="en" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" RUBY_TARGETS="ruby20" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Konrad Rzeszutek Wilk]

> >>Your analysis was absolutely spot on. After re-thinking this for a
> >>moment, I thought going down that route first would make a lot of sense
> >>as PV guests still do work and one of the differences to HVM domUs is
> >>that the former do _not_ require SeaBIOS. Looking at my log files of
> >>installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
> >>the relevant timeframe which obviously had not made it to the hvmloader
> >>of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
> >>
> >>So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
> >>1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
> >>to strongly indicate that there indeed might be an issue with SeaBIOS as
> >>this probably was the only variable that had changed from the original
> >>install of xen-4.5.1.

I recall seeing this way back in Fedora 20 days. I narrowed it down the 
SeaBIOS version that was a standalone package to not have CONFIG_XEN.

Having that fixed in the SeaBIOS package fixed it.

> >>
> >>My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
> >>xen-4.5.1. Voila, the system was again up and running. While still
> >>having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
> >>probably guessed it ... the problem was gone: The system boots up with
> >>no issues and everything is fine again.
> >>
> >>So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
> >>preventing HVM doamins from successfully starting up. I don't know what
> >>this is triggered from, if this is specific to my hardware or whether
> >>something else in my environment is to blame.
> >>
> >>In any case, I am again more than happy to provide data / run a few
> >>tests should you wish to get to the grounds of this.
> >>
> >>I do owe you a beer (or any other drink) should you ever be at my
> >>location (i.e. Vienna, Austria).
> >>
> >>Many thanks again for your analysis and your first class support. Xen
> >>and their people absolutely rock!
> >>
> >>Atom2
> >I'm a little late to the thread but can you send me (you can do it
> >off-list if you'd like) the USE flags you used for xen, xen-tools and
> >seabios? Also emerge --info. You can kill two birds with one stone by
> >using emerge --info xen.
> Hi Doug,
> here you go:
> USE flags:
> app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi -flask
> -xsm"
> app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu
> screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml) -ovmf
> -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
> sys-firmware/seabios-1.7.5::gentoo  USE="binary"
> emerge --info: Please see the attached file
> >I'm not too familiar with the xen ebuilds but I was pretty sure that
> >xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
> >and builds it so that it remains consistent. But obviously your
> >experience shows otherwise.
> You are right, it's xen-tools that builds hvmloader. If I remember
> correctly, the "system-seabios" USE flag (for xen-tools) specifies whether
> sys-firmware/seabios is used and the latter downloads SeaBIOS in it's binary
> form provided its "binary" USE flag is set. At least that's my
> understanding.
> >I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
> >your info would be helpful.
> Great. Whatever makes gentoo and xen stronger will be awesome. What
> immediately springs to mind is to create a separate hvmloader package and
> slot that (that's just an idea and probably not fully thought through, but
> ss far as I understood Andrew, it would then be possible to specify the
> specific firmware version [i.e. hvmloader] to use on xl's command line by
> using firmware_override="full/path/to/firmware").
> 
> I also found out that an upgrade to sys-firmware/seabios obviously does not
> trigger an automatic re-emerge of xen-tools and thus hvmloader. Shouldn't
> this also happen automatically as xen-tools depends on seabios?

No and yes. You can use 'seabios' also with a standalone QEMU (so not
hardware virtualization).

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>> moment, I thought going down that route first would make a lot of sense
>>>> as PV guests still do work and one of the differences to HVM domUs is
>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
>>>> the relevant timeframe which obviously had not made it to the hvmloader
>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
>>>>
>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
>>>> to strongly indicate that there indeed might be an issue with SeaBIOS as
>>>> this probably was the only variable that had changed from the original
>>>> install of xen-4.5.1.
> I recall seeing this way back in Fedora 20 days. I narrowed it down the
> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>
> Having that fixed in the SeaBIOS package fixed it.
Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
Konrad, you might have found an interesting point. I did have a look at 
the ebuild for the failing version and in there I found the following 
comment:
====== comment from ebuild =======
     # Upstream hasn't released a new binary.  We snipe ours from Fedora 
for now.
     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
====== end comment from ebuild =======
which might in fact underline that there might be an issue similar to 
what you described above.

What is also pretty interesting is the fact that the old (working) 
SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios 
is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS 
1.8.2 installed in the same location is only half as big: 131.072 bytes.

I checked at the download site and the 1.8.2 binary version is indeed 
not available from http://code.coreboot.org/p/seabios/downloads/. But 
both the binary versions for 1.7.5 and 1.8.0 are available and both are 
acutually 262.144 bytes in size, so I'd be very surprised if the 1.8.2 
version is really only half that size. By the way, the old working 
version (according to the ebuild) was directly downloaded from the above 
url and also shows an identical SHA1 digest to that version available 
for download there.

To me this looks as if something is really wrong here. If anybody of you 
has access to a 1.8.2 version, could you please confirm whether there's 
really that big a size difference between the 1.7.5 and the 1.8.2 
versions? Or is that difference probably attributable to the missing 
CONFIG_XEN option?

Andrew: I havent't gotten around to run the debug version of the 
hypervisor again, but if the current suspicion turns out to be true, 
there's probably not much value in that anyways. Would you agree?

Thanks Atom2
>>>> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
>>>> xen-4.5.1. Voila, the system was again up and running. While still
>>>> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
>>>> probably guessed it ... the problem was gone: The system boots up with
>>>> no issues and everything is fine again.
>>>>
>>>> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
>>>> preventing HVM doamins from successfully starting up. I don't know what
>>>> this is triggered from, if this is specific to my hardware or whether
>>>> something else in my environment is to blame.
>>>>
>>>> In any case, I am again more than happy to provide data / run a few
>>>> tests should you wish to get to the grounds of this.
>>>>
>>>> I do owe you a beer (or any other drink) should you ever be at my
>>>> location (i.e. Vienna, Austria).
>>>>
>>>> Many thanks again for your analysis and your first class support. Xen
>>>> and their people absolutely rock!
>>>>
>>>> Atom2
>>> I'm a little late to the thread but can you send me (you can do it
>>> off-list if you'd like) the USE flags you used for xen, xen-tools and
>>> seabios? Also emerge --info. You can kill two birds with one stone by
>>> using emerge --info xen.
>> Hi Doug,
>> here you go:
>> USE flags:
>> app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi -flask
>> -xsm"
>> app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu
>> screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml) -ovmf
>> -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
>> sys-firmware/seabios-1.7.5::gentoo  USE="binary"
>> emerge --info: Please see the attached file
>>> I'm not too familiar with the xen ebuilds but I was pretty sure that
>>> xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
>>> and builds it so that it remains consistent. But obviously your
>>> experience shows otherwise.
>> You are right, it's xen-tools that builds hvmloader. If I remember
>> correctly, the "system-seabios" USE flag (for xen-tools) specifies whether
>> sys-firmware/seabios is used and the latter downloads SeaBIOS in it's binary
>> form provided its "binary" USE flag is set. At least that's my
>> understanding.
>>> I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
>>> your info would be helpful.
>> Great. Whatever makes gentoo and xen stronger will be awesome. What
>> immediately springs to mind is to create a separate hvmloader package and
>> slot that (that's just an idea and probably not fully thought through, but
>> ss far as I understood Andrew, it would then be possible to specify the
>> specific firmware version [i.e. hvmloader] to use on xl's command line by
>> using firmware_override="full/path/to/firmware").
>>
>> I also found out that an upgrade to sys-firmware/seabios obviously does not
>> trigger an automatic re-emerge of xen-tools and thus hvmloader. Shouldn't
>> this also happen automatically as xen-tools depends on seabios?
> No and yes. You can use 'seabios' also with a standalone QEMU (so not
> hardware virtualization).
> en.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Konrad Rzeszutek Wilk]

On Mon, Nov 16, 2015 at 08:16:33PM +0100, Atom2 wrote:
> 
> 
> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
> >>>>Your analysis was absolutely spot on. After re-thinking this for a
> >>>>moment, I thought going down that route first would make a lot of sense
> >>>>as PV guests still do work and one of the differences to HVM domUs is
> >>>>that the former do _not_ require SeaBIOS. Looking at my log files of
> >>>>installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
> >>>>the relevant timeframe which obviously had not made it to the hvmloader
> >>>>of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
> >>>>
> >>>>So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
> >>>>1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
> >>>>to strongly indicate that there indeed might be an issue with SeaBIOS as
> >>>>this probably was the only variable that had changed from the original
> >>>>install of xen-4.5.1.
> >I recall seeing this way back in Fedora 20 days. I narrowed it down the
> >SeaBIOS version that was a standalone package to not have CONFIG_XEN.
> >
> >Having that fixed in the SeaBIOS package fixed it.
> Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
> Konrad, you might have found an interesting point. I did have a look at the
> ebuild for the failing version and in there I found the following comment:
> ====== comment from ebuild =======
>     # Upstream hasn't released a new binary.  We snipe ours from Fedora for
> now.
>     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
> ====== end comment from ebuild =======
> which might in fact underline that there might be an issue similar to what
> you described above.
> 
> What is also pretty interesting is the fact that the old (working) SeaBIOS
> version 1.7.5 installed as "bios.bin" under /usr/share/seabios is actually
> 262.144 bytes in size whereas the new (invalid) SeaBIOS 1.8.2 installed in
> the same location is only half as big: 131.072 bytes.
> 
> I checked at the download site and the 1.8.2 binary version is indeed not
> available from http://code.coreboot.org/p/seabios/downloads/. But both the
> binary versions for 1.7.5 and 1.8.0 are available and both are acutually
> 262.144 bytes in size, so I'd be very surprised if the 1.8.2 version is
> really only half that size. By the way, the old working version (according
> to the ebuild) was directly downloaded from the above url and also shows an
> identical SHA1 digest to that version available for download there.

<blinks>I thought Gentoo was all about rebuilding from source and not
taking binary blobs.
> 
> To me this looks as if something is really wrong here. If anybody of you has
> access to a 1.8.2 version, could you please confirm whether there's really
> that big a size difference between the 1.7.5 and the 1.8.2 versions? Or is
> that difference probably attributable to the missing CONFIG_XEN option?

It may be other options too - like CONFIG_XHCI, or a huge amount of other
ones.
> 
> Andrew: I havent't gotten around to run the debug version of the hypervisor
> again, but if the current suspicion turns out to be true, there's probably
> not much value in that anyways. Would you agree?

I am not Andrew and can't really speak for him, but I am going to take a 
leap here and say he will agree with you.

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Doug Goldstein]

[-- Attachment #1.1: Type: text/plain, Size: 4534 bytes --]

On 11/16/15 1:25 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 16, 2015 at 08:16:33PM +0100, Atom2 wrote:
>>
>>
>> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>>>> moment, I thought going down that route first would make a lot of sense
>>>>>> as PV guests still do work and one of the differences to HVM domUs is
>>>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
>>>>>> the relevant timeframe which obviously had not made it to the hvmloader
>>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
>>>>>>
>>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
>>>>>> to strongly indicate that there indeed might be an issue with SeaBIOS as
>>>>>> this probably was the only variable that had changed from the original
>>>>>> install of xen-4.5.1.
>>> I recall seeing this way back in Fedora 20 days. I narrowed it down the
>>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>>>
>>> Having that fixed in the SeaBIOS package fixed it.
>> Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
>> Konrad, you might have found an interesting point. I did have a look at the
>> ebuild for the failing version and in there I found the following comment:
>> ====== comment from ebuild =======
>>     # Upstream hasn't released a new binary.  We snipe ours from Fedora for
>> now.
>>     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
>> ====== end comment from ebuild =======
>> which might in fact underline that there might be an issue similar to what
>> you described above.
>>
>> What is also pretty interesting is the fact that the old (working) SeaBIOS
>> version 1.7.5 installed as "bios.bin" under /usr/share/seabios is actually
>> 262.144 bytes in size whereas the new (invalid) SeaBIOS 1.8.2 installed in
>> the same location is only half as big: 131.072 bytes.
>>
>> I checked at the download site and the 1.8.2 binary version is indeed not
>> available from http://code.coreboot.org/p/seabios/downloads/. But both the
>> binary versions for 1.7.5 and 1.8.0 are available and both are acutually
>> 262.144 bytes in size, so I'd be very surprised if the 1.8.2 version is
>> really only half that size. By the way, the old working version (according
>> to the ebuild) was directly downloaded from the above url and also shows an
>> identical SHA1 digest to that version available for download there.
> 
> <blinks>I thought Gentoo was all about rebuilding from source and not
> taking binary blobs.

So since SeaBIOS and friends (the blobs) are so sensitive to compilers
and environments and to avoid as much problems for people as possible. I
setup the ebuilds in Gentoo to grab the binary blobs by default and if
the user disabled the binary option it would build from source. The idea
was that Fedora doesn't ship any blobs that can't be rebuilt so I would
follow so the same approach and even use their built blobs. But it
appears to be that there are definitely differences between what QEMU
needs/uses and what upstream ships.

Its pretty common for saying Gentoo is about building from source but
really its about user choice.

That said I haven't been maintaining these for some time now and I've
looked at the state of the way that SeaBIOS and friends are built and I
believe there's an issue and I intend on remedying things soon to avoid
issues like this.

>>
>> To me this looks as if something is really wrong here. If anybody of you has
>> access to a 1.8.2 version, could you please confirm whether there's really
>> that big a size difference between the 1.7.5 and the 1.8.2 versions? Or is
>> that difference probably attributable to the missing CONFIG_XEN option?
> 
> It may be other options too - like CONFIG_XHCI, or a huge amount of other
> ones.

Yes. There's definitely differences.

>>
>> Andrew: I havent't gotten around to run the debug version of the hypervisor
>> again, but if the current suspicion turns out to be true, there's probably
>> not much value in that anyways. Would you agree?
> 
> I am not Andrew and can't really speak for him, but I am going to take a 
> leap here and say he will agree with you.
> 


-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Konrad Rzeszutek Wilk]

On Mon, Nov 16, 2015 at 01:39:08PM -0600, Doug Goldstein wrote:
> On 11/16/15 1:25 PM, Konrad Rzeszutek Wilk wrote:
> > On Mon, Nov 16, 2015 at 08:16:33PM +0100, Atom2 wrote:
> >>
> >>
> >> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
> >>>>>> Your analysis was absolutely spot on. After re-thinking this for a
> >>>>>> moment, I thought going down that route first would make a lot of sense
> >>>>>> as PV guests still do work and one of the differences to HVM domUs is
> >>>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
> >>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
> >>>>>> the relevant timeframe which obviously had not made it to the hvmloader
> >>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
> >>>>>>
> >>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
> >>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
> >>>>>> to strongly indicate that there indeed might be an issue with SeaBIOS as
> >>>>>> this probably was the only variable that had changed from the original
> >>>>>> install of xen-4.5.1.
> >>> I recall seeing this way back in Fedora 20 days. I narrowed it down the
> >>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
> >>>
> >>> Having that fixed in the SeaBIOS package fixed it.
> >> Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
> >> Konrad, you might have found an interesting point. I did have a look at the
> >> ebuild for the failing version and in there I found the following comment:
> >> ====== comment from ebuild =======
> >>     # Upstream hasn't released a new binary.  We snipe ours from Fedora for
> >> now.
> >>     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
> >> ====== end comment from ebuild =======
> >> which might in fact underline that there might be an issue similar to what
> >> you described above.
> >>
> >> What is also pretty interesting is the fact that the old (working) SeaBIOS
> >> version 1.7.5 installed as "bios.bin" under /usr/share/seabios is actually
> >> 262.144 bytes in size whereas the new (invalid) SeaBIOS 1.8.2 installed in
> >> the same location is only half as big: 131.072 bytes.
> >>
> >> I checked at the download site and the 1.8.2 binary version is indeed not
> >> available from http://code.coreboot.org/p/seabios/downloads/. But both the
> >> binary versions for 1.7.5 and 1.8.0 are available and both are acutually
> >> 262.144 bytes in size, so I'd be very surprised if the 1.8.2 version is
> >> really only half that size. By the way, the old working version (according
> >> to the ebuild) was directly downloaded from the above url and also shows an
> >> identical SHA1 digest to that version available for download there.
> > 
> > <blinks>I thought Gentoo was all about rebuilding from source and not
> > taking binary blobs.
> 
> So since SeaBIOS and friends (the blobs) are so sensitive to compilers
> and environments and to avoid as much problems for people as possible. I
> setup the ebuilds in Gentoo to grab the binary blobs by default and if
> the user disabled the binary option it would build from source. The idea
> was that Fedora doesn't ship any blobs that can't be rebuilt so I would
> follow so the same approach and even use their built blobs. But it
> appears to be that there are definitely differences between what QEMU
> needs/uses and what upstream ships.
> 
> Its pretty common for saying Gentoo is about building from source but
> really its about user choice.

Aaah! Thank you for educating me!
> 
> That said I haven't been maintaining these for some time now and I've
> looked at the state of the way that SeaBIOS and friends are built and I
> believe there's an issue and I intend on remedying things soon to avoid
> issues like this.

Woot!
> 
> >>
> >> To me this looks as if something is really wrong here. If anybody of you has
> >> access to a 1.8.2 version, could you please confirm whether there's really
> >> that big a size difference between the 1.7.5 and the 1.8.2 versions? Or is
> >> that difference probably attributable to the missing CONFIG_XEN option?
> > 
> > It may be other options too - like CONFIG_XHCI, or a huge amount of other
> > ones.
> 
> Yes. There's definitely differences.
> 
> >>
> >> Andrew: I havent't gotten around to run the debug version of the hypervisor
> >> again, but if the current suspicion turns out to be true, there's probably
> >> not much value in that anyways. Would you agree?
> > 
> > I am not Andrew and can't really speak for him, but I am going to take a 
> > leap here and say he will agree with you.
> > 
> 
> 
> -- 
> Doug Goldstein
> 

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 16.11.15 um 20:25 schrieb Konrad Rzeszutek Wilk:
[snip]
> <blinks>I thought Gentoo was all about rebuilding from source and not
> taking binary blobs.
Well, you probably caught Gentoo on the wrong foot here, but generally 
it is what you thought it was. And IMHO that's the beauty of it. 
Applying a patch is therefore very easy, you just store the patch file 
in the appropriate directory and it will automatically be picked up by 
the build process.

You could also decide to build SeaBIOS from source (it's just a change 
of a USE flag), but there's a reason why this is not the default and 
that warning pops up if you have decided to build from source:

==== warning from ebuild =====
You have decided to compile your own SeaBIOS. This is not supported by 
upstream unless you use their recommended toolchain (which you are not).

If you are intending to use this build with QEMU, realize you will not 
receive any support if you have compiled your own SeaBIOS. Virtual 
machines subtly fail based on changes in SeaBIOS.
==== end warning from ebuild =====

In any case thanks for your support, your humour (:-) and I am glad you 
at least were not shocked by this information,

Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 16/11/2015 19:16, Atom2 wrote:
>
>
> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>>> moment, I thought going down that route first would make a lot of
>>>>> sense
>>>>> as PV guests still do work and one of the differences to HVM domUs is
>>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
>>>>> 1.8.2 in
>>>>> the relevant timeframe which obviously had not made it to the
>>>>> hvmloader
>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of
>>>>> SeaBIOS.
>>>>>
>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that
>>>>> seemed
>>>>> to strongly indicate that there indeed might be an issue with
>>>>> SeaBIOS as
>>>>> this probably was the only variable that had changed from the
>>>>> original
>>>>> install of xen-4.5.1.
>> I recall seeing this way back in Fedora 20 days. I narrowed it down the
>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>>
>> Having that fixed in the SeaBIOS package fixed it.
> Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
> Konrad, you might have found an interesting point. I did have a look
> at the ebuild for the failing version and in there I found the
> following comment:
> ====== comment from ebuild =======
>     # Upstream hasn't released a new binary.  We snipe ours from
> Fedora for now.
>     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
> ====== end comment from ebuild =======
> which might in fact underline that there might be an issue similar to
> what you described above.
>
> What is also pretty interesting is the fact that the old (working)
> SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
> is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
> 1.8.2 installed in the same location is only half as big: 131.072 bytes.
>
> I checked at the download site and the 1.8.2 binary version is indeed
> not available from http://code.coreboot.org/p/seabios/downloads/. But
> both the binary versions for 1.7.5 and 1.8.0 are available and both
> are acutually 262.144 bytes in size, so I'd be very surprised if the
> 1.8.2 version is really only half that size. By the way, the old
> working version (according to the ebuild) was directly downloaded from
> the above url and also shows an identical SHA1 digest to that version
> available for download there.
>
> To me this looks as if something is really wrong here. If anybody of
> you has access to a 1.8.2 version, could you please confirm whether
> there's really that big a size difference between the 1.7.5 and the
> 1.8.2 versions? Or is that difference probably attributable to the
> missing CONFIG_XEN option?
>
> Andrew: I havent't gotten around to run the debug version of the
> hypervisor again, but if the current suspicion turns out to be true,
> there's probably not much value in that anyways. Would you agree?

Sadly not.

I accept that this issue is possibly fixed in newer SeaBIOS by working
around the issue.

However, I stand by my original point.  *There is no way the guest
should be able to get into this situation in the first place*, and its
implication of *there is a genuine hypervisor bug which we should track
down*, irrespective of whether the issue has been worked around elsehow.

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 17.11.15 um 00:01 schrieb Andrew Cooper:
> On 16/11/2015 19:16, Atom2 wrote:
>>
>> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>>>> moment, I thought going down that route first would make a lot of
>>>>>> sense
>>>>>> as PV guests still do work and one of the differences to HVM domUs is
>>>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
>>>>>> 1.8.2 in
>>>>>> the relevant timeframe which obviously had not made it to the
>>>>>> hvmloader
>>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of
>>>>>> SeaBIOS.
>>>>>>
>>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that
>>>>>> seemed
>>>>>> to strongly indicate that there indeed might be an issue with
>>>>>> SeaBIOS as
>>>>>> this probably was the only variable that had changed from the
>>>>>> original
>>>>>> install of xen-4.5.1.
>>> I recall seeing this way back in Fedora 20 days. I narrowed it down the
>>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>>>
>>> Having that fixed in the SeaBIOS package fixed it.
>> Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
>> Konrad, you might have found an interesting point. I did have a look
>> at the ebuild for the failing version and in there I found the
>> following comment:
>> ====== comment from ebuild =======
>>      # Upstream hasn't released a new binary.  We snipe ours from
>> Fedora for now.
>>      # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
>> ====== end comment from ebuild =======
>> which might in fact underline that there might be an issue similar to
>> what you described above.
>>
>> What is also pretty interesting is the fact that the old (working)
>> SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
>> is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
>> 1.8.2 installed in the same location is only half as big: 131.072 bytes.
>>
>> I checked at the download site and the 1.8.2 binary version is indeed
>> not available from http://code.coreboot.org/p/seabios/downloads/. But
>> both the binary versions for 1.7.5 and 1.8.0 are available and both
>> are acutually 262.144 bytes in size, so I'd be very surprised if the
>> 1.8.2 version is really only half that size. By the way, the old
>> working version (according to the ebuild) was directly downloaded from
>> the above url and also shows an identical SHA1 digest to that version
>> available for download there.
>>
>> To me this looks as if something is really wrong here. If anybody of
>> you has access to a 1.8.2 version, could you please confirm whether
>> there's really that big a size difference between the 1.7.5 and the
>> 1.8.2 versions? Or is that difference probably attributable to the
>> missing CONFIG_XEN option?
>>
>> Andrew: I havent't gotten around to run the debug version of the
>> hypervisor again, but if the current suspicion turns out to be true,
>> there's probably not much value in that anyways. Would you agree?
> Sadly not.
Fair enough. I'll try to get things done, hopefully somewhen tomorrow 
or, in case that doesn't work out, on Wednesday and will send you the 
requested information.

Many thanks for your support, Atom2
> I accept that this issue is possibly fixed in newer SeaBIOS by working
> around the issue.
>
> However, I stand by my original point.  *There is no way the guest
> should be able to get into this situation in the first place*, and its
> implication of *there is a genuine hypervisor bug which we should track
> down*, irrespective of whether the issue has been worked around elsehow

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 6662 bytes --]

Am 17.11.15 um 00:10 schrieb Atom2:
> Am 17.11.15 um 00:01 schrieb Andrew Cooper:
>> On 16/11/2015 19:16, Atom2 wrote:
>>>
>>> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>>>>> moment, I thought going down that route first would make a lot of
>>>>>>> sense
>>>>>>> as PV guests still do work and one of the differences to HVM 
>>>>>>> domUs is
>>>>>>> that the former do _not_ require SeaBIOS. Looking at my log 
>>>>>>> files of
>>>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
>>>>>>> 1.8.2 in
>>>>>>> the relevant timeframe which obviously had not made it to the
>>>>>>> hvmloader
>>>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of
>>>>>>> SeaBIOS.
>>>>>>>
>>>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed 
>>>>>>> SeaBIOS
>>>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that
>>>>>>> seemed
>>>>>>> to strongly indicate that there indeed might be an issue with
>>>>>>> SeaBIOS as
>>>>>>> this probably was the only variable that had changed from the
>>>>>>> original
>>>>>>> install of xen-4.5.1.
>>>> I recall seeing this way back in Fedora 20 days. I narrowed it down 
>>>> the
>>>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>>>>
>>>> Having that fixed in the SeaBIOS package fixed it.
>>> Hi Konrad, Doug, Andrew (specifically added to this part of the 
>>> thread)!
>>> Konrad, you might have found an interesting point. I did have a look
>>> at the ebuild for the failing version and in there I found the
>>> following comment:
>>> ====== comment from ebuild =======
>>>      # Upstream hasn't released a new binary.  We snipe ours from
>>> Fedora for now.
>>>      # 
>>> http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
>>> ====== end comment from ebuild =======
>>> which might in fact underline that there might be an issue similar to
>>> what you described above.
>>>
>>> What is also pretty interesting is the fact that the old (working)
>>> SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
>>> is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
>>> 1.8.2 installed in the same location is only half as big: 131.072 
>>> bytes.
>>>
>>> I checked at the download site and the 1.8.2 binary version is indeed
>>> not available from http://code.coreboot.org/p/seabios/downloads/. But
>>> both the binary versions for 1.7.5 and 1.8.0 are available and both
>>> are acutually 262.144 bytes in size, so I'd be very surprised if the
>>> 1.8.2 version is really only half that size. By the way, the old
>>> working version (according to the ebuild) was directly downloaded from
>>> the above url and also shows an identical SHA1 digest to that version
>>> available for download there.
>>>
>>> To me this looks as if something is really wrong here. If anybody of
>>> you has access to a 1.8.2 version, could you please confirm whether
>>> there's really that big a size difference between the 1.7.5 and the
>>> 1.8.2 versions? Or is that difference probably attributable to the
>>> missing CONFIG_XEN option?
>>>
>>> Andrew: I havent't gotten around to run the debug version of the
>>> hypervisor again, but if the current suspicion turns out to be true,
>>> there's probably not much value in that anyways. Would you agree?
>> Sadly not.
> Fair enough. I'll try to get things done, hopefully somewhen tomorrow 
> or, in case that doesn't work out, on Wednesday and will send you the 
> requested information.
>
> Many thanks for your support, Atom2
>> I accept that this issue is possibly fixed in newer SeaBIOS by working
>> around the issue.
>>
>> However, I stand by my original point.  *There is no way the guest
>> should be able to get into this situation in the first place*, and its
>> implication of *there is a genuine hypervisor bug which we should track
>> down*, irrespective of whether the issue has been worked around elsehow
Hi Andrew,
as promised I have again tried with a debug build and the results are 
very mixed. I initially tried to better understand what the debug USE 
flag actually does in gentoo and my understanding (after reading the so 
called ebuilds) is now that the XEN hypervisor will be built by adding a 
gcc option of "debug=y" - and that's what should compile a debug build - 
right? So I went on and again enabled the debug USE flag plus gdb 
symbols and rebuilt the hypervisor in the hope that this created a valid 
and working debug build.

It, however, seems there's another problem lurking somewhere which only 
manifests itself when I boot from the debug build of the hypervisor. The 
system crashes early on with a DOUBLE FAULT in doIRQ - we have had this 
already earlier in that thread. I am however a step further as the 
disass in gdb now seems to provide not just an empty page full of NULL 
values but rather something that might give you a hint why it crashes 
that early on: Please see the attached disass file (doIRQ) together with 
the serial console output (serial.dbg). The old NULL value file was 
probably because I did not include gdb symbols in the debug build at 
that time - my bad.

So I am at loss why a debug build actually results in a crash early on, 
but probably Doug could step in here and explain that ... it initially 
thought that it might have to do with the hardened sources that I am 
using, but discarded that option because linux does not play any role 
that early on in the boot process (albeit all lines in the serial log 
are prefixed with "(XEN)").

Because using a debug build of the hypervisor did not work out, I at 
least used the debug build of the xen-tools (which includes hvmloader) 
and run this on the working hypervisor (i.e. the standard, non-debug 
build) in the hope that this might prove useful.

Please find attached the "xl dmesg" from an attempt to start a HVM domU. 
There's a bit more information included compared to the previous xl 
dmesg output, but I don't know whether that's of any help. Unfortunately 
I could not spot any lines "prefixed with [HVM:$DOMID.$VCPUID]" which 
you had expected. But I assume that's because I was forced to boot from 
a non-debug build of the hypervisor in order to get the system up at all.

So to me it looks as if we are currently stuck - unless you (or Doug) 
are able to find the root cause of the early crash when using the debug 
build of the hypervisor. If that is resolved, I could then move on to 
the next step and hopefully be able to provide the information that you 
are after.

Sorry for not being able to provide better input right now.

Thanks Atom2

[-- Attachment #2: serial.dbg --]
[-- Type: text/plain, Size: 15119 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=y Wed Nov 18 21:44:54 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot hvm_debug=0xc3f dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.504529] Detected 2394.615 MHz processor.
(XEN) [    1.512357] Initing memory sharing.
(XEN) [    1.516684] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.523531] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.533596] Intel machine check reporting enabled
(XEN) [    1.539144] alt table ffff82d0802e8810 -> ffff82d0802e9830
(XEN) [    1.545540] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.554027] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.560246] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.566578] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.572817] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.579133] Intel VT-d Snoop Control not enabled.
(XEN) [    1.584680] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.590827] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.596638] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.602347] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.616800] I/O virtualisation enabled
(XEN) [    1.621384]  - Dom0 mode: Relaxed
(XEN) [    1.625666] Interrupt remapping enabled
(XEN) [    1.630341] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.636789] ENABLING IO-APIC IRQs
(XEN) [    1.641153]  -> Using old ACK method
(XEN) [    1.645893] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.854527] TSC deadline timer enabled
(XEN) [    2.149527] Platform timer is 14.318MHz HPET
(XEN) [    2.154466] 'A' pressed -> using alternative key handling
(XEN) [    2.155163] Allocated console ring of 64 KiB.
(XEN) [    2.156284] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.157337] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.157858] mwait-idle: v0.4 model 0x2a
(XEN) [    2.158376] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.158943] VMX: Supported advanced features:
(XEN) [    2.159463]  - APIC MMIO access virtualisation
(XEN) [    2.159983]  - APIC TPR shadow
(XEN) [    2.160526]  - Extended Page Tables (EPT)
(XEN) [    2.161046]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.161570]  - Virtual NMI
(XEN) [    2.162110]  - MSR direct-access bitmap
(XEN) [    2.162626]  - Unrestricted Guest
(XEN) [    2.163132] HVM: ASIDs enabled.
(XEN) [    2.163672] HVM: VMX enabled
(XEN) [    2.164175] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.164691] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.165873] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.167380] microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.169195] microcode: CPU6 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.170248] Brought up 8 CPUs
(XEN) [    2.176718] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.197373] ACPI sleep modes: S3
(XEN) [    2.197880] mcheck_poll: Machine check polling timer started.
(XEN) [    2.198412] Dom0 has maximum 792 PIRQs
(XEN) [    2.199006] *** LOADING DOMAIN 0 ***
(XEN) [    2.382223] elf_parse_binary: phdr: paddr=0x1000000 memsz=0x65033b
(XEN) [    2.382784] elf_parse_binary: phdr: paddr=0x1651000 memsz=0x1e5000
(XEN) [    2.383306] elf_parse_binary: phdr: paddr=0x1836000 memsz=0x98000
(XEN) [    2.383829] elf_parse_binary: phdr: paddr=0x18ce000 memsz=0x1000
(XEN) [    2.384387] elf_parse_binary: phdr: paddr=0x18cf000 memsz=0x13c98
(XEN) [    2.384908] elf_parse_binary: phdr: paddr=0x18e3000 memsz=0x3d000
(XEN) [    2.385431] elf_parse_binary: phdr: paddr=0x1920000 memsz=0x1120
(XEN) [    2.385988] elf_parse_binary: phdr: paddr=0x1922000 memsz=0x2de000
(XEN) [    2.386512] elf_parse_binary: memory: 0x1000000 -> 0x1c00000
(XEN) [    2.387035] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [    2.387584] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [    2.388100] elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) [    2.388616] elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) [    2.389173] elf_xen_parse_note: ENTRY = 0xffffffff818e31f0
(XEN) [    2.389692] elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) [    2.390219] elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) [    2.391793] elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) [    2.392311] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [    2.392826] elf_xen_parse_note: LOADER = "generic"
(XEN) [    2.393597] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [    2.394117] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [    2.394630] elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) [    2.395180] elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) [    2.395704] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [    2.396219] elf_xen_addr_calc_check: addresses:
(XEN) [    2.396766]     virt_base        = 0xffffffff80000000
(XEN) [    2.397282]     elf_paddr_offset = 0x0
(XEN) [    2.397791]     virt_offset      = 0xffffffff80000000
(XEN) [    2.398342]     virt_kstart      = 0xffffffff81000000
(XEN) [    2.398858]     virt_kend        = 0xffffffff81c00000
(XEN) [    2.399374]     virt_entry       = 0xffffffff818e31f0
(XEN) [    2.399923]     p2m_base         = 0xffffffffffffffff
(XEN) [    2.400439]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    2.400952]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    2.402210] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    2.402717]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    2.403739]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    2.404298] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    2.404807]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    2.405327]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    2.405846]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    2.406364]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    2.406883]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    2.407402]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    2.407920]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    2.408474]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    2.410055] Dom0 has maximum 8 VCPUs
(XEN) [    2.410603] elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff8165033b
(XEN) [    2.412981] elf_load_binary: phdr 1 at 0xffffffff81651000 -> 0xffffffff81836000
(XEN) [    2.414414] elf_load_binary: phdr 2 at 0xffffffff81836000 -> 0xffffffff818ce000
(XEN) [    2.415548] elf_load_binary: phdr 3 at 0xffffffff818ce000 -> 0xffffffff818cf000
(XEN) [    2.416597] elf_load_binary: phdr 4 at 0xffffffff818cf000 -> 0xffffffff818e2c98
(XEN) [    2.417627] elf_load_binary: phdr 5 at 0xffffffff818e3000 -> 0xffffffff81920000
(XEN) [    2.418723] elf_load_binary: phdr 6 at 0xffffffff81920000 -> 0xffffffff81921120
(XEN) [    2.419737] elf_load_binary: phdr 7 at 0xffffffff81922000 -> 0xffffffff819dd000
(XEN) [    2.743635] *** DOUBLE FAULT ***
(XEN) [    2.747762] ----[ Xen-4.5.2  x86_64  debug=y  Not tainted ]----
(XEN) [    2.754520] CPU:    3
(XEN) [    2.757634] RIP:    e008:[<ffff82d08017658c>] do_IRQ+0x15/0x64c
(XEN) [    2.764468] RFLAGS: 0000000000010087   CONTEXT: hypervisor
(XEN) [    2.770790] rax: 0000000000000246   rbx: ffff83080ca9a200   rcx: 0000000000000001
(XEN) [    2.779543] rdx: ffff82d080318080   rsi: fffffffffffffed4   rdi: ffff83080ca8ed88
(XEN) [    2.788209] rbp: ffff83080ca8ed78   rsp: ffff83080ca8dcf8   r8:  ffff83080ca9d558
(XEN) [    2.796948] r9:  00000000000001c8   r10: 0000000000000001   r11: 0080000000000000
(XEN) [    2.805617] r12: ffff83080ca967d0   r13: ffffffffffffffff   r14: ffff83080ca88000
(XEN) [    2.814356] r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) [    2.823026] cr3: 00000000db69f000   cr2: ffff83080ca8dce8
(XEN) [    2.829327] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [    2.837474] Valid stack range: ffff83080ca8e000-ffff83080ca90000, sp=ffff83080ca8dcf8, tss.esp0=ffff83080ca8ffc0
(XEN) [    2.848969] No stack overflow detected. Skipping stack trace.
(XEN) [    2.855556] 
(XEN) [    2.857949] ****************************************
(XEN) [    2.863754] Panic on CPU 3:
(XEN) [    2.867381] DOUBLE FAULT -- system shutdown
(XEN) [    2.872403] ****************************************
(XEN) [    2.878305] 
(XEN) [    2.880632] Reboot in five seconds...

[-- Attachment #3: doIRQ --]
[-- Type: text/plain, Size: 23069 bytes --]

Dump of assembler code for function do_IRQ:
   0xffff82d080176577 <+0>:	push   %rbp
   0xffff82d080176578 <+1>:	mov    %rsp,%rbp
   0xffff82d08017657b <+4>:	push   %r15
   0xffff82d08017657d <+6>:	push   %r14
   0xffff82d08017657f <+8>:	push   %r13
   0xffff82d080176581 <+10>:	push   %r12
   0xffff82d080176583 <+12>:	push   %rbx
   0xffff82d080176584 <+13>:	lea    -0x1058(%rsp),%rsp
   0xffff82d08017658c <+21>:	orq    $0x0,(%rsp)
   0xffff82d080176591 <+26>:	lea    0x1020(%rsp),%rsp
   0xffff82d080176599 <+34>:	mov    %rdi,%r12
   0xffff82d08017659c <+37>:	movzbl 0x7c(%rdi),%r13d
   0xffff82d0801765a1 <+42>:	mov    %r13d,%edx
   0xffff82d0801765a4 <+45>:	lea    0x1c3515(%rip),%rax        # 0xffff82d080339ac0 <per_cpu__vector_irq>
   0xffff82d0801765ab <+52>:	lea    (%rax,%rdx,4),%rdx
   0xffff82d0801765af <+56>:	mov    %rsp,%rcx
   0xffff82d0801765b2 <+59>:	and    $0xffffffffffff8000,%rcx
   0xffff82d0801765b9 <+66>:	mov    0x7ff0(%rcx),%rax
   0xffff82d0801765c0 <+73>:	mov    (%rdx,%rax,1),%esi
   0xffff82d0801765c3 <+76>:	mov    %esi,-0x44(%rbp)
   0xffff82d0801765c6 <+79>:	lea    0x1c34bb(%rip),%rdx        # 0xffff82d080339a88 <per_cpu____irq_regs>
   0xffff82d0801765cd <+86>:	add    %rdx,%rax
   0xffff82d0801765d0 <+89>:	mov    (%rax),%rbx
   0xffff82d0801765d3 <+92>:	mov    %rbx,-0x50(%rbp)
   0xffff82d0801765d7 <+96>:	mov    %rdi,(%rax)
   0xffff82d0801765da <+99>:	lea    0x1c349f(%rip),%rax        # 0xffff82d080339a80 <per_cpu__irq_count>
   0xffff82d0801765e1 <+106>:	add    0x7ff0(%rcx),%rax
   0xffff82d0801765e8 <+113>:	addl   $0x1,(%rax)
   0xffff82d0801765eb <+116>:	lea    0x199a8e(%rip),%rdx        # 0xffff82d080310080 <irq_stat>
   0xffff82d0801765f2 <+123>:	mov    0x7fe0(%rcx),%eax
   0xffff82d0801765f8 <+129>:	shl    $0x7,%rax
   0xffff82d0801765fc <+133>:	addl   $0x1,0x4(%rdx,%rax,1)
   0xffff82d080176601 <+138>:	test   %esi,%esi
   0xffff82d080176603 <+140>:	jns    0xffff82d0801767af <do_IRQ+568>
   0xffff82d080176609 <+146>:	mov    %r13d,%eax
   0xffff82d08017660c <+149>:	lea    0x1ab3ad(%rip),%rdx        # 0xffff82d0803219c0 <direct_apic_vector>
   0xffff82d080176613 <+156>:	mov    (%rdx,%rax,8),%rax
   0xffff82d080176617 <+160>:	test   %rax,%rax
   0xffff82d08017661a <+163>:	je     0xffff82d080176623 <do_IRQ+172>
   0xffff82d08017661c <+165>:	callq  *%rax
   0xffff82d08017661e <+167>:	jmpq   0xffff82d080176b7e <do_IRQ+1543>
   0xffff82d080176623 <+172>:	mov    %r13d,%eax
   0xffff82d080176626 <+175>:	and    $0xffffffe0,%eax
   0xffff82d080176629 <+178>:	sar    %eax
   0xffff82d08017662b <+180>:	add    $0x100,%eax
   0xffff82d080176630 <+185>:	cltq   
   0xffff82d080176632 <+187>:	cmpb   $0x0,0x116a4e(%rip)        # 0xffff82d08028d087 <x2apic_enabled>
   0xffff82d080176639 <+194>:	je     0xffff82d080176667 <do_IRQ+240>
   0xffff82d08017663b <+196>:	shr    $0x4,%rax
   0xffff82d08017663f <+200>:	lea    0x800(%rax),%rcx
   0xffff82d080176646 <+207>:	rdmsr  
   0xffff82d080176648 <+209>:	mov    %rdx,%rcx
   0xffff82d08017664b <+212>:	shl    $0x20,%rcx
   0xffff82d08017664f <+216>:	or     %rcx,%rax
   0xffff82d080176652 <+219>:	mov    %r13d,%ecx
   0xffff82d080176655 <+222>:	and    $0x1f,%ecx
   0xffff82d080176658 <+225>:	shr    %cl,%eax
   0xffff82d08017665a <+227>:	lea    0xf24c4(%rip),%rbx        # 0xffff82d080268b25
   0xffff82d080176661 <+234>:	test   $0x1,%al
   0xffff82d080176663 <+236>:	je     0xffff82d0801766bb <do_IRQ+324>
   0xffff82d080176665 <+238>:	jmp    0xffff82d08017668a <do_IRQ+275>
   0xffff82d080176667 <+240>:	movabs $0xffff82cfffdfb000,%rdx
   0xffff82d080176671 <+250>:	add    %rdx,%rax
   0xffff82d080176674 <+253>:	mov    (%rax),%edx
   0xffff82d080176676 <+255>:	mov    %r13d,%eax
   0xffff82d080176679 <+258>:	and    $0x1f,%eax
   0xffff82d08017667c <+261>:	lea    0xf24a2(%rip),%rbx        # 0xffff82d080268b25
   0xffff82d080176683 <+268>:	bt     %eax,%edx
   0xffff82d080176686 <+271>:	jae    0xffff82d0801766bb <do_IRQ+324>
   0xffff82d080176688 <+273>:	jmp    0xffff82d0801766a4 <do_IRQ+301>
   0xffff82d08017668a <+275>:	mov    $0x0,%edx
   0xffff82d08017668f <+280>:	mov    $0x0,%eax
   0xffff82d080176694 <+285>:	mov    $0x80b,%ecx
   0xffff82d080176699 <+290>:	wrmsr  
   0xffff82d08017669b <+292>:	lea    0xf514b(%rip),%rbx        # 0xffff82d08026b7ed
   0xffff82d0801766a2 <+299>:	jmp    0xffff82d0801766bb <do_IRQ+324>
   0xffff82d0801766a4 <+301>:	movabs $0xffff82cfffdfb0b0,%rax
   0xffff82d0801766ae <+311>:	movl   $0x0,(%rax)
   0xffff82d0801766b4 <+317>:	lea    0xf5132(%rip),%rbx        # 0xffff82d08026b7ed
   0xffff82d0801766bb <+324>:	lea    -0xe0(%r13),%edi
   0xffff82d0801766c2 <+331>:	cmp    $0xf,%edi
   0xffff82d0801766c5 <+334>:	ja     0xffff82d0801766d4 <do_IRQ+349>
   0xffff82d0801766c7 <+336>:	callq  0xffff82d08016f1ea <bogus_8259A_irq>
   0xffff82d0801766cc <+341>:	test   %al,%al
   0xffff82d0801766ce <+343>:	jne    0xffff82d080176781 <do_IRQ+522>
   0xffff82d0801766d4 <+349>:	mov    %rsp,%rax
   0xffff82d0801766d7 <+352>:	and    $0xffffffffffff8000,%rax
   0xffff82d0801766dd <+358>:	mov    0x7fe0(%rax),%esi
   0xffff82d0801766e3 <+364>:	mov    %rbx,%r8
   0xffff82d0801766e6 <+367>:	mov    -0x44(%rbp),%r12d
   0xffff82d0801766ea <+371>:	mov    %r12d,%ecx
   0xffff82d0801766ed <+374>:	mov    %r13d,%edx
   0xffff82d0801766f0 <+377>:	lea    0x103971(%rip),%rdi        # 0xffff82d08027a068
   0xffff82d0801766f7 <+384>:	mov    $0x0,%eax
   0xffff82d0801766fc <+389>:	callq  0xffff82d080146cfb <printk>
   0xffff82d080176701 <+394>:	mov    0x117108(%rip),%rdx        # 0xffff82d08028d810 <irq_desc>
   0xffff82d080176708 <+401>:	not    %r12d
   0xffff82d08017670b <+404>:	cmp    0x11710a(%rip),%r12d        # 0xffff82d08028d81c <nr_irqs>
   0xffff82d080176712 <+411>:	jae    0xffff82d080176781 <do_IRQ+522>
   0xffff82d080176714 <+413>:	movslq %r12d,%rbx
   0xffff82d080176717 <+416>:	mov    %rbx,%rax
   0xffff82d08017671a <+419>:	shl    $0x8,%rax
   0xffff82d08017671e <+423>:	lea    (%rdx,%rax,1),%rbx
   0xffff82d080176722 <+427>:	cmpq   $0x0,0x8(%rbx)
   0xffff82d080176727 <+432>:	je     0xffff82d080176781 <do_IRQ+522>
   0xffff82d080176729 <+434>:	lea    0x24(%rbx),%r14
   0xffff82d08017672d <+438>:	mov    %r14,%rdi
   0xffff82d080176730 <+441>:	callq  0xffff82d08012e8cd <_spin_lock>
   0xffff82d080176735 <+446>:	sub    $0x8,%rsp
   0xffff82d080176739 <+450>:	mov    0x40(%rbx),%rcx
   0xffff82d08017673d <+454>:	mov    0x38(%rbx),%rdx
   0xffff82d080176741 <+458>:	mov    0x68(%rbx),%rax
   0xffff82d080176745 <+462>:	mov    (%rbx),%esi
   0xffff82d080176747 <+464>:	push   %rsi
   0xffff82d080176748 <+465>:	mov    0x8(%rbx),%rsi
   0xffff82d08017674c <+469>:	pushq  (%rsi)
   0xffff82d08017674e <+471>:	movswl 0x32(%rbx),%esi
   0xffff82d080176752 <+475>:	push   %rsi
   0xffff82d080176753 <+476>:	movswl 0x30(%rbx),%r9d
   0xffff82d080176758 <+481>:	mov    (%rcx),%r8
   0xffff82d08017675b <+484>:	mov    (%rdx),%rcx
   0xffff82d08017675e <+487>:	mov    (%rax),%rdx
   0xffff82d080176761 <+490>:	mov    %r12d,%esi
   0xffff82d080176764 <+493>:	lea    0x103935(%rip),%rdi        # 0xffff82d08027a0a0
   0xffff82d08017676b <+500>:	mov    $0x0,%eax
   0xffff82d080176770 <+505>:	callq  0xffff82d080146cfb <printk>
   0xffff82d080176775 <+510>:	add    $0x20,%rsp
   0xffff82d080176779 <+514>:	mov    %r14,%rdi
   0xffff82d08017677c <+517>:	callq  0xffff82d08012ea4f <_spin_unlock>
   0xffff82d080176781 <+522>:	cmpl   $0x0,0x113d34(%rip)        # 0xffff82d08028a4bc <tb_init_done>
   0xffff82d080176788 <+529>:	je     0xffff82d080176b7e <do_IRQ+1543>
   0xffff82d08017678e <+535>:	mov    %r13d,-0x3c(%rbp)
   0xffff82d080176792 <+539>:	lea    -0x3c(%rbp),%rcx
   0xffff82d080176796 <+543>:	mov    $0x4,%edx
   0xffff82d08017679b <+548>:	mov    $0x1,%esi
   0xffff82d0801767a0 <+553>:	mov    $0x802007,%edi
   0xffff82d0801767a5 <+558>:	callq  0xffff82d08013369f <__trace_var>
   0xffff82d0801767aa <+563>:	jmpq   0xffff82d080176b7e <do_IRQ+1543>
   0xffff82d0801767af <+568>:	movslq -0x44(%rbp),%r15
   0xffff82d0801767b3 <+572>:	mov    %r15,%r14
   0xffff82d0801767b6 <+575>:	shl    $0x8,%r14
   0xffff82d0801767ba <+579>:	mov    %r14,%rbx
   0xffff82d0801767bd <+582>:	add    0x11704c(%rip),%rbx        # 0xffff82d08028d810 <irq_desc>
   0xffff82d0801767c4 <+589>:	lea    0x24(%rbx),%rax
   0xffff82d0801767c8 <+593>:	mov    %rax,-0x58(%rbp)
   0xffff82d0801767cc <+597>:	mov    %rax,%rdi
   0xffff82d0801767cf <+600>:	callq  0xffff82d08012e8cd <_spin_lock>
   0xffff82d0801767d4 <+605>:	mov    0x8(%rbx),%rax
   0xffff82d0801767d8 <+609>:	mov    %rbx,%rdi
   0xffff82d0801767db <+612>:	callq  *0x28(%rax)
   0xffff82d0801767de <+615>:	mov    (%rbx),%edx
   0xffff82d0801767e0 <+617>:	test   $0x10,%dl
   0xffff82d0801767e3 <+620>:	je     0xffff82d080176aca <do_IRQ+1363>
   0xffff82d0801767e9 <+626>:	cmpq   $0x0,0x1ab9e7(%rip)        # 0xffff82d0803221d8 <irq_ratelimit_timer+24>
   0xffff82d0801767f1 <+634>:	je     0xffff82d0801768b4 <do_IRQ+829>
   0xffff82d0801767f7 <+640>:	mov    0x78(%rbx),%eax
   0xffff82d0801767fa <+643>:	lea    0x1(%rax),%edx
   0xffff82d0801767fd <+646>:	mov    %edx,0x78(%rbx)
   0xffff82d080176800 <+649>:	cmp    0x117002(%rip),%eax        # 0xffff82d08028d808 <irq_ratelimit_threshold>
   0xffff82d080176806 <+655>:	jb     0xffff82d0801768b4 <do_IRQ+829>
   0xffff82d08017680c <+661>:	callq  0xffff82d080190b82 <get_s_time>
   0xffff82d080176811 <+666>:	mov    %rax,%r12
   0xffff82d080176814 <+669>:	mov    0x70(%rbx),%rax
   0xffff82d080176818 <+673>:	add    $0x989680,%rax
   0xffff82d08017681e <+679>:	cmp    %rax,%r12
   0xffff82d080176821 <+682>:	jge    0xffff82d0801768a9 <do_IRQ+818>
   0xffff82d080176827 <+688>:	mov    0x8(%rbx),%rax
   0xffff82d08017682b <+692>:	mov    %rbx,%rdi
   0xffff82d08017682e <+695>:	callq  *0x20(%rax)
   0xffff82d080176831 <+698>:	lea    0x80(%rbx),%r14
   0xffff82d080176838 <+705>:	cmp    0x80(%rbx),%r14
   0xffff82d08017683f <+712>:	jne    0xffff82d080176b60 <do_IRQ+1513>
   0xffff82d080176845 <+718>:	lea    0x12b2f4(%rip),%rdi        # 0xffff82d0802a1b40 <irq_ratelimit_lock>
   0xffff82d08017684c <+725>:	callq  0xffff82d08012e8cd <_spin_lock>
   0xffff82d080176851 <+730>:	lea    0x12c668(%rip),%rax        # 0xffff82d0802a2ec0 <irq_ratelimit_list>
   0xffff82d080176858 <+737>:	cmp    %rax,(%rax)
   0xffff82d08017685b <+740>:	jne    0xffff82d080176871 <do_IRQ+762>
   0xffff82d08017685d <+742>:	lea    0x989680(%r12),%rsi
   0xffff82d080176865 <+750>:	lea    0x1ab954(%rip),%rdi        # 0xffff82d0803221c0 <irq_ratelimit_timer>
   0xffff82d08017686c <+757>:	callq  0xffff82d080132023 <set_timer>
   0xffff82d080176871 <+762>:	mov    0x12c648(%rip),%rax        # 0xffff82d0802a2ec0 <irq_ratelimit_list>
   0xffff82d080176878 <+769>:	mov    %r14,0x8(%rax)
   0xffff82d08017687c <+773>:	mov    %rax,0x80(%rbx)
   0xffff82d080176883 <+780>:	lea    0x12c636(%rip),%rax        # 0xffff82d0802a2ec0 <irq_ratelimit_list>
   0xffff82d08017688a <+787>:	mov    %rax,0x88(%rbx)
   0xffff82d080176891 <+794>:	mov    %r14,0x12c628(%rip)        # 0xffff82d0802a2ec0 <irq_ratelimit_list>
   0xffff82d080176898 <+801>:	lea    0x12b2a1(%rip),%rdi        # 0xffff82d0802a1b40 <irq_ratelimit_lock>
   0xffff82d08017689f <+808>:	callq  0xffff82d08012ea4f <_spin_unlock>
   0xffff82d0801768a4 <+813>:	jmpq   0xffff82d080176b60 <do_IRQ+1513>
   0xffff82d0801768a9 <+818>:	movl   $0x0,0x78(%rbx)
   0xffff82d0801768b0 <+825>:	mov    %r12,0x70(%rbx)
   0xffff82d0801768b4 <+829>:	movl   $0x0,-0x60(%rbp)
   0xffff82d0801768bb <+836>:	cmpl   $0x0,0x113bfa(%rip)        # 0xffff82d08028a4bc <tb_init_done>
   0xffff82d0801768c2 <+843>:	je     0xffff82d0801768c9 <do_IRQ+850>
   0xffff82d0801768c4 <+845>:	rdtsc  
   0xffff82d0801768c6 <+847>:	mov    %eax,-0x60(%rbp)
   0xffff82d0801768c9 <+850>:	mov    %r14,%rdi
   0xffff82d0801768cc <+853>:	add    0x116f3d(%rip),%rdi        # 0xffff82d08028d810 <irq_desc>
   0xffff82d0801768d3 <+860>:	mov    0x18(%rdi),%r12
   0xffff82d0801768d7 <+864>:	lea    0x1c2ea2(%rip),%rdx        # 0xffff82d080339780 <per_cpu__pending_eoi>
   0xffff82d0801768de <+871>:	mov    %rsp,%rax
   0xffff82d0801768e1 <+874>:	and    $0xffffffffffff8000,%rax
   0xffff82d0801768e7 <+880>:	mov    0x7ff0(%rax),%rcx
   0xffff82d0801768ee <+887>:	lea    0x1c3193(%rip),%rax        # 0xffff82d080339a88 <per_cpu____irq_regs>
   0xffff82d0801768f5 <+894>:	mov    (%rax,%rcx,1),%rax
   0xffff82d0801768f9 <+898>:	movzbl 0x7c(%rax),%esi
   0xffff82d0801768fd <+902>:	cmpb   $0x0,(%r12)
   0xffff82d080176902 <+907>:	jne    0xffff82d08017692d <do_IRQ+950>
   0xffff82d080176904 <+909>:	cmpb   $0x2,0x3(%r12)
   0xffff82d08017690a <+915>:	je     0xffff82d08017690e <do_IRQ+919>
   0xffff82d08017690c <+917>:	ud2    
   0xffff82d08017690e <+919>:	testb  $0x2,(%rdi)
   0xffff82d080176911 <+922>:	jne    0xffff82d080176915 <do_IRQ+926>
   0xffff82d080176913 <+924>:	ud2    
   0xffff82d080176915 <+926>:	mov    0x8(%rdi),%rax
   0xffff82d080176919 <+930>:	mov    0x30(%rax),%rax
   0xffff82d08017691d <+934>:	test   %rax,%rax
   0xffff82d080176920 <+937>:	je     0xffff82d080176a8f <do_IRQ+1304>
   0xffff82d080176926 <+943>:	callq  *%rax
   0xffff82d080176928 <+945>:	jmpq   0xffff82d080176a8f <do_IRQ+1304>
   0xffff82d08017692d <+950>:	cmpb   $0x2,0x3(%r12)
   0xffff82d080176933 <+956>:	jne    0xffff82d0801769b1 <do_IRQ+1082>
   0xffff82d080176935 <+958>:	add    %rcx,%rdx
   0xffff82d080176938 <+961>:	movzbl 0x2ff(%rdx),%ecx
   0xffff82d08017693f <+968>:	movzbl %cl,%eax
   0xffff82d080176942 <+971>:	test   %eax,%eax
   0xffff82d080176944 <+973>:	je     0xffff82d08017695d <do_IRQ+998>
   0xffff82d080176946 <+975>:	movzbl %cl,%edi
   0xffff82d080176949 <+978>:	movzbl -0x1(%rdx,%rdi,4),%edi
   0xffff82d08017694e <+983>:	cmp    %edi,%esi
   0xffff82d080176950 <+985>:	ja     0xffff82d080176954 <do_IRQ+989>
   0xffff82d080176952 <+987>:	ud2    
   0xffff82d080176954 <+989>:	cmp    $0xbe,%eax
   0xffff82d080176959 <+994>:	jle    0xffff82d08017695d <do_IRQ+998>
   0xffff82d08017695b <+996>:	ud2    
   0xffff82d08017695d <+998>:	movzbl %cl,%eax
   0xffff82d080176960 <+1001>:	lea    (%rdx,%rax,4),%rdi
   0xffff82d080176964 <+1005>:	mov    -0x44(%rbp),%eax
   0xffff82d080176967 <+1008>:	and    $0x7fffff,%eax
   0xffff82d08017696c <+1013>:	add    %eax,%eax
   0xffff82d08017696e <+1015>:	mov    (%rdi),%r8d
   0xffff82d080176971 <+1018>:	and    $0xff000001,%r8d
   0xffff82d080176978 <+1025>:	or     %r8d,%eax
   0xffff82d08017697b <+1028>:	mov    %eax,(%rdi)
   0xffff82d08017697d <+1030>:	mov    %sil,0x3(%rdi)
   0xffff82d080176981 <+1034>:	and    $0xfffffffe,%eax
   0xffff82d080176984 <+1037>:	mov    %al,(%rdi)
   0xffff82d080176986 <+1039>:	add    $0x1,%ecx
   0xffff82d080176989 <+1042>:	mov    %cl,0x2ff(%rdx)
   0xffff82d08017698f <+1048>:	mov    %rsp,%rax
   0xffff82d080176992 <+1051>:	and    $0xffffffffffff8000,%rax
   0xffff82d080176998 <+1057>:	mov    0x7fe0(%rax),%eax
   0xffff82d08017699e <+1063>:	mov    0x8(%r12),%rdx
   0xffff82d0801769a3 <+1068>:	cmp    0x113057(%rip),%eax        # 0xffff82d080289a00 <nr_cpu_ids>
   0xffff82d0801769a9 <+1074>:	jb     0xffff82d0801769ad <do_IRQ+1078>
   0xffff82d0801769ab <+1076>:	ud2    
   0xffff82d0801769ad <+1078>:	lock bts %eax,(%rdx)
   0xffff82d0801769b1 <+1082>:	cmpb   $0x0,(%r12)
   0xffff82d0801769b6 <+1087>:	je     0xffff82d080176a50 <do_IRQ+1241>
   0xffff82d0801769bc <+1093>:	mov    $0x0,%r13d
   0xffff82d0801769c2 <+1099>:	movslq %r13d,%rax
   0xffff82d0801769c5 <+1102>:	mov    0x40(%r12,%rax,8),%rbx
   0xffff82d0801769ca <+1107>:	lea    0xb70(%rbx),%rdi
   0xffff82d0801769d1 <+1114>:	mov    %r15,%rsi
   0xffff82d0801769d4 <+1117>:	callq  0xffff82d08013ebb6 <radix_tree_lookup>
   0xffff82d0801769d9 <+1122>:	test   %rax,%rax
   0xffff82d0801769dc <+1125>:	je     0xffff82d0801769f2 <do_IRQ+1147>
   0xffff82d0801769de <+1127>:	mov    %rax,%rdx
   0xffff82d0801769e1 <+1130>:	and    $0x3,%edx
   0xffff82d0801769e4 <+1133>:	cmp    $0x2,%rdx
   0xffff82d0801769e8 <+1137>:	je     0xffff82d0801769ec <do_IRQ+1141>
   0xffff82d0801769ea <+1139>:	ud2    
   0xffff82d0801769ec <+1141>:	shr    $0x2,%rax
   0xffff82d0801769f0 <+1145>:	jmp    0xffff82d0801769f7 <do_IRQ+1152>
   0xffff82d0801769f2 <+1147>:	mov    $0x0,%eax
   0xffff82d0801769f7 <+1152>:	movslq %eax,%rsi
   0xffff82d0801769fa <+1155>:	lea    0xe0(%rbx),%rdi
   0xffff82d080176a01 <+1162>:	callq  0xffff82d08013ebb6 <radix_tree_lookup>
   0xffff82d080176a06 <+1167>:	mov    %rax,%r14
   0xffff82d080176a09 <+1170>:	cmpb   $0x0,0x3(%r12)
   0xffff82d080176a0f <+1176>:	je     0xffff82d080176a24 <do_IRQ+1197>
   0xffff82d080176a11 <+1178>:	mov    $0x1,%eax
   0xffff82d080176a16 <+1183>:	xchg   %al,0x6(%r14)
   0xffff82d080176a1a <+1187>:	test   %al,%al
   0xffff82d080176a1c <+1189>:	jne    0xffff82d080176a24 <do_IRQ+1197>
   0xffff82d080176a1e <+1191>:	addb   $0x1,0x1(%r12)
   0xffff82d080176a24 <+1197>:	mov    %r14,%rsi
   0xffff82d080176a27 <+1200>:	mov    %rbx,%rdi
   0xffff82d080176a2a <+1203>:	callq  0xffff82d08014d73f <hvm_do_IRQ_dpci>
   0xffff82d080176a2f <+1208>:	test   %eax,%eax
   0xffff82d080176a31 <+1210>:	jne    0xffff82d080176a3e <do_IRQ+1223>
   0xffff82d080176a33 <+1212>:	mov    %r14,%rsi
   0xffff82d080176a36 <+1215>:	mov    %rbx,%rdi
   0xffff82d080176a39 <+1218>:	callq  0xffff82d080108647 <send_guest_pirq>
   0xffff82d080176a3e <+1223>:	add    $0x1,%r13d
   0xffff82d080176a42 <+1227>:	movzbl (%r12),%eax
   0xffff82d080176a47 <+1232>:	cmp    %eax,%r13d
   0xffff82d080176a4a <+1235>:	jl     0xffff82d0801769c2 <do_IRQ+1099>
   0xffff82d080176a50 <+1241>:	cmpb   $0x0,0x3(%r12)
   0xffff82d080176a56 <+1247>:	je     0xffff82d080176a8f <do_IRQ+1304>
   0xffff82d080176a58 <+1249>:	add    $0x10,%r12
   0xffff82d080176a5c <+1253>:	mov    %r12,%rdi
   0xffff82d080176a5f <+1256>:	callq  0xffff82d080132231 <stop_timer>
   0xffff82d080176a64 <+1261>:	mov    %rsp,%rax
   0xffff82d080176a67 <+1264>:	and    $0xffffffffffff8000,%rax
   0xffff82d080176a6d <+1270>:	mov    0x7fe0(%rax),%esi
   0xffff82d080176a73 <+1276>:	mov    %r12,%rdi
   0xffff82d080176a76 <+1279>:	callq  0xffff82d0801323d3 <migrate_timer>
   0xffff82d080176a7b <+1284>:	callq  0xffff82d080190b82 <get_s_time>
   0xffff82d080176a80 <+1289>:	lea    0xf4240(%rax),%rsi
   0xffff82d080176a87 <+1296>:	mov    %r12,%rdi
   0xffff82d080176a8a <+1299>:	callq  0xffff82d080132023 <set_timer>
   0xffff82d080176a8f <+1304>:	cmpl   $0x0,0x113a26(%rip)        # 0xffff82d08028a4bc <tb_init_done>
   0xffff82d080176a96 <+1311>:	je     0xffff82d080176b75 <do_IRQ+1534>
   0xffff82d080176a9c <+1317>:	mov    -0x44(%rbp),%eax
   0xffff82d080176a9f <+1320>:	mov    %eax,-0x3c(%rbp)
   0xffff82d080176aa2 <+1323>:	mov    -0x60(%rbp),%eax
   0xffff82d080176aa5 <+1326>:	mov    %eax,-0x38(%rbp)
   0xffff82d080176aa8 <+1329>:	rdtsc  
   0xffff82d080176aaa <+1331>:	mov    %eax,-0x34(%rbp)
   0xffff82d080176aad <+1334>:	lea    -0x3c(%rbp),%rcx
   0xffff82d080176ab1 <+1338>:	mov    $0xc,%edx
   0xffff82d080176ab6 <+1343>:	mov    $0x1,%esi
   0xffff82d080176abb <+1348>:	mov    $0x802008,%edi
   0xffff82d080176ac0 <+1353>:	callq  0xffff82d08013369f <__trace_var>
   0xffff82d080176ac5 <+1358>:	jmpq   0xffff82d080176b75 <do_IRQ+1534>
   0xffff82d080176aca <+1363>:	mov    %edx,%eax
   0xffff82d080176acc <+1365>:	and    $0xfffffff7,%eax
   0xffff82d080176acf <+1368>:	test   $0x3,%dl
   0xffff82d080176ad2 <+1371>:	je     0xffff82d080176ade <do_IRQ+1383>
   0xffff82d080176ad4 <+1373>:	or     $0x4,%eax
   0xffff82d080176ad7 <+1376>:	mov    %eax,(%rbx)
   0xffff82d080176ad9 <+1378>:	jmpq   0xffff82d080176b60 <do_IRQ+1513>
   0xffff82d080176ade <+1383>:	or     $0x5,%eax
   0xffff82d080176ae1 <+1386>:	mov    0x18(%rbx),%r14
   0xffff82d080176ae5 <+1390>:	lea    -0x3c(%rbp),%rcx
   0xffff82d080176ae9 <+1394>:	mov    %rcx,-0x60(%rbp)
   0xffff82d080176aed <+1398>:	and    $0xfffffffb,%eax
   0xffff82d080176af0 <+1401>:	mov    %eax,(%rbx)
   0xffff82d080176af2 <+1403>:	mov    -0x58(%rbp),%rdi
   0xffff82d080176af6 <+1407>:	callq  0xffff82d08012ea7f <_spin_unlock_irq>
   0xffff82d080176afb <+1412>:	mov    $0x0,%r15d
   0xffff82d080176b01 <+1418>:	cmpl   $0x0,0x1139b4(%rip)        # 0xffff82d08028a4bc <tb_init_done>
   0xffff82d080176b08 <+1425>:	je     0xffff82d080176b0f <do_IRQ+1432>
   0xffff82d080176b0a <+1427>:	rdtsc  
   0xffff82d080176b0c <+1429>:	mov    %eax,%r15d
   0xffff82d080176b0f <+1432>:	mov    0x10(%r14),%rsi
   0xffff82d080176b13 <+1436>:	mov    %r12,%rdx
   0xffff82d080176b16 <+1439>:	mov    -0x44(%rbp),%edi
   0xffff82d080176b19 <+1442>:	callq  *(%r14)
   0xffff82d080176b1c <+1445>:	cmpl   $0x0,0x113999(%rip)        # 0xffff82d08028a4bc <tb_init_done>
   0xffff82d080176b23 <+1452>:	je     0xffff82d080176b4c <do_IRQ+1493>
   0xffff82d080176b25 <+1454>:	mov    -0x44(%rbp),%eax
   0xffff82d080176b28 <+1457>:	mov    %eax,-0x3c(%rbp)
   0xffff82d080176b2b <+1460>:	mov    %r15d,-0x38(%rbp)
   0xffff82d080176b2f <+1464>:	rdtsc  
   0xffff82d080176b31 <+1466>:	mov    %eax,-0x34(%rbp)
   0xffff82d080176b34 <+1469>:	mov    -0x60(%rbp),%rcx
   0xffff82d080176b38 <+1473>:	mov    $0xc,%edx
   0xffff82d080176b3d <+1478>:	mov    $0x1,%esi
   0xffff82d080176b42 <+1483>:	mov    $0x802008,%edi
   0xffff82d080176b47 <+1488>:	callq  0xffff82d08013369f <__trace_var>
   0xffff82d080176b4c <+1493>:	mov    -0x58(%rbp),%rdi
   0xffff82d080176b50 <+1497>:	callq  0xffff82d08012e936 <_spin_lock_irq>
   0xffff82d080176b55 <+1502>:	mov    (%rbx),%eax
   0xffff82d080176b57 <+1504>:	test   $0x4,%al
   0xffff82d080176b59 <+1506>:	jne    0xffff82d080176aed <do_IRQ+1398>
   0xffff82d080176b5b <+1508>:	and    $0xfffffffe,%eax
   0xffff82d080176b5e <+1511>:	mov    %eax,(%rbx)
   0xffff82d080176b60 <+1513>:	mov    0x8(%rbx),%rax
   0xffff82d080176b64 <+1517>:	mov    0x30(%rax),%rax
   0xffff82d080176b68 <+1521>:	test   %rax,%rax
   0xffff82d080176b6b <+1524>:	je     0xffff82d080176b75 <do_IRQ+1534>
   0xffff82d080176b6d <+1526>:	mov    %r13d,%esi
   0xffff82d080176b70 <+1529>:	mov    %rbx,%rdi
   0xffff82d080176b73 <+1532>:	callq  *%rax
   0xffff82d080176b75 <+1534>:	mov    -0x58(%rbp),%rdi
   0xffff82d080176b79 <+1538>:	callq  0xffff82d08012ea4f <_spin_unlock>
   0xffff82d080176b7e <+1543>:	mov    %rsp,%rdx
   0xffff82d080176b81 <+1546>:	and    $0xffffffffffff8000,%rdx
   0xffff82d080176b88 <+1553>:	lea    0x1994f1(%rip),%rcx        # 0xffff82d080310080 <irq_stat>
   0xffff82d080176b8f <+1560>:	mov    0x7fe0(%rdx),%eax
   0xffff82d080176b95 <+1566>:	shl    $0x7,%rax
   0xffff82d080176b99 <+1570>:	subl   $0x1,0x4(%rcx,%rax,1)
   0xffff82d080176b9e <+1575>:	lea    0x1c2ee3(%rip),%rax        # 0xffff82d080339a88 <per_cpu____irq_regs>
   0xffff82d080176ba5 <+1582>:	mov    0x7ff0(%rdx),%rdx
   0xffff82d080176bac <+1589>:	mov    -0x50(%rbp),%rcx
   0xffff82d080176bb0 <+1593>:	mov    %rcx,(%rax,%rdx,1)
   0xffff82d080176bb4 <+1597>:	lea    -0x28(%rbp),%rsp
   0xffff82d080176bb8 <+1601>:	pop    %rbx
   0xffff82d080176bb9 <+1602>:	pop    %r12
   0xffff82d080176bbb <+1604>:	pop    %r13
   0xffff82d080176bbd <+1606>:	pop    %r14
   0xffff82d080176bbf <+1608>:	pop    %r15
   0xffff82d080176bc1 <+1610>:	pop    %rbp
   0xffff82d080176bc2 <+1611>:	retq   
End of assembler dump.

[-- Attachment #4: xl.dmesg --]
[-- Type: text/plain, Size: 35402 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Wed Nov 18 22:55:24 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga console_timestamps=boot hvm_debug=0xc3f dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    1.444848] Detected 2394.579 MHz processor.
(XEN) [    1.452444] Initing memory sharing.
(XEN) [    1.456772] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    1.463620] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    1.473725] Intel machine check reporting enabled
(XEN) [    1.479272] alt table ffff82d0802cfa90 -> ffff82d0802d0bd0
(XEN) [    1.485705] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    1.494189] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    1.500368] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    1.506774] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    1.513009] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    1.519367] Intel VT-d Snoop Control not enabled.
(XEN) [    1.524910] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    1.531173] Intel VT-d Queued Invalidation enabled.
(XEN) [    1.536893] Intel VT-d Interrupt Remapping enabled.
(XEN) [    1.542605] Intel VT-d Shared EPT tables not enabled.
(XEN) [    1.556868] I/O virtualisation enabled
(XEN) [    1.561576]  - Dom0 mode: Relaxed
(XEN) [    1.565734] Interrupt remapping enabled
(XEN) [    1.570408] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    1.576852] ENABLING IO-APIC IRQs
(XEN) [    1.581081]  -> Using old ACK method
(XEN) [    1.585804] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.794848] TSC deadline timer enabled
(XEN) [    2.799471] Platform timer is 14.318MHz HPET
(XEN) [    2.804268] 'A' pressed -> using alternative key handling
(XEN) [    2.804963] Allocated console ring of 64 KiB.
(XEN) [    2.806146] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.807391] mwait-idle: MWAIT substates: 0x1120
(XEN) [    2.807944] mwait-idle: v0.4 model 0x2a
(XEN) [    2.808461] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    2.808995] VMX: Supported advanced features:
(XEN) [    2.809545]  - APIC MMIO access virtualisation
(XEN) [    2.810066]  - APIC TPR shadow
(XEN) [    2.810580]  - Extended Page Tables (EPT)
(XEN) [    2.811128]  - Virtual-Processor Identifiers (VPID)
(XEN) [    2.811652]  - Virtual NMI
(XEN) [    2.812163]  - MSR direct-access bitmap
(XEN) [    2.812711]  - Unrestricted Guest
(XEN) [    2.813218] HVM: ASIDs enabled.
(XEN) [    2.813724] HVM: VMX enabled
(XEN) [    2.814263] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    2.814779] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    2.815929] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.816967] Brought up 8 CPUs
(XEN) [    2.817487] microcode: CPU5 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.824356] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    2.824874] microcode: CPU7 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    2.846051] ACPI sleep modes: S3
(XEN) [    2.846556] mcheck_poll: Machine check polling timer started.
(XEN) [    2.847083] Dom0 has maximum 792 PIRQs
(XEN) [    2.847668] *** LOADING DOMAIN 0 ***
(XEN) [    3.029255]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    3.029805]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    3.031289] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    3.031796]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    3.032816]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    3.033377] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    3.033886]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    3.034406]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    3.034926]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    3.035444]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    3.035964]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    3.036485]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    3.037003]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    3.037559]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    3.038892] Dom0 has maximum 8 VCPUs
(XEN) [    6.204144] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    6.212447] Std. Loglevel: All
(XEN) [    6.212988] Guest Loglevel: All
(XEN) [    6.213492] Xen is relinquishing VGA console.
(XEN) [    6.214566] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) [    6.214605] Freed 316kB init memory.
(XEN) [    6.214613] '0' pressed -> dumping Dom0's registers
(XEN) [    6.214616] *** Dumping Dom0 vcpu#0 state: ***
(XEN) [    6.214619] RIP:    e033:[<ffffffff818e31f0>]
(XEN) [    6.214621] RFLAGS: 0000000000000200   EM: 1   CONTEXT: pv guest (d0v0)
(XEN) [    6.214624] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214626] rdx: 0000000000000000   rsi: ffffffff82400000   rdi: 0000000000000000
(XEN) [    6.214628] rbp: 0000000000000000   rsp: ffffffff82419000   r8:  0000000000000000
(XEN) [    6.214630] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214632] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214633] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214635] cr3: 0000000802401000   cr2: 0000000000000000
(XEN) [    6.214637] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) [    6.214639] Guest stack trace from rsp=ffffffff82419000:
(XEN) [    6.214640]   Stack empty.
(XEN) [    6.214649] *** Dumping Dom0 vcpu#1 state: ***
(XEN) [    6.214652] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214654] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v1)
(XEN) [    6.214658] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214660] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214662] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214665] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214667] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214669] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214671] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214674] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214707] *** Dumping Dom0 vcpu#2 state: ***
(XEN) [    6.214710] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214712] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v2)
(XEN) [    6.214715] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214717] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214719] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214720] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214722] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214723] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214725] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214727] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214733] *** Dumping Dom0 vcpu#3 state: ***
(XEN) [    6.214735] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214737] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v3)
(XEN) [    6.214740] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214742] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214743] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214745] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214747] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214748] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214750] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214752] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214780] *** Dumping Dom0 vcpu#4 state: ***
(XEN) [    6.214783] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214785] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v4)
(XEN) [    6.214788] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214790] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214791] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214793] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214795] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214797] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214799] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214800] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214807] *** Dumping Dom0 vcpu#5 state: ***
(XEN) [    6.214809] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214810] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v5)
(XEN) [    6.214813] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214815] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214816] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214818] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214820] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214821] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214823] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214824] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214853] *** Dumping Dom0 vcpu#6 state: ***
(XEN) [    6.214856] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214858] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v6)
(XEN) [    6.214861] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214863] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214864] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214866] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214868] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214869] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214871] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214873] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.214879] *** Dumping Dom0 vcpu#7 state: ***
(XEN) [    6.214881] RIP:    0000:[<0000000000000000>]
(XEN) [    6.214882] RFLAGS: 0000000000000000   EM: 1   CONTEXT: pv guest (d0v7)
(XEN) [    6.214885] rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) [    6.214887] rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) [    6.214888] rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) [    6.214890] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [    6.214892] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [    6.214893] r15: 0000000000000000   cr0: 0000000000000000   cr4: 0000000000002660
(XEN) [    6.214895] cr3: 0000000000000000   cr2: 0000000000000000
(XEN) [    6.214897] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) [    6.470939] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.470943] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) [    6.470947] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) [    6.470950] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.470954] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) [    6.470957] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.470960] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.555997] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.556004] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080cab3000 to 0xffffffff81646d10.
(XEN) [    6.556009] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080cab3080 to 0xffffffff81648e70.
(XEN) [    6.556014] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.556019] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83080cab7fc0 to 0x0000000000000000.
(XEN) [    6.556024] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.556029] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.556330] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.556335] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff83080caa3000 to 0xffffffff81646d10.
(XEN) [    6.556338] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff83080caa3080 to 0xffffffff81648e70.
(XEN) [    6.556342] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.556345] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff83080caa7fc0 to 0x0000000000000000.
(XEN) [    6.556348] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.556352] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.556648] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.556653] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca93000 to 0xffffffff81646d10.
(XEN) [    6.556657] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca93080 to 0xffffffff81648e70.
(XEN) [    6.556660] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.556663] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca97fc0 to 0x0000000000000000.
(XEN) [    6.556667] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.556670] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.556942] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.556947] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca83000 to 0xffffffff81646d10.
(XEN) [    6.556950] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca83080 to 0xffffffff81648e70.
(XEN) [    6.556953] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.556957] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca87fc0 to 0x0000000000000000.
(XEN) [    6.556960] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.556963] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.557196] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.557200] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063f3000 to 0xffffffff81646d10.
(XEN) [    6.557204] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063f3080 to 0xffffffff81648e70.
(XEN) [    6.557207] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.557210] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308063f7fc0 to 0x0000000000000000.
(XEN) [    6.557213] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.557216] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.557430] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.557464] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063eb000 to 0xffffffff81646d10.
(XEN) [    6.557468] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063eb080 to 0xffffffff81648e70.
(XEN) [    6.557471] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.557475] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff8308063effc0 to 0x0000000000000000.
(XEN) [    6.557478] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.557481] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.557678] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    6.557682] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063db000 to 0xffffffff81646d10.
(XEN) [    6.557685] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063db080 to 0xffffffff81648e70.
(XEN) [    6.557688] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    6.557691] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000175 from 0xffff8308063dffc0 to 0x0000000000000000.
(XEN) [    6.557695] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0.
(XEN) [    6.557698] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    7.700727] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    7.700730] PCI add device 0000:00:00.0
(XEN) [    7.700921] PCI add device 0000:00:01.0
(XEN) [    7.701104] PCI add device 0000:00:01.1
(XEN) [    7.701334] PCI add device 0000:00:02.0
(XEN) [    7.701539] PCI add device 0000:00:06.0
(XEN) [    7.701824] PCI add device 0000:00:16.0
(XEN) [    7.702109] PCI add device 0000:00:19.0
(XEN) [    7.702406] PCI add device 0000:00:1a.0
(XEN) [    7.702670] PCI add device 0000:00:1b.0
(XEN) [    7.702950] PCI add device 0000:00:1c.0
(XEN) [    7.703213] PCI add device 0000:00:1c.4
(XEN) [    7.703469] PCI add device 0000:00:1c.6
(XEN) [    7.703766] PCI add device 0000:00:1d.0
(XEN) [    7.703941] PCI add device 0000:00:1e.0
(XEN) [    7.704193] PCI add device 0000:00:1f.0
(XEN) [    7.704489] PCI add device 0000:00:1f.2
(XEN) [    7.704671] PCI add device 0000:00:1f.3
(XEN) [    7.704944] PCI add device 0000:01:00.0
(XEN) [    7.726878] PCI add device 0000:02:00.0
(XEN) [    7.746884] PCI add device 0000:03:00.0
(XEN) [    7.747319] PCI add device 0000:04:00.0
(XEN) [    7.767088] PCI add device 0000:05:00.0
(XEN) [    7.767513] PCI add device 0000:06:00.0
(XEN) [    7.768234] PCI add device 0000:07:00.0
(XEN) [    7.787118] PCI add device 0000:08:00.0
(XEN) [    7.787441] PCI add device 0000:09:00.0
(XEN) [    7.787915] PCI add device 0000:0a:08.0
(XEN) [    7.788201] PCI add device 0000:0a:09.0
(XEN) [    7.788486] PCI add device 0000:0a:0a.0
(XEN) [    7.788767] PCI add device 0000:0a:0b.0
(XEN) [    7.850118] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850138] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850183] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850202] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850209] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850226] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850273] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(XEN) [    7.850293] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d080239753
(d1) [   35.999994] HVM Loader
(d1) [   36.000293] Detected Xen v4.5.2
(d1) [   36.000685] Xenbus rings @0xfeffc000, event channel 1
(d1) [   36.001748] System requested SeaBIOS
(d1) [   36.001952] CPU speed is 2395 MHz
(d1) [   36.002577] Relocating guest memory for lowmem MMIO space disabled
(XEN) [   36.003796] irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(d1) [   36.004196] PCI-ISA link 0 routed to IRQ5
(XEN) [   36.004355] irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(d1) [   36.004738] PCI-ISA link 1 routed to IRQ10
(XEN) [   36.004885] irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(d1) [   36.005268] PCI-ISA link 2 routed to IRQ11
(XEN) [   36.005441] irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(d1) [   36.005826] PCI-ISA link 3 routed to IRQ5
(d1) [   36.021936] pci dev 01:3 INTA->IRQ10
(d1) [   36.024582] pci dev 02:0 INTA->IRQ11
(d1) [   36.030503] pci dev 04:0 INTA->IRQ5
(d1) [   36.033877] pci dev 05:0 INTA->IRQ10
(d1) [   36.037262] pci dev 06:0 INTA->IRQ11
(d1) [   36.057653] No RAM in high memory; setting high_mem resource base to 100000000
(d1) [   36.058139] pci dev 03:0 bar 10 size 002000000: 0f0000008
(d1) [   36.059845] pci dev 02:0 bar 14 size 001000000: 0f2000008
(d1) [   36.061531] pci dev 04:0 bar 10 size 000020000: 0f3000004
(XEN) [   36.061685] memory_map:add: dom1 gfn=f3000 mfn=f7c00 nr=20
(d1) [   36.064837] pci dev 03:0 bar 30 size 000010000: 0f3020000
(d1) [   36.065415] pci dev 04:0 bar 30 size 000010000: 0f3030000
(d1) [   36.066009] pci dev 05:0 bar 30 size 000010000: 0f3040000
(d1) [   36.066606] pci dev 06:0 bar 10 size 000010000: 0f3050000
(XEN) [   36.066787] memory_map:add: dom1 gfn=f3050 mfn=f7800 nr=10
(d1) [   36.071651] pci dev 03:0 bar 14 size 000001000: 0f3060000
(XEN) [   36.071944] memory_map:add: dom1 gfn=f3061 mfn=f7842 nr=1
(d1) [   36.075268] pci dev 05:0 bar 14 size 000001000: 0f3061000
(d1) [   36.075833] pci dev 02:0 bar 10 size 000000100: 00000c001
(d1) [   36.077937] pci dev 05:0 bar 10 size 000000100: 00000c101
(XEN) [   36.078289] ioport_map:add: dom1 gport=c100 mport=c200 nr=100
(d1) [   36.081618] pci dev 01:1 bar 20 size 000000010: 00000c201
(d1) [   36.083526] Multiprocessor initialisation:
(d1) [   36.106024]  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   36.125629]  - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   36.125696] Testing HVM environment:
(d1) [   36.129259]  - REP INSB across page boundaries ... passed
(d1) [   36.131211]  - GS base MSRs and SWAPGS ... passed
(d1) [   36.131231] Passed 2 of 2 tests
(d1) [   36.131257] Writing SMBIOS tables ...
(d1) [   36.131983] Loading SeaBIOS ...
(d1) [   36.132018] Creating MP tables ...
(d1) [   36.132098] Loading ACPI ...
(d1) [   36.133403] vm86 TSS at fc00a380
(d1) [   36.133666] BIOS map:
(d1) [   36.133693]  10000-100d3: Scratch space
(d1) [   36.133717]  e0000-fffff: Main BIOS
(d1) [   36.133730] E820 table:
(d1) [   36.133777]  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d1) [   36.133820]  HOLE: 00000000:000a0000 - 00000000:000e0000
(d1) [   36.133871]  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d1) [   36.133921]  [02]: 00000000:00100000 - 00000000:27800000: RAM
(d1) [   36.133964]  HOLE: 00000000:27800000 - 00000000:fc000000
(d1) [   36.134016]  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d1) [   36.134228] Invoking SeaBIOS ...
(XEN) [   36.993929] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) [   36.993931] ************* VMCS Area **************
(XEN) [   36.993933] *** Guest State ***
(XEN) [   36.993935] CR0: actual=0x0000000000000039, shadow=0x0000000000000011, gh_mask=ffffffffffffffff
(XEN) [   36.993937] CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff
(XEN) [   36.993938] CR3: actual=0x0000000000800000, target_count=0
(XEN) [   36.993940]      target0=0000000000000000, target1=0000000000000000
(XEN) [   36.993941]      target2=0000000000000000, target3=0000000000000000
(XEN) [   36.993943] RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 0x0000000100000000 (0x0000000100000000)
(XEN) [   36.993944] RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 0x0000000000000400
(XEN) [   36.993946] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) [   36.993948] CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993950] DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993952] SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993953] ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993955] FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993957] GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   36.993958] GDTR:                           limit=0x00000037, base=0x00000000000f6d80
(XEN) [   36.993960] LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, base=0x0000000000000000
(XEN) [   36.993961] IDTR:                           limit=0x00000000, base=0x00000000000f6dbe
(XEN) [   36.993963] TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, base=0x0000000000000000
(XEN) [   36.993965] Guest PAT = 0x0007040600070406
(XEN) [   36.993966] TSC Offset = ffffffe521a87e2a
(XEN) [   36.993967] DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) [   36.993969] Interruptibility=0000 ActivityState=0000
(XEN) [   36.993970] *** Host State ***
(XEN) [   36.993971] RSP = 0xffff83080caa7f90  RIP = 0xffff82d0801f47a0
(XEN) [   36.993973] CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) [   36.993974] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83080caaec80
(XEN) [   36.993976] GDTBase=ffff83080ca9f000 IDTBase=ffff83080caab000
(XEN) [   36.993978] CR0=000000008005003b CR3=000000070bf71000 CR4=00000000000426f0
(XEN) [   36.993980] Sysenter RSP=ffff83080caa7fc0 CS:RIP=e008:ffff82d080235550
(XEN) [   36.993981] Host PAT = 0x0000050100070406
(XEN) [   36.993982] *** Control State ***
(XEN) [   36.993983] PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) [   36.993985] EntryControls=000051ff ExitControls=000fefff
(XEN) [   36.993986] ExceptionBitmap=000600c2
(XEN) [   36.993987] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.993989] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   36.993990]         reason=80000021 qualification=00000000
(XEN) [   36.993991] IDTVectoring: info=00000000 errcode=00000000
(XEN) [   36.993992] TPR Threshold = 0x00
(XEN) [   36.993994] EPT pointer = 0x000000070bf3f01e
(XEN) [   36.993995] Virtual processor ID = 0x0007
(XEN) [   36.993996] **************************************
(XEN) [   36.993997] domain_crash called from vmx.c:2505
(XEN) [   36.993999] Domain 1 (vcpu#0) crashed on cpu#2:
(XEN) [   36.994001] ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) [   36.994003] CPU:    2
(XEN) [   36.994004] RIP:    0008:[<0000000100000000>]
(XEN) [   36.994005] RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) [   36.994008] rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) [   36.994009] rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) [   36.994011] rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) [   36.994012] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [   36.994014] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [   36.994015] r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) [   36.994016] cr3: 0000000000800000   cr2: 0000000000000000
(XEN) [   36.994018] ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008

[-- Attachment #5: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 18/11/2015 22:51, Atom2 wrote:
> Am 17.11.15 um 00:10 schrieb Atom2:
>> Am 17.11.15 um 00:01 schrieb Andrew Cooper:
>>> On 16/11/2015 19:16, Atom2 wrote:
>>>>
>>>> Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>>>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>>>>>> moment, I thought going down that route first would make a lot of
>>>>>>>> sense
>>>>>>>> as PV guests still do work and one of the differences to HVM
>>>>>>>> domUs is
>>>>>>>> that the former do _not_ require SeaBIOS. Looking at my log
>>>>>>>> files of
>>>>>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
>>>>>>>> 1.8.2 in
>>>>>>>> the relevant timeframe which obviously had not made it to the
>>>>>>>> hvmloader
>>>>>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of
>>>>>>>> SeaBIOS.
>>>>>>>>
>>>>>>>> So I re-compiled xen-4.5.1 (obviously now using the installed
>>>>>>>> SeaBIOS
>>>>>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that
>>>>>>>> seemed
>>>>>>>> to strongly indicate that there indeed might be an issue with
>>>>>>>> SeaBIOS as
>>>>>>>> this probably was the only variable that had changed from the
>>>>>>>> original
>>>>>>>> install of xen-4.5.1.
>>>>> I recall seeing this way back in Fedora 20 days. I narrowed it
>>>>> down the
>>>>> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>>>>>
>>>>> Having that fixed in the SeaBIOS package fixed it.
>>>> Hi Konrad, Doug, Andrew (specifically added to this part of the
>>>> thread)!
>>>> Konrad, you might have found an interesting point. I did have a look
>>>> at the ebuild for the failing version and in there I found the
>>>> following comment:
>>>> ====== comment from ebuild =======
>>>>      # Upstream hasn't released a new binary.  We snipe ours from
>>>> Fedora for now.
>>>>      #
>>>> http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
>>>> ====== end comment from ebuild =======
>>>> which might in fact underline that there might be an issue similar to
>>>> what you described above.
>>>>
>>>> What is also pretty interesting is the fact that the old (working)
>>>> SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
>>>> is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
>>>> 1.8.2 installed in the same location is only half as big: 131.072
>>>> bytes.
>>>>
>>>> I checked at the download site and the 1.8.2 binary version is indeed
>>>> not available from http://code.coreboot.org/p/seabios/downloads/. But
>>>> both the binary versions for 1.7.5 and 1.8.0 are available and both
>>>> are acutually 262.144 bytes in size, so I'd be very surprised if the
>>>> 1.8.2 version is really only half that size. By the way, the old
>>>> working version (according to the ebuild) was directly downloaded from
>>>> the above url and also shows an identical SHA1 digest to that version
>>>> available for download there.
>>>>
>>>> To me this looks as if something is really wrong here. If anybody of
>>>> you has access to a 1.8.2 version, could you please confirm whether
>>>> there's really that big a size difference between the 1.7.5 and the
>>>> 1.8.2 versions? Or is that difference probably attributable to the
>>>> missing CONFIG_XEN option?
>>>>
>>>> Andrew: I havent't gotten around to run the debug version of the
>>>> hypervisor again, but if the current suspicion turns out to be true,
>>>> there's probably not much value in that anyways. Would you agree?
>>> Sadly not.
>> Fair enough. I'll try to get things done, hopefully somewhen tomorrow
>> or, in case that doesn't work out, on Wednesday and will send you the
>> requested information.
>>
>> Many thanks for your support, Atom2
>>> I accept that this issue is possibly fixed in newer SeaBIOS by working
>>> around the issue.
>>>
>>> However, I stand by my original point.  *There is no way the guest
>>> should be able to get into this situation in the first place*, and its
>>> implication of *there is a genuine hypervisor bug which we should track
>>> down*, irrespective of whether the issue has been worked around elsehow
> Hi Andrew,
> as promised I have again tried with a debug build and the results are
> very mixed. I initially tried to better understand what the debug USE
> flag actually does in gentoo and my understanding (after reading the
> so called ebuilds) is now that the XEN hypervisor will be built by
> adding a gcc option of "debug=y" - and that's what should compile a
> debug build - right?

Yes indeed.

> So I went on and again enabled the debug USE flag plus gdb symbols and
> rebuilt the hypervisor in the hope that this created a valid and
> working debug build.
>
> It, however, seems there's another problem lurking somewhere which
> only manifests itself when I boot from the debug build of the hypervisor.

You did manage to get at least one decent log from a properly debugbuild.

However, all we need is the hvm_debug output.  This patch:

---8<---
diff --git a/xen/include/asm-x86/hvm/support.h
b/xen/include/asm-x86/hvm/support.h
index 05ef5c5..7a8fbb5 100644
--- a/xen/include/asm-x86/hvm/support.h
+++ b/xen/include/asm-x86/hvm/support.h
@@ -28,7 +28,7 @@

 #define HVM_DELIVER_NO_ERROR_CODE  -1

-#ifndef NDEBUG
+#if 1
 #define DBG_LEVEL_0                 (1 << 0)
 #define DBG_LEVEL_1                 (1 << 1)
 #define DBG_LEVEL_2                 (1 << 2)
---8<---

Will enable hvm_debug in a non-debug build of hypervisor.  Can you try
that please?


> The system crashes early on with a DOUBLE FAULT in doIRQ - we have had
> this already earlier in that thread. I am however a step further as
> the disass in gdb now seems to provide not just an empty page full of
> NULL values but rather something that might give you a hint why it
> crashes that early on: Please see the attached disass file (doIRQ)
> together with the serial console output (serial.dbg). The old NULL
> value file was probably because I did not include gdb symbols in the
> debug build at that time - my bad.

The fact that it is completely consistent is useful from a debugging
point of view.

The disassembly of do_IRQ now looks like a plausible function, but the
consistently faulting address has no plausible way of generating a
double fault.  I suspect therefore that something has caused memory
corruption in Xen .text section.

As an experiment, could you try booting with the minimum available
command line options, which look to be just "com1=115200,8n1,0x3f8,4
console=com1,vga dom0_mem=4G,max:4G" to see whether it is an interaction
of the options you have enabled.

If the issue still reproduces, I will rework the previous debugging
patch I gave you to definitely dump the actual code being run at the
time of the fault.

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Am 19.11.15 um 00:17 schrieb Andrew Cooper:
> On 18/11/2015 22:51, Atom2 wrote:
>> Am 17.11.15 um 00:10 schrieb Atom2:
[big snip]
>> Hi Andrew,
>> as promised I have again tried with a debug build and the results are
>> very mixed. I initially tried to better understand what the debug USE
>> flag actually does in gentoo and my understanding (after reading the
>> so called ebuilds) is now that the XEN hypervisor will be built by
>> adding a gcc option of "debug=y" - and that's what should compile a
>> debug build - right?
> Yes indeed.
>
>> So I went on and again enabled the debug USE flag plus gdb symbols and
>> rebuilt the hypervisor in the hope that this created a valid and
>> working debug build.
>>
>> It, however, seems there's another problem lurking somewhere which
>> only manifests itself when I boot from the debug build of the hypervisor.
> You did manage to get at least one decent log from a properly debugbuild.
>
> However, all we need is the hvm_debug output.  This patch:
>
> ---8<---
> diff --git a/xen/include/asm-x86/hvm/support.h
> b/xen/include/asm-x86/hvm/support.h
> index 05ef5c5..7a8fbb5 100644
> --- a/xen/include/asm-x86/hvm/support.h
> +++ b/xen/include/asm-x86/hvm/support.h
> @@ -28,7 +28,7 @@
>
>   #define HVM_DELIVER_NO_ERROR_CODE  -1
>
> -#ifndef NDEBUG
> +#if 1
>   #define DBG_LEVEL_0                 (1 << 0)
>   #define DBG_LEVEL_1                 (1 << 1)
>   #define DBG_LEVEL_2                 (1 << 2)
> ---8<---
>
> Will enable hvm_debug in a non-debug build of hypervisor.  Can you try
> that please?
Done. Please find the xl dmesg output attached to this mail. I guess 
this time it is really what you were expecting. Whether it does make 
sense though, might be a different issue. But I am confident in your 
abilities to figure out what's going on.
>> The system crashes early on with a DOUBLE FAULT in doIRQ - we have had
>> this already earlier in that thread. I am however a step further as
>> the disass in gdb now seems to provide not just an empty page full of
>> NULL values but rather something that might give you a hint why it
>> crashes that early on: Please see the attached disass file (doIRQ)
>> together with the serial console output (serial.dbg). The old NULL
>> value file was probably because I did not include gdb symbols in the
>> debug build at that time - my bad.
> The fact that it is completely consistent is useful from a debugging
> point of view.
>
> The disassembly of do_IRQ now looks like a plausible function, but the
> consistently faulting address has no plausible way of generating a
> double fault.  I suspect therefore that something has caused memory
> corruption in Xen .text section.
>
> As an experiment, could you try booting with the minimum available
> command line options, which look to be just "com1=115200,8n1,0x3f8,4
> console=com1,vga dom0_mem=4G,max:4G" to see whether it is an interaction
> of the options you have enabled.
I haven't done that yet as this will again require a re-compile of the 
non-working debug build. I'll probably give that a try tomorrow.
> If the issue still reproduces, I will rework the previous debugging
> patch I gave you to definitely dump the actual code being run at the
> time of the fault.
>
> ~Andrew
Thanks Atom2

[-- Attachment #2: xl.dmesg.dbg --]
[-- Type: text/plain, Size: 56193 bytes --]

 Xen 4.5.2
(XEN) [    0.000000] Xen version 4.5.2 (@herrenhauspark.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Thu Nov 19 00:54:45 CET 2015
(XEN) [    0.000000] Latest ChangeSet: 
(XEN) [    0.000000] Bootloader: GRUB 2.00
(XEN) [    0.000000] Command line: placeholder ucode=-1 loglvl=all guest_loglvl=all hvm_debug=0xc3f console_timestamps=boot dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0
(XEN) [    0.000000] Video information:
(XEN) [    0.000000]  VGA is text mode 80x25, font 8x16
(XEN) [    0.000000]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [    0.000000] Disc information:
(XEN) [    0.000000]  Found 2 MBR signatures
(XEN) [    0.000000]  Found 2 EDD information structures
(XEN) [    0.000000] Xen-e820 RAM map:
(XEN) [    0.000000]  0000000000000000 - 000000000009d800 (usable)
(XEN) [    0.000000]  000000000009d800 - 00000000000a0000 (reserved)
(XEN) [    0.000000]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [    0.000000]  0000000000100000 - 0000000020000000 (usable)
(XEN) [    0.000000]  0000000020000000 - 0000000020200000 (reserved)
(XEN) [    0.000000]  0000000020200000 - 0000000040000000 (usable)
(XEN) [    0.000000]  0000000040000000 - 0000000040200000 (reserved)
(XEN) [    0.000000]  0000000040200000 - 00000000db9f0000 (usable)
(XEN) [    0.000000]  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN) [    0.000000]  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN) [    0.000000]  00000000dc651000 - 00000000dc652000 (usable)
(XEN) [    0.000000]  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN) [    0.000000]  00000000dc695000 - 00000000dcdba000 (usable)
(XEN) [    0.000000]  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN) [    0.000000]  00000000dcff2000 - 00000000dd000000 (usable)
(XEN) [    0.000000]  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) [    0.000000]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [    0.000000]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [    0.000000]  00000000fed00000 - 00000000fed04000 (reserved)
(XEN) [    0.000000]  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) [    0.000000]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [    0.000000]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [    0.000000]  0000000100000000 - 000000081e600000 (usable)
(XEN) [    0.000000] ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) [    0.000000] ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) [    0.000000] ACPI: FACS DC1F7F80, 0040
(XEN) [    0.000000] ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [    0.000000] ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) [    0.000000] ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) [    0.000000] ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) [    0.000000] ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) [    0.000000] ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) [    0.000000] ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) [    0.000000] System RAM: 32674MB (33458948kB)
(XEN) [    0.000000] No NUMA configuration found
(XEN) [    0.000000] Faking a node at 0000000000000000-000000081e600000
(XEN) [    0.000000] Domain heap initialised
(XEN) [    0.000000] found SMP MP-table at 000fd8d0
(XEN) [    0.000000] DMI 2.7 present.
(XEN) [    0.000000] Using APIC driver default
(XEN) [    0.000000] ACPI: PM-Timer IO Port: 0x408
(XEN) [    0.000000] ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) [    0.000000] ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32
(XEN) [    0.000000] ACPI:             wakeup_vec[dc1f7f8c], vec_size[20]
(XEN) [    0.000000] ACPI: Local APIC address 0xfee00000
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [    0.000000] Processor #0 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [    0.000000] Processor #2 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [    0.000000] Processor #4 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [    0.000000] Processor #6 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [    0.000000] Processor #1 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [    0.000000] Processor #3 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [    0.000000] Processor #5 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [    0.000000] Processor #7 6:10 APIC version 21
(XEN) [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [    0.000000] ACPI: IRQ0 used by override.
(XEN) [    0.000000] ACPI: IRQ2 used by override.
(XEN) [    0.000000] ACPI: IRQ9 used by override.
(XEN) [    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) [    0.000000] ERST table was not found
(XEN) [    0.000000] Using ACPI (MADT) for SMP configuration information
(XEN) [    0.000000] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [    0.000000] IRQ limits: 24 GSI, 1528 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
(XEN) [    0.000000] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [    0.875010] Detected 2394.655 MHz processor.
(XEN) [    0.877796] Initing memory sharing.
(XEN) [    0.878310] xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) [    0.878835] mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) [    0.879866] Intel machine check reporting enabled
(XEN) [    0.880385] alt table ffff82d0802d1a90 -> ffff82d0802d2bd0
(XEN) [    0.881135] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [    0.882146] PCI: MCFG area at f8000000 reserved in E820
(XEN) [    0.882665] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [    0.883609] Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) [    0.884130] Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) [    0.884647] Intel VT-d Snoop Control not enabled.
(XEN) [    0.885166] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    0.885683] Intel VT-d Queued Invalidation enabled.
(XEN) [    0.886197] Intel VT-d Interrupt Remapping enabled.
(XEN) [    0.886711] Intel VT-d Shared EPT tables not enabled.
(XEN) [    0.895408] I/O virtualisation enabled
(XEN) [    0.895915]  - Dom0 mode: Relaxed
(XEN) [    0.896419] Interrupt remapping enabled
(XEN) [    0.897155] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    0.897961] ENABLING IO-APIC IRQs
(XEN) [    0.898466]  -> Using old ACK method
(XEN) [    0.899281] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    1.105002] TSC deadline timer enabled
(XEN) [    1.712306] Platform timer is 14.318MHz HPET
(XEN) [    1.712831] Allocated console ring of 64 KiB.
(XEN) [    1.713952] microcode: CPU0 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    1.714965] mwait-idle: MWAIT substates: 0x1120
(XEN) [    1.715478] mwait-idle: v0.4 model 0x2a
(XEN) [    1.715986] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [    1.716512] VMX: Supported advanced features:
(XEN) [    1.717023]  - APIC MMIO access virtualisation
(XEN) [    1.717532]  - APIC TPR shadow
(XEN) [    1.718035]  - Extended Page Tables (EPT)
(XEN) [    1.718544]  - Virtual-Processor Identifiers (VPID)
(XEN) [    1.719057]  - Virtual NMI
(XEN) [    1.719559]  - MSR direct-access bitmap
(XEN) [    1.720067]  - Unrestricted Guest
(XEN) [    1.720575] HVM: ASIDs enabled.
(XEN) [    1.721080] HVM: VMX enabled
(XEN) [    1.721582] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    1.722100] HVM: HAP page sizes: 4kB, 2MB
(XEN) [    1.723221] microcode: CPU2 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    1.724258] Brought up 8 CPUs
(XEN) [    1.724766] microcode: CPU4 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    1.731583] tmem: initialized comp=1 dedup=1 tze=0
(XEN) [    1.732100] microcode: CPU7 updated from revision 0x28 to 0x29, date = 2013-06-12 
(XEN) [    1.753207] ACPI sleep modes: S3
(XEN) [    1.753711] mcheck_poll: Machine check polling timer started.
(XEN) [    1.754237] Dom0 has maximum 792 PIRQs
(XEN) [    1.754787] *** LOADING DOMAIN 0 ***
(XEN) [    1.933380]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    1.933894]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) [    1.935348] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    1.935857]  Dom0 alloc.:   0000000800000000->0000000804000000 (1029890 pages to be allocated)
(XEN) [    1.936878]  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) [    1.937403] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    1.937911]  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN) [    1.938432]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    1.938951]  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN) [    1.939472]  Start info:    ffffffff82400000->ffffffff824004b4
(XEN) [    1.939991]  Page tables:   ffffffff82401000->ffffffff82418000
(XEN) [    1.940512]  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN) [    1.941032]  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN) [    1.941552]  ENTRY ADDRESS: ffffffff818e31f0
(XEN) [    1.942866] Dom0 has maximum 8 VCPUs
(XEN) [    5.106001] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    5.114635] Std. Loglevel: All
(XEN) [    5.115139] Guest Loglevel: All
(XEN) [    5.115643] Xen is relinquishing VGA console.
(XEN) [    5.116683] *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) [    5.116719] Freed 308kB init memory.
(XEN) [    5.366129] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.366133] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10.
(XEN) [    5.366135] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70.
(XEN) [    5.366138] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.366140] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000.
(XEN) [    5.366142] traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.366144] traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.460630] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.460635] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080cab3000 to 0xffffffff81646d10.
(XEN) [    5.460639] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080cab3080 to 0xffffffff81648e70.
(XEN) [    5.460642] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.460645] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83080cab7fc0 to 0x0000000000000000.
(XEN) [    5.460648] traps.c:2590:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.460652] traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.461026] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.461029] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff83080caab000 to 0xffffffff81646d10.
(XEN) [    5.461032] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff83080caab080 to 0xffffffff81648e70.
(XEN) [    5.461034] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.461036] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff83080caaffc0 to 0x0000000000000000.
(XEN) [    5.461039] traps.c:2590:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.461041] traps.c:2590:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.461361] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.461364] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca93000 to 0xffffffff81646d10.
(XEN) [    5.461366] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca93080 to 0xffffffff81648e70.
(XEN) [    5.461368] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.461370] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca97fc0 to 0x0000000000000000.
(XEN) [    5.461373] traps.c:2590:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.461375] traps.c:2590:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.461663] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.461666] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca8b000 to 0xffffffff81646d10.
(XEN) [    5.461669] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca8b080 to 0xffffffff81648e70.
(XEN) [    5.461671] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.461673] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff83080ca8ffc0 to 0x0000000000000000.
(XEN) [    5.461676] traps.c:2590:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.461678] traps.c:2590:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.461949] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.461952] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063fb000 to 0xffffffff81646d10.
(XEN) [    5.461954] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063fb080 to 0xffffffff81648e70.
(XEN) [    5.461956] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.461958] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308063fffc0 to 0x0000000000000000.
(XEN) [    5.461961] traps.c:2590:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.461963] traps.c:2590:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.462228] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.462232] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063eb000 to 0xffffffff81646d10.
(XEN) [    5.462234] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063eb080 to 0xffffffff81648e70.
(XEN) [    5.462236] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.462239] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff8308063effc0 to 0x0000000000000000.
(XEN) [    5.462241] traps.c:2590:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.462243] traps.c:2590:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    5.462510] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) [    5.462513] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff8308063db000 to 0xffffffff81646d10.
(XEN) [    5.462515] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff8308063db080 to 0xffffffff81648e70.
(XEN) [    5.462517] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) [    5.462519] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000175 from 0xffff8308063dffc0 to 0x0000000000000000.
(XEN) [    5.462522] traps.c:2590:d0v7 Domain attempted WRMSR 0000000000000176 from 0xffff82d080237550 to 0xffffffff81648cf0.
(XEN) [    5.462524] traps.c:2590:d0v7 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) [    6.611987] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [    6.611990] PCI add device 0000:00:00.0
(XEN) [    6.612216] PCI add device 0000:00:01.0
(XEN) [    6.612434] PCI add device 0000:00:01.1
(XEN) [    6.612652] PCI add device 0000:00:02.0
(XEN) [    6.612878] PCI add device 0000:00:06.0
(XEN) [    6.613162] PCI add device 0000:00:16.0
(XEN) [    6.613481] PCI add device 0000:00:19.0
(XEN) [    6.613812] PCI add device 0000:00:1a.0
(XEN) [    6.614112] PCI add device 0000:00:1b.0
(XEN) [    6.614480] PCI add device 0000:00:1c.0
(XEN) [    6.614855] PCI add device 0000:00:1c.4
(XEN) [    6.615220] PCI add device 0000:00:1c.6
(XEN) [    6.615553] PCI add device 0000:00:1d.0
(XEN) [    6.615764] PCI add device 0000:00:1e.0
(XEN) [    6.616014] PCI add device 0000:00:1f.0
(XEN) [    6.616291] PCI add device 0000:00:1f.2
(XEN) [    6.616473] PCI add device 0000:00:1f.3
(XEN) [    6.616743] PCI add device 0000:01:00.0
(XEN) [    6.631514] PCI add device 0000:02:00.0
(XEN) [    6.651593] PCI add device 0000:03:00.0
(XEN) [    6.652040] PCI add device 0000:04:00.0
(XEN) [    6.671872] PCI add device 0000:05:00.0
(XEN) [    6.672389] PCI add device 0000:06:00.0
(XEN) [    6.673158] PCI add device 0000:07:00.0
(XEN) [    6.691995] PCI add device 0000:08:00.0
(XEN) [    6.692326] PCI add device 0000:09:00.0
(XEN) [    6.692835] PCI add device 0000:0a:08.0
(XEN) [    6.693120] PCI add device 0000:0a:09.0
(XEN) [    6.693403] PCI add device 0000:0a:0a.0
(XEN) [    6.693682] PCI add device 0000:0a:0b.0
(XEN) [    6.767732] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767751] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767796] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767812] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767819] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767836] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767884] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [    6.767901] traps.c:3162: GPF (0000): ffff82d08019917c -> ffff82d08023b74d
(XEN) [   32.801381] [HVM:0.1] <do_hvm_op> set param 4 = 1
(XEN) [   32.801388] [HVM:0.1] <do_hvm_op> set param 11 = 1
(XEN) [   32.801392] [HVM:0.1] <do_hvm_op> set param 10 = 1
(XEN) [   32.801397] [HVM:0.1] <do_hvm_op> set param 16 = 1
(XEN) [   32.801402] [HVM:0.1] <do_hvm_op> set param 24 = 0
(XEN) [   32.885059] [HVM:0.2] <do_hvm_op> set param 1 = feffc
(XEN) [   32.885062] [HVM:0.2] <do_hvm_op> set param 6 = feffb
(XEN) [   32.885065] [HVM:0.2] <do_hvm_op> set param 5 = feffd
(XEN) [   32.885067] [HVM:0.2] <do_hvm_op> set param 17 = fefff
(XEN) [   32.885069] [HVM:0.2] <do_hvm_op> set param 27 = feff8
(XEN) [   32.885071] [HVM:0.2] <do_hvm_op> set param 28 = feff9
(XEN) [   32.885073] [HVM:0.2] <do_hvm_op> set param 29 = feffa
(XEN) [   32.885105] [HVM:0.2] <do_hvm_op> set param 32 = feff0
(XEN) [   32.885108] [HVM:0.2] <do_hvm_op> set param 33 = 8
(XEN) [   32.885127] [HVM:0.2] <do_hvm_op> set param 12 = feffe000
(XEN) [   32.885546] [HVM:0.2] <do_hvm_op> get param 1 = feffc
(XEN) [   32.885549] [HVM:0.2] <do_hvm_op> get param 17 = fefff
(XEN) [   32.885551] [HVM:0.2] <do_hvm_op> set param 2 = 1
(XEN) [   32.885554] [HVM:0.2] <do_hvm_op> set param 18 = 2
(XEN) [   32.885838] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885848] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885860] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885868] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885878] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885891] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885896] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885900] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885904] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885907] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885912] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885916] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885921] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885925] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885930] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885933] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885938] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885942] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885947] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885951] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885955] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885958] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885962] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885965] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885969] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885973] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885977] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885980] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885984] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885987] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.885993] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.885996] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886000] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886004] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886007] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886011] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886022] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886025] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886031] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886034] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886039] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886043] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886047] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886050] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886054] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886057] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886061] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886065] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886068] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886072] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886076] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886079] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886083] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886087] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886090] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886094] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886098] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886101] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886105] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886109] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886112] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886116] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886120] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886123] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886127] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886131] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886134] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886138] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886142] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886145] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886149] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886152] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886156] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886160] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886164] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886167] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886171] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886174] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886178] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886182] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886186] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886189] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886193] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886196] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886200] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886204] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886207] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886211] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886215] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886218] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886222] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886226] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886229] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886233] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886237] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886240] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886244] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886247] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886251] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886255] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886258] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886262] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886266] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886269] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886273] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886277] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886280] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886284] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886288] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886291] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886295] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886298] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886302] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886306] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886309] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886313] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886317] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886320] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886324] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886328] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886331] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886335] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886339] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886342] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886346] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886349] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886353] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886357] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886360] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886364] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886368] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886371] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886375] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886379] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886382] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886386] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886390] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886393] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886397] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886400] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886404] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886408] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886411] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886415] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886419] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886422] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886426] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886430] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886433] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886437] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886441] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886444] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886448] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886451] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886455] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886459] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886463] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886466] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886470] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886473] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886477] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886481] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886484] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886488] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886493] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886497] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886502] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886506] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886511] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886515] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886520] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886524] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886529] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886532] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886536] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886540] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886545] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886548] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   32.886553] [HVM:0.2] <do_hvm_op> get param 4 = 1
(XEN) [   32.886557] [HVM:0.2] <do_hvm_op> get param 24 = 0
(XEN) [   33.597994] [HVM:0.4] <do_hvm_op> get param 5 = feffd
(XEN) [   33.598028] [HVM:0.4] <do_hvm_op> get param 6 = feffb
(XEN) [   33.598110] [HVM:0.4] <do_hvm_op> get param 26 = 4
(XEN) [   34.504382] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.505934] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.506207] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.507894] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.508004] HVM Loader
(XEN) [   34.508093] [HVM:1.0] <vmx_msr_write_intercept> ecx=0x40000000, msr_value=0x80000
(XEN) [   34.508096] [HVM:1.0] <long_mode_do_msr_write> msr 0x40000000 content 0x80000
(XEN) [   34.508116] [HVM:1.0] <hvm_do_hypercall> hcall17(1, 19c690, 19c690, 1, 19c67c, 19c6b8)
(XEN) [   34.508121] [HVM:1.0] <hvm_do_hypercall> hcall17 -> 0
(d1) [   34.508308] Detected Xen v4.5.2
(XEN) [   34.508322] [HVM:1.0] <hvm_do_hypercall> hcall34(1, 19c5f0, e9, 1, 19c67c, 19c608)
(XEN) [   34.508326] [HVM:1.0] <do_hvm_op> get param 1 = feffc
(XEN) [   34.508327] [HVM:1.0] <hvm_do_hypercall> hcall34 -> 0
(XEN) [   34.508338] [HVM:1.0] <hvm_do_hypercall> hcall34(1, 19c5f0, 19c628, 1, 19c67c, 19c608)
(XEN) [   34.508341] [HVM:1.0] <do_hvm_op> get param 2 = 1
(XEN) [   34.508342] [HVM:1.0] <hvm_do_hypercall> hcall34 -> 0
(d1) [   34.508707] Xenbus rings @0xfeffc000, event channel 1
(XEN) [   34.508805] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c5ac, 1, 2, f, 19c5f8)
(XEN) [   34.508809] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.508832] [HVM:1.0] <hvm_do_hypercall> hcall12(7, 19c540, fdfff000, 0, 19c67c, 19c558)
(XEN) [   34.508851] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 0
(d1) [   34.509105] System requested SeaBIOS
(d1) [   34.509307] CPU speed is 2395 MHz
(XEN) [   34.509460] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4bc, 1, 2, 20, 19c508)
(XEN) [   34.509464] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.509524] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c480, fdfff800, 803a0, 19c67c, 19c498)
(XEN) [   34.509527] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.509564] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.510010] Relocating guest memory for lowmem MMIO space disabled
(XEN) [   34.510107] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4bc, 1, 2, 18, 19c508)
(XEN) [   34.510111] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.510147] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c480, fdfff800, 803a0, 19c67c, 19c498)
(XEN) [   34.510150] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.510399] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.510433] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c480, fdfff800, 803a0, 19c67c, 19c498)
(XEN) [   34.510436] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.510696] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.510751] irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(XEN) [   34.510827] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.511078] PCI-ISA link 0 routed to IRQ5
(XEN) [   34.511199] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.511237] irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(d1) [   34.511587] PCI-ISA link 1 routed to IRQ10
(XEN) [   34.511730] irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(XEN) [   34.511961] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.512075] PCI-ISA link 2 routed to IRQ11
(XEN) [   34.512208] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.512259] irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(d1) [   34.512595] PCI-ISA link 3 routed to IRQ5
(XEN) [   34.512743] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.512853] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.512935] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.513001] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.513114] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.513192] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.514425] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.514587] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.516800] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.516860] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.516941] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.517001] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.531521] pci dev 01:3 INTA->IRQ10
(d1) [   34.536813] pci dev 02:0 INTA->IRQ11
(d1) [   34.546361] pci dev 04:0 INTA->IRQ5
(d1) [   34.551110] pci dev 05:0 INTA->IRQ10
(XEN) [   34.553102] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.555727] pci dev 06:0 INTA->IRQ11
(XEN) [   34.583097] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(d1) [   34.584700] No RAM in high memory; setting high_mem resource base to 100000000
(d1) [   34.585267] pci dev 03:0 bar 10 size 002000000: 0f0000008
(d1) [   34.587398] pci dev 02:0 bar 14 size 001000000: 0f2000008
(d1) [   34.589320] pci dev 04:0 bar 10 size 000020000: 0f3000004
(XEN) [   34.589517] memory_map:add: dom1 gfn=f3000 mfn=f7c00 nr=20
(d1) [   34.593122] pci dev 03:0 bar 30 size 000010000: 0f3020000
(d1) [   34.593737] pci dev 04:0 bar 30 size 000010000: 0f3030000
(d1) [   34.594372] pci dev 05:0 bar 30 size 000010000: 0f3040000
(d1) [   34.595004] pci dev 06:0 bar 10 size 000010000: 0f3050000
(XEN) [   34.595202] memory_map:add: dom1 gfn=f3050 mfn=f7800 nr=10
(d1) [   34.600185] pci dev 03:0 bar 14 size 000001000: 0f3060000
(XEN) [   34.600512] memory_map:add: dom1 gfn=f3061 mfn=f7842 nr=1
(d1) [   34.603811] pci dev 05:0 bar 14 size 000001000: 0f3061000
(d1) [   34.604435] pci dev 02:0 bar 10 size 000000100: 00000c001
(d1) [   34.606840] pci dev 05:0 bar 10 size 000000100: 00000c101
(XEN) [   34.607280] ioport_map:add: dom1 gport=c100 mport=c200 nr=100
(d1) [   34.610787] pci dev 01:1 bar 20 size 000000010: 00000c201
(d1) [   34.612870] Multiprocessor initialisation:
(d1) [   34.637565]  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(XEN) [   34.637648] [HVM:1.1] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.637660] [HVM:1.1] <hvm_set_cr0> Update CR0 value = 11
(d1) [   34.654138]  - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) [   34.654198] Testing HVM environment:
(XEN) [   34.656118] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 80000011
(XEN) [   34.656120] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 80000011
(XEN) [   34.656646] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 11
(XEN) [   34.656647] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 11
(d1) [   34.657630]  - REP INSB across page boundaries ... passed
(XEN) [   34.659566] [HVM:1.0] <hvm_mov_to_cr> CR4, value = 20
(XEN) [   34.659573] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 80000011
(XEN) [   34.659574] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 80000011
(XEN) [   34.659575] [HVM:1.0] <hvm_set_cr0> Enabling long mode
(XEN) [   34.659578] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 11
(XEN) [   34.659579] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 11
(XEN) [   34.659583] [HVM:1.0] <hvm_mov_to_cr> CR4, value = 0
(d1) [   34.659592]  - GS base MSRs and SWAPGS ... passed
(d1) [   34.659612] Passed 2 of 2 tests
(d1) [   34.659637] Writing SMBIOS tables ...
(XEN) [   34.659641] [HVM:1.0] <hvm_do_hypercall> hcall17(8, 19c5f0, 80220, 152240, 19c67c, 19c618)
(XEN) [   34.659643] [HVM:1.0] <hvm_do_hypercall> hcall17 -> 0
(XEN) [   34.659645] [HVM:1.0] <hvm_do_hypercall> hcall17(0, 0, 80220, 152240, 19c67c, 19c618)
(XEN) [   34.659646] [HVM:1.0] <hvm_do_hypercall> hcall17 -> 40005
(XEN) [   34.659649] [HVM:1.0] <hvm_do_hypercall> hcall17(1, 19c5e0, 80220, 40005, 4, 19c618)
(XEN) [   34.659650] [HVM:1.0] <hvm_do_hypercall> hcall17 -> 0
(XEN) [   34.659655] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 19, 19c468)
(XEN) [   34.659657] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.659664] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659667] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659693] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659695] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659756] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 19, 19c468)
(XEN) [   34.659758] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.659761] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659763] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659776] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659777] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659817] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 1a, 19c468)
(XEN) [   34.659819] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.659822] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659824] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659837] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 4, 19c3f8)
(XEN) [   34.659839] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659877] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 21, 19c468)
(XEN) [   34.659880] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.659883] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 1002e, 19c3f8)
(XEN) [   34.659885] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659899] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 1002e, 19c3f8)
(XEN) [   34.659901] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659940] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 21, 19c468)
(XEN) [   34.659942] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.659945] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107052, 19c3f8)
(XEN) [   34.659947] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659960] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107052, 19c3f8)
(XEN) [   34.659961] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.659999] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 1c, 19c468)
(XEN) [   34.660002] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660006] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107094, 19c3f8)
(XEN) [   34.660007] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660019] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107094, 19c3f8)
(XEN) [   34.660021] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660059] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 22, 19c468)
(XEN) [   34.660062] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660065] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 19c530, 19c3f8)
(XEN) [   34.660066] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660081] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 19c530, 19c3f8)
(XEN) [   34.660083] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660121] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 24, 19c468)
(XEN) [   34.660124] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660127] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 100b0, 19c3f8)
(XEN) [   34.660129] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660144] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 100b0, 19c3f8)
(XEN) [   34.660145] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660184] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 25, 19c468)
(XEN) [   34.660187] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660190] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107052, 19c3f8)
(XEN) [   34.660191] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660206] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 107052, 19c3f8)
(XEN) [   34.660208] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660248] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 13, 19c468)
(XEN) [   34.660250] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660253] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 10115, 19c3f8)
(XEN) [   34.660255] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660270] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 10115, 19c3f8)
(XEN) [   34.660272] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660310] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c41c, 1, 2, 21, 19c468)
(XEN) [   34.660313] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660316] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 54, 19c3f8)
(XEN) [   34.660318] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660332] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c3e0, fdfff800, 803a0, 54, 19c3f8)
(XEN) [   34.660334] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660372] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c44c, fc000, 1, fc001152, 19c478)
(XEN) [   34.660389] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(d1) [   34.660413] Loading SeaBIOS ...
(d1) [   34.660452] Creating MP tables ...
(XEN) [   34.660481] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c5ac, 1, 2, e, 19c5f8)
(XEN) [   34.660484] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.660487] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c570, fdfff800, 803a0, 9f800, 19c588)
(XEN) [   34.660488] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.660503] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c570, fdfff800, 803a0, 9f800, 19c588)
(XEN) [   34.660505] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(d1) [   34.660542] Loading ACPI ...
(XEN) [   34.660545] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c52c, 100d3, 1, 9f800, 19c558)
(XEN) [   34.660549] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660554] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc001, 1, fc009990, 19c528)
(XEN) [   34.660557] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660560] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc002, 1, fc009990, 19c528)
(XEN) [   34.660563] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660565] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc003, 1, fc009990, 19c528)
(XEN) [   34.660568] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660571] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc004, 1, fc009990, 19c528)
(XEN) [   34.660574] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660576] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc005, 1, fc009990, 19c528)
(XEN) [   34.660579] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660581] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc006, 1, fc009990, 19c528)
(XEN) [   34.660584] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660586] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc007, 1, fc009990, 19c528)
(XEN) [   34.660589] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.660591] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc008, 1, fc009990, 19c528)
(XEN) [   34.660594] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.661187] [HVM:1.0] <hvm_do_hypercall> hcall12(6, 19c4fc, fc009, 1, fc00a037, 19c528)
(XEN) [   34.661191] [HVM:1.0] <hvm_do_hypercall> hcall12 -> 1
(XEN) [   34.661234] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4cc, 1, 2, 11, 19c518)
(XEN) [   34.661236] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.661239] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661241] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661255] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661257] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661281] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4cc, 1, 2, 11, 19c518)
(XEN) [   34.661284] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.661287] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661288] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661302] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661304] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661359] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4cc, 1, 2, 17, 19c518)
(XEN) [   34.661361] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.661365] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661366] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661381] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, 8, 19c4a8)
(XEN) [   34.661382] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661438] [HVM:1.0] <hvm_do_hypercall> hcall32(4, 19c4cc, 1, 2, 17, 19c518)
(XEN) [   34.661440] [HVM:1.0] <hvm_do_hypercall> hcall32 -> 0
(XEN) [   34.661444] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, fc00a0f0, 19c4a8)
(XEN) [   34.661445] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661460] [HVM:1.0] <hvm_do_hypercall> hcall29(3, 19c490, fdfff800, 803a0, fc00a0f0, 19c4a8)
(XEN) [   34.661462] [HVM:1.0] <hvm_do_hypercall> hcall29 -> 0
(XEN) [   34.661855] [HVM:1.0] <hvm_do_hypercall> hcall34(0, 19c624, 0, 152240, 9f800, 19c638)
(XEN) [   34.661858] [HVM:1.0] <do_hvm_op> set param 19 = 1
(XEN) [   34.661858] [HVM:1.0] <hvm_do_hypercall> hcall34 -> 0
(XEN) [   34.661867] [HVM:1.0] <hvm_do_hypercall> hcall34(0, 19c624, 0, 152240, 9f800, 19c638)
(XEN) [   34.661868] [HVM:1.0] <do_hvm_op> set param 15 = fc00a380
(XEN) [   34.661869] [HVM:1.0] <hvm_do_hypercall> hcall34 -> 0
(d1) [   34.661890] vm86 TSS at fc00a380
(d1) [   34.662226] BIOS map:
(d1) [   34.662261]  10000-100d3: Scratch space
(d1) [   34.662282]  e0000-fffff: Main BIOS
(d1) [   34.662293] E820 table:
(d1) [   34.662335]  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d1) [   34.662373]  HOLE: 00000000:000a0000 - 00000000:000e0000
(d1) [   34.662419]  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d1) [   34.662461]  [02]: 00000000:00100000 - 00000000:27800000: RAM
(d1) [   34.662499]  HOLE: 00000000:27800000 - 00000000:fc000000
(d1) [   34.662545]  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d1) [   34.662739] Invoking SeaBIOS ...
(XEN) [   34.662742] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 0
(XEN) [   34.662742] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 0
(XEN) [   34.662844] [HVM:1.0] <hvm_mov_to_cr> CR0, value = 11
(XEN) [   34.662845] [HVM:1.0] <hvm_set_cr0> Update CR0 value = 11
(XEN) [   34.763093] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.823098] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.823161] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   34.823989] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   35.373074] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   35.373188] [HVM:1.0] <vmx_restore_guest_msrs> restore guest's EFER with value 0
(XEN) [   35.464356] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) [   35.464357] ************* VMCS Area **************
(XEN) [   35.464358] *** Guest State ***
(XEN) [   35.464359] CR0: actual=0x0000000000000039, shadow=0x0000000000000011, gh_mask=ffffffffffffffff
(XEN) [   35.464361] CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff
(XEN) [   35.464361] CR3: actual=0x0000000000800000, target_count=0
(XEN) [   35.464362]      target0=0000000000000000, target1=0000000000000000
(XEN) [   35.464363]      target2=0000000000000000, target3=0000000000000000
(XEN) [   35.464364] RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP = 0x0000000100000000 (0x0000000100000000)
(XEN) [   35.464365] RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 0x0000000000000400
(XEN) [   35.464366] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) [   35.464368] CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464369] DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464370] SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464371] ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464372] FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464373] GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000
(XEN) [   35.464374] GDTR:                           limit=0x00000037, base=0x00000000000f6d80
(XEN) [   35.464375] LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, base=0x0000000000000000
(XEN) [   35.464376] IDTR:                           limit=0x00000000, base=0x00000000000f6dbe
(XEN) [   35.464377] TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, base=0x0000000000000000
(XEN) [   35.464378] Guest PAT = 0x0007040600070406
(XEN) [   35.464379] TSC Offset = ffffff71374d1298
(XEN) [   35.464379] DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) [   35.464380] Interruptibility=0000 ActivityState=0000
(XEN) [   35.464381] *** Host State ***
(XEN) [   35.464382] RSP = 0xffff83080ca97f90  RIP = 0xffff82d0801f5ae0
(XEN) [   35.464383] CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) [   35.464384] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83080caa2c80
(XEN) [   35.464385] GDTBase=ffff83080ca96000 IDTBase=ffff83080caa1000
(XEN) [   35.464386] CR0=000000008005003b CR3=000000070bf75000 CR4=00000000000426f0
(XEN) [   35.464388] Sysenter RSP=ffff83080ca97fc0 CS:RIP=e008:ffff82d080237550
(XEN) [   35.464388] Host PAT = 0x0000050100070406
(XEN) [   35.464389] *** Control State ***
(XEN) [   35.464390] PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) [   35.464391] EntryControls=000051ff ExitControls=000fefff
(XEN) [   35.464392] ExceptionBitmap=000600c2
(XEN) [   35.464392] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   35.464393] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [   35.464394]         reason=80000021 qualification=00000000
(XEN) [   35.464395] IDTVectoring: info=00000000 errcode=00000000
(XEN) [   35.464395] TPR Threshold = 0x00
(XEN) [   35.464396] EPT pointer = 0x0000000707f0301e
(XEN) [   35.464397] Virtual processor ID = 0x0007
(XEN) [   35.464398] **************************************
(XEN) [   35.464398] domain_crash called from vmx.c:2505
(XEN) [   35.464399] Domain 1 (vcpu#0) crashed on cpu#3:
(XEN) [   35.464401] ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) [   35.464402] CPU:    3
(XEN) [   35.464403] RIP:    0008:[<0000000100000000>]
(XEN) [   35.464404] RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) [   35.464406] rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) [   35.464407] rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) [   35.464408] rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) [   35.464409] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [   35.464409] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [   35.464410] r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) [   35.464411] cr3: 0000000000800000   cr2: 0000000000000000
(XEN) [   35.464412] ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 19/11/2015 00:31, Atom2 wrote:
> Am 19.11.15 um 00:17 schrieb Andrew Cooper:
>> On 18/11/2015 22:51, Atom2 wrote:
>>> Am 17.11.15 um 00:10 schrieb Atom2:
> [big snip]
>>> Hi Andrew,
>>> as promised I have again tried with a debug build and the results are
>>> very mixed. I initially tried to better understand what the debug USE
>>> flag actually does in gentoo and my understanding (after reading the
>>> so called ebuilds) is now that the XEN hypervisor will be built by
>>> adding a gcc option of "debug=y" - and that's what should compile a
>>> debug build - right?
>> Yes indeed.
>>
>>> So I went on and again enabled the debug USE flag plus gdb symbols and
>>> rebuilt the hypervisor in the hope that this created a valid and
>>> working debug build.
>>>
>>> It, however, seems there's another problem lurking somewhere which
>>> only manifests itself when I boot from the debug build of the
>>> hypervisor.
>> You did manage to get at least one decent log from a properly
>> debugbuild.
>>
>> However, all we need is the hvm_debug output.  This patch:
>>
>> ---8<---
>> diff --git a/xen/include/asm-x86/hvm/support.h
>> b/xen/include/asm-x86/hvm/support.h
>> index 05ef5c5..7a8fbb5 100644
>> --- a/xen/include/asm-x86/hvm/support.h
>> +++ b/xen/include/asm-x86/hvm/support.h
>> @@ -28,7 +28,7 @@
>>
>>   #define HVM_DELIVER_NO_ERROR_CODE  -1
>>
>> -#ifndef NDEBUG
>> +#if 1
>>   #define DBG_LEVEL_0                 (1 << 0)
>>   #define DBG_LEVEL_1                 (1 << 1)
>>   #define DBG_LEVEL_2                 (1 << 2)
>> ---8<---
>>
>> Will enable hvm_debug in a non-debug build of hypervisor.  Can you try
>> that please?
> Done. Please find the xl dmesg output attached to this mail. I guess
> this time it is really what you were expecting. Whether it does make
> sense though, might be a different issue. But I am confident in your
> abilities to figure out what's going on.

Thanks! That is what I was looking for.

Sadly, it is less useful than I was hoping.  The guest is not appearing
to do anything interesting which causes the bad state; it is almost a
full second between the previous action of note, and the crash.

Can you email me the bad HVMLoader binary please?

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 19.11.15 um 02:06 schrieb Andrew Cooper:
> Thanks! That is what I was looking for.
>
> Sadly, it is less useful than I was hoping.  The guest is not appearing
> to do anything interesting which causes the bad state; it is almost a
> full second between the previous action of note, and the crash.
>
> Can you email me the bad HVMLoader binary please?
Hi Andrew,
Sure I can email the binary. What version of the HVMLoader do you want 
me to send to you? Should this be one with the debug option and debug 
symbols or a standard build without any of those? I assume we are 
talking about amd64 - right?

Given the recent findings do you still want me to boot with minimal 
command line options to rule out an interaction with the options as 
suggested in an earlier answer?

I can certainly do this, but would need to reconfigure certain things as 
both my HVM domUs require PCI passthrough devices (which you have 
stripped off the suggested minimal command line) to be available. 
Failing to see those devices puts those domUs into an unconfigured state 
and they re-start their initial setup process.

Thanks Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 19/11/2015 20:02, Atom2 wrote:
> Am 19.11.15 um 02:06 schrieb Andrew Cooper:
>> Thanks! That is what I was looking for.
>>
>> Sadly, it is less useful than I was hoping.  The guest is not appearing
>> to do anything interesting which causes the bad state; it is almost a
>> full second between the previous action of note, and the crash.
>>
>> Can you email me the bad HVMLoader binary please?
> Hi Andrew,
> Sure I can email the binary. What version of the HVMLoader do you want
> me to send to you? Should this be one with the debug option and debug
> symbols or a standard build without any of those? I assume we are
> talking about amd64 - right?

hvmloader is always a 32bit binary, no matter what your host environment
is.  I would like any of the ones which reliably crash for  you,
preferably debug if that is available.

>
> Given the recent findings do you still want me to boot with minimal
> command line options to rule out an interaction with the options as
> suggested in an earlier answer?

No - this is unnecessary now.

>
> I can certainly do this, but would need to reconfigure certain things
> as both my HVM domUs require PCI passthrough devices (which you have
> stripped off the suggested minimal command line) to be available.
> Failing to see those devices puts those domUs into an unconfigured
> state and they re-start their initial setup process.

Don't worry - Jan figured this out from the disassembly, which I
admittedly missed.

It would be nice to see your vm's .cfg file to help me reproduce the issue.

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 2445 bytes --]

Am 20.11.15 um 00:53 schrieb Andrew Cooper:
> On 19/11/2015 20:02, Atom2 wrote:
>> Am 19.11.15 um 02:06 schrieb Andrew Cooper:
>>> Thanks! That is what I was looking for.
>>>
>>> Sadly, it is less useful than I was hoping.  The guest is not appearing
>>> to do anything interesting which causes the bad state; it is almost a
>>> full second between the previous action of note, and the crash.
>>>
>>> Can you email me the bad HVMLoader binary please?
>> Hi Andrew,
>> Sure I can email the binary. What version of the HVMLoader do you want
>> me to send to you? Should this be one with the debug option and debug
>> symbols or a standard build without any of those? I assume we are
>> talking about amd64 - right?
> hvmloader is always a 32bit binary, no matter what your host environment
> is.  I would like any of the ones which reliably crash for  you,
> preferably debug if that is available.
Hi Andrew,
appologies for my delay in responding. I have just mailed you the 
crashing hvmloader off-list using my main mail address. It is the debug 
version from xen-4.5.2 together with symbols (as splitdebug) using the 
non-working seabios-1.8.2 - all packed up in a compressed tar file using 
relative names and maintaining the directory structure required for 
splitdebug to work.

If you would also like me to send you the failing seabios binary 
separately, just let me know. My understanding, however, is that seabios 
is anyways emebedded in the hvmloader binary.
>> Given the recent findings do you still want me to boot with minimal
>> command line options to rule out an interaction with the options as
>> suggested in an earlier answer?
> No - this is unnecessary now.
>
>> I can certainly do this, but would need to reconfigure certain things
>> as both my HVM domUs require PCI passthrough devices (which you have
>> stripped off the suggested minimal command line) to be available.
>> Failing to see those devices puts those domUs into an unconfigured
>> state and they re-start their initial setup process.
> Don't worry - Jan figured this out from the disassembly, which I
> admittedly missed.
>
> It would be nice to see your vm's .cfg file to help me reproduce the issue.
Attached please find the config file for one of the HVM virtual machines 
that crashes with the hmvloader binary sent to you by mail. The vm, btw, 
is running the pfsense firewall using FreeBSD. Any more information 
required - just ask.

Many thanks Atom2

[-- Attachment #2: pfsense.cfg --]
[-- Type: text/plain, Size: 549 bytes --]

builder			= 'hvm'
cpus			= '2-7'
vcpus			= 2
cpu_weight		= 512
memory			= 640
name			= 'pfsense'

disk			= [ 'vdev=xvda,format=raw,access=rw,target=/etc/xen/guests/disk.d/pfsense.disk' ]

vif			= [ 'mac=00:16:3e:a1:64:01,bridge=xenbr0,type=vif,vifname=pfsense.0,script=vif-bridge.noTXoffload' ]

on_poweroff		= 'destroy'
on_reboot		= 'restart'
on_crash		= 'destroy'
localtime		= 0

boot			= 'c'

vnc			= 1
vnclisten		= '127.0.0.1'
vncpasswd		= ''
keymap			= 'de'
nographic		= 1
serial			= 'pty'
nx			= 1

pci			= [ '04:00.0', '0a:08.0', '0a:0b.0' ]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 19.11.15 at 00:17, <andrew.cooper3@citrix.com> wrote:
> The disassembly of do_IRQ now looks like a plausible function, but the
> consistently faulting address has no plausible way of generating a
> double fault.  I suspect therefore that something has caused memory
> corruption in Xen .text section.

Dump of assembler code for function do_IRQ:
   0xffff82d080176577 <+0>:	push   %rbp
   0xffff82d080176578 <+1>:	mov    %rsp,%rbp
   0xffff82d08017657b <+4>:	push   %r15
   0xffff82d08017657d <+6>:	push   %r14
   0xffff82d08017657f <+8>:	push   %r13
   0xffff82d080176581 <+10>:	push   %r12
   0xffff82d080176583 <+12>:	push   %rbx
   0xffff82d080176584 <+13>:	lea    -0x1058(%rsp),%rsp
   0xffff82d08017658c <+21>:	orq    $0x0,(%rsp)
   0xffff82d080176591 <+26>:	lea    0x1020(%rsp),%rsp

The orq surely has potential for causing a double fault, if %rsp is
near the stack limit. The two LEAs look suspect, presumably a
result of some non-standard option passed to gcc. Removing that
option might already be a step forward.

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Andrew Cooper]

On 19/11/15 10:24, Jan Beulich wrote:
>>>> On 19.11.15 at 00:17, <andrew.cooper3@citrix.com> wrote:
>> The disassembly of do_IRQ now looks like a plausible function, but the
>> consistently faulting address has no plausible way of generating a
>> double fault.  I suspect therefore that something has caused memory
>> corruption in Xen .text section.
> Dump of assembler code for function do_IRQ:
>    0xffff82d080176577 <+0>:	push   %rbp
>    0xffff82d080176578 <+1>:	mov    %rsp,%rbp
>    0xffff82d08017657b <+4>:	push   %r15
>    0xffff82d08017657d <+6>:	push   %r14
>    0xffff82d08017657f <+8>:	push   %r13
>    0xffff82d080176581 <+10>:	push   %r12
>    0xffff82d080176583 <+12>:	push   %rbx
>    0xffff82d080176584 <+13>:	lea    -0x1058(%rsp),%rsp
>    0xffff82d08017658c <+21>:	orq    $0x0,(%rsp)
>    0xffff82d080176591 <+26>:	lea    0x1020(%rsp),%rsp
>
> The orq surely has potential for causing a double fault, if %rsp is
> near the stack limit. The two LEAs look suspect, presumably a
> result of some non-standard option passed to gcc. Removing that
> option might already be a step forward.

Actually yes - that is a huge quantity of stack usage.

(The actual behaviour looks very suspect - it appears to be completely
pointless).

The #DF handler reports that %rsp in the exception frame is within
range.  Having said that,

(XEN) [    2.788209] rbp: ffff83080ca8ed78   rsp: ffff83080ca8dcf8  
r8:  ffff83080ca9d558
...
(XEN) [    2.837474] Valid stack range:
ffff83080ca8e000-ffff83080ca90000, sp=ffff83080ca8dcf8,
tss.esp0=ffff83080ca8ffc0
(XEN) [    2.848969] No stack overflow detected. Skipping stack trace.

In this case, the stack pointer *is* out of range, and has hit the guard
page.

This means:
1) There is some bug in the stack overflow detection in the #DF handler.
2) Whatever options Gentoo compiles Xen with is sufficient to overflow
the 8K hypervisor stack.

~Andrew

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 19.11.15 um 11:38 schrieb Andrew Cooper:
> On 19/11/15 10:24, Jan Beulich wrote:
>>>>> On 19.11.15 at 00:17, <andrew.cooper3@citrix.com> wrote:
>>> The disassembly of do_IRQ now looks like a plausible function, but the
>>> consistently faulting address has no plausible way of generating a
>>> double fault.  I suspect therefore that something has caused memory
>>> corruption in Xen .text section.
>> Dump of assembler code for function do_IRQ:
>>     0xffff82d080176577 <+0>:	push   %rbp
>>     0xffff82d080176578 <+1>:	mov    %rsp,%rbp
>>     0xffff82d08017657b <+4>:	push   %r15
>>     0xffff82d08017657d <+6>:	push   %r14
>>     0xffff82d08017657f <+8>:	push   %r13
>>     0xffff82d080176581 <+10>:	push   %r12
>>     0xffff82d080176583 <+12>:	push   %rbx
>>     0xffff82d080176584 <+13>:	lea    -0x1058(%rsp),%rsp
>>     0xffff82d08017658c <+21>:	orq    $0x0,(%rsp)
>>     0xffff82d080176591 <+26>:	lea    0x1020(%rsp),%rsp
>>
>> The orq surely has potential for causing a double fault, if %rsp is
>> near the stack limit. The two LEAs look suspect, presumably a
>> result of some non-standard option passed to gcc. Removing that
>> option might already be a step forward.

Andrew, Jan - thanks again.
In terms of non-standard options passed to gcc I have tried to make sense of what flags are actually being used during the build process. I am not absolutely sure, but I think the options passed to gcc are as follows:

I do have system wide flags which are used for non-debug builds:
CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"

for builds with debug symbols (using splitdebug) there are system wide overrides as follows:
CFLAGS="-march=native -O2 -pipe -ggdb"
CXXFLAGS="${CFLAGS}"
LDFLAGS: I'd assume that this inherits its value from the system wide setting of LDFLAGS

for xen (the hypervisor) the build system seems to do the following:
CFLAGS="" (i.e. unset CFLAGS)
to me this indicates that the rest stays untouched (i.e. either standard or debug flags)

for xen-tools (includes e.g. hvmloader) the build system appears to to the following:
CFLAGS="" (i.e. unset CFLAGS)
CXXFLAGS="${CXXFLAGS} -fno-strict-overflow"
LDFLAGS="" (i.e. unset LDFLAGS)

So I think there's probably nothing really fancy in my options to gcc.

> Actually yes - that is a huge quantity of stack usage.
>
> (The actual behaviour looks very suspect - it appears to be completely
> pointless).
>
> The #DF handler reports that %rsp in the exception frame is within
> range.  Having said that,
>
> (XEN) [    2.788209] rbp: ffff83080ca8ed78   rsp: ffff83080ca8dcf8
> r8:  ffff83080ca9d558
> ...
> (XEN) [    2.837474] Valid stack range:
> ffff83080ca8e000-ffff83080ca90000, sp=ffff83080ca8dcf8,
> tss.esp0=ffff83080ca8ffc0
> (XEN) [    2.848969] No stack overflow detected. Skipping stack trace.
>
> In this case, the stack pointer *is* out of range, and has hit the guard
> page.
>
> This means:
> 1) There is some bug in the stack overflow detection in the #DF handler.
> 2) Whatever options Gentoo compiles Xen with is sufficient to overflow
> the 8K hypervisor stack.
Thanks Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 19.11.15 at 20:51, <ariel.atom2@web2web.at> wrote:
> In terms of non-standard options passed to gcc I have tried to make sense of 
> what flags are actually being used during the build process. I am not 
> absolutely sure, but I think the options passed to gcc are as follows:
> 
> I do have system wide flags which are used for non-debug builds:
> CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
> CXXFLAGS="${CFLAGS}"
> LDFLAGS="-Wl,-O1 -Wl,--as-needed"
> 
> for builds with debug symbols (using splitdebug) there are system wide 
> overrides as follows:
> CFLAGS="-march=native -O2 -pipe -ggdb"
> CXXFLAGS="${CFLAGS}"
> LDFLAGS: I'd assume that this inherits its value from the system wide 
> setting of LDFLAGS
> 
> for xen (the hypervisor) the build system seems to do the following:
> CFLAGS="" (i.e. unset CFLAGS)
> to me this indicates that the rest stays untouched (i.e. either standard or 
> debug flags)
> 
> for xen-tools (includes e.g. hvmloader) the build system appears to to the 
> following:
> CFLAGS="" (i.e. unset CFLAGS)
> CXXFLAGS="${CXXFLAGS} -fno-strict-overflow"
> LDFLAGS="" (i.e. unset LDFLAGS)
> 
> So I think there's probably nothing really fancy in my options to gcc.

What you list above seems pretty normal indeed. However, why don't
you simply look at the options actually passed to gcc when building the
hypervisor. And you should also not forget that your distro's gcc build
itself may have certain non-standard option settings. For the odd
stack accesses at function entry I think you should be able to see
whether this is a compiler default by looking at some random, not
overly trivial object file known to have been compiled by the exact
same compiler.

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Am 20.11.15 um 08:57 schrieb Jan Beulich:
> What you list above seems pretty normal indeed. However, why don't you 
> simply look at the options actually passed to gcc when building the 
> hypervisor. And you should also not forget that your distro's gcc 
> build itself may have certain non-standard option settings. For the 
> odd stack accesses at function entry I think you should be able to see 
> whether this is a compiler default by looking at some random, not 
> overly trivial object file known to have been compiled by the exact 
> same compiler. Jan 
Hi Jan,
sorry for my delay in replying. Given that your answer suggests it might 
not be as trivial as I thought, I decided to simply re-compile the 
crashing xen (including symbols and the debug option) and attach the 
complete build log which to me looks as if it includes all flags used 
for every object built. The file is long, thus compressed, but I am sure 
your are able to immeditately spot what to look for.

Thanks Atom2

[-- Attachment #2: build.log.gz --]
[-- Type: application/gzip, Size: 16872 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 24.11.15 at 11:32, <ariel.atom2@web2web.at> wrote:
> Am 20.11.15 um 08:57 schrieb Jan Beulich:
>> What you list above seems pretty normal indeed. However, why don't you 
>> simply look at the options actually passed to gcc when building the 
>> hypervisor. And you should also not forget that your distro's gcc 
>> build itself may have certain non-standard option settings. For the 
>> odd stack accesses at function entry I think you should be able to see 
>> whether this is a compiler default by looking at some random, not 
>> overly trivial object file known to have been compiled by the exact 
>> same compiler. Jan 
> Hi Jan,
> sorry for my delay in replying. Given that your answer suggests it might 
> not be as trivial as I thought, I decided to simply re-compile the 
> crashing xen (including symbols and the debug option) and attach the 
> complete build log which to me looks as if it includes all flags used 
> for every object built. The file is long, thus compressed, but I am sure 
> your are able to immeditately spot what to look for.

Taking a random object out of that log, I can't see any non-standard
option passed to the compiler, so I have to assume this is its default
behavior (i.e. determined at build time, or established by extra
patches). Did you check the result of a random, non-trivial C file from
other than the Xen tree?

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 24.11.15 um 11:43 schrieb Jan Beulich:
> Taking a random object out of that log, I can't see any non-standard
> option passed to the compiler, so I have to assume this is its default
> behavior (i.e. determined at build time, or established by extra
> patches). Did you check the result of a random, non-trivial C file from
> other than the Xen tree?
Hi Jan,
today I update a few packages like net-tools, curl, lipcre, and python 
and none of these packages did have any strange compiler options. It 
appeared that actually all had "-march=native -O2 -pipe 
-fomit-frame-pointer" which is the globally defined standard on my 
system and thus fully expected. A few obviously had package specific 
additions (i.e. python added -fPIC -fwrapv), but nothing really strange 
in my view.

Any more information you require or would you want to see any such build 
log for yourself?

Thanks Atom2

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Jan Beulich]

>>> On 27.11.15 at 23:51, <ariel.atom2@web2web.at> wrote:
> Am 24.11.15 um 11:43 schrieb Jan Beulich:
>> Taking a random object out of that log, I can't see any non-standard
>> option passed to the compiler, so I have to assume this is its default
>> behavior (i.e. determined at build time, or established by extra
>> patches). Did you check the result of a random, non-trivial C file from
>> other than the Xen tree?
> Hi Jan,
> today I update a few packages like net-tools, curl, lipcre, and python 
> and none of these packages did have any strange compiler options. It 
> appeared that actually all had "-march=native -O2 -pipe 
> -fomit-frame-pointer" which is the globally defined standard on my 
> system and thus fully expected. A few obviously had package specific 
> additions (i.e. python added -fPIC -fwrapv), but nothing really strange 
> in my view.
> 
> Any more information you require or would you want to see any such build 
> log for yourself?

I definitely do not want to see any build logs; I also do not _require_
and more information: I think I've provided enough guidance for you
to investigate the odd compiler behavior at your end.

Jan

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Doug Goldstein]

[-- Attachment #1.1: Type: text/plain, Size: 10371 bytes --]

On 11/15/15 7:05 PM, Atom2 wrote:
> Am 15.11.15 um 21:12 schrieb Doug Goldstein:
>> On 11/14/15 6:14 PM, Atom2 wrote:
>>> Am 14.11.15 um 21:32 schrieb Andrew Cooper:
>>>> On 14/11/2015 00:16, Atom2 wrote:
>>>>> Am 13.11.15 um 11:09 schrieb Andrew Cooper:
>>>>>> On 13/11/15 07:25, Jan Beulich wrote:
>>>>>>>>>> On 13.11.15 at 00:00, <ariel.atom2@web2web.at> wrote:
>>>>>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper:
>>>>>>>>> On 12/11/15 14:29, Atom2 wrote:
>>>>>>>>>> Hi Andrew,
>>>>>>>>>> thanks for your reply. Answers are inline further down.
>>>>>>>>>>
>>>>>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper:
>>>>>>>>>>> On 12/11/15 12:52, Jan Beulich wrote:
>>>>>>>>>>>>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>>>>>>>>>>>>> After the upgrade HVM domUs appear to no longer work -
>>>>>>>>>>>>> regardless
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0
>>>>>>>>>>>>> kernel); PV
>>>>>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels.
>>>>>>>>>>>>>
>>>>>>>>>>>>> xl dmesg shows the following information after the first
>>>>>>>>>>>>> crashed HVM
>>>>>>>>>>>>> domU which is started as part of the machine booting up:
>>>>>>>>>>>>> [...]
>>>>>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by
>>>>>>>>>>>>> invalid guest
>>>>>>>>>>>>> state (0).
>>>>>>>>>>>>> (XEN) ************* VMCS Area **************
>>>>>>>>>>>>> (XEN) *** Guest State ***
>>>>>>>>>>>>> (XEN) CR0: actual=0x0000000000000039,
>>>>>>>>>>>>> shadow=0x0000000000000011,
>>>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>>>> (XEN) CR4: actual=0x0000000000002050,
>>>>>>>>>>>>> shadow=0x0000000000000000,
>>>>>>>>>>>>> gh_mask=ffffffffffffffff
>>>>>>>>>>>>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>>>>>>>>>>>>> (XEN)      target0=0000000000000000, target1=0000000000000000
>>>>>>>>>>>>> (XEN)      target2=0000000000000000, target3=0000000000000000
>>>>>>>>>>>>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>>>>>>>>>>>>> 0x0000000100000000 (0x0000000100000000)
>>>>>>>>>>>> Other than RIP looking odd for a guest still in non-paged
>>>>>>>>>>>> protected
>>>>>>>>>>>> mode I can't seem to spot anything wrong with guest state.
>>>>>>>>>>> odd? That will be the source of the failure.
>>>>>>>>>>>
>>>>>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero,
>>>>>>>>>>> and it
>>>>>>>>>>> should not be possible to set any of them.
>>>>>>>>>>>
>>>>>>>>>>> I suspect that the guest has exited for emulation, and there
>>>>>>>>>>> has been a
>>>>>>>>>>> bad update to %rip.  The alternative (which I hope is not the
>>>>>>>>>>> case) is
>>>>>>>>>>> that there is a hardware errata which allows the guest to
>>>>>>>>>>> accidentally
>>>>>>>>>>> get it self into this condition.
>>>>>>>>>>>
>>>>>>>>>>> Are you able to rerun with a debug build of the hypervisor?
>>> [big snip]
>>>>>>>>>> Now _without_ the debug USE flag, but with debug information in
>>>>>>>>>>          the binary (I used splitdebug), all is back to where
>>>>>>>>>> the problem
>>>>>>>>>>          started off (i.e. the system boots without issues
>>>>>>>>>> until such
>>>>>>>>>>          time it starts a HVM domU which then crashes; PV
>>>>>>>>>> domUs are
>>>>>>>>>>          working). I have attached the latest "xl dmesg"
>>>>>>>>>> output with the
>>>>>>>>>>          timing information included.
>>>>>>>>>>   
>>>>> I hope any of this makes sense to you.
>>>>>
>>>>> Again many thanks and best regards
>>>>>
>>>> Right - it would appear that the USE flag is definitely not what you
>>>> wanted, and causes bad compilation for Xen.  The do_IRQ disassembly
>>>> you sent is a the result of disassembling a whole block of zeroes.
>>>> Sorry for leading you on a goose chase - the double faults will be the
>>>> product of bad compilation, rather than anything to do with your
>>>> specific problem.
>>> Hi Andrew,
>>> there's absolutely no need to appologize as it is me who asked for help
>>> and you who generously stepped in and provided it. I really do
>>> appreciate your help and it is for me, as the one seeking help, to
>>> provide all the information you deem necessary and you ask for.
>>>> However, the final log you sent (dmesg) is using a debug Xen, which is
>>>> what I was attempting to get you to do originally.
>>> Next time I know better how to arrive at a debug XEN. It's all about
>>> learning.
>>>> We still observe that the VM ends up in 32bit non-paged mode but with
>>>> an RIP with bit 32 set, which is an invalid state to be in.  However,
>>>> there was nothing particularly interesting in the extra log
>>>> information.
>>>>
>>>> Please can you rerun with "hvm_debug=0xc3f", which will cause far more
>>>> logging to occur to the console while the HVM guest is running.  That
>>>> might show some hints.
>>> I haven't done that yet - but please see my next paragraph. If you are
>>> still interested in this, for whatever reason, I am clearly more than
>>> happy to rerun with your suggested option and provide that information
>>> as well.
>>>> Also, the fact that this occurs just after starting SeaBIOS is
>>>> interesting.  As you have switched versions of Xen, you have also
>>>> switched hvmloader, which contains the SeaBIOS binary embedded in it.
>>>> Would you be able to compile both 4.5.1 and 4.5.2 and switch the
>>>> hvmloader binaries in use.  It would be very interesting to see
>>>> whether the failure is caused by the hvmloader binary or the
>>>> hypervisor.  (With `xl`, you can use
>>>> firmware_override="/full/path/to/firmware" to override the default
>>>> hvmloader).
>>> Your analysis was absolutely spot on. After re-thinking this for a
>>> moment, I thought going down that route first would make a lot of sense
>>> as PV guests still do work and one of the differences to HVM domUs is
>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
>>> the relevant timeframe which obviously had not made it to the hvmloader
>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
>>>
>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
>>> to strongly indicate that there indeed might be an issue with SeaBIOS as
>>> this probably was the only variable that had changed from the original
>>> install of xen-4.5.1.
>>>
>>> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
>>> xen-4.5.1. Voila, the system was again up and running. While still
>>> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
>>> probably guessed it ... the problem was gone: The system boots up with
>>> no issues and everything is fine again.
>>>
>>> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
>>> preventing HVM doamins from successfully starting up. I don't know what
>>> this is triggered from, if this is specific to my hardware or whether
>>> something else in my environment is to blame.
>>>
>>> In any case, I am again more than happy to provide data / run a few
>>> tests should you wish to get to the grounds of this.
>>>
>>> I do owe you a beer (or any other drink) should you ever be at my
>>> location (i.e. Vienna, Austria).
>>>
>>> Many thanks again for your analysis and your first class support. Xen
>>> and their people absolutely rock!
>>>
>>> Atom2
>> I'm a little late to the thread but can you send me (you can do it
>> off-list if you'd like) the USE flags you used for xen, xen-tools and
>> seabios? Also emerge --info. You can kill two birds with one stone by
>> using emerge --info xen.
> Hi Doug,
> here you go:

Thanks. I'll use your configuration as a test point to update a few
things with regard to the Gentoo ebuilds. I'm not the maintainer of Xen
and SeaBIOS but I don't think the maintainers will have much issue with
the changes.

> USE flags:
> app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi
> -flask -xsm"
> app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu
> screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml)
> -ovmf -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
> sys-firmware/seabios-1.7.5::gentoo  USE="binary"

So looking at how SeaBIOS and friends are built I think we have an issue
that needs to be addressed. That being said, you wouldn't have this
issue if you did USE="-system-seabios -system-qemu". I believe you would
also be ok if you had done USE="system-seabios system-qemu". But after a
quick look at everything USE="system-seabios -system-qemu" will
definitely do the wrong thing.

> emerge --info: Please see the attached file
>> I'm not too familiar with the xen ebuilds but I was pretty sure that
>> xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
>> and builds it so that it remains consistent. But obviously your
>> experience shows otherwise.
> You are right, it's xen-tools that builds hvmloader. If I remember
> correctly, the "system-seabios" USE flag (for xen-tools) specifies
> whether sys-firmware/seabios is used and the latter downloads SeaBIOS in
> it's binary form provided its "binary" USE flag is set. At least that's
> my understanding.
>> I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
>> your info would be helpful.
> Great. Whatever makes gentoo and xen stronger will be awesome. What
> immediately springs to mind is to create a separate hvmloader package
> and slot that (that's just an idea and probably not fully thought
> through, but ss far as I understood Andrew, it would then be possible to
> specify the specific firmware version [i.e. hvmloader] to use on xl's
> command line by using firmware_override="full/path/to/firmware").
> 
> I also found out that an upgrade to sys-firmware/seabios obviously does
> not trigger an automatic re-emerge of xen-tools and thus hvmloader.
> Shouldn't this also happen automatically as xen-tools depends on seabios?
> 
> Thanks and best regards Atom2
> 
> 
> P.S. If you prefer to take this off-list, just reply to my mail address.


-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Am 16.11.15 um 20:47 schrieb Doug Goldstein:
> I'm a little late to the thread but can you send me (you can do it
> off-list if you'd like) the USE flags you used for xen, xen-tools and
> seabios? Also emerge --info. You can kill two birds with one stone by
> using emerge --info xen.
>> Hi Doug,
>> here you go:
> Thanks. I'll use your configuration as a test point to update a few
> things with regard to the Gentoo ebuilds. I'm not the maintainer of Xen
> and SeaBIOS but I don't think the maintainers will have much issue with
> the changes.
Hi Doug,
that sounds great. If you require any more information, please do not 
hesitate to contact me - either by mail or through this thread.
>> USE flags:
>> app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi
>> -flask -xsm"
>> app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu
>> screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml)
>> -ovmf -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
>> sys-firmware/seabios-1.7.5::gentoo  USE="binary"
> So looking at how SeaBIOS and friends are built I think we have an issue
> that needs to be addressed. That being said, you wouldn't have this
> issue if you did USE="-system-seabios -system-qemu". I believe you would
> also be ok if you had done USE="system-seabios system-qemu". But after a
> quick look at everything USE="system-seabios -system-qemu" will
> definitely do the wrong thing.
There was a reason why I did not use -system-seabios. The reason (and a 
discussion about the issues I was facing back then) can be found in this 
thread: 
http://lists.xenproject.org/archives/html/xen-users/2015-01/msg00103.html. 
This problem was then sorted by changing to the upstream SeaBIOS instead 
of the bundled version (described in 
http://lists.xenproject.org/archives/html/xen-users/2015-01/msg00131.html) 
by changing the USE flag.

I don't know whether there were any relevant changes since xen-4.3.3, 
but at that point in time it appeared to be my only chance to keep my 
system up and running. Since then (January 2015) I did not find any 
reason to revert the USE flag back to -system-seabios as everything was 
working as expected - obviously up until now.

Thanks Atom2
>> emerge --info: Please see the attached file
>>> I'm not too familiar with the xen ebuilds but I was pretty sure that
>>> xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
>>> and builds it so that it remains consistent. But obviously your
>>> experience shows otherwise.
>> You are right, it's xen-tools that builds hvmloader. If I remember
>> correctly, the "system-seabios" USE flag (for xen-tools) specifies
>> whether sys-firmware/seabios is used and the latter downloads SeaBIOS in
>> it's binary form provided its "binary" USE flag is set. At least that's
>> my understanding.
>>> I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
>>> your info would be helpful.
>> Great. Whatever makes gentoo and xen stronger will be awesome. What
>> immediately springs to mind is to create a separate hvmloader package
>> and slot that (that's just an idea and probably not fully thought
>> through, but ss far as I understood Andrew, it would then be possible to
>> specify the specific firmware version [i.e. hvmloader] to use on xl's
>> command line by using firmware_override="full/path/to/firmware").
>>
>> I also found out that an upgrade to sys-firmware/seabios obviously does
>> not trigger an automatic re-emerge of xen-tools and thus hvmloader.
>> Shouldn't this also happen automatically as xen-tools depends on seabios?
>>
>> Thanks and best regards Atom2
>>
>>
>> P.S. If you prefer to take this off-list, just reply to my mail address.

Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2]

Hi Jan,
many thanks for your reply. Answers are further down inline.
Am 12.11.15 um 13:52 schrieb Jan Beulich:
>>>> On 12.11.15 at 02:08, <ariel.atom2@web2web.at> wrote:
>> After the upgrade HVM domUs appear to no longer work - regardless of the
>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV
>> domUs, however, work just fine as before on both dom0 kernels.
>>
>> xl dmesg shows the following information after the first crashed HVM
>> domU which is started as part of the machine booting up:
>> [...]
>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>> state (0).
>> (XEN) ************* VMCS Area **************
>> (XEN) *** Guest State ***
>> (XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011,
>> gh_mask=ffffffffffffffff
>> (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000,
>> gh_mask=ffffffffffffffff
>> (XEN) CR3: actual=0x0000000000800000, target_count=0
>> (XEN)      target0=0000000000000000, target1=0000000000000000
>> (XEN)      target2=0000000000000000, target3=0000000000000000
>> (XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc)  RIP =
>> 0x0000000100000000 (0x0000000100000000)
> Other than RIP looking odd for a guest still in non-paged protected
> mode I can't seem to spot anything wrong with guest state.
> Considering that there was just a single HVM-related commit
> between the two releases (which looks completely unrelated) I
> wonder whether you're observing a problem that's a side effect
> of something else, e.g. a build system change (compiler update or
> alike). If that can be ruled out, I guess the only chance would be
> for you to bisect for the offending commit.
A few weeks ago there was an update of the gcc compiler from version 
4.8.5 to 4.9.3. The old version of XEN (i.e. 4.5.1) was compiled in 
August on gcc-4.8.5. The current version xen-4.5.2 was compiled 
yesterday and obviously used gcc-4.9.3.
I might try to re-compile with 4.8.5 again in case that makes any sense 
(gcc-4.8.5 is still installed in another slot on my system), but will 
first reply to Andrew as well.
>
> Are you observing this on more than one kind of system?
Unfortunately this system is currently my only XEN system and it was 
rock-solid over the last couple of years with virtually no down-time 
once it was up and running with various XEN and dom0 kernel versions.
>
> Jan
Many thanks Atom