[Xen-devel] XenDom0/FreeBSD: guest crash when nested virtualization is used

* [Xen-devel] XenDom0/FreeBSD: guest crash when nested virtualization is used
@ 2019-07-24 18:02 Oleg Ginzburg
  2019-07-24 18:19 ` Andrew Cooper
  0 siblings, 1 reply; 3+ messages in thread
From: Oleg Ginzburg @ 2019-07-24 18:02 UTC (permalink / raw)
  To: xen-devel

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

Hello maillist,

I use XEN on the FreeBSD platform. Everything worked fine until I
needed to use nested virtualization (for testing purposes).

After some communication with Roger Pau Monné, maintainer of XEN port
in FreeBSD ( https://www.freshports.org/emulators/xen-kernel )  it was
suggested that this might be a common Xen problem and not related to
FreeBSD.

My case:

Host env: FreeBSD 13-HEAD/amd64 r349909
Xen kernel version: 4.12.0_3  ( FreeBSD ports version )
Xen tools version: 4.12.0_2 (FreeBSD ports version )
OS Inside guest vm: Linux Debian 9.9.0 amd64

When I try to run a virtual machine inside a virtual machine ( via
virsh/virt-manager ), the guest simply terminates.

I would appreciate any help and comments.
Please let me know if I can help debug this issue.

attached files:
  dmesg.boot: from /var/run/dmesg.boot of the host
  gdb.txt: messages from gdb session with 'xl create' .. (no any bt)
  messages: from /var/log/messages of the host (syslog output)
  qemu-dm-debian1.log: from /var/log/xen/ directory
  xen.cfg: config file for 'xl create'
  xl.info:  xl info output from the host
  xl-debian.1.log: from /var/log/xen/ directory
  xl-dmesg.txt: output from 'xl dmesg' command after guest crash

[-- Attachment #2: gdb.txt --]
[-- Type: text/plain, Size: 2523 bytes --]

libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/local/etc/xen/scripts/vif-bridge /local/domain/0/backend/vif/15/0 
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x800988dd0: deregister unregistered
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 15:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x800988dd0: deregister unregistered
libxl: debug: libxl_event.c:2200:libxl__ao_progress_report: ao 0x8009390a0: progress report: callback queued aop=0x800999970
libxl: debug: libxl_event.c:1873:libxl__ao_complete: ao 0x8009390a0: complete, rc=0
libxl: debug: libxl_event.c:1408:egc_run_callbacks: ao 0x8009390a0: progress report: callback aop=0x800999970
libxl: debug: libxl_event.c:1842:libxl__ao__destroy: ao 0x8009390a0: destroy
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 15:connected to /var/run/xen/qmp-libxl-15
                                                                                                     libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 15:message type: qmp
                                                                                                                                                                                   libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 15:QEMU version: 3.0.0
                                                 libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 15:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
 libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 15:message type: return
                                                                                  libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 15:next qmp command: '{"execute":"cont","id":2}
'
 libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 15:message type: return
                                                                                  xencall:buffer: debug: total allocations:674 total releases:674
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:658 misses:3 toobig:13
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0
[New Thread 800904a00 (LWP 101338/xl)]

Program exited normally.
(gdb)

[-- Attachment #3: xen.cfg --]
[-- Type: application/octet-stream, Size: 459 bytes --]

name = "debian1"
memory = 4096
vcpus = 2

vif = [ 'bridge=bridge0' ]

boot="c"

stdvga=1
videoram=16

opengl=1
vncconsole=1
vncunused=0

on_poweroff="destroy";
on_reboot="restart";
on_crash="destroy";

hap=1
nestedhvm=1

spice=1
spicehost="0.0.0.0"
spiceport="6000"
spicepasswd="cbsd"
disk = [
'/usr/jails/jails-data/debian1-data/dsk1.vhd,raw,hda,rw', '/usr/jails/src/iso/cbsd-iso-debian-9.9.0-amd64-DVD-1.iso,raw,hdb:cdrom,r'
]

builder = "hvm"
serial='pty'

[-- Attachment #4: dmesg.boot --]
[-- Type: application/octet-stream, Size: 17108 bytes --]

---<<BOOT>>---
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-CURRENT r349909 GENERIC amd64
FreeBSD clang version 8.0.1 (branches/release_80 364487) (based on LLVM 8.0.1)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
XEN: Hypervisor version 4.12 detected.
CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2194.94-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfefa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c2fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,RDSEED,ADX,SMAP>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 4421480448 (4216 MB)
avail memory = 1967239168 (1876 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-47 on motherboard
Launching APs: 1 2 3
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff82c77670, 0) error 6
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8111e330, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000043 [4254] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
hpet0: HPET never increments, disabling
device_attach: hpet0 attach returned 6
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <dasp, performance counters> at device 11.1 (no driver attached)
pci0: <dasp, performance counters> at device 11.2 (no driver attached)
pci0: <dasp, performance counters> at device 16.1 (no driver attached)
pci0: <dasp, performance counters> at device 16.6 (no driver attached)
pci0: <dasp, performance counters> at device 18.1 (no driver attached)
acpi_syscontainer0: <System Container> on acpi0
pcib1: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 26 at device 1.0 on pci1
pci2: <ACPI PCI bus> on pcib2
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xe020-0xe03f mem 0xfb720000-0xfb73ffff,0xfb744000-0xfb747fff irq 26 at device 0.0 on pci2
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: a8:1e:84:93:d3:00
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xe000-0xe01f mem 0xfb700000-0xfb71ffff,0xfb740000-0xfb743fff irq 28 at device 0.1 on pci2
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: a8:1e:84:93:d3:01
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> irq 32 at device 2.0 on pci1
pci3: <ACPI PCI bus> on pcib3
mpr0: <Avago Technologies (LSI) SAS3008> port 0xd000-0xd0ff mem 0xfb640000-0xfb64ffff,0xfb600000-0xfb63ffff irq 32 at device 0.0 on pci3
mpr0: Firmware: 09.00.00.00, Driver: 23.00.00.00-fbsd
mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
pcib4: <ACPI PCI-PCI bridge> irq 40 at device 3.0 on pci1
pci4: <ACPI PCI bus> on pcib4
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xc020-0xc03f mem 0xfb200000-0xfb2fffff,0xfb404000-0xfb407fff irq 40 at device 0.0 on pci4
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 68:05:ca:87:8b:40
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: Error 2 setting up SR-IOV
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xc000-0xc01f mem 0xfb100000-0xfb1fffff,0xfb400000-0xfb403fff irq 44 at device 0.1 on pci4
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: 68:05:ca:87:8b:41
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: Error 2 setting up SR-IOV
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
pci1: <unknown> at device 17.0 (no driver attached)
ahci0: <Intel Wellsburg AHCI SATA controller> port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb818000-0xfb8187ff irq 16 at device 17.4 on pci1
ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pci1: <simple comms> at device 22.0 (no driver attached)
pci1: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Wellsburg USB 2.0 controller> mem 0xfb814000-0xfb8143ff irq 18 at device 26.0 on pci1
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci1
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xfa800000-0xfaffffff,0xfb000000-0xfb01ffff irq 16 at device 0.0 on pci6
vgapci0: Boot video device
ehci1: <Intel Wellsburg USB 2.0 controller> mem 0xfb813000-0xfb8133ff irq 18 at device 29.0 on pci1
usbus1: EHCI version 1.0
usbus1 on ehci1
usbus1: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 31.0 on pci1
isa0: <ISA bus> on isab0
ahci1: <Intel Wellsburg AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb812000-0xfb8127ff irq 16 at device 31.2 on pci1
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
ahcich8: <AHCI channel> at channel 4 on ahci1
ahcich9: <AHCI channel> at channel 5 on ahci1
ahciem1: <AHCI enclosure management bridge> on ahci1
acpi_button0: <Power Button> on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xc0: <Xen Console> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff pnpid PNP0900 on isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
ugen0.1: <Intel EHCI root HUB> at usbus0
ugen1.1: <Intel EHCI root HUB> at usbus1
uhub0 on usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub1 on usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x8087 product 0x8002> at usbus1
uhub2 on uhub1
uhub2: <vendor 0x8087 product 0x8002, class 9/0, rev 2.00/0.05, addr 2> on usbus1
ugen0.2: <vendor 0x8087 product 0x800a> at usbus0
uhub3 on uhub0
uhub3: <vendor 0x8087 product 0x800a, class 9/0, rev 2.00/0.05, addr 2> on usbus0
uhub3: 6 ports with 6 removable, self powered
uhub2: 8 ports with 8 removable, self powered
ugen0.3: <American Megatrends Inc. Virtual Hub> at usbus0
uhub4 on uhub3
uhub4: <7-port Hub> on usbus0
uhub4: 5 ports with 5 removable, self powered
ugen0.4: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0
ukbd0 on uhub4
ukbd0: <Keyboard Interface> on usbus0
kbd2 at ukbd0
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (    )
lock order reversal:
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
 1st 0xfffff80004dcfcd0 CAM device lock (CAM device lock) @ /usr/jails/src/src_13/src/sys/cam/cam_xpt.c:831
 2nd 0xfffffe00006da218 MPR lock (MPR lock) @ /usr/jails/src/src_13/src/sys/cam/cam_xpt.c:2816
stack backtrace:
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
#0 0xffffffff80c35073 at witnesmpr0: At enclosure level 0 and connector name (    )
s_checkorder+0xd33
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
#1 0xffffffff80c34dbd at witness_checkorder+0xa7d
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
#2 0xffffffff80ba9515 at __mtx_lock_flags+0x95
#3 0xffffffff80375662 at xpt_action_default+0x1c2
#4 0xffffffff8038abd4 at nvme_get_identify_ns+0x4104
#5 0xffffffff8037c191 at xpt_register_async+0x19a1
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
#6 0xffffffff80b89044 at fork_exit+0x84
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
#7 0xffffffff8107176e at fork_trampoline+0xe
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (    )
ses0 at mpr0 bus 0 scbus0 target 16 lun 0
ses0: <QUANTA S2B 1U 2.5 1000> Fixed Enclosure Services SPC-3 SCSI device
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
ses1 at ahciem0 bus 0 scbus5 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses2 at ahciem1 bus 0 scbus12 target 0 lun 0
ses2: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses2: SEMB SES Device
ses0: (none) in 'DISK00', SAS Slot: 1+ phys at slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027280
ses0: da1,pass2 in 'DISK01', SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027281
da1 at mpr0 bus 0 scbus0 target 17 lun 0
da1: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number B97807891B5300070060
da1: 1200.000MB/s transfers
da1: Command Queueing enabled
da1: 228936MB (468862128 512 byte sectors)
da3 at mpr0 bus 0 scbus0 target 19 lun 0
da3: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number BA1B07891A5300080861
da3: 1200.000MB/s transfers
da3: Command Queueing enabled
da3: 228936MB (468862128 512 byte sectors)
da2 at mpr0 bus 0 scbus0 target 18 lun 0
da2: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number BA1B07891A5300080852
da2: 1200.000MB/s transfers
da2: Command Queueing enabled
da2: 228936MB (468862128 512 byte sectors)
da0 at mpr0 bus 0 scbus0 target 10 lun 0
da0: <ATA Patriot Burst 61.3> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number B7E0078C1B5300063606
da0: 1200.000MB/s transfers
da0: Command Queueing enabled
da0: 457862MB (937703088 512 byte sectors)
ses0: da2,pass3 in 'DISK02', SAS Slot: 1+ phys at slot 2
da4 at mpr0 bus 0 scbus0 target 20 lun 0
da4: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number BF3307891A5300122021
da4: 1200.000MB/s transfers
da4: Command Queueing enabled
da4: 228936MB (468862128 512 byte sectors)
da8 at mpr0 bus 0 scbus0 target 24 lun 0
da8: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number BF3307891A5300122018
da8: 1200.000MB/s transfers
da8: Command Queueing enabled
da8: 228936MB (468862128 512 byte sectors)
da9 at mpr0 bus 0 scbus0 target 25 lun 0
da9: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number BA1B07891A5300080846
da9: 1200.000MB/s transfers
da9: Command Queueing enabled
da9: 228936MB (468862128 512 byte sectors)
da6 at mpr0 bus 0 scbus0 target 22 lun 0
da6: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number B73607891A5300098424
da6: 1200.000MB/s transfers
da6: Command Queueing enabled
da6: 228936MB (468862128 512 byte sectors)
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027282
da7 at mpr0 bus 0 scbus0 target 23 lun 0
da7: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number BF330789195300120676
da7: 1200.000MB/s transfers
da7: Command Queueing enabled
da7: 228936MB (468862128 512 byte sectors)
da5 at mpr0 bus 0 scbus0 target 21 lun 0
da5: <ATA Patriot Burst 61.2> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number BA1B07891A5300080869
da5: 1200.000MB/s transfers
da5: Command Queueing enabled
da5: 228936MB (468862128 512 byte sectors)
ses0: da3,pass4 in 'DISK03', SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027283
ses0: da4,pass5 in 'DISK04', SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027284
ses0: da5,pass6 in 'DISK05', SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027285
ses0: da6,pass7 in 'DISK06', SAS Slot: 1+ phys at slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027286
ses0: da7,pass8 in 'DISK07', SAS Slot: 1+ phys at slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027287
ses0: da8,pass9 in 'DISK08', SAS Slot: 1+ phys at slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027288
ses0: da9,pass10 in 'DISK09', SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500605b0000272bf addr 500605b000027289
Trying to mount root from zfs:tank/ROOT/default []...
WARNING: WITNESS option enabled, expect reduced performance.
bridge0: Ethernet address: 02:ba:40:e5:ca:00
lo0: link state changed to UP
igb1: promiscuous mode enabled
bridge0: link state changed to UP
igb1: link state changed to UP
ums0 on uhub4
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0

[-- Attachment #5: messages --]
[-- Type: application/octet-stream, Size: 1406 bytes --]

Jul 12 16:55:36 q2u005 kernel: tap0: Ethernet address: 58:9c:fc:00:33:03
Jul 12 16:55:36 q2u005 kernel: tap0: link state changed to UP
Jul 12 16:55:36 q2u005 kernel: tap0: changing name to 'xnb15.0-emu'
Jul 12 16:55:36 q2u005 kernel: xnb(xnb_probe:1129): Claiming device 0, xnb
Jul 12 16:55:36 q2u005 kernel: xnb(xnb_attach:1273): Attaching to backend/vif/15/0
Jul 12 16:55:36 q2u005 kernel: xnb(xnb_frontend_changed:1397): frontend_state=Initialising, xnb_state=InitWait
Jul 12 16:55:36 q2u005 kernel: xnb15.0: 2 link states coalesced
Jul 12 16:55:36 q2u005 kernel: xnb15.0: link state changed to UP
Jul 12 16:55:36 q2u005 kernel: xnb15.0: promiscuous mode enabled
Jul 12 16:55:36 q2u005 kernel: xnb15.0: link state changed to DOWN
Jul 12 16:55:36 q2u005 kernel: xnb15.0: link state changed to UP
Jul 12 16:55:36 q2u005 kernel: xnb15.0-emu: promiscuous mode enabled
Jul 12 16:55:59 q2u005 kernel: xnb15.0-emu: link state changed to DOWN
Jul 12 16:56:00 q2u005 kernel: xnb(xnb_frontend_changed:1397): frontend_state=Connected, xnb_state=InitWait
Jul 12 16:56:00 q2u005 kernel: xnb(xnb_connect_comms:793): rings connected!
Jul 12 16:57:00 q2u005 kernel: xnb(xnb_frontend_changed:1397): frontend_state=Closed, xnb_state=Connected
Jul 12 16:57:00 q2u005 kernel: xnb15.0-emu: promiscuous mode disabled
Jul 12 16:57:00 q2u005 kernel: xnb(xnb_detach:1336): 
Jul 12 16:57:00 q2u005 kernel: xnb(xnb_detach:1345): 

[-- Attachment #6: qemu-dm-debian1.log --]
[-- Type: text/x-log, Size: 60 bytes --]

qemu-system-i386: terminating on signal 1 from pid 295 (xl)

[-- Attachment #7: xl-dmesg.txt --]
[-- Type: text/plain, Size: 3529 bytes --]

(d2) Booting from Hard Disk...
(d2) Booting from 0000:7c00
(XEN) d2v0 VMLAUNCH error: 0x7
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000080050033, shadow=0x0000000060000010, gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=fffffffffffff871
(XEN) CR3 = 0x00000000feffc000
(XEN) RSP = 0x0000000000000000 (0x0000000000000000)  RIP = 0x000000000000fff0 (0x000000000000fff0)
(XEN) RFLAGS=0x00010002 (0x00010002)  DR7 = 0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN)        sel  attr  limit   base
(XEN)   CS: f000 0009b 0000ffff 00000000ffff0000
(XEN)   DS: 0000 00093 0000ffff 0000000000000000
(XEN)   SS: 0000 00093 0000ffff 0000000000000000
(XEN)   ES: 0000 00093 0000ffff 0000000000000000
(XEN)   FS: 0000 00093 0000ffff 0000000000000000
(XEN)   GS: 0000 00093 0000ffff 0000000000000000
(XEN) GDTR:            0000ffff 0000000000000000
(XEN) LDTR: 0000 00082 0000ffff 0000000000000000
(XEN) IDTR:            0000ffff 0000000000000000
(XEN)   TR: 0000 0008b 0000ffff 0000000000000000
(XEN) EFER(VMCS) = 0x0000000000000000  PAT = 0x0000050100070406
(XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000
(XEN) DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000
(XEN) Interruptibility = 00000000  ActivityState = 00000000
(XEN) InterruptStatus = 0000
(XEN) *** Host State ***
(XEN) RIP = 0xffff82d08030f8b0 (vmac.c#vmx_asm_vmexit_handler)  RSP = 0xffff8320259bff70
(XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff8320259c2c00
(XEN) GDTBase=ffff8320259b2000 IDTBase=ffff8320259b6000
(XEN) CR0=0000000080050033 CR3=000000201bc45000 CR4=00000000003526e0
(XEN) Sysenter RSP=ffff8320259bffa0 CS:RIP=e008:ffff82d080354420
(XEN) EFER = 0x0000000000000d01  PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6a1edfa SecondaryExec=000214eb
(XEN) EntryControls=000011ff ExitControls=002fefff
(XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000
(XEN) VMEntry: intr_info=8000030d errcode=00000000 ilen=00000000
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000005
(XEN)         reason=00000030 qualification=0000000000000181
(XEN) IDTVectoring: info=80000b0d errcode=0000f000
(XEN) TSC Offset = 0xfff9d10bc60f4ad6  TSC Multiplier = 0x0000000000000000
(XEN) TPR Threshold = 0x00  PostedIntrVec = 0xf4
(XEN) EPT pointer = 0x000000207dd3b01e  EPTP index = 0x0000
(XEN) PLE Gap=00000080 Window=00001000
(XEN) Virtual processor ID = 0x3540 VMfunc controls = 0000000000000000
(XEN) domain_crash called from vmcs.c:1777
(XEN) Domain 2 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.12.0  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    f000:[<000000000000fff0>]
(XEN) RFLAGS: 0000000000010002   CONTEXT: hvm guest (d2v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) rdx: 00000000000206c1   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000002050
(XEN) cr3: 00000000feffc000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: ffff90e74ec00000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: f000

[-- Attachment #8: xl-debian1.log --]
[-- Type: text/x-log, Size: 8284 bytes --]

libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x80090a3e0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 15:[evg=0x8009b3000] nentries=1 rc=1 15..15
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 15:[evg=0x8009b3000]   got=domaininfos[0] got->domain=15
libxl: debug: libxl_domain.c:804:domain_death_xswatch_callback: Domain 15:Exists shutdown_reported=0 dominf.flags=30106
libxl: debug: libxl_domain.c:816:domain_death_xswatch_callback:  shutdown reporting
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
Domain 15 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is destroy
Domain 15 needs to be cleaned up: destroying the domain
libxl: debug: libxl_domain.c:902:libxl_domain_destroy: Domain 15:ao 0x800939280: create: how=0x0 callback=0x0 poller=0x8009334b0
libxl: debug: libxl_dm.c:3008:libxl__destroy_device_model: Domain 15:Didn't find dm UID; destroying by pid
libxl: debug: libxl_dm.c:2877:kill_device_model: Device Model signaled
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x8009889d0 wpath=/local/domain/0/backend/vbd/15/768/state token=1/2: register slotnum=1
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x8009880d0 wpath=/local/domain/0/backend/vif/15/0/state token=0/3: register slotnum=0
libxl: debug: libxl_domain.c:911:libxl_domain_destroy: Domain 15:ao 0x800939280: inprogress: poller=0x8009334b0, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x8009889d0 wpath=/local/domain/0/backend/vbd/15/768/state token=1/2: event epath=/local/domain/0/backend/vbd/15/768/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vbd/15/768/state wanted state 6 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x8009889d0 wpath=/local/domain/0/backend/vbd/15/768/state token=1/2: deregister slotnum=1
libxl: debug: libxl_device.c:1117:device_backend_callback: Domain 15:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009889d0: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 15:calling hotplug script: /usr/local/etc/xen/scripts/block /local/domain/0/backend/vbd/15/768
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 15:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 15:	remove
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 15:env:
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/local/etc/xen/scripts/block /local/domain/0/backend/vbd/15/768 
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/15/768/state token=1/2: empty slot
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x8009880d0 wpath=/local/domain/0/backend/vif/15/0/state token=0/3: event epath=/local/domain/0/backend/vif/15/0/state
libxl: debug: libxl_event.c:881:devstate_callback: backend /local/domain/0/backend/vif/15/0/state wanted state 6 still waiting state 5
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x8009880d0 wpath=/local/domain/0/backend/vif/15/0/state token=0/3: event epath=/local/domain/0/backend/vif/15/0/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vif/15/0/state wanted state 6 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x8009880d0 wpath=/local/domain/0/backend/vif/15/0/state token=0/3: deregister slotnum=0
libxl: debug: libxl_device.c:1117:device_backend_callback: Domain 15:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009880d0: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 15:calling hotplug script: /usr/local/etc/xen/scripts/vif-bridge /local/domain/0/backend/vif/15/0
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 15:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 15:	remove
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 15:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 15:	iface_dev: xnb15.0
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 15:	emulated: 0
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/local/etc/xen/scripts/vif-bridge /local/domain/0/backend/vif/15/0 
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vif/15/0/state token=0/3: empty slot
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x800988ad0: deregister unregistered
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 15:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x800988ad0: deregister unregistered
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009881d0: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 15:calling hotplug script: /usr/local/etc/xen/scripts/vif-bridge /local/domain/0/backend/vif/15/0
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 15:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 15:	remove
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 15:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 15:	iface_dev: xnb15.0-emu
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 15:	emulated: 1
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /usr/local/etc/xen/scripts/vif-bridge /local/domain/0/backend/vif/15/0 
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009881d0: deregister unregistered
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 15:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009881d0: deregister unregistered
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 15:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009887d0: deregister unregistered
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 15:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x8009884d0: deregister unregistered
libxl: debug: libxl_domain.c:1194:devices_destroy_cb: Domain 15:Forked pid 382 for destroy of domain
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x800933500 wpath=/local/domain/15/device/vbd/832/eject token=2/1: event epath=/local/domain/15/device/vbd/832/eject
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x80090a3e0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 15:[evg=0x8009b3000] nentries=1 rc=0 0..0
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 15:[evg=0x8009b3000]   got=domaininfos[0] got->domain=-1
libxl: debug: libxl_domain.c:724:domain_death_occurred: Domain 15:empty list
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
libxl: debug: libxl_event.c:1873:libxl__ao_complete: ao 0x800939280: complete, rc=0
libxl: debug: libxl_event.c:1842:libxl__ao__destroy: ao 0x800939280: destroy
Done. Exiting now
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x80090a3e0 wpath=@releaseDomain token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x800933500 wpath=/local/domain/15/device/vbd/832/eject token=2/1: deregister slotnum=2
xencall:buffer: debug: total allocations:37 total releases:37
xencall:buffer: debug: current allocations:0 maximum allocations:2
xencall:buffer: debug: cache current size:2
xencall:buffer: debug: cache hits:29 misses:2 toobig:6
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0

[-- Attachment #9: xl.info --]
[-- Type: application/octet-stream, Size: 1456 bytes --]

host                   : q2u005.z.vstack.com
release                : 13.0-CURRENT
version                : FreeBSD 13.0-CURRENT r349909 GENERIC
machine                : amd64
nr_cpus                : 20
max_cpu_id             : 19
nr_nodes               : 1
cores_per_socket       : 10
threads_per_core       : 2
cpu_mhz                : 2194.935
hw_caps                : bfebfbff:76fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
virt_caps              : pv hvm hvm_directio pv_directio
total_memory           : 130951
free_memory            : 127467
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 12
xen_extra              : .0
xen_version            : 4.12.0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit2
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : dom0_mem=2048M dom0_max_vcpus=4 dom0=pvh com1=115200,8n1 guest_loglvl=all loglvl=all vga=keep noreboot
cc_compiler            : FreeBSD clang version 8.0.1 (branches/release_80 364487) (based
cc_compile_by          : root
cc_compile_domain      : 
cc_compile_date        : Thu Jul 11 16:26:43 MSK 2019
build_id               : ad8bed1bf8c86449af541ddc1bb1418cd4244b03
xend_config_format     : 4

[-- Attachment #10: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 3+ messages in thread