* [Bug 1876678] [NEW] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve
@ 2020-05-04 8:48 John Hartley
2020-05-05 6:49 ` [Bug 1876678] Re: Ubuntu 20.04 KVM / " John Hartley
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: John Hartley @ 2020-05-04 8:48 UTC (permalink / raw)
To: qemu-devel
Public bug reported:
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
Pausing with "Emulation Failure"
TESTING:
My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>
<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>
Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:
Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>
<<LIBVIRT CONFIG SNIPPET>>
...
...
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>
Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:
<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64
# grep Features /var/run/dmesg.boot
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>
On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..
<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>
Result is that Layer 1 - FreeBSD Host guest "paused".
To Layer 1 machines freezes I cannot get any further diagnostics from
this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host
<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT= 0000000000000000 00000000
IDT= 0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>
I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870
However in this case my VM Layer 1 VM is using OVMF.
NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
** Affects: qemu
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1876678
Title:
Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve
Status in QEMU:
New
Bug description:
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
Pausing with "Emulation Failure"
TESTING:
My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>
<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>
Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:
Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>
<<LIBVIRT CONFIG SNIPPET>>
...
...
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>
Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:
<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64
# grep Features /var/run/dmesg.boot
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>
On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..
<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>
Result is that Layer 1 - FreeBSD Host guest "paused".
To Layer 1 machines freezes I cannot get any further diagnostics from
this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host
<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT= 0000000000000000 00000000
IDT= 0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>
I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870
However in this case my VM Layer 1 VM is using OVMF.
NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1876678/+subscriptions
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
2020-05-04 8:48 [Bug 1876678] [NEW] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve John Hartley
@ 2020-05-05 6:49 ` John Hartley
2020-05-10 2:13 ` John Hartley
2021-05-06 14:18 ` Thomas Huth
2 siblings, 0 replies; 4+ messages in thread
From: John Hartley @ 2020-05-05 6:49 UTC (permalink / raw)
To: qemu-devel
** Summary changed:
- Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve
+ Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1876678
Title:
Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Status in QEMU:
New
Bug description:
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
Pausing with "Emulation Failure"
TESTING:
My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>
<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>
Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:
Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>
<<LIBVIRT CONFIG SNIPPET>>
...
...
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>
Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:
<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64
# grep Features /var/run/dmesg.boot
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>
On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..
<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>
Result is that Layer 1 - FreeBSD Host guest "paused".
To Layer 1 machines freezes I cannot get any further diagnostics from
this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host
<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT= 0000000000000000 00000000
IDT= 0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>
I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870
However in this case my VM Layer 1 VM is using OVMF.
NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1876678/+subscriptions
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
2020-05-04 8:48 [Bug 1876678] [NEW] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve John Hartley
2020-05-05 6:49 ` [Bug 1876678] Re: Ubuntu 20.04 KVM / " John Hartley
@ 2020-05-10 2:13 ` John Hartley
2021-05-06 14:18 ` Thomas Huth
2 siblings, 0 replies; 4+ messages in thread
From: John Hartley @ 2020-05-10 2:13 UTC (permalink / raw)
To: qemu-devel
Hi Ubuntu / KVM Maintainers,
I have now done additional diagnostics on this bug and it appears to be
triggered in nested virtualization case when apic virtualisation is
available in Layer 0 HW and then passed forward to Layer 1 VM via
Libvirt: <cpu mode='host-model' check='partial'> .
Testing found that in case where Layer 1 FreeBSD host had this feature,
see "VID,PostIntr" in "VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr"
from CPU Feature below:
<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>>
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME>
AMD Extended Feature Extensions ID EBX=0x1001000
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
...
<END LAYER 1 - dimes.log>>
In my case with Intel Broadwell chipset this is available, in case of
desktop "core i5-8250U" chip- this reports as:
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
For this case HW case, nested:
Layer 0 - Ubuntu 20.04, Layer 1 - FreeBSD 12.1 with bhyve, Layer 2 - FreeBSD 12.1
Works.
Workaround is to disable APIC virtual interrupt delivery:
1. Add entry into Layer 1 - FreeBSD Guest / Host: /boot/loader.conf:
hw.vmm.vmx.use_apic_vid=0
2. Reboot
3. Check via sysctl that virtual_interupt_delivery is disabled:
# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 <- should be zero
Questions is:
While FreeBSD triggers this bug, is this a KVM issue or a FreeBSD bhyve
one ?
In doing some searching on Web I see that there is already work being
done with KVM 5.6 around APIC virtualisation and its handling. So not
sure if this a potentially know problem:
https://events19.linuxfoundation.org/wp-content/uploads/2017/12
/Improving-KVM-x86-Nested-Virtualization-Liran-Alon-Oracle.pdf
APIC Virtualisation support was introduced back in FreeBSD 11.0 way back
in Sept 2016:
https://www.freebsd.org/releases/11.0R/relnotes.html#hardware-
virtualization
Thanks to Peter Graham on FreeBSD virtualization bug tracker for helping
to find source of problem.
Should this BUG go to KVM / QEMU upstream ?
Cheers,
John Hartley.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1876678
Title:
Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Status in QEMU:
New
Bug description:
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
Pausing with "Emulation Failure"
TESTING:
My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>
<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>
Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:
Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>
<<LIBVIRT CONFIG SNIPPET>>
...
...
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>
Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:
<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64
# grep Features /var/run/dmesg.boot
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>
On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..
<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>
Result is that Layer 1 - FreeBSD Host guest "paused".
To Layer 1 machines freezes I cannot get any further diagnostics from
this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host
<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT= 0000000000000000 00000000
IDT= 0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>
I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870
However in this case my VM Layer 1 VM is using OVMF.
NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1876678/+subscriptions
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
2020-05-04 8:48 [Bug 1876678] [NEW] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve John Hartley
2020-05-05 6:49 ` [Bug 1876678] Re: Ubuntu 20.04 KVM / " John Hartley
2020-05-10 2:13 ` John Hartley
@ 2021-05-06 14:18 ` Thomas Huth
2 siblings, 0 replies; 4+ messages in thread
From: Thomas Huth @ 2021-05-06 14:18 UTC (permalink / raw)
To: qemu-devel
Since you were talking about Ubuntu, I moved this to the Ubuntu tracker
now. If you can reproduce the problem with upstream QEMU (currently
v6.0), then please open a new ticket in the new QEMU issue tracker at
gitlab.com.
** Project changed: qemu => qemu (Ubuntu)
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1876678
Title:
Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Status in qemu package in Ubuntu:
New
Bug description:
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
Pausing with "Emulation Failure"
TESTING:
My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>
<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>
Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:
Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>
<<LIBVIRT CONFIG SNIPPET>>
...
...
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>
Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:
<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64
# grep Features /var/run/dmesg.boot
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>
On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..
<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>
Result is that Layer 1 - FreeBSD Host guest "paused".
To Layer 1 machines freezes I cannot get any further diagnostics from
this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host
<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT= 0000000000000000 00000000
IDT= 0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>
I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870
However in this case my VM Layer 1 VM is using OVMF.
NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678/+subscriptions
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-05-06 14:36 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-04 8:48 [Bug 1876678] [NEW] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve John Hartley
2020-05-05 6:49 ` [Bug 1876678] Re: Ubuntu 20.04 KVM / " John Hartley
2020-05-10 2:13 ` John Hartley
2021-05-06 14:18 ` Thomas Huth
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.