[Bug 1876678] Re: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve

[John Hartley <1876678@bugs.launchpad.net>]

Hi Ubuntu / KVM Maintainers,

I have now done additional diagnostics on this bug and it appears to be
triggered in nested virtualization case when apic virtualisation is
available in Layer 0 HW and then passed forward to Layer 1 VM via
Libvirt: <cpu mode='host-model' check='partial'> .

Testing found that in case where Layer 1 FreeBSD host had this feature,
see "VID,PostIntr" in "VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr"
from CPU Feature below:

<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>>
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306d2  Family=0x6  Model=0x3d  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME>
  AMD Extended Feature Extensions ID EBX=0x1001000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
...
<END LAYER 1 - dimes.log>>

In my case with Intel Broadwell chipset this is available, in case of
desktop "core i5-8250U" chip- this reports as:

VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

For this case HW case, nested:
Layer 0 - Ubuntu 20.04, Layer 1 - FreeBSD 12.1 with bhyve, Layer 2 - FreeBSD 12.1
Works.

Workaround is to disable APIC virtual interrupt delivery:

1. Add entry into Layer 1 - FreeBSD Guest / Host: /boot/loader.conf:
hw.vmm.vmx.use_apic_vid=0

2. Reboot

3. Check via sysctl that virtual_interupt_delivery is disabled:
# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0          <- should be zero


Questions is:

While FreeBSD triggers this bug, is this a KVM issue or a FreeBSD bhyve
one ?

In doing some searching on Web I see that there is already work being
done with KVM 5.6 around APIC virtualisation and its handling. So not
sure if this a potentially know problem:
https://events19.linuxfoundation.org/wp-content/uploads/2017/12
/Improving-KVM-x86-Nested-Virtualization-Liran-Alon-Oracle.pdf

APIC Virtualisation support was introduced back in FreeBSD 11.0 way back
in Sept 2016:

https://www.freebsd.org/releases/11.0R/relnotes.html#hardware-
virtualization

Thanks to Peter Graham on FreeBSD virtualization bug tracker for helping
to find source of problem.

Should this BUG go to KVM / QEMU upstream ?

Cheers,

John Hartley.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1876678

Title:
  Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve

Status in QEMU:
  New

Bug description:
  BUG:

  Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on
  Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host
  Pausing with "Emulation Failure"

  TESTING:

  My test scenario is nested virtualisation:
  Layer 0 - Ubuntu 20.04 Host
  Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
  Layer 2 - FreeBSD 12.1 guest

  Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt

  <<START QEMU VERSION>>
  $ virsh -c qemu:///system version --daemon
  Compiled against library: libvirt 6.0.0
  Using library: libvirt 6.0.0
  Using API: QEMU 6.0.0
  Running hypervisor: QEMU 4.2.0
  Running against daemon: 6.0.0
  <<END QEMU VERSION>

  <<START Intel VMX Support & Nesting Enabled>>
  $ cat /proc/cpuinfo | grep -c vmx
  64
  $ cat /sys/module/kvm_intel/parameters/nested
  Y
  <<END Intel VMS>>


  Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:

  Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>

  <<LIBVIRT CONFIG SNIPPET>>
  ...
  ...
    <os>
      <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
      <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
      <nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
    </os>
    <features>
      <acpi/>
      <apic/>
      <vmport state='off'/>
    </features>
    <cpu mode='host-model' check='partial'/>
  ...
  ...
  <END LIBVIRT CONFIG SNIPPET>>

  Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:

  <<LAYER 1 - FreeBSD CPU Features>>
  # uname -a
  FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC  amd64

  # grep Features /var/run/dmesg.boot 
    Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
    Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
    AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
    AMD Features2=0x121<LAHF,ABM,Prefetch>
    Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
    Structured Extended Features2=0x4<UMIP>
    Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
    XSAVE Features=0x1<XSAVEOPT>
  <<END LAYER 1 - FreeBSD CPU Features>

  On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..

  <<START LAYER 2 GUEST START>>
  # ls
  FreeBSD-11.2-RELEASE-amd64-bootonly.iso	FreeBSD-12.1-RELEASE-amd64-dvd1.iso	bee-hd1-01.img
  # /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
  <<END LAYER 2 GUEST START>>

  Result is that Layer 1 - FreeBSD Host guest "paused".

  To Layer 1 machines freezes I cannot get any further diagnostics from
  this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host

  <<LAYER 0 LOG TAIL>>
  char device redirected to /dev/pts/29 (label charserial0)
  2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
  2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
  2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
  2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
  2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
  2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
  2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
  2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
  KVM internal error. Suberror: 1
  emulation failure
  EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
  ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
  EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
  CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
  SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
  DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
  FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
  GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
  LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
  TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
  GDT=     0000000000000000 00000000
  IDT=     0000000000000000 00000000
  CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
  DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
  DR6=00000000ffff0ff0 DR7=0000000000000400
  EFER=0000000000000d01
  Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
  2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
  2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
  <<END LAYER 0 LOG TAIL>>

  
  I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870

  However in this case my VM Layer 1 VM is using OVMF.

  NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
  NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled".  This is known bug in FreeBSD that I have reported separately.
  NOTE 3: I will cross posted this bug report on FreeBSD bugzilla as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
  NOTE 4: Have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts  and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1876678/+subscriptions