On 02/10/2019 17:50, Chris Brannon wrote:
Jan Beulich <jbeulich@suse.com> writes:


On 21.09.2019 01:14, Sarah Newman wrote:

With nestedhvm=1, the L2 HVM guest is either hanging (Xen 4.8) or
crashing (Xen 4.12.1) the L1 Xen hypervisor with recent versions of
Linux. We
isolated the commit to:

commit 093ae8f9a86a974c920b613860f1f7fd5bbd70ab
Author: Borislav Petkov <bp@suse.de>
Date:   Thu Apr 12 13:11:36 2018 +0200

Thanks for doing the bisection on the Linux side. Would you please also
provide the actual data about the L1 Xen crash (i.e. a maximum
verbosity L1 log of a debug build of Xen)? It might also turn out
helpful if you could provide the corresponding L0 Xen log, assuming you
have host access.

Typescripts are attached.
typescript-l1.txt contains the log of domain creation on the L1.
typescript-l0.txt contains both the log of domain creation on the L0,
and xl dmesg output from the L0.

Both the L0 and L1 are running debug builds of Xen 4.12.1.

-- Chris

(XEN) traps.c:1576: GPF (0000): ffff82d08031a80f [vmx.c#vmx_msr_read_intercept+0x387/0x3fd] -> ffff82d08037c9f2
(XEN) traps.c:1576: GPF (0000): ffff82d08031a80f [vmx.c#vmx_msr_read_intercept+0x387/0x3fd] -> ffff82d08037c9f2
(d2) xs_write(/vm/95f11fc0-b9e7-47ff-8523-bacab35b96b6/rtc/timeoffset): EACCES
(XEN) d1v1 Unhandled nested vmexit: reason 51
(XEN) domain_crash called from vvmx.c:2671
(XEN) Domain 1 (vcpu#1) crashed on cpu#2:

51 is the RDTSCP intercept.  It seems that noone has ever tried executing a RDTSCP instruction in nested virt yet... which goes to show how untested this all is.

Can you see about giving this patch a spin?

~Andrew

diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
index fdf449bfd1..6696bd6240 100644
--- a/xen/arch/x86/hvm/vmx/vvmx.c
+++ b/xen/arch/x86/hvm/vmx/vvmx.c
@@ -2491,6 +2491,7 @@ int nvmx_n2_vmexit_handler(struct cpu_user_regs *regs,
             nvcpu->nv_vmexit_pending = 1;
         break;
     case EXIT_REASON_RDTSC:
+    case EXIT_REASON_RDTSCP:
         ctrl = __n2_exec_control(v);
         if ( ctrl & CPU_BASED_RDTSC_EXITING )
             nvcpu->nv_vmexit_pending = 1;
@@ -2501,6 +2502,8 @@ int nvmx_n2_vmexit_handler(struct cpu_user_regs *regs,
              * avoiding changing guest_tsc and messing up timekeeping in L1
              */
             msr_split(regs, hvm_get_guest_tsc(v) + get_vvmcs(v, TSC_OFFSET));
+            if ( exit_reason == EXIT_REASON_RDTSCP )
+                regs->rcx = v->arch.msrs->tsc_aux;
             update_guest_eip();
 
             return 1;