[PATCH v2 0/5] KVM: VMX: Optimize VMX instrs error/fault handling

[PATCH v2 0/5] KVM: VMX: Optimize VMX instrs error/fault handling

[Sean Christopherson]

A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with
objtool.  An unfortunate side effect is that JMP is now inserted after
most VMX instructions so that the reboot macro can use an actual CALL to
kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.

Rework the low level VMX instruction helpers to handle unexpected faults
manually instead of relying on the "fault on reboot" macro.  By using
asm-goto, most helpers can branch directly to an in-function call to
kvm_spurious_fault(), which can then be optimized by compilers to reside
out-of-line at the end of the function instead of inline as done by
"fault on reboot".

The net impact relative to the current code base is more or less a nop
when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__.
A bunch of code that was previously in .fixup gets moved into the slow
paths of functions, but the fast paths are more basically unchanged.

Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net
positive as CC_SET() without compiler support almost always generates a
SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.

A small bonus is that the Jcc instrs are hinted to predict that the VMX
instr will be successful.

v2:
  - Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
  - Reworded changelogs to reference the commit instead lkml link for
    the recent changes to __kvm_handle_fault_on_reboot().
  - Added Paolo's acks for patch 1-4
  - Added patch 5 to do more cleanup, which was made possible by rebasing
    on top of the __kvm_handle_fault_on_reboot() changes.
  
Sean Christopherson (5):
  objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
  KVM: VMX: Optimize VMX instruction error and fault handling
  KVM: VMX: Add error handling to VMREAD helper
  KVM: x86: Drop ____kvm_handle_fault_on_reboot()
  KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

 arch/x86/include/asm/kvm_host.h | 16 ++----
 arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
 arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
 arch/x86/kvm/x86.c              |  3 +-
 tools/objtool/check.c           |  1 -
 5 files changed, 104 insertions(+), 51 deletions(-)

-- 
2.22.0

[PATCH v2 1/5] objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()

[Sean Christopherson]

Explicitly check kvm_reboot in kvm_spurious_fault() prior to invoking
BUG(), as opposed to assuming the caller has already done so.  Letting
kvm_spurious_fault() be called "directly" will allow VMX to better
optimize its low level assembly flows.

As a happy side effect, kvm_spurious_fault() no longer needs to be
marked as a dead end since it doesn't unconditionally BUG().

Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/include/asm/kvm_host.h | 2 +-
 arch/x86/kvm/x86.c              | 3 ++-
 tools/objtool/check.c           | 1 -
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 8282b8d41209..9739ed615faf 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1496,7 +1496,7 @@ enum {
 #define kvm_arch_vcpu_memslots_id(vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)
 #define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, (role).smm)
 
-asmlinkage void __noreturn kvm_spurious_fault(void);
+asmlinkage void kvm_spurious_fault(void);
 
 /*
  * Hardware virtualization extension instructions may fault if a
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4a0b74ecd1de..6bc012afb86a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -356,7 +356,8 @@ EXPORT_SYMBOL_GPL(kvm_set_apic_base);
 asmlinkage __visible void kvm_spurious_fault(void)
 {
 	/* Fault while not rebooting.  We want the trace. */
-	BUG();
+	if (!kvm_rebooting)
+		BUG();
 }
 EXPORT_SYMBOL_GPL(kvm_spurious_fault);
 
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 5f26620f13f5..688a9af8124d 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -138,7 +138,6 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func,
 		"do_task_dead",
 		"__module_put_and_exit",
 		"complete_and_exit",
-		"kvm_spurious_fault",
 		"__reiserfs_panic",
 		"lbug_with_loc",
 		"fortify_panic",
-- 
2.22.0

[PATCH v2 2/5] KVM: VMX: Optimize VMX instruction error and fault handling

[Sean Christopherson]

Rework the VMX instruction helpers using asm-goto to branch directly
to error/fault "handlers" in lieu of using __ex(), i.e. the generic
____kvm_handle_fault_on_reboot().  Branching directly to fault handling
code during fixup avoids the extra JMP that is inserted after every VMX
instruction when using the generic "fault on reboot" (see commit
3901336ed9887, "x86/kvm: Don't call kvm_spurious_fault() from .fixup").

Opportunistically clean up the helpers so that they all have consistent
error handling and messages.

Leave the usage of ____kvm_handle_fault_on_reboot() (via __ex()) in
kvm_cpu_vmxoff() and nested_vmx_check_vmentry_hw() as is.  The VMXOFF
case is not a fast path, i.e. the cleanliness of __ex() is worth the
JMP, and the extra JMP in nested_vmx_check_vmentry_hw() is unavoidable.

Note, VMREAD cannot get the asm-goto treatment as output operands aren't
compatible with GCC's asm-goto due to internal compiler restrictions.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/kvm/vmx/ops.h | 72 +++++++++++++++++++++++-------------------
 arch/x86/kvm/vmx/vmx.c | 34 ++++++++++++++++++++
 2 files changed, 74 insertions(+), 32 deletions(-)

diff --git a/arch/x86/kvm/vmx/ops.h b/arch/x86/kvm/vmx/ops.h
index 2200fb698dd0..79e25d49d4d9 100644
--- a/arch/x86/kvm/vmx/ops.h
+++ b/arch/x86/kvm/vmx/ops.h
@@ -14,6 +14,12 @@
 #define __ex_clear(x, reg) \
 	____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg)
 
+void vmwrite_error(unsigned long field, unsigned long value);
+void vmclear_error(struct vmcs *vmcs, u64 phys_addr);
+void vmptrld_error(struct vmcs *vmcs, u64 phys_addr);
+void invvpid_error(unsigned long ext, u16 vpid, gva_t gva);
+void invept_error(unsigned long ext, u64 eptp, gpa_t gpa);
+
 static __always_inline void vmcs_check16(unsigned long field)
 {
 	BUILD_BUG_ON_MSG(__builtin_constant_p(field) && ((field) & 0x6001) == 0x2000,
@@ -103,21 +109,39 @@ static __always_inline unsigned long vmcs_readl(unsigned long field)
 	return __vmcs_readl(field);
 }
 
-static noinline void vmwrite_error(unsigned long field, unsigned long value)
-{
-	printk(KERN_ERR "vmwrite error: reg %lx value %lx (err %d)\n",
-	       field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
-	dump_stack();
-}
+#define vmx_asm1(insn, op1, error_args...)				\
+do {									\
+	asm_volatile_goto("1: " __stringify(insn) " %0\n\t"		\
+			  ".byte 0x2e\n\t" /* branch not taken hint */	\
+			  "jna %l[error]\n\t"				\
+			  _ASM_EXTABLE(1b, %l[fault])			\
+			  : : op1 : "cc" : error, fault);		\
+	return;								\
+error:									\
+	insn##_error(error_args);					\
+	return;								\
+fault:									\
+	kvm_spurious_fault();						\
+} while (0)
+
+#define vmx_asm2(insn, op1, op2, error_args...)				\
+do {									\
+	asm_volatile_goto("1: "  __stringify(insn) " %1, %0\n\t"	\
+			  ".byte 0x2e\n\t" /* branch not taken hint */	\
+			  "jna %l[error]\n\t"				\
+			  _ASM_EXTABLE(1b, %l[fault])			\
+			  : : op1, op2 : "cc" : error, fault);		\
+	return;								\
+error:									\
+	insn##_error(error_args);					\
+	return;								\
+fault:									\
+	kvm_spurious_fault();						\
+} while (0)
 
 static __always_inline void __vmcs_writel(unsigned long field, unsigned long value)
 {
-	bool error;
-
-	asm volatile (__ex("vmwrite %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(field), "rm"(value));
-	if (unlikely(error))
-		vmwrite_error(field, value);
+	vmx_asm2(vmwrite, "r"(field), "rm"(value), field, value);
 }
 
 static __always_inline void vmcs_write16(unsigned long field, u16 value)
@@ -182,28 +206,18 @@ static __always_inline void vmcs_set_bits(unsigned long field, u32 mask)
 static inline void vmcs_clear(struct vmcs *vmcs)
 {
 	u64 phys_addr = __pa(vmcs);
-	bool error;
 
-	asm volatile (__ex("vmclear %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "m"(phys_addr));
-	if (unlikely(error))
-		printk(KERN_ERR "kvm: vmclear fail: %p/%llx\n",
-		       vmcs, phys_addr);
+	vmx_asm1(vmclear, "m"(phys_addr), vmcs, phys_addr);
 }
 
 static inline void vmcs_load(struct vmcs *vmcs)
 {
 	u64 phys_addr = __pa(vmcs);
-	bool error;
 
 	if (static_branch_unlikely(&enable_evmcs))
 		return evmcs_load(phys_addr);
 
-	asm volatile (__ex("vmptrld %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "m"(phys_addr));
-	if (unlikely(error))
-		printk(KERN_ERR "kvm: vmptrld %p/%llx failed\n",
-		       vmcs, phys_addr);
+	vmx_asm1(vmptrld, "m"(phys_addr), vmcs, phys_addr);
 }
 
 static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva)
@@ -213,11 +227,8 @@ static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva)
 		u64 rsvd : 48;
 		u64 gva;
 	} operand = { vpid, 0, gva };
-	bool error;
 
-	asm volatile (__ex("invvpid %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(ext), "m"(operand));
-	BUG_ON(error);
+	vmx_asm2(invvpid, "r"(ext), "m"(operand), ext, vpid, gva);
 }
 
 static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)
@@ -225,11 +236,8 @@ static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)
 	struct {
 		u64 eptp, gpa;
 	} operand = {eptp, gpa};
-	bool error;
 
-	asm volatile (__ex("invept %2, %1") CC_SET(na)
-		      : CC_OUT(na) (error) : "r"(ext), "m"(operand));
-	BUG_ON(error);
+	vmx_asm2(invept, "r"(ext), "m"(operand), ext, eptp, gpa);
 }
 
 static inline bool vpid_sync_vcpu_addr(int vpid, gva_t addr)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 69536553446d..46689019ebf7 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -343,6 +343,40 @@ static __always_inline void vmx_disable_intercept_for_msr(unsigned long *msr_bit
 
 void vmx_vmexit(void);
 
+#define vmx_insn_failed(fmt...)		\
+do {					\
+	WARN_ONCE(1, fmt);		\
+	pr_warn_ratelimited(fmt);	\
+} while (0)
+
+noinline void vmwrite_error(unsigned long field, unsigned long value)
+{
+	vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%d\n",
+			field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
+}
+
+noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr)
+{
+	vmx_insn_failed("kvm: vmclear failed: %p/%llx\n", vmcs, phys_addr);
+}
+
+noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr)
+{
+	vmx_insn_failed("kvm: vmptrld failed: %p/%llx\n", vmcs, phys_addr);
+}
+
+noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva)
+{
+	vmx_insn_failed("kvm: invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",
+			ext, vpid, gva);
+}
+
+noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa)
+{
+	vmx_insn_failed("kvm: invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",
+			ext, eptp, gpa);
+}
+
 static DEFINE_PER_CPU(struct vmcs *, vmxarea);
 DEFINE_PER_CPU(struct vmcs *, current_vmcs);
 /*
-- 
2.22.0

[PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper

[Sean Christopherson]

Now that VMREAD flows require a taken branch, courtesy of commit

  3901336ed9887 ("x86/kvm: Don't call kvm_spurious_fault() from .fixup")

bite the bullet and add full error handling to VMREAD, i.e. replace the
JMP added by __ex()/____kvm_handle_fault_on_reboot() with a hinted Jcc.

To minimize the code footprint, add a helper function, vmread_error(),
to handle both faults and failures so that the inline flow has a single
CALL.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/kvm/vmx/ops.h | 21 +++++++++++++++++----
 arch/x86/kvm/vmx/vmx.c |  8 ++++++++
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/vmx/ops.h b/arch/x86/kvm/vmx/ops.h
index 79e25d49d4d9..45eaedee2ac0 100644
--- a/arch/x86/kvm/vmx/ops.h
+++ b/arch/x86/kvm/vmx/ops.h
@@ -11,9 +11,8 @@
 #include "vmcs.h"
 
 #define __ex(x) __kvm_handle_fault_on_reboot(x)
-#define __ex_clear(x, reg) \
-	____kvm_handle_fault_on_reboot(x, "xor " reg ", " reg)
 
+asmlinkage void vmread_error(unsigned long field, bool fault);
 void vmwrite_error(unsigned long field, unsigned long value);
 void vmclear_error(struct vmcs *vmcs, u64 phys_addr);
 void vmptrld_error(struct vmcs *vmcs, u64 phys_addr);
@@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
 {
 	unsigned long value;
 
-	asm volatile (__ex_clear("vmread %1, %0", "%k0")
-		      : "=r"(value) : "r"(field));
+	asm volatile("1: vmread %2, %1\n\t"
+		     ".byte 0x3e\n\t" /* branch taken hint */
+		     "ja 3f\n\t"
+		     "mov %2, %%" _ASM_ARG1 "\n\t"
+		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
+		     "2: call vmread_error\n\t"
+		     "xor %k1, %k1\n\t"
+		     "3:\n\t"
+
+		     ".pushsection .fixup, \"ax\"\n\t"
+		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
+		     "mov $1, %%" _ASM_ARG2 "\n\t"
+		     "jmp 2b\n\t"
+		     ".popsection\n\t"
+		     _ASM_EXTABLE(1b, 4b)
+		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");
 	return value;
 }
 
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 46689019ebf7..9acf3d6395d3 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -349,6 +349,14 @@ do {					\
 	pr_warn_ratelimited(fmt);	\
 } while (0)
 
+asmlinkage void vmread_error(unsigned long field, bool fault)
+{
+	if (fault)
+		kvm_spurious_fault();
+	else
+		vmx_insn_failed("kvm: vmread failed: field=%lx\n", field);
+}
+
 noinline void vmwrite_error(unsigned long field, unsigned long value)
 {
 	vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%d\n",
-- 
2.22.0

[PATCH v2 4/5] KVM: x86: Drop ____kvm_handle_fault_on_reboot()

[Sean Christopherson]

Remove the variation of __kvm_handle_fault_on_reboot() that accepts a
post-fault cleanup instruction now that its sole user (VMREAD) uses
a different method for handling faults.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/include/asm/kvm_host.h | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 9739ed615faf..92c59cd923b6 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1504,7 +1504,7 @@ asmlinkage void kvm_spurious_fault(void);
  * Usually after catching the fault we just panic; during reboot
  * instead the instruction is ignored.
  */
-#define ____kvm_handle_fault_on_reboot(insn, cleanup_insn)		\
+#define __kvm_handle_fault_on_reboot(insn)				\
 	"666: \n\t"							\
 	insn "\n\t"							\
 	"jmp	668f \n\t"						\
@@ -1513,16 +1513,12 @@ asmlinkage void kvm_spurious_fault(void);
 	"668: \n\t"							\
 	".pushsection .fixup, \"ax\" \n\t"				\
 	"700: \n\t"							\
-	cleanup_insn "\n\t"						\
 	"cmpb	$0, kvm_rebooting\n\t"					\
 	"je	667b \n\t"						\
 	"jmp	668b \n\t"						\
 	".popsection \n\t"						\
 	_ASM_EXTABLE(666b, 700b)
 
-#define __kvm_handle_fault_on_reboot(insn)		\
-	____kvm_handle_fault_on_reboot(insn, "")
-
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
 int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);
-- 
2.22.0

[PATCH v2 5/5] KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

[Sean Christopherson]

Remove the kvm_rebooting check from VMX/SVM instruction exception fixup
now that kvm_spurious_fault() conditions its BUG() on !kvm_rebooting.
Because the 'cleanup_insn' functionally is also gone, deferring to
kvm_spurious_fault() means __kvm_handle_fault_on_reboot() can eliminate
its .fixup code entirely and have its exception table entry branch
directly to the call to kvm_spurious_fault().

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/include/asm/kvm_host.h | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 92c59cd923b6..a5ae5562ce0a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);
 	"667: \n\t"							\
 	"call	kvm_spurious_fault \n\t"				\
 	"668: \n\t"							\
-	".pushsection .fixup, \"ax\" \n\t"				\
-	"700: \n\t"							\
-	"cmpb	$0, kvm_rebooting\n\t"					\
-	"je	667b \n\t"						\
-	"jmp	668b \n\t"						\
-	".popsection \n\t"						\
-	_ASM_EXTABLE(666b, 700b)
+	_ASM_EXTABLE(666b, 667b)
 
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
-- 
2.22.0

Re: [PATCH v2 5/5] KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()

[Paolo Bonzini]

On 19/07/19 22:41, Sean Christopherson wrote:
> Remove the kvm_rebooting check from VMX/SVM instruction exception fixup
> now that kvm_spurious_fault() conditions its BUG() on !kvm_rebooting.
> Because the 'cleanup_insn' functionally is also gone, deferring to
> kvm_spurious_fault() means __kvm_handle_fault_on_reboot() can eliminate
> its .fixup code entirely and have its exception table entry branch
> directly to the call to kvm_spurious_fault().
> 
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> ---
>  arch/x86/include/asm/kvm_host.h | 8 +-------
>  1 file changed, 1 insertion(+), 7 deletions(-)
> 
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 92c59cd923b6..a5ae5562ce0a 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1511,13 +1511,7 @@ asmlinkage void kvm_spurious_fault(void);
>  	"667: \n\t"							\
>  	"call	kvm_spurious_fault \n\t"				\
>  	"668: \n\t"							\
> -	".pushsection .fixup, \"ax\" \n\t"				\
> -	"700: \n\t"							\
> -	"cmpb	$0, kvm_rebooting\n\t"					\
> -	"je	667b \n\t"						\
> -	"jmp	668b \n\t"						\
> -	".popsection \n\t"						\
> -	_ASM_EXTABLE(666b, 700b)
> +	_ASM_EXTABLE(666b, 667b)
>  
>  #define KVM_ARCH_WANT_MMU_NOTIFIER
>  int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end);
> 

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper

[Josh Poimboeuf]

On Fri, Jul 19, 2019 at 01:41:08PM -0700, Sean Christopherson wrote:
> @@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
>  {
>  	unsigned long value;
>  
> -	asm volatile (__ex_clear("vmread %1, %0", "%k0")
> -		      : "=r"(value) : "r"(field));
> +	asm volatile("1: vmread %2, %1\n\t"
> +		     ".byte 0x3e\n\t" /* branch taken hint */
> +		     "ja 3f\n\t"
> +		     "mov %2, %%" _ASM_ARG1 "\n\t"
> +		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
> +		     "2: call vmread_error\n\t"
> +		     "xor %k1, %k1\n\t"
> +		     "3:\n\t"
> +
> +		     ".pushsection .fixup, \"ax\"\n\t"
> +		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
> +		     "mov $1, %%" _ASM_ARG2 "\n\t"
> +		     "jmp 2b\n\t"
> +		     ".popsection\n\t"
> +		     _ASM_EXTABLE(1b, 4b)
> +		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");

Was there a reason you didn't do the asm goto thing here like you did
for the previous patch?  That seemed cleaner, and needs less asm.  

I think the branch hints aren't needed -- they're ignored on modern
processors.  Ditto for the previous patch.

Also please use named asm operands whereever you can, like "%[field]"
instead of "%2".  It helps a lot with readability.

-- 
Josh

Re: [PATCH v2 3/5] KVM: VMX: Add error handling to VMREAD helper

[Paolo Bonzini]

On 28/07/19 21:36, Josh Poimboeuf wrote:
> On Fri, Jul 19, 2019 at 01:41:08PM -0700, Sean Christopherson wrote:
>> @@ -68,8 +67,22 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field)
>>  {
>>  	unsigned long value;
>>  
>> -	asm volatile (__ex_clear("vmread %1, %0", "%k0")
>> -		      : "=r"(value) : "r"(field));
>> +	asm volatile("1: vmread %2, %1\n\t"
>> +		     ".byte 0x3e\n\t" /* branch taken hint */
>> +		     "ja 3f\n\t"
>> +		     "mov %2, %%" _ASM_ARG1 "\n\t"
>> +		     "xor %%" _ASM_ARG2 ", %%" _ASM_ARG2 "\n\t"
>> +		     "2: call vmread_error\n\t"
>> +		     "xor %k1, %k1\n\t"
>> +		     "3:\n\t"
>> +
>> +		     ".pushsection .fixup, \"ax\"\n\t"
>> +		     "4: mov %2, %%" _ASM_ARG1 "\n\t"
>> +		     "mov $1, %%" _ASM_ARG2 "\n\t"
>> +		     "jmp 2b\n\t"
>> +		     ".popsection\n\t"
>> +		     _ASM_EXTABLE(1b, 4b)
>> +		     : ASM_CALL_CONSTRAINT, "=r"(value) : "r"(field) : "cc");
> 
> Was there a reason you didn't do the asm goto thing here like you did
> for the previous patch?  That seemed cleaner, and needs less asm.  

It's because asm goto doesn't support outputs.

Paolo

> I think the branch hints aren't needed -- they're ignored on modern
> processors.  Ditto for the previous patch.
> 
> Also please use named asm operands whereever you can, like "%[field]"
> instead of "%2".  It helps a lot with readability.
> 

Re: [PATCH v2 0/5] KVM: VMX: Optimize VMX instrs error/fault handling

[Sean Christopherson]

Paolo,

Any chance this can be picked up for 5.4?  Josh's kvm_spurious_fault()
changes went into 5.3, so this can be taken through the KVM tree, which
probably makes the most sense since the non-KVM change is a one line
deletion.

On Fri, Jul 19, 2019 at 01:41:05PM -0700, Sean Christopherson wrote:
> A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with
> objtool.  An unfortunate side effect is that JMP is now inserted after
> most VMX instructions so that the reboot macro can use an actual CALL to
> kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.
> 
> Rework the low level VMX instruction helpers to handle unexpected faults
> manually instead of relying on the "fault on reboot" macro.  By using
> asm-goto, most helpers can branch directly to an in-function call to
> kvm_spurious_fault(), which can then be optimized by compilers to reside
> out-of-line at the end of the function instead of inline as done by
> "fault on reboot".
> 
> The net impact relative to the current code base is more or less a nop
> when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__.
> A bunch of code that was previously in .fixup gets moved into the slow
> paths of functions, but the fast paths are more basically unchanged.
> 
> Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net
> positive as CC_SET() without compiler support almost always generates a
> SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.
> 
> A small bonus is that the Jcc instrs are hinted to predict that the VMX
> instr will be successful.
> 
> v2:
>   - Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
>   - Reworded changelogs to reference the commit instead lkml link for
>     the recent changes to __kvm_handle_fault_on_reboot().
>   - Added Paolo's acks for patch 1-4
>   - Added patch 5 to do more cleanup, which was made possible by rebasing
>     on top of the __kvm_handle_fault_on_reboot() changes.
>   
> Sean Christopherson (5):
>   objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
>   KVM: VMX: Optimize VMX instruction error and fault handling
>   KVM: VMX: Add error handling to VMREAD helper
>   KVM: x86: Drop ____kvm_handle_fault_on_reboot()
>   KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()
> 
>  arch/x86/include/asm/kvm_host.h | 16 ++----
>  arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
>  arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
>  arch/x86/kvm/x86.c              |  3 +-
>  tools/objtool/check.c           |  1 -
>  5 files changed, 104 insertions(+), 51 deletions(-)
> 
> -- 
> 2.22.0
> 

Re: [PATCH v2 0/5] KVM: VMX: Optimize VMX instrs error/fault handling

[Paolo Bonzini]

On 19/07/19 22:41, Sean Christopherson wrote:
> A recent commit reworked __kvm_handle_fault_on_reboot() to play nice with
> objtool.  An unfortunate side effect is that JMP is now inserted after
> most VMX instructions so that the reboot macro can use an actual CALL to
> kvm_spurious_fault() instead of a funky PUSH+JMP facsimile in .fixup.
> 
> Rework the low level VMX instruction helpers to handle unexpected faults
> manually instead of relying on the "fault on reboot" macro.  By using
> asm-goto, most helpers can branch directly to an in-function call to
> kvm_spurious_fault(), which can then be optimized by compilers to reside
> out-of-line at the end of the function instead of inline as done by
> "fault on reboot".
> 
> The net impact relative to the current code base is more or less a nop
> when building with a compiler that supports __GCC_ASM_FLAG_OUTPUTS__.
> A bunch of code that was previously in .fixup gets moved into the slow
> paths of functions, but the fast paths are more basically unchanged.
> 
> Without __GCC_ASM_FLAG_OUTPUTS__, manually coding the Jcc is a net
> positive as CC_SET() without compiler support almost always generates a
> SETcc+CMP+Jcc sequence, which is now replaced with a single Jcc.
> 
> A small bonus is that the Jcc instrs are hinted to predict that the VMX
> instr will be successful.
> 
> v2:
>   - Rebased to x86/master, commit eceffd88ca20 ("Merge branch 'x86/urgent'")
>   - Reworded changelogs to reference the commit instead lkml link for
>     the recent changes to __kvm_handle_fault_on_reboot().
>   - Added Paolo's acks for patch 1-4
>   - Added patch 5 to do more cleanup, which was made possible by rebasing
>     on top of the __kvm_handle_fault_on_reboot() changes.
>   
> Sean Christopherson (5):
>   objtool: KVM: x86: Check kvm_rebooting in kvm_spurious_fault()
>   KVM: VMX: Optimize VMX instruction error and fault handling
>   KVM: VMX: Add error handling to VMREAD helper
>   KVM: x86: Drop ____kvm_handle_fault_on_reboot()
>   KVM: x86: Don't check kvm_rebooting in __kvm_handle_fault_on_reboot()
> 
>  arch/x86/include/asm/kvm_host.h | 16 ++----
>  arch/x86/kvm/vmx/ops.h          | 93 ++++++++++++++++++++-------------
>  arch/x86/kvm/vmx/vmx.c          | 42 +++++++++++++++
>  arch/x86/kvm/x86.c              |  3 +-
>  tools/objtool/check.c           |  1 -
>  5 files changed, 104 insertions(+), 51 deletions(-)
> 

Queued, thanks.

Paolo