[PATCH 0/3] Few nSVM bugfixes

[PATCH 0/3] Few nSVM bugfixes

[Maxim Levitsky]

This patch series contains few nested SVM fixes from
testing I did this weekend.

Patch #1 fixes issue where we were setting the GIF (global interrupt flag)
on first nested VMexit, after migration thus making the nested guest crash
from unexpected interrupts.

Patch #2 is my observation that we never setup nesed msr bitmap on nested
state load after migration.

Patch #3 was 'migrated' ;-) from my other patch series to make it smaller,
which is about more strict checks when we about to return to a nested guest,
from SMM.

Best regards,
	Maxim Levitsky

Maxim Levitsky (3):
  SVM: nSVM: correctly restore GIF on vmexit from nesting after
    migration
  SVM: nSVM: setup nested msr permission bitmap on nested state load
  KVM: nSVM: more strict SMM checks when returning to nested guest

 arch/x86/kvm/svm/nested.c |  7 ++++++-
 arch/x86/kvm/svm/svm.c    | 29 ++++++++++++++++++-----------
 2 files changed, 24 insertions(+), 12 deletions(-)

-- 
2.26.2

[PATCH 1/3] SVM: nSVM: correctly restore GIF on vmexit from nesting after migration

[Maxim Levitsky]

Currently code in svm_set_nested_state copies the current vmcb control
area to L1 control area (hsave->control), under assumption that
it mostly reflects the defaults that kvm choose, and later qemu
overrides  these defaults with L2 state using standard KVM interfaces,
like KVM_SET_REGS.

However nested GIF (which is AMD specific thing) is by default is true,
and it is copied to hsave area as such.

This alone is not a big deal since on VMexit, GIF is always set to false,
regardless of what it was on VM entry.

However in nested_svm_vmexit we were first were setting GIF to false,
but then we overwrite this with value from the hsave area.

Now on normal vm entry this is not a problem, since GIF is false
prior to normal vm entry,
and this is the value that copied to hsave, and then restored,
but this is not always the case when the nested state is loaded as
explained above.

Anyway to fix this issue, move svm_set_gif after we restore the L1 control
state in nested_svm_vmexit, so that even with wrong GIF in the
saved L1 control area, we still clear GIF as the spec says.

All of this is only relevant when GIF virtualization is enabled,
(otherwise nested GIF doesn't reside in the vmcb).

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
---
 arch/x86/kvm/svm/nested.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index fb68467e60496..95fdf068fe4c1 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -586,7 +586,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 	svm->vcpu.arch.mp_state = KVM_MP_STATE_RUNNABLE;
 
 	/* Give the current vmcb to the guest */
-	svm_set_gif(svm, false);
 
 	nested_vmcb->save.es     = vmcb->save.es;
 	nested_vmcb->save.cs     = vmcb->save.cs;
@@ -632,6 +631,9 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 	/* Restore the original control entries */
 	copy_vmcb_control_area(&vmcb->control, &hsave->control);
 
+	/* On vmexit the  GIF is set to false */
+	svm_set_gif(svm, false);
+
 	svm->vmcb->control.tsc_offset = svm->vcpu.arch.tsc_offset =
 		svm->vcpu.arch.l1_tsc_offset;
 
-- 
2.26.2

[PATCH 2/3] SVM: nSVM: setup nested msr permission bitmap on nested state load

[Maxim Levitsky]

This code was missing and was forcing the L2 run with L1's msr
permission bitmap

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 95fdf068fe4c1..e90bc436f5849 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1134,6 +1134,9 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	load_nested_vmcb_control(svm, &ctl);
 	nested_prepare_vmcb_control(svm);
 
+	if (!nested_svm_vmrun_msrpm(svm))
+		return -EINVAL;
+
 out_set_gif:
 	svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
 	return 0;
-- 
2.26.2

[PATCH 3/3] KVM: nSVM: more strict SMM checks when returning to nested guest

[Maxim Levitsky]

* check that guest is 64 bit guest, otherwise the SVM related fields
  in the smm state area are not defined

* If the SMM area indicates that SMM interrupted a running guest,
  check that EFER.SVME which is also saved in this area is set, otherwise
  the guest might have tampered with SMM save area, and so indicate
  emulation failure which should triple fault the guest.

* Check that that guest CPUID supports SVM (due to the same issue as above)

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
---
 arch/x86/kvm/svm/svm.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 03dd7bac80348..0cfb8c08e744e 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3900,21 +3900,28 @@ static int svm_pre_enter_smm(struct kvm_vcpu *vcpu, char *smstate)
 static int svm_pre_leave_smm(struct kvm_vcpu *vcpu, const char *smstate)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
-	struct vmcb *nested_vmcb;
 	struct kvm_host_map map;
-	u64 guest;
-	u64 vmcb;
 	int ret = 0;
 
-	guest = GET_SMSTATE(u64, smstate, 0x7ed8);
-	vmcb = GET_SMSTATE(u64, smstate, 0x7ee0);
+	if (guest_cpuid_has(vcpu, X86_FEATURE_LM)) {
+		u64 saved_efer = GET_SMSTATE(u64, smstate, 0x7ed0);
+		u64 guest = GET_SMSTATE(u64, smstate, 0x7ed8);
+		u64 vmcb = GET_SMSTATE(u64, smstate, 0x7ee0);
 
-	if (guest) {
-		if (kvm_vcpu_map(&svm->vcpu, gpa_to_gfn(vmcb), &map) == -EINVAL)
-			return 1;
-		nested_vmcb = map.hva;
-		ret = enter_svm_guest_mode(svm, vmcb, nested_vmcb);
-		kvm_vcpu_unmap(&svm->vcpu, &map, true);
+		if (guest) {
+			if (!guest_cpuid_has(vcpu, X86_FEATURE_SVM))
+				return 1;
+
+			if (!(saved_efer && EFER_SVME))
+				return 1;
+
+			if (kvm_vcpu_map(&svm->vcpu,
+					 gpa_to_gfn(vmcb), &map) == -EINVAL)
+				return 1;
+
+			ret = enter_svm_guest_mode(svm, vmcb, map.hva);
+			kvm_vcpu_unmap(&svm->vcpu, &map, true);
+		}
 	}
 
 	return ret;
-- 
2.26.2

Re: [PATCH 3/3] KVM: nSVM: more strict SMM checks when returning to nested guest

[Dan Carpenter]

[-- Attachment #1: Type: text/plain, Size: 4048 bytes --]

Hi Maxim,

url:    https://github.com/0day-ci/linux/commits/Maxim-Levitsky/Few-nSVM-bugfixes/20200828-003025
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next
config: x86_64-randconfig-m001-20200827 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>

smatch warnings:
arch/x86/kvm/svm/svm.c:3915 svm_pre_leave_smm() warn: should this be a bitwise op?

# https://github.com/0day-ci/linux/commit/e2317f8eb1f0e9f731ddbe66ab175be19f3bdaf1
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Maxim-Levitsky/Few-nSVM-bugfixes/20200828-003025
git checkout e2317f8eb1f0e9f731ddbe66ab175be19f3bdaf1
vim +3915 arch/x86/kvm/svm/svm.c

ed19321fb657121 arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3900  static int svm_pre_leave_smm(struct kvm_vcpu *vcpu, const char *smstate)
0234bf885236a41 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3901  {
05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3902  	struct vcpu_svm *svm = to_svm(vcpu);
8c5fbf1a7231078 arch/x86/kvm/svm.c     KarimAllah Ahmed    2019-01-31  3903  	struct kvm_host_map map;
59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3904  	int ret = 0;
05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3905  
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3906  	if (guest_cpuid_has(vcpu, X86_FEATURE_LM)) {
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3907  		u64 saved_efer = GET_SMSTATE(u64, smstate, 0x7ed0);
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3908  		u64 guest = GET_SMSTATE(u64, smstate, 0x7ed8);
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3909  		u64 vmcb = GET_SMSTATE(u64, smstate, 0x7ee0);
05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3910  
ed19321fb657121 arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3911  		if (guest) {
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3912  			if (!guest_cpuid_has(vcpu, X86_FEATURE_SVM))
9ec19493fb86d6d arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3913  				return 1;
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3914  
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27 @3915  			if (!(saved_efer && EFER_SVME))
                                                                                                                 ^^
It looks like bitwise AND was intended.

e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3916  				return 1;
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3917  
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3918  			if (kvm_vcpu_map(&svm->vcpu,
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3919  					 gpa_to_gfn(vmcb), &map) == -EINVAL)
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3920  				return 1;
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3921  
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3922  			ret = enter_svm_guest_mode(svm, vmcb, map.hva);
69c9dfa24bb7bac arch/x86/kvm/svm/svm.c Paolo Bonzini       2020-05-13  3923  			kvm_vcpu_unmap(&svm->vcpu, &map, true);
05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3924  		}
e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3925  	}
59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3926  
59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3927  	return ret;
0234bf885236a41 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3928  }

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 43482 bytes --]

Re: [PATCH 3/3] KVM: nSVM: more strict SMM checks when returning to nested guest

[Maxim Levitsky]

On Mon, 2020-08-31 at 15:01 +0300, Dan Carpenter wrote:
> Hi Maxim,
> 
> url:    https://github.com/0day-ci/linux/commits/Maxim-Levitsky/Few-nSVM-bugfixes/20200828-003025
> base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git linux-next
> config: x86_64-randconfig-m001-20200827 (attached as .config)
> compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> Reported-by: Dan Carpenter <dan.carpenter@oracle.com>
> 
> smatch warnings:
> arch/x86/kvm/svm/svm.c:3915 svm_pre_leave_smm() warn: should this be a bitwise op?
> 
> # https://github.com/0day-ci/linux/commit/e2317f8eb1f0e9f731ddbe66ab175be19f3bdaf1
> git remote add linux-review https://github.com/0day-ci/linux
> git fetch --no-tags linux-review Maxim-Levitsky/Few-nSVM-bugfixes/20200828-003025
> git checkout e2317f8eb1f0e9f731ddbe66ab175be19f3bdaf1
> vim +3915 arch/x86/kvm/svm/svm.c
> 
> ed19321fb657121 arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3900  static int svm_pre_leave_smm(struct kvm_vcpu *vcpu, const char *smstate)
> 0234bf885236a41 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3901  {
> 05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3902  	struct vcpu_svm *svm = to_svm(vcpu);
> 8c5fbf1a7231078 arch/x86/kvm/svm.c     KarimAllah Ahmed    2019-01-31  3903  	struct kvm_host_map map;
> 59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3904  	int ret = 0;
> 05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3905  
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3906  	if (guest_cpuid_has(vcpu, X86_FEATURE_LM)) {
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3907  		u64 saved_efer = GET_SMSTATE(u64, smstate, 0x7ed0);
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3908  		u64 guest = GET_SMSTATE(u64, smstate, 0x7ed8);
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3909  		u64 vmcb = GET_SMSTATE(u64, smstate, 0x7ee0);
> 05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3910  
> ed19321fb657121 arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3911  		if (guest) {
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3912  			if (!guest_cpuid_has(vcpu, X86_FEATURE_SVM))
> 9ec19493fb86d6d arch/x86/kvm/svm.c     Sean Christopherson 2019-04-02  3913  				return 1;
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3914  
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27 @3915  			if (!(saved_efer && EFER_SVME))
>                                                                                                                  ^^
> It looks like bitwise AND was intended.

Oops. Thanks!

Best regards,
	Maxim Levitskky
> 
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3916  				return 1;
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3917  
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3918  			if (kvm_vcpu_map(&svm->vcpu,
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3919  					 gpa_to_gfn(vmcb), &map) == -EINVAL)
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3920  				return 1;
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3921  
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3922  			ret = enter_svm_guest_mode(svm, vmcb, map.hva);
> 69c9dfa24bb7bac arch/x86/kvm/svm/svm.c Paolo Bonzini       2020-05-13  3923  			kvm_vcpu_unmap(&svm->vcpu, &map, true);
> 05cade71cf3b925 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3924  		}
> e2317f8eb1f0e9f arch/x86/kvm/svm/svm.c Maxim Levitsky      2020-08-27  3925  	}
> 59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3926  
> 59cd9bc5b03f0ba arch/x86/kvm/svm/svm.c Vitaly Kuznetsov    2020-07-10  3927  	return ret;
> 0234bf885236a41 arch/x86/kvm/svm.c     Ladi Prosek         2017-10-11  3928  }
> 
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

Re: [PATCH 1/3] SVM: nSVM: correctly restore GIF on vmexit from nesting after migration

[Paolo Bonzini]

On 27/08/20 18:27, Maxim Levitsky wrote:
> However in nested_svm_vmexit we were first were setting GIF to false,
> but then we overwrite this with value from the hsave area.

Do you mean we are overwriting the resulting intercepts with values from
the hsave area?  If so, I can rewrite the commit message but the patch
is good!

Paolo

Re: [PATCH 0/3] Few nSVM bugfixes

[Paolo Bonzini]

On 27/08/20 18:27, Maxim Levitsky wrote:
> This patch series contains few nested SVM fixes from
> testing I did this weekend.
> 
> Patch #1 fixes issue where we were setting the GIF (global interrupt flag)
> on first nested VMexit, after migration thus making the nested guest crash
> from unexpected interrupts.
> 
> Patch #2 is my observation that we never setup nesed msr bitmap on nested
> state load after migration.
> 
> Patch #3 was 'migrated' ;-) from my other patch series to make it smaller,
> which is about more strict checks when we about to return to a nested guest,
> from SMM.
> 
> Best regards,
> 	Maxim Levitsky
> 
> Maxim Levitsky (3):
>   SVM: nSVM: correctly restore GIF on vmexit from nesting after
>     migration
>   SVM: nSVM: setup nested msr permission bitmap on nested state load
>   KVM: nSVM: more strict SMM checks when returning to nested guest
> 
>  arch/x86/kvm/svm/nested.c |  7 ++++++-
>  arch/x86/kvm/svm/svm.c    | 29 ++++++++++++++++++-----------
>  2 files changed, 24 insertions(+), 12 deletions(-)
> 

Queued, thanks.

Paolo