[PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Paolo Bonzini]

32-bit KVM has extra complications in the code due to:

- different ways to write 64-bit values in VMCS

- different handling of DS and ES selectors as well as FS/GS bases

- lack of CR8 and EFER

- lack of XFD

- impossibility of writing 64-bit PTEs atomically

The last is the big one, because it prevents from using the TDP MMU
unconditionally.

Since 32-bit processors with virtualization extensions are a historical
curiosity, and even 32-bit userspace can only deal with small guests
due to limited address space, it makes sense to restrict KVM to 64-bit
hosts and kernels.

In preparation for that make 32-bit KVM depend on CONFIG_BROKEN, and
opportunistically rephrase the help message---processors with
virtualization extensions are supported even if they are not
"fairly recent".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/Kconfig | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index e3cbd7706136..a107df22ffee 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -21,6 +21,7 @@ if VIRTUALIZATION
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
 	depends on HAVE_KVM
+	depends on X86_64 || BROKEN
 	depends on HIGH_RES_TIMERS
 	depends on X86_LOCAL_APIC
 	select PREEMPT_NOTIFIERS
@@ -50,12 +51,13 @@ config KVM
 	select HAVE_KVM_PM_NOTIFIER if PM
 	help
 	  Support hosting fully virtualized guest machines using hardware
-	  virtualization extensions.  You will need a fairly recent
-	  processor equipped with virtualization extensions. You will also
-	  need to select one or more of the processor modules below.
+	  virtualization extensions. You will also need to select one or
+	  more of the processor modules below.
 
 	  This module provides access to the hardware capabilities through
-	  a character device node named /dev/kvm.
+	  a character device node named /dev/kvm.  You might have to enable
+	  virtualization extensions in the firmware setup if the device is
+	  not visible.
 
 	  To compile this as a module, choose M here: the module
 	  will be called kvm.
-- 
2.31.1

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Sean Christopherson]

On Mon, Sep 26, 2022, Paolo Bonzini wrote:
> 32-bit KVM has extra complications in the code due to:
> 
> - different ways to write 64-bit values in VMCS
> 
> - different handling of DS and ES selectors as well as FS/GS bases
> 
> - lack of CR8 and EFER
> 
> - lack of XFD
> 

More for the list:

  - SVM is effectively restricted to PAE kernels due to NX requirements

> - impossibility of writing 64-bit PTEs atomically

It's not impossible, just ugly.  KVM could use CMPXCHG8B to do all of the accesses
for the TDP MMU, including the non-atomic reads and writes.

> The last is the big one, because it prevents from using the TDP MMU
> unconditionally.

As above, if the TDP MMU really is the sticking point, that's solvable.

The real justification for deprecating 32-bit KVM is that, outside of KVM developers,
literally no one uses 32-bit KVM.  I.e. any amount of effort that is required to
continue supporting 32-bit kernels is a complete waste of resources.

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Maxim Levitsky]

On Tue, 2022-09-27 at 17:07 +0000, Sean Christopherson wrote:
> On Mon, Sep 26, 2022, Paolo Bonzini wrote:
> > 32-bit KVM has extra complications in the code due to:
> > 
> > - different ways to write 64-bit values in VMCS
> > 
> > - different handling of DS and ES selectors as well as FS/GS bases
> > 
> > - lack of CR8 and EFER
> > 
> > - lack of XFD
> > 
> 
> More for the list:
> 
>   - SVM is effectively restricted to PAE kernels due to NX requirements
> 
> > - impossibility of writing 64-bit PTEs atomically
> 
> It's not impossible, just ugly.  KVM could use CMPXCHG8B to do all of the accesses
> for the TDP MMU, including the non-atomic reads and writes.
> 
> > The last is the big one, because it prevents from using the TDP MMU
> > unconditionally.
> 
> As above, if the TDP MMU really is the sticking point, that's solvable.
> 
> The real justification for deprecating 32-bit KVM is that, outside of KVM developers,
> literally no one uses 32-bit KVM.  I.e. any amount of effort that is required to
> continue supporting 32-bit kernels is a complete waste of resources.
> 

I also think that outside KVM developers nobody should be using KVM on 32 bit host.

However for _developement_ I think that 32 bit KVM support is very useful, as it
allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
and can even probably be useful to some users (e.g running some legacy stuff in a VM,
which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
other 32 bit nested hypervisors can be useful.

Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
as current kernel works, it is useful to use the same kernel on host and guest.

Best regards,
	Maxim Levitsky

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Paolo Bonzini]

On 9/28/22 09:10, Maxim Levitsky wrote:
> I also think that outside KVM developers nobody should be using KVM on 32 bit host.
> 
> However for_developement_  I think that 32 bit KVM support is very useful, as it
> allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
> and can even probably be useful to some users (e.g running some legacy stuff in a VM,
> which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
> using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
> other 32 bit nested hypervisors can be useful.
> 
> Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
> as current kernel works, it is useful to use the same kernel on host and guest.

Yeah, I would use older 32 bit kernels just like I use RHEL4 to test PIT 
reinjection. :)  But really the ultimate solution to this would be to 
improve kvm-unit-tests so that we can compile vmx.c and svm.c for 32-bit.

Paolo

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Paolo Bonzini]

On 9/27/22 19:07, Sean Christopherson wrote:
> On Mon, Sep 26, 2022, Paolo Bonzini wrote:
>> 32-bit KVM has extra complications in the code due to:
>>
>> - different ways to write 64-bit values in VMCS
>>
>> - different handling of DS and ES selectors as well as FS/GS bases
>>
>> - lack of CR8 and EFER
>>
>> - lack of XFD
>>
> 
> More for the list:
> 
>    - SVM is effectively restricted to PAE kernels due to NX requirements

True, but I'm not sure how it complicates the code?  Do you mean having 
to kmap/kunmap, and if so are you thinking of making KVM depend on !HIGHMEM?

>> - impossibility of writing 64-bit PTEs atomically
> 
> It's not impossible, just ugly.  KVM could use CMPXCHG8B to do all of the accesses
> for the TDP MMU, including the non-atomic reads and writes.

Ok, rephrased:

==========
Breakage in 32-bit KVM is somewhat rare, but it did happen and
developers themselves found out a few months later.  This means that it
is very unlikely that it has any users.  32-bit processors with
virtualization extensions are a historical curiosity; 32-bit userspace
can only deal with small guests due to limited address space.

Hence, it makes sense to restrict x86 KVM to 64-bit hosts and kernels.
This removes all the conditional code to deal with the above
differences, and it also enables unconditional use of the TDP MMU:
making it work on 32-bit systems would require contortions using
CMPXCHG8B to write 64-bit PTEs atomically, and they are simply not
worth it.
==========

>> The last is the big one, because it prevents from using the TDP MMU
>> unconditionally.
> 
> As above, if the TDP MMU really is the sticking point, that's solvable.

Yeah, but until now the maintainability cost of keeping 32-bit on life 
support was small.  Using CMPXCHG8B in the TDP MMU is possibly worse 
than having two MMUs for the two-dimensional paging case, therefore 
dropping the old x86 direct MMU for me is what tips the balance in favor 
of removal.

(Once David submits his MMU callbacks work, I still want to see if it's 
possible to preserve the "old" x86 direct MMU, for example for use in 
pKVM; however, it would not be in arch/x86/).

Paolo

> The real justification for deprecating 32-bit KVM is that, outside of KVM developers,
> literally no one uses 32-bit KVM.  I.e. any amount of effort that is required to
> continue supporting 32-bit kernels is a complete waste of resources.
> 

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Sean Christopherson]

On Wed, Sep 28, 2022, Paolo Bonzini wrote:
> On 9/28/22 09:10, Maxim Levitsky wrote:
> > I also think that outside KVM developers nobody should be using KVM on 32 bit host.
> > 
> > However for_developement_  I think that 32 bit KVM support is very useful, as it
> > allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
> > and can even probably be useful to some users (e.g running some legacy stuff in a VM,
> > which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
> > using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
> > other 32 bit nested hypervisors can be useful.
> > 
> > Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
> > as current kernel works, it is useful to use the same kernel on host and guest.
> 
> Yeah, I would use older 32 bit kernels just like I use RHEL4 to test PIT
> reinjection. :)  But really the ultimate solution to this would be to
> improve kvm-unit-tests so that we can compile vmx.c and svm.c for 32-bit.

Agreed.  I too use 32-bit KVM to validate KVM's handling of 32-bit L1 hypervisors,
but the maintenance cost is painfully high.

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Maxim Levitsky]

On Wed, 2022-09-28 at 16:12 +0000, Sean Christopherson wrote:
> On Wed, Sep 28, 2022, Paolo Bonzini wrote:
> > On 9/28/22 09:10, Maxim Levitsky wrote:
> > > I also think that outside KVM developers nobody should be using KVM on 32 bit host.
> > > 
> > > However for_developement_  I think that 32 bit KVM support is very useful, as it
> > > allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
> > > and can even probably be useful to some users (e.g running some legacy stuff in a VM,
> > > which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
> > > using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
> > > other 32 bit nested hypervisors can be useful.
> > > 
> > > Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
> > > as current kernel works, it is useful to use the same kernel on host and guest.
> > 
> > Yeah, I would use older 32 bit kernels just like I use RHEL4 to test PIT
> > reinjection. :)  But really the ultimate solution to this would be to
> > improve kvm-unit-tests so that we can compile vmx.c and svm.c for 32-bit.
> 
> Agreed.  I too use 32-bit KVM to validate KVM's handling of 32-bit L1 hypervisors,
> but the maintenance cost is painfully high.
> 

But is is actually? I test it routinely and it it does work quite well IMHO.

As far as my opinion goes I do volunteer to test this code more often,
and I do not want to see the 32 bit KVM support be removed *yet*.

Best regards,
	Maxim Levitsky

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Maxim Levitsky]

On Wed, 2022-09-28 at 16:12 +0000, Sean Christopherson wrote:
> On Wed, Sep 28, 2022, Paolo Bonzini wrote:
> > On 9/28/22 09:10, Maxim Levitsky wrote:
> > > I also think that outside KVM developers nobody should be using KVM on 32 bit host.
> > > 
> > > However for_developement_  I think that 32 bit KVM support is very useful, as it
> > > allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
> > > and can even probably be useful to some users (e.g running some legacy stuff in a VM,
> > > which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
> > > using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
> > > other 32 bit nested hypervisors can be useful.
> > > 
> > > Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
> > > as current kernel works, it is useful to use the same kernel on host and guest.
> > 
> > Yeah, I would use older 32 bit kernels just like I use RHEL4 to test PIT
> > reinjection. :)  But really the ultimate solution to this would be to
> > improve kvm-unit-tests so that we can compile vmx.c and svm.c for 32-bit.
> 
> Agreed.  I too use 32-bit KVM to validate KVM's handling of 32-bit L1 hypervisors,
> but the maintenance cost is painfully high.
> 

But is it actually? I test it routinely and it it does work quite well IMHO.
I don't remember that there were that much breakage lately in this area.

As far as my opinion goes I do volunteer to test this code more often,
and I do not want to see the 32 bit KVM support be removed *yet*.

Best regards,
	Maxim Levitsky

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Sean Christopherson]

On Wed, Sep 28, 2022, Maxim Levitsky wrote:
> On Wed, 2022-09-28 at 16:12 +0000, Sean Christopherson wrote:
> > On Wed, Sep 28, 2022, Paolo Bonzini wrote:
> > > On 9/28/22 09:10, Maxim Levitsky wrote:
> > > > I also think that outside KVM developers nobody should be using KVM on 32 bit host.
> > > > 
> > > > However for_developement_  I think that 32 bit KVM support is very useful, as it
> > > > allows to smoke test the support for 32 bit nested hypervisors, which I do once in a while,
> > > > and can even probably be useful to some users (e.g running some legacy stuff in a VM,
> > > > which includes a hypervisor, especially to run really legacy OSes / custom bare metal software,
> > > > using an old hypervisor) - or in other words, 32 bit nested KVM is mostly useless, but
> > > > other 32 bit nested hypervisors can be useful.
> > > > 
> > > > Yes, I can always use an older 32 bit kernel in a guest with KVM support, but as long
> > > > as current kernel works, it is useful to use the same kernel on host and guest.
> > > 
> > > Yeah, I would use older 32 bit kernels just like I use RHEL4 to test PIT
> > > reinjection. :)  But really the ultimate solution to this would be to
> > > improve kvm-unit-tests so that we can compile vmx.c and svm.c for 32-bit.
> > 
> > Agreed.  I too use 32-bit KVM to validate KVM's handling of 32-bit L1 hypervisors,
> > but the maintenance cost is painfully high.
> > 
> 
> But is it actually? I test it routinely and it it does work quite well IMHO.
> I don't remember that there were that much breakage lately in this area.

Oh, I didn't mean that it actually requires a lot of attention in terms of bug
fixes, what I meant by "maintenance cost" is the cost of testing that all the
flavors of 32-bit KVM actually work.  That can be automated to some extent, but
there's a non-trivial cost to maintaining all that automation.

> As far as my opinion goes I do volunteer to test this code more often,
> and I do not want to see the 32 bit KVM support be removed *yet*.

Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Paolo Bonzini]

On 9/28/22 19:55, Sean Christopherson wrote:
>> As far as my opinion goes I do volunteer to test this code more often,
>> and I do not want to see the 32 bit KVM support be removed*yet*.
>
> Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
> coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
> BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?

Yeah, maybe EXPERT is better but I'm not sure of the equivalent test 
coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but 
what's wrong with booting an older guest?

Paolo

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Maxim Levitsky]

On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
> On 9/28/22 19:55, Sean Christopherson wrote:
> > > As far as my opinion goes I do volunteer to test this code more often,
> > > and I do not want to see the 32 bit KVM support be removed*yet*.
> > 
> > Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
> > coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
> > BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
> 
> Yeah, maybe EXPERT is better but I'm not sure of the equivalent test 
> coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but 
> what's wrong with booting an older guest?

From my point of view, using the same kernel source for host and the guest
is easier because you know that both kernels behave the same.

About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
these days compiles a 32 bit kernel, knows what they are doing.

I personally would wait few more releases when there is a pressing reason to remove
this support.

AFAIK, it is not really possible to remove most of the legacy direct mmu
because shadowing mmu still can use it (I think Sean told me that once).

Best regards,
	Maxim Levitsky

> 
> Paolo
> 

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Paolo Bonzini]

On 9/29/22 15:52, Maxim Levitsky wrote:
> On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
>> On 9/28/22 19:55, Sean Christopherson wrote:
>>>> As far as my opinion goes I do volunteer to test this code more often,
>>>> and I do not want to see the 32 bit KVM support be removed*yet*.
>>>
>>> Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
>>> coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
>>> BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
>>
>> Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
>> coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
>> what's wrong with booting an older guest?
> 
> From my point of view, using the same kernel source for host and the guest
> is easier because you know that both kernels behave the same.

It is certainly easier, but it is less correct.  You don't cover 
anything that KVM doesn't use.

> About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
> one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
> these days compiles a 32 bit kernel, knows what they are doing.
> 
> I personally would wait few more releases when there is a pressing reason to remove
> this support.
> 
> AFAIK, it is not really possible to remove most of the legacy direct mmu
> because shadowing mmu still can use it (I think Sean told me that once).

Yeah, it won't let us remove a lot of code but there are several logic 
cleanups that become possible if the TDP case can just assume the TDP 
MMU is there.  For example, there is no reason to have a cpu_role (as 
opposed to an mmu_page_role for the root) if you are building HPA->GPA 
page tables.

(Which reminds me that toggling CR0.WP is still a hog with the TDP MMU).

Paolo

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Thomas Huth]

On 29/09/2022 15.52, Maxim Levitsky wrote:
> On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
>> On 9/28/22 19:55, Sean Christopherson wrote:
>>>> As far as my opinion goes I do volunteer to test this code more often,
>>>> and I do not want to see the 32 bit KVM support be removed*yet*.
>>>
>>> Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
>>> coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
>>> BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
>>
>> Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
>> coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
>> what's wrong with booting an older guest?
> 
>>From my point of view, using the same kernel source for host and the guest
> is easier because you know that both kernels behave the same.
> 
> About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
> one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
> these days compiles a 32 bit kernel, knows what they are doing.
> 
> I personally would wait few more releases when there is a pressing reason to remove
> this support.

FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit 
KVM support from the kernel. The QEMU project currently struggles badly with 
keeping everything tested in the CI in a reasonable amount of time. The 
32-bit KVM kernel support is the only reason to keep the qemu-system-i386 
binary around - everything else can be covered with the qemu-system-x86_64 
binary that is a superset of the -i386 variant (except for the KVM part as 
far as I know).
Sure, we could also drop qemu-system-i386 from the CI without dropping the 
32-bit KVM code in the kernel, but I guess things will rather bitrot there 
even faster in that case, so I'd appreciate if the kernel could drop the 
32-bit in the near future, too.

  Thomas

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Sean Christopherson]

On Fri, Feb 17, 2023, Thomas Huth wrote:
> On 29/09/2022 15.52, Maxim Levitsky wrote:
> > On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
> > > On 9/28/22 19:55, Sean Christopherson wrote:
> > > > > As far as my opinion goes I do volunteer to test this code more often,
> > > > > and I do not want to see the 32 bit KVM support be removed*yet*.
> > > > 
> > > > Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
> > > > coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
> > > > BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
> > > 
> > > Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
> > > coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
> > > what's wrong with booting an older guest?
> > 
> > > From my point of view, using the same kernel source for host and the guest
> > is easier because you know that both kernels behave the same.
> > 
> > About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
> > one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
> > these days compiles a 32 bit kernel, knows what they are doing.
> > 
> > I personally would wait few more releases when there is a pressing reason to remove
> > this support.
> 
> FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit
> KVM support from the kernel. The QEMU project currently struggles badly with
> keeping everything tested in the CI in a reasonable amount of time. The
> 32-bit KVM kernel support is the only reason to keep the qemu-system-i386
> binary around - everything else can be covered with the qemu-system-x86_64
> binary that is a superset of the -i386 variant (except for the KVM part as
> far as I know).
> Sure, we could also drop qemu-system-i386 from the CI without dropping the
> 32-bit KVM code in the kernel, but I guess things will rather bitrot there
> even faster in that case, so I'd appreciate if the kernel could drop the
> 32-bit in the near future, too.

Ya, I would happily drop support for 32-bit kernels today, the only sticking point
is the lack of 32-bit shadow paging test coverage, which unfortunately is a rather
large point.  :-(

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Thomas Huth]

On 22/02/2023 23.27, Sean Christopherson wrote:
> On Fri, Feb 17, 2023, Thomas Huth wrote:
>> On 29/09/2022 15.52, Maxim Levitsky wrote:
>>> On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
>>>> On 9/28/22 19:55, Sean Christopherson wrote:
>>>>>> As far as my opinion goes I do volunteer to test this code more often,
>>>>>> and I do not want to see the 32 bit KVM support be removed*yet*.
>>>>>
>>>>> Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
>>>>> coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
>>>>> BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
>>>>
>>>> Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
>>>> coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
>>>> what's wrong with booting an older guest?
>>>
>>>>  From my point of view, using the same kernel source for host and the guest
>>> is easier because you know that both kernels behave the same.
>>>
>>> About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
>>> one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
>>> these days compiles a 32 bit kernel, knows what they are doing.
>>>
>>> I personally would wait few more releases when there is a pressing reason to remove
>>> this support.
>>
>> FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit
>> KVM support from the kernel. The QEMU project currently struggles badly with
>> keeping everything tested in the CI in a reasonable amount of time. The
>> 32-bit KVM kernel support is the only reason to keep the qemu-system-i386
>> binary around - everything else can be covered with the qemu-system-x86_64
>> binary that is a superset of the -i386 variant (except for the KVM part as
>> far as I know).
>> Sure, we could also drop qemu-system-i386 from the CI without dropping the
>> 32-bit KVM code in the kernel, but I guess things will rather bitrot there
>> even faster in that case, so I'd appreciate if the kernel could drop the
>> 32-bit in the near future, too.
> 
> Ya, I would happily drop support for 32-bit kernels today, the only sticking point
> is the lack of 32-bit shadow paging test coverage, which unfortunately is a rather
> large point.  :-(

 From your point of view, would it be OK if QEMU dropped qemu-system-i386? 
I.e. would it be fine to use older versions of QEMU only for that test 
coverage (or do you even use a different userspace for testing that)?

  Thomas

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Maxim Levitsky]

On Thu, 2023-02-23 at 08:01 +0100, Thomas Huth wrote:
> On 22/02/2023 23.27, Sean Christopherson wrote:
> > On Fri, Feb 17, 2023, Thomas Huth wrote:
> > > On 29/09/2022 15.52, Maxim Levitsky wrote:
> > > > On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
> > > > > On 9/28/22 19:55, Sean Christopherson wrote:
> > > > > > > As far as my opinion goes I do volunteer to test this code more often,
> > > > > > > and I do not want to see the 32 bit KVM support be removed*yet*.
> > > > > > 
> > > > > > Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
> > > > > > coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
> > > > > > BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
> > > > > 
> > > > > Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
> > > > > coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
> > > > > what's wrong with booting an older guest?
> > > > >  From my point of view, using the same kernel source for host and the guest
> > > > is easier because you know that both kernels behave the same.
> > > > 
> > > > About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
> > > > one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
> > > > these days compiles a 32 bit kernel, knows what they are doing.
> > > > 
> > > > I personally would wait few more releases when there is a pressing reason to remove
> > > > this support.
> > > 
> > > FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit
> > > KVM support from the kernel. The QEMU project currently struggles badly with
> > > keeping everything tested in the CI in a reasonable amount of time. The
> > > 32-bit KVM kernel support is the only reason to keep the qemu-system-i386
> > > binary around - everything else can be covered with the qemu-system-x86_64
> > > binary that is a superset of the -i386 variant (except for the KVM part as
> > > far as I know).
> > > Sure, we could also drop qemu-system-i386 from the CI without dropping the
> > > 32-bit KVM code in the kernel, but I guess things will rather bitrot there
> > > even faster in that case, so I'd appreciate if the kernel could drop the
> > > 32-bit in the near future, too.
> > 
> > Ya, I would happily drop support for 32-bit kernels today, the only sticking point
> > is the lack of 32-bit shadow paging test coverage, which unfortunately is a rather
> > large point.  :-(
> 
>  From your point of view, would it be OK if QEMU dropped qemu-system-i386? 
> I.e. would it be fine to use older versions of QEMU only for that test 
> coverage (or do you even use a different userspace for testing that)?
> 
>   Thomas
> 

From my point of view qemu-system-x86_64 does run 32 bit guests just fine.

The only exception that I know is that gdbstub is somewhat broken, but that can be probably
fixed.


Best regards,
	Maxim Levitsky

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Sean Christopherson]

On Thu, Feb 23, 2023, Maxim Levitsky wrote:
> On Thu, 2023-02-23 at 08:01 +0100, Thomas Huth wrote:
> > On 22/02/2023 23.27, Sean Christopherson wrote:
> > > On Fri, Feb 17, 2023, Thomas Huth wrote:
> > > > On 29/09/2022 15.52, Maxim Levitsky wrote:
> > > > > On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
> > > > > > On 9/28/22 19:55, Sean Christopherson wrote:
> > > > > > > > As far as my opinion goes I do volunteer to test this code more often,
> > > > > > > > and I do not want to see the 32 bit KVM support be removed*yet*.
> > > > > > > 
> > > > > > > Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
> > > > > > > coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
> > > > > > > BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
> > > > > > 
> > > > > > Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
> > > > > > coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
> > > > > > what's wrong with booting an older guest?
> > > > > >  From my point of view, using the same kernel source for host and the guest
> > > > > is easier because you know that both kernels behave the same.
> > > > > 
> > > > > About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
> > > > > one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
> > > > > these days compiles a 32 bit kernel, knows what they are doing.
> > > > > 
> > > > > I personally would wait few more releases when there is a pressing reason to remove
> > > > > this support.
> > > > 
> > > > FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit
> > > > KVM support from the kernel. The QEMU project currently struggles badly with
> > > > keeping everything tested in the CI in a reasonable amount of time. The
> > > > 32-bit KVM kernel support is the only reason to keep the qemu-system-i386
> > > > binary around - everything else can be covered with the qemu-system-x86_64
> > > > binary that is a superset of the -i386 variant (except for the KVM part as
> > > > far as I know).
> > > > Sure, we could also drop qemu-system-i386 from the CI without dropping the
> > > > 32-bit KVM code in the kernel, but I guess things will rather bitrot there
> > > > even faster in that case, so I'd appreciate if the kernel could drop the
> > > > 32-bit in the near future, too.
> > > 
> > > Ya, I would happily drop support for 32-bit kernels today, the only sticking point
> > > is the lack of 32-bit shadow paging test coverage, which unfortunately is a rather
> > > large point.  :-(
> > 
> >  From your point of view, would it be OK if QEMU dropped qemu-system-i386? 
> > I.e. would it be fine to use older versions of QEMU only for that test 
> > coverage (or do you even use a different userspace for testing that)?

For me personally, I have no objection to dropping qemu-system-i386 support in
future QEMU releases.  I update my 32-bit images very, very infrequently, so I
probably wouldn't even notice for like 5 years :-)

> From my point of view qemu-system-x86_64 does run 32 bit guests just fine.

Right, but unless I seriously misunderstand what qemu-system-x86_64 ecompasses,
it can't be used to run guests of 32-bit _hosts_, which is what we need to test
shadowing of 32-bit NPT.

Re: [PATCH] KVM: x86: disable on 32-bit unless CONFIG_BROKEN

[Thomas Huth]

On 23/02/2023 23.10, Sean Christopherson wrote:
> On Thu, Feb 23, 2023, Maxim Levitsky wrote:
>> On Thu, 2023-02-23 at 08:01 +0100, Thomas Huth wrote:
>>> On 22/02/2023 23.27, Sean Christopherson wrote:
>>>> On Fri, Feb 17, 2023, Thomas Huth wrote:
>>>>> On 29/09/2022 15.52, Maxim Levitsky wrote:
>>>>>> On Thu, 2022-09-29 at 15:26 +0200, Paolo Bonzini wrote:
>>>>>>> On 9/28/22 19:55, Sean Christopherson wrote:
>>>>>>>>> As far as my opinion goes I do volunteer to test this code more often,
>>>>>>>>> and I do not want to see the 32 bit KVM support be removed*yet*.
>>>>>>>>
>>>>>>>> Yeah, I 100% agree that it shouldn't be removed until we have equivalent test
>>>>>>>> coverage.  But I do think it should an "off-by-default" sort of thing.  Maybe
>>>>>>>> BROKEN is the wrong dependency though?  E.g. would EXPERT be a better option?
>>>>>>>
>>>>>>> Yeah, maybe EXPERT is better but I'm not sure of the equivalent test
>>>>>>> coverage.  32-bit VMX/SVM kvm-unit-tests are surely a good idea, but
>>>>>>> what's wrong with booting an older guest?
>>>>>>>   From my point of view, using the same kernel source for host and the guest
>>>>>> is easier because you know that both kernels behave the same.
>>>>>>
>>>>>> About EXPERT, IMHO these days most distros already dropped 32 bit suport thus anyway
>>>>>> one needs to compile a recent 32 bit kernel manually - thus IMHO whoever
>>>>>> these days compiles a 32 bit kernel, knows what they are doing.
>>>>>>
>>>>>> I personally would wait few more releases when there is a pressing reason to remove
>>>>>> this support.
>>>>>
>>>>> FWIW, from the QEMU perspective, it would be very helpful to remove 32-bit
>>>>> KVM support from the kernel. The QEMU project currently struggles badly with
>>>>> keeping everything tested in the CI in a reasonable amount of time. The
>>>>> 32-bit KVM kernel support is the only reason to keep the qemu-system-i386
>>>>> binary around - everything else can be covered with the qemu-system-x86_64
>>>>> binary that is a superset of the -i386 variant (except for the KVM part as
>>>>> far as I know).
>>>>> Sure, we could also drop qemu-system-i386 from the CI without dropping the
>>>>> 32-bit KVM code in the kernel, but I guess things will rather bitrot there
>>>>> even faster in that case, so I'd appreciate if the kernel could drop the
>>>>> 32-bit in the near future, too.
>>>>
>>>> Ya, I would happily drop support for 32-bit kernels today, the only sticking point
>>>> is the lack of 32-bit shadow paging test coverage, which unfortunately is a rather
>>>> large point.  :-(
>>>
>>>   From your point of view, would it be OK if QEMU dropped qemu-system-i386?
>>> I.e. would it be fine to use older versions of QEMU only for that test
>>> coverage (or do you even use a different userspace for testing that)?
> 
> For me personally, I have no objection to dropping qemu-system-i386 support in
> future QEMU releases.  I update my 32-bit images very, very infrequently, so I
> probably wouldn't even notice for like 5 years :-)
> 
>>  From my point of view qemu-system-x86_64 does run 32 bit guests just fine.
> 
> Right, but unless I seriously misunderstand what qemu-system-x86_64 ecompasses,
> it can't be used to run guests of 32-bit _hosts_, which is what we need to test
> shadowing of 32-bit NPT.

That's what I've been told in the past, too, and that's why I asked. Thanks 
for the clarification!

To summarize: My takeaway is that nobody really needs qemu-system-i386 
anymore for recent development - the remaining 32-bit KVM use cases can be 
done with older versions of QEMU instead, thus it should be fine for the 
QEMU project to drop qemu-system-i386 nowadays.

  Thanks,
   Thomas