KVM_COMPAT support on aarch64

KVM_COMPAT support on aarch64

[Daniel Verkamp]

Hi Marc,

I noticed that in recent kernels (4.19 in our case), KVM no longer
works for 32-bit processes running on arm64 machines. We
(Crostini/crosvm, Linux VM support on Chromebooks) use this KVM_COMPAT
on all arm64-based Chromebooks that support VMs, since our entire
userspace (including crosvm) is 32-bit.

I found this commit (KVM: arm64: Prevent KVM_COMPAT from being
selected): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c

Is there any particular reason the compat ioctl is disabled (e.g.
actual bugs/breakage), or would it be possible to re-enable KVM_COMPAT
for ARM64?

In my brief testing with a 4.19-stable-based kernel on a new arm64
board, everything seems to work fine with the commit above reverted.

Thanks,
-- Daniel
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM_COMPAT support on aarch64

[Marc Zyngier]

Hi Daniel,

[please use my kernel.org address, as I'm not employed by ARM anymore]

On 2019-11-21 20:16, Daniel Verkamp wrote:
> Hi Marc,
>
> I noticed that in recent kernels (4.19 in our case), KVM no longer
> works for 32-bit processes running on arm64 machines. We
> (Crostini/crosvm, Linux VM support on Chromebooks) use this 
> KVM_COMPAT
> on all arm64-based Chromebooks that support VMs, since our entire
> userspace (including crosvm) is 32-bit.
>
> I found this commit (KVM: arm64: Prevent KVM_COMPAT from being
> selected):
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c
>
> Is there any particular reason the compat ioctl is disabled (e.g.
> actual bugs/breakage), or would it be possible to re-enable 
> KVM_COMPAT
> for ARM64?

Well, it never worked the first place! :( The whole API is entirely 
different,
and you should feel extremely lucky that it ever seemed to work!

> In my brief testing with a 4.19-stable-based kernel on a new arm64
> board, everything seems to work fine with the commit above reverted.

The key word here is *seems*. It may happen that the userspace ABI
somehow align for some of the basic stuff, but it will fail for
anything moderately serious (I really doubt QEMU would be able to
run at all).

I suppose to tried this with crosvm? 32 or 64bit guest?

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM_COMPAT support on aarch64

[Daniel Verkamp]

On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Daniel,
>
> [please use my kernel.org address, as I'm not employed by ARM anymore]
>
> On 2019-11-21 20:16, Daniel Verkamp wrote:
> > Hi Marc,
> >
> > I noticed that in recent kernels (4.19 in our case), KVM no longer
> > works for 32-bit processes running on arm64 machines. We
> > (Crostini/crosvm, Linux VM support on Chromebooks) use this
> > KVM_COMPAT
> > on all arm64-based Chromebooks that support VMs, since our entire
> > userspace (including crosvm) is 32-bit.
> >
> > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being
> > selected):
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c
> >
> > Is there any particular reason the compat ioctl is disabled (e.g.
> > actual bugs/breakage), or would it be possible to re-enable
> > KVM_COMPAT
> > for ARM64?
>
> Well, it never worked the first place! :( The whole API is entirely
> different,
> and you should feel extremely lucky that it ever seemed to work!

I hadn't looked at this before, but it seems like the Rust bindings
for KVM we are using are actually generated from the aarch64 C headers
(e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs):
https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs
So we are really passing the native 64-bit structs to the 32-bit
compat ioctl, which is probably bogus.

I can see how this is unlikely to work in a normal C program compiled
against the 32-bit ARM uapi headers.

> > In my brief testing with a 4.19-stable-based kernel on a new arm64
> > board, everything seems to work fine with the commit above reverted.
>
> The key word here is *seems*. It may happen that the userspace ABI
> somehow align for some of the basic stuff, but it will fail for
> anything moderately serious (I really doubt QEMU would be able to
> run at all).
>
> I suppose to tried this with crosvm? 32 or 64bit guest?

Yes, I am able to run the full Crostini environment using 32-bit
crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable.  The
guest kernel and userspace is 64 bit in this case (we don't support
32-bit guest).

I also just tried a 32-bit build of kvmtool (lkvm), since we have a
convenient ebuild to cross-compile it in the Chrome OS SDK, but I
wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG).
I didn't attempt to debug it, but I'm assuming it's due to the ABI
mismatch you mentioned.

It sounds like disabling the compat ioctl is probably the right thing
to do in this case; I'll have to see if we can build crosvm as a
64-bit executable instead in the future (not sure how easily this will
be given the way the Chromium OS build system works).

Thanks,
-- Daniel
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM_COMPAT support on aarch64

[Marc Zyngier]

On Fri, 22 Nov 2019 09:50:50 -0800
Daniel Verkamp <dverkamp@chromium.org> wrote:

> On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Daniel,
> >
> > [please use my kernel.org address, as I'm not employed by ARM anymore]
> >
> > On 2019-11-21 20:16, Daniel Verkamp wrote:  
> > > Hi Marc,
> > >
> > > I noticed that in recent kernels (4.19 in our case), KVM no longer
> > > works for 32-bit processes running on arm64 machines. We
> > > (Crostini/crosvm, Linux VM support on Chromebooks) use this
> > > KVM_COMPAT
> > > on all arm64-based Chromebooks that support VMs, since our entire
> > > userspace (including crosvm) is 32-bit.
> > >
> > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being
> > > selected):
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c
> > >
> > > Is there any particular reason the compat ioctl is disabled (e.g.
> > > actual bugs/breakage), or would it be possible to re-enable
> > > KVM_COMPAT
> > > for ARM64?  
> >
> > Well, it never worked the first place! :( The whole API is entirely
> > different,
> > and you should feel extremely lucky that it ever seemed to work!  
> 
> I hadn't looked at this before, but it seems like the Rust bindings
> for KVM we are using are actually generated from the aarch64 C headers
> (e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs):
> https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs
> So we are really passing the native 64-bit structs to the 32-bit
> compat ioctl, which is probably bogus.

Wow. It's amazing it did anything sensible!

> I can see how this is unlikely to work in a normal C program compiled
> against the 32-bit ARM uapi headers.

Indeed.

> > > In my brief testing with a 4.19-stable-based kernel on a new arm64
> > > board, everything seems to work fine with the commit above reverted.  
> >
> > The key word here is *seems*. It may happen that the userspace ABI
> > somehow align for some of the basic stuff, but it will fail for
> > anything moderately serious (I really doubt QEMU would be able to
> > run at all).
> >
> > I suppose to tried this with crosvm? 32 or 64bit guest?  
> 
> Yes, I am able to run the full Crostini environment using 32-bit
> crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable.  The
> guest kernel and userspace is 64 bit in this case (we don't support
> 32-bit guest).
> 
> I also just tried a 32-bit build of kvmtool (lkvm), since we have a
> convenient ebuild to cross-compile it in the Chrome OS SDK, but I
> wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG).
> I didn't attempt to debug it, but I'm assuming it's due to the ABI
> mismatch you mentioned.

That's what I was expecting indeed. The registers are identified
differently to make my life easier (and at some point I considered
writing a compat interface before making up my mind and canning the
thing altogether).

> It sounds like disabling the compat ioctl is probably the right thing
> to do in this case; I'll have to see if we can build crosvm as a
> 64-bit executable instead in the future (not sure how easily this will
> be given the way the Chromium OS build system works).

You may have to end-up with a static binary, which is not very
appealing...


	M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm