KVM works on RPi4

KVM works on RPi4

[Jan Kiszka]

Hi all,

just got KVM running on the Raspberry Pi4. Seems they now embedded all
required logic into that new SoC.

However, as the Raspberry kernel is not yet ready for 64-bit (and
upstream is not in sight), I had to use legacy 32-bit mode. And there we
stumble over the core detection. This little patch made it work, though:

diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
index 2b8de885b2bf..01606aad73cc 100644
--- a/arch/arm/kvm/guest.c
+++ b/arch/arm/kvm/guest.c
@@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
 	case ARM_CPU_PART_CORTEX_A7:
 		return KVM_ARM_TARGET_CORTEX_A7;
 	case ARM_CPU_PART_CORTEX_A15:
+	case ARM_CPU_PART_CORTEX_A72:
 		return KVM_ARM_TARGET_CORTEX_A15;
 	default:
 		return -EINVAL;

That raises the question if this is hack or a valid change and if there
is general interest in mapping 64-bit cores on 32-bit if they happen to
run in 32-bit mode.

Jan

PS: The RPi device tree lacks description of the GICH maintenance
interrupts. Seems KVM is fine without that - because it has the
information hard-coded or because it can live without that interrupt?
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Marc Zyngier]

On Sat, 29 Jun 2019 19:09:37 +0200
Jan Kiszka <jan.kiszka@web.de> wrote:

Hi Jan,

> Hi all,
> 
> just got KVM running on the Raspberry Pi4. Seems they now embedded all
> required logic into that new SoC.

Yeah, someone saw the light and decided to enter the 21st century by
attaching a GICv2 to the thing. Who knows, they may plug a GICv3 and a
SMMU in 2050 at that rate! ;-)

> However, as the Raspberry kernel is not yet ready for 64-bit (and
> upstream is not in sight), I had to use legacy 32-bit mode. And there we
> stumble over the core detection. This little patch made it work, though:
> 
> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
> index 2b8de885b2bf..01606aad73cc 100644
> --- a/arch/arm/kvm/guest.c
> +++ b/arch/arm/kvm/guest.c
> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
>  	case ARM_CPU_PART_CORTEX_A7:
>  		return KVM_ARM_TARGET_CORTEX_A7;
>  	case ARM_CPU_PART_CORTEX_A15:
> +	case ARM_CPU_PART_CORTEX_A72:
>  		return KVM_ARM_TARGET_CORTEX_A15;
>  	default:
>  		return -EINVAL;
> 
> That raises the question if this is hack or a valid change and if there
> is general interest in mapping 64-bit cores on 32-bit if they happen to
> run in 32-bit mode.

The real thing to do here would be to move to a generic target, much
like we did on the 64bit side. Could you investigate that instead? It
would also allow KVM to be used on other 32bit cores such as
A12/A17/A32.

Although some would argue that the *real* real thing to do would be "rm
-rf arch/arm/kvm" and be done with it, but that's a discussion for next
week... ;-)

> Jan
> 
> PS: The RPi device tree lacks description of the GICH maintenance
> interrupts. Seems KVM is fine without that - because it has the
> information hard-coded or because it can live without that interrupt?

Nah, it really should have an interrupt here. You can end-up in
situation where new virtual interrupts are delayed until the next
natural exit if you don't get a maintenance interrupt. Feels like a bug.

Anyway, if you know of any effort to get a 64bit kernel on that thing,
I'm interested in helping. I bought one on Monday, but didn't get a
change to do any hacking on it just yet...

Thanks,

	M.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Jan Kiszka]

On 30.06.19 00:42, Marc Zyngier wrote:
> On Sat, 29 Jun 2019 19:09:37 +0200
> Jan Kiszka <jan.kiszka@web.de> wrote:
>
> Hi Jan,
>
>> Hi all,
>>
>> just got KVM running on the Raspberry Pi4. Seems they now embedded all
>> required logic into that new SoC.
>
> Yeah, someone saw the light and decided to enter the 21st century by
> attaching a GICv2 to the thing. Who knows, they may plug a GICv3 and a
> SMMU in 2050 at that rate! ;-)
>

Optimistic.

>> However, as the Raspberry kernel is not yet ready for 64-bit (and
>> upstream is not in sight), I had to use legacy 32-bit mode. And there we
>> stumble over the core detection. This little patch made it work, though:
>>
>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
>> index 2b8de885b2bf..01606aad73cc 100644
>> --- a/arch/arm/kvm/guest.c
>> +++ b/arch/arm/kvm/guest.c
>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
>>   	case ARM_CPU_PART_CORTEX_A7:
>>   		return KVM_ARM_TARGET_CORTEX_A7;
>>   	case ARM_CPU_PART_CORTEX_A15:
>> +	case ARM_CPU_PART_CORTEX_A72:
>>   		return KVM_ARM_TARGET_CORTEX_A15;
>>   	default:
>>   		return -EINVAL;
>>
>> That raises the question if this is hack or a valid change and if there
>> is general interest in mapping 64-bit cores on 32-bit if they happen to
>> run in 32-bit mode.
>
> The real thing to do here would be to move to a generic target, much
> like we did on the 64bit side. Could you investigate that instead? It
> would also allow KVM to be used on other 32bit cores such as
> A12/A17/A32.

You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that...

>
> Although some would argue that the *real* real thing to do would be "rm
> -rf arch/arm/kvm" and be done with it, but that's a discussion for next
> week... ;-)
>
>> Jan
>>
>> PS: The RPi device tree lacks description of the GICH maintenance
>> interrupts. Seems KVM is fine without that - because it has the
>> information hard-coded or because it can live without that interrupt?
>
> Nah, it really should have an interrupt here. You can end-up in
> situation where new virtual interrupts are delayed until the next
> natural exit if you don't get a maintenance interrupt. Feels like a bug.

Probably just in their DT. How can I check if the maintenance IRQ is working?

>
> Anyway, if you know of any effort to get a 64bit kernel on that thing,
> I'm interested in helping. I bought one on Monday, but didn't get a
> change to do any hacking on it just yet...

I played with compiling the rpi kernel for 64-bit. Lots of pieces from the
graphic drivers are falling from the truck, but you can make it build at least.
Not that it boots so far or gives any early messages. Probably that is the reason:

https://github.com/raspberrypi/linux/issues/3032

Jan
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Jan Kiszka]

On 30.06.19 11:34, Jan Kiszka wrote:
> On 30.06.19 00:42, Marc Zyngier wrote:
>> On Sat, 29 Jun 2019 19:09:37 +0200
>> Jan Kiszka <jan.kiszka@web.de> wrote:
>>> However, as the Raspberry kernel is not yet ready for 64-bit (and
>>> upstream is not in sight), I had to use legacy 32-bit mode. And there we
>>> stumble over the core detection. This little patch made it work, though:
>>>
>>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
>>> index 2b8de885b2bf..01606aad73cc 100644
>>> --- a/arch/arm/kvm/guest.c
>>> +++ b/arch/arm/kvm/guest.c
>>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
>>>       case ARM_CPU_PART_CORTEX_A7:
>>>           return KVM_ARM_TARGET_CORTEX_A7;
>>>       case ARM_CPU_PART_CORTEX_A15:
>>> +    case ARM_CPU_PART_CORTEX_A72:
>>>           return KVM_ARM_TARGET_CORTEX_A15;
>>>       default:
>>>           return -EINVAL;
>>>
>>> That raises the question if this is hack or a valid change and if there
>>> is general interest in mapping 64-bit cores on 32-bit if they happen to
>>> run in 32-bit mode.
>>
>> The real thing to do here would be to move to a generic target, much
>> like we did on the 64bit side. Could you investigate that instead? It
>> would also allow KVM to be used on other 32bit cores such as
>> A12/A17/A32.
>
> You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that...
>

Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I
found nothing so far:

kvm_reset_vcpu:
         switch (vcpu->arch.target) {
         case KVM_ARM_TARGET_CORTEX_A7:
         case KVM_ARM_TARGET_CORTEX_A15:
                 reset_regs = &cortexa_regs_reset;
                 vcpu->arch.midr = read_cpuid_id();
                 break;

And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some
symbols renamed.

What's the purpose of all that? Planned for something bigger but never
implemented? From that perspective, there seems to be no need to arch.target and
kvm_coproc_target_table at all.

Jan
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Jan Kiszka]

On 30.06.19 12:18, Jan Kiszka wrote:
> On 30.06.19 11:34, Jan Kiszka wrote:
>> On 30.06.19 00:42, Marc Zyngier wrote:
>>> On Sat, 29 Jun 2019 19:09:37 +0200
>>> Jan Kiszka <jan.kiszka@web.de> wrote:
>>>> However, as the Raspberry kernel is not yet ready for 64-bit (and
>>>> upstream is not in sight), I had to use legacy 32-bit mode. And there we
>>>> stumble over the core detection. This little patch made it work, though:
>>>>
>>>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
>>>> index 2b8de885b2bf..01606aad73cc 100644
>>>> --- a/arch/arm/kvm/guest.c
>>>> +++ b/arch/arm/kvm/guest.c
>>>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
>>>>       case ARM_CPU_PART_CORTEX_A7:
>>>>           return KVM_ARM_TARGET_CORTEX_A7;
>>>>       case ARM_CPU_PART_CORTEX_A15:
>>>> +    case ARM_CPU_PART_CORTEX_A72:
>>>>           return KVM_ARM_TARGET_CORTEX_A15;
>>>>       default:
>>>>           return -EINVAL;
>>>>
>>>> That raises the question if this is hack or a valid change and if there
>>>> is general interest in mapping 64-bit cores on 32-bit if they happen to
>>>> run in 32-bit mode.
>>>
>>> The real thing to do here would be to move to a generic target, much
>>> like we did on the 64bit side. Could you investigate that instead? It
>>> would also allow KVM to be used on other 32bit cores such as
>>> A12/A17/A32.
>>
>> You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that...
>>
>
> Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I
> found nothing so far:
>
> kvm_reset_vcpu:
>          switch (vcpu->arch.target) {
>          case KVM_ARM_TARGET_CORTEX_A7:
>          case KVM_ARM_TARGET_CORTEX_A15:
>                  reset_regs = &cortexa_regs_reset;
>                  vcpu->arch.midr = read_cpuid_id();
>                  break;
>
> And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some
> symbols renamed.

OK, found it: The reset values of SCTLR differ, in one bit. A15 starts with
branch prediction (11) off, A7 with that feature enabled. Quite some boilerplate
code for managing a single bit.

For a generic target, can we simply assume A15 reset behaviour?

Jan
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Vladimir Murzin]

On 6/30/19 11:49 AM, Jan Kiszka wrote:
> On 30.06.19 12:18, Jan Kiszka wrote:
>> On 30.06.19 11:34, Jan Kiszka wrote:
>>> On 30.06.19 00:42, Marc Zyngier wrote:
>>>> On Sat, 29 Jun 2019 19:09:37 +0200
>>>> Jan Kiszka <jan.kiszka@web.de> wrote:
>>>>> However, as the Raspberry kernel is not yet ready for 64-bit (and
>>>>> upstream is not in sight), I had to use legacy 32-bit mode. And there we
>>>>> stumble over the core detection. This little patch made it work, though:
>>>>>
>>>>> diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
>>>>> index 2b8de885b2bf..01606aad73cc 100644
>>>>> --- a/arch/arm/kvm/guest.c
>>>>> +++ b/arch/arm/kvm/guest.c
>>>>> @@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
>>>>>       case ARM_CPU_PART_CORTEX_A7:
>>>>>           return KVM_ARM_TARGET_CORTEX_A7;
>>>>>       case ARM_CPU_PART_CORTEX_A15:
>>>>> +    case ARM_CPU_PART_CORTEX_A72:
>>>>>           return KVM_ARM_TARGET_CORTEX_A15;
>>>>>       default:
>>>>>           return -EINVAL;
>>>>>
>>>>> That raises the question if this is hack or a valid change and if there
>>>>> is general interest in mapping 64-bit cores on 32-bit if they happen to
>>>>> run in 32-bit mode.
>>>>
>>>> The real thing to do here would be to move to a generic target, much
>>>> like we did on the 64bit side. Could you investigate that instead? It
>>>> would also allow KVM to be used on other 32bit cores such as
>>>> A12/A17/A32.
>>>
>>> You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that...
>>>
>>
>> Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I
>> found nothing so far:
>>
>> kvm_reset_vcpu:
>>          switch (vcpu->arch.target) {
>>          case KVM_ARM_TARGET_CORTEX_A7:
>>          case KVM_ARM_TARGET_CORTEX_A15:
>>                  reset_regs = &cortexa_regs_reset;
>>                  vcpu->arch.midr = read_cpuid_id();
>>                  break;
>>
>> And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some
>> symbols renamed.
> 
> OK, found it: The reset values of SCTLR differ, in one bit. A15 starts with
> branch prediction (11) off, A7 with that feature enabled. Quite some boilerplate
> code for managing a single bit.
> 
> For a generic target, can we simply assume A15 reset behaviour?

IIUC, it'd work only for ARCH_VIRT guest, which is known not to touch IMP_DEF
registers. Unfortunately, other variants of guests (ARCH_VEXPRESS?) might touch
such registers, for instance, l2ctlr is often used for querying number of populated
cpus, and it might not be present at all on v8. Also, content of IMP_DEF register
is not fixed and meaning of the bits may wary between different implementations, so
guests may react differently.

Just my 2p.

Cheers
Vladimir

> 
> Jan
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Christoffer Dall]

Hi Jan,

On Sun, Jun 30, 2019 at 12:18:59PM +0200, Jan Kiszka wrote:
> On 30.06.19 11:34, Jan Kiszka wrote:
> >On 30.06.19 00:42, Marc Zyngier wrote:
> >>On Sat, 29 Jun 2019 19:09:37 +0200
> >>Jan Kiszka <jan.kiszka@web.de> wrote:
> >>>However, as the Raspberry kernel is not yet ready for 64-bit (and
> >>>upstream is not in sight), I had to use legacy 32-bit mode. And there we
> >>>stumble over the core detection. This little patch made it work, though:
> >>>
> >>>diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
> >>>index 2b8de885b2bf..01606aad73cc 100644
> >>>--- a/arch/arm/kvm/guest.c
> >>>+++ b/arch/arm/kvm/guest.c
> >>>@@ -290,6 +290,7 @@ int __attribute_const__ kvm_target_cpu(void)
> >>>      case ARM_CPU_PART_CORTEX_A7:
> >>>          return KVM_ARM_TARGET_CORTEX_A7;
> >>>      case ARM_CPU_PART_CORTEX_A15:
> >>>+    case ARM_CPU_PART_CORTEX_A72:
> >>>          return KVM_ARM_TARGET_CORTEX_A15;
> >>>      default:
> >>>          return -EINVAL;
> >>>
> >>>That raises the question if this is hack or a valid change and if there
> >>>is general interest in mapping 64-bit cores on 32-bit if they happen to
> >>>run in 32-bit mode.
> >>
> >>The real thing to do here would be to move to a generic target, much
> >>like we did on the 64bit side. Could you investigate that instead? It
> >>would also allow KVM to be used on other 32bit cores such as
> >>A12/A17/A32.
> >
> >You mean something like KVM_ARM_TARGET_GENERIC_V8? Need to study that...
> >
> 
> Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I
> found nothing so far:
> 
> kvm_reset_vcpu:
>         switch (vcpu->arch.target) {
>         case KVM_ARM_TARGET_CORTEX_A7:
>         case KVM_ARM_TARGET_CORTEX_A15:
>                 reset_regs = &cortexa_regs_reset;
>                 vcpu->arch.midr = read_cpuid_id();
>                 break;
> 
> And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some
> symbols renamed.
> 
> What's the purpose of all that? Planned for something bigger but never
> implemented? From that perspective, there seems to be no need to arch.target and
> kvm_coproc_target_table at all.
> 

There was some speculation involved here, and we needed to figure out
how we would deal with implementation defined behavior, so we built this
support for each type of CPU etc.

In reality, most CPUs that we support are pretty similar and that's why
we did the generic CPU type instead.  In practice, there might be a more
light-weight appraoch to handling the minor differences between CPU
implementations than what we have here.


Thanks,

    Christoffer
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM works on RPi4

[Peter Maydell]

On Mon, 26 Aug 2019 at 09:38, Christoffer Dall <christoffer.dall@arm.com> wrote:
> On Sun, Jun 30, 2019 at 12:18:59PM +0200, Jan Kiszka wrote:
> > Hmm, looking at what KVM_ARM_TARGET_CORTEX_A7 and ..._A15 differentiates, I
> > found nothing so far:
> >
> > kvm_reset_vcpu:
> >         switch (vcpu->arch.target) {
> >         case KVM_ARM_TARGET_CORTEX_A7:
> >         case KVM_ARM_TARGET_CORTEX_A15:
> >                 reset_regs = &cortexa_regs_reset;
> >                 vcpu->arch.midr = read_cpuid_id();
> >                 break;
> >
> > And arch/arm/kvm/coproc_a15.c looks like a copy of coproc_a7.c, just with some
> > symbols renamed.
> >
> > What's the purpose of all that? Planned for something bigger but never
> > implemented? From that perspective, there seems to be no need to arch.target and
> > kvm_coproc_target_table at all.
> >
>
> There was some speculation involved here, and we needed to figure out
> how we would deal with implementation defined behavior, so we built this
> support for each type of CPU etc.
>
> In reality, most CPUs that we support are pretty similar and that's why
> we did the generic CPU type instead.  In practice, there might be a more
> light-weight appraoch to handling the minor differences between CPU
> implementations than what we have here.

The other future-direction I think we were thinking about was that
one day we'd want to support showing the guest a CPU other than
what the host is, at which point you would want to be able to
say specifically "give me a Cortex-A7" and have it work even if the
host was a Cortex-A15. But there are significant unresolved design
issues if we ever did want to go in that direction...

thanks
-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm