Re: [PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[Stephan Gerhold <stephan@gerhold.net>]

Hi Marc,

On Thu, Apr 08, 2021 at 02:06:17PM +0100, Marc Zyngier wrote:
> Hi Stephan,
> 
> On Wed, 07 Apr 2021 17:36:48 +0100,
> Stephan Gerhold <stephan@gerhold.net> wrote:
> > 
> > The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
> > (an implementation of the ARM GIC 2.0 specification) used in MSM8916
> > support virtualization, e.g. for KVM on Linux. However, so far it was
> > not possible to make use of this functionality, because Qualcomm's
> > proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
> > allows booting Linux in EL1.
> > 
> > However, on devices without (firmware) secure boot there is no need
> > to rely on all of Qualcomm's firmware. The "hyp" firmware on MSM8916
> > seems simple enough that it can be replaced with an open-source
> > alternative created only based on trial and error - with some similar
> > EL2/EL1 initialization code adapted from Linux and U-Boot.
> 
> Ay, Quaramba! That's great news!
> 
> > 
> > qhypstub [1] is such an open-source firmware for MSM8916 that
> > can be used as drop-in replacement for Qualcomm's "hyp" firmware.
> > It does not implement any hypervisor functionality.
> > Instead, it allows booting Linux/KVM (or other hypervisors) in EL2.
> 
> Do you happen to know if the same method would apply to other SoCs
> from the same vendor? /me eyes the Lenovo c630 that is getting bored
> with EL1 only...
> 

I think there is still a separate "hyp" firmware on newer SoCs, which
suggests that a similar approach should be possible. However, newer SoCs
also seem to have "larger" hyp firmwares (just judging from the file
size). This suggests that there is more functionality implemented there.
Perhaps it's not important code or it can be replicated easily, but it
would definitely require some investigation.

However, I suspect the main problem for your Lenovo c630 is that it
likely has (firmware) secure boot enabled, which means that all firmware
must be signed with a private key from Lenovo. Unless you can somehow
convince Lenovo to sign firmware for you it is pretty much impossible to
tinker with the firmware there. Sorry :(

This is even a problem for qhypstub; out of 20+ MSM8916 smartphones
with mainline support (not all of them upstream yet), only ~5 can make
use of my firmware, all others also have firmware secure boot enabled.

For MSM8916, this even means no PSCI support (= no SMP, no CPU idle)
because Qualcomm never added support for that in firmwares used on
Android devices. I have to workaround that with very annoying hacks... :(

(To clarify this: Qualcomm made a PSCI firmware for the DragonBoard 410c,
 but it can only be installed on devices without firmware secure boot.)

> > 
> > With Linux booting in EL2, KVM seems to be working just fine on MSM8916.
> > However, so far it is not possible to make use of the virtualization
> > features in the GICv2. To use KVM's VGICv2 code, the QGIC2 device tree
> > node needs additional resources (according to binding documentation):
> > 
> >   - The CPU interface region (second reg) must be at least 8 KiB large
> >     to access the GICC_DIR register (mapped at 0x1000 offset)
> >   - Virtual control/CPU interface register base and size
> >   - Hypervisor maintenance interrupt
> > 
> > Fortunately, the public APQ8016E TRM [2] provides the required information:
> > 
> >   - The CPU interface region (at 0x0B002000) actually has a size of 8 KiB
> >   - Virtual control/CPU interface register is at 0x0B001000/0x0B004000
> >   - Hypervisor maintenance interrupt is "PPI #0"
> >       Note: This is a bit strange since almost all other ARM SoCs use
> >             GIC_PPI 9 for this. However, I have verified that this is
> >             indeed the interrupt that fires when bits are set in GICH_HCR.
> 
> Other SoCs have their maintenance interrupt wired to weird and
> wonderful interrupts. Given QC's lack of appetite for standards, I'm
> not totally surprised.
> 

To be honest, I was kind of positively surprised that Qualcomm
implemented the standards well enough to make KVM and the GIC
virtualization work without any quirks in the code. :)
I doubt that the Qualcomm firmware makes use of the GIC virtualization
functionality, so it is really nice that it works without any problems.

Thanks for the review!
Stephan