[PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[Stephan Gerhold]

The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
(an implementation of the ARM GIC 2.0 specification) used in MSM8916
support virtualization, e.g. for KVM on Linux. However, so far it was
not possible to make use of this functionality, because Qualcomm's
proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
allows booting Linux in EL1.

However, on devices without (firmware) secure boot there is no need
to rely on all of Qualcomm's firmware. The "hyp" firmware on MSM8916
seems simple enough that it can be replaced with an open-source
alternative created only based on trial and error - with some similar
EL2/EL1 initialization code adapted from Linux and U-Boot.

qhypstub [1] is such an open-source firmware for MSM8916 that
can be used as drop-in replacement for Qualcomm's "hyp" firmware.
It does not implement any hypervisor functionality.
Instead, it allows booting Linux/KVM (or other hypervisors) in EL2.

With Linux booting in EL2, KVM seems to be working just fine on MSM8916.
However, so far it is not possible to make use of the virtualization
features in the GICv2. To use KVM's VGICv2 code, the QGIC2 device tree
node needs additional resources (according to binding documentation):

  - The CPU interface region (second reg) must be at least 8 KiB large
    to access the GICC_DIR register (mapped at 0x1000 offset)
  - Virtual control/CPU interface register base and size
  - Hypervisor maintenance interrupt

Fortunately, the public APQ8016E TRM [2] provides the required information:

  - The CPU interface region (at 0x0B002000) actually has a size of 8 KiB
  - Virtual control/CPU interface register is at 0x0B001000/0x0B004000
  - Hypervisor maintenance interrupt is "PPI #0"
      Note: This is a bit strange since almost all other ARM SoCs use
            GIC_PPI 9 for this. However, I have verified that this is
            indeed the interrupt that fires when bits are set in GICH_HCR.

Add the additional resources to the QGIC2 device tree node in msm8916.dtsi.
There is no functional difference when Linux is started in EL1 since the
additional resources are ignored in that case.

With these changes (and qhypstub), KVM seems to be fully working on
the DragonBoard 410c (apq8016-sbc) and BQ Aquaris X5 (longcheer-l8910).

[1]: https://github.com/msm8916-mainline/qhypstub
[2]: https://developer.qualcomm.com/download/sd410/snapdragon-410e-technical-reference-manual.pdf

Signed-off-by: Stephan Gerhold <stephan@gerhold.net>
---
 arch/arm64/boot/dts/qcom/msm8916.dtsi | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi
index 4c155735fbc9..4f06c0a9c425 100644
--- a/arch/arm64/boot/dts/qcom/msm8916.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi
@@ -1766,7 +1766,9 @@ intc: interrupt-controller@b000000 {
 			compatible = "qcom,msm-qgic2";
 			interrupt-controller;
 			#interrupt-cells = <3>;
-			reg = <0x0b000000 0x1000>, <0x0b002000 0x1000>;
+			reg = <0x0b000000 0x1000>, <0x0b002000 0x2000>,
+			      <0x0b001000 0x1000>, <0x0b004000 0x2000>;
+			interrupts = <GIC_PPI 0 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
 		};
 
 		apcs: mailbox@b011000 {
-- 
2.31.1

Re: [PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[Bjorn Andersson]

On Wed 07 Apr 11:36 CDT 2021, Stephan Gerhold wrote:

> The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
> (an implementation of the ARM GIC 2.0 specification) used in MSM8916
> support virtualization, e.g. for KVM on Linux. However, so far it was
> not possible to make use of this functionality, because Qualcomm's
> proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
> allows booting Linux in EL1.
> 
> However, on devices without (firmware) secure boot there is no need
> to rely on all of Qualcomm's firmware. The "hyp" firmware on MSM8916
> seems simple enough that it can be replaced with an open-source
> alternative created only based on trial and error - with some similar
> EL2/EL1 initialization code adapted from Linux and U-Boot.
> 
> qhypstub [1] is such an open-source firmware for MSM8916 that
> can be used as drop-in replacement for Qualcomm's "hyp" firmware.
> It does not implement any hypervisor functionality.
> Instead, it allows booting Linux/KVM (or other hypervisors) in EL2.
> 
> With Linux booting in EL2, KVM seems to be working just fine on MSM8916.
> However, so far it is not possible to make use of the virtualization
> features in the GICv2. To use KVM's VGICv2 code, the QGIC2 device tree
> node needs additional resources (according to binding documentation):
> 
>   - The CPU interface region (second reg) must be at least 8 KiB large
>     to access the GICC_DIR register (mapped at 0x1000 offset)
>   - Virtual control/CPU interface register base and size
>   - Hypervisor maintenance interrupt
> 
> Fortunately, the public APQ8016E TRM [2] provides the required information:
> 
>   - The CPU interface region (at 0x0B002000) actually has a size of 8 KiB
>   - Virtual control/CPU interface register is at 0x0B001000/0x0B004000
>   - Hypervisor maintenance interrupt is "PPI #0"
>       Note: This is a bit strange since almost all other ARM SoCs use
>             GIC_PPI 9 for this. However, I have verified that this is
>             indeed the interrupt that fires when bits are set in GICH_HCR.
> 
> Add the additional resources to the QGIC2 device tree node in msm8916.dtsi.
> There is no functional difference when Linux is started in EL1 since the
> additional resources are ignored in that case.
> 
> With these changes (and qhypstub), KVM seems to be fully working on
> the DragonBoard 410c (apq8016-sbc) and BQ Aquaris X5 (longcheer-l8910).
> 
> [1]: https://github.com/msm8916-mainline/qhypstub
> [2]: https://developer.qualcomm.com/download/sd410/snapdragon-410e-technical-reference-manual.pdf

This is awesome!

Thanks,
Bjorn

> 
> Signed-off-by: Stephan Gerhold <stephan@gerhold.net>
> ---
>  arch/arm64/boot/dts/qcom/msm8916.dtsi | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi
> index 4c155735fbc9..4f06c0a9c425 100644
> --- a/arch/arm64/boot/dts/qcom/msm8916.dtsi
> +++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi
> @@ -1766,7 +1766,9 @@ intc: interrupt-controller@b000000 {
>  			compatible = "qcom,msm-qgic2";
>  			interrupt-controller;
>  			#interrupt-cells = <3>;
> -			reg = <0x0b000000 0x1000>, <0x0b002000 0x1000>;
> +			reg = <0x0b000000 0x1000>, <0x0b002000 0x2000>,
> +			      <0x0b001000 0x1000>, <0x0b004000 0x2000>;
> +			interrupts = <GIC_PPI 0 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
>  		};
>  
>  		apcs: mailbox@b011000 {
> -- 
> 2.31.1
> 

Re: [PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[patchwork-bot+linux-arm-msm]

Hello:

This patch was applied to qcom/linux.git (refs/heads/for-next):

On Wed,  7 Apr 2021 18:36:48 +0200 you wrote:
> The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
> (an implementation of the ARM GIC 2.0 specification) used in MSM8916
> support virtualization, e.g. for KVM on Linux. However, so far it was
> not possible to make use of this functionality, because Qualcomm's
> proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
> allows booting Linux in EL1.
> 
> [...]

Here is the summary with links:
  - arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt
    https://git.kernel.org/qcom/c/4579058fecb6

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[Marc Zyngier]

Hi Stephan,

On Wed, 07 Apr 2021 17:36:48 +0100,
Stephan Gerhold <stephan@gerhold.net> wrote:
> 
> The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
> (an implementation of the ARM GIC 2.0 specification) used in MSM8916
> support virtualization, e.g. for KVM on Linux. However, so far it was
> not possible to make use of this functionality, because Qualcomm's
> proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
> allows booting Linux in EL1.
> 
> However, on devices without (firmware) secure boot there is no need
> to rely on all of Qualcomm's firmware. The "hyp" firmware on MSM8916
> seems simple enough that it can be replaced with an open-source
> alternative created only based on trial and error - with some similar
> EL2/EL1 initialization code adapted from Linux and U-Boot.

Ay, Quaramba! That's great news!

> 
> qhypstub [1] is such an open-source firmware for MSM8916 that
> can be used as drop-in replacement for Qualcomm's "hyp" firmware.
> It does not implement any hypervisor functionality.
> Instead, it allows booting Linux/KVM (or other hypervisors) in EL2.

Do you happen to know if the same method would apply to other SoCs
from the same vendor? /me eyes the Lenovo c630 that is getting bored
with EL1 only...

> 
> With Linux booting in EL2, KVM seems to be working just fine on MSM8916.
> However, so far it is not possible to make use of the virtualization
> features in the GICv2. To use KVM's VGICv2 code, the QGIC2 device tree
> node needs additional resources (according to binding documentation):
> 
>   - The CPU interface region (second reg) must be at least 8 KiB large
>     to access the GICC_DIR register (mapped at 0x1000 offset)
>   - Virtual control/CPU interface register base and size
>   - Hypervisor maintenance interrupt
> 
> Fortunately, the public APQ8016E TRM [2] provides the required information:
> 
>   - The CPU interface region (at 0x0B002000) actually has a size of 8 KiB
>   - Virtual control/CPU interface register is at 0x0B001000/0x0B004000
>   - Hypervisor maintenance interrupt is "PPI #0"
>       Note: This is a bit strange since almost all other ARM SoCs use
>             GIC_PPI 9 for this. However, I have verified that this is
>             indeed the interrupt that fires when bits are set in GICH_HCR.

Other SoCs have their maintenance interrupt wired to weird and
wonderful interrupts. Given QC's lack of appetite for standards, I'm
not totally surprised.

>
> Add the additional resources to the QGIC2 device tree node in msm8916.dtsi.
> There is no functional difference when Linux is started in EL1 since the
> additional resources are ignored in that case.
> 
> With these changes (and qhypstub), KVM seems to be fully working on
> the DragonBoard 410c (apq8016-sbc) and BQ Aquaris X5 (longcheer-l8910).
> 
> [1]: https://github.com/msm8916-mainline/qhypstub
> [2]: https://developer.qualcomm.com/download/sd410/snapdragon-410e-technical-reference-manual.pdf
> 
> Signed-off-by: Stephan Gerhold <stephan@gerhold.net>
> ---
>  arch/arm64/boot/dts/qcom/msm8916.dtsi | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi
> index 4c155735fbc9..4f06c0a9c425 100644
> --- a/arch/arm64/boot/dts/qcom/msm8916.dtsi
> +++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi
> @@ -1766,7 +1766,9 @@ intc: interrupt-controller@b000000 {
>  			compatible = "qcom,msm-qgic2";
>  			interrupt-controller;
>  			#interrupt-cells = <3>;
> -			reg = <0x0b000000 0x1000>, <0x0b002000 0x1000>;
> +			reg = <0x0b000000 0x1000>, <0x0b002000 0x2000>,
> +			      <0x0b001000 0x1000>, <0x0b004000 0x2000>;
> +			interrupts = <GIC_PPI 0 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
>  		};
>  
>  		apcs: mailbox@b011000 {

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] arm64: dts: qcom: msm8916: Add GICv2 hypervisor registers/interrupt

[Stephan Gerhold]

Hi Marc,

On Thu, Apr 08, 2021 at 02:06:17PM +0100, Marc Zyngier wrote:
> Hi Stephan,
> 
> On Wed, 07 Apr 2021 17:36:48 +0100,
> Stephan Gerhold <stephan@gerhold.net> wrote:
> > 
> > The ARM Cortex-A53 CPU cores and QGIC2 interrupt controller
> > (an implementation of the ARM GIC 2.0 specification) used in MSM8916
> > support virtualization, e.g. for KVM on Linux. However, so far it was
> > not possible to make use of this functionality, because Qualcomm's
> > proprietary "hyp" firmware blocks the EL2 mode of the CPU and only
> > allows booting Linux in EL1.
> > 
> > However, on devices without (firmware) secure boot there is no need
> > to rely on all of Qualcomm's firmware. The "hyp" firmware on MSM8916
> > seems simple enough that it can be replaced with an open-source
> > alternative created only based on trial and error - with some similar
> > EL2/EL1 initialization code adapted from Linux and U-Boot.
> 
> Ay, Quaramba! That's great news!
> 
> > 
> > qhypstub [1] is such an open-source firmware for MSM8916 that
> > can be used as drop-in replacement for Qualcomm's "hyp" firmware.
> > It does not implement any hypervisor functionality.
> > Instead, it allows booting Linux/KVM (or other hypervisors) in EL2.
> 
> Do you happen to know if the same method would apply to other SoCs
> from the same vendor? /me eyes the Lenovo c630 that is getting bored
> with EL1 only...
> 

I think there is still a separate "hyp" firmware on newer SoCs, which
suggests that a similar approach should be possible. However, newer SoCs
also seem to have "larger" hyp firmwares (just judging from the file
size). This suggests that there is more functionality implemented there.
Perhaps it's not important code or it can be replicated easily, but it
would definitely require some investigation.

However, I suspect the main problem for your Lenovo c630 is that it
likely has (firmware) secure boot enabled, which means that all firmware
must be signed with a private key from Lenovo. Unless you can somehow
convince Lenovo to sign firmware for you it is pretty much impossible to
tinker with the firmware there. Sorry :(

This is even a problem for qhypstub; out of 20+ MSM8916 smartphones
with mainline support (not all of them upstream yet), only ~5 can make
use of my firmware, all others also have firmware secure boot enabled.

For MSM8916, this even means no PSCI support (= no SMP, no CPU idle)
because Qualcomm never added support for that in firmwares used on
Android devices. I have to workaround that with very annoying hacks... :(

(To clarify this: Qualcomm made a PSCI firmware for the DragonBoard 410c,
 but it can only be installed on devices without firmware secure boot.)

> > 
> > With Linux booting in EL2, KVM seems to be working just fine on MSM8916.
> > However, so far it is not possible to make use of the virtualization
> > features in the GICv2. To use KVM's VGICv2 code, the QGIC2 device tree
> > node needs additional resources (according to binding documentation):
> > 
> >   - The CPU interface region (second reg) must be at least 8 KiB large
> >     to access the GICC_DIR register (mapped at 0x1000 offset)
> >   - Virtual control/CPU interface register base and size
> >   - Hypervisor maintenance interrupt
> > 
> > Fortunately, the public APQ8016E TRM [2] provides the required information:
> > 
> >   - The CPU interface region (at 0x0B002000) actually has a size of 8 KiB
> >   - Virtual control/CPU interface register is at 0x0B001000/0x0B004000
> >   - Hypervisor maintenance interrupt is "PPI #0"
> >       Note: This is a bit strange since almost all other ARM SoCs use
> >             GIC_PPI 9 for this. However, I have verified that this is
> >             indeed the interrupt that fires when bits are set in GICH_HCR.
> 
> Other SoCs have their maintenance interrupt wired to weird and
> wonderful interrupts. Given QC's lack of appetite for standards, I'm
> not totally surprised.
> 

To be honest, I was kind of positively surprised that Qualcomm
implemented the standards well enough to make KVM and the GIC
virtualization work without any quirks in the code. :)
I doubt that the Qualcomm firmware makes use of the GIC virtualization
functionality, so it is really nice that it works without any problems.

Thanks for the review!
Stephan