[PATCH 0/2] arm64: booting.rst: Clarify EL2 and cover v8-R64

[PATCH 0/2] arm64: booting.rst: Clarify EL2 and cover v8-R64

[Andre Przywara]

Hi,

clarify the wording of the ARM64 boot protocol requirements, to both
cater for the introduction of secure EL2 in Armv8.4, but also for the
addition of the Armv8-R64 architecture branch.
There is an ARM ARM supplement for v8-R64[1], and also a free model[2].

On top of those clarifications, there are at least two other issues to
consider for v8-R64 on Linux, I wonder if those should be documented
somewhere, although I doubt that booting.rst is the right place:
- Linux clears the NS/NSTable bit in the PTEs. In non-secure world those
  bits are ignored, but when in secure world this means "secure PA".
  That luckily matches the design here (secure-only), but we should avoid
  re-purposing those bits in the future (which would be technically
  possible when running only non-secure).
- The GIC needs to be implemented using a "single Security state" for
  the Linux GIC driver to work. The model mentioned above defaults to
  a security aware GIC, so needs to be told off using the command line.
  I wonder if this is should be explicitly mentioned somewhere, to avoid
  disappointment by implementors later.

I'd be grateful for any hint whether to state those things and where to
put them.

Cheers,
Andre

[1] https://developer.arm.com/documentation/ddi0600/latest/
[2] https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms/arm-ecosystem-models

Andre Przywara (2):
  arm64: booting.rst: Clarify on requiring non-secure EL2
  arm64: booting.rst: Cover Armv8-R64

 Documentation/arm64/booting.rst | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

-- 
2.25.1

[PATCH 1/2] arm64: booting.rst: Clarify on requiring non-secure EL2

[Andre Przywara]

The ARMv8.4 architecture revision introduced the EL2 exception level
to the secure world. Clarify the existing wording to make sure that
Linux relies on being executed in the non-secure state.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/arm64/booting.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
index 52d060caf8bb..07cb34ed4200 100644
--- a/Documentation/arm64/booting.rst
+++ b/Documentation/arm64/booting.rst
@@ -10,9 +10,9 @@ This document is based on the ARM booting document by Russell King and
 is relevant to all public releases of the AArch64 Linux kernel.
 
 The AArch64 exception model is made up of a number of exception levels
-(EL0 - EL3), with EL0 and EL1 having a secure and a non-secure
-counterpart.  EL2 is the hypervisor level and exists only in non-secure
-mode. EL3 is the highest priority level and exists only in secure mode.
+(EL0 - EL3), with EL0, EL1 and EL2 having a secure and a non-secure
+counterpart.  EL2 is the hypervisor level, EL3 is the highest priority
+level and exists only in secure mode. Both are architecturally optional.
 
 For the purposes of this document, we will use the term `boot loader`
 simply to define all software that executes on the CPU(s) before control
@@ -167,8 +167,8 @@ Before jumping into the kernel, the following conditions must be met:
 
   All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
   IRQ and FIQ).
-  The CPU must be in either EL2 (RECOMMENDED in order to have access to
-  the virtualisation extensions) or non-secure EL1.
+  The CPU must be in non-secure state, either in EL2 (RECOMMENDED in order
+  to have access  to the virtualisation extensions), or in EL1.
 
 - Caches, MMUs
 
-- 
2.25.1

Re: [PATCH 1/2] arm64: booting.rst: Clarify on requiring non-secure EL2

[Mark Rutland]

On Fri, Jan 07, 2022 at 04:00:55PM +0000, Andre Przywara wrote:
> The ARMv8.4 architecture revision introduced the EL2 exception level
> to the secure world. Clarify the existing wording to make sure that
> Linux relies on being executed in the non-secure state.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/arm64/booting.rst | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
> index 52d060caf8bb..07cb34ed4200 100644
> --- a/Documentation/arm64/booting.rst
> +++ b/Documentation/arm64/booting.rst
> @@ -10,9 +10,9 @@ This document is based on the ARM booting document by Russell King and
>  is relevant to all public releases of the AArch64 Linux kernel.
>  
>  The AArch64 exception model is made up of a number of exception levels
> -(EL0 - EL3), with EL0 and EL1 having a secure and a non-secure
> -counterpart.  EL2 is the hypervisor level and exists only in non-secure
> -mode. EL3 is the highest priority level and exists only in secure mode.
> +(EL0 - EL3), with EL0, EL1 and EL2 having a secure and a non-secure
> +counterpart.  EL2 is the hypervisor level, EL3 is the highest priority
> +level and exists only in secure mode. Both are architecturally optional.
>  
>  For the purposes of this document, we will use the term `boot loader`
>  simply to define all software that executes on the CPU(s) before control
> @@ -167,8 +167,8 @@ Before jumping into the kernel, the following conditions must be met:
>  
>    All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
>    IRQ and FIQ).
> -  The CPU must be in either EL2 (RECOMMENDED in order to have access to
> -  the virtualisation extensions) or non-secure EL1.
> +  The CPU must be in non-secure state, either in EL2 (RECOMMENDED in order
> +  to have access  to the virtualisation extensions), or in EL1.
                   ^^

Nit: double space

It might be clearer to explicitly say "non-secure EL2" and "non-secure EL1"
here, but either way this looks good to me, so with the whitespace fixed:

Reviewed-by: Mark Rutland <mark.rutland@arm.com>

Thanks,
Mark.

[PATCH 2/2] arm64: booting.rst: Cover Armv8-R64

[Andre Przywara]

There is a new revision of the ARMv8-R architecture [1], which
optionally introduces kernel compatibility - by introducing an MMU
into EL1 and EL0.
Linux can run on such an implementation, if it is entered in EL1 and
VMSA is both implemented and enabled for that exception level.

Clarify our kernel boot protocol to make this an officially supported
mode of operation, but also limit the expectations about running in
secure state (which is the only security state in v8-R).

Also we heavily rely on the Virtual Memory System Architecture (VMSA),
make this explicit in the text, as this allows to cover v8-R64 as well.

[1] https://developer.arm.com/documentation/ddi0600/latest/

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/arm64/booting.rst | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
index 07cb34ed4200..99fab4d7e7ad 100644
--- a/Documentation/arm64/booting.rst
+++ b/Documentation/arm64/booting.rst
@@ -167,8 +167,13 @@ Before jumping into the kernel, the following conditions must be met:
 
   All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
   IRQ and FIQ).
-  The CPU must be in non-secure state, either in EL2 (RECOMMENDED in order
-  to have access  to the virtualisation extensions), or in EL1.
+  If the CPU supports two security states, Linux must be entered in
+  non-secure state, either in EL2 (RECOMMENDED in order to have access
+  to the virtualisation extensions) or in EL1.
+  If the CPU only supports a single security state, Linux can be run even
+  when this single state is "secure".
+  The exception level the kernel is entered in must support the VMSA
+  memory model.
 
 - Caches, MMUs
 
-- 
2.25.1

Re: [PATCH 2/2] arm64: booting.rst: Cover Armv8-R64

[Mark Rutland]

On Fri, Jan 07, 2022 at 04:00:56PM +0000, Andre Przywara wrote:
> There is a new revision of the ARMv8-R architecture [1], which
> optionally introduces kernel compatibility - by introducing an MMU
> into EL1 and EL0.
> Linux can run on such an implementation, if it is entered in EL1 and
> VMSA is both implemented and enabled for that exception level.
> 
> Clarify our kernel boot protocol to make this an officially supported
> mode of operation, but also limit the expectations about running in
> secure state (which is the only security state in v8-R).
> 
> Also we heavily rely on the Virtual Memory System Architecture (VMSA),
> make this explicit in the text, as this allows to cover v8-R64 as well.
> 
> [1] https://developer.arm.com/documentation/ddi0600/latest/
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/arm64/booting.rst | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
> index 07cb34ed4200..99fab4d7e7ad 100644
> --- a/Documentation/arm64/booting.rst
> +++ b/Documentation/arm64/booting.rst
> @@ -167,8 +167,13 @@ Before jumping into the kernel, the following conditions must be met:
>  
>    All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
>    IRQ and FIQ).
> -  The CPU must be in non-secure state, either in EL2 (RECOMMENDED in order
> -  to have access  to the virtualisation extensions), or in EL1.
> +  If the CPU supports two security states, Linux must be entered in
> +  non-secure state, either in EL2 (RECOMMENDED in order to have access
> +  to the virtualisation extensions) or in EL1.
> +  If the CPU only supports a single security state, Linux can be run even
> +  when this single state is "secure".

Hmm... we've never supported running on the secure side so far, so are we
certain that everything actually works in such configs?

I know that some control fields (e.g. for filtering debug/tracing and so on)
differ across S/NS, and IIRC there's a bunch of GIC configuration that could
differ (but I could be mistaken).

Is there anything we need to have initialized differently by firmware?

Thanks,
Mark.

> +  The exception level the kernel is entered in must support the VMSA
> +  memory model.
>  
>  - Caches, MMUs
>  
> -- 
> 2.25.1
> 

Re: [PATCH 0/2] arm64: booting.rst: Clarify EL2 and cover v8-R64

[Will Deacon]

On Fri, 7 Jan 2022 16:00:54 +0000, Andre Przywara wrote:
> clarify the wording of the ARM64 boot protocol requirements, to both
> cater for the introduction of secure EL2 in Armv8.4, but also for the
> addition of the Armv8-R64 architecture branch.
> There is an ARM ARM supplement for v8-R64[1], and also a free model[2].
> 
> On top of those clarifications, there are at least two other issues to
> consider for v8-R64 on Linux, I wonder if those should be documented
> somewhere, although I doubt that booting.rst is the right place:
> - Linux clears the NS/NSTable bit in the PTEs. In non-secure world those
>   bits are ignored, but when in secure world this means "secure PA".
>   That luckily matches the design here (secure-only), but we should avoid
>   re-purposing those bits in the future (which would be technically
>   possible when running only non-secure).
> - The GIC needs to be implemented using a "single Security state" for
>   the Linux GIC driver to work. The model mentioned above defaults to
>   a security aware GIC, so needs to be told off using the command line.
>   I wonder if this is should be explicitly mentioned somewhere, to avoid
>   disappointment by implementors later.
> 
> [...]

Applied first patch to arm64 (for-next/docs), thanks!

[1/2] arm64: booting.rst: Clarify on requiring non-secure EL2
      https://git.kernel.org/arm64/c/b8ac4ee08d48

Cheers,
-- 
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev