[PATCH v2 0/4] arm64: Booting clarifications and fine grained traps setup

[PATCH v2 0/4] arm64: Booting clarifications and fine grained traps setup

[Mark Brown]

Currently there are a number of areas where we are not explicit about
how system registers should be configured when booting the kernel,
provide more detailed guidance to help ensure people get things right.
I'm not entirely happy with the wording on use of specified reset values
so it is a separate patch at the end, I worry that the requirement is
too strict.

This was triggered by noticing that it was in spec to leave fine grained
traps to EL2 enabled so explicitly disable them at boot, this is sent as
part of the same series since doing so means we require that EL3 provide
access to fine grained traps in the same document.

v2:
 - Pull booting and FGT setup into a single series.
 - Require system registers at or below the level the kernel is entered
   to be initialised, not just those at that level.
 - Rather than documenting the desired EL2 setup for FGT add a general
   statement about how we expect registers to be set up.
 - Reorder register initialisation in el2_setup.h.
 - Document requirement for SCR_EL3.FGTEn to be 1.

Mark Brown (4):
  arm64: Document requirements for fine grained traps at boot
  arm64: Disable fine grained traps on boot
  arm64: Require that system registers at all visible ELs be initialized
  arm64: Document values for system registers on boot

 Documentation/arm64/booting.rst    | 15 ++++++++++++---
 arch/arm64/include/asm/el2_setup.h | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sysreg.h    |  6 ++++++
 3 files changed, 39 insertions(+), 3 deletions(-)


base-commit: 1e28eed17697bcf343c6743f0028cc3b5dd88bf0
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 1/4] arm64: Document requirements for fine grained traps at boot

[Mark Brown]

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses
to small sets of registers and instructions from EL1 and EL0, access to
which is controlled by EL3.  Require access to it so that it is
available to us in future and so that we can ensure these traps are
disabled during boot.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 Documentation/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
index 7552dbc1cc54..92ec0bea1af5 100644
--- a/Documentation/arm64/booting.rst
+++ b/Documentation/arm64/booting.rst
@@ -270,6 +270,12 @@ Before jumping into the kernel, the following conditions must be met:
       having 0b1 set for the corresponding bit for each of the auxiliary
       counters present.
 
+  For CPUs with the Fine Grained Traps (FEAT_FGT) extension present:
+
+  - If EL3 is present and the kernel is entered at EL2:
+
+    - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 2/4] arm64: Disable fine grained traps on boot

[Mark Brown]

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses
to small sets of registers and instructions from EL1 and EL0.  Currently
Linux makes no use of this feature, ensure that it is not active at boot by
disabling the traps during EL2 setup.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/include/asm/el2_setup.h | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sysreg.h    |  6 ++++++
 2 files changed, 27 insertions(+)

diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index d77d358f9395..b3f2d3bb0938 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -131,6 +131,26 @@
 .Lskip_sve_\@:
 .endm
 
+/* Disable any fine grained traps */
+.macro __init_el2_fgt
+	mrs	x1, id_aa64mmfr0_el1
+	ubfx	x1, x1, #ID_AA64MMFR0_FGT_SHIFT, #4
+	cbz	x1, .Lskip_fgt_\@
+
+	msr_s	SYS_HDFGRTR_EL2, xzr
+	msr_s	SYS_HDFGWTR_EL2, xzr
+	msr_s	SYS_HFGRTR_EL2, xzr
+	msr_s	SYS_HFGWTR_EL2, xzr
+	msr_s	SYS_HFGITR_EL2, xzr
+
+	mrs	x1, id_aa64pfr0_el1		// AMU traps UNDEF without AMU
+	ubfx	x1, x1, #ID_AA64PFR0_AMU_SHIFT, #4
+	cbz	x1, .Lskip_fgt_\@
+
+	msr_s	SYS_HAFGRTR_EL2, xzr
+.Lskip_fgt_\@:
+.endm
+
 .macro __init_el2_nvhe_prepare_eret
 	mov	x0, #INIT_PSTATE_EL1
 	msr	spsr_el2, x0
@@ -155,6 +175,7 @@
 	__init_el2_nvhe_idregs
 	__init_el2_nvhe_cptr
 	__init_el2_nvhe_sve
+	__init_el2_fgt
 	__init_el2_nvhe_prepare_eret
 .endm
 
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index d4a5fca984c3..b35468927363 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -475,9 +475,15 @@
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
 #define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
+#define SYS_HFGRTR_EL2			sys_reg(3, 4, 1, 1, 4)
+#define SYS_HFGWTR_EL2			sys_reg(3, 4, 1, 1, 5)
+#define SYS_HFGITR_EL2			sys_reg(3, 4, 1, 1, 6)
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
+#define SYS_HDFGRTR_EL2			sys_reg(3, 4, 3, 1, 4)
+#define SYS_HDFGWTR_EL2			sys_reg(3, 4, 3, 1, 5)
+#define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 3/4] arm64: Require that system registers at all visible ELs be initialized

[Mark Brown]

Currently we require that software at a higher exception level initialise
all registers at the exception level the kernel will be entered prior to
starting the kernel in order to ensure that there is nothing uninitialised
which could result in an UNKNOWN state while running the kernel. The
expectation is that the software running at the highest exception levels
will be tightly coupled to the system and can ensure that all available
features are appropriately initialised and that the kernel can initialise
anything else.

There is a gap here in the case where new registers are added to lower
exception levels that require initialisation but the kernel does not yet
understand them. Extend the requirement to also include exception levels
below the one where the kernel is entered to cover this.

Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Mark Brown <broonie@kernel.org>
---
 Documentation/arm64/booting.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
index 92ec0bea1af5..4fcc00add117 100644
--- a/Documentation/arm64/booting.rst
+++ b/Documentation/arm64/booting.rst
@@ -202,9 +202,10 @@ Before jumping into the kernel, the following conditions must be met:
 
 - System registers
 
-  All writable architected system registers at the exception level where
-  the kernel image will be entered must be initialised by software at a
-  higher exception level to prevent execution in an UNKNOWN state.
+  All writable architected system registers at or below the exception
+  level where the kernel image will be entered must be initialised by
+  software at a higher exception level to prevent execution in an UNKNOWN
+  state.
 
   - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
     executing on.
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 4/4] arm64: Document values for system registers on boot

[Mark Brown]

When booting require that the system registers available at or below the
exception level the kernel is entered be initialised but do not specify
what values should be used in the general case, creating some potential
for issues if the kernel does not subsequently configure those registers
explicitly (for example if they are not yet used by the kernel) or where
their effects may create issues during early configuration.

Specify that where the architecture provides a reset value that value
must be used.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 Documentation/arm64/booting.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
index 4fcc00add117..d49f568eb79d 100644
--- a/Documentation/arm64/booting.rst
+++ b/Documentation/arm64/booting.rst
@@ -205,7 +205,9 @@ Before jumping into the kernel, the following conditions must be met:
   All writable architected system registers at or below the exception
   level where the kernel image will be entered must be initialised by
   software at a higher exception level to prevent execution in an UNKNOWN
-  state.
+  state.  Where these architected system registers have reset values
+  specified by the architecture they must be initialised to those values
+  unless specified more specifically.
 
   - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
     executing on.
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/4] arm64: Document requirements for fine grained traps at boot

[Marc Zyngier]

On Thu, 01 Apr 2021 19:09:39 +0100,
Mark Brown <broonie@kernel.org> wrote:
> 
> The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses
> to small sets of registers and instructions from EL1 and EL0, access to
> which is controlled by EL3.  Require access to it so that it is
> available to us in future and so that we can ensure these traps are
> disabled during boot.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> ---
>  Documentation/arm64/booting.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
> index 7552dbc1cc54..92ec0bea1af5 100644
> --- a/Documentation/arm64/booting.rst
> +++ b/Documentation/arm64/booting.rst
> @@ -270,6 +270,12 @@ Before jumping into the kernel, the following conditions must be met:
>        having 0b1 set for the corresponding bit for each of the auxiliary
>        counters present.
>  
> +  For CPUs with the Fine Grained Traps (FEAT_FGT) extension present:
> +
> +  - If EL3 is present and the kernel is entered at EL2:
> +
> +    - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.
> +
>  The requirements described above for CPU mode, caches, MMUs, architected
>  timers, coherency and system registers apply to all CPUs.  All CPUs must
>  enter the kernel in the same exception level.

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/4] arm64: Disable fine grained traps on boot

[Marc Zyngier]

On Thu, 01 Apr 2021 19:09:40 +0100,
Mark Brown <broonie@kernel.org> wrote:
> 
> The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses
> to small sets of registers and instructions from EL1 and EL0.  Currently
> Linux makes no use of this feature, ensure that it is not active at boot by
> disabling the traps during EL2 setup.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>

Reviewed-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 3/4] arm64: Require that system registers at all visible ELs be initialized

[Marc Zyngier]

On Thu, 01 Apr 2021 19:09:41 +0100,
Mark Brown <broonie@kernel.org> wrote:
> 
> Currently we require that software at a higher exception level initialise
> all registers at the exception level the kernel will be entered prior to
> starting the kernel in order to ensure that there is nothing uninitialised
> which could result in an UNKNOWN state while running the kernel. The
> expectation is that the software running at the highest exception levels
> will be tightly coupled to the system and can ensure that all available
> features are appropriately initialised and that the kernel can initialise
> anything else.
> 
> There is a gap here in the case where new registers are added to lower
> exception levels that require initialisation but the kernel does not yet
> understand them. Extend the requirement to also include exception levels
> below the one where the kernel is entered to cover this.
> 
> Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
> Signed-off-by: Mark Brown <broonie@kernel.org>

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 4/4] arm64: Document values for system registers on boot

[Marc Zyngier]

Hi Mark,

On Thu, 01 Apr 2021 19:09:42 +0100,
Mark Brown <broonie@kernel.org> wrote:
> 
> When booting require that the system registers available at or below the
> exception level the kernel is entered be initialised but do not specify
> what values should be used in the general case, creating some potential
> for issues if the kernel does not subsequently configure those registers
> explicitly (for example if they are not yet used by the kernel) or where
> their effects may create issues during early configuration.
> 
> Specify that where the architecture provides a reset value that value
> must be used.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> ---
>  Documentation/arm64/booting.rst | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst
> index 4fcc00add117..d49f568eb79d 100644
> --- a/Documentation/arm64/booting.rst
> +++ b/Documentation/arm64/booting.rst
> @@ -205,7 +205,9 @@ Before jumping into the kernel, the following conditions must be met:
>    All writable architected system registers at or below the exception
>    level where the kernel image will be entered must be initialised by
>    software at a higher exception level to prevent execution in an UNKNOWN
> -  state.
> +  state.  Where these architected system registers have reset values
> +  specified by the architecture they must be initialised to those values
> +  unless specified more specifically.
>  
>    - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
>      executing on.
> -- 
> 2.20.1
> 
> 

Is that always enforceable? Is that even desirable?

Take for example ICC_SRE_EL2.SRE. The reset value for that bit is
0. But it is extremely likely that the FW has set this bit to 1 in
order to be able to use interrupts with the sysreg interface. However,
as outlined in the GIC spec:

<quote>
If software changes this bit from 1 to 0, the results are UNPREDICTABLE.
</quote>

We could go and specify this one, but I fear there is a lot of things
we'd have to make explicit...

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 4/4] arm64: Document values for system registers on boot

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 1960 bytes --]

On Thu, Apr 08, 2021 at 06:16:03PM +0100, Marc Zyngier wrote:
> Mark Brown <broonie@kernel.org> wrote:

> > Specify that where the architecture provides a reset value that value
> > must be used.

> >    level where the kernel image will be entered must be initialised by
> >    software at a higher exception level to prevent execution in an UNKNOWN
> > -  state.
> > +  state.  Where these architected system registers have reset values
> > +  specified by the architecture they must be initialised to those values
> > +  unless specified more specifically.

> Is that always enforceable? Is that even desirable?

I'm not convinced on either front frankly, like I said in the cover
letter I was worrying that it is too strict a requirement.  I couldn't
think of a better/more appropriate wording and figured it might start a
discussion.

> Take for example ICC_SRE_EL2.SRE. The reset value for that bit is
> 0. But it is extremely likely that the FW has set this bit to 1 in
> order to be able to use interrupts with the sysreg interface. However,
> as outlined in the GIC spec:

> <quote>
> If software changes this bit from 1 to 0, the results are UNPREDICTABLE.
> </quote>

That definitely means it's unenforcable without an explict override for
that case, having now seen that case I'm sure there's others.

> We could go and specify this one, but I fear there is a lot of things
> we'd have to make explicit...

Yeah, that's pushing me towards dropping this patch.  We could add a
general statement recommending that people use the defaults without
actually requiring it but that's just vague and unenforcable instead of
strict and unimplementable which isn't particularly better.  If we're
going to have to keep updating the document it's going to be less effort
all round to just update for cases we can't handle in code rather than
keep on adding exceptions and require firmware to reset lots of stuff we
don't *really* have strict requirements for.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: (subset) [PATCH v2 0/4] arm64: Booting clarifications and fine grained traps setup

[Catalin Marinas]

On Thu, 1 Apr 2021 19:09:38 +0100, Mark Brown wrote:
> Currently there are a number of areas where we are not explicit about
> how system registers should be configured when booting the kernel,
> provide more detailed guidance to help ensure people get things right.
> I'm not entirely happy with the wording on use of specified reset values
> so it is a separate patch at the end, I worry that the requirement is
> too strict.
> 
> [...]

Applied to arm64 (for-next/fgt-boot-init), thanks!

[1/4] arm64: Document requirements for fine grained traps at boot
      https://git.kernel.org/arm64/c/3e237387bb76
[2/4] arm64: Disable fine grained traps on boot
      https://git.kernel.org/arm64/c/31c00d2aeaa2
[3/4] arm64: Require that system registers at all visible ELs be initialized
      https://git.kernel.org/arm64/c/230800cd315c

-- 
Catalin


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel