[PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Wei Chen]

The Armv-R architecture profile was designed to support use cases
that have a high sensitivity to deterministic execution. (e.g.
Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm
architecture targeted at the Real-time profile. It introduces
virtualization at the highest security level while retaining the
Protected Memory System Architecture (PMSA) based on a Memory
Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
The latest Armv8-R64 document can be found [1]. And the features of
Armv8-R64 architecture:
  - An exception model that is compatible with the Armv8-A model
  - Virtualization with support for guest operating systems
  - PMSA virtualization using MPUs In EL2.
  - Adds support for the 64-bit A64 instruction set.
  - Supports up to 48-bit physical addressing.
  - Supports three Exception Levels (ELs)
        - Secure EL2 - The Highest Privilege
        - Secure EL1 - RichOS (MMU) or RTOS (MPU)
        - Secure EL0 - Application Workloads
 - Supports only a single Security state - Secure.
 - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support
for Xen, which are based on the discussion of
"Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.

We will split these patches to several parts, this series is the
part#1, the full PoC can be found in [3]. More software for
Armv8-R64 can be found in [4];

[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/

Penny Zheng (3):
  xen/arm64: create boot-time MPU protection regions
  xen/arm64: introduce helpers for MPU enable/disable
  xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Wei Chen (8):
  xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
  xen/arm: add iounmap after initrd has been loaded in domain_build
  xen/arm: disable EFI boot services for MPU systems
  xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
  xen/arm: define Xen start address for FVP BaseR platform
  xen/arm: split MMU and MPU config files from config.h
  xen/arm: implement FIXMAP_ADDR for MPU systems
  xen/arm64: move MMU related code from head.S to head_mmu.S

 xen/arch/arm/Kconfig                          |  15 +-
 xen/arch/arm/arm64/Makefile                   |   5 +
 xen/arch/arm/arm64/head.S                     | 429 ++----------------
 xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
 xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
 xen/arch/arm/domain_build.c                   |   2 +
 xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
 xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
 xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
 xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
 xen/arch/arm/include/asm/config.h             |  99 +---
 xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
 xen/arch/arm/include/asm/config_mpu.h         |  29 ++
 xen/arch/arm/include/asm/fixmap.h             |  25 +
 xen/arch/arm/include/asm/flushtlb.h           |  22 +
 .../arm/include/asm/platforms/fvp_baser.h     |  18 +
 xen/arch/arm/platforms/Kconfig                |  16 +-
 17 files changed, 976 insertions(+), 500 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

-- 
2.25.1

[PATCH v6 01/11] xen/arm: remove xen_phys_start and xenheap_phys_end from config.h

[Wei Chen]

These two variables are stale variables, they only have declarations
in config.h, they don't have any definition and no any code is using
these two variables. So in this patch, we remove them from config.h.

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/include/asm/config.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/xen/arch/arm/include/asm/config.h b/xen/arch/arm/include/asm/config.h
index 0fefed1b8a..25a625ff08 100644
--- a/xen/arch/arm/include/asm/config.h
+++ b/xen/arch/arm/include/asm/config.h
@@ -172,8 +172,6 @@
 #define STACK_SIZE  (PAGE_SIZE << STACK_ORDER)
 
 #ifndef __ASSEMBLY__
-extern unsigned long xen_phys_start;
-extern unsigned long xenheap_phys_end;
 extern unsigned long frametable_virt_end;
 #endif
 
-- 
2.25.1

Re: [PATCH v6 01/11] xen/arm: remove xen_phys_start and xenheap_phys_end from config.h

[Julien Grall]

Hi,

On 04/11/2022 10:07, Wei Chen wrote:
> These two variables are stale variables, they only have declarations

AFAICT, this has always been the case. I am guessing this was because 
the header was mostly likely copied from x86...

> in config.h, they don't have any definition and no any code is using
> these two variables. So in this patch, we remove them from config.h.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>

Acked-by: Julien Grall <jgrall@amazon.com>

Cheers,

-- 
Julien Grall

[PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Wei Chen]

domain_build use ioremap_wc to map a new non-cacheable virtual
address for initrd. After Xen copy initrd from this address to
guest, this new allocated virtual address has not been unmapped.

So in this patch, we add an iounmap to the end of domain_build,
after Xen loaded initrd to guest memory.

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/domain_build.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 4fb5c20b13..bd30d3798c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo)
                                           initrd, len);
     if ( res != 0 )
         panic("Unable to copy the initrd in the hwdom memory\n");
+
+    iounmap(initrd);
 }
 
 /*
-- 
2.25.1

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Julien Grall]

(+ Henry)

Hi,

On 04/11/2022 10:07, Wei Chen wrote:
> domain_build use ioremap_wc to map a new non-cacheable virtual

s/use/uses/

> address for initrd. After Xen copy initrd from this address to
> guest, this new allocated virtual address has not been unmapped.
> 
> So in this patch, we add an iounmap to the end of domain_build,
> after Xen loaded initrd to guest memory.
> 

Please a fixes tag. The issue was introduced by commit bb7e6d565d92.

> Signed-off-by: Wei Chen <wei.chen@arm.com>
> ---
>   xen/arch/arm/domain_build.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 4fb5c20b13..bd30d3798c 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info *kinfo)
>                                             initrd, len);
>       if ( res != 0 )
>           panic("Unable to copy the initrd in the hwdom memory\n");
> +
> +    iounmap(initrd);

This looks good to me. But I am wondering whether using ioremap_wc() is 
actually correct because we are reading the region. So it seems strang 
to map it with write-combine.

So I would consider to use ioremap_cache(). That said, this would be a 
separate patch.

I think this wants to be in 4.17. This will avoid Xen to have two 
mappings with different caching attribute (initrd is part of the RAM and 
therefore directmap).

Cheers,

-- 
Julien Grall

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Henry Wang]

Hi Julien,

> -----Original Message-----
> Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been
> loaded in domain_build
> 
> (+ Henry)
> I think this wants to be in 4.17. This will avoid Xen to have two
> mappings with different caching attribute (initrd is part of the RAM and
> therefore directmap).

Sounds good to me, I am wondering if we need to include also patch #1 though.

If this patch wants to be in 4.17:

Release-acked-by: Henry Wang <Henry.Wang@arm.com>

Kind regards,
Henry

> 
> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Julien Grall]

On 07/11/2022 01:33, Henry Wang wrote:
> Hi Julien,

Hi Henry,

> 
>> -----Original Message-----
>> Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been
>> loaded in domain_build
>>
>> (+ Henry)
>> I think this wants to be in 4.17. This will avoid Xen to have two
>> mappings with different caching attribute (initrd is part of the RAM and
>> therefore directmap).
> 
> Sounds good to me, I am wondering if we need to include also patch #1 though.

If we were earlier in the release, I would have say yes. But now, it is 
a no as this just a cleanup (having extra definitions is harmless).

> 
> If this patch wants to be in 4.17:
> 
> Release-acked-by: Henry Wang <Henry.Wang@arm.com>

Thanks!

Cheers,

-- 
Julien Grall

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Henry Wang]

Hi Julien,

> -----Original Message-----
> Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been
> loaded in domain_build
> >> (+ Henry)
> >> I think this wants to be in 4.17. This will avoid Xen to have two
> >> mappings with different caching attribute (initrd is part of the RAM and
> >> therefore directmap).
> >
> > Sounds good to me, I am wondering if we need to include also patch #1
> though.
> 
> If we were earlier in the release, I would have say yes. But now, it is
> a no as this just a cleanup (having extra definitions is harmless).

Thanks for your confirmation :) No problem.

Kind regards,
Henry

Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Julien Grall]

On 06/11/2022 18:55, Julien Grall wrote:
> (+ Henry)
> 
> Hi,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
>> domain_build use ioremap_wc to map a new non-cacheable virtual
> 
> s/use/uses/
> 
>> address for initrd. After Xen copy initrd from this address to
>> guest, this new allocated virtual address has not been unmapped.
>>
>> So in this patch, we add an iounmap to the end of domain_build,
>> after Xen loaded initrd to guest memory.
>>
> 
> Please a fixes tag. The issue was introduced by commit bb7e6d565d92.

Well I forgot to add it on commit :/. I will try to remember to backport it.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
> Julien Grall
> Sent: 2022年11月7日 2:55
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>; Henry Wang <Henry.Wang@arm.com>
> Subject: Re: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been
> loaded in domain_build
> 
> (+ Henry)
> 
> Hi,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > domain_build use ioremap_wc to map a new non-cacheable virtual
> 
> s/use/uses/
> 
> > address for initrd. After Xen copy initrd from this address to
> > guest, this new allocated virtual address has not been unmapped.
> >
> > So in this patch, we add an iounmap to the end of domain_build,
> > after Xen loaded initrd to guest memory.
> >
> 
> Please a fixes tag. The issue was introduced by commit bb7e6d565d92.
> 
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > ---
> >   xen/arch/arm/domain_build.c | 2 ++
> >   1 file changed, 2 insertions(+)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 4fb5c20b13..bd30d3798c 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -3418,6 +3418,8 @@ static void __init initrd_load(struct kernel_info
> *kinfo)
> >                                             initrd, len);
> >       if ( res != 0 )
> >           panic("Unable to copy the initrd in the hwdom memory\n");
> > +
> > +    iounmap(initrd);
> 
> This looks good to me. But I am wondering whether using ioremap_wc() is
> actually correct because we are reading the region. So it seems strang
> to map it with write-combine.
> 
> So I would consider to use ioremap_cache(). That said, this would be a
> separate patch.
>

Ok, we will try to use ioremap_cache and test it. If everything works
well we will introduce a separate patch in next version.

Cheers,
Wei Chen


> I think this wants to be in 4.17. This will avoid Xen to have two
> mappings with different caching attribute (initrd is part of the RAM and
> therefore directmap).
> 
> Cheers,
> 
> --
> Julien Grall

RE: [PATCH v6 02/11] xen/arm: add iounmap after initrd has been loaded in domain_build

[Wei Chen]

> > >   1 file changed, 2 insertions(+)
> > >
> > > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > > index 4fb5c20b13..bd30d3798c 100644
> > > --- a/xen/arch/arm/domain_build.c
> > > +++ b/xen/arch/arm/domain_build.c
> > > @@ -3418,6 +3418,8 @@ static void __init initrd_load(struct
> kernel_info
> > *kinfo)
> > >                                             initrd, len);
> > >       if ( res != 0 )
> > >           panic("Unable to copy the initrd in the hwdom memory\n");
> > > +
> > > +    iounmap(initrd);
> >
> > This looks good to me. But I am wondering whether using ioremap_wc() is
> > actually correct because we are reading the region. So it seems strang
> > to map it with write-combine.
> >
> > So I would consider to use ioremap_cache(). That said, this would be a
> > separate patch.
> >
> 
> Ok, we will try to use ioremap_cache and test it. If everything works
> well we will introduce a separate patch in next version.
> 

Or is it better to send a separate patch for this? Because I think we
might need something to address the v1 comments.

Cheers,
Wei Chen

> Cheers,
> Wei Chen
> 
> 
> > I think this wants to be in 4.17. This will avoid Xen to have two
> > mappings with different caching attribute (initrd is part of the RAM and
> > therefore directmap).
> >
> > Cheers,
> >
> > --
> > Julien Grall

[PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

[Wei Chen]

Current EFI boot services support of Arm64 could not
work well for Armv8-R64 system that only has MPU in
EL2. That is because EFI boot services may need some
relocation support or partial PIE/PIC support. But
these will not be supported in the initial stage of
porting Xen to MPU systems. So in this patch, we
disable EFI boot services support for Arm MPU systems.

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/Kconfig      | 2 +-
 xen/arch/arm/arm64/head.S | 8 ++++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 1fe5faf847..ad592367bd 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -7,7 +7,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
-	select ARM_EFI
+	select ARM_EFI if !HAS_MPU
 	select HAS_FAST_MULTIPLY
 
 config ARM
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index ad014716db..ccedf20dc7 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -172,8 +172,10 @@ efi_head:
         .byte   0x52
         .byte   0x4d
         .byte   0x64
-        .long   pe_header - efi_head        /* Offset to the PE header. */
-
+#ifndef CONFIG_ARM_EFI
+        .long   0                    /* 0 means no PE header. */
+#else
+        .long   pe_header - efi_head /* Offset to the PE header. */
         /*
          * Add the PE/COFF header to the file.  The address of this header
          * is at offset 0x3c in the file, and is part of Linux "Image"
@@ -279,6 +281,8 @@ section_table:
         .short  0                /* NumberOfLineNumbers  (0 for executables) */
         .long   0xe0500020       /* Characteristics (section flags) */
         .align  5
+#endif /* CONFIG_ARM_EFI */
+
 real_start:
         /* BSS should be zeroed when booting without EFI */
         mov   x26, #0                /* x26 := skip_zero_bss */
-- 
2.25.1

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> Current EFI boot services support of Arm64 could not
> work well for Armv8-R64 system that only has MPU in
> EL2. That is because EFI boot services may need some
> relocation support or partial PIE/PIC support.

I am a bit confused with argument. We have nothing in Xen today to deal 
with relocation/partial PIE/PIC support. So what is the exact problem? 
Is it because UEFI can load Xen anywwhere?

> But these will not be supported in the initial stage of
> porting Xen to MPU systems. So in this patch, we
> disable EFI boot services support for Arm MPU systems.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> ---
>   xen/arch/arm/Kconfig      | 2 +-
>   xen/arch/arm/arm64/head.S | 8 ++++++--
>   2 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 1fe5faf847..ad592367bd 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -7,7 +7,7 @@ config ARM_64
>   	def_bool y
>   	depends on !ARM_32
>   	select 64BIT
> -	select ARM_EFI
> +	select ARM_EFI if !HAS_MPU

I think it would make sense to allow ARM_EFI to be disabled even without 
the MPU support. So this would remove nearly 3K lines (just using wc -l 
*.c in the efi directories) for someone that don't need to boot using EFI.

>   	select HAS_FAST_MULTIPLY
>   
>   config ARM
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index ad014716db..ccedf20dc7 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -172,8 +172,10 @@ efi_head:
>           .byte   0x52
>           .byte   0x4d
>           .byte   0x64
> -        .long   pe_header - efi_head        /* Offset to the PE header. */
> -
> +#ifndef CONFIG_ARM_EFI
> +        .long   0                    /* 0 means no PE header. */
> +#else
> +        .long   pe_header - efi_head /* Offset to the PE header. */
>           /*
>            * Add the PE/COFF header to the file.  The address of this header
>            * is at offset 0x3c in the file, and is part of Linux "Image"
> @@ -279,6 +281,8 @@ section_table:
>           .short  0                /* NumberOfLineNumbers  (0 for executables) */
>           .long   0xe0500020       /* Characteristics (section flags) */
>           .align  5
> +#endif /* CONFIG_ARM_EFI */
> +
>   real_start:
>           /* BSS should be zeroed when booting without EFI */
>           mov   x26, #0                /* x26 := skip_zero_bss */

Shouldn't the function efi_xen_start be stubbed as well?

Cheers,

-- 
Julien Grall

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

[Julien Grall]

On 06/11/2022 19:12, Julien Grall wrote:
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
>> Current EFI boot services support of Arm64 could not
>> work well for Armv8-R64 system that only has MPU in
>> EL2. That is because EFI boot services may need some
>> relocation support or partial PIE/PIC support.
> 
> I am a bit confused with argument. We have nothing in Xen today to deal 
> with relocation/partial PIE/PIC support. So what is the exact problem? 
> Is it because UEFI can load Xen anywwhere?
> 
>> But these will not be supported in the initial stage of
>> porting Xen to MPU systems. So in this patch, we
>> disable EFI boot services support for Arm MPU systems.
>>
>> Signed-off-by: Wei Chen <wei.chen@arm.com>
>> ---
>>   xen/arch/arm/Kconfig      | 2 +-
>>   xen/arch/arm/arm64/head.S | 8 ++++++--
>>   2 files changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 1fe5faf847..ad592367bd 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -7,7 +7,7 @@ config ARM_64
>>       def_bool y
>>       depends on !ARM_32
>>       select 64BIT
>> -    select ARM_EFI
>> +    select ARM_EFI if !HAS_MPU
> 
> I think it would make sense to allow ARM_EFI to be disabled even without 
> the MPU support. So this would remove nearly 3K lines (just using wc -l 
> *.c in the efi directories) for someone that don't need to boot using EFI.
> 
>>       select HAS_FAST_MULTIPLY
>>   config ARM
>> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>> index ad014716db..ccedf20dc7 100644
>> --- a/xen/arch/arm/arm64/head.S
>> +++ b/xen/arch/arm/arm64/head.S
>> @@ -172,8 +172,10 @@ efi_head:
>>           .byte   0x52
>>           .byte   0x4d
>>           .byte   0x64
>> -        .long   pe_header - efi_head        /* Offset to the PE 
>> header. */
>> -
>> +#ifndef CONFIG_ARM_EFI
>> +        .long   0                    /* 0 means no PE header. */
>> +#else
>> +        .long   pe_header - efi_head /* Offset to the PE header. */
>>           /*
>>            * Add the PE/COFF header to the file.  The address of this 
>> header
>>            * is at offset 0x3c in the file, and is part of Linux "Image"
>> @@ -279,6 +281,8 @@ section_table:
>>           .short  0                /* NumberOfLineNumbers  (0 for 
>> executables) */
>>           .long   0xe0500020       /* Characteristics (section flags) */
>>           .align  5
>> +#endif /* CONFIG_ARM_EFI */
>> +
>>   real_start:
>>           /* BSS should be zeroed when booting without EFI */
>>           mov   x26, #0                /* x26 := skip_zero_bss */
> 
> Shouldn't the function efi_xen_start be stubbed as well?

Sorry, I mean protected rather than stubbed because there will be no 
user when !CONFIG_ARM_EFI.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 3:12
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU
> systems
> 
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > Current EFI boot services support of Arm64 could not
> > work well for Armv8-R64 system that only has MPU in
> > EL2. That is because EFI boot services may need some
> > relocation support or partial PIE/PIC support.
> 
> I am a bit confused with argument. We have nothing in Xen today to deal
> with relocation/partial PIE/PIC support. So what is the exact problem?
> Is it because UEFI can load Xen anywwhere?
> 
> > But these will not be supported in the initial stage of
> > porting Xen to MPU systems. So in this patch, we
> > disable EFI boot services support for Arm MPU systems.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > ---
> >   xen/arch/arm/Kconfig      | 2 +-
> >   xen/arch/arm/arm64/head.S | 8 ++++++--
> >   2 files changed, 7 insertions(+), 3 deletions(-)
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index 1fe5faf847..ad592367bd 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -7,7 +7,7 @@ config ARM_64
> >   	def_bool y
> >   	depends on !ARM_32
> >   	select 64BIT
> > -	select ARM_EFI
> > +	select ARM_EFI if !HAS_MPU
> 
> I think it would make sense to allow ARM_EFI to be disabled even without
> the MPU support. So this would remove nearly 3K lines (just using wc -l
> *.c in the efi directories) for someone that don't need to boot using EFI.
> 

Ok, how about address this comment in NUMA patch set#3 (Arm implementation),
because about making ARM_EFI invisible to users, we had some discussions
for NUMA RFC and V1.

> >   	select HAS_FAST_MULTIPLY
> >
> >   config ARM
> > diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> > index ad014716db..ccedf20dc7 100644
> > --- a/xen/arch/arm/arm64/head.S
> > +++ b/xen/arch/arm/arm64/head.S
> > @@ -172,8 +172,10 @@ efi_head:
> >           .byte   0x52
> >           .byte   0x4d
> >           .byte   0x64
> > -        .long   pe_header - efi_head        /* Offset to the PE header.
> */
> > -
> > +#ifndef CONFIG_ARM_EFI
> > +        .long   0                    /* 0 means no PE header. */
> > +#else
> > +        .long   pe_header - efi_head /* Offset to the PE header. */
> >           /*
> >            * Add the PE/COFF header to the file.  The address of this
> header
> >            * is at offset 0x3c in the file, and is part of Linux "Image"
> > @@ -279,6 +281,8 @@ section_table:
> >           .short  0                /* NumberOfLineNumbers  (0 for
> executables) */
> >           .long   0xe0500020       /* Characteristics (section flags) */
> >           .align  5
> > +#endif /* CONFIG_ARM_EFI */
> > +
> >   real_start:
> >           /* BSS should be zeroed when booting without EFI */
> >           mov   x26, #0                /* x26 := skip_zero_bss */
> 
> Shouldn't the function efi_xen_start be stubbed as well?
> 

Reply for your next email:
Yes, this is a good point, efi_xen_start should be protected. I will fix it.

Cheers,
Wei Chen


> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 03/11] xen/arm: disable EFI boot services for MPU systems

[Wei Chen]

Hi Julien,

On 2022/11/7 3:12, Julien Grall wrote:
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
>> Current EFI boot services support of Arm64 could not
>> work well for Armv8-R64 system that only has MPU in
>> EL2. That is because EFI boot services may need some
>> relocation support or partial PIE/PIC support.
> 
> I am a bit confused with argument. We have nothing in Xen today to deal 
> with relocation/partial PIE/PIC support. So what is the exact problem? 
> Is it because UEFI can load Xen anywwhere?
> 

Sorry, I had missed this comment. Yes, you're right we had
tried to boot Xen R82 Image from EFI loader, but it failed.
UEFI loader will load Xen to a random address, which is
not supported by Xen R82 now.

Cheers,
Wei Chen


>> But these will not be supported in the initial stage of
>> porting Xen to MPU systems. So in this patch, we
>> disable EFI boot services support for Arm MPU systems.
>>

[PATCH v6 04/11] xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA

[Wei Chen]

From Arm ARM Supplement of Armv8-R AArch64 (DDI 0600A) [1],
section D1.6.2 TLB maintenance instructions. We know that
Armv8-R AArch64 permits an implementation to cache stage 1
VMSAv8-64 and stage 2 PMSAv8-64 attributes as a common entry
for the Secure EL1&0 translation regime. But for Xen itself,
it's running with stage 1 PMSAv8-64 on Armv8-R AArch64. The
EL2 MPU updates for stage1 PMSAv8-64 will not be cached in
TLB entries. So we don't need any TLB invalidation for Xen
itself in EL2.

So in this patch, we use empty macros to stub Xen TLB helpers
for MPU system (PMSA), but still keep the Guest TLB helpers.
Because when a guest running in EL1 with VMSAv8-64 (MMU), guest
TLB invalidation is still needed. But we need some policy to
distinguish MPU and MMU guest, this will be done in guest
support of Armv8-R64 later.

[1] https://developer.arm.com/documentation/ddi0600/ac

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/include/asm/arm64/flushtlb.h | 25 +++++++++++++++++++++++
 xen/arch/arm/include/asm/flushtlb.h       | 22 ++++++++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/xen/arch/arm/include/asm/arm64/flushtlb.h b/xen/arch/arm/include/asm/arm64/flushtlb.h
index 7c54315187..fe445f6831 100644
--- a/xen/arch/arm/include/asm/arm64/flushtlb.h
+++ b/xen/arch/arm/include/asm/arm64/flushtlb.h
@@ -51,6 +51,8 @@ TLB_HELPER(flush_all_guests_tlb_local, alle1);
 /* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */
 TLB_HELPER(flush_all_guests_tlb, alle1is);
 
+#ifndef CONFIG_HAS_MPU
+
 /* Flush all hypervisor mappings from the TLB of the local processor. */
 TLB_HELPER(flush_xen_tlb_local, alle2);
 
@@ -66,6 +68,29 @@ static inline void __flush_xen_tlb_one(vaddr_t va)
     asm volatile("tlbi vae2is, %0;" : : "r" (va>>PAGE_SHIFT) : "memory");
 }
 
+#else
+
+/*
+ * When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU
+ * updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we
+ * don't need any TLB invalidation for Xen itself in EL2. See Arm ARM
+ * Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance
+ * instructions for more details.
+ */
+static inline void flush_xen_tlb_local(void)
+{
+}
+
+static inline void  __flush_xen_tlb_one_local(vaddr_t va)
+{
+}
+
+static inline void __flush_xen_tlb_one(vaddr_t va)
+{
+}
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */
 /*
  * Local variables:
diff --git a/xen/arch/arm/include/asm/flushtlb.h b/xen/arch/arm/include/asm/flushtlb.h
index 125a141975..4b8bf65281 100644
--- a/xen/arch/arm/include/asm/flushtlb.h
+++ b/xen/arch/arm/include/asm/flushtlb.h
@@ -28,6 +28,7 @@ static inline void page_set_tlbflush_timestamp(struct page_info *page)
 /* Flush specified CPUs' TLBs */
 void arch_flush_tlb_mask(const cpumask_t *mask);
 
+#ifndef CONFIG_HAS_MPU
 /*
  * Flush a range of VA's hypervisor mappings from the TLB of the local
  * processor.
@@ -66,6 +67,27 @@ static inline void flush_xen_tlb_range_va(vaddr_t va,
     isb();
 }
 
+#else
+
+/*
+ * When Xen is running with stage 1 PMSAv8-64 on MPU systems. The EL2 MPU
+ * updates for stage1 PMSAv8-64 will not be cached in TLB entries. So we
+ * don't need any TLB invalidation for Xen itself in EL2. See Arm ARM
+ * Supplement of Armv8-R AArch64 (DDI 0600A), section D1.6.2 TLB maintenance
+ * instructions for more details.
+ */
+static inline void flush_xen_tlb_range_va_local(vaddr_t va,
+                                                unsigned long size)
+{
+}
+
+static inline void flush_xen_tlb_range_va(vaddr_t va,
+                                          unsigned long size)
+{
+}
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_ARM_FLUSHTLB_H__ */
 /*
  * Local variables:
-- 
2.25.1

[PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Wei Chen]

On Armv8-A, Xen has a fixed virtual start address (link address
too) for all Armv8-A platforms. In an MMU based system, Xen can
map its loaded address to this virtual start address. So, on
Armv8-A platforms, the Xen start address does not need to be
configurable. But on Armv8-R platforms, there is no MMU to map
loaded address to a fixed virtual address and different platforms
will have very different address space layout. So Xen cannot use
a fixed physical address on MPU based system and need to have it
configurable.

So in this patch, we reuse the existing arm/platforms to store
Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
kind of FVP BaseR platform's parameters. So we define default
`XEN_START_ADDRESS` for FVP BaseR in its platform file.

We also introduce one Kconfig option for users to override the
default Xen start address of selected platform, if they think
the default address doesn't suit their scenarios. For this
Kconfig option, we use an unaligned address "0xffffffff" as the
default value to indicate that users haven't used a customized
Xen start address.

And as we introduced Armv8-R platforms to Xen, that means the
existed Arm64 platforms should not be listed in Armv8-R platform
list, so we add !ARM_V8R dependency for these platforms.

Signed-off-by: Wei Chen <wei.chen@arm.com>
Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
---
 xen/arch/arm/Kconfig                           | 11 +++++++++++
 xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
 xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
 3 files changed, 38 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index ad592367bd..ac276307d6 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -138,6 +138,17 @@ config TEE
 	  This option enables generic TEE mediators support. It allows guests
 	  to access real TEE via one of TEE mediators implemented in XEN.
 
+config XEN_START_ADDRESS
+	hex "Xen start address: keep default to use platform defined address"
+	default 0xFFFFFFFF
+	depends on HAS_MPU
+	help
+	  This option allows to set the customized address at which Xen will be
+	  linked on MPU systems. This address must be aligned to a page size.
+	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
+	  customized this address, and Xen use use the default value that has
+	  been defined in platform files.
+
 source "arch/arm/tee/Kconfig"
 
 config STATIC_SHM
diff --git a/xen/arch/arm/include/asm/platforms/fvp_baser.h b/xen/arch/arm/include/asm/platforms/fvp_baser.h
new file mode 100644
index 0000000000..9450a411a9
--- /dev/null
+++ b/xen/arch/arm/include/asm/platforms/fvp_baser.h
@@ -0,0 +1,14 @@
+#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
+#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
+
+/*
+ * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
+ * we will use platform defined default address.
+ */
+#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
+#define XEN_START_ADDRESS 0x200000
+#else
+#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
+#endif
+
+#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
diff --git a/xen/arch/arm/platforms/Kconfig b/xen/arch/arm/platforms/Kconfig
index c93a6b2756..0904793a0b 100644
--- a/xen/arch/arm/platforms/Kconfig
+++ b/xen/arch/arm/platforms/Kconfig
@@ -1,6 +1,7 @@
 choice
 	prompt "Platform Support"
 	default ALL_PLAT
+	default FVP_BASER if ARM_V8R
 	---help---
 	Choose which hardware platform to enable in Xen.
 
@@ -8,13 +9,14 @@ choice
 
 config ALL_PLAT
 	bool "All Platforms"
+	depends on !ARM_V8R
 	---help---
 	Enable support for all available hardware platforms. It doesn't
 	automatically select any of the related drivers.
 
 config QEMU
 	bool "QEMU aarch virt machine support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select GICV3
 	select HAS_PL011
 	---help---
@@ -23,7 +25,7 @@ config QEMU
 
 config RCAR3
 	bool "Renesas RCar3 support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select HAS_SCIF
 	select IPMMU_VMSA
 	---help---
@@ -31,14 +33,22 @@ config RCAR3
 
 config MPSOC
 	bool "Xilinx Ultrascale+ MPSoC support"
-	depends on ARM_64
+	depends on ARM_64 && !ARM_V8R
 	select HAS_CADENCE_UART
 	select ARM_SMMU
 	---help---
 	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
 
+config FVP_BASER
+	bool "Fixed Virtual Platform BaseR support"
+	depends on ARM_V8R
+	help
+	  Enable platform specific configurations for Fixed Virtual
+	  Platform BaseR
+
 config NO_PLAT
 	bool "No Platforms"
+	depends on !ARM_V8R
 	---help---
 	Do not enable specific support for any platform.
 
-- 
2.25.1

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Julien Grall]

On 04/11/2022 10:07, Wei Chen wrote:
> On Armv8-A, Xen has a fixed virtual start address (link address
> too) for all Armv8-A platforms. In an MMU based system, Xen can
> map its loaded address to this virtual start address. So, on
> Armv8-A platforms, the Xen start address does not need to be
> configurable. But on Armv8-R platforms, there is no MMU to map
> loaded address to a fixed virtual address and different platforms
> will have very different address space layout. So Xen cannot use
> a fixed physical address on MPU based system and need to have it
> configurable.
> 
> So in this patch, we reuse the existing arm/platforms to store
> Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> kind of FVP BaseR platform's parameters. So we define default
> `XEN_START_ADDRESS` for FVP BaseR in its platform file.
> 
> We also introduce one Kconfig option for users to override the
> default Xen start address of selected platform, if they think
> the default address doesn't suit their scenarios. For this
> Kconfig option, we use an unaligned address "0xffffffff" as the
> default value to indicate that users haven't used a customized
> Xen start address.
> 
> And as we introduced Armv8-R platforms to Xen, that means the
> existed Arm64 platforms should not be listed in Armv8-R platform
> list, so we add !ARM_V8R dependency for these platforms.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> ---
>   xen/arch/arm/Kconfig                           | 11 +++++++++++
>   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++

I looked at the content of fvp_baser.h after this series is applied. 
There are a bit of boiler plate that I expect to be part for other 
platforms. In particular...

>   xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
>   3 files changed, 38 insertions(+), 3 deletions(-)
>   create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
> 
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index ad592367bd..ac276307d6 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -138,6 +138,17 @@ config TEE
>   	  This option enables generic TEE mediators support. It allows guests
>   	  to access real TEE via one of TEE mediators implemented in XEN.
>   
> +config XEN_START_ADDRESS
> +	hex "Xen start address: keep default to use platform defined address"
> +	default 0xFFFFFFFF

... this default value will need to be tested everywhere. At least for 
now, I think you can avoid the per platform header by using the Kconfig 
to select the proper address (see the config for selecting early printk 
address).

This will also avoids to use an invalid value here.

> +	depends on HAS_MPU
> +	help
> +	  This option allows to set the customized address at which Xen will be
> +	  linked on MPU systems. This address must be aligned to a page size.
> +	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
> +	  customized this address, and Xen use use the default value that has
> +	  been defined in platform files.
> +
>   source "arch/arm/tee/Kconfig"
>   
>   config STATIC_SHM
> diff --git a/xen/arch/arm/include/asm/platforms/fvp_baser.h b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> new file mode 100644
> index 0000000000..9450a411a9
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> @@ -0,0 +1,14 @@
> +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
> +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
> +
> +/*
> + * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
> + * we will use platform defined default address.
> + */
> +#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
> +#define XEN_START_ADDRESS 0x200000
> +#else
> +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
> +#endif
> +
> +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
> diff --git a/xen/arch/arm/platforms/Kconfig b/xen/arch/arm/platforms/Kconfig
> index c93a6b2756..0904793a0b 100644
> --- a/xen/arch/arm/platforms/Kconfig
> +++ b/xen/arch/arm/platforms/Kconfig
> @@ -1,6 +1,7 @@
>   choice
>   	prompt "Platform Support"
>   	default ALL_PLAT
> +	default FVP_BASER if ARM_V8R

Is there any reason to create a new Kconfig rather than using MPU?

>   	---help---
>   	Choose which hardware platform to enable in Xen.
>   
> @@ -8,13 +9,14 @@ choice
>   
>   config ALL_PLAT
>   	bool "All Platforms"
> +	depends on !ARM_V8R
>   	---help---
>   	Enable support for all available hardware platforms. It doesn't
>   	automatically select any of the related drivers.
>   
>   config QEMU
>   	bool "QEMU aarch virt machine support"
> -	depends on ARM_64
> +	depends on ARM_64 && !ARM_V8R
>   	select GICV3
>   	select HAS_PL011
>   	---help---
> @@ -23,7 +25,7 @@ config QEMU
>   
>   config RCAR3
>   	bool "Renesas RCar3 support"
> -	depends on ARM_64
> +	depends on ARM_64 && !ARM_V8R
>   	select HAS_SCIF
>   	select IPMMU_VMSA
>   	---help---
> @@ -31,14 +33,22 @@ config RCAR3
>   
>   config MPSOC
>   	bool "Xilinx Ultrascale+ MPSoC support"
> -	depends on ARM_64
> +	depends on ARM_64 && !ARM_V8R
>   	select HAS_CADENCE_UART
>   	select ARM_SMMU
>   	---help---
>   	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
>   
> +config FVP_BASER
> +	bool "Fixed Virtual Platform BaseR support"
> +	depends on ARM_V8R
> +	help
> +	  Enable platform specific configurations for Fixed Virtual
> +	  Platform BaseR
> +
>   config NO_PLAT
>   	bool "No Platforms"
> +	depends on !ARM_V8R
>   	---help---
>   	Do not enable specific support for any platform.
>   

Cheers,

-- 
Julien Grall

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 3:20
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP
> BaseR platform
> 
> 
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > On Armv8-A, Xen has a fixed virtual start address (link address
> > too) for all Armv8-A platforms. In an MMU based system, Xen can
> > map its loaded address to this virtual start address. So, on
> > Armv8-A platforms, the Xen start address does not need to be
> > configurable. But on Armv8-R platforms, there is no MMU to map
> > loaded address to a fixed virtual address and different platforms
> > will have very different address space layout. So Xen cannot use
> > a fixed physical address on MPU based system and need to have it
> > configurable.
> >
> > So in this patch, we reuse the existing arm/platforms to store
> > Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> > kind of FVP BaseR platform's parameters. So we define default
> > `XEN_START_ADDRESS` for FVP BaseR in its platform file.
> >
> > We also introduce one Kconfig option for users to override the
> > default Xen start address of selected platform, if they think
> > the default address doesn't suit their scenarios. For this
> > Kconfig option, we use an unaligned address "0xffffffff" as the
> > default value to indicate that users haven't used a customized
> > Xen start address.
> >
> > And as we introduced Armv8-R platforms to Xen, that means the
> > existed Arm64 platforms should not be listed in Armv8-R platform
> > list, so we add !ARM_V8R dependency for these platforms.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> > ---
> >   xen/arch/arm/Kconfig                           | 11 +++++++++++
> >   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
> 
> I looked at the content of fvp_baser.h after this series is applied.
> There are a bit of boiler plate that I expect to be part for other
> platforms. In particular...
> 
> >   xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
> >   3 files changed, 38 insertions(+), 3 deletions(-)
> >   create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index ad592367bd..ac276307d6 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -138,6 +138,17 @@ config TEE
> >   	  This option enables generic TEE mediators support. It allows
> guests
> >   	  to access real TEE via one of TEE mediators implemented in XEN.
> >
> > +config XEN_START_ADDRESS
> > +	hex "Xen start address: keep default to use platform defined
> address"
> > +	default 0xFFFFFFFF
> 
> ... this default value will need to be tested everywhere. At least for
> now, I think you can avoid the per platform header by using the Kconfig
> to select the proper address (see the config for selecting early printk
> address).
> 
> This will also avoids to use an invalid value here.
> 

We had considered to use Kconfig to define the start addresses of v8R64
platforms (prompt users to input the address). But we also want to provide
a default start address for each platform (Discussed in [1], header for
default value, Kconfig option for customized address).

We also had thought to use Kconfig to define a default start address
for each platform like what we had done for early printk in RFC[2].
But this method has been deprecated.

So if we don’t use header files, just use the Kconfig, we can't
provide a default start address for platforms, and have to force users
to enter the start address.

[1] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[2] https://gitlab.com/xen-project/fusa/xen-integration/-/blob/integration/mpu/xen/arch/arm/Kconfig.debug

> > +	depends on HAS_MPU
> > +	help
> > +	  This option allows to set the customized address at which Xen will
> be
> > +	  linked on MPU systems. This address must be aligned to a page size.
> > +	  Use 0xFFFFFFFF as the default value to indicate that user hasn't
> > +	  customized this address, and Xen use use the default value that
> has
> > +	  been defined in platform files.
> > +
> >   source "arch/arm/tee/Kconfig"
> >
> >   config STATIC_SHM
> > diff --git a/xen/arch/arm/include/asm/platforms/fvp_baser.h
> b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> > new file mode 100644
> > index 0000000000..9450a411a9
> > --- /dev/null
> > +++ b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> > @@ -0,0 +1,14 @@
> > +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
> > +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
> > +
> > +/*
> > + * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
> > + * we will use platform defined default address.
> > + */
> > +#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
> > +#define XEN_START_ADDRESS 0x200000
> > +#else
> > +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
> > +#endif
> > +
> > +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
> > diff --git a/xen/arch/arm/platforms/Kconfig
> b/xen/arch/arm/platforms/Kconfig
> > index c93a6b2756..0904793a0b 100644
> > --- a/xen/arch/arm/platforms/Kconfig
> > +++ b/xen/arch/arm/platforms/Kconfig
> > @@ -1,6 +1,7 @@
> >   choice
> >   	prompt "Platform Support"
> >   	default ALL_PLAT
> > +	default FVP_BASER if ARM_V8R
> 
> Is there any reason to create a new Kconfig rather than using MPU?
> 

Did you mean FVP_BASER? If yes, we want to give each board a MACRO
to indicate its specific configurations. In current series, this MACRO
only be used for board specific start address.

If you meant Armv8R, that's because Armv8R does not equal to MPU.
We only allow Armv8R code to detect MPU features. MPU is configurable
In EL2.

Cheers,
Wei Chen

> >   	---help---
> >   	Choose which hardware platform to enable in Xen.
> >
> > @@ -8,13 +9,14 @@ choice
> >
> >   config ALL_PLAT
> >   	bool "All Platforms"
> > +	depends on !ARM_V8R
> >   	---help---
> >   	Enable support for all available hardware platforms. It doesn't
> >   	automatically select any of the related drivers.
> >
> >   config QEMU
> >   	bool "QEMU aarch virt machine support"
> > -	depends on ARM_64
> > +	depends on ARM_64 && !ARM_V8R
> >   	select GICV3
> >   	select HAS_PL011
> >   	---help---
> > @@ -23,7 +25,7 @@ config QEMU
> >
> >   config RCAR3
> >   	bool "Renesas RCar3 support"
> > -	depends on ARM_64
> > +	depends on ARM_64 && !ARM_V8R
> >   	select HAS_SCIF
> >   	select IPMMU_VMSA
> >   	---help---
> > @@ -31,14 +33,22 @@ config RCAR3
> >
> >   config MPSOC
> >   	bool "Xilinx Ultrascale+ MPSoC support"
> > -	depends on ARM_64
> > +	depends on ARM_64 && !ARM_V8R
> >   	select HAS_CADENCE_UART
> >   	select ARM_SMMU
> >   	---help---
> >   	Enable all the required drivers for Xilinx Ultrascale+ MPSoC
> >
> > +config FVP_BASER
> > +	bool "Fixed Virtual Platform BaseR support"
> > +	depends on ARM_V8R
> > +	help
> > +	  Enable platform specific configurations for Fixed Virtual
> > +	  Platform BaseR
> > +
> >   config NO_PLAT
> >   	bool "No Platforms"
> > +	depends on !ARM_V8R
> >   	---help---
> >   	Do not enable specific support for any platform.
> >
> 
> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Julien Grall]

On 09/11/2022 04:55, Wei Chen wrote:
> Hi Julien,

Hi Wei,

> 
>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: 2022年11月7日 3:20
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
>> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
>> Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP
>> BaseR platform
>>
>>
>>
>> On 04/11/2022 10:07, Wei Chen wrote:
>>> On Armv8-A, Xen has a fixed virtual start address (link address
>>> too) for all Armv8-A platforms. In an MMU based system, Xen can
>>> map its loaded address to this virtual start address. So, on
>>> Armv8-A platforms, the Xen start address does not need to be
>>> configurable. But on Armv8-R platforms, there is no MMU to map
>>> loaded address to a fixed virtual address and different platforms
>>> will have very different address space layout. So Xen cannot use
>>> a fixed physical address on MPU based system and need to have it
>>> configurable.
>>>
>>> So in this patch, we reuse the existing arm/platforms to store
>>> Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
>>> kind of FVP BaseR platform's parameters. So we define default
>>> `XEN_START_ADDRESS` for FVP BaseR in its platform file.
>>>
>>> We also introduce one Kconfig option for users to override the
>>> default Xen start address of selected platform, if they think
>>> the default address doesn't suit their scenarios. For this
>>> Kconfig option, we use an unaligned address "0xffffffff" as the
>>> default value to indicate that users haven't used a customized
>>> Xen start address.
>>>
>>> And as we introduced Armv8-R platforms to Xen, that means the
>>> existed Arm64 platforms should not be listed in Armv8-R platform
>>> list, so we add !ARM_V8R dependency for these platforms.
>>>
>>> Signed-off-by: Wei Chen <wei.chen@arm.com>
>>> Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
>>> ---
>>>    xen/arch/arm/Kconfig                           | 11 +++++++++++
>>>    xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
>>
>> I looked at the content of fvp_baser.h after this series is applied.
>> There are a bit of boiler plate that I expect to be part for other
>> platforms. In particular...
>>
>>>    xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
>>>    3 files changed, 38 insertions(+), 3 deletions(-)
>>>    create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
>>>
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index ad592367bd..ac276307d6 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -138,6 +138,17 @@ config TEE
>>>    	  This option enables generic TEE mediators support. It allows
>> guests
>>>    	  to access real TEE via one of TEE mediators implemented in XEN.
>>>
>>> +config XEN_START_ADDRESS
>>> +	hex "Xen start address: keep default to use platform defined
>> address"
>>> +	default 0xFFFFFFFF
>>
>> ... this default value will need to be tested everywhere. At least for
>> now, I think you can avoid the per platform header by using the Kconfig
>> to select the proper address (see the config for selecting early printk
>> address).
>>
>> This will also avoids to use an invalid value here.
>>
> 
> We had considered to use Kconfig to define the start addresses of v8R64
> platforms (prompt users to input the address). But we also want to provide
> a default start address for each platform (Discussed in [1], header for
> default value, Kconfig option for customized address).
Why do you want to provide a default value? And how it is guaranteed 
that it will work for most of the users?

> 
> We also had thought to use Kconfig to define a default start address
> for each platform like what we had done for early printk in RFC[2].
> But this method has been deprecated.

Most of the current Xen is board agnostic except the UART. We push back 
the addition of new one because the address can be found in the firmware 
table and I wanted to avoid increase the number of option (there are 
dozens of platform out...).

> 
> So if we don’t use header files, just use the Kconfig, we can't
> provide a default start address for platforms, and have to force users
> to enter the start address.

I am not sure I see the problem to force the user to enter the start 
address. My worry with per-platform default value is we end up to force 
each vendor to provide an header in order to boot Xen.

I think it would be better to provide a config tailored for that 
platform (whether it is part of Xen can be debatable). This would allow 
a user to try a release Xen on their platform with zero changes in the code.

>>> diff --git a/xen/arch/arm/platforms/Kconfig
>> b/xen/arch/arm/platforms/Kconfig
>>> index c93a6b2756..0904793a0b 100644
>>> --- a/xen/arch/arm/platforms/Kconfig
>>> +++ b/xen/arch/arm/platforms/Kconfig
>>> @@ -1,6 +1,7 @@
>>>    choice
>>>    	prompt "Platform Support"
>>>    	default ALL_PLAT
>>> +	default FVP_BASER if ARM_V8R
>>
>> Is there any reason to create a new Kconfig rather than using MPU?
>>
> 
> Did you mean FVP_BASER? If yes, we want to give each board a MACRO
> to indicate its specific configurations. In current series, this MACRO
> only be used for board specific start address.

See above for this.

> 
> If you meant Armv8R, that's because Armv8R does not equal to MPU.

I am not entirely sure to understand. Are you saying that an existing 
Xen can boot on Armv8R?

Cheers,

-- 
Julien Grall

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5938 bytes --]

On Wed, 9 Nov 2022, Julien Grall wrote:
> > > -----Original Message-----
> > > From: Julien Grall <julien@xen.org>
> > > Sent: 2022年11月7日 3:20
> > > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> > > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > > <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> > > Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP
> > > BaseR platform
> > > 
> > > 
> > > 
> > > On 04/11/2022 10:07, Wei Chen wrote:
> > > > On Armv8-A, Xen has a fixed virtual start address (link address
> > > > too) for all Armv8-A platforms. In an MMU based system, Xen can
> > > > map its loaded address to this virtual start address. So, on
> > > > Armv8-A platforms, the Xen start address does not need to be
> > > > configurable. But on Armv8-R platforms, there is no MMU to map
> > > > loaded address to a fixed virtual address and different platforms
> > > > will have very different address space layout. So Xen cannot use
> > > > a fixed physical address on MPU based system and need to have it
> > > > configurable.
> > > > 
> > > > So in this patch, we reuse the existing arm/platforms to store
> > > > Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> > > > kind of FVP BaseR platform's parameters. So we define default
> > > > `XEN_START_ADDRESS` for FVP BaseR in its platform file.
> > > > 
> > > > We also introduce one Kconfig option for users to override the
> > > > default Xen start address of selected platform, if they think
> > > > the default address doesn't suit their scenarios. For this
> > > > Kconfig option, we use an unaligned address "0xffffffff" as the
> > > > default value to indicate that users haven't used a customized
> > > > Xen start address.
> > > > 
> > > > And as we introduced Armv8-R platforms to Xen, that means the
> > > > existed Arm64 platforms should not be listed in Armv8-R platform
> > > > list, so we add !ARM_V8R dependency for these platforms.
> > > > 
> > > > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > > > Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> > > > ---
> > > >    xen/arch/arm/Kconfig                           | 11 +++++++++++
> > > >    xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
> > > 
> > > I looked at the content of fvp_baser.h after this series is applied.
> > > There are a bit of boiler plate that I expect to be part for other
> > > platforms. In particular...
> > > 
> > > >    xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
> > > >    3 files changed, 38 insertions(+), 3 deletions(-)
> > > >    create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
> > > > 
> > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > index ad592367bd..ac276307d6 100644
> > > > --- a/xen/arch/arm/Kconfig
> > > > +++ b/xen/arch/arm/Kconfig
> > > > @@ -138,6 +138,17 @@ config TEE
> > > >    	  This option enables generic TEE mediators support. It allows
> > > guests
> > > >    	  to access real TEE via one of TEE mediators implemented in
> > > > XEN.
> > > > 
> > > > +config XEN_START_ADDRESS
> > > > +	hex "Xen start address: keep default to use platform defined
> > > address"
> > > > +	default 0xFFFFFFFF
> > > 
> > > ... this default value will need to be tested everywhere. At least for
> > > now, I think you can avoid the per platform header by using the Kconfig
> > > to select the proper address (see the config for selecting early printk
> > > address).
> > > 
> > > This will also avoids to use an invalid value here.
> > > 
> > 
> > We had considered to use Kconfig to define the start addresses of v8R64
> > platforms (prompt users to input the address). But we also want to provide
> > a default start address for each platform (Discussed in [1], header for
> > default value, Kconfig option for customized address).
> Why do you want to provide a default value? And how it is guaranteed that it
> will work for most of the users?
> 
> > 
> > We also had thought to use Kconfig to define a default start address
> > for each platform like what we had done for early printk in RFC[2].
> > But this method has been deprecated.
> 
> Most of the current Xen is board agnostic except the UART. We push back the
> addition of new one because the address can be found in the firmware table and
> I wanted to avoid increase the number of option (there are dozens of platform
> out...).
> 
> > 
> > So if we don’t use header files, just use the Kconfig, we can't
> > provide a default start address for platforms, and have to force users
> > to enter the start address.
> 
> I am not sure I see the problem to force the user to enter the start address.
> My worry with per-platform default value is we end up to force each vendor to
> provide an header in order to boot Xen.
> 
> I think it would be better to provide a config tailored for that platform
> (whether it is part of Xen can be debatable). This would allow a user to try a
> release Xen on their platform with zero changes in the code.

I agree with Julien, especially on this last point.

Of course we need a default configuration for a given platform, we don't
want every user of the same platform to have to go and look at the
manual to find the right address to use.

The question is where to put the per-platform default value. The kconfig
"default" keyword is not great for that and it is not realistic to have
a single address that works everywhere.

Instead, we could have a prepopulated kconfig under
xen/arch/arm/configs, or something under ImageBuilder, or maybe expand
the existing "Platform Support" kconfig menu.

If this was just XEN_START_ADDRESS, I would suggest to keep it in
xen.git somewhere. But given that there are a few addresses and sizes to
provide/calculate for Xen on MPU to work, using ImageBuilder could be a
good idea.

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Wei Chen]

Hi Stefano, Julien,

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
> Stefano Stabellini
> Sent: 2022年11月11日 6:13
> To: Julien Grall <julien@xen.org>
> Cc: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org; nd
> <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP
> BaseR platform
> 
> On Wed, 9 Nov 2022, Julien Grall wrote:
> > > > -----Original Message-----
> > > > From: Julien Grall <julien@xen.org>
> > > > Sent: 2022年11月7日 3:20
> > > > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > > > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Bertrand
> > > > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > > > <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> > > > Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for
> FVP
> > > > BaseR platform
> > > >
> > > >
> > > >
> > > > On 04/11/2022 10:07, Wei Chen wrote:
> > > > > On Armv8-A, Xen has a fixed virtual start address (link address
> > > > > too) for all Armv8-A platforms. In an MMU based system, Xen can
> > > > > map its loaded address to this virtual start address. So, on
> > > > > Armv8-A platforms, the Xen start address does not need to be
> > > > > configurable. But on Armv8-R platforms, there is no MMU to map
> > > > > loaded address to a fixed virtual address and different platforms
> > > > > will have very different address space layout. So Xen cannot use
> > > > > a fixed physical address on MPU based system and need to have it
> > > > > configurable.
> > > > >
> > > > > So in this patch, we reuse the existing arm/platforms to store
> > > > > Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> > > > > kind of FVP BaseR platform's parameters. So we define default
> > > > > `XEN_START_ADDRESS` for FVP BaseR in its platform file.
> > > > >
> > > > > We also introduce one Kconfig option for users to override the
> > > > > default Xen start address of selected platform, if they think
> > > > > the default address doesn't suit their scenarios. For this
> > > > > Kconfig option, we use an unaligned address "0xffffffff" as the
> > > > > default value to indicate that users haven't used a customized
> > > > > Xen start address.
> > > > >
> > > > > And as we introduced Armv8-R platforms to Xen, that means the
> > > > > existed Arm64 platforms should not be listed in Armv8-R platform
> > > > > list, so we add !ARM_V8R dependency for these platforms.
> > > > >
> > > > > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > > > > Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> > > > > ---
> > > > >    xen/arch/arm/Kconfig                           | 11 +++++++++++
> > > > >    xen/arch/arm/include/asm/platforms/fvp_baser.h | 14
> ++++++++++++++
> > > >
> > > > I looked at the content of fvp_baser.h after this series is applied.
> > > > There are a bit of boiler plate that I expect to be part for other
> > > > platforms. In particular...
> > > >
> > > > >    xen/arch/arm/platforms/Kconfig                 | 16
> +++++++++++++---
> > > > >    3 files changed, 38 insertions(+), 3 deletions(-)
> > > > >    create mode 100644
> xen/arch/arm/include/asm/platforms/fvp_baser.h
> > > > >
> > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > index ad592367bd..ac276307d6 100644
> > > > > --- a/xen/arch/arm/Kconfig
> > > > > +++ b/xen/arch/arm/Kconfig
> > > > > @@ -138,6 +138,17 @@ config TEE
> > > > >    	  This option enables generic TEE mediators support. It allows
> > > > guests
> > > > >    	  to access real TEE via one of TEE mediators implemented in
> > > > > XEN.
> > > > >
> > > > > +config XEN_START_ADDRESS
> > > > > +	hex "Xen start address: keep default to use platform defined
> > > > address"
> > > > > +	default 0xFFFFFFFF
> > > >
> > > > ... this default value will need to be tested everywhere. At least
> for
> > > > now, I think you can avoid the per platform header by using the
> Kconfig
> > > > to select the proper address (see the config for selecting early
> printk
> > > > address).
> > > >
> > > > This will also avoids to use an invalid value here.
> > > >
> > >
> > > We had considered to use Kconfig to define the start addresses of
> v8R64
> > > platforms (prompt users to input the address). But we also want to
> provide
> > > a default start address for each platform (Discussed in [1], header
> for
> > > default value, Kconfig option for customized address).
> > Why do you want to provide a default value? And how it is guaranteed
> that it
> > will work for most of the users?
> >
> > >
> > > We also had thought to use Kconfig to define a default start address
> > > for each platform like what we had done for early printk in RFC[2].
> > > But this method has been deprecated.
> >
> > Most of the current Xen is board agnostic except the UART. We push back
> the
> > addition of new one because the address can be found in the firmware
> table and
> > I wanted to avoid increase the number of option (there are dozens of
> platform
> > out...).
> >
> > >
> > > So if we don’t use header files, just use the Kconfig, we can't
> > > provide a default start address for platforms, and have to force users
> > > to enter the start address.
> >
> > I am not sure I see the problem to force the user to enter the start
> address.
> > My worry with per-platform default value is we end up to force each
> vendor to
> > provide an header in order to boot Xen.
> >
> > I think it would be better to provide a config tailored for that
> platform
> > (whether it is part of Xen can be debatable). This would allow a user to
> try a
> > release Xen on their platform with zero changes in the code.
> 
> I agree with Julien, especially on this last point.
> 
> Of course we need a default configuration for a given platform, we don't
> want every user of the same platform to have to go and look at the
> manual to find the right address to use.
> 
> The question is where to put the per-platform default value. The kconfig
> "default" keyword is not great for that and it is not realistic to have
> a single address that works everywhere.
> 
> Instead, we could have a prepopulated kconfig under
> xen/arch/arm/configs, or something under ImageBuilder, or maybe expand

Do you mean we can keep a config like armv8r_fvp_baser_config in
xen/arch/arm/configs for users to generate a default config?
If yes I think this method might be better for now. And about ImageBuilder
solution we can do it after MPU support be merged?

Cheers,
Wei Chen

> the existing "Platform Support" kconfig menu.
> 
> If this was just XEN_START_ADDRESS, I would suggest to keep it in
> xen.git somewhere. But given that there are a few addresses and sizes to
> provide/calculate for Xen on MPU to work, using ImageBuilder could be a
> good idea.

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7111 bytes --]

On Fri, 11 Nov 2022, Wei Chen wrote:
> Hi Stefano, Julien,
> 
> > -----Original Message-----
> > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
> > Stefano Stabellini
> > Sent: 2022年11月11日 6:13
> > To: Julien Grall <julien@xen.org>
> > Cc: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org; nd
> > <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> > Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP
> > BaseR platform
> > 
> > On Wed, 9 Nov 2022, Julien Grall wrote:
> > > > > -----Original Message-----
> > > > > From: Julien Grall <julien@xen.org>
> > > > > Sent: 2022年11月7日 3:20
> > > > > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > > > > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> > Bertrand
> > > > > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > > > > <Volodymyr_Babchuk@epam.com>; Jiamei Xie <Jiamei.Xie@arm.com>
> > > > > Subject: Re: [PATCH v6 05/11] xen/arm: define Xen start address for
> > FVP
> > > > > BaseR platform
> > > > >
> > > > >
> > > > >
> > > > > On 04/11/2022 10:07, Wei Chen wrote:
> > > > > > On Armv8-A, Xen has a fixed virtual start address (link address
> > > > > > too) for all Armv8-A platforms. In an MMU based system, Xen can
> > > > > > map its loaded address to this virtual start address. So, on
> > > > > > Armv8-A platforms, the Xen start address does not need to be
> > > > > > configurable. But on Armv8-R platforms, there is no MMU to map
> > > > > > loaded address to a fixed virtual address and different platforms
> > > > > > will have very different address space layout. So Xen cannot use
> > > > > > a fixed physical address on MPU based system and need to have it
> > > > > > configurable.
> > > > > >
> > > > > > So in this patch, we reuse the existing arm/platforms to store
> > > > > > Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> > > > > > kind of FVP BaseR platform's parameters. So we define default
> > > > > > `XEN_START_ADDRESS` for FVP BaseR in its platform file.
> > > > > >
> > > > > > We also introduce one Kconfig option for users to override the
> > > > > > default Xen start address of selected platform, if they think
> > > > > > the default address doesn't suit their scenarios. For this
> > > > > > Kconfig option, we use an unaligned address "0xffffffff" as the
> > > > > > default value to indicate that users haven't used a customized
> > > > > > Xen start address.
> > > > > >
> > > > > > And as we introduced Armv8-R platforms to Xen, that means the
> > > > > > existed Arm64 platforms should not be listed in Armv8-R platform
> > > > > > list, so we add !ARM_V8R dependency for these platforms.
> > > > > >
> > > > > > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > > > > > Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> > > > > > ---
> > > > > >    xen/arch/arm/Kconfig                           | 11 +++++++++++
> > > > > >    xen/arch/arm/include/asm/platforms/fvp_baser.h | 14
> > ++++++++++++++
> > > > >
> > > > > I looked at the content of fvp_baser.h after this series is applied.
> > > > > There are a bit of boiler plate that I expect to be part for other
> > > > > platforms. In particular...
> > > > >
> > > > > >    xen/arch/arm/platforms/Kconfig                 | 16
> > +++++++++++++---
> > > > > >    3 files changed, 38 insertions(+), 3 deletions(-)
> > > > > >    create mode 100644
> > xen/arch/arm/include/asm/platforms/fvp_baser.h
> > > > > >
> > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > index ad592367bd..ac276307d6 100644
> > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > @@ -138,6 +138,17 @@ config TEE
> > > > > >    	  This option enables generic TEE mediators support. It allows
> > > > > guests
> > > > > >    	  to access real TEE via one of TEE mediators implemented in
> > > > > > XEN.
> > > > > >
> > > > > > +config XEN_START_ADDRESS
> > > > > > +	hex "Xen start address: keep default to use platform defined
> > > > > address"
> > > > > > +	default 0xFFFFFFFF
> > > > >
> > > > > ... this default value will need to be tested everywhere. At least
> > for
> > > > > now, I think you can avoid the per platform header by using the
> > Kconfig
> > > > > to select the proper address (see the config for selecting early
> > printk
> > > > > address).
> > > > >
> > > > > This will also avoids to use an invalid value here.
> > > > >
> > > >
> > > > We had considered to use Kconfig to define the start addresses of
> > v8R64
> > > > platforms (prompt users to input the address). But we also want to
> > provide
> > > > a default start address for each platform (Discussed in [1], header
> > for
> > > > default value, Kconfig option for customized address).
> > > Why do you want to provide a default value? And how it is guaranteed
> > that it
> > > will work for most of the users?
> > >
> > > >
> > > > We also had thought to use Kconfig to define a default start address
> > > > for each platform like what we had done for early printk in RFC[2].
> > > > But this method has been deprecated.
> > >
> > > Most of the current Xen is board agnostic except the UART. We push back
> > the
> > > addition of new one because the address can be found in the firmware
> > table and
> > > I wanted to avoid increase the number of option (there are dozens of
> > platform
> > > out...).
> > >
> > > >
> > > > So if we don’t use header files, just use the Kconfig, we can't
> > > > provide a default start address for platforms, and have to force users
> > > > to enter the start address.
> > >
> > > I am not sure I see the problem to force the user to enter the start
> > address.
> > > My worry with per-platform default value is we end up to force each
> > vendor to
> > > provide an header in order to boot Xen.
> > >
> > > I think it would be better to provide a config tailored for that
> > platform
> > > (whether it is part of Xen can be debatable). This would allow a user to
> > try a
> > > release Xen on their platform with zero changes in the code.
> > 
> > I agree with Julien, especially on this last point.
> > 
> > Of course we need a default configuration for a given platform, we don't
> > want every user of the same platform to have to go and look at the
> > manual to find the right address to use.
> > 
> > The question is where to put the per-platform default value. The kconfig
> > "default" keyword is not great for that and it is not realistic to have
> > a single address that works everywhere.
> > 
> > Instead, we could have a prepopulated kconfig under
> > xen/arch/arm/configs, or something under ImageBuilder, or maybe expand
> 
> Do you mean we can keep a config like armv8r_fvp_baser_config in
> xen/arch/arm/configs for users to generate a default config?

Yes


> If yes I think this method might be better for now. And about ImageBuilder
> solution we can do it after MPU support be merged?

That's fine

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Wei Chen]

Hi Julien,

I thought I had replied to your email, but when I am checking to see if 
I have addressed all your comments, I didn't find my reply on the 
mailing list. Maybe there is something wrong with my mail system, I 
didn't find it sooner. Sorry about it. Hope my reply is not too late.


On 2022/11/10 2:24, Julien Grall wrote:
> 
> 
> On 09/11/2022 04:55, Wei Chen wrote:
>> Hi Julien,
> 
> Hi Wei,
> 
>>>
>>
>> We had considered to use Kconfig to define the start addresses of v8R64
>> platforms (prompt users to input the address). But we also want to 
>> provide
>> a default start address for each platform (Discussed in [1], header for
>> default value, Kconfig option for customized address).
> Why do you want to provide a default value? And how it is guaranteed 
> that it will work for most of the users?
> 
>>
>> We also had thought to use Kconfig to define a default start address
>> for each platform like what we had done for early printk in RFC[2].
>> But this method has been deprecated.
> 
> Most of the current Xen is board agnostic except the UART. We push back 
> the addition of new one because the address can be found in the firmware 
> table and I wanted to avoid increase the number of option (there are 
> dozens of platform out...).
> 
>>
>> So if we don’t use header files, just use the Kconfig, we can't
>> provide a default start address for platforms, and have to force users
>> to enter the start address.
> 
> I am not sure I see the problem to force the user to enter the start 
> address. My worry with per-platform default value is we end up to force 
> each vendor to provide an header in order to boot Xen.
> 
> I think it would be better to provide a config tailored for that 
> platform (whether it is part of Xen can be debatable). This would allow 
> a user to try a release Xen on their platform with zero changes in the 
> code.
> 

Above comments have been answered in my reply to you and Stefano.

>>>> diff --git a/xen/arch/arm/platforms/Kconfig
>>> b/xen/arch/arm/platforms/Kconfig
>>>> index c93a6b2756..0904793a0b 100644
>>>> --- a/xen/arch/arm/platforms/Kconfig
>>>> +++ b/xen/arch/arm/platforms/Kconfig
>>>> @@ -1,6 +1,7 @@
>>>>    choice
>>>>        prompt "Platform Support"
>>>>        default ALL_PLAT
>>>> +    default FVP_BASER if ARM_V8R
>>>
>>> Is there any reason to create a new Kconfig rather than using MPU?
>>>
>>
>> Did you mean FVP_BASER? If yes, we want to give each board a MACRO
>> to indicate its specific configurations. In current series, this MACRO
>> only be used for board specific start address.
> 
> See above for this.
> 

If we move board specific information to tailored config file, I think
we don't need FVP_BASER.

>>
>> If you meant Armv8R, that's because Armv8R does not equal to MPU.
> 
> I am not entirely sure to understand. Are you saying that an existing 
> Xen can boot on Armv8R?
> 

No, I didn't mean that. I just think we can't use only one MPU or one
ARM_V8R to cover all our changes in this series. For example, some
changes like new system registers are brought by Armv8R not the MPU.

Cheers,
Wei Chen




> Cheers,
> 

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Julien Grall]

Hi,

On 05/12/2022 10:17, Wei Chen wrote:
> On 2022/11/10 2:24, Julien Grall wrote:
> diff --git a/xen/arch/arm/platforms/Kconfig
>>>> b/xen/arch/arm/platforms/Kconfig
>>>>> index c93a6b2756..0904793a0b 100644
>>>>> --- a/xen/arch/arm/platforms/Kconfig
>>>>> +++ b/xen/arch/arm/platforms/Kconfig
>>>>> @@ -1,6 +1,7 @@
>>>>>    choice
>>>>>        prompt "Platform Support"
>>>>>        default ALL_PLAT
>>>>> +    default FVP_BASER if ARM_V8R
>>>>
>>>> Is there any reason to create a new Kconfig rather than using MPU?
>>>>
>>>
>>> Did you mean FVP_BASER? If yes, we want to give each board a MACRO
>>> to indicate its specific configurations. In current series, this MACRO
>>> only be used for board specific start address.
>>
>> See above for this.
>>
> 
> If we move board specific information to tailored config file, I think
> we don't need FVP_BASER.
> 
>>>
>>> If you meant Armv8R, that's because Armv8R does not equal to MPU.
>>
>> I am not entirely sure to understand. Are you saying that an existing 
>> Xen can boot on Armv8R?
>>
> 
> No, I didn't mean that. I just think we can't use only one MPU or one
> ARM_V8R to cover all our changes in this series. For example, some
> changes like new system registers are brought by Armv8R not the MPU.

I understand the theory. But in practice this needs to be a balance 
between finer grain and too much Kconfig.

 From this series alone, it doesn't seem to be make sense to introduce 
the two. Furthermore, I am not entirely sure you will be able to make 
the MPU work without enable the ARMv8-R Kconfig.

Cheers,

-- 
Julien Grall

Re: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Ayan Kumar Halder]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On Armv8-A, Xen has a fixed virtual start address (link address
> too) for all Armv8-A platforms. In an MMU based system, Xen can
> map its loaded address to this virtual start address. So, on
> Armv8-A platforms, the Xen start address does not need to be
> configurable. But on Armv8-R platforms, there is no MMU to map
> loaded address to a fixed virtual address and different platforms
> will have very different address space layout. So Xen cannot use
> a fixed physical address on MPU based system and need to have it
> configurable.
>
> So in this patch, we reuse the existing arm/platforms to store
> Armv8-R platforms' parameters. And `XEN_START_ADDRESS` is one
> kind of FVP BaseR platform's parameters. So we define default
> `XEN_START_ADDRESS` for FVP BaseR in its platform file.
>
> We also introduce one Kconfig option for users to override the
> default Xen start address of selected platform, if they think
> the default address doesn't suit their scenarios. For this
> Kconfig option, we use an unaligned address "0xffffffff" as the
> default value to indicate that users haven't used a customized
> Xen start address.
>
> And as we introduced Armv8-R platforms to Xen, that means the
> existed Arm64 platforms should not be listed in Armv8-R platform
> list, so we add !ARM_V8R dependency for these platforms.
>
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> Signed-off-by: Jiamei.Xie <jiamei.xie@arm.com>
> ---
>   xen/arch/arm/Kconfig                           | 11 +++++++++++
>   xen/arch/arm/include/asm/platforms/fvp_baser.h | 14 ++++++++++++++
>   xen/arch/arm/platforms/Kconfig                 | 16 +++++++++++++---
>   3 files changed, 38 insertions(+), 3 deletions(-)
>   create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index ad592367bd..ac276307d6 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -138,6 +138,17 @@ config TEE
>            This option enables generic TEE mediators support. It allows guests
>            to access real TEE via one of TEE mediators implemented in XEN.
>
> +config XEN_START_ADDRESS
> +       hex "Xen start address: keep default to use platform defined address"
> +       default 0xFFFFFFFF
> +       depends on HAS_MPU
> +       help
> +         This option allows to set the customized address at which Xen will be
> +         linked on MPU systems. This address must be aligned to a page size.
> +         Use 0xFFFFFFFF as the default value to indicate that user hasn't
> +         customized this address, and Xen use use the default value that has
> +         been defined in platform files.
> +
>   source "arch/arm/tee/Kconfig"
>
>   config STATIC_SHM
> diff --git a/xen/arch/arm/include/asm/platforms/fvp_baser.h b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> new file mode 100644
> index 0000000000..9450a411a9
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/platforms/fvp_baser.h
> @@ -0,0 +1,14 @@
> +#ifndef __ASM_ARM_PLATFORMS_FVP_BASER_H__
> +#define __ASM_ARM_PLATFORMS_FVP_BASER_H__
> +
> +/*
> + * 0xFFFFFFFF indicates users haven't customized XEN_START_ADDRESS,
> + * we will use platform defined default address.
> + */
> +#if CONFIG_XEN_START_ADDRESS == 0xFFFFFFFF
> +#define XEN_START_ADDRESS 0x200000
> +#else
> +#define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
> +#endif
> +
> +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
> diff --git a/xen/arch/arm/platforms/Kconfig b/xen/arch/arm/platforms/Kconfig
> index c93a6b2756..0904793a0b 100644
> --- a/xen/arch/arm/platforms/Kconfig
> +++ b/xen/arch/arm/platforms/Kconfig
> @@ -1,6 +1,7 @@
>   choice
>          prompt "Platform Support"
>          default ALL_PLAT
> +       default FVP_BASER if ARM_V8R

I could not spot the patch which introduced ARM_V8R.

Can you rename this to ARM64_V8R ? The reason being the underlying code 
is specific to R82 ie 64 bit V8R.

- Ayan

>          ---help---
>          Choose which hardware platform to enable in Xen.
>
> @@ -8,13 +9,14 @@ choice
>
>   config ALL_PLAT
>          bool "All Platforms"
> +       depends on !ARM_V8R
>          ---help---
>          Enable support for all available hardware platforms. It doesn't
>          automatically select any of the related drivers.
>
>   config QEMU
>          bool "QEMU aarch virt machine support"
> -       depends on ARM_64
> +       depends on ARM_64 && !ARM_V8R
>          select GICV3
>          select HAS_PL011
>          ---help---
> @@ -23,7 +25,7 @@ config QEMU
>
>   config RCAR3
>          bool "Renesas RCar3 support"
> -       depends on ARM_64
> +       depends on ARM_64 && !ARM_V8R
>          select HAS_SCIF
>          select IPMMU_VMSA
>          ---help---
> @@ -31,14 +33,22 @@ config RCAR3
>
>   config MPSOC
>          bool "Xilinx Ultrascale+ MPSoC support"
> -       depends on ARM_64
> +       depends on ARM_64 && !ARM_V8R
>          select HAS_CADENCE_UART
>          select ARM_SMMU
>          ---help---
>          Enable all the required drivers for Xilinx Ultrascale+ MPSoC
>
> +config FVP_BASER
> +       bool "Fixed Virtual Platform BaseR support"
> +       depends on ARM_V8R
> +       help
> +         Enable platform specific configurations for Fixed Virtual
> +         Platform BaseR
> +
>   config NO_PLAT
>          bool "No Platforms"
> +       depends on !ARM_V8R
>          ---help---
>          Do not enable specific support for any platform.
>
> --
> 2.25.1
>
>

RE: [PATCH v6 05/11] xen/arm: define Xen start address for FVP BaseR platform

[Wei Chen]

Hi Ayan,

> > +
> > +#endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
> > diff --git a/xen/arch/arm/platforms/Kconfig
> b/xen/arch/arm/platforms/Kconfig
> > index c93a6b2756..0904793a0b 100644
> > --- a/xen/arch/arm/platforms/Kconfig
> > +++ b/xen/arch/arm/platforms/Kconfig
> > @@ -1,6 +1,7 @@
> >   choice
> >          prompt "Platform Support"
> >          default ALL_PLAT
> > +       default FVP_BASER if ARM_V8R
> 
> I could not spot the patch which introduced ARM_V8R.
> 

That patch is not in this part, it will be the last one of MPU support
patch series. You can find it gitlab branch's full series.

> Can you rename this to ARM64_V8R ? The reason being the underlying code
> is specific to R82 ie 64 bit V8R.
> 

I renamed ARM64_V8R (in RFC patch) to ARM_V8R is because "Arm64 v8r" is
not an official Arm architecture name. The Arm official name is Armv8-R
AArch32/AArch64. And currently, MPU will only be selected by Arm64, so
current MPU code can only work in AArch64 state. When you're trying to
enable Armv8-R AArch32 like R52, you can remove this limitation, and use
CONFIG_ARM64 or CONFIG_ARM32 to distinguish code between r82 and r52 code.

Cheers,
Wei Chen

> - Ayan
> 
> >          ---help---
> >          Choose which hardware platform to enable in Xen.
> >
> > @@ -8,13 +9,14 @@ choice
> >
> >   config ALL_PLAT
> >          bool "All Platforms"
> > +       depends on !ARM_V8R
> >          ---help---
> >          Enable support for all available hardware platforms. It doesn't
> >          automatically select any of the related drivers.
> >
> >   config QEMU
> >          bool "QEMU aarch virt machine support"
> > -       depends on ARM_64
> > +       depends on ARM_64 && !ARM_V8R
> >          select GICV3
> >          select HAS_PL011
> >          ---help---
> > @@ -23,7 +25,7 @@ config QEMU
> >
> >   config RCAR3
> >          bool "Renesas RCar3 support"
> > -       depends on ARM_64
> > +       depends on ARM_64 && !ARM_V8R
> >          select HAS_SCIF
> >          select IPMMU_VMSA
> >          ---help---
> > @@ -31,14 +33,22 @@ config RCAR3
> >
> >   config MPSOC
> >          bool "Xilinx Ultrascale+ MPSoC support"
> > -       depends on ARM_64
> > +       depends on ARM_64 && !ARM_V8R
> >          select HAS_CADENCE_UART
> >          select ARM_SMMU
> >          ---help---
> >          Enable all the required drivers for Xilinx Ultrascale+ MPSoC
> >
> > +config FVP_BASER
> > +       bool "Fixed Virtual Platform BaseR support"
> > +       depends on ARM_V8R
> > +       help
> > +         Enable platform specific configurations for Fixed Virtual
> > +         Platform BaseR
> > +
> >   config NO_PLAT
> >          bool "No Platforms"
> > +       depends on !ARM_V8R
> >          ---help---
> >          Do not enable specific support for any platform.
> >
> > --
> > 2.25.1
> >
> >

[PATCH v6 06/11] xen/arm: split MMU and MPU config files from config.h

[Wei Chen]

Xen defines some global configuration macros for Arm in
config.h. We still want to use it for Armv8-R systems, but
there are some address related macros that are defined for
MMU systems. These macros will not be used by MPU systems,
Adding ifdefery with CONFIG_HAS_MPU to gate these macros
will result in a messy and hard to read/maintain code.

So we keep some common definitions still in config.h, but
move virtual address related definitions to a new file -
config_mmu.h. And use a new file config_mpu.h to store
definitions for MPU systems. To avoid spreading #ifdef
everywhere, we keep the same definition names for MPU
systems, like XEN_VIRT_START and HYPERVISOR_VIRT_START,
but the definition contents are MPU specific.

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/include/asm/config.h     |  97 +--------------------
 xen/arch/arm/include/asm/config_mmu.h | 119 ++++++++++++++++++++++++++
 xen/arch/arm/include/asm/config_mpu.h |  27 ++++++
 3 files changed, 150 insertions(+), 93 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h

diff --git a/xen/arch/arm/include/asm/config.h b/xen/arch/arm/include/asm/config.h
index 25a625ff08..189e932605 100644
--- a/xen/arch/arm/include/asm/config.h
+++ b/xen/arch/arm/include/asm/config.h
@@ -71,99 +71,10 @@
 #include <xen/const.h>
 #include <xen/page-size.h>
 
-/*
- * Common ARM32 and ARM64 layout:
- *   0  -   2M   Unmapped
- *   2M -   4M   Xen text, data, bss
- *   4M -   6M   Fixmap: special-purpose 4K mapping slots
- *   6M -  10M   Early boot mapping of FDT
- *   10M - 12M   Livepatch vmap (if compiled in)
- *
- * ARM32 layout:
- *   0  -  12M   <COMMON>
- *
- *  32M - 128M   Frametable: 24 bytes per page for 16GB of RAM
- * 256M -   1G   VMAP: ioremap and early_ioremap use this virtual address
- *                    space
- *
- *   1G -   2G   Xenheap: always-mapped memory
- *   2G -   4G   Domheap: on-demand-mapped
- *
- * ARM64 layout:
- * 0x0000000000000000 - 0x0000007fffffffff (512GB, L0 slot [0])
- *   0  -  12M   <COMMON>
- *
- *   1G -   2G   VMAP: ioremap and early_ioremap
- *
- *  32G -  64G   Frametable: 24 bytes per page for 5.3TB of RAM
- *
- * 0x0000008000000000 - 0x00007fffffffffff (127.5TB, L0 slots [1..255])
- *  Unused
- *
- * 0x0000800000000000 - 0x000084ffffffffff (5TB, L0 slots [256..265])
- *  1:1 mapping of RAM
- *
- * 0x0000850000000000 - 0x0000ffffffffffff (123TB, L0 slots [266..511])
- *  Unused
- */
-
-#define XEN_VIRT_START         _AT(vaddr_t,0x00200000)
-#define FIXMAP_ADDR(n)        (_AT(vaddr_t,0x00400000) + (n) * PAGE_SIZE)
-
-#define BOOT_FDT_VIRT_START    _AT(vaddr_t,0x00600000)
-#define BOOT_FDT_VIRT_SIZE     _AT(vaddr_t, MB(4))
-
-#ifdef CONFIG_LIVEPATCH
-#define LIVEPATCH_VMAP_START   _AT(vaddr_t,0x00a00000)
-#define LIVEPATCH_VMAP_SIZE    _AT(vaddr_t, MB(2))
-#endif
-
-#define HYPERVISOR_VIRT_START  XEN_VIRT_START
-
-#ifdef CONFIG_ARM_32
-
-#define CONFIG_SEPARATE_XENHEAP 1
-
-#define FRAMETABLE_VIRT_START  _AT(vaddr_t,0x02000000)
-#define FRAMETABLE_SIZE        MB(128-32)
-#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
-#define FRAMETABLE_VIRT_END    (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1)
-
-#define VMAP_VIRT_START        _AT(vaddr_t,0x10000000)
-#define VMAP_VIRT_SIZE         _AT(vaddr_t, GB(1) - MB(256))
-
-#define XENHEAP_VIRT_START     _AT(vaddr_t,0x40000000)
-#define XENHEAP_VIRT_SIZE      _AT(vaddr_t, GB(1))
-
-#define DOMHEAP_VIRT_START     _AT(vaddr_t,0x80000000)
-#define DOMHEAP_VIRT_SIZE      _AT(vaddr_t, GB(2))
-
-#define DOMHEAP_ENTRIES        1024  /* 1024 2MB mapping slots */
-
-/* Number of domheap pagetable pages required at the second level (2MB mappings) */
-#define DOMHEAP_SECOND_PAGES (DOMHEAP_VIRT_SIZE >> FIRST_SHIFT)
-
-#else /* ARM_64 */
-
-#define SLOT0_ENTRY_BITS  39
-#define SLOT0(slot) (_AT(vaddr_t,slot) << SLOT0_ENTRY_BITS)
-#define SLOT0_ENTRY_SIZE  SLOT0(1)
-
-#define VMAP_VIRT_START  GB(1)
-#define VMAP_VIRT_SIZE   GB(1)
-
-#define FRAMETABLE_VIRT_START  GB(32)
-#define FRAMETABLE_SIZE        GB(32)
-#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
-
-#define DIRECTMAP_VIRT_START   SLOT0(256)
-#define DIRECTMAP_SIZE         (SLOT0_ENTRY_SIZE * (265-256))
-#define DIRECTMAP_VIRT_END     (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1)
-
-#define XENHEAP_VIRT_START     directmap_virt_start
-
-#define HYPERVISOR_VIRT_END    DIRECTMAP_VIRT_END
-
+#ifdef CONFIG_HAS_MPU
+#include <asm/config_mpu.h>
+#else
+#include <asm/config_mmu.h>
 #endif
 
 #define NR_hypercalls 64
diff --git a/xen/arch/arm/include/asm/config_mmu.h b/xen/arch/arm/include/asm/config_mmu.h
new file mode 100644
index 0000000000..444223f4a1
--- /dev/null
+++ b/xen/arch/arm/include/asm/config_mmu.h
@@ -0,0 +1,119 @@
+/******************************************************************************
+ * config_mmu.h
+ *
+ * A Linux-style configuration list, only can be included by config.h
+ */
+
+#ifndef __ARM_CONFIG_MMU_H__
+#define __ARM_CONFIG_MMU_H__
+
+/*
+ * Common ARM32 and ARM64 layout:
+ *   0  -   2M   Unmapped
+ *   2M -   4M   Xen text, data, bss
+ *   4M -   6M   Fixmap: special-purpose 4K mapping slots
+ *   6M -  10M   Early boot mapping of FDT
+ *   10M - 12M   Livepatch vmap (if compiled in)
+ *
+ * ARM32 layout:
+ *   0  -  12M   <COMMON>
+ *
+ *  32M - 128M   Frametable: 24 bytes per page for 16GB of RAM
+ * 256M -   1G   VMAP: ioremap and early_ioremap use this virtual address
+ *                    space
+ *
+ *   1G -   2G   Xenheap: always-mapped memory
+ *   2G -   4G   Domheap: on-demand-mapped
+ *
+ * ARM64 layout:
+ * 0x0000000000000000 - 0x0000007fffffffff (512GB, L0 slot [0])
+ *   0  -  12M   <COMMON>
+ *
+ *   1G -   2G   VMAP: ioremap and early_ioremap
+ *
+ *  32G -  64G   Frametable: 24 bytes per page for 5.3TB of RAM
+ *
+ * 0x0000008000000000 - 0x00007fffffffffff (127.5TB, L0 slots [1..255])
+ *  Unused
+ *
+ * 0x0000800000000000 - 0x000084ffffffffff (5TB, L0 slots [256..265])
+ *  1:1 mapping of RAM
+ *
+ * 0x0000850000000000 - 0x0000ffffffffffff (123TB, L0 slots [266..511])
+ *  Unused
+ */
+
+#define XEN_VIRT_START         _AT(vaddr_t,0x00200000)
+#define FIXMAP_ADDR(n)        (_AT(vaddr_t,0x00400000) + (n) * PAGE_SIZE)
+
+#define BOOT_FDT_VIRT_START    _AT(vaddr_t,0x00600000)
+#define BOOT_FDT_VIRT_SIZE     _AT(vaddr_t, MB(4))
+
+#ifdef CONFIG_LIVEPATCH
+#define LIVEPATCH_VMAP_START   _AT(vaddr_t,0x00a00000)
+#define LIVEPATCH_VMAP_SIZE    _AT(vaddr_t, MB(2))
+#endif
+
+#define HYPERVISOR_VIRT_START  XEN_VIRT_START
+
+#ifdef CONFIG_ARM_32
+
+#define CONFIG_SEPARATE_XENHEAP 1
+
+#define FRAMETABLE_VIRT_START  _AT(vaddr_t,0x02000000)
+#define FRAMETABLE_SIZE        MB(128-32)
+#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
+#define FRAMETABLE_VIRT_END    (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1)
+
+#define VMAP_VIRT_START        _AT(vaddr_t,0x10000000)
+#define VMAP_VIRT_SIZE         _AT(vaddr_t, GB(1) - MB(256))
+
+#define XENHEAP_VIRT_START     _AT(vaddr_t,0x40000000)
+#define XENHEAP_VIRT_SIZE      _AT(vaddr_t, GB(1))
+
+#define DOMHEAP_VIRT_START     _AT(vaddr_t,0x80000000)
+#define DOMHEAP_VIRT_SIZE      _AT(vaddr_t, GB(2))
+
+#define DOMHEAP_ENTRIES        1024  /* 1024 2MB mapping slots */
+
+/* Number of domheap pagetable pages required at the second level (2MB mappings) */
+#define DOMHEAP_SECOND_PAGES (DOMHEAP_VIRT_SIZE >> FIRST_SHIFT)
+
+#else /* ARM_64 */
+
+#define SLOT0_ENTRY_BITS  39
+#define SLOT0(slot) (_AT(vaddr_t,slot) << SLOT0_ENTRY_BITS)
+#define SLOT0_ENTRY_SIZE  SLOT0(1)
+
+#define VMAP_VIRT_START  GB(1)
+#define VMAP_VIRT_SIZE   GB(1)
+
+#define FRAMETABLE_VIRT_START  GB(32)
+#define FRAMETABLE_SIZE        GB(32)
+#define FRAMETABLE_NR          (FRAMETABLE_SIZE / sizeof(*frame_table))
+
+#define DIRECTMAP_VIRT_START   SLOT0(256)
+#define DIRECTMAP_SIZE         (SLOT0_ENTRY_SIZE * (265-256))
+#define DIRECTMAP_VIRT_END     (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1)
+
+#define XENHEAP_VIRT_START     directmap_virt_start
+
+#define HYPERVISOR_VIRT_END    DIRECTMAP_VIRT_END
+
+#endif
+
+/* Fixmap slots */
+#define FIXMAP_CONSOLE  0  /* The primary UART */
+#define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
+#define FIXMAP_ACPI_BEGIN  2  /* Start mappings of ACPI tables */
+#define FIXMAP_ACPI_END    (FIXMAP_ACPI_BEGIN + NUM_FIXMAP_ACPI_PAGES - 1)  /* End mappings of ACPI tables */
+
+#endif /* __ARM_CONFIG_MMU_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/include/asm/config_mpu.h b/xen/arch/arm/include/asm/config_mpu.h
new file mode 100644
index 0000000000..530abb8302
--- /dev/null
+++ b/xen/arch/arm/include/asm/config_mpu.h
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * config_mpu.h: A Linux-style configuration list for Arm MPU systems,
+ *               only can be included by config.h
+ */
+
+#ifndef __ARM_CONFIG_MPU_H__
+#define __ARM_CONFIG_MPU_H__
+
+#ifdef CONFIG_FVP_BASER
+#include <asm/platforms/fvp_baser.h>
+#endif
+
+/*
+ * All MPU platforms need to provide a XEN_START_ADDRESS for linker. This
+ * address indicates where Xen image will be loaded and run from. This
+ * address must be aligned to a PAGE_SIZE.
+ */
+#if (XEN_START_ADDRESS % PAGE_SIZE) != 0
+#error "XEN_START_ADDRESS must be aligned to PAGE_SIZE"
+#endif
+
+#define XEN_VIRT_START         _AT(paddr_t, XEN_START_ADDRESS)
+
+#define HYPERVISOR_VIRT_START  XEN_VIRT_START
+
+#endif /* __ARM_CONFIG_MPU_H__ */
-- 
2.25.1

[PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Wei Chen]

FIXMAP is a special virtual address section for Xen to map some
physical ram or device memory temporarily in initialization for
MMU systems. FIXMAP_ADDR will return a virtual address by index
for special purpose phys-to-virt mapping usage. For example,
FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

But in MPU systems, we can't map physical address to any virtual
address. So we want the code that is using FIXMAP_ADDR to return
the input physical address in MPU systems. So in MPU version,
FIXMAP_ADDR will trim physical address to PAGE alignment. This
will return an offset which is similar to MMU version FIXMAP_ADDR.
But it's a physical offset got from input physical address, plus
to an offset inside page (which is also got from physical address
mask with PAGE_MASK). The caller can return the input physical
address directly.

As pmap depends on FIXAMP, so we disable pmap for Arm with MPU
enabled systems.

Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/Kconfig                  |  2 +-
 xen/arch/arm/include/asm/config_mpu.h |  2 ++
 xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index ac276307d6..1458ffa777 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -16,7 +16,7 @@ config ARM
 	select HAS_DEVICE_TREE
 	select HAS_PASSTHROUGH
 	select HAS_PDX
-	select HAS_PMAP
+	select HAS_PMAP if !HAS_MPU
 	select IOMMU_FORCE_PT_SHARE
 
 config ARCH_DEFCONFIG
diff --git a/xen/arch/arm/include/asm/config_mpu.h b/xen/arch/arm/include/asm/config_mpu.h
index 530abb8302..eee60dcffc 100644
--- a/xen/arch/arm/include/asm/config_mpu.h
+++ b/xen/arch/arm/include/asm/config_mpu.h
@@ -24,4 +24,6 @@
 
 #define HYPERVISOR_VIRT_START  XEN_VIRT_START
 
+#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
+
 #endif /* __ARM_CONFIG_MPU_H__ */
diff --git a/xen/arch/arm/include/asm/fixmap.h b/xen/arch/arm/include/asm/fixmap.h
index d0c9a52c8c..1e338759e9 100644
--- a/xen/arch/arm/include/asm/fixmap.h
+++ b/xen/arch/arm/include/asm/fixmap.h
@@ -7,6 +7,8 @@
 #include <xen/acpi.h>
 #include <xen/pmap.h>
 
+#ifndef CONFIG_HAS_MPU
+
 /* Fixmap slots */
 #define FIXMAP_CONSOLE  0  /* The primary UART */
 #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
@@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
 
 #endif /* __ASSEMBLY__ */
 
+#else
+
+/*
+ * FIXMAP_ADDR will trim physical address to PAGE alignment.
+ * This will return an offset which is similar to MMU version
+ * FIXMAP_ADDR.
+ * For example:
+ * EARLY_UART_VIRTUAL_ADDRESS is defined by:
+ *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
+ *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
+ * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
+ * EARLY_UART_VIRTUAL_ADDRESS can be restore to
+ * CONFIG_EARLY_UART_BASE_ADDRESS.
+ * In this case, we don't need to use #ifdef MPU in the code
+ * where are using FIXMAP_ADDR to make them to use physical
+ * address explicitily.
+ */
+#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
+#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
+#endif
+
+#endif /* CONFIG_HAS_MPU */
+
 #endif /* __ASM_FIXMAP_H */
-- 
2.25.1

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> FIXMAP is a special virtual address section for Xen to map some
> physical ram or device memory temporarily in initialization for
> MMU systems. FIXMAP_ADDR will return a virtual address by index
> for special purpose phys-to-virt mapping usage. For example,
> FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
> FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.

To me, we are bending quite a bit the definition of the fixmap. There 
are not many use of the FIXMAP within the code and I think it would 
simply be better to abstract the use (or removing it when possible) and 
avoid defining FIXMAP_ADDR() & co for MPU.

> 
> But in MPU systems, we can't map physical address to any virtual
> address. So we want the code that is using FIXMAP_ADDR to return
> the input physical address in MPU systems. So in MPU version,
> FIXMAP_ADDR will trim physical address to PAGE alignment. This
> will return an offset which is similar to MMU version FIXMAP_ADDR.
> But it's a physical offset got from input physical address, plus
> to an offset inside page (which is also got from physical address
> mask with PAGE_MASK). The caller can return the input physical
> address directly.
> 
> As pmap depends on FIXAMP, so we disable pmap for Arm with MPU
> enabled systems.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> ---
>   xen/arch/arm/Kconfig                  |  2 +-
>   xen/arch/arm/include/asm/config_mpu.h |  2 ++
>   xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
>   3 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index ac276307d6..1458ffa777 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -16,7 +16,7 @@ config ARM
>   	select HAS_DEVICE_TREE
>   	select HAS_PASSTHROUGH
>   	select HAS_PDX
> -	select HAS_PMAP
> +	select HAS_PMAP if !HAS_MPU

I can't find any change of mm.c in this series. So surely this will 
break the build?

>   	select IOMMU_FORCE_PT_SHARE
>   
>   config ARCH_DEFCONFIG
> diff --git a/xen/arch/arm/include/asm/config_mpu.h b/xen/arch/arm/include/asm/config_mpu.h
> index 530abb8302..eee60dcffc 100644
> --- a/xen/arch/arm/include/asm/config_mpu.h
> +++ b/xen/arch/arm/include/asm/config_mpu.h
> @@ -24,4 +24,6 @@
>   
>   #define HYPERVISOR_VIRT_START  XEN_VIRT_START
>   
> +#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
> +
>   #endif /* __ARM_CONFIG_MPU_H__ */
> diff --git a/xen/arch/arm/include/asm/fixmap.h b/xen/arch/arm/include/asm/fixmap.h
> index d0c9a52c8c..1e338759e9 100644
> --- a/xen/arch/arm/include/asm/fixmap.h
> +++ b/xen/arch/arm/include/asm/fixmap.h
> @@ -7,6 +7,8 @@
>   #include <xen/acpi.h>
>   #include <xen/pmap.h>
>   
> +#ifndef CONFIG_HAS_MPU
> +
>   /* Fixmap slots */
>   #define FIXMAP_CONSOLE  0  /* The primary UART */
>   #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
> @@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
>   
>   #endif /* __ASSEMBLY__ */
>   
> +#else
> +
> +/*
> + * FIXMAP_ADDR will trim physical address to PAGE alignment.
> + * This will return an offset which is similar to MMU version
> + * FIXMAP_ADDR.
> + * For example:
> + * EARLY_UART_VIRTUAL_ADDRESS is defined by:
> + *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
> + *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
> + * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
> + * EARLY_UART_VIRTUAL_ADDRESS can be restore to
> + * CONFIG_EARLY_UART_BASE_ADDRESS.
> + * In this case, we don't need to use #ifdef MPU in the code
> + * where are using FIXMAP_ADDR to make them to use physical
> + * address explicitily.
> + */
> +#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
> +#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
> +#endif
> +
> +#endif /* CONFIG_HAS_MPU */
> +
>   #endif /* __ASM_FIXMAP_H */

Cheers,

-- 
Julien Grall

RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 3:45
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
> systems
> 
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > FIXMAP is a special virtual address section for Xen to map some
> > physical ram or device memory temporarily in initialization for
> > MMU systems. FIXMAP_ADDR will return a virtual address by index
> > for special purpose phys-to-virt mapping usage. For example,
> > FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
> > FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.
> 
> To me, we are bending quite a bit the definition of the fixmap. There
> are not many use of the FIXMAP within the code and I think it would
> simply be better to abstract the use (or removing it when possible) and
> avoid defining FIXMAP_ADDR() & co for MPU.
> 

I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic code. 
I also prefer this way. Frankly, I really think FIXMAP is awkward in MPU
System.

> >
> > But in MPU systems, we can't map physical address to any virtual
> > address. So we want the code that is using FIXMAP_ADDR to return
> > the input physical address in MPU systems. So in MPU version,
> > FIXMAP_ADDR will trim physical address to PAGE alignment. This
> > will return an offset which is similar to MMU version FIXMAP_ADDR.
> > But it's a physical offset got from input physical address, plus
> > to an offset inside page (which is also got from physical address
> > mask with PAGE_MASK). The caller can return the input physical
> > address directly.
> >
> > As pmap depends on FIXAMP, so we disable pmap for Arm with MPU
> > enabled systems.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > ---
> >   xen/arch/arm/Kconfig                  |  2 +-
> >   xen/arch/arm/include/asm/config_mpu.h |  2 ++
> >   xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
> >   3 files changed, 28 insertions(+), 1 deletion(-)
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index ac276307d6..1458ffa777 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -16,7 +16,7 @@ config ARM
> >   	select HAS_DEVICE_TREE
> >   	select HAS_PASSTHROUGH
> >   	select HAS_PDX
> > -	select HAS_PMAP
> > +	select HAS_PMAP if !HAS_MPU
> 
> I can't find any change of mm.c in this series. So surely this will
> break the build?

Yes, in our internal testing, open PMAP for MPU will cause building
failed, except we add some new stubs for MPU system.

> 
> >   	select IOMMU_FORCE_PT_SHARE
> >
> >   config ARCH_DEFCONFIG
> > diff --git a/xen/arch/arm/include/asm/config_mpu.h
> b/xen/arch/arm/include/asm/config_mpu.h
> > index 530abb8302..eee60dcffc 100644
> > --- a/xen/arch/arm/include/asm/config_mpu.h
> > +++ b/xen/arch/arm/include/asm/config_mpu.h
> > @@ -24,4 +24,6 @@
> >
> >   #define HYPERVISOR_VIRT_START  XEN_VIRT_START
> >
> > +#define FIXMAP_ADDR(n)         (_AT(paddr_t, n) & (PAGE_MASK))
> > +
> >   #endif /* __ARM_CONFIG_MPU_H__ */
> > diff --git a/xen/arch/arm/include/asm/fixmap.h
> b/xen/arch/arm/include/asm/fixmap.h
> > index d0c9a52c8c..1e338759e9 100644
> > --- a/xen/arch/arm/include/asm/fixmap.h
> > +++ b/xen/arch/arm/include/asm/fixmap.h
> > @@ -7,6 +7,8 @@
> >   #include <xen/acpi.h>
> >   #include <xen/pmap.h>
> >
> > +#ifndef CONFIG_HAS_MPU
> > +
> >   /* Fixmap slots */
> >   #define FIXMAP_CONSOLE  0  /* The primary UART */
> >   #define FIXMAP_MISC     1  /* Ephemeral mappings of hardware */
> > @@ -45,4 +47,27 @@ static inline unsigned int virt_to_fix(vaddr_t vaddr)
> >
> >   #endif /* __ASSEMBLY__ */
> >
> > +#else
> > +
> > +/*
> > + * FIXMAP_ADDR will trim physical address to PAGE alignment.
> > + * This will return an offset which is similar to MMU version
> > + * FIXMAP_ADDR.
> > + * For example:
> > + * EARLY_UART_VIRTUAL_ADDRESS is defined by:
> > + *     (FIXMAP_ADDR(FIXMAP_CONSOLE) + \
> > + *     (CONFIG_EARLY_UART_BASE_ADDRESS & ~PAGE_MASK))
> > + * With MPU version FIXMAP_CONSOLE and FIXMAP_ADDR definitions,
> > + * EARLY_UART_VIRTUAL_ADDRESS can be restore to
> > + * CONFIG_EARLY_UART_BASE_ADDRESS.
> > + * In this case, we don't need to use #ifdef MPU in the code
> > + * where are using FIXMAP_ADDR to make them to use physical
> > + * address explicitily.
> > + */
> > +#ifdef CONFIG_EARLY_UART_BASE_ADDRESS
> > +#define FIXMAP_CONSOLE         CONFIG_EARLY_UART_BASE_ADDRESS
> > +#endif
> > +
> > +#endif /* CONFIG_HAS_MPU */
> > +
> >   #endif /* __ASM_FIXMAP_H */
> 
> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Julien Grall]

On 09/11/2022 06:46, Wei Chen wrote:
> Hi Julien,

Hi Wei,

> 
>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: 2022年11月7日 3:45
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
>> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>
>> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
>> systems
>>
>> Hi Wei,
>>
>> On 04/11/2022 10:07, Wei Chen wrote:
>>> FIXMAP is a special virtual address section for Xen to map some
>>> physical ram or device memory temporarily in initialization for
>>> MMU systems. FIXMAP_ADDR will return a virtual address by index
>>> for special purpose phys-to-virt mapping usage. For example,
>>> FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
>>> FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.
>>
>> To me, we are bending quite a bit the definition of the fixmap. There
>> are not many use of the FIXMAP within the code and I think it would
>> simply be better to abstract the use (or removing it when possible) and
>> avoid defining FIXMAP_ADDR() & co for MPU.
>>
> 
> I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic code.

FAOD, this is not what I had in mind. Instead, it was to provide helper 
which for !HAS_MPU would call fixmap and for HAS_MPU would do the work 
to map the region in the MPU.

[...]

>>>    xen/arch/arm/Kconfig                  |  2 +-
>>>    xen/arch/arm/include/asm/config_mpu.h |  2 ++
>>>    xen/arch/arm/include/asm/fixmap.h     | 25 +++++++++++++++++++++++++
>>>    3 files changed, 28 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index ac276307d6..1458ffa777 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -16,7 +16,7 @@ config ARM
>>>    	select HAS_DEVICE_TREE
>>>    	select HAS_PASSTHROUGH
>>>    	select HAS_PDX
>>> -	select HAS_PMAP
>>> +	select HAS_PMAP if !HAS_MPU
>>
>> I can't find any change of mm.c in this series. So surely this will
>> break the build?
> 
> Yes, in our internal testing, open PMAP for MPU will cause building
> failed, except we add some new stubs for MPU system.

Do you mean you added some stubs for PMAP? If so, I would not expect any 
caller for the pmap() to be used for the MPU. Therefore, why would they 
be necessary?

Cheers,

-- 
Julien Grall

RE: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月10日 2:30
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
> systems
> 
> 
> 
> On 09/11/2022 06:46, Wei Chen wrote:
> > Hi Julien,
> 
> Hi Wei,
> 
> >
> >> -----Original Message-----
> >> From: Julien Grall <julien@xen.org>
> >> Sent: 2022年11月7日 3:45
> >> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> >> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Bertrand
> >> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> >> <Volodymyr_Babchuk@epam.com>
> >> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
> >> systems
> >>
> >> Hi Wei,
> >>
> >> On 04/11/2022 10:07, Wei Chen wrote:
> >>> FIXMAP is a special virtual address section for Xen to map some
> >>> physical ram or device memory temporarily in initialization for
> >>> MMU systems. FIXMAP_ADDR will return a virtual address by index
> >>> for special purpose phys-to-virt mapping usage. For example,
> >>> FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
> >>> FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.
> >>
> >> To me, we are bending quite a bit the definition of the fixmap. There
> >> are not many use of the FIXMAP within the code and I think it would
> >> simply be better to abstract the use (or removing it when possible) and
> >> avoid defining FIXMAP_ADDR() & co for MPU.
> >>
> >
> > I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic
> code.
> 
> FAOD, this is not what I had in mind. Instead, it was to provide helper
> which for !HAS_MPU would call fixmap and for HAS_MPU would do the work
> to map the region in the MPU.
> 

Sorry, I am still confused about this comment, did you mean we can provider
Some generic helpers like: early_map_console / eary_map_guest_memory.
For non-MPU system, we still can call fixmap in these callers, but for
MPU system, we have to map the region to MPU region?

> [...]
> 
> >>>    xen/arch/arm/Kconfig                  |  2 +-
> >>>    xen/arch/arm/include/asm/config_mpu.h |  2 ++
> >>>    xen/arch/arm/include/asm/fixmap.h     | 25
> +++++++++++++++++++++++++
> >>>    3 files changed, 28 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> >>> index ac276307d6..1458ffa777 100644
> >>> --- a/xen/arch/arm/Kconfig
> >>> +++ b/xen/arch/arm/Kconfig
> >>> @@ -16,7 +16,7 @@ config ARM
> >>>    	select HAS_DEVICE_TREE
> >>>    	select HAS_PASSTHROUGH
> >>>    	select HAS_PDX
> >>> -	select HAS_PMAP
> >>> +	select HAS_PMAP if !HAS_MPU
> >>
> >> I can't find any change of mm.c in this series. So surely this will
> >> break the build?
> >
> > Yes, in our internal testing, open PMAP for MPU will cause building
> > failed, except we add some new stubs for MPU system.
> 
> Do you mean you added some stubs for PMAP? If so, I would not expect any
> caller for the pmap() to be used for the MPU. Therefore, why would they
> be necessary?
> 

No, I mean if we want to make pmap can be built successfully for MPU,
we have to implement some stubs like: fix_to_virt, xen_fixmap and
write_pte, to make compiling success. But just as you said, we would
not expect MPU to use any PMAP function, so we have not implemented
them for MPU. Instead we disable PMAP for MPU. 

Cheer,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU systems

[Julien Grall]

On 11/11/2022 07:56, Wei Chen wrote:
> Hi Julien,
> 
>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: 2022年11月10日 2:30
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
>> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>
>> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
>> systems
>>
>>
>>
>> On 09/11/2022 06:46, Wei Chen wrote:
>>> Hi Julien,
>>
>> Hi Wei,
>>
>>>
>>>> -----Original Message-----
>>>> From: Julien Grall <julien@xen.org>
>>>> Sent: 2022年11月7日 3:45
>>>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>>>> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
>> Bertrand
>>>> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>>>> <Volodymyr_Babchuk@epam.com>
>>>> Subject: Re: [PATCH v6 07/11] xen/arm: implement FIXMAP_ADDR for MPU
>>>> systems
>>>>
>>>> Hi Wei,
>>>>
>>>> On 04/11/2022 10:07, Wei Chen wrote:
>>>>> FIXMAP is a special virtual address section for Xen to map some
>>>>> physical ram or device memory temporarily in initialization for
>>>>> MMU systems. FIXMAP_ADDR will return a virtual address by index
>>>>> for special purpose phys-to-virt mapping usage. For example,
>>>>> FIXMAP_ADDR(FIXMAP_CONSOLE) for early console mapping and
>>>>> FIXMAP_ADDR(FIXMAP_MISC) for copy_from_paddr.
>>>>
>>>> To me, we are bending quite a bit the definition of the fixmap. There
>>>> are not many use of the FIXMAP within the code and I think it would
>>>> simply be better to abstract the use (or removing it when possible) and
>>>> avoid defining FIXMAP_ADDR() & co for MPU.
>>>>
>>>
>>> I agree, if we don't mind to add some CONFIG_HAS_MPU in some generic
>> code.
>>
>> FAOD, this is not what I had in mind. Instead, it was to provide helper
>> which for !HAS_MPU would call fixmap and for HAS_MPU would do the work
>> to map the region in the MPU.
>>
> 
> Sorry, I am still confused about this comment, did you mean we can provider
> Some generic helpers like: early_map_console / eary_map_guest_memory.
> For non-MPU system, we still can call fixmap in these callers, but for
> MPU system, we have to map the region to MPU region?

Yes.

Cheers,

-- 
Julien Grall

[PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Wei Chen]

There are lots of MMU specific code in head.S. This code will not
be used in MPU systems. If we use #ifdef to gate them, the code
will become messy and hard to maintain. So we move MMU related
code to head_mmu.S, and keep common code still in head.S.

As we need to access "fail" and "puts" functions out of assembly
file, so we have to export them in this patch. And the assembly
macros: adr_l and load_paddr will be used by MMU and MPU later,
so we move them to macros.h.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Signed-off-by: Wei Chen <wei.chen@arm.com>
---
 xen/arch/arm/arm64/Makefile             |   3 +
 xen/arch/arm/arm64/head.S               | 407 +-----------------------
 xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
 xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
 4 files changed, 432 insertions(+), 394 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S

diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
index 6d507da0d4..22da2f54b5 100644
--- a/xen/arch/arm/arm64/Makefile
+++ b/xen/arch/arm/arm64/Makefile
@@ -8,6 +8,9 @@ obj-y += domctl.o
 obj-y += domain.o
 obj-y += entry.o
 obj-y += head.o
+ifneq ($(CONFIG_HAS_MPU),y)
+obj-y += head_mmu.o
+endif
 obj-y += insn.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o
 obj-y += smc.o
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index ccedf20dc7..d9a8da9120 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -25,17 +25,6 @@
 #include <efi/efierr.h>
 #include <asm/arm64/efibind.h>
 
-#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
-#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
-#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
-
-/* Convenience defines to get slot used by Xen mapping. */
-#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
-#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
-#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
-
 #define __HEAD_FLAG_PAGE_SIZE   ((PAGE_SHIFT - 10) / 2)
 
 #define __HEAD_FLAG_PHYS_BASE   1
@@ -82,73 +71,22 @@
  *  x30 - lr
  */
 
-#ifdef CONFIG_EARLY_PRINTK
-/*
- * Macro to print a string to the UART, if there is one.
- *
- * Clobbers x0 - x3
- */
-#define PRINT(_s)          \
-        mov   x3, lr ;     \
-        adr   x0, 98f ;    \
-        bl    puts    ;    \
-        mov   lr, x3 ;     \
-        RODATA_STR(98, _s)
+.section .text.header, "ax", %progbits
+/*.aarch64*/
 
 /*
- * Macro to print the value of register \xb
+ * Kernel startup entry point.
+ * ---------------------------
  *
- * Clobbers x0 - x4
- */
-.macro print_reg xb
-        mov   x0, \xb
-        mov   x4, lr
-        bl    putn
-        mov   lr, x4
-.endm
-
-#else /* CONFIG_EARLY_PRINTK */
-#define PRINT(s)
-
-.macro print_reg xb
-.endm
-
-#endif /* !CONFIG_EARLY_PRINTK */
-
-/*
- * Pseudo-op for PC relative adr <reg>, <symbol> where <symbol> is
- * within the range +/- 4GB of the PC.
+ * The requirements are:
+ *   MMU = off, D-cache = off, I-cache = on or off,
+ *   x0 = physical address to the FDT blob.
  *
- * @dst: destination register (64 bit wide)
- * @sym: name of the symbol
+ * This must be the very first address in the loaded image.
+ * It should be linked at XEN_VIRT_START, and loaded at any
+ * 4K-aligned address.  All of text+data+bss must fit in 2MB,
+ * or the initial pagetable code below will need adjustment.
  */
-.macro  adr_l, dst, sym
-        adrp \dst, \sym
-        add  \dst, \dst, :lo12:\sym
-.endm
-
-/* Load the physical address of a symbol into xb */
-.macro load_paddr xb, sym
-        ldr \xb, =\sym
-        add \xb, \xb, x20
-.endm
-
-        .section .text.header, "ax", %progbits
-        /*.aarch64*/
-
-        /*
-         * Kernel startup entry point.
-         * ---------------------------
-         *
-         * The requirements are:
-         *   MMU = off, D-cache = off, I-cache = on or off,
-         *   x0 = physical address to the FDT blob.
-         *
-         * This must be the very first address in the loaded image.
-         * It should be linked at XEN_VIRT_START, and loaded at any
-         * 4K-aligned address.  All of text+data+bss must fit in 2MB,
-         * or the initial pagetable code below will need adjustment.
-         */
 
 GLOBAL(start)
         /*
@@ -497,296 +435,6 @@ cpu_init:
         ret
 ENDPROC(cpu_init)
 
-/*
- * Macro to find the slot number at a given page-table level
- *
- * slot:     slot computed
- * virt:     virtual address
- * lvl:      page-table level
- */
-.macro get_table_slot, slot, virt, lvl
-        ubfx  \slot, \virt, #XEN_PT_LEVEL_SHIFT(\lvl), #XEN_PT_LPAE_SHIFT
-.endm
-
-/*
- * Macro to create a page table entry in \ptbl to \tbl
- *
- * ptbl:    table symbol where the entry will be created
- * tbl:     table symbol to point to
- * virt:    virtual address
- * lvl:     page-table level
- * tmp1:    scratch register
- * tmp2:    scratch register
- * tmp3:    scratch register
- *
- * Preserves \virt
- * Clobbers \tmp1, \tmp2, \tmp3
- *
- * Also use x20 for the phys offset.
- *
- * Note that all parameters using registers should be distinct.
- */
-.macro create_table_entry, ptbl, tbl, virt, lvl, tmp1, tmp2, tmp3
-        get_table_slot \tmp1, \virt, \lvl   /* \tmp1 := slot in \tlb */
-
-        load_paddr \tmp2, \tbl
-        mov   \tmp3, #PT_PT                 /* \tmp3 := right for linear PT */
-        orr   \tmp3, \tmp3, \tmp2           /*          + \tlb paddr */
-
-        adr_l \tmp2, \ptbl
-
-        str   \tmp3, [\tmp2, \tmp1, lsl #3]
-.endm
-
-/*
- * Macro to create a mapping entry in \tbl to \phys. Only mapping in 3rd
- * level table (i.e page granularity) is supported.
- *
- * ptbl:     table symbol where the entry will be created
- * virt:    virtual address
- * phys:    physical address (should be page aligned)
- * tmp1:    scratch register
- * tmp2:    scratch register
- * tmp3:    scratch register
- * type:    mapping type. If not specified it will be normal memory (PT_MEM_L3)
- *
- * Preserves \virt, \phys
- * Clobbers \tmp1, \tmp2, \tmp3
- *
- * Note that all parameters using registers should be distinct.
- */
-.macro create_mapping_entry, ptbl, virt, phys, tmp1, tmp2, tmp3, type=PT_MEM_L3
-        and   \tmp3, \phys, #THIRD_MASK     /* \tmp3 := PAGE_ALIGNED(phys) */
-
-        get_table_slot \tmp1, \virt, 3      /* \tmp1 := slot in \tlb */
-
-        mov   \tmp2, #\type                 /* \tmp2 := right for section PT */
-        orr   \tmp2, \tmp2, \tmp3           /*          + PAGE_ALIGNED(phys) */
-
-        adr_l \tmp3, \ptbl
-
-        str   \tmp2, [\tmp3, \tmp1, lsl #3]
-.endm
-
-/*
- * Rebuild the boot pagetable's first-level entries. The structure
- * is described in mm.c.
- *
- * After the CPU enables paging it will add the fixmap mapping
- * to these page tables, however this may clash with the 1:1
- * mapping. So each CPU must rebuild the page tables here with
- * the 1:1 in place.
- *
- * Inputs:
- *   x19: paddr(start)
- *   x20: phys offset
- *
- * Clobbers x0 - x4
- */
-create_page_tables:
-        /* Prepare the page-tables for mapping Xen */
-        ldr   x0, =XEN_VIRT_START
-        create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
-        create_table_entry boot_first, boot_second, x0, 1, x1, x2, x3
-        create_table_entry boot_second, boot_third, x0, 2, x1, x2, x3
-
-        /* Map Xen */
-        adr_l x4, boot_third
-
-        lsr   x2, x19, #THIRD_SHIFT  /* Base address for 4K mapping */
-        lsl   x2, x2, #THIRD_SHIFT
-        mov   x3, #PT_MEM_L3         /* x2 := Section map */
-        orr   x2, x2, x3
-
-        /* ... map of vaddr(start) in boot_third */
-        mov   x1, xzr
-1:      str   x2, [x4, x1]           /* Map vaddr(start) */
-        add   x2, x2, #PAGE_SIZE     /* Next page */
-        add   x1, x1, #8             /* Next slot */
-        cmp   x1, #(XEN_PT_LPAE_ENTRIES<<3) /* 512 entries per page */
-        b.lt  1b
-
-        /*
-         * If Xen is loaded at exactly XEN_VIRT_START then we don't
-         * need an additional 1:1 mapping, the virtual mapping will
-         * suffice.
-         */
-        cmp   x19, #XEN_VIRT_START
-        bne   1f
-        ret
-1:
-        /*
-         * Setup the 1:1 mapping so we can turn the MMU on. Note that
-         * only the first page of Xen will be part of the 1:1 mapping.
-         */
-
-        /*
-         * Find the zeroeth slot used. If the slot is not
-         * XEN_ZEROETH_SLOT, then the 1:1 mapping will use its own set of
-         * page-tables from the first level.
-         */
-        get_table_slot x0, x19, 0       /* x0 := zeroeth slot */
-        cmp   x0, #XEN_ZEROETH_SLOT
-        beq   1f
-        create_table_entry boot_pgtable, boot_first_id, x19, 0, x0, x1, x2
-        b     link_from_first_id
-
-1:
-        /*
-         * Find the first slot used. If the slot is not XEN_FIRST_SLOT,
-         * then the 1:1 mapping will use its own set of page-tables from
-         * the second level.
-         */
-        get_table_slot x0, x19, 1      /* x0 := first slot */
-        cmp   x0, #XEN_FIRST_SLOT
-        beq   1f
-        create_table_entry boot_first, boot_second_id, x19, 1, x0, x1, x2
-        b     link_from_second_id
-
-1:
-        /*
-         * Find the second slot used. If the slot is XEN_SECOND_SLOT, then the
-         * 1:1 mapping will use its own set of page-tables from the
-         * third level. For slot XEN_SECOND_SLOT, Xen is not yet able to handle
-         * it.
-         */
-        get_table_slot x0, x19, 2     /* x0 := second slot */
-        cmp   x0, #XEN_SECOND_SLOT
-        beq   virtphys_clash
-        create_table_entry boot_second, boot_third_id, x19, 2, x0, x1, x2
-        b     link_from_third_id
-
-link_from_first_id:
-        create_table_entry boot_first_id, boot_second_id, x19, 1, x0, x1, x2
-link_from_second_id:
-        create_table_entry boot_second_id, boot_third_id, x19, 2, x0, x1, x2
-link_from_third_id:
-        create_mapping_entry boot_third_id, x19, x19, x0, x1, x2
-        ret
-
-virtphys_clash:
-        /* Identity map clashes with boot_third, which we cannot handle yet */
-        PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
-        b     fail
-ENDPROC(create_page_tables)
-
-/*
- * Turn on the Data Cache and the MMU. The function will return on the 1:1
- * mapping. In other word, the caller is responsible to switch to the runtime
- * mapping.
- *
- * Clobbers x0 - x3
- */
-enable_mmu:
-        PRINT("- Turning on paging -\r\n")
-
-        /*
-         * The state of the TLBs is unknown before turning on the MMU.
-         * Flush them to avoid stale one.
-         */
-        tlbi  alle2                  /* Flush hypervisor TLBs */
-        dsb   nsh
-
-        /* Write Xen's PT's paddr into TTBR0_EL2 */
-        load_paddr x0, boot_pgtable
-        msr   TTBR0_EL2, x0
-        isb
-
-        mrs   x0, SCTLR_EL2
-        orr   x0, x0, #SCTLR_Axx_ELx_M  /* Enable MMU */
-        orr   x0, x0, #SCTLR_Axx_ELx_C  /* Enable D-cache */
-        dsb   sy                     /* Flush PTE writes and finish reads */
-        msr   SCTLR_EL2, x0          /* now paging is enabled */
-        isb                          /* Now, flush the icache */
-        ret
-ENDPROC(enable_mmu)
-
-/*
- * Remove the 1:1 map from the page-tables. It is not easy to keep track
- * where the 1:1 map was mapped, so we will look for the top-level entry
- * exclusive to the 1:1 map and remove it.
- *
- * Inputs:
- *   x19: paddr(start)
- *
- * Clobbers x0 - x1
- */
-remove_identity_mapping:
-        /*
-         * Find the zeroeth slot used. Remove the entry from zeroeth
-         * table if the slot is not XEN_ZEROETH_SLOT.
-         */
-        get_table_slot x1, x19, 0       /* x1 := zeroeth slot */
-        cmp   x1, #XEN_ZEROETH_SLOT
-        beq   1f
-        /* It is not in slot XEN_ZEROETH_SLOT, remove the entry. */
-        ldr   x0, =boot_pgtable         /* x0 := root table */
-        str   xzr, [x0, x1, lsl #3]
-        b     identity_mapping_removed
-
-1:
-        /*
-         * Find the first slot used. Remove the entry for the first
-         * table if the slot is not XEN_FIRST_SLOT.
-         */
-        get_table_slot x1, x19, 1       /* x1 := first slot */
-        cmp   x1, #XEN_FIRST_SLOT
-        beq   1f
-        /* It is not in slot XEN_FIRST_SLOT, remove the entry. */
-        ldr   x0, =boot_first           /* x0 := first table */
-        str   xzr, [x0, x1, lsl #3]
-        b     identity_mapping_removed
-
-1:
-        /*
-         * Find the second slot used. Remove the entry for the first
-         * table if the slot is not XEN_SECOND_SLOT.
-         */
-        get_table_slot x1, x19, 2       /* x1 := second slot */
-        cmp   x1, #XEN_SECOND_SLOT
-        beq   identity_mapping_removed
-        /* It is not in slot 1, remove the entry */
-        ldr   x0, =boot_second          /* x0 := second table */
-        str   xzr, [x0, x1, lsl #3]
-
-identity_mapping_removed:
-        /* See asm/arm64/flushtlb.h for the explanation of the sequence. */
-        dsb   nshst
-        tlbi  alle2
-        dsb   nsh
-        isb
-
-        ret
-ENDPROC(remove_identity_mapping)
-
-/*
- * Map the UART in the fixmap (when earlyprintk is used) and hook the
- * fixmap table in the page tables.
- *
- * The fixmap cannot be mapped in create_page_tables because it may
- * clash with the 1:1 mapping.
- *
- * Inputs:
- *   x20: Physical offset
- *   x23: Early UART base physical address
- *
- * Clobbers x0 - x3
- */
-setup_fixmap:
-#ifdef CONFIG_EARLY_PRINTK
-        /* Add UART to the fixmap table */
-        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
-        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3, type=PT_DEV_L3
-#endif
-        /* Map fixmap into boot_second */
-        ldr   x0, =FIXMAP_ADDR(0)
-        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
-        /* Ensure any page table updates made above have occurred. */
-        dsb   nshst
-
-        ret
-ENDPROC(setup_fixmap)
-
 /*
  * Setup the initial stack and jump to the C world
  *
@@ -810,41 +458,14 @@ launch:
 ENDPROC(launch)
 
 /* Fail-stop */
-fail:   PRINT("- Boot failed -\r\n")
+ENTRY(fail)
+        PRINT("- Boot failed -\r\n")
 1:      wfe
         b     1b
 ENDPROC(fail)
 
 GLOBAL(_end_boot)
 
-/*
- * Switch TTBR
- *
- * x0    ttbr
- *
- * TODO: This code does not comply with break-before-make.
- */
-ENTRY(switch_ttbr)
-        dsb   sy                     /* Ensure the flushes happen before
-                                      * continuing */
-        isb                          /* Ensure synchronization with previous
-                                      * changes to text */
-        tlbi   alle2                 /* Flush hypervisor TLB */
-        ic     iallu                 /* Flush I-cache */
-        dsb    sy                    /* Ensure completion of TLB flush */
-        isb
-
-        msr    TTBR0_EL2, x0
-
-        isb                          /* Ensure synchronization with previous
-                                      * changes to text */
-        tlbi   alle2                 /* Flush hypervisor TLB */
-        ic     iallu                 /* Flush I-cache */
-        dsb    sy                    /* Ensure completion of TLB flush */
-        isb
-
-        ret
-ENDPROC(switch_ttbr)
 
 #ifdef CONFIG_EARLY_PRINTK
 /*
@@ -868,7 +489,7 @@ ENDPROC(init_uart)
  * x0: Nul-terminated string to print.
  * x23: Early UART base address
  * Clobbers x0-x1 */
-puts:
+ENTRY(puts)
         early_uart_ready x23, 1
         ldrb  w1, [x0], #1           /* Load next char */
         cbz   w1, 1f                 /* Exit on nul */
diff --git a/xen/arch/arm/arm64/head_mmu.S b/xen/arch/arm/arm64/head_mmu.S
new file mode 100644
index 0000000000..1a3df81a38
--- /dev/null
+++ b/xen/arch/arm/arm64/head_mmu.S
@@ -0,0 +1,364 @@
+/*
+ * xen/arch/arm/head_mmu.S
+ *
+ * Start-of-day code for an ARMv8-A.
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2012 Citrix Systems.
+ *
+ * Based on ARMv7-A head.S by
+ * Tim Deegan <tim@xen.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <asm/page.h>
+#include <asm/early_printk.h>
+
+#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
+#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
+#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
+#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
+#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
+
+/* Convenience defines to get slot used by Xen mapping. */
+#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
+#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
+#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
+
+/*
+ * Macro to find the slot number at a given page-table level
+ *
+ * slot:     slot computed
+ * virt:     virtual address
+ * lvl:      page-table level
+ */
+.macro get_table_slot, slot, virt, lvl
+        ubfx  \slot, \virt, #XEN_PT_LEVEL_SHIFT(\lvl), #XEN_PT_LPAE_SHIFT
+.endm
+
+/*
+ * Macro to create a page table entry in \ptbl to \tbl
+ *
+ * ptbl:    table symbol where the entry will be created
+ * tbl:     table symbol to point to
+ * virt:    virtual address
+ * lvl:     page-table level
+ * tmp1:    scratch register
+ * tmp2:    scratch register
+ * tmp3:    scratch register
+ *
+ * Preserves \virt
+ * Clobbers \tmp1, \tmp2, \tmp3
+ *
+ * Also use x20 for the phys offset.
+ *
+ * Note that all parameters using registers should be distinct.
+ */
+.macro create_table_entry, ptbl, tbl, virt, lvl, tmp1, tmp2, tmp3
+        get_table_slot \tmp1, \virt, \lvl   /* \tmp1 := slot in \tlb */
+
+        load_paddr \tmp2, \tbl
+        mov   \tmp3, #PT_PT                 /* \tmp3 := right for linear PT */
+        orr   \tmp3, \tmp3, \tmp2           /*          + \tlb paddr */
+
+        adr_l \tmp2, \ptbl
+
+        str   \tmp3, [\tmp2, \tmp1, lsl #3]
+.endm
+
+/*
+ * Macro to create a mapping entry in \tbl to \phys. Only mapping in 3rd
+ * level table (i.e page granularity) is supported.
+ *
+ * ptbl:     table symbol where the entry will be created
+ * virt:    virtual address
+ * phys:    physical address (should be page aligned)
+ * tmp1:    scratch register
+ * tmp2:    scratch register
+ * tmp3:    scratch register
+ * type:    mapping type. If not specified it will be normal memory (PT_MEM_L3)
+ *
+ * Preserves \virt, \phys
+ * Clobbers \tmp1, \tmp2, \tmp3
+ *
+ * Note that all parameters using registers should be distinct.
+ */
+.macro create_mapping_entry, ptbl, virt, phys, tmp1, tmp2, tmp3, type=PT_MEM_L3
+        and   \tmp3, \phys, #THIRD_MASK     /* \tmp3 := PAGE_ALIGNED(phys) */
+
+        get_table_slot \tmp1, \virt, 3      /* \tmp1 := slot in \tlb */
+
+        mov   \tmp2, #\type                 /* \tmp2 := right for section PT */
+        orr   \tmp2, \tmp2, \tmp3           /*          + PAGE_ALIGNED(phys) */
+
+        adr_l \tmp3, \ptbl
+
+        str   \tmp2, [\tmp3, \tmp1, lsl #3]
+.endm
+
+.section .text.header, "ax", %progbits
+/*.aarch64*/
+
+/*
+ * Rebuild the boot pagetable's first-level entries. The structure
+ * is described in mm.c.
+ *
+ * After the CPU enables paging it will add the fixmap mapping
+ * to these page tables, however this may clash with the 1:1
+ * mapping. So each CPU must rebuild the page tables here with
+ * the 1:1 in place.
+ *
+ * Inputs:
+ *   x19: paddr(start)
+ *   x20: phys offset
+ *
+ * Clobbers x0 - x4
+ */
+ENTRY(create_page_tables)
+        /* Prepare the page-tables for mapping Xen */
+        ldr   x0, =XEN_VIRT_START
+        create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
+        create_table_entry boot_first, boot_second, x0, 1, x1, x2, x3
+        create_table_entry boot_second, boot_third, x0, 2, x1, x2, x3
+
+        /* Map Xen */
+        adr_l x4, boot_third
+
+        lsr   x2, x19, #THIRD_SHIFT  /* Base address for 4K mapping */
+        lsl   x2, x2, #THIRD_SHIFT
+        mov   x3, #PT_MEM_L3         /* x2 := Section map */
+        orr   x2, x2, x3
+
+        /* ... map of vaddr(start) in boot_third */
+        mov   x1, xzr
+1:      str   x2, [x4, x1]           /* Map vaddr(start) */
+        add   x2, x2, #PAGE_SIZE     /* Next page */
+        add   x1, x1, #8             /* Next slot */
+        cmp   x1, #(XEN_PT_LPAE_ENTRIES<<3) /* 512 entries per page */
+        b.lt  1b
+
+        /*
+         * If Xen is loaded at exactly XEN_VIRT_START then we don't
+         * need an additional 1:1 mapping, the virtual mapping will
+         * suffice.
+         */
+        cmp   x19, #XEN_VIRT_START
+        bne   1f
+        ret
+1:
+        /*
+         * Setup the 1:1 mapping so we can turn the MMU on. Note that
+         * only the first page of Xen will be part of the 1:1 mapping.
+         */
+
+        /*
+         * Find the zeroeth slot used. If the slot is not
+         * XEN_ZEROETH_SLOT, then the 1:1 mapping will use its own set of
+         * page-tables from the first level.
+         */
+        get_table_slot x0, x19, 0       /* x0 := zeroeth slot */
+        cmp   x0, #XEN_ZEROETH_SLOT
+        beq   1f
+        create_table_entry boot_pgtable, boot_first_id, x19, 0, x0, x1, x2
+        b     link_from_first_id
+
+1:
+        /*
+         * Find the first slot used. If the slot is not XEN_FIRST_SLOT,
+         * then the 1:1 mapping will use its own set of page-tables from
+         * the second level.
+         */
+        get_table_slot x0, x19, 1      /* x0 := first slot */
+        cmp   x0, #XEN_FIRST_SLOT
+        beq   1f
+        create_table_entry boot_first, boot_second_id, x19, 1, x0, x1, x2
+        b     link_from_second_id
+
+1:
+        /*
+         * Find the second slot used. If the slot is XEN_SECOND_SLOT, then the
+         * 1:1 mapping will use its own set of page-tables from the
+         * third level. For slot XEN_SECOND_SLOT, Xen is not yet able to handle
+         * it.
+         */
+        get_table_slot x0, x19, 2     /* x0 := second slot */
+        cmp   x0, #XEN_SECOND_SLOT
+        beq   virtphys_clash
+        create_table_entry boot_second, boot_third_id, x19, 2, x0, x1, x2
+        b     link_from_third_id
+
+link_from_first_id:
+        create_table_entry boot_first_id, boot_second_id, x19, 1, x0, x1, x2
+link_from_second_id:
+        create_table_entry boot_second_id, boot_third_id, x19, 2, x0, x1, x2
+link_from_third_id:
+        create_mapping_entry boot_third_id, x19, x19, x0, x1, x2
+        ret
+
+virtphys_clash:
+        /* Identity map clashes with boot_third, which we cannot handle yet */
+        PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
+        b     fail
+ENDPROC(create_page_tables)
+
+/*
+ * Turn on the Data Cache and the MMU. The function will return on the 1:1
+ * mapping. In other word, the caller is responsible to switch to the runtime
+ * mapping.
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(enable_mmu)
+        PRINT("- Turning on paging -\r\n")
+
+        /*
+         * The state of the TLBs is unknown before turning on the MMU.
+         * Flush them to avoid stale one.
+         */
+        tlbi  alle2                  /* Flush hypervisor TLBs */
+        dsb   nsh
+
+        /* Write Xen's PT's paddr into TTBR0_EL2 */
+        load_paddr x0, boot_pgtable
+        msr   TTBR0_EL2, x0
+        isb
+
+        mrs   x0, SCTLR_EL2
+        orr   x0, x0, #SCTLR_Axx_ELx_M  /* Enable MMU */
+        orr   x0, x0, #SCTLR_Axx_ELx_C  /* Enable D-cache */
+        dsb   sy                     /* Flush PTE writes and finish reads */
+        msr   SCTLR_EL2, x0          /* now paging is enabled */
+        isb                          /* Now, flush the icache */
+        ret
+ENDPROC(enable_mmu)
+
+/*
+ * Remove the 1:1 map from the page-tables. It is not easy to keep track
+ * where the 1:1 map was mapped, so we will look for the top-level entry
+ * exclusive to the 1:1 map and remove it.
+ *
+ * Inputs:
+ *   x19: paddr(start)
+ *
+ * Clobbers x0 - x1
+ */
+ENTRY(remove_identity_mapping)
+        /*
+         * Find the zeroeth slot used. Remove the entry from zeroeth
+         * table if the slot is not XEN_ZEROETH_SLOT.
+         */
+        get_table_slot x1, x19, 0       /* x1 := zeroeth slot */
+        cmp   x1, #XEN_ZEROETH_SLOT
+        beq   1f
+        /* It is not in slot XEN_ZEROETH_SLOT, remove the entry. */
+        ldr   x0, =boot_pgtable         /* x0 := root table */
+        str   xzr, [x0, x1, lsl #3]
+        b     identity_mapping_removed
+
+1:
+        /*
+         * Find the first slot used. Remove the entry for the first
+         * table if the slot is not XEN_FIRST_SLOT.
+         */
+        get_table_slot x1, x19, 1       /* x1 := first slot */
+        cmp   x1, #XEN_FIRST_SLOT
+        beq   1f
+        /* It is not in slot XEN_FIRST_SLOT, remove the entry. */
+        ldr   x0, =boot_first           /* x0 := first table */
+        str   xzr, [x0, x1, lsl #3]
+        b     identity_mapping_removed
+
+1:
+        /*
+         * Find the second slot used. Remove the entry for the first
+         * table if the slot is not XEN_SECOND_SLOT.
+         */
+        get_table_slot x1, x19, 2       /* x1 := second slot */
+        cmp   x1, #XEN_SECOND_SLOT
+        beq   identity_mapping_removed
+        /* It is not in slot 1, remove the entry */
+        ldr   x0, =boot_second          /* x0 := second table */
+        str   xzr, [x0, x1, lsl #3]
+
+identity_mapping_removed:
+        /* See asm/arm64/flushtlb.h for the explanation of the sequence. */
+        dsb   nshst
+        tlbi  alle2
+        dsb   nsh
+        isb
+
+        ret
+ENDPROC(remove_identity_mapping)
+
+/*
+ * Map the UART in the fixmap (when earlyprintk is used) and hook the
+ * fixmap table in the page tables.
+ *
+ * The fixmap cannot be mapped in create_page_tables because it may
+ * clash with the 1:1 mapping.
+ *
+ * Inputs:
+ *   x20: Physical offset
+ *   x23: Early UART base physical address
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(setup_fixmap)
+#ifdef CONFIG_EARLY_PRINTK
+        /* Add UART to the fixmap table */
+        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
+        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3, type=PT_DEV_L3
+#endif
+        /* Map fixmap into boot_second */
+        ldr   x0, =FIXMAP_ADDR(0)
+        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
+        /* Ensure any page table updates made above have occurred. */
+        dsb   nshst
+
+        ret
+ENDPROC(setup_fixmap)
+
+/*
+ * Switch TTBR
+ *
+ * x0    ttbr
+ *
+ * TODO: This code does not comply with break-before-make.
+ */
+ENTRY(switch_ttbr)
+        dsb   sy                     /* Ensure the flushes happen before
+                                      * continuing */
+        isb                          /* Ensure synchronization with previous
+                                      * changes to text */
+        tlbi   alle2                 /* Flush hypervisor TLB */
+        ic     iallu                 /* Flush I-cache */
+        dsb    sy                    /* Ensure completion of TLB flush */
+        isb
+
+        msr    TTBR0_EL2, x0
+
+        isb                          /* Ensure synchronization with previous
+                                      * changes to text */
+        tlbi   alle2                 /* Flush hypervisor TLB */
+        ic     iallu                 /* Flush I-cache */
+        dsb    sy                    /* Ensure completion of TLB flush */
+        isb
+
+        ret
+ENDPROC(switch_ttbr)
+
+/*
+ * Local variables:
+ * mode: ASM
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/include/asm/arm64/macros.h b/xen/arch/arm/include/asm/arm64/macros.h
index 140e223b4c..462dc70335 100644
--- a/xen/arch/arm/include/asm/arm64/macros.h
+++ b/xen/arch/arm/include/asm/arm64/macros.h
@@ -32,10 +32,60 @@
         hint    #22
     .endm
 
+    /*
+     * Pseudo-op for PC relative adr <reg>, <symbol> where <symbol> is
+     * within the range +/- 4GB of the PC.
+     *
+     * @dst: destination register (64 bit wide)
+     * @sym: name of the symbol
+     */
+    .macro  adr_l, dst, sym
+        adrp \dst, \sym
+        add  \dst, \dst, :lo12:\sym
+    .endm
+
+    /* Load the physical address of a symbol into xb */
+    .macro load_paddr xb, sym
+        ldr \xb, =\sym
+        add \xb, \xb, x20
+    .endm
+
 /*
  * Register aliases.
  */
 lr      .req    x30             /* link register */
 
-#endif /* __ASM_ARM_ARM64_MACROS_H */
+#ifdef CONFIG_EARLY_PRINTK
+/*
+ * Macro to print a string to the UART, if there is one.
+ *
+ * Clobbers x0 - x3
+ */
+#define PRINT(_s)          \
+        mov   x3, lr ;     \
+        adr   x0, 98f ;    \
+        bl    puts    ;    \
+        mov   lr, x3 ;     \
+        RODATA_STR(98, _s)
 
+    /*
+     * Macro to print the value of register \xb
+     *
+     * Clobbers x0 - x4
+     */
+    .macro print_reg xb
+    mov   x0, \xb
+    mov   x4, lr
+    bl    putn
+    mov   lr, x4
+    .endm
+
+#else /* CONFIG_EARLY_PRINTK */
+#define PRINT(s)
+
+.macro print_reg xb
+.endm
+
+#endif /* !CONFIG_EARLY_PRINTK */
+
+#endif /* __ASM_ARM_ARM64_MACROS_H */
-- 
2.25.1

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> There are lots of MMU specific code in head.S. This code will not
> be used in MPU systems. If we use #ifdef to gate them, the code
> will become messy and hard to maintain. So we move MMU related
> code to head_mmu.S, and keep common code still in head.S.

I am afraid that you can't simply move the MMU code out of head.S 
because this will break Xen when running using the identity map.

This is because we only map the first 4KB of Xen with PA == VA. At the 
moment, we guarantee it by having everything that needs to be used in 
the identity mapping before _end_boot and checking at link time if this 
fits in 4KB.

Now that you moved the MMU code outside of head.S. We need to find a 
different way to guarantee it. One way to do it would be to create a 
section that would be used for everything that needs to be identity mapped.

> 
> As we need to access "fail" and "puts" functions out of assembly
> file, so we have to export them in this patch. And the assembly
> macros: adr_l and load_paddr will be used by MMU and MPU later,
> so we move them to macros.h.
> 
> Signed-off-by: Henry Wang <Henry.Wang@arm.com>
> Signed-off-by: Wei Chen <wei.chen@arm.com>

In general, the first signed-off should match the author. So who is who 
here?

> ---
>   xen/arch/arm/arm64/Makefile             |   3 +
>   xen/arch/arm/arm64/head.S               | 407 +-----------------------
>   xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
>   xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
>   4 files changed, 432 insertions(+), 394 deletions(-)
>   create mode 100644 xen/arch/arm/arm64/head_mmu.S
> 
> diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
> index 6d507da0d4..22da2f54b5 100644
> --- a/xen/arch/arm/arm64/Makefile
> +++ b/xen/arch/arm/arm64/Makefile
> @@ -8,6 +8,9 @@ obj-y += domctl.o
>   obj-y += domain.o
>   obj-y += entry.o
>   obj-y += head.o
> +ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o
> +endif
>   obj-y += insn.o
>   obj-$(CONFIG_LIVEPATCH) += livepatch.o
>   obj-y += smc.o
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index ccedf20dc7..d9a8da9120 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -25,17 +25,6 @@
>   #include <efi/efierr.h>
>   #include <asm/arm64/efibind.h>
>   
> -#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
> -#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
> -#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
> -#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
> -#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
> -
> -/* Convenience defines to get slot used by Xen mapping. */
> -#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
> -#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
> -#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
> -
>   #define __HEAD_FLAG_PAGE_SIZE   ((PAGE_SHIFT - 10) / 2)
>   
>   #define __HEAD_FLAG_PHYS_BASE   1
> @@ -82,73 +71,22 @@
>    *  x30 - lr
>    */
>   
> -#ifdef CONFIG_EARLY_PRINTK
> -/*
> - * Macro to print a string to the UART, if there is one.
> - *
> - * Clobbers x0 - x3
> - */
> -#define PRINT(_s)          \
> -        mov   x3, lr ;     \
> -        adr   x0, 98f ;    \
> -        bl    puts    ;    \
> -        mov   lr, x3 ;     \
> -        RODATA_STR(98, _s)
> +.section .text.header, "ax", %progbits
> +/*.aarch64*/

The patch is already quite difficult to read. So I would rather prefer 
if the indentation is changed separately.

Furthermore, I think it would be best if the functions moved in the 
header are done separately to help checking (I would be able to diff the 
source with the destination more easily).

>   
>   /*
> - * Macro to print the value of register \xb
> + * Kernel startup entry point.
> + * ---------------------------

Same here about the indentation. I will not comment everywhere where the 
indentation was changed. So please look at it.

[...]

> -/*
> - * Map the UART in the fixmap (when earlyprintk is used) and hook the
> - * fixmap table in the page tables.
> - *
> - * The fixmap cannot be mapped in create_page_tables because it may
> - * clash with the 1:1 mapping.
> - *
> - * Inputs:
> - *   x20: Physical offset
> - *   x23: Early UART base physical address
> - *
> - * Clobbers x0 - x3
> - */
> -setup_fixmap:
> -#ifdef CONFIG_EARLY_PRINTK
> -        /* Add UART to the fixmap table */
> -        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
> -        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3, type=PT_DEV_L3
> -#endif
> -        /* Map fixmap into boot_second */
> -        ldr   x0, =FIXMAP_ADDR(0)
> -        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
> -        /* Ensure any page table updates made above have occurred. */
> -        dsb   nshst
> -
> -        ret
> -ENDPROC(setup_fixmap)
> -
>   /*
>    * Setup the initial stack and jump to the C world
>    *
> @@ -810,41 +458,14 @@ launch:
>   ENDPROC(launch)
>   
>   /* Fail-stop */
> -fail:   PRINT("- Boot failed -\r\n")
> +ENTRY(fail)

This name is a bit too generic to be exposed. But it would be better to 
duplicate it.

Cheers,

-- 
Julien Grall

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Julien Grall]

On 06/11/2022 20:06, Julien Grall wrote:
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
>> There are lots of MMU specific code in head.S. This code will not
>> be used in MPU systems. If we use #ifdef to gate them, the code
>> will become messy and hard to maintain. So we move MMU related
>> code to head_mmu.S, and keep common code still in head.S.
> 
> I am afraid that you can't simply move the MMU code out of head.S 
> because this will break Xen when running using the identity map.
> 
> This is because we only map the first 4KB of Xen with PA == VA. At the 
> moment, we guarantee it by having everything that needs to be used in 
> the identity mapping before _end_boot and checking at link time if this 
> fits in 4KB.
> 
> Now that you moved the MMU code outside of head.S. We need to find a 
> different way to guarantee it. One way to do it would be to create a 
> section that would be used for everything that needs to be identity mapped.

Looking at the code this morning, I noticed that we already have the 
section ".text.header". For now, that should do the job. So we just need
to check the size of .text.header.

Ideally, checking the size should be done in a separate pre-patch so it 
is easier to review.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 4:06
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>; Henry Wang <Henry.Wang@arm.com>
> Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S
> to head_mmu.S
> 
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > There are lots of MMU specific code in head.S. This code will not
> > be used in MPU systems. If we use #ifdef to gate them, the code
> > will become messy and hard to maintain. So we move MMU related
> > code to head_mmu.S, and keep common code still in head.S.
> 
> I am afraid that you can't simply move the MMU code out of head.S
> because this will break Xen when running using the identity map.
> 
> This is because we only map the first 4KB of Xen with PA == VA. At the
> moment, we guarantee it by having everything that needs to be used in
> the identity mapping before _end_boot and checking at link time if this
> fits in 4KB.
> 
> Now that you moved the MMU code outside of head.S. We need to find a
> different way to guarantee it. One way to do it would be to create a
> section that would be used for everything that needs to be identity mapped.
> 

Quote from next email
"
Looking at the code this morning, I noticed that we already have the 
section ".text.header". For now, that should do the job. So we just need
to check the size of .text.header.

Ideally, checking the size should be done in a separate pre-patch so it 
is easier to review.
"

OK. We will create a patch to check the size, and place it in the
head of the series.

> >
> > As we need to access "fail" and "puts" functions out of assembly
> > file, so we have to export them in this patch. And the assembly
> > macros: adr_l and load_paddr will be used by MMU and MPU later,
> > so we move them to macros.h.
> >
> > Signed-off-by: Henry Wang <Henry.Wang@arm.com>
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> 
> In general, the first signed-off should match the author. So who is who
> here?
> 

I will adjust this order.

> > ---
> >   xen/arch/arm/arm64/Makefile             |   3 +
> >   xen/arch/arm/arm64/head.S               | 407 +-----------------------
> >   xen/arch/arm/arm64/head_mmu.S           | 364 +++++++++++++++++++++
> >   xen/arch/arm/include/asm/arm64/macros.h |  52 ++-
> >   4 files changed, 432 insertions(+), 394 deletions(-)
> >   create mode 100644 xen/arch/arm/arm64/head_mmu.S
> >
> > diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
> > index 6d507da0d4..22da2f54b5 100644
> > --- a/xen/arch/arm/arm64/Makefile
> > +++ b/xen/arch/arm/arm64/Makefile
> > @@ -8,6 +8,9 @@ obj-y += domctl.o
> >   obj-y += domain.o
> >   obj-y += entry.o
> >   obj-y += head.o
> > +ifneq ($(CONFIG_HAS_MPU),y) > +obj-y += head_mmu.o
> > +endif
> >   obj-y += insn.o
> >   obj-$(CONFIG_LIVEPATCH) += livepatch.o
> >   obj-y += smc.o
> > diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> > index ccedf20dc7..d9a8da9120 100644
> > --- a/xen/arch/arm/arm64/head.S
> > +++ b/xen/arch/arm/arm64/head.S
> > @@ -25,17 +25,6 @@
> >   #include <efi/efierr.h>
> >   #include <asm/arm64/efibind.h>
> >
> > -#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1
> */
> > -#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1
> */
> > -#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1
> */
> > -#define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1
> */
> > -#define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1
> */
> > -
> > -/* Convenience defines to get slot used by Xen mapping. */
> > -#define XEN_ZEROETH_SLOT    zeroeth_table_offset(XEN_VIRT_START)
> > -#define XEN_FIRST_SLOT      first_table_offset(XEN_VIRT_START)
> > -#define XEN_SECOND_SLOT     second_table_offset(XEN_VIRT_START)
> > -
> >   #define __HEAD_FLAG_PAGE_SIZE   ((PAGE_SHIFT - 10) / 2)
> >
> >   #define __HEAD_FLAG_PHYS_BASE   1
> > @@ -82,73 +71,22 @@
> >    *  x30 - lr
> >    */
> >
> > -#ifdef CONFIG_EARLY_PRINTK
> > -/*
> > - * Macro to print a string to the UART, if there is one.
> > - *
> > - * Clobbers x0 - x3
> > - */
> > -#define PRINT(_s)          \
> > -        mov   x3, lr ;     \
> > -        adr   x0, 98f ;    \
> > -        bl    puts    ;    \
> > -        mov   lr, x3 ;     \
> > -        RODATA_STR(98, _s)
> > +.section .text.header, "ax", %progbits
> > +/*.aarch64*/
> 
> The patch is already quite difficult to read. So I would rather prefer
> if the indentation is changed separately.
> 

Ok.

> Furthermore, I think it would be best if the functions moved in the
> header are done separately to help checking (I would be able to diff the
> source with the destination more easily).
> 

Did you mean to create a separate patch for moving the functions in macro.h?
Or in header section?

> >
> >   /*
> > - * Macro to print the value of register \xb
> > + * Kernel startup entry point.
> > + * ---------------------------
> 
> Same here about the indentation. I will not comment everywhere where the
> indentation was changed. So please look at it.
> 

OK. we will use a separate patch to clean them.

> [...]
> 
> > -/*
> > - * Map the UART in the fixmap (when earlyprintk is used) and hook the
> > - * fixmap table in the page tables.
> > - *
> > - * The fixmap cannot be mapped in create_page_tables because it may
> > - * clash with the 1:1 mapping.
> > - *
> > - * Inputs:
> > - *   x20: Physical offset
> > - *   x23: Early UART base physical address
> > - *
> > - * Clobbers x0 - x3
> > - */
> > -setup_fixmap:
> > -#ifdef CONFIG_EARLY_PRINTK
> > -        /* Add UART to the fixmap table */
> > -        ldr   x0, =EARLY_UART_VIRTUAL_ADDRESS
> > -        create_mapping_entry xen_fixmap, x0, x23, x1, x2, x3,
> type=PT_DEV_L3
> > -#endif
> > -        /* Map fixmap into boot_second */
> > -        ldr   x0, =FIXMAP_ADDR(0)
> > -        create_table_entry boot_second, xen_fixmap, x0, 2, x1, x2, x3
> > -        /* Ensure any page table updates made above have occurred. */
> > -        dsb   nshst
> > -
> > -        ret
> > -ENDPROC(setup_fixmap)
> > -
> >   /*
> >    * Setup the initial stack and jump to the C world
> >    *
> > @@ -810,41 +458,14 @@ launch:
> >   ENDPROC(launch)
> >
> >   /* Fail-stop */
> > -fail:   PRINT("- Boot failed -\r\n")
> > +ENTRY(fail)
> 
> This name is a bit too generic to be exposed. But it would be better to
> duplicate it.

Yes, duplicate it might be better. We will do it in next version.

Cheers,
Wei Chen

> 
> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Julien Grall]

On 09/11/2022 07:36, Wei Chen wrote:
> Hi Julien,

Hi Wei,

>> The patch is already quite difficult to read. So I would rather prefer
>> if the indentation is changed separately.
>>
> 
> Ok.
> 
>> Furthermore, I think it would be best if the functions moved in the
>> header are done separately to help checking (I would be able to diff the
>> source with the destination more easily).
>>
> 
> Did you mean to create a separate patch for moving the functions in macro.h?

I mean the macros in macro.h. Sorry, I misquoted it.

Cheers,

-- 
Julien Grall

Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Julien Grall]

On 09/11/2022 07:36, Wei Chen wrote:
> Hi Julien,

Hi Wei,

> 
>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: 2022年11月7日 4:06
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
>> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>; Henry Wang <Henry.Wang@arm.com>
>> Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S
>> to head_mmu.S
>>
>> Hi Wei,
>>
>> On 04/11/2022 10:07, Wei Chen wrote:
>>> There are lots of MMU specific code in head.S. This code will not
>>> be used in MPU systems. If we use #ifdef to gate them, the code
>>> will become messy and hard to maintain. So we move MMU related
>>> code to head_mmu.S, and keep common code still in head.S.
>>
>> I am afraid that you can't simply move the MMU code out of head.S
>> because this will break Xen when running using the identity map.
>>
>> This is because we only map the first 4KB of Xen with PA == VA. At the
>> moment, we guarantee it by having everything that needs to be used in
>> the identity mapping before _end_boot and checking at link time if this
>> fits in 4KB.
>>
>> Now that you moved the MMU code outside of head.S. We need to find a
>> different way to guarantee it. One way to do it would be to create a
>> section that would be used for everything that needs to be identity mapped.
>>
> 
> Quote from next email
> "
> Looking at the code this morning, I noticed that we already have the
> section ".text.header". For now, that should do the job. So we just need
> to check the size of .text.header.
> 
> Ideally, checking the size should be done in a separate pre-patch so it
> is easier to review.
> "
> 
> OK. We will create a patch to check the size, and place it in the
> head of the series.

I thought a bit more about what I wrote. Xen binary should always start 
with the Image/Zimage header. At the moment, this is guaranteed because 
there is only one object using the section .text.header.

With the change introduced in this patch, there will be multiple objects 
using with the sections .text.header. This means we are relying on the 
compiler to always put the content of head.S first.

This is basically reverting to the behavior before commit 4267a33b19d 
("xen/build: put image header into a separate section").

Therefore we do need a separate section to be used for head_*.S and 
maybe part of head.S. This new section could be called text.idmap so we 
know what the section is used for.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S to head_mmu.S

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月14日 5:43
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>; Henry Wang <Henry.Wang@arm.com>
> Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from head.S
> to head_mmu.S
> 
> 
> 
> On 09/11/2022 07:36, Wei Chen wrote:
> > Hi Julien,
> 
> Hi Wei,
> 
> >
> >> -----Original Message-----
> >> From: Julien Grall <julien@xen.org>
> >> Sent: 2022年11月7日 4:06
> >> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> >> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Bertrand
> >> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> >> <Volodymyr_Babchuk@epam.com>; Henry Wang <Henry.Wang@arm.com>
> >> Subject: Re: [PATCH v6 08/11] xen/arm64: move MMU related code from
> head.S
> >> to head_mmu.S
> >>
> >> Hi Wei,
> >>
> >> On 04/11/2022 10:07, Wei Chen wrote:
> >>> There are lots of MMU specific code in head.S. This code will not
> >>> be used in MPU systems. If we use #ifdef to gate them, the code
> >>> will become messy and hard to maintain. So we move MMU related
> >>> code to head_mmu.S, and keep common code still in head.S.
> >>
> >> I am afraid that you can't simply move the MMU code out of head.S
> >> because this will break Xen when running using the identity map.
> >>
> >> This is because we only map the first 4KB of Xen with PA == VA. At the
> >> moment, we guarantee it by having everything that needs to be used in
> >> the identity mapping before _end_boot and checking at link time if this
> >> fits in 4KB.
> >>
> >> Now that you moved the MMU code outside of head.S. We need to find a
> >> different way to guarantee it. One way to do it would be to create a
> >> section that would be used for everything that needs to be identity
> mapped.
> >>
> >
> > Quote from next email
> > "
> > Looking at the code this morning, I noticed that we already have the
> > section ".text.header". For now, that should do the job. So we just need
> > to check the size of .text.header.
> >
> > Ideally, checking the size should be done in a separate pre-patch so it
> > is easier to review.
> > "
> >
> > OK. We will create a patch to check the size, and place it in the
> > head of the series.
> 
> I thought a bit more about what I wrote. Xen binary should always start
> with the Image/Zimage header. At the moment, this is guaranteed because
> there is only one object using the section .text.header.
> 
> With the change introduced in this patch, there will be multiple objects
> using with the sections .text.header. This means we are relying on the
> compiler to always put the content of head.S first.
> 

Yes, now it depends on the objects' order in Makefile. This should be
a risk.

> This is basically reverting to the behavior before commit 4267a33b19d
> ("xen/build: put image header into a separate section").
> 
> Therefore we do need a separate section to be used for head_*.S and
> maybe part of head.S. This new section could be called text.idmap so we
> know what the section is used for.
> 

Yes, we might place this new section after text.header.

Cheers,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

[PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

[Wei Chen]

From: Penny Zheng <penny.zheng@arm.com>

Like boot-time page table in MMU system, we need a boot-time
MPU protection region configuration in MPU system so Xen can
fetch code and data from normal memory.

This operation need to access Armv8-R MPU system registers, but
these system registers are not supported in GCC version < 11.
So we have to encode these Armv8-R MPU system registers in header
file explicitly.

As MMU system and MPU system have different functions to create
the boot MMU/MPU data, this will introduce extra #ifdef in code
flow, so we introduce a neutral name prepare_early_mappings to
replace create_page_tables for MMU and MPU.

Signed-off-by: Wei Chen <wei.chen@arm.com>
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 xen/arch/arm/arm64/Makefile              |  2 +
 xen/arch/arm/arm64/head.S                | 13 ++--
 xen/arch/arm/arm64/head_mmu.S            |  4 +-
 xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
 xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
 xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++
 6 files changed, 185 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h

diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
index 22da2f54b5..438c9737ad 100644
--- a/xen/arch/arm/arm64/Makefile
+++ b/xen/arch/arm/arm64/Makefile
@@ -10,6 +10,8 @@ obj-y += entry.o
 obj-y += head.o
 ifneq ($(CONFIG_HAS_MPU),y)
 obj-y += head_mmu.o
+else
+obj-y += head_mpu.o
 endif
 obj-y += insn.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index d9a8da9120..6c1a5f74a1 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -79,12 +79,12 @@
  * ---------------------------
  *
  * The requirements are:
- *   MMU = off, D-cache = off, I-cache = on or off,
+ *   MMU/MPU = off, D-cache = off, I-cache = on or off,
  *   x0 = physical address to the FDT blob.
  *
  * This must be the very first address in the loaded image.
  * It should be linked at XEN_VIRT_START, and loaded at any
- * 4K-aligned address.  All of text+data+bss must fit in 2MB,
+ * 4K-aligned address. All of text+data+bss must fit in 2MB,
  * or the initial pagetable code below will need adjustment.
  */
 
@@ -249,7 +249,12 @@ real_start_efi:
 
         bl    check_cpu_mode
         bl    cpu_init
-        bl    create_page_tables
+
+        /*
+         * Create boot memory management data, pagetable for MMU systems
+         * and protection regions for MPU systems.
+         */
+        bl    prepare_early_mappings
         bl    enable_mmu
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
@@ -307,7 +312,7 @@ GLOBAL(init_secondary)
 #endif
         bl    check_cpu_mode
         bl    cpu_init
-        bl    create_page_tables
+        bl    prepare_early_mappings
         bl    enable_mmu
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
diff --git a/xen/arch/arm/arm64/head_mmu.S b/xen/arch/arm/arm64/head_mmu.S
index 1a3df81a38..fc64819a98 100644
--- a/xen/arch/arm/arm64/head_mmu.S
+++ b/xen/arch/arm/arm64/head_mmu.S
@@ -123,7 +123,7 @@
  *
  * Clobbers x0 - x4
  */
-ENTRY(create_page_tables)
+ENTRY(prepare_early_mappings)
         /* Prepare the page-tables for mapping Xen */
         ldr   x0, =XEN_VIRT_START
         create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
@@ -208,7 +208,7 @@ virtphys_clash:
         /* Identity map clashes with boot_third, which we cannot handle yet */
         PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
         b     fail
-ENDPROC(create_page_tables)
+ENDPROC(prepare_early_mappings)
 
 /*
  * Turn on the Data Cache and the MMU. The function will return on the 1:1
diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
new file mode 100644
index 0000000000..f60611b556
--- /dev/null
+++ b/xen/arch/arm/arm64/head_mpu.S
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Start-of-day code for an Armv8-R MPU system.
+ */
+
+#include <asm/arm64/mpu.h>
+#include <asm/page.h>
+#include <asm/early_printk.h>
+
+/*
+ * From the requirements of head.S we know that Xen image should
+ * be linked at XEN_START_ADDRESS, and all of text + data + bss
+ * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the
+ * address that Xen image should be loaded at. So for initial MPU
+ * regions setup, we use 2MB for Xen data memory to setup boot
+ * region, or the create boot regions code below will need adjustment.
+ */
+#define XEN_START_MEM_SIZE      0x200000
+
+/*
+ * In boot stage, we will use 1 MPU region:
+ * Region#0: Normal memory for Xen text + data + bss (2MB)
+ */
+#define BOOT_NORMAL_REGION_IDX  0x0
+
+/* MPU normal memory attributes. */
+#define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
+#define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
+
+.macro write_pr, sel, prbar, prlar
+    msr   PRSELR_EL2, \sel
+    dsb   sy
+    msr   PRBAR_EL2, \prbar
+    msr   PRLAR_EL2, \prlar
+    dsb   sy
+    isb
+.endm
+
+.section .text.header, "ax", %progbits
+
+/*
+ * Static start-of-day EL2 MPU memory layout.
+ *
+ * It has a very simple structure, including:
+ *  - 2MB normal memory mappings of xen at XEN_START_ADDRESS, which
+ * is the address where Xen was loaded by the bootloader.
+ */
+ENTRY(prepare_early_mappings)
+    /* Map Xen start memory to a normal memory region. */
+    mov x0, #BOOT_NORMAL_REGION_IDX
+    ldr x1, =XEN_START_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_NORMAL_MEM
+    orr x1, x1, x3
+
+    ldr x2, =XEN_START_ADDRESS
+    mov x3, #(XEN_START_MEM_SIZE - 1)
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_NORMAL_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar
+     */
+    write_pr x0, x1, x2
+
+    ret
+ENDPROC(prepare_early_mappings)
diff --git a/xen/arch/arm/include/asm/arm64/mpu.h b/xen/arch/arm/include/asm/arm64/mpu.h
new file mode 100644
index 0000000000..d209eef6db
--- /dev/null
+++ b/xen/arch/arm/include/asm/arm64/mpu.h
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * mpu.h: Arm Memory Protection Unit definitions.
+ */
+
+#ifndef __ARM64_MPU_H__
+#define __ARM64_MPU_H__
+
+#define MPU_REGION_SHIFT  6
+#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
+#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
+
+#endif /* __ARM64_MPU_H__ */
diff --git a/xen/arch/arm/include/asm/arm64/sysregs.h b/xen/arch/arm/include/asm/arm64/sysregs.h
index 54670084c3..a596042d6c 100644
--- a/xen/arch/arm/include/asm/arm64/sysregs.h
+++ b/xen/arch/arm/include/asm/arm64/sysregs.h
@@ -458,6 +458,95 @@
 #define ZCR_ELx_LEN_SIZE             9
 #define ZCR_ELx_LEN_MASK             0x1ff
 
+/* System registers for AArch64 with PMSA */
+#ifdef CONFIG_HAS_MPU
+
+/* EL1 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL1   S3_0_C6_C8_0
+#define PRBAR1_EL1  S3_0_C6_C8_4
+#define PRBAR2_EL1  S3_0_C6_C9_0
+#define PRBAR3_EL1  S3_0_C6_C9_4
+#define PRBAR4_EL1  S3_0_C6_C10_0
+#define PRBAR5_EL1  S3_0_C6_C10_4
+#define PRBAR6_EL1  S3_0_C6_C11_0
+#define PRBAR7_EL1  S3_0_C6_C11_4
+#define PRBAR8_EL1  S3_0_C6_C12_0
+#define PRBAR9_EL1  S3_0_C6_C12_4
+#define PRBAR10_EL1 S3_0_C6_C13_0
+#define PRBAR11_EL1 S3_0_C6_C13_4
+#define PRBAR12_EL1 S3_0_C6_C14_0
+#define PRBAR13_EL1 S3_0_C6_C14_4
+#define PRBAR14_EL1 S3_0_C6_C15_0
+#define PRBAR15_EL1 S3_0_C6_C15_4
+
+/* EL1 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL1   S3_0_C6_C8_1
+#define PRLAR1_EL1  S3_0_C6_C8_5
+#define PRLAR2_EL1  S3_0_C6_C9_1
+#define PRLAR3_EL1  S3_0_C6_C9_5
+#define PRLAR4_EL1  S3_0_C6_C10_1
+#define PRLAR5_EL1  S3_0_C6_C10_5
+#define PRLAR6_EL1  S3_0_C6_C11_1
+#define PRLAR7_EL1  S3_0_C6_C11_5
+#define PRLAR8_EL1  S3_0_C6_C12_1
+#define PRLAR9_EL1  S3_0_C6_C12_5
+#define PRLAR10_EL1 S3_0_C6_C13_1
+#define PRLAR11_EL1 S3_0_C6_C13_5
+#define PRLAR12_EL1 S3_0_C6_C14_1
+#define PRLAR13_EL1 S3_0_C6_C14_5
+#define PRLAR14_EL1 S3_0_C6_C15_1
+#define PRLAR15_EL1 S3_0_C6_C15_5
+
+/* EL2 MPU Protection Region Base Address Register encode */
+#define PRBAR_EL2   S3_4_C6_C8_0
+#define PRBAR1_EL2  S3_4_C6_C8_4
+#define PRBAR2_EL2  S3_4_C6_C9_0
+#define PRBAR3_EL2  S3_4_C6_C9_4
+#define PRBAR4_EL2  S3_4_C6_C10_0
+#define PRBAR5_EL2  S3_4_C6_C10_4
+#define PRBAR6_EL2  S3_4_C6_C11_0
+#define PRBAR7_EL2  S3_4_C6_C11_4
+#define PRBAR8_EL2  S3_4_C6_C12_0
+#define PRBAR9_EL2  S3_4_C6_C12_4
+#define PRBAR10_EL2 S3_4_C6_C13_0
+#define PRBAR11_EL2 S3_4_C6_C13_4
+#define PRBAR12_EL2 S3_4_C6_C14_0
+#define PRBAR13_EL2 S3_4_C6_C14_4
+#define PRBAR14_EL2 S3_4_C6_C15_0
+#define PRBAR15_EL2 S3_4_C6_C15_4
+
+/* EL2 MPU Protection Region Limit Address Register encode */
+#define PRLAR_EL2   S3_4_C6_C8_1
+#define PRLAR1_EL2  S3_4_C6_C8_5
+#define PRLAR2_EL2  S3_4_C6_C9_1
+#define PRLAR3_EL2  S3_4_C6_C9_5
+#define PRLAR4_EL2  S3_4_C6_C10_1
+#define PRLAR5_EL2  S3_4_C6_C10_5
+#define PRLAR6_EL2  S3_4_C6_C11_1
+#define PRLAR7_EL2  S3_4_C6_C11_5
+#define PRLAR8_EL2  S3_4_C6_C12_1
+#define PRLAR9_EL2  S3_4_C6_C12_5
+#define PRLAR10_EL2 S3_4_C6_C13_1
+#define PRLAR11_EL2 S3_4_C6_C13_5
+#define PRLAR12_EL2 S3_4_C6_C14_1
+#define PRLAR13_EL2 S3_4_C6_C14_5
+#define PRLAR14_EL2 S3_4_C6_C15_1
+#define PRLAR15_EL2 S3_4_C6_C15_5
+
+/* MPU Protection Region Enable Register encode */
+#define PRENR_EL1 S3_0_C6_C1_1
+#define PRENR_EL2 S3_4_C6_C1_1
+
+/* MPU Protection Region Selection Register encode */
+#define PRSELR_EL1 S3_0_C6_C2_1
+#define PRSELR_EL2 S3_4_C6_C2_1
+
+/* MPU Type registers encode */
+#define MPUIR_EL1 S3_0_C0_C0_4
+#define MPUIR_EL2 S3_4_C0_C0_4
+
+#endif
+
 /* Access to system registers */
 
 #define WRITE_SYSREG64(v, name) do {                    \
-- 
2.25.1

Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> From: Penny Zheng <penny.zheng@arm.com>
> 
> Like boot-time page table in MMU system, we need a boot-time
> MPU protection region configuration in MPU system so Xen can
> fetch code and data from normal memory.
> 
> This operation need to access Armv8-R MPU system registers, but
> these system registers are not supported in GCC version < 11.
> So we have to encode these Armv8-R MPU system registers in header
> file explicitly.
> 
> As MMU system and MPU system have different functions to create
> the boot MMU/MPU data, this will introduce extra #ifdef in code
> flow, so we introduce a neutral name prepare_early_mappings to
> replace create_page_tables for MMU and MPU.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>

If Penny is the original author, then her signed-off-by should be first.

> ---
>   xen/arch/arm/arm64/Makefile              |  2 +
>   xen/arch/arm/arm64/head.S                | 13 ++--
>   xen/arch/arm/arm64/head_mmu.S            |  4 +-
>   xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
>   xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
>   xen/arch/arm/include/asm/arm64/sysregs.h | 89 ++++++++++++++++++++++++
>   6 files changed, 185 insertions(+), 6 deletions(-)
>   create mode 100644 xen/arch/arm/arm64/head_mpu.S
>   create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
> 
> diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
> index 22da2f54b5..438c9737ad 100644
> --- a/xen/arch/arm/arm64/Makefile
> +++ b/xen/arch/arm/arm64/Makefile
> @@ -10,6 +10,8 @@ obj-y += entry.o
>   obj-y += head.o
>   ifneq ($(CONFIG_HAS_MPU),y)
>   obj-y += head_mmu.o
> +else
> +obj-y += head_mpu.o
>   endif
>   obj-y += insn.o
>   obj-$(CONFIG_LIVEPATCH) += livepatch.o
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index d9a8da9120..6c1a5f74a1 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -79,12 +79,12 @@
>    * ---------------------------
>    *
>    * The requirements are:
> - *   MMU = off, D-cache = off, I-cache = on or off,
> + *   MMU/MPU = off, D-cache = off, I-cache = on or off,
>    *   x0 = physical address to the FDT blob.
>    *
>    * This must be the very first address in the loaded image.
>    * It should be linked at XEN_VIRT_START, and loaded at any
> - * 4K-aligned address.  All of text+data+bss must fit in 2MB,
> + * 4K-aligned address. All of text+data+bss must fit in 2MB,

The double space after the final point was valid. This is fairly common 
to use it and this is a spurious change.


>    * or the initial pagetable code below will need adjustment.
>    */
>   
> @@ -249,7 +249,12 @@ real_start_efi:
>   
>           bl    check_cpu_mode
>           bl    cpu_init
> -        bl    create_page_tables
> +
> +        /*
> +         * Create boot memory management data, pagetable for MMU systems
> +         * and protection regions for MPU systems.
> +         */

head.S is now meant to be generic. So I would prefer if we keep comment 
as generic as possible. IOW, anything after the first comma should be 
dropped.

> +        bl    prepare_early_mappings
>           bl    enable_mmu
>   
>           /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
> @@ -307,7 +312,7 @@ GLOBAL(init_secondary)
>   #endif
>           bl    check_cpu_mode
>           bl    cpu_init
> -        bl    create_page_tables
> +        bl    prepare_early_mappings
>           bl    enable_mmu
>   
>           /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
> diff --git a/xen/arch/arm/arm64/head_mmu.S b/xen/arch/arm/arm64/head_mmu.S
> index 1a3df81a38..fc64819a98 100644
> --- a/xen/arch/arm/arm64/head_mmu.S
> +++ b/xen/arch/arm/arm64/head_mmu.S
> @@ -123,7 +123,7 @@
>    *
>    * Clobbers x0 - x4
>    */
> -ENTRY(create_page_tables)
> +ENTRY(prepare_early_mappings)
>           /* Prepare the page-tables for mapping Xen */
>           ldr   x0, =XEN_VIRT_START
>           create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
> @@ -208,7 +208,7 @@ virtphys_clash:
>           /* Identity map clashes with boot_third, which we cannot handle yet */
>           PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
>           b     fail
> -ENDPROC(create_page_tables)
> +ENDPROC(prepare_early_mappings)
>   
>   /*
>    * Turn on the Data Cache and the MMU. The function will return on the 1:1
> diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
> new file mode 100644
> index 0000000000..f60611b556
> --- /dev/null
> +++ b/xen/arch/arm/arm64/head_mpu.S
> @@ -0,0 +1,70 @@
> +// SPDX-License-Identifier: GPL-2.0-only

Coding style:

/* SPDX ... */

> +/*
> + * Start-of-day code for an Armv8-R MPU system.
> + */
> +
> +#include <asm/arm64/mpu.h>
> +#include <asm/page.h>
> +#include <asm/early_printk.h>

Headers should be included in alphabetical order.

> +
> +/*
> + * From the requirements of head.S we know that Xen image should
> + * be linked at XEN_START_ADDRESS, and all of text + data + bss
> + * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the
> + * address that Xen image should be loaded at. So for initial MPU
> + * regions setup, we use 2MB for Xen data memory to setup boot
> + * region, or the create boot regions code below will need adjustment.
> + */
> +#define XEN_START_MEM_SIZE      0x200000

It sounds like something that should be defined in the header. Also, I 
think the size should be common between MPU and MMU.

In [1], I was going to name it XEN_VIRT_SIZE. I would be OK to remove 
"VIRT" in the name.

> +
> +/*
> + * In boot stage, we will use 1 MPU region:
> + * Region#0: Normal memory for Xen text + data + bss (2MB)
> + */

Are we only going to modify the MPU in head.S? If not, then I would 
define the layout in config_mpu.h so there are a single point where you 
can read how this works.

> +#define BOOT_NORMAL_REGION_IDX  0x0
> +
> +/* MPU normal memory attributes. */
> +#define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */

IIUC, this means that Xen will be mapped write-executable. Is this going 
to be forever? If not, when can't we already mapped Xen properly?

> +#define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */

To me, it feels like this should be fined outside of head.S because this 
could be re-used by other part of Xen.

> +
> +.macro write_pr, sel, prbar, prlar
> +    msr   PRSELR_EL2, \sel
> +    dsb   sy

Is it really necessary to use "sy"  here? Also, it would be good to 
explain the logic. I.e. why do you need two dsb but only one isb?

In fact, I was expecting an "isb" here than "dsb" to wait for the 
completion of the instruction.

> +    msr   PRBAR_EL2, \prbar
> +    msr   PRLAR_EL2, \prlar
> +    dsb   sy
> +    isb
> +.endm
> +
> +.section .text.header, "ax", %progbits
> +
> +/*
> + * Static start-of-day EL2 MPU memory layout.
> + *
> + * It has a very simple structure, including:
> + *  - 2MB normal memory mappings of xen at XEN_START_ADDRESS, which
> + * is the address where Xen was loaded by the bootloader.

Missing details on the clobberred registers.

> + */
> +ENTRY(prepare_early_mappings)
> +    /* Map Xen start memory to a normal memory region. */
> +    mov x0, #BOOT_NORMAL_REGION_IDX
> +    ldr x1, =XEN_START_ADDRESS
> +    and x1, x1, #MPU_REGION_MASK
> +    mov x3, #PRBAR_NORMAL_MEM
> +    orr x1, x1, x3

It looks like to me there are a potential for a macro to compute the 
register.

> +
> +    ldr x2, =XEN_START_ADDRESS
> +    mov x3, #(XEN_START_MEM_SIZE - 1)

XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be smaller 
and you will map memory that may not be part of Xen. Therefore, you 
likely want to compute the real size to avoid mapping too much.


> +    add x2, x2, x3
> +    and x2, x2, #MPU_REGION_MASK
> +    mov x3, #PRLAR_NORMAL_MEM
> +    orr x2, x2, x3
> +
> +    /*
> +     * Write to MPU protection region:
> +     * x0 for pr_sel, x1 for prbar x2 for prlar

This is not a very useful comment because this can be inferred from the 
prototype of write_pr. What would be more interesting is to explain the 
logic within this function in the same way we do in head.S and head_mmu.S.

> +     */
> +    write_pr x0, x1, x2
> +
> +    ret
> +ENDPROC(prepare_early_mappings)
> diff --git a/xen/arch/arm/include/asm/arm64/mpu.h b/xen/arch/arm/include/asm/arm64/mpu.h
> new file mode 100644
> index 0000000000..d209eef6db
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/arm64/mpu.h
> @@ -0,0 +1,13 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * mpu.h: Arm Memory Protection Unit definitions.
> + */
> +
> +#ifndef __ARM64_MPU_H__
> +#define __ARM64_MPU_H__
> +
> +#define MPU_REGION_SHIFT  6
> +#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
> +#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
> +
> +#endif /* __ARM64_MPU_H__ */
> diff --git a/xen/arch/arm/include/asm/arm64/sysregs.h b/xen/arch/arm/include/asm/arm64/sysregs.h
> index 54670084c3..a596042d6c 100644
> --- a/xen/arch/arm/include/asm/arm64/sysregs.h
> +++ b/xen/arch/arm/include/asm/arm64/sysregs.h

In the context of this patch, it would be better to only define the 
registers you need. If you want to define all of them, then please 
define them in a separate patch before this one.

> @@ -458,6 +458,95 @@
>   #define ZCR_ELx_LEN_SIZE             9
>   #define ZCR_ELx_LEN_MASK             0x1ff
>   
> +/* System registers for AArch64 with PMSA */
> +#ifdef CONFIG_HAS_MPU

The #ifdef here seems unnecessary.

> +
> +/* EL1 MPU Protection Region Base Address Register encode */
> +#define PRBAR_EL1   S3_0_C6_C8_0
> +#define PRBAR1_EL1  S3_0_C6_C8_4
> +#define PRBAR2_EL1  S3_0_C6_C9_0
> +#define PRBAR3_EL1  S3_0_C6_C9_4
> +#define PRBAR4_EL1  S3_0_C6_C10_0
> +#define PRBAR5_EL1  S3_0_C6_C10_4
> +#define PRBAR6_EL1  S3_0_C6_C11_0
> +#define PRBAR7_EL1  S3_0_C6_C11_4
> +#define PRBAR8_EL1  S3_0_C6_C12_0
> +#define PRBAR9_EL1  S3_0_C6_C12_4
> +#define PRBAR10_EL1 S3_0_C6_C13_0
> +#define PRBAR11_EL1 S3_0_C6_C13_4
> +#define PRBAR12_EL1 S3_0_C6_C14_0
> +#define PRBAR13_EL1 S3_0_C6_C14_4
> +#define PRBAR14_EL1 S3_0_C6_C15_0
> +#define PRBAR15_EL1 S3_0_C6_C15_4
> +
> +/* EL1 MPU Protection Region Limit Address Register encode */
> +#define PRLAR_EL1   S3_0_C6_C8_1
> +#define PRLAR1_EL1  S3_0_C6_C8_5
> +#define PRLAR2_EL1  S3_0_C6_C9_1
> +#define PRLAR3_EL1  S3_0_C6_C9_5
> +#define PRLAR4_EL1  S3_0_C6_C10_1
> +#define PRLAR5_EL1  S3_0_C6_C10_5
> +#define PRLAR6_EL1  S3_0_C6_C11_1
> +#define PRLAR7_EL1  S3_0_C6_C11_5
> +#define PRLAR8_EL1  S3_0_C6_C12_1
> +#define PRLAR9_EL1  S3_0_C6_C12_5
> +#define PRLAR10_EL1 S3_0_C6_C13_1
> +#define PRLAR11_EL1 S3_0_C6_C13_5
> +#define PRLAR12_EL1 S3_0_C6_C14_1
> +#define PRLAR13_EL1 S3_0_C6_C14_5
> +#define PRLAR14_EL1 S3_0_C6_C15_1
> +#define PRLAR15_EL1 S3_0_C6_C15_5
> +
> +/* EL2 MPU Protection Region Base Address Register encode */
> +#define PRBAR_EL2   S3_4_C6_C8_0
> +#define PRBAR1_EL2  S3_4_C6_C8_4
> +#define PRBAR2_EL2  S3_4_C6_C9_0
> +#define PRBAR3_EL2  S3_4_C6_C9_4
> +#define PRBAR4_EL2  S3_4_C6_C10_0
> +#define PRBAR5_EL2  S3_4_C6_C10_4
> +#define PRBAR6_EL2  S3_4_C6_C11_0
> +#define PRBAR7_EL2  S3_4_C6_C11_4
> +#define PRBAR8_EL2  S3_4_C6_C12_0
> +#define PRBAR9_EL2  S3_4_C6_C12_4
> +#define PRBAR10_EL2 S3_4_C6_C13_0
> +#define PRBAR11_EL2 S3_4_C6_C13_4
> +#define PRBAR12_EL2 S3_4_C6_C14_0
> +#define PRBAR13_EL2 S3_4_C6_C14_4
> +#define PRBAR14_EL2 S3_4_C6_C15_0
> +#define PRBAR15_EL2 S3_4_C6_C15_4
> +
> +/* EL2 MPU Protection Region Limit Address Register encode */
> +#define PRLAR_EL2   S3_4_C6_C8_1
> +#define PRLAR1_EL2  S3_4_C6_C8_5
> +#define PRLAR2_EL2  S3_4_C6_C9_1
> +#define PRLAR3_EL2  S3_4_C6_C9_5
> +#define PRLAR4_EL2  S3_4_C6_C10_1
> +#define PRLAR5_EL2  S3_4_C6_C10_5
> +#define PRLAR6_EL2  S3_4_C6_C11_1
> +#define PRLAR7_EL2  S3_4_C6_C11_5
> +#define PRLAR8_EL2  S3_4_C6_C12_1
> +#define PRLAR9_EL2  S3_4_C6_C12_5
> +#define PRLAR10_EL2 S3_4_C6_C13_1
> +#define PRLAR11_EL2 S3_4_C6_C13_5
> +#define PRLAR12_EL2 S3_4_C6_C14_1
> +#define PRLAR13_EL2 S3_4_C6_C14_5
> +#define PRLAR14_EL2 S3_4_C6_C15_1
> +#define PRLAR15_EL2 S3_4_C6_C15_5
> +
> +/* MPU Protection Region Enable Register encode */
> +#define PRENR_EL1 S3_0_C6_C1_1
> +#define PRENR_EL2 S3_4_C6_C1_1
> +
> +/* MPU Protection Region Selection Register encode */
> +#define PRSELR_EL1 S3_0_C6_C2_1
> +#define PRSELR_EL2 S3_4_C6_C2_1
> +
> +/* MPU Type registers encode */
> +#define MPUIR_EL1 S3_0_C0_C0_4
> +#define MPUIR_EL2 S3_4_C0_C0_4
> +
> +#endif
> +
>   /* Access to system registers */
>   
>   #define WRITE_SYSREG64(v, name) do {                    \

Cheers,

[1] https://lore.kernel.org/all/20221022150422.17707-2-julien@xen.org/

-- 
Julien Grall

RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Monday, November 7, 2022 4:47 AM
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection
> regions
> 
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > From: Penny Zheng <penny.zheng@arm.com>
> >
> > Like boot-time page table in MMU system, we need a boot-time MPU
> > protection region configuration in MPU system so Xen can fetch code
> > and data from normal memory.
> >
> > This operation need to access Armv8-R MPU system registers, but these
> > system registers are not supported in GCC version < 11.
> > So we have to encode these Armv8-R MPU system registers in header file
> > explicitly.
> >
> > As MMU system and MPU system have different functions to create the
> > boot MMU/MPU data, this will introduce extra #ifdef in code flow, so
> > we introduce a neutral name prepare_early_mappings to replace
> > create_page_tables for MMU and MPU.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> 
> If Penny is the original author, then her signed-off-by should be first.
> 
> > ---
> >   xen/arch/arm/arm64/Makefile              |  2 +
> >   xen/arch/arm/arm64/head.S                | 13 ++--
> >   xen/arch/arm/arm64/head_mmu.S            |  4 +-
> >   xen/arch/arm/arm64/head_mpu.S            | 70 +++++++++++++++++++
> >   xen/arch/arm/include/asm/arm64/mpu.h     | 13 ++++
> >   xen/arch/arm/include/asm/arm64/sysregs.h | 89
> ++++++++++++++++++++++++
> >   6 files changed, 185 insertions(+), 6 deletions(-)
> >   create mode 100644 xen/arch/arm/arm64/head_mpu.S
> >   create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
> >
> > diff --git a/xen/arch/arm/arm64/Makefile
> b/xen/arch/arm/arm64/Makefile
> > index 22da2f54b5..438c9737ad 100644
> > --- a/xen/arch/arm/arm64/Makefile
> > +++ b/xen/arch/arm/arm64/Makefile
> > @@ -10,6 +10,8 @@ obj-y += entry.o
> >   obj-y += head.o
> >   ifneq ($(CONFIG_HAS_MPU),y)
> >   obj-y += head_mmu.o
> > +else
> > +obj-y += head_mpu.o
> >   endif
> >   obj-y += insn.o
> >   obj-$(CONFIG_LIVEPATCH) += livepatch.o diff --git
> > a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index
> > d9a8da9120..6c1a5f74a1 100644
> > --- a/xen/arch/arm/arm64/head.S
> > +++ b/xen/arch/arm/arm64/head.S
> > @@ -79,12 +79,12 @@
> >    * ---------------------------
> >    *
> >    * The requirements are:
> > - *   MMU = off, D-cache = off, I-cache = on or off,
> > + *   MMU/MPU = off, D-cache = off, I-cache = on or off,
> >    *   x0 = physical address to the FDT blob.
> >    *
> >    * This must be the very first address in the loaded image.
> >    * It should be linked at XEN_VIRT_START, and loaded at any
> > - * 4K-aligned address.  All of text+data+bss must fit in 2MB,
> > + * 4K-aligned address. All of text+data+bss must fit in 2MB,
> 
> The double space after the final point was valid. This is fairly common to use
> it and this is a spurious change.
> 
> 
> >    * or the initial pagetable code below will need adjustment.
> >    */
> >
> > @@ -249,7 +249,12 @@ real_start_efi:
> >
> >           bl    check_cpu_mode
> >           bl    cpu_init
> > -        bl    create_page_tables
> > +
> > +        /*
> > +         * Create boot memory management data, pagetable for MMU
> systems
> > +         * and protection regions for MPU systems.
> > +         */
> 
> head.S is now meant to be generic. So I would prefer if we keep comment
> as generic as possible. IOW, anything after the first comma should be
> dropped.
> 
> > +        bl    prepare_early_mappings
> >           bl    enable_mmu
> >
> >           /* We are still in the 1:1 mapping. Jump to the runtime Virtual
> Address. */
> > @@ -307,7 +312,7 @@ GLOBAL(init_secondary)
> >   #endif
> >           bl    check_cpu_mode
> >           bl    cpu_init
> > -        bl    create_page_tables
> > +        bl    prepare_early_mappings
> >           bl    enable_mmu
> >
> >           /* We are still in the 1:1 mapping. Jump to the runtime Virtual
> Address. */
> > diff --git a/xen/arch/arm/arm64/head_mmu.S
> b/xen/arch/arm/arm64/head_mmu.S
> > index 1a3df81a38..fc64819a98 100644
> > --- a/xen/arch/arm/arm64/head_mmu.S
> > +++ b/xen/arch/arm/arm64/head_mmu.S
> > @@ -123,7 +123,7 @@
> >    *
> >    * Clobbers x0 - x4
> >    */
> > -ENTRY(create_page_tables)
> > +ENTRY(prepare_early_mappings)
> >           /* Prepare the page-tables for mapping Xen */
> >           ldr   x0, =XEN_VIRT_START
> >           create_table_entry boot_pgtable, boot_first, x0, 0, x1, x2, x3
> > @@ -208,7 +208,7 @@ virtphys_clash:
> >           /* Identity map clashes with boot_third, which we cannot handle yet
> */
> >           PRINT("- Unable to build boot page tables - virt and phys addresses
> clash. -\r\n")
> >           b     fail
> > -ENDPROC(create_page_tables)
> > +ENDPROC(prepare_early_mappings)
> >
> >   /*
> >    * Turn on the Data Cache and the MMU. The function will return on the
> 1:1
> > diff --git a/xen/arch/arm/arm64/head_mpu.S
> b/xen/arch/arm/arm64/head_mpu.S
> > new file mode 100644
> > index 0000000000..f60611b556
> > --- /dev/null
> > +++ b/xen/arch/arm/arm64/head_mpu.S
> > @@ -0,0 +1,70 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> 
> Coding style:
> 
> /* SPDX ... */
> 
> > +/*
> > + * Start-of-day code for an Armv8-R MPU system.
> > + */
> > +
> > +#include <asm/arm64/mpu.h>
> > +#include <asm/page.h>
> > +#include <asm/early_printk.h>
> 
> Headers should be included in alphabetical order.
> 
> > +
> > +/*
> > + * From the requirements of head.S we know that Xen image should
> > + * be linked at XEN_START_ADDRESS, and all of text + data + bss
> > + * must fit in 2MB. On MPU systems, XEN_START_ADDRESS is also the
> > + * address that Xen image should be loaded at. So for initial MPU
> > + * regions setup, we use 2MB for Xen data memory to setup boot
> > + * region, or the create boot regions code below will need adjustment.
> > + */
> > +#define XEN_START_MEM_SIZE      0x200000
> 
> It sounds like something that should be defined in the header. Also, I
> think the size should be common between MPU and MMU.
> 
> In [1], I was going to name it XEN_VIRT_SIZE. I would be OK to remove
> "VIRT" in the name.
> 

Thx and please, then I will replace it with XEN_SIZE

> > +
> > +/*
> > + * In boot stage, we will use 1 MPU region:
> > + * Region#0: Normal memory for Xen text + data + bss (2MB)
> > + */
> 
> Are we only going to modify the MPU in head.S? If not, then I would
> define the layout in config_mpu.h so there are a single point where you
> can read how this works.
> 

We will remap Xen in C codes in setup_mm().
The whole strategy is aligned with MMU: a very simple setup(map xen
with the maximum size, 2M) at start-of-the-day, and a fit map in
setup_mm.

All the following variables will be only used at head_mpu.S.
It is not very generic, later, when introducing MPU memory region
management in C codes, we will define different macros for
various memory attributes.

> > +#define BOOT_NORMAL_REGION_IDX  0x0
> > +
> > +/* MPU normal memory attributes. */
> > +#define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
> 
> IIUC, this means that Xen will be mapped write-executable. Is this going
> to be forever? If not, when can't we already mapped Xen properly?
> 

No, this setup is the start-of-day setup, and will only last for a very short time.

To be aligned with MMU system, in which L3 memory attributes is as follows:
#define PT_MEM_L3 0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
Xen shall be mapped read-only-executable here, and I will fix it.

> > +#define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
> 
> To me, it feels like this should be fined outside of head.S because this
> could be re-used by other part of Xen.
> 

It is like PT_MEM_L3 in head_mmu.S. It will be only used in compiler codes.

MMU is defining macro like PAGE_HYPERVISOR_RW, for memory attributes
management in C codes, and we intend to follow the same strategy.

> > +
> > +.macro write_pr, sel, prbar, prlar
> > +    msr   PRSELR_EL2, \sel
> > +    dsb   sy
> 
> Is it really necessary to use "sy"  here? Also, it would be good to
> explain the logic. I.e. why do you need two dsb but only one isb?
> 
> In fact, I was expecting an "isb" here than "dsb" to wait for the
> completion of the instruction.
> 
> > +    msr   PRBAR_EL2, \prbar
> > +    msr   PRLAR_EL2, \prlar
> > +    dsb   sy
> > +    isb
> > +.endm
> > +
> > +.section .text.header, "ax", %progbits
> > +
> > +/*
> > + * Static start-of-day EL2 MPU memory layout.
> > + *
> > + * It has a very simple structure, including:
> > + *  - 2MB normal memory mappings of xen at XEN_START_ADDRESS,
> which
> > + * is the address where Xen was loaded by the bootloader.
> 
> Missing details on the clobberred registers.
> 
> > + */
> > +ENTRY(prepare_early_mappings)
> > +    /* Map Xen start memory to a normal memory region. */
> > +    mov x0, #BOOT_NORMAL_REGION_IDX
> > +    ldr x1, =XEN_START_ADDRESS
> > +    and x1, x1, #MPU_REGION_MASK
> > +    mov x3, #PRBAR_NORMAL_MEM
> > +    orr x1, x1, x3
> 
> It looks like to me there are a potential for a macro to compute the
> register.
> 
> > +
> > +    ldr x2, =XEN_START_ADDRESS
> > +    mov x3, #(XEN_START_MEM_SIZE - 1)
> 
> XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be
> smaller
> and you will map memory that may not be part of Xen. Therefore, you
> likely want to compute the real size to avoid mapping too much.
> 

Later, in setup_mm we will map XEN components by components, such as,
one MPU memory region for read-only-executable text section, one
MPU memory region for read-only data section, etc, etc.
So in there, XEN will be mapped fitly.

IMHO, the mapping in compiler with maximum size of Xen is also what
MMU does.

>
> > +    add x2, x2, x
> > +    and x2, x2, #MPU_REGION_MASK
> > +    mov x3, #PRLAR_NORMAL_MEM
> > +    orr x2, x2, x3
> > +
> > +    /*
> > +     * Write to MPU protection region:
> > +     * x0 for pr_sel, x1 for prbar x2 for prlar
> 
> This is not a very useful comment because this can be inferred from the
> prototype of write_pr. What would be more interesting is to explain the
> logic within this function in the same way we do in head.S and head_mmu.S.
> 
> > +     */
> > +    write_pr x0, x1, x2
> > +
> > +    ret
> > +ENDPROC(prepare_early_mappings)
> > diff --git a/xen/arch/arm/include/asm/arm64/mpu.h
> b/xen/arch/arm/include/asm/arm64/mpu.h
> > new file mode 100644
> > index 0000000000..d209eef6db
> > --- /dev/null
> > +++ b/xen/arch/arm/include/asm/arm64/mpu.h
> > @@ -0,0 +1,13 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * mpu.h: Arm Memory Protection Unit definitions.
> > + */
> > +
> > +#ifndef __ARM64_MPU_H__
> > +#define __ARM64_MPU_H__
> > +
> > +#define MPU_REGION_SHIFT  6
> > +#define MPU_REGION_ALIGN  (_AC(1, UL) << MPU_REGION_SHIFT)
> > +#define MPU_REGION_MASK   (~(MPU_REGION_ALIGN - 1))
> > +
> > +#endif /* __ARM64_MPU_H__ */
> > diff --git a/xen/arch/arm/include/asm/arm64/sysregs.h
> b/xen/arch/arm/include/asm/arm64/sysregs.h
> > index 54670084c3..a596042d6c 100644
> > --- a/xen/arch/arm/include/asm/arm64/sysregs.h
> > +++ b/xen/arch/arm/include/asm/arm64/sysregs.h
> 
> In the context of this patch, it would be better to only define the
> registers you need. If you want to define all of them, then please
> define them in a separate patch before this one.
> 
> > @@ -458,6 +458,95 @@
> >   #define ZCR_ELx_LEN_SIZE             9
> >   #define ZCR_ELx_LEN_MASK             0x1ff
> >
> > +/* System registers for AArch64 with PMSA */
> > +#ifdef CONFIG_HAS_MPU
> 
> The #ifdef here seems unnecessary.
> 
> > +
> > +/* EL1 MPU Protection Region Base Address Register encode */
> > +#define PRBAR_EL1   S3_0_C6_C8_0
> > +#define PRBAR1_EL1  S3_0_C6_C8_4
> > +#define PRBAR2_EL1  S3_0_C6_C9_0
> > +#define PRBAR3_EL1  S3_0_C6_C9_4
> > +#define PRBAR4_EL1  S3_0_C6_C10_0
> > +#define PRBAR5_EL1  S3_0_C6_C10_4
> > +#define PRBAR6_EL1  S3_0_C6_C11_0
> > +#define PRBAR7_EL1  S3_0_C6_C11_4
> > +#define PRBAR8_EL1  S3_0_C6_C12_0
> > +#define PRBAR9_EL1  S3_0_C6_C12_4
> > +#define PRBAR10_EL1 S3_0_C6_C13_0
> > +#define PRBAR11_EL1 S3_0_C6_C13_4
> > +#define PRBAR12_EL1 S3_0_C6_C14_0
> > +#define PRBAR13_EL1 S3_0_C6_C14_4
> > +#define PRBAR14_EL1 S3_0_C6_C15_0
> > +#define PRBAR15_EL1 S3_0_C6_C15_4
> > +
> > +/* EL1 MPU Protection Region Limit Address Register encode */
> > +#define PRLAR_EL1   S3_0_C6_C8_1
> > +#define PRLAR1_EL1  S3_0_C6_C8_5
> > +#define PRLAR2_EL1  S3_0_C6_C9_1
> > +#define PRLAR3_EL1  S3_0_C6_C9_5
> > +#define PRLAR4_EL1  S3_0_C6_C10_1
> > +#define PRLAR5_EL1  S3_0_C6_C10_5
> > +#define PRLAR6_EL1  S3_0_C6_C11_1
> > +#define PRLAR7_EL1  S3_0_C6_C11_5
> > +#define PRLAR8_EL1  S3_0_C6_C12_1
> > +#define PRLAR9_EL1  S3_0_C6_C12_5
> > +#define PRLAR10_EL1 S3_0_C6_C13_1
> > +#define PRLAR11_EL1 S3_0_C6_C13_5
> > +#define PRLAR12_EL1 S3_0_C6_C14_1
> > +#define PRLAR13_EL1 S3_0_C6_C14_5
> > +#define PRLAR14_EL1 S3_0_C6_C15_1
> > +#define PRLAR15_EL1 S3_0_C6_C15_5
> > +
> > +/* EL2 MPU Protection Region Base Address Register encode */
> > +#define PRBAR_EL2   S3_4_C6_C8_0
> > +#define PRBAR1_EL2  S3_4_C6_C8_4
> > +#define PRBAR2_EL2  S3_4_C6_C9_0
> > +#define PRBAR3_EL2  S3_4_C6_C9_4
> > +#define PRBAR4_EL2  S3_4_C6_C10_0
> > +#define PRBAR5_EL2  S3_4_C6_C10_4
> > +#define PRBAR6_EL2  S3_4_C6_C11_0
> > +#define PRBAR7_EL2  S3_4_C6_C11_4
> > +#define PRBAR8_EL2  S3_4_C6_C12_0
> > +#define PRBAR9_EL2  S3_4_C6_C12_4
> > +#define PRBAR10_EL2 S3_4_C6_C13_0
> > +#define PRBAR11_EL2 S3_4_C6_C13_4
> > +#define PRBAR12_EL2 S3_4_C6_C14_0
> > +#define PRBAR13_EL2 S3_4_C6_C14_4
> > +#define PRBAR14_EL2 S3_4_C6_C15_0
> > +#define PRBAR15_EL2 S3_4_C6_C15_4
> > +
> > +/* EL2 MPU Protection Region Limit Address Register encode */
> > +#define PRLAR_EL2   S3_4_C6_C8_1
> > +#define PRLAR1_EL2  S3_4_C6_C8_5
> > +#define PRLAR2_EL2  S3_4_C6_C9_1
> > +#define PRLAR3_EL2  S3_4_C6_C9_5
> > +#define PRLAR4_EL2  S3_4_C6_C10_1
> > +#define PRLAR5_EL2  S3_4_C6_C10_5
> > +#define PRLAR6_EL2  S3_4_C6_C11_1
> > +#define PRLAR7_EL2  S3_4_C6_C11_5
> > +#define PRLAR8_EL2  S3_4_C6_C12_1
> > +#define PRLAR9_EL2  S3_4_C6_C12_5
> > +#define PRLAR10_EL2 S3_4_C6_C13_1
> > +#define PRLAR11_EL2 S3_4_C6_C13_5
> > +#define PRLAR12_EL2 S3_4_C6_C14_1
> > +#define PRLAR13_EL2 S3_4_C6_C14_5
> > +#define PRLAR14_EL2 S3_4_C6_C15_1
> > +#define PRLAR15_EL2 S3_4_C6_C15_5
> > +
> > +/* MPU Protection Region Enable Register encode */
> > +#define PRENR_EL1 S3_0_C6_C1_1
> > +#define PRENR_EL2 S3_4_C6_C1_1
> > +
> > +/* MPU Protection Region Selection Register encode */
> > +#define PRSELR_EL1 S3_0_C6_C2_1
> > +#define PRSELR_EL2 S3_4_C6_C2_1
> > +
> > +/* MPU Type registers encode */
> > +#define MPUIR_EL1 S3_0_C0_C0_4
> > +#define MPUIR_EL2 S3_4_C0_C0_4
> > +
> > +#endif
> > +
> >   /* Access to system registers */
> >
> >   #define WRITE_SYSREG64(v, name) do {                    \
> 
> Cheers,
> 
> [1] https://lore.kernel.org/all/20221022150422.17707-2-julien@xen.org/
> 
> --
> Julien Grall

Cheers,

--
Penny Zheng

Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

[Julien Grall]

On 07/11/2022 06:59, Penny Zheng wrote:
> Hi Julien

Hi Penny,

>>> +
>>> +/*
>>> + * In boot stage, we will use 1 MPU region:
>>> + * Region#0: Normal memory for Xen text + data + bss (2MB)
>>> + */
>>
>> Are we only going to modify the MPU in head.S? If not, then I would
>> define the layout in config_mpu.h so there are a single point where you
>> can read how this works.
>>
> 
> We will remap Xen in C codes in setup_mm().
> The whole strategy is aligned with MMU: a very simple setup(map xen
> with the maximum size, 2M) at start-of-the-day, and a fit map in
> setup_mm.

The strategy we are using for the MMU is completely broken (not 
compliant with the Arm Arm) and unnecessary. My long term goal is to 
actually remove the switch_ttbr() and most of setup_pagetables() for all 
setup but cache coloring. This means the concept of boot pages will not 
exist anymore.

For the MPU, we should aim to do better than what was done for the MMU. 
Right now, I see no argument for switching MPUs table. I am only seen 
argument against it because you need to disable the cache and is quite 
fragile.

[...]

>>> +
>>> +    ldr x2, =XEN_START_ADDRESS
>>> +    mov x3, #(XEN_START_MEM_SIZE - 1)
>>
>> XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be
>> smaller
>> and you will map memory that may not be part of Xen. Therefore, you
>> likely want to compute the real size to avoid mapping too much.
>>
> 
> Later, in setup_mm we will map XEN components by components, such as,
> one MPU memory region for read-only-executable text section, one
> MPU memory region for read-only data section, etc, etc.
> So in there, XEN will be mapped fitly.

But what prevents you to do this now?

> 
> IMHO, the mapping in compiler with maximum size of Xen is also what
> MMU does.

Which is broken because we don't know what located after Xen binary. 
This could be reserved RAM, device which may requires non-caching 
attribute. Mapping those regions with caching attributes is going to break.

As I hinted above, there are a very long list of issues in the MMU boot 
code. So don't take that code for granted. Your MPU code should first be 
compliant with the Arm Arm. If it is close to the MMU code then that's a 
bonus. But bear in mind that the code may look very different soon (see 
[1]).

Cheers,

[1] 20221022150422.17707-1-julien@xen.org

-- 
Julien Grall

RE: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection regions

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Monday, November 7, 2022 5:30 PM
> To: Penny Zheng <Penny.Zheng@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 09/11] xen/arm64: create boot-time MPU protection
> regions
> 
> On 07/11/2022 06:59, Penny Zheng wrote:
> > Hi Julien
> 
> Hi Penny,
> 
> >>> +
> >>> +/*
> >>> + * In boot stage, we will use 1 MPU region:
> >>> + * Region#0: Normal memory for Xen text + data + bss (2MB)  */
> >>
> >> Are we only going to modify the MPU in head.S? If not, then I would
> >> define the layout in config_mpu.h so there are a single point where
> >> you can read how this works.
> >>
> >
> > We will remap Xen in C codes in setup_mm().
> > The whole strategy is aligned with MMU: a very simple setup(map xen
> > with the maximum size, 2M) at start-of-the-day, and a fit map in
> > setup_mm.
> 
> The strategy we are using for the MMU is completely broken (not compliant
> with the Arm Arm) and unnecessary. My long term goal is to actually remove
> the switch_ttbr() and most of setup_pagetables() for all setup but cache
> coloring. This means the concept of boot pages will not exist anymore.
> 
> For the MPU, we should aim to do better than what was done for the MMU.
> Right now, I see no argument for switching MPUs table. I am only seen
> argument against it because you need to disable the cache and is quite fragile.
> 

> [...]
> 
> >>> +
> >>> +    ldr x2, =XEN_START_ADDRESS
> >>> +    mov x3, #(XEN_START_MEM_SIZE - 1)
> >>
> >> XEN_START_MEM_SIZE is the maximum size of Xen. IOW, Xen may be
> >> smaller and you will map memory that may not be part of Xen.
> >> Therefore, you likely want to compute the real size to avoid mapping
> >> too much.
> >>
> >
> > Later, in setup_mm we will map XEN components by components, such as,
> > one MPU memory region for read-only-executable text section, one MPU
> > memory region for read-only data section, etc, etc.
> > So in there, XEN will be mapped fitly.
> 
> But what prevents you to do this now?
> 
> >
> > IMHO, the mapping in compiler with maximum size of Xen is also what
> > MMU does.
> 
> Which is broken because we don't know what located after Xen binary.
> This could be reserved RAM, device which may requires non-caching
> attribute. Mapping those regions with caching attributes is going to break.
> 

Understood!
Then I will map Xen components by components in the compile time,
then later the first MPU/Cache disabled will be eliminated, I guess.

> 
> Cheers,
> 
> [1] 20221022150422.17707-1-julien@xen.org
> 
> --
> Julien Grall

[PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

[Wei Chen]

From: Penny Zheng <penny.zheng@arm.com>

We need some helpers for Xen to enable/disable MPU in boot-time
and runtime. For MPU enable helper, we know that it's an
essential requirement of MPU system. But for MPU disable,
we need to use it for some special situations. For example,
in the progress of tranferring from boot-time to runtime,
we need to update the MPU protection regions configuration,
but we can't modify an MPU protection region if there is some
data accessed by Xen. But in boot-time all of Xen text, data
and BSS are in one MPU protection region, if Xen want to update
this protection region, above restriction will be triggered.
So in this situation, we need to disable the whole MPU to update
the protection regions.

In these helper, enable/disable MPU will also enable/disable
the D-cache. There are two reasons for it:
1. Make the function semantic be consistent with enable_mmu.
   For MMU systems, enable_mmu will turn MMU and D-Cache at
   the same time.
2. When MPU is disabled, the MPU background attributes will
   be used. On some platforms, the background will treat all
   memory as device memory. The access to device memory will
   bypass the cache, even if the C bit is enabled in SCTLR.
   To avoid this implicit behavior, we disable cache with MPU
   explicitly to tell user that when MPU is disabled, the
   memory access is uncacheable.

In this patch, we also introduce a neutral name enable_mm for
Xen to enable MMU/MPU. This can help us to keep one code flow
in head.S

Signed-off-by: Wei Chen <wei.chen@arm.com>
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 xen/arch/arm/arm64/head.S     |  5 +++--
 xen/arch/arm/arm64/head_mmu.S |  4 ++--
 xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 6c1a5f74a1..228f01db69 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -255,7 +255,8 @@ real_start_efi:
          * and protection regions for MPU systems.
          */
         bl    prepare_early_mappings
-        bl    enable_mmu
+        /* Turn on MMU or MPU */
+        bl    enable_mm
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
         ldr   x0, =primary_switched
@@ -313,7 +314,7 @@ GLOBAL(init_secondary)
         bl    check_cpu_mode
         bl    cpu_init
         bl    prepare_early_mappings
-        bl    enable_mmu
+        bl    enable_mm
 
         /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
         ldr   x0, =secondary_switched
diff --git a/xen/arch/arm/arm64/head_mmu.S b/xen/arch/arm/arm64/head_mmu.S
index fc64819a98..b542755bd2 100644
--- a/xen/arch/arm/arm64/head_mmu.S
+++ b/xen/arch/arm/arm64/head_mmu.S
@@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
  *
  * Clobbers x0 - x3
  */
-ENTRY(enable_mmu)
+ENTRY(enable_mm)
         PRINT("- Turning on paging -\r\n")
 
         /*
@@ -239,7 +239,7 @@ ENTRY(enable_mmu)
         msr   SCTLR_EL2, x0          /* now paging is enabled */
         isb                          /* Now, flush the icache */
         ret
-ENDPROC(enable_mmu)
+ENDPROC(enable_mm)
 
 /*
  * Remove the 1:1 map from the page-tables. It is not easy to keep track
diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
index f60611b556..5a1b03e293 100644
--- a/xen/arch/arm/arm64/head_mpu.S
+++ b/xen/arch/arm/arm64/head_mpu.S
@@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
 
     ret
 ENDPROC(prepare_early_mappings)
+
+/*
+ * Enable EL2 MPU and data cache. Because we will disable cache
+ * with MPU at the same time, in accordance with that, we have
+ * to enable cache with MPU at the same time in this function.
+ * When MPU is disabled, the MPU background attributes will
+ * be used. On some platform, the background will treat all
+ * memory as IO memory. The access to IO memory will bypass
+ * the cache, even you have enabled the C bit in SCTLR.
+ * To avoid this implicit behavior, we disable cache with MPU
+ * explicitly to tell user that when MPU is disabled, the memory
+ * access is uncacheable.
+ */
+ENTRY(enable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
+    /* Enable EL2 MPU and D-cache */
+    orr   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(enable_mm)
+
+/* Disable MPU system, including data cache. */
+ENTRY(disable_mm)
+    mrs   x0, SCTLR_EL2
+    mov   x1, #~(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
+    /* Disable EL2 MPU and D-cache */
+    and   x0, x0, x1
+    dsb   sy
+    msr   SCTLR_EL2, x0
+    isb
+    ret
+ENDPROC(disable_mm)
-- 
2.25.1

Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> From: Penny Zheng <penny.zheng@arm.com>
> 
> We need some helpers for Xen to enable/disable MPU in boot-time
> and runtime. For MPU enable helper, we know that it's an
> essential requirement of MPU system. But for MPU disable,
> we need to use it for some special situations. For example,
> in the progress of tranferring from boot-time to runtime,
> we need to update the MPU protection regions configuration,
> but we can't modify an MPU protection region if there is some
> data accessed by Xen. But in boot-time all of Xen text, data
> and BSS are in one MPU protection region, if Xen want to update
> this protection region, above restriction will be triggered.

This raises the following question: Why can't we create the split 
regions right now?

In particular, disabling the MMU/Cache is fairly risky because you need 
to ensure that anything in the cache you care about have been written 
back to the RAM).

> So in this situation, we need to disable the whole MPU to update
> the protection regions.
> 
> In these helper, enable/disable MPU will also enable/disable
> the D-cache. There are two reasons for it:
> 1. Make the function semantic be consistent with enable_mmu.
>     For MMU systems, enable_mmu will turn MMU and D-Cache at
>     the same time.
> 2. When MPU is disabled, the MPU background attributes will
>     be used. On some platforms, the background will treat all
>     memory as device memory. The access to device memory will
>     bypass the cache, even if the C bit is enabled in SCTLR.
>     To avoid this implicit behavior, we disable cache with MPU
>     explicitly to tell user that when MPU is disabled, the
>     memory access is uncacheable.
> 
> In this patch, we also introduce a neutral name enable_mm for
> Xen to enable MMU/MPU. This can help us to keep one code flow
> in head.S
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> ---
>   xen/arch/arm/arm64/head.S     |  5 +++--
>   xen/arch/arm/arm64/head_mmu.S |  4 ++--
>   xen/arch/arm/arm64/head_mpu.S | 35 +++++++++++++++++++++++++++++++++++
>   3 files changed, 40 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index 6c1a5f74a1..228f01db69 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -255,7 +255,8 @@ real_start_efi:
>            * and protection regions for MPU systems.
>            */
>           bl    prepare_early_mappings
> -        bl    enable_mmu
> +        /* Turn on MMU or MPU */
> +        bl    enable_mm
>   
>           /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
>           ldr   x0, =primary_switched
> @@ -313,7 +314,7 @@ GLOBAL(init_secondary)
>           bl    check_cpu_mode
>           bl    cpu_init
>           bl    prepare_early_mappings
> -        bl    enable_mmu
> +        bl    enable_mm
>   
>           /* We are still in the 1:1 mapping. Jump to the runtime Virtual Address. */
>           ldr   x0, =secondary_switched
> diff --git a/xen/arch/arm/arm64/head_mmu.S b/xen/arch/arm/arm64/head_mmu.S
> index fc64819a98..b542755bd2 100644
> --- a/xen/arch/arm/arm64/head_mmu.S
> +++ b/xen/arch/arm/arm64/head_mmu.S
> @@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
>    *
>    * Clobbers x0 - x3
>    */
> -ENTRY(enable_mmu)
> +ENTRY(enable_mm)
>           PRINT("- Turning on paging -\r\n")
>   
>           /*
> @@ -239,7 +239,7 @@ ENTRY(enable_mmu)
>           msr   SCTLR_EL2, x0          /* now paging is enabled */
>           isb                          /* Now, flush the icache */
>           ret
> -ENDPROC(enable_mmu)
> +ENDPROC(enable_mm)
>   
>   /*
>    * Remove the 1:1 map from the page-tables. It is not easy to keep track
> diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
> index f60611b556..5a1b03e293 100644
> --- a/xen/arch/arm/arm64/head_mpu.S
> +++ b/xen/arch/arm/arm64/head_mpu.S
> @@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
>   
>       ret
>   ENDPROC(prepare_early_mappings)
> +
> +/*
> + * Enable EL2 MPU and data cache. Because we will disable cache
> + * with MPU at the same time, in accordance with that, we have
> + * to enable cache with MPU at the same time in this function.
> + * When MPU is disabled, the MPU background attributes will
> + * be used. On some platform, the background will treat all
> + * memory as IO memory.

I was under the impression that all access would be treated as Device 
Memory when the MMU is off. Isn't it the case for the MPU?

Also, I think the correct wording is "device memory" rather than "IO 
memory".

> The access to IO memory will bypass

Ditto.

> + * the cache, even you have enabled the C bit in SCTLR.
> + * To avoid this implicit behavior, we disable cache with MPU
> + * explicitly to tell user that when MPU is disabled, the memory
> + * access is uncacheable.
> + */
> +ENTRY(enable_mm)
> +    mrs   x0, SCTLR_EL2
> +    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
> +    /* Enable EL2 MPU and D-cache */
> +    orr   x0, x0, x1
> +    dsb   sy
> +    msr   SCTLR_EL2, x0
> +    isb
> +    ret
> +ENDPROC(enable_mm)
> +
> +/* Disable MPU system, including data cache. */
> +ENTRY(disable_mm)

I would rather not introduce this function until there is a caller. This 
is because, I believe, there are some assumptions on the state of the 
cache before we can turn off the MMU. So I would like to see the caller 
in order to assess whether this function makes sense.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
> Julien Grall
> Sent: Monday, November 7, 2022 4:56 AM
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU
> enable/disable
> 
> Hi Wei,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > From: Penny Zheng <penny.zheng@arm.com>
> >
> > We need some helpers for Xen to enable/disable MPU in boot-time and
> > runtime. For MPU enable helper, we know that it's an essential
> > requirement of MPU system. But for MPU disable, we need to use it for
> > some special situations. For example, in the progress of tranferring
> > from boot-time to runtime, we need to update the MPU protection
> > regions configuration, but we can't modify an MPU protection region if
> > there is some data accessed by Xen. But in boot-time all of Xen text,
> > data and BSS are in one MPU protection region, if Xen want to update
> > this protection region, above restriction will be triggered.
> 
> This raises the following question: Why can't we create the split regions right
> now?
> 

The reason why we are not creating the split regions right now is that we
are trying to go the same path MMU goes. Then we could reuse as much
same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU
all over the place.

> In particular, disabling the MMU/Cache is fairly risky because you need to
> ensure that anything in the cache you care about have been written back to
> the RAM).
>

Hope I could understand your concern totally, you are worrying about stale info left in
the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could
be different before and after? 
So it is never enough that we only flush the variables which we will use during the disabling
time, it should be everything in the cache...:/

Since in current design, there are two time points in boot time where we will disable
MPU/Cache to configure MPU.

One is in setup_mm, here, we will map XEN components by components, each MPU memory
region for a different component.
The other is near the end of boot time, we will reorg the whole MPU memory region layout
before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear.
Otherwise it is hard and complex to maintain on runtime, especially during context switch.

Unchanging regions like xen text will not change during vcpu context switch. Flexible regions, like
guest memory region, will display different contents during vcpu context switch.

> > So in this situation, we need to disable the whole MPU to update the
> > protection regions.
> >
> > In these helper, enable/disable MPU will also enable/disable the
> > D-cache. There are two reasons for it:
> > 1. Make the function semantic be consistent with enable_mmu.
> >     For MMU systems, enable_mmu will turn MMU and D-Cache at
> >     the same time.
> > 2. When MPU is disabled, the MPU background attributes will
> >     be used. On some platforms, the background will treat all
> >     memory as device memory. The access to device memory will
> >     bypass the cache, even if the C bit is enabled in SCTLR.
> >     To avoid this implicit behavior, we disable cache with MPU
> >     explicitly to tell user that when MPU is disabled, the
> >     memory access is uncacheable.
> >
> > In this patch, we also introduce a neutral name enable_mm for Xen to
> > enable MMU/MPU. This can help us to keep one code flow in head.S
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> > ---
> >   xen/arch/arm/arm64/head.S     |  5 +++--
> >   xen/arch/arm/arm64/head_mmu.S |  4 ++--
> >   xen/arch/arm/arm64/head_mpu.S | 35
> +++++++++++++++++++++++++++++++++++
> >   3 files changed, 40 insertions(+), 4 deletions(-)
> >
> > diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> > index 6c1a5f74a1..228f01db69 100644
> > --- a/xen/arch/arm/arm64/head.S
> > +++ b/xen/arch/arm/arm64/head.S
> > @@ -255,7 +255,8 @@ real_start_efi:
> >            * and protection regions for MPU systems.
> >            */
> >           bl    prepare_early_mappings
> > -        bl    enable_mmu
> > +        /* Turn on MMU or MPU */
> > +        bl    enable_mm
> >
> >           /* We are still in the 1:1 mapping. Jump to the runtime Virtual
> Address. */
> >           ldr   x0, =primary_switched
> > @@ -313,7 +314,7 @@ GLOBAL(init_secondary)
> >           bl    check_cpu_mode
> >           bl    cpu_init
> >           bl    prepare_early_mappings
> > -        bl    enable_mmu
> > +        bl    enable_mm
> >
> >           /* We are still in the 1:1 mapping. Jump to the runtime Virtual
> Address. */
> >           ldr   x0, =secondary_switched
> > diff --git a/xen/arch/arm/arm64/head_mmu.S
> > b/xen/arch/arm/arm64/head_mmu.S index fc64819a98..b542755bd2
> 100644
> > --- a/xen/arch/arm/arm64/head_mmu.S
> > +++ b/xen/arch/arm/arm64/head_mmu.S
> > @@ -217,7 +217,7 @@ ENDPROC(prepare_early_mappings)
> >    *
> >    * Clobbers x0 - x3
> >    */
> > -ENTRY(enable_mmu)
> > +ENTRY(enable_mm)
> >           PRINT("- Turning on paging -\r\n")
> >
> >           /*
> > @@ -239,7 +239,7 @@ ENTRY(enable_mmu)
> >           msr   SCTLR_EL2, x0          /* now paging is enabled */
> >           isb                          /* Now, flush the icache */
> >           ret
> > -ENDPROC(enable_mmu)
> > +ENDPROC(enable_mm)
> >
> >   /*
> >    * Remove the 1:1 map from the page-tables. It is not easy to keep
> > track diff --git a/xen/arch/arm/arm64/head_mpu.S
> > b/xen/arch/arm/arm64/head_mpu.S index f60611b556..5a1b03e293
> 100644
> > --- a/xen/arch/arm/arm64/head_mpu.S
> > +++ b/xen/arch/arm/arm64/head_mpu.S
> > @@ -68,3 +68,38 @@ ENTRY(prepare_early_mappings)
> >
> >       ret
> >   ENDPROC(prepare_early_mappings)
> > +
> > +/*
> > + * Enable EL2 MPU and data cache. Because we will disable cache
> > + * with MPU at the same time, in accordance with that, we have
> > + * to enable cache with MPU at the same time in this function.
> > + * When MPU is disabled, the MPU background attributes will
> > + * be used. On some platform, the background will treat all
> > + * memory as IO memory.
> 
> I was under the impression that all access would be treated as Device
> Memory when the MMU is off. Isn't it the case for the MPU?
> 

Yes, without background region. Right now, background region is disabled
here.
But if the Background region is enabled, then the MPU uses the default memory
map as the Background region for generating the memory attributes when MPU is disabled.
And it is also IMPLENMENTATION DEFINED~

> Also, I think the correct wording is "device memory" rather than "IO
> memory".
> 
> > The access to IO memory will bypass
> 
> Ditto.
> 
> > + * the cache, even you have enabled the C bit in SCTLR.
> > + * To avoid this implicit behavior, we disable cache with MPU
> > + * explicitly to tell user that when MPU is disabled, the memory
> > + * access is uncacheable.
> > + */
> > +ENTRY(enable_mm)
> > +    mrs   x0, SCTLR_EL2
> > +    mov   x1, #(SCTLR_Axx_ELx_M | SCTLR_Axx_ELx_C)
> > +    /* Enable EL2 MPU and D-cache */
> > +    orr   x0, x0, x1
> > +    dsb   sy
> > +    msr   SCTLR_EL2, x0
> > +    isb
> > +    ret
> > +ENDPROC(enable_mm)
> > +
> > +/* Disable MPU system, including data cache. */
> > +ENTRY(disable_mm)
> 
> I would rather not introduce this function until there is a caller. This is
> because, I believe, there are some assumptions on the state of the cache
> before we can turn off the MMU. So I would like to see the caller in order to
> assess whether this function makes sense.
> 
> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

[Julien Grall]

On 07/11/2022 09:57, Penny Zheng wrote:
> Hi Julien

Hi Penny,

> 
>> -----Original Message-----
>> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
>> Julien Grall
>> Sent: Monday, November 7, 2022 4:56 AM
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
>> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
>> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>
>> Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU
>> enable/disable
>>
>> Hi Wei,
>>
>> On 04/11/2022 10:07, Wei Chen wrote:
>>> From: Penny Zheng <penny.zheng@arm.com>
>>>
>>> We need some helpers for Xen to enable/disable MPU in boot-time and
>>> runtime. For MPU enable helper, we know that it's an essential
>>> requirement of MPU system. But for MPU disable, we need to use it for
>>> some special situations. For example, in the progress of tranferring
>>> from boot-time to runtime, we need to update the MPU protection
>>> regions configuration, but we can't modify an MPU protection region if
>>> there is some data accessed by Xen. But in boot-time all of Xen text,
>>> data and BSS are in one MPU protection region, if Xen want to update
>>> this protection region, above restriction will be triggered.
>>
>> This raises the following question: Why can't we create the split regions right
>> now?
>>
> 
> The reason why we are not creating the split regions right now is that we
> are trying to go the same path MMU goes. 

The MMU code is going to change pretty soon (see [1] for some ground 
work). The runtime page-tables for CPU0 will be created in assembly code 
and never switched after (aside when using cache coloring).

Although, I don't think I will apply the proper permissions in assembly 
(this is a bit trickier than with the MPU).

> Then we could reuse as much
> same interfaces as we could, in order to not leave #ifdef CONFIG_HAS_MPU
> all over the place.
Do you have a list of those interfaces that would require #ifdef?

> 
>> In particular, disabling the MMU/Cache is fairly risky because you need to
>> ensure that anything in the cache you care about have been written back to
>> the RAM).
>>
> 
> Hope I could understand your concern totally, you are worrying about stale info left in
> the cache, even if it's always 1:1 mapping on the MPU system, memory attributes could
> be different before and after?

No. I am more concerned about...

> So it is never enough that we only flush the variables which we will use during the disabling
> time, it should be everything in the cache...:/

... this. We don't only need to flush before they are accessed but also 
after if they are modified.

It is possible to do it correctly, but it requires to be very careful. 
So if we can avoid disabling the cache/MPU then it will be a lot better.

> 
> Since in current design, there are two time points in boot time where we will disable
> MPU/Cache to configure MPU.
> 
> One is in setup_mm, here, we will map XEN components by components, each MPU memory
> region for a different component.
> The other is near the end of boot time, we will reorg the whole MPU memory region layout
> before going runtime, and we will keep unchanging regions in the front and flexible ones in the rear.

You should not need any reorg if you map the boot-only section towards 
in the higher slot index (or just after the fixed ones).

Cheers,

[1] 20221022150422.17707-1-julien@xen.org

-- 
Julien Grall

RE: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU enable/disable

[Penny Zheng]

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Monday, November 7, 2022 6:38 PM
> To: Penny Zheng <Penny.Zheng@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU
> enable/disable
> 
> 
> 
> On 07/11/2022 09:57, Penny Zheng wrote:
> > Hi Julien
> 
> Hi Penny,

Hi Julien

> 
> >
> >> -----Original Message-----
> >> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
> >> Julien Grall
> >> Sent: Monday, November 7, 2022 4:56 AM
> >> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> >> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
> >> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
> >> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> >> <Volodymyr_Babchuk@epam.com>
> >> Subject: Re: [PATCH v6 10/11] xen/arm64: introduce helpers for MPU
> >> enable/disable
> >>
> >> Hi Wei,
> >>
> >> On 04/11/2022 10:07, Wei Chen wrote:
> >>> From: Penny Zheng <penny.zheng@arm.com>
> >>>
> >>> We need some helpers for Xen to enable/disable MPU in boot-time and
> >>> runtime. For MPU enable helper, we know that it's an essential
> >>> requirement of MPU system. But for MPU disable, we need to use it
> >>> for some special situations. For example, in the progress of
> >>> tranferring from boot-time to runtime, we need to update the MPU
> >>> protection regions configuration, but we can't modify an MPU
> >>> protection region if there is some data accessed by Xen. But in
> >>> boot-time all of Xen text, data and BSS are in one MPU protection
> >>> region, if Xen want to update this protection region, above restriction will
> be triggered.
> >>
> >> This raises the following question: Why can't we create the split
> >> regions right now?
> >>
> >
> > The reason why we are not creating the split regions right now is that
> > we are trying to go the same path MMU goes.
> 
> The MMU code is going to change pretty soon (see [1] for some ground
> work). The runtime page-tables for CPU0 will be created in assembly code
> and never switched after (aside when using cache coloring).
> 
> Although, I don't think I will apply the proper permissions in assembly (this is
> a bit trickier than with the MPU).
> 
> > Then we could reuse as much
> > same interfaces as we could, in order to not leave #ifdef
> > CONFIG_HAS_MPU all over the place.
> Do you have a list of those interfaces that would require #ifdef?
> 
> >
> >> In particular, disabling the MMU/Cache is fairly risky because you
> >> need to ensure that anything in the cache you care about have been
> >> written back to the RAM).
> >>
> >
> > Hope I could understand your concern totally, you are worrying about
> > stale info left in the cache, even if it's always 1:1 mapping on the
> > MPU system, memory attributes could be different before and after?
> 
> No. I am more concerned about...
> 
> > So it is never enough that we only flush the variables which we will
> > use during the disabling time, it should be everything in the
> > cache...:/
> 
> ... this. We don't only need to flush before they are accessed but also after if
> they are modified.
> 
> It is possible to do it correctly, but it requires to be very careful.
> So if we can avoid disabling the cache/MPU then it will be a lot better.
> 
> >
> > Since in current design, there are two time points in boot time where
> > we will disable MPU/Cache to configure MPU.
> >
> > One is in setup_mm, here, we will map XEN components by components,
> > each MPU memory region for a different component.
> > The other is near the end of boot time, we will reorg the whole MPU
> > memory region layout before going runtime, and we will keep unchanging
> regions in the front and flexible ones in the rear.
> 
> You should not need any reorg if you map the boot-only section towards in
> the higher slot index (or just after the fixed ones).
> 

"in the higher slot index" is really shining a light in my mind ;) And I'll try to enable it
in v2.

> Cheers,
> 
> [1] 20221022150422.17707-1-julien@xen.org
> 
> --
> Julien Grall

[PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

[Wei Chen]

From: Penny Zheng <penny.zheng@arm.com>

setup_fixmap and remove_identity_mapping are two functions that
are used in Xen boot-time code flow. We implement these two
functions for MPU system, in this case, the code flow in head.S
doesn't need to use #ifdef to gate MPU/MMU code.

In MMU system, setup_fixmap is used for Xen to map some essentail
data or devices in boot-time. For MPU system, we still have this
requirement, we map the early UART to MPU protection region when
earlyprintk is enabled. This also means PRINT can't be used after
turning on MPU but before setup_fixmap. This restriction is the
same as MMU system.

For remove_identity_mapping, we just need an empty function to
make head.S code flow happy.

Signed-off-by: Wei Chen <wei.chen@arm.com>
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
 .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
 2 files changed, 53 insertions(+)

diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
index 5a1b03e293..336c0a630f 100644
--- a/xen/arch/arm/arm64/head_mpu.S
+++ b/xen/arch/arm/arm64/head_mpu.S
@@ -20,13 +20,20 @@
 /*
  * In boot stage, we will use 1 MPU region:
  * Region#0: Normal memory for Xen text + data + bss (2MB)
+ * Region#1: Device memory for EARLY UART, size is defined
+ *           by platform's EARLY_UART_SIZE
  */
 #define BOOT_NORMAL_REGION_IDX  0x0
+#define BOOT_DEVICE_REGION_IDX  0x1
 
 /* MPU normal memory attributes. */
 #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
 #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
 
+/* MPU device memory attributes. */
+#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
+#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
+
 .macro write_pr, sel, prbar, prlar
     msr   PRSELR_EL2, \sel
     dsb   sy
@@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
     ret
 ENDPROC(prepare_early_mappings)
 
+/*
+ * In MMU system, setup_fixmap is used for Xen to map some essential data
+ * or devices in boot-time. In order to be consistent with MMU system, we
+ * inherit the function name for MPU system.
+ * setup_fixmap of MPU system will:
+ * - Map the early UART to MPU protection region when earlyprintk is
+ *   enabled (The PRINT can't be used after turning on MPU but before
+ *   setup_fixmap).
+ *
+ * Clobbers x0 - x3
+ */
+ENTRY(setup_fixmap)
+#ifdef CONFIG_EARLY_PRINTK
+    /* Map early uart to MPU device region for early printk. */
+    mov x0, #BOOT_DEVICE_REGION_IDX
+    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
+    and x1, x1, #MPU_REGION_MASK
+    mov x3, #PRBAR_DEVICE_MEM
+    orr x1, x1, x3
+
+    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
+    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
+    add x2, x2, x3
+    and x2, x2, #MPU_REGION_MASK
+    mov x3, #PRLAR_DEVICE_MEM
+    orr x2, x2, x3
+
+    /*
+     * Write to MPU protection region:
+     * x0 for pr_sel, x1 for prbar x2 for prlar
+     */
+    write_pr x0, x1, x2
+#endif
+
+    ret
+ENDPROC(setup_fixmap)
+
+/* Stub of remove_identity_mapping for MPU systems */
+ENTRY(remove_identity_mapping)
+    ret
+ENDPROC(remove_identity_mapping)
+
 /*
  * Enable EL2 MPU and data cache. Because we will disable cache
  * with MPU at the same time, in accordance with that, we have
diff --git a/xen/arch/arm/include/asm/platforms/fvp_baser.h b/xen/arch/arm/include/asm/platforms/fvp_baser.h
index 9450a411a9..acde3541a1 100644
--- a/xen/arch/arm/include/asm/platforms/fvp_baser.h
+++ b/xen/arch/arm/include/asm/platforms/fvp_baser.h
@@ -11,4 +11,8 @@
 #define XEN_START_ADDRESS CONFIG_XEN_START_ADDRESS
 #endif
 
+#ifdef CONFIG_EARLY_PRINTK
+#define EARLY_UART_SIZE   0x1000
+#endif
+
 #endif /* __ASM_ARM_PLATFORMS_FVP_BASER_H__ */
-- 
2.25.1

Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

[Julien Grall]

Hi,

On 04/11/2022 10:07, Wei Chen wrote:
> From: Penny Zheng <penny.zheng@arm.com>
> 
> setup_fixmap and remove_identity_mapping are two functions that
> are used in Xen boot-time code flow. We implement these two
> functions for MPU system, in this case, the code flow in head.S
> doesn't need to use #ifdef to gate MPU/MMU code.
> 
> In MMU system, setup_fixmap is used for Xen to map some essentail
> data or devices in boot-time. For MPU system, we still have this
> requirement, we map the early UART to MPU protection region when
> earlyprintk is enabled. This also means PRINT can't be used after
> turning on MPU but before setup_fixmap. This restriction is the
> same as MMU system.
> 
> For remove_identity_mapping, we just need an empty function to
> make head.S code flow happy.
> 
> Signed-off-by: Wei Chen <wei.chen@arm.com>
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> ---
>   xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
>   .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
>   2 files changed, 53 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/head_mpu.S b/xen/arch/arm/arm64/head_mpu.S
> index 5a1b03e293..336c0a630f 100644
> --- a/xen/arch/arm/arm64/head_mpu.S
> +++ b/xen/arch/arm/arm64/head_mpu.S
> @@ -20,13 +20,20 @@
>   /*
>    * In boot stage, we will use 1 MPU region:
>    * Region#0: Normal memory for Xen text + data + bss (2MB)
> + * Region#1: Device memory for EARLY UART, size is defined
> + *           by platform's EARLY_UART_SIZE
>    */
>   #define BOOT_NORMAL_REGION_IDX  0x0
> +#define BOOT_DEVICE_REGION_IDX  0x1
>   
>   /* MPU normal memory attributes. */
>   #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
>   #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
>   
> +/* MPU device memory attributes. */
> +#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
> +#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
> +
>   .macro write_pr, sel, prbar, prlar
>       msr   PRSELR_EL2, \sel
>       dsb   sy
> @@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
>       ret
>   ENDPROC(prepare_early_mappings)
>   
> +/*
> + * In MMU system, setup_fixmap is used for Xen to map some essential data
> + * or devices in boot-time. In order to be consistent with MMU system, we
> + * inherit the function name for MPU system.
> + * setup_fixmap of MPU system will:
> + * - Map the early UART to MPU protection region when earlyprintk is
> + *   enabled (The PRINT can't be used after turning on MPU but before
> + *   setup_fixmap).

For the MMU, we have this restriction because the fixmap could clash 
with the identity mapping. I don't think there are such restrictions for 
the MPU and therefore it seems strange to pertain the same behavior.

In fact, I have plan to get rid of this restriction even for the MMU. So 
better this restriction is not spread if we can.

> + *
> + * Clobbers x0 - x3
> + */
> +ENTRY(setup_fixmap)
> +#ifdef CONFIG_EARLY_PRINTK
> +    /* Map early uart to MPU device region for early printk. */
> +    mov x0, #BOOT_DEVICE_REGION_IDX
> +    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
> +    and x1, x1, #MPU_REGION_MASK
> +    mov x3, #PRBAR_DEVICE_MEM
> +    orr x1, x1, x3
> +
> +    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
> +    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
> +    add x2, x2, x3
> +    and x2, x2, #MPU_REGION_MASK
> +    mov x3, #PRLAR_DEVICE_MEM
> +    orr x2, x2, x3
> +
> +    /*
> +     * Write to MPU protection region:
> +     * x0 for pr_sel, x1 for prbar x2 for prlar
> +     */
> +    write_pr x0, x1, x2
> +#endif
> +
> +    ret
> +ENDPROC(setup_fixmap)
> +
> +/* Stub of remove_identity_mapping for MPU systems */
> +ENTRY(remove_identity_mapping)
> +    ret
> +ENDPROC(remove_identity_mapping)

This stub could be avoided if you move the call to 
remove_identity_mapping in enable_mmu() as I did for arm32. See [1].

[1] https://lore.kernel.org/all/20221022150422.17707-3-julien@xen.org/

Cheers,

-- 
Julien Grall

RE: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Monday, November 7, 2022 5:02 AM
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and
> remove_identity_mapping for MPU
> 
> Hi,
> 
> On 04/11/2022 10:07, Wei Chen wrote:
> > From: Penny Zheng <penny.zheng@arm.com>
> >
> > setup_fixmap and remove_identity_mapping are two functions that are
> > used in Xen boot-time code flow. We implement these two functions for
> > MPU system, in this case, the code flow in head.S doesn't need to use
> > #ifdef to gate MPU/MMU code.
> >
> > In MMU system, setup_fixmap is used for Xen to map some essentail data
> > or devices in boot-time. For MPU system, we still have this
> > requirement, we map the early UART to MPU protection region when
> > earlyprintk is enabled. This also means PRINT can't be used after
> > turning on MPU but before setup_fixmap. This restriction is the same
> > as MMU system.
> >
> > For remove_identity_mapping, we just need an empty function to make
> > head.S code flow happy.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> > ---
> >   xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
> >   .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
> >   2 files changed, 53 insertions(+)
> >
> > diff --git a/xen/arch/arm/arm64/head_mpu.S
> > b/xen/arch/arm/arm64/head_mpu.S index 5a1b03e293..336c0a630f
> 100644
> > --- a/xen/arch/arm/arm64/head_mpu.S
> > +++ b/xen/arch/arm/arm64/head_mpu.S
> > @@ -20,13 +20,20 @@
> >   /*
> >    * In boot stage, we will use 1 MPU region:
> >    * Region#0: Normal memory for Xen text + data + bss (2MB)
> > + * Region#1: Device memory for EARLY UART, size is defined
> > + *           by platform's EARLY_UART_SIZE
> >    */
> >   #define BOOT_NORMAL_REGION_IDX  0x0
> > +#define BOOT_DEVICE_REGION_IDX  0x1
> >
> >   /* MPU normal memory attributes. */
> >   #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
> >   #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
> >
> > +/* MPU device memory attributes. */
> > +#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
> > +#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
> > +
> >   .macro write_pr, sel, prbar, prlar
> >       msr   PRSELR_EL2, \sel
> >       dsb   sy
> > @@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
> >       ret
> >   ENDPROC(prepare_early_mappings)
> >
> > +/*
> > + * In MMU system, setup_fixmap is used for Xen to map some essential
> > +data
> > + * or devices in boot-time. In order to be consistent with MMU
> > +system, we
> > + * inherit the function name for MPU system.
> > + * setup_fixmap of MPU system will:
> > + * - Map the early UART to MPU protection region when earlyprintk is
> > + *   enabled (The PRINT can't be used after turning on MPU but before
> > + *   setup_fixmap).
> 
> For the MMU, we have this restriction because the fixmap could clash with
> the identity mapping. I don't think there are such restrictions for the MPU
> and therefore it seems strange to pertain the same behavior.
> 

Yes, both removing identity mapping and using fixmap virtual memory layout are
not applied to the MPU system.

And in MMU system, the setup_fixmap function has a behavior to map the UART
for early printk. And we are only trying to pertain this behavior on MPU system. 

> In fact, I have plan to get rid of this restriction even for the MMU. So better
> this restriction is not spread if we can.

Hmm, I'm a bit confused here. Which restriction you are trying to remove? The whole
fixmap virtual memory layout?
 
> 
> > + *
> > + * Clobbers x0 - x3
> > + */
> > +ENTRY(setup_fixmap)
> > +#ifdef CONFIG_EARLY_PRINTK
> > +    /* Map early uart to MPU device region for early printk. */
> > +    mov x0, #BOOT_DEVICE_REGION_IDX
> > +    ldr x1, =CONFIG_EARLY_UART_BASE_ADDRESS
> > +    and x1, x1, #MPU_REGION_MASK
> > +    mov x3, #PRBAR_DEVICE_MEM
> > +    orr x1, x1, x3
> > +
> > +    ldr x2, =CONFIG_EARLY_UART_BASE_ADDRESS
> > +    ldr x3, =(CONFIG_EARLY_UART_BASE_ADDRESS + EARLY_UART_SIZE - 1)
> > +    add x2, x2, x3
> > +    and x2, x2, #MPU_REGION_MASK
> > +    mov x3, #PRLAR_DEVICE_MEM
> > +    orr x2, x2, x3
> > +
> > +    /*
> > +     * Write to MPU protection region:
> > +     * x0 for pr_sel, x1 for prbar x2 for prlar
> > +     */
> > +    write_pr x0, x1, x2
> > +#endif
> > +
> > +    ret
> > +ENDPROC(setup_fixmap)
> > +
> > +/* Stub of remove_identity_mapping for MPU systems */
> > +ENTRY(remove_identity_mapping)
> > +    ret
> > +ENDPROC(remove_identity_mapping)
> 
> This stub could be avoided if you move the call to remove_identity_mapping
> in enable_mmu() as I did for arm32. See [1].
> 
> [1] https://lore.kernel.org/all/20221022150422.17707-3-julien@xen.org/
> 

Thx! Understood, and I'll use the same logic for enable_mmu.

> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

[Julien Grall]

Hi Penny,

On 07/11/2022 08:13, Penny Zheng wrote:
>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: Monday, November 7, 2022 5:02 AM
>> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
>> Cc: nd <nd@arm.com>; Penny Zheng <Penny.Zheng@arm.com>; Stefano
>> Stabellini <sstabellini@kernel.org>; Bertrand Marquis
>> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
>> <Volodymyr_Babchuk@epam.com>
>> Subject: Re: [PATCH v6 11/11] xen/arm64: add setup_fixmap and
>> remove_identity_mapping for MPU
>>
>> Hi,
>>
>> On 04/11/2022 10:07, Wei Chen wrote:
>>> From: Penny Zheng <penny.zheng@arm.com>
>>>
>>> setup_fixmap and remove_identity_mapping are two functions that are
>>> used in Xen boot-time code flow. We implement these two functions for
>>> MPU system, in this case, the code flow in head.S doesn't need to use
>>> #ifdef to gate MPU/MMU code.
>>>
>>> In MMU system, setup_fixmap is used for Xen to map some essentail data
>>> or devices in boot-time. For MPU system, we still have this
>>> requirement, we map the early UART to MPU protection region when
>>> earlyprintk is enabled. This also means PRINT can't be used after
>>> turning on MPU but before setup_fixmap. This restriction is the same
>>> as MMU system.
>>>
>>> For remove_identity_mapping, we just need an empty function to make
>>> head.S code flow happy.
>>>
>>> Signed-off-by: Wei Chen <wei.chen@arm.com>
>>> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
>>> ---
>>>    xen/arch/arm/arm64/head_mpu.S                 | 49 +++++++++++++++++++
>>>    .../arm/include/asm/platforms/fvp_baser.h     |  4 ++
>>>    2 files changed, 53 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/arm64/head_mpu.S
>>> b/xen/arch/arm/arm64/head_mpu.S index 5a1b03e293..336c0a630f
>> 100644
>>> --- a/xen/arch/arm/arm64/head_mpu.S
>>> +++ b/xen/arch/arm/arm64/head_mpu.S
>>> @@ -20,13 +20,20 @@
>>>    /*
>>>     * In boot stage, we will use 1 MPU region:
>>>     * Region#0: Normal memory for Xen text + data + bss (2MB)
>>> + * Region#1: Device memory for EARLY UART, size is defined
>>> + *           by platform's EARLY_UART_SIZE
>>>     */
>>>    #define BOOT_NORMAL_REGION_IDX  0x0
>>> +#define BOOT_DEVICE_REGION_IDX  0x1
>>>
>>>    /* MPU normal memory attributes. */
>>>    #define PRBAR_NORMAL_MEM        0x30    /* SH=11 AP=00 XN=00 */
>>>    #define PRLAR_NORMAL_MEM        0x0f    /* NS=0 ATTR=111 EN=1 */
>>>
>>> +/* MPU device memory attributes. */
>>> +#define PRBAR_DEVICE_MEM        0x20    /* SH=10 AP=00 XN=00 */
>>> +#define PRLAR_DEVICE_MEM        0x09    /* NS=0 ATTR=100 EN=1 */
>>> +
>>>    .macro write_pr, sel, prbar, prlar
>>>        msr   PRSELR_EL2, \sel
>>>        dsb   sy
>>> @@ -69,6 +76,48 @@ ENTRY(prepare_early_mappings)
>>>        ret
>>>    ENDPROC(prepare_early_mappings)
>>>
>>> +/*
>>> + * In MMU system, setup_fixmap is used for Xen to map some essential
>>> +data
>>> + * or devices in boot-time. In order to be consistent with MMU
>>> +system, we
>>> + * inherit the function name for MPU system.
>>> + * setup_fixmap of MPU system will:
>>> + * - Map the early UART to MPU protection region when earlyprintk is
>>> + *   enabled (The PRINT can't be used after turning on MPU but before
>>> + *   setup_fixmap).
>>
>> For the MMU, we have this restriction because the fixmap could clash with
>> the identity mapping. I don't think there are such restrictions for the MPU
>> and therefore it seems strange to pertain the same behavior.
>>
> 
> Yes, both removing identity mapping and using fixmap virtual memory layout are
> not applied to the MPU system.
> 
> And in MMU system, the setup_fixmap function has a behavior to map the UART
> for early printk. And we are only trying to pertain this behavior on MPU system.
> 
>> In fact, I have plan to get rid of this restriction even for the MMU. So better
>> this restriction is not spread if we can.
> 
> Hmm, I'm a bit confused here. Which restriction you are trying to remove? 

The fact that the fixmap entry is not created in create_page_tables() 
(now prepare_early_mappings()).

Cheers,

-- 
Julien Grall

RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Wei Chen]

Sorry for the wrong patch version. This is the first version of this series.

Cheers,
Wei Chen

> -----Original Message-----
> From: Wei Chen <wei.chen@arm.com>
> Sent: 2022年11月4日 18:08
> To: xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Wei Chen <Wei.Chen@arm.com>; Stefano Stabellini
> <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>
> Subject: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> The Armv-R architecture profile was designed to support use cases
> that have a high sensitivity to deterministic execution. (e.g.
> Fuel Injection, Brake control, Drive trains, Motor control etc)
> 
> Arm announced Armv8-R in 2013, it is the latest generation Arm
> architecture targeted at the Real-time profile. It introduces
> virtualization at the highest security level while retaining the
> Protected Memory System Architecture (PMSA) based on a Memory
> Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
> which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
> The latest Armv8-R64 document can be found [1]. And the features of
> Armv8-R64 architecture:
>   - An exception model that is compatible with the Armv8-A model
>   - Virtualization with support for guest operating systems
>   - PMSA virtualization using MPUs In EL2.
>   - Adds support for the 64-bit A64 instruction set.
>   - Supports up to 48-bit physical addressing.
>   - Supports three Exception Levels (ELs)
>         - Secure EL2 - The Highest Privilege
>         - Secure EL1 - RichOS (MMU) or RTOS (MPU)
>         - Secure EL0 - Application Workloads
>  - Supports only a single Security state - Secure.
>  - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> 
> These patch series are implementing the Armv8-R64 MPU support
> for Xen, which are based on the discussion of
> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> 
> We will implement the Armv8-R64 and MPU support in three stages:
> 1. Boot Xen itself to idle thread, do not create any guests on it.
> 2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
> 3. SMP and other advanced features of Xen support on Armv8-R64.
> 
> We will split these patches to several parts, this series is the
> part#1, the full PoC can be found in [3]. More software for
> Armv8-R64 can be found in [4];
> 
> [1] https://developer.arm.com/documentation/ddi0600/latest
> [2] https://lists.xenproject.org/archives/html/xen-devel/2022-
> 05/msg00643.html
> [3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-
> /tree/poc/r82-mpu-v2
> [4] https://armv8r64-refstack.docs.arm.com/en/v4.0/
> 
> Penny Zheng (3):
>   xen/arm64: create boot-time MPU protection regions
>   xen/arm64: introduce helpers for MPU enable/disable
>   xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
> 
> Wei Chen (8):
>   xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
>   xen/arm: add iounmap after initrd has been loaded in domain_build
>   xen/arm: disable EFI boot services for MPU systems
>   xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
>   xen/arm: define Xen start address for FVP BaseR platform
>   xen/arm: split MMU and MPU config files from config.h
>   xen/arm: implement FIXMAP_ADDR for MPU systems
>   xen/arm64: move MMU related code from head.S to head_mmu.S
> 
>  xen/arch/arm/Kconfig                          |  15 +-
>  xen/arch/arm/arm64/Makefile                   |   5 +
>  xen/arch/arm/arm64/head.S                     | 429 ++----------------
>  xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
>  xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
>  xen/arch/arm/domain_build.c                   |   2 +
>  xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
>  xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
>  xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
>  xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
>  xen/arch/arm/include/asm/config.h             |  99 +---
>  xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
>  xen/arch/arm/include/asm/config_mpu.h         |  29 ++
>  xen/arch/arm/include/asm/fixmap.h             |  25 +
>  xen/arch/arm/include/asm/flushtlb.h           |  22 +
>  .../arm/include/asm/platforms/fvp_baser.h     |  18 +
>  xen/arch/arm/platforms/Kconfig                |  16 +-
>  17 files changed, 976 insertions(+), 500 deletions(-)
>  create mode 100644 xen/arch/arm/arm64/head_mmu.S
>  create mode 100644 xen/arch/arm/arm64/head_mpu.S
>  create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
>  create mode 100644 xen/arch/arm/include/asm/config_mmu.h
>  create mode 100644 xen/arch/arm/include/asm/config_mpu.h
>  create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
> 
> --
> 2.25.1

Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Julien Grall]

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> The Armv-R architecture profile was designed to support use cases
> that have a high sensitivity to deterministic execution. (e.g.
> Fuel Injection, Brake control, Drive trains, Motor control etc)
> 
> Arm announced Armv8-R in 2013, it is the latest generation Arm
> architecture targeted at the Real-time profile. It introduces
> virtualization at the highest security level while retaining the
> Protected Memory System Architecture (PMSA) based on a Memory
> Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
> which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
> The latest Armv8-R64 document can be found [1]. And the features of
> Armv8-R64 architecture:
>    - An exception model that is compatible with the Armv8-A model
>    - Virtualization with support for guest operating systems
>    - PMSA virtualization using MPUs In EL2.
>    - Adds support for the 64-bit A64 instruction set.
>    - Supports up to 48-bit physical addressing.
>    - Supports three Exception Levels (ELs)
>          - Secure EL2 - The Highest Privilege
>          - Secure EL1 - RichOS (MMU) or RTOS (MPU)
>          - Secure EL0 - Application Workloads
>   - Supports only a single Security state - Secure.
>   - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> 
> These patch series are implementing the Armv8-R64 MPU support
> for Xen, which are based on the discussion of
> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> 
> We will implement the Armv8-R64 and MPU support in three stages:
> 1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). 
However... HAS_MPU is not defined and I was expecting mm.c to be 
modified to cater the MPU support. So I am a bit ensure what the series 
is actually doing.

Cheers,

-- 
Julien Grall

RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Wei Chen]

Hi Julien,

> >   - Supports only a single Security state - Secure.
> >   - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> >
> > These patch series are implementing the Armv8-R64 MPU support
> > for Xen, which are based on the discussion of
> > "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> >
> > We will implement the Armv8-R64 and MPU support in three stages:
> > 1. Boot Xen itself to idle thread, do not create any guests on it.
> 
> I read this as I can build Xen and see it boots (not creating domain).
> However... HAS_MPU is not defined and I was expecting mm.c to be
> modified to cater the MPU support. So I am a bit ensure what the series
> is actually doing.
> 

These 11 patches are part#1 of stage#1, the full stage#1 has about 30
patches. We have some concerns if we send so many patches at once, the
review pressure of maintainers may be very high, so we only choose about
10 to send as part of it. But this also means that we can't do a relatively
complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches can be
Reviewed efficiently. Can we send the patches by stages, even the stage#1
will have about 30 patches?

Cheers,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Julien Grall]

On 07/11/2022 09:52, Wei Chen wrote:
> Hi Julien,

Hi,

> 
>>>    - Supports only a single Security state - Secure.
>>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
>>>
>>> These patch series are implementing the Armv8-R64 MPU support
>>> for Xen, which are based on the discussion of
>>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
>>>
>>> We will implement the Armv8-R64 and MPU support in three stages:
>>> 1. Boot Xen itself to idle thread, do not create any guests on it.
>>
>> I read this as I can build Xen and see it boots (not creating domain).
>> However... HAS_MPU is not defined and I was expecting mm.c to be
>> modified to cater the MPU support. So I am a bit ensure what the series
>> is actually doing.
>>
> 
> These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> patches. We have some concerns if we send so many patches at once, the
> review pressure of maintainers may be very high, so we only choose about
> 10 to send as part of it. But this also means that we can't do a relatively
> complete thing in this part#1 series.
> 
> We want to hear some suggestions from you to make so many patches can be
> Reviewed efficiently. Can we send the patches by stages, even the stage#1
> will have about 30 patches?

30 patches in a go is no too bad. I would personally prefer that because 
at least I have better idea of the shape of the code after stage#1 and 
also possibly test it (I need to check if I have access for the ARMv8-R 
model).

Cheers,

-- 
Julien Grall

RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Wei Chen]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 18:16
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> 
> 
> On 07/11/2022 09:52, Wei Chen wrote:
> > Hi Julien,
> 
> Hi,
> 
> >
> >>>    - Supports only a single Security state - Secure.
> >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> >>>
> >>> These patch series are implementing the Armv8-R64 MPU support
> >>> for Xen, which are based on the discussion of
> >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> >>>
> >>> We will implement the Armv8-R64 and MPU support in three stages:
> >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> >>
> >> I read this as I can build Xen and see it boots (not creating domain).
> >> However... HAS_MPU is not defined and I was expecting mm.c to be
> >> modified to cater the MPU support. So I am a bit ensure what the series
> >> is actually doing.
> >>
> >
> > These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> > patches. We have some concerns if we send so many patches at once, the
> > review pressure of maintainers may be very high, so we only choose about
> > 10 to send as part of it. But this also means that we can't do a
> relatively
> > complete thing in this part#1 series.
> >
> > We want to hear some suggestions from you to make so many patches can be
> > Reviewed efficiently. Can we send the patches by stages, even the
> stage#1
> > will have about 30 patches?
> 
> 30 patches in a go is no too bad. I would personally prefer that because
> at least I have better idea of the shape of the code after stage#1 and
> also possibly test it (I need to check if I have access for the ARMv8-R
> model).
> 

I also prefer to this way. After we have addressed the comments in
this series, we will send the full stage#1 patches together in v2.

For Armv8-R model, you can download Armv8-R AEM FVP model from link[1].
It's license free.

[1] https://developer.arm.com/downloads/-/arm-ecosystem-models

Cheers,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2903 bytes --]

On Mon, 7 Nov 2022, Wei Chen wrote:
> Hi Julien,
> 
> > -----Original Message-----
> > From: Julien Grall <julien@xen.org>
> > Sent: 2022年11月7日 18:16
> > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > <Volodymyr_Babchuk@epam.com>
> > Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> > Part#1
> > 
> > 
> > 
> > On 07/11/2022 09:52, Wei Chen wrote:
> > > Hi Julien,
> > 
> > Hi,
> > 
> > >
> > >>>    - Supports only a single Security state - Secure.
> > >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> > >>>
> > >>> These patch series are implementing the Armv8-R64 MPU support
> > >>> for Xen, which are based on the discussion of
> > >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> > >>>
> > >>> We will implement the Armv8-R64 and MPU support in three stages:
> > >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> > >>
> > >> I read this as I can build Xen and see it boots (not creating domain).
> > >> However... HAS_MPU is not defined and I was expecting mm.c to be
> > >> modified to cater the MPU support. So I am a bit ensure what the series
> > >> is actually doing.
> > >>
> > >
> > > These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> > > patches. We have some concerns if we send so many patches at once, the
> > > review pressure of maintainers may be very high, so we only choose about
> > > 10 to send as part of it. But this also means that we can't do a
> > relatively
> > > complete thing in this part#1 series.
> > >
> > > We want to hear some suggestions from you to make so many patches can be
> > > Reviewed efficiently. Can we send the patches by stages, even the
> > stage#1
> > > will have about 30 patches?
> > 
> > 30 patches in a go is no too bad. I would personally prefer that because
> > at least I have better idea of the shape of the code after stage#1 and
> > also possibly test it (I need to check if I have access for the ARMv8-R
> > model).
> > 
> 
> I also prefer to this way. After we have addressed the comments in
> this series, we will send the full stage#1 patches together in v2.


One suggestion to make things easier to review and to commit is to
organize the series in a way so that the first 10 patches can still be
committed first independently, even if all 30 patches are sent together.

Or alternatively only send 10 patches but also add a link to a
github/gitlab tree with all the 30+ patches so that maintainers can have
a look how the whole work fit together.

I think we are all on the same page -- I just wanted to highlight that
we don't have to finish the review of all 30 patches before we can start
committing some of the initial patches in the series.

Cheers,

Stefano

RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

[Wei Chen]

Hi Stefano, Julien,

> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: 2022年11月11日 6:26
> To: Wei Chen <Wei.Chen@arm.com>
> Cc: Julien Grall <julien@xen.org>; xen-devel@lists.xenproject.org; nd
> <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> On Mon, 7 Nov 2022, Wei Chen wrote:
> > Hi Julien,
> >
> > > -----Original Message-----
> > > From: Julien Grall <julien@xen.org>
> > > Sent: 2022年11月7日 18:16
> > > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Bertrand
> > > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > > <Volodymyr_Babchuk@epam.com>
> > > Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to
> Xen -
> > > Part#1
> > >
> > >
> > >
> > > On 07/11/2022 09:52, Wei Chen wrote:
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > >
> > > >>>    - Supports only a single Security state - Secure.
> > > >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> > > >>>
> > > >>> These patch series are implementing the Armv8-R64 MPU support
> > > >>> for Xen, which are based on the discussion of
> > > >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> > > >>>
> > > >>> We will implement the Armv8-R64 and MPU support in three stages:
> > > >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> > > >>
> > > >> I read this as I can build Xen and see it boots (not creating
> domain).
> > > >> However... HAS_MPU is not defined and I was expecting mm.c to be
> > > >> modified to cater the MPU support. So I am a bit ensure what the
> series
> > > >> is actually doing.
> > > >>
> > > >
> > > > These 11 patches are part#1 of stage#1, the full stage#1 has about
> 30
> > > > patches. We have some concerns if we send so many patches at once,
> the
> > > > review pressure of maintainers may be very high, so we only choose
> about
> > > > 10 to send as part of it. But this also means that we can't do a
> > > relatively
> > > > complete thing in this part#1 series.
> > > >
> > > > We want to hear some suggestions from you to make so many patches
> can be
> > > > Reviewed efficiently. Can we send the patches by stages, even the
> > > stage#1
> > > > will have about 30 patches?
> > >
> > > 30 patches in a go is no too bad. I would personally prefer that
> because
> > > at least I have better idea of the shape of the code after stage#1 and
> > > also possibly test it (I need to check if I have access for the ARMv8-
> R
> > > model).
> > >
> >
> > I also prefer to this way. After we have addressed the comments in
> > this series, we will send the full stage#1 patches together in v2.
> 
> 
> One suggestion to make things easier to review and to commit is to
> organize the series in a way so that the first 10 patches can still be
> committed first independently, even if all 30 patches are sent together.
> 

I think this is foreseeable, and we have done in this way internally.
Every patch can be built and will not broken other architectures.

> Or alternatively only send 10 patches but also add a link to a
> github/gitlab tree with all the 30+ patches so that maintainers can have
> a look how the whole work fit together.
> 

In this series we have linked the gitlab branch with the full patches.

> I think we are all on the same page -- I just wanted to highlight that
> we don't have to finish the review of all 30 patches before we can start
> committing some of the initial patches in the series.
> 

Yes, I agree with it. And above solutions are ok for us. There will not
be much difference in efforts for these two ways for us, so if you guys
think which method is the most efficient, I will follow it.

Cheers,
Wei Chen

> Cheers,
> 
> Stefano