[PATCH for-6.0 0/2] arm: Make M-profile VTOR loads on reset handle memory aliasing

[PATCH for-6.0 0/2] arm: Make M-profile VTOR loads on reset handle memory aliasing

[Peter Maydell]

For Arm M-profile CPUs, on reset the CPU must load its initial PC and
SP from a vector table in guest memory.  Because we can't guarantee
reset ordering, we have to handle the possibility that the ROM blob
loader's reset function has not yet run when the CPU resets, in which
case the data in an ELF file specified by the user won't be in guest
memory to be read yet.

We work around the reset ordering problem by checking whether the ROM
blob loader has any data for the address where the vector table is,
using rom_ptr().  Unfortunately this does not handle the possibility
of memory aliasing.  For many M-profile boards, memory can be accessed
via multiple possible physical addresses; if the board has the vector
table at address X but the user's ELF file loads data via a different
address Y which is an alias to the same underlying guest RAM then
rom_ptr() will not find it.

This series handles the possibility of aliasing by iterating through
the whole FlatView of the CPU's address space checking for other
mappings of the MemoryRegion corresponding to the location of the
vector table.  If we find any aliases we use rom_ptr() to see if the
ROM blob loader has any data there.

I still think that long-term the preferable option is going to be to
sort out our reset handling so that we can use three-phase-reset
everywhere and then have the rom blob loader write data in phase 2
which the CPU reset can read in phase 3. But that doesn't work today
because CPUs are not on any qbus and so they must be manually reset
outside the standard "system reset by resetting the qbus tree", using
qemu_register_reset(). The ROM blob loader also registers its reset
via qemu_register_reset(). Sadly qemu_register_reset() has no support
for three-phase-reset currently. That all adds up to "fixing reset
handling sufficiently for this approach to solve the problem is going
to take a long time", so for the moment this series is a reasonably
self-contained way to allow QEMU to run guest images linked to an
alias of the vector table address.

thanks
-- PMM

Peter Maydell (2):
  memory: Add offset_in_region to flatview_cb arguments
  target/arm: Make M-profile VTOR loads on reset handle memory aliasing

 include/exec/memory.h           |  4 +-
 softmmu/memory.c                |  3 +-
 target/arm/cpu.c                | 68 ++++++++++++++++++++++++++++++++-
 tests/qtest/fuzz/generic_fuzz.c |  4 +-
 4 files changed, 75 insertions(+), 4 deletions(-)

-- 
2.20.1

[PATCH for-6.0 1/2] memory: Add offset_in_region to flatview_cb arguments

[Peter Maydell]

The function flatview_for_each_range() calls a callback for each
range in a FlatView.  Currently the callback gets the start and
length of the range and the MemoryRegion involved, but not the offset
within the MemoryRegion.  Add this to the callback's arguments; we're
going to want it for a new use in the next commit.

While we're editing the flatview_cb typedef, add names for the
arguments that were missing names.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 include/exec/memory.h           | 4 +++-
 softmmu/memory.c                | 3 ++-
 tests/qtest/fuzz/generic_fuzz.c | 4 +++-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/include/exec/memory.h b/include/exec/memory.h
index 54ccf1a5f09..0c3ddf2bb9a 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -778,7 +778,9 @@ static inline FlatView *address_space_to_flatview(AddressSpace *as)
 
 typedef int (*flatview_cb)(Int128 start,
                            Int128 len,
-                           const MemoryRegion*, void*);
+                           const MemoryRegion *mr,
+                           hwaddr offset_in_region,
+                           void *opaque);
 
 void flatview_for_each_range(FlatView *fv, flatview_cb cb , void *opaque);
 
diff --git a/softmmu/memory.c b/softmmu/memory.c
index 9db47b7db6b..3cc78862c79 100644
--- a/softmmu/memory.c
+++ b/softmmu/memory.c
@@ -671,7 +671,8 @@ void flatview_for_each_range(FlatView *fv, flatview_cb cb , void *opaque)
     assert(cb);
 
     FOR_EACH_FLAT_RANGE(fr, fv) {
-        if (cb(fr->addr.start, fr->addr.size, fr->mr, opaque))
+        if (cb(fr->addr.start, fr->addr.size, fr->mr,
+               fr->offset_in_region, opaque))
             break;
     }
 }
diff --git a/tests/qtest/fuzz/generic_fuzz.c b/tests/qtest/fuzz/generic_fuzz.c
index ee8c17a04c4..c10a19316f5 100644
--- a/tests/qtest/fuzz/generic_fuzz.c
+++ b/tests/qtest/fuzz/generic_fuzz.c
@@ -96,7 +96,9 @@ struct get_io_cb_info {
 };
 
 static int get_io_address_cb(Int128 start, Int128 size,
-                          const MemoryRegion *mr, void *opaque) {
+                             const MemoryRegion *mr,
+                             hwaddr offset_in_region,
+                             void *opaque) {
     struct get_io_cb_info *info = opaque;
     if (g_hash_table_lookup(fuzzable_memoryregions, mr)) {
         if (info->index == 0) {
-- 
2.20.1

Re: [PATCH for-6.0 1/2] memory: Add offset_in_region to flatview_cb arguments

[Philippe Mathieu-Daudé]

On 3/12/21 6:29 PM, Peter Maydell wrote:
> The function flatview_for_each_range() calls a callback for each
> range in a FlatView.  Currently the callback gets the start and
> length of the range and the MemoryRegion involved, but not the offset
> within the MemoryRegion.  Add this to the callback's arguments; we're
> going to want it for a new use in the next commit.
> 
> While we're editing the flatview_cb typedef, add names for the
> arguments that were missing names.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
>  include/exec/memory.h           | 4 +++-
>  softmmu/memory.c                | 3 ++-
>  tests/qtest/fuzz/generic_fuzz.c | 4 +++-
>  3 files changed, 8 insertions(+), 3 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

[PATCH for-6.0 2/2] target/arm: Make M-profile VTOR loads on reset handle memory aliasing

[Peter Maydell]

For Arm M-profile CPUs, on reset the CPU must load its initial PC and
SP from a vector table in guest memory.  Because we can't guarantee
reset ordering, we have to handle the possibility that the ROM blob
loader's reset function has not yet run when the CPU resets, in which
case the data in an ELF file specified by the user won't be in guest
memory to be read yet.

We work around the reset ordering problem by checking whether the ROM
blob loader has any data for the address where the vector table is,
using rom_ptr().  Unfortunately this does not handle the possibility
of memory aliasing.  For many M-profile boards, memory can be
accessed via multiple possible physical addresses; if the board has
the vector table at address X but the user's ELF file loads data via
a different address Y which is an alias to the same underlying guest
RAM then rom_ptr() will not find it.

Handle the possibility of aliasing by iterating through the whole
FlatView of the CPU's address space checking for other mappings of
the MemoryRegion corresponding to the location of the vector table.
If we find any aliases we use rom_ptr() to see if the ROM blob loader
has any data there.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 target/arm/cpu.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 67 insertions(+), 1 deletion(-)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index ae04884408c..aac78ae6623 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -161,6 +161,72 @@ static void cp_reg_check_reset(gpointer key, gpointer value,  gpointer opaque)
     assert(oldvalue == newvalue);
 }
 
+#ifndef CONFIG_USER_ONLY
+typedef struct FindRomCBData {
+    size_t size; /* Amount of data we want from ROM, in bytes */
+    MemoryRegion *mr; /* MR at the unaliased guest addr */
+    hwaddr xlat; /* Offset of addr within mr */
+    uint8_t *rom; /* Output: rom data pointer, if found */
+} FindRomCBData;
+
+static int find_rom_cb(Int128 start, Int128 len, const MemoryRegion *mr,
+                       hwaddr offset_in_region, void *opaque)
+{
+    FindRomCBData *cbdata = opaque;
+    hwaddr alias_addr;
+
+    if (mr != cbdata->mr) {
+        return 0;
+    }
+
+    alias_addr = int128_get64(start) + cbdata->xlat - offset_in_region;
+    cbdata->rom = rom_ptr(alias_addr, cbdata->size);
+    if (!cbdata->rom) {
+        return 0;
+    }
+    /* Found a match, stop iterating */
+    return 1;
+}
+
+static uint8_t *find_rom_for_addr(AddressSpace *as, hwaddr addr, size_t size)
+{
+    /*
+     * Find any ROM data for the given guest address range.  If there
+     * is a ROM blob then return a pointer to the host memory
+     * corresponding to 'addr'; otherwise return NULL.
+     *
+     * This is like rom_ptr(), except that it handles possible aliases
+     * within the CPU's address space, so that we still find a ROM
+     * blob even if it was loaded to an address that aliases addr
+     * rather than to addr itself.
+     */
+    FlatView *fv;
+    uint8_t *rom;
+    hwaddr len_unused;
+    FindRomCBData cbdata = {};
+
+    /* Easy case: there's data at the actual address */
+    rom = rom_ptr(addr, size);
+    if (rom) {
+        return rom;
+    }
+
+    RCU_READ_LOCK_GUARD();
+
+    fv = address_space_to_flatview(as);
+    cbdata.mr = flatview_translate(fv, addr, &cbdata.xlat, &len_unused,
+                                   false, MEMTXATTRS_UNSPECIFIED);
+    if (!cbdata.mr) {
+        /* Nothing at this address, so there can't be any aliasing */
+        return NULL;
+    }
+
+    cbdata.size = size;
+    flatview_for_each_range(fv, find_rom_cb, &cbdata);
+    return cbdata.rom;
+}
+#endif
+
 static void arm_cpu_reset(DeviceState *dev)
 {
     CPUState *s = CPU(dev);
@@ -331,7 +397,7 @@ static void arm_cpu_reset(DeviceState *dev)
 
         /* Load the initial SP and PC from offset 0 and 4 in the vector table */
         vecbase = env->v7m.vecbase[env->v7m.secure];
-        rom = rom_ptr(vecbase, 8);
+        rom = find_rom_for_addr(s->as, vecbase, 8);
         if (rom) {
             /* Address zero is covered by ROM which hasn't yet been
              * copied into physical memory.
-- 
2.20.1

Re: [PATCH for-6.0 2/2] target/arm: Make M-profile VTOR loads on reset handle memory aliasing

[Philippe Mathieu-Daudé]

Hi Peter,

On 3/12/21 6:29 PM, Peter Maydell wrote:
> For Arm M-profile CPUs, on reset the CPU must load its initial PC and
> SP from a vector table in guest memory.  Because we can't guarantee
> reset ordering, we have to handle the possibility that the ROM blob
> loader's reset function has not yet run when the CPU resets, in which
> case the data in an ELF file specified by the user won't be in guest
> memory to be read yet.
> 
> We work around the reset ordering problem by checking whether the ROM
> blob loader has any data for the address where the vector table is,
> using rom_ptr().  Unfortunately this does not handle the possibility
> of memory aliasing.  For many M-profile boards, memory can be
> accessed via multiple possible physical addresses; if the board has
> the vector table at address X but the user's ELF file loads data via
> a different address Y which is an alias to the same underlying guest
> RAM then rom_ptr() will not find it.
> 
> Handle the possibility of aliasing by iterating through the whole
> FlatView of the CPU's address space checking for other mappings of
> the MemoryRegion corresponding to the location of the vector table.
> If we find any aliases we use rom_ptr() to see if the ROM blob loader
> has any data there.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
>  target/arm/cpu.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 67 insertions(+), 1 deletion(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index ae04884408c..aac78ae6623 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -161,6 +161,72 @@ static void cp_reg_check_reset(gpointer key, gpointer value,  gpointer opaque)
>      assert(oldvalue == newvalue);
>  }
>  
> +#ifndef CONFIG_USER_ONLY
> +typedef struct FindRomCBData {
> +    size_t size; /* Amount of data we want from ROM, in bytes */
> +    MemoryRegion *mr; /* MR at the unaliased guest addr */
> +    hwaddr xlat; /* Offset of addr within mr */
> +    uint8_t *rom; /* Output: rom data pointer, if found */
> +} FindRomCBData;
> +
> +static int find_rom_cb(Int128 start, Int128 len, const MemoryRegion *mr,
> +                       hwaddr offset_in_region, void *opaque)

Return bool maybe?

> +{
> +    FindRomCBData *cbdata = opaque;
> +    hwaddr alias_addr;
> +
> +    if (mr != cbdata->mr) {
> +        return 0;
> +    }
> +
> +    alias_addr = int128_get64(start) + cbdata->xlat - offset_in_region;
> +    cbdata->rom = rom_ptr(alias_addr, cbdata->size);
> +    if (!cbdata->rom) {
> +        return 0;
> +    }
> +    /* Found a match, stop iterating */
> +    return 1;
> +}
> +
> +static uint8_t *find_rom_for_addr(AddressSpace *as, hwaddr addr, size_t size)
> +{
> +    /*
> +     * Find any ROM data for the given guest address range.  If there
> +     * is a ROM blob then return a pointer to the host memory
> +     * corresponding to 'addr'; otherwise return NULL.
> +     *
> +     * This is like rom_ptr(), except that it handles possible aliases
> +     * within the CPU's address space, so that we still find a ROM
> +     * blob even if it was loaded to an address that aliases addr
> +     * rather than to addr itself.
> +     */
> +    FlatView *fv;
> +    uint8_t *rom;
> +    hwaddr len_unused;
> +    FindRomCBData cbdata = {};
> +
> +    /* Easy case: there's data at the actual address */
> +    rom = rom_ptr(addr, size);
> +    if (rom) {
> +        return rom;
> +    }
> +
> +    RCU_READ_LOCK_GUARD();
> +
> +    fv = address_space_to_flatview(as);
> +    cbdata.mr = flatview_translate(fv, addr, &cbdata.xlat, &len_unused,
> +                                   false, MEMTXATTRS_UNSPECIFIED);
> +    if (!cbdata.mr) {
> +        /* Nothing at this address, so there can't be any aliasing */
> +        return NULL;
> +    }
> +
> +    cbdata.size = size;
> +    flatview_for_each_range(fv, find_rom_cb, &cbdata);
> +    return cbdata.rom;
> +}
> +#endif
> +
>  static void arm_cpu_reset(DeviceState *dev)
>  {
>      CPUState *s = CPU(dev);
> @@ -331,7 +397,7 @@ static void arm_cpu_reset(DeviceState *dev)
>  
>          /* Load the initial SP and PC from offset 0 and 4 in the vector table */
>          vecbase = env->v7m.vecbase[env->v7m.secure];
> -        rom = rom_ptr(vecbase, 8);
> +        rom = find_rom_for_addr(s->as, vecbase, 8);
>          if (rom) {
>              /* Address zero is covered by ROM which hasn't yet been
>               * copied into physical memory.
> 

Your solution seems generic, and the problem is not resticted
to Cortex-M:

$ git grep rom_ptr target
target/arm/cpu.c:334:        rom = rom_ptr(vecbase, 8);
target/rx/cpu.c:61:    resetvec = rom_ptr(0xfffffffc, 4);

Some thoughs:
- make find_rom_for_addr() generic ("hw/loader.h" again?)
- poison rom_ptr() under target/ to restrict it to hw/

Regards,

Phil.

Re: [PATCH for-6.0 2/2] target/arm: Make M-profile VTOR loads on reset handle memory aliasing

[Richard Henderson]

On 3/12/21 2:17 PM, Philippe Mathieu-Daudé wrote:
>> +static int find_rom_cb(Int128 start, Int128 len, const MemoryRegion *mr,
>> +                       hwaddr offset_in_region, void *opaque)
> Return bool maybe?
> 

Yes, it might be worth changing with the first patch, while we're changing all 
(one) user of flatview_for_each_range().

But he can't simply change it here -- must match flatview_cb type.


r~

Re: [PATCH for-6.0 2/2] target/arm: Make M-profile VTOR loads on reset handle memory aliasing

[Peter Maydell]

On Fri, 12 Mar 2021 at 20:17, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> Your solution seems generic, and the problem is not resticted
> to Cortex-M:
>
> $ git grep rom_ptr target
> target/arm/cpu.c:334:        rom = rom_ptr(vecbase, 8);
> target/rx/cpu.c:61:    resetvec = rom_ptr(0xfffffffc, 4);
>
> Some thoughs:
> - make find_rom_for_addr() generic ("hw/loader.h" again?)
> - poison rom_ptr() under target/ to restrict it to hw/

This problem isn't specific to code in target/, that's just
the particular case I care about. The uses in hw/ are mostly
in board init code, which (like cpu reset code) runs before
the rom blob reset function, and so wants to directly look
at the blob data. If a board has multiple aliases for a piece
of RAM then rom_ptr() will cause problems for code in hw/ too.
The only advantage that uses of rom_ptr() in hw/ have is that
they can know which board model they're running on and so either
(a) be sure that the address they're looking at doesn't have any
aliases elsewhere in the AddressSpace or (b) manually look at
every alias address in turn.

thanks
-- PMM

Re: [PATCH for-6.0 0/2] arm: Make M-profile VTOR loads on reset handle memory aliasing

[Peter Maydell]

On Fri, 12 Mar 2021 at 17:29, Peter Maydell <peter.maydell@linaro.org> wrote:
> This series handles the possibility of aliasing by iterating through
> the whole FlatView of the CPU's address space checking for other
> mappings of the MemoryRegion corresponding to the location of the
> vector table.  If we find any aliases we use rom_ptr() to see if the
> ROM blob loader has any data there.

The other possible place we could put this code would be
to put it into rom_ptr() itself. You'd have to change the
callsites to pass an AddressSpace to rom_ptr(), but really
we ought to do that anyway, because a Rom has an AddressSpace
that we should be checking as well as the address.

-- PMM

Re: [PATCH for-6.0 0/2] arm: Make M-profile VTOR loads on reset handle memory aliasing

[Richard Henderson]

On 3/12/21 12:59 PM, Peter Maydell wrote:
> On Fri, 12 Mar 2021 at 17:29, Peter Maydell <peter.maydell@linaro.org> wrote:
>> This series handles the possibility of aliasing by iterating through
>> the whole FlatView of the CPU's address space checking for other
>> mappings of the MemoryRegion corresponding to the location of the
>> vector table.  If we find any aliases we use rom_ptr() to see if the
>> ROM blob loader has any data there.
> 
> The other possible place we could put this code would be
> to put it into rom_ptr() itself. You'd have to change the
> callsites to pass an AddressSpace to rom_ptr(), but really
> we ought to do that anyway, because a Rom has an AddressSpace
> that we should be checking as well as the address.

I like this as the solution.


r~

Re: [PATCH for-6.0 0/2] arm: Make M-profile VTOR loads on reset handle memory aliasing

[Peter Maydell]

On Sat, 13 Mar 2021 at 19:05, Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 3/12/21 12:59 PM, Peter Maydell wrote:
> > On Fri, 12 Mar 2021 at 17:29, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> This series handles the possibility of aliasing by iterating through
> >> the whole FlatView of the CPU's address space checking for other
> >> mappings of the MemoryRegion corresponding to the location of the
> >> vector table.  If we find any aliases we use rom_ptr() to see if the
> >> ROM blob loader has any data there.
> >
> > The other possible place we could put this code would be
> > to put it into rom_ptr() itself. You'd have to change the
> > callsites to pass an AddressSpace to rom_ptr(), but really
> > we ought to do that anyway, because a Rom has an AddressSpace
> > that we should be checking as well as the address.
>
> I like this as the solution.

I realized that checking against the Rom's AddressSpace
is likely to be a bad plan, though -- in some cases the AS is
used to allow the loader to say "load this image to AS
such-and-such", and we don't want to fail to find the ROM
blob because the rom_ptr() code is reading it from a different
AS that is an alternate view onto the same actual MemoryRegions
(eg Secure vs NonSecure and the RAM is in the same place.)

thanks
-- PMM