TARGET_SYS_HEAPINFO and Cortex-A15 memory map

TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

I'm experiencing some issues with the startup code for an emulated Cortex-a15 machine I plan to use for running unit-tests.

I'm starting QEMU with:

.../qemu-system-arm  "--machine" "virt" "--cpu" "cortex-a15" "--nographic" "-d" "unimp,guest_errors" "--semihosting-config" "enable=on,target=native,arg=sample-test,arg=one,arg=two" -s -S


At 0x0 I'm loading the application that uses the newlib semihosting library and startup.


The application starts and I can use GDB to step into the code from the very beginning. In crt0 the first thing I see is a call to SYS_HEAPINFO, followed by setting the heap and stack.

The values returned are:

0x04000000 - heap base
0x08000000 - heap limit
0x08000000 - stack base
0x0 - stack limit

This sets the SP at 0x08000000, which I'm not sure it is a valid memory address, since writes to it seem ineffective, and in the first function called, when it tries to pop registers from the stack, everything is zero, and the program jumps to 0x0.

I'm not very familiar with the Cortex-A15 memory map and initialisation; if the memory below 0x08000000 is indeed valid for a stack, probably I need to enable something more during the reset sequence, to make it writable.

Any suggestion?


Liviu

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

> On 2 Jun 2022, at 21:36, Liviu Ionescu <ilg@livius.net> wrote:
> 
> ... SYS_HEAPINFO...
> 
> 0x04000000 - heap base
> 0x08000000 - heap limit
> 0x08000000 - stack base
> 0x0 - stack limit

For Cortex-A72 I see similar values:

0x44000000
0x48000000
0x48000000
0x40000000

just that in this case the memory is writable, and the startup proceeds as expected.

Any idea why in the Cortex-A15 case the memory below 0x08000000 is not writable?


Liviu

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Peter Maydell]

On Fri, 3 Jun 2022 at 07:17, Liviu Ionescu <ilg@livius.net> wrote:
> > On 2 Jun 2022, at 21:36, Liviu Ionescu <ilg@livius.net> wrote:
> >
> > ... SYS_HEAPINFO...
> >
> > 0x04000000 - heap base
> > 0x08000000 - heap limit
> > 0x08000000 - stack base
> > 0x0 - stack limit
>
> For Cortex-A72 I see similar values:
>
> 0x44000000
> 0x48000000
> 0x48000000
> 0x40000000
>
> just that in this case the memory is writable, and the startup proceeds as expected.
>
> Any idea why in the Cortex-A15 case the memory below 0x08000000 is not writable?

Sorry I didn't get back to this before -- I marked it in my email client
but then forgot about it :-(

Anyway, the memory map for the 'virt' board doesn't change for the a72
versus the a15. In both cases, the memory from 0x0 to 0x08000000
is the flash memory. So QEMU is incorrect to have reported that as
the place to put the heap in SYS_HEAPINFO.

I'm not sure why the A72 gives different results, but the heuristic
that tries to figure out where the heap goes is basically trying to
do "find the biggest lump of memory which doesn't have some bit of
the guest code in it", so it's going to be sensitive to things like
just how big that guest binary happens to be and where it's loaded.
Perhaps that is why it's ended up different.

I suspect the fix to this bug is going to be "make sure that
SYS_HEAPINFO doesn't consider memory regions that are rom-devices"
(we already ignore read-only memory, but flash devices are
technically not read-only).

You don't mention which QEMU version you are using. We
rewrote the whole SYS_HEAPINFO handling in commit 5fc983af8b20d6927
(Feb 2022, should be in 7.0.0) because the old implementation
could produce some completely wrong results. Looking at the code
I guess this was with the new implementation, though.

Do you have a test binary I can reproduce this with? That would save
me a little time.

thanks
-- PMM

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

> On 25 Jul 2022, at 18:43, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... the memory map for the 'virt' board doesn't change for the a72
> versus the a15. In both cases, the memory from 0x0 to 0x08000000
> is the flash memory. So QEMU is incorrect to have reported that as
> the place to put the heap in SYS_HEAPINFO.

Could you also update the documentation pages to reflect the actual memory map for the Arm devices? It took me some time and experimentation to find this for my tests.

> I suspect the fix to this bug is going to be "make sure that
> SYS_HEAPINFO doesn't consider memory regions that are rom-devices"

Sounds ok.

> You don't mention which QEMU version you are using.

7.0.0

> Do you have a test binary I can reproduce this with? That would save
> me a little time.

I have a template project that can generate several binaries intended to run on QEMU:

- https://github.com/micro-os-plus/hello-world-qemu-template-xpack/

The only prerequisite is npm/xpm, the rest of the dependencies are handled automatically.

If this does not work for you, please tell me exactly which binary you need, and I can try to generate it.


Thank you,

Liviu

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Peter Maydell]

On Mon, 25 Jul 2022 at 16:56, Liviu Ionescu <ilg@livius.net> wrote:
>
>
>
> > On 25 Jul 2022, at 18:43, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > ... the memory map for the 'virt' board doesn't change for the a72
> > versus the a15. In both cases, the memory from 0x0 to 0x08000000
> > is the flash memory. So QEMU is incorrect to have reported that as
> > the place to put the heap in SYS_HEAPINFO.
>
> Could you also update the documentation pages to reflect the actual memory map for the Arm devices? It took me some time and experimentation to find this for my tests.

We document what the guest can assume about the virt board
memory layout here:

https://www.qemu.org/docs/master/system/arm/virt.html#hardware-configuration-information-for-bare-metal-programming

Flash at 0, RAM at 0x4000_0000, must look in the DTB blob
for all other information.

> > I suspect the fix to this bug is going to be "make sure that
> > SYS_HEAPINFO doesn't consider memory regions that are rom-devices"
>
> Sounds ok.
>
> > You don't mention which QEMU version you are using.
>
> 7.0.0
>
> > Do you have a test binary I can reproduce this with? That would save
> > me a little time.
>
> I have a template project that can generate several binaries intended to run on QEMU:
>
> - https://github.com/micro-os-plus/hello-world-qemu-template-xpack/
>
> The only prerequisite is npm/xpm, the rest of the dependencies are handled automatically.
>
> If this does not work for you, please tell me exactly which binary you need, and I can try to generate it.

The one where SYS_HEAPINFO produces the bogus result putting the
heap at 0x04000000, that you mentioned in the original report with
the command line

.../qemu-system-arm  "--machine" "virt" "--cpu" "cortex-a15"
"--nographic" "-d" "unimp,guest_errors" "--semihosting-config"
"enable=on,target=native,arg=sample-test,arg=one,arg=two" -s -S

thanks
-- PMM

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

> On 25 Jul 2022, at 19:02, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> 
> We document what the guest can assume about the virt board
> memory layout here:
> 
> https://www.qemu.org/docs/master/system/arm/virt.html#hardware-configuration-information-for-bare-metal-programming
> 
> Flash at 0, RAM at 0x4000_0000, must look in the DTB blob
> for all other information.

ah, ok, I probably missed this. :-(

btw, looking in the dtb blob is probably not a problem for a linux kernel, but if I want only to write a simple test that needs a timer, getting the timer address is probably more complicated than my entire test... :-(

> The one where SYS_HEAPINFO produces the bogus result putting the
> heap at 0x04000000, that you mentioned in the original report with
> the command line
> 
> .../qemu-system-arm "--machine" "virt" "--cpu" "cortex-a15"
> "--nographic" "-d" "unimp,guest_errors" "--semihosting-config"
> "enable=on,target=native,arg=sample-test,arg=one,arg=two" -s -S

ah, the bogus one... that's a bit more complicated, since it happened in the early tests, and I don't remember how I did it, it might be that I tried to load my code in flash and my data in ram, but I'm not sure.

try to check the logic and avoid the cases when flash addresses are returned for heap, if possible.

btw, this might be a different topic, but on Cortex-M devices I'm used to configure the linker scripts to allocate the text in flash and the data+bss in ram; for qemu aarch32/64 devices I could not make this work, and I had to allocate everything in ram, which is functional, but probably not very accurate for some tests, that might fail when running from flash.


regards,

Liviu

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Peter Maydell]

On Mon, 25 Jul 2022 at 18:02, Liviu Ionescu <ilg@livius.net> wrote:
> > On 25 Jul 2022, at 19:02, Peter Maydell <peter.maydell@linaro.org> wrote:
> > The one where SYS_HEAPINFO produces the bogus result putting the
> > heap at 0x04000000, that you mentioned in the original report with
> > the command line
> >
> > .../qemu-system-arm "--machine" "virt" "--cpu" "cortex-a15"
> > "--nographic" "-d" "unimp,guest_errors" "--semihosting-config"
> > "enable=on,target=native,arg=sample-test,arg=one,arg=two" -s -S
>
> ah, the bogus one... that's a bit more complicated, since it happened in the early tests, and I don't remember how I did it, it might be that I tried to load my code in flash and my data in ram, but I'm not sure.
>
> try to check the logic and avoid the cases when flash addresses are returned for heap, if possible.

Yeah, that's my plan. I think it'll be a one-line fix. But it'll
be much easier to be sure it works if I have a test case that
triggers the problem. I'll try a setup with code in flash and
data in RAM, that sounds like it might be a way to cause the
problem if there's not much code and a lot of data (because
then the remaining space in the flash is larger than the
remaining space in the RAM.)

> btw, this might be a different topic, but on Cortex-M devices
> I'm used to configure the linker scripts to allocate the text
> in flash and the data+bss in ram; for qemu aarch32/64 devices
> I could not make this work, and I had to allocate everything
> in ram, which is functional, but probably not very accurate
> for some tests, that might fail when running from flash.

I'm not a linker/toolchain expert, I'm afraid. Certainly in
theory it should be possible to have a split text/data setup
on A-profile devices, the same as M-profile ones.

thanks
-- PMM

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

> On 25 Jul 2022, at 20:33, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... I'll try a setup with code in flash and
> data in RAM, that sounds like it might be a way to cause the
> problem if there's not much code and a lot of data (because
> then the remaining space in the flash is larger than the
> remaining space in the RAM.)

My tests use several hundred KB of text (the C++ runtime) and a few tens of KB of bss+data; I don't know how to compute the remaining space, since I don't know the default sizes of flash & ram.

I do have some experience with linker scripts for Cortex-M devices, and I can tweak the projects generated by my template to allocate text in flash, but I have no experience with Cortex-A devices; if you tell me exactly at what address you want the program started, I can try to provide you a binary.


Liviu

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Peter Maydell]

On Mon, 25 Jul 2022 at 18:02, Liviu Ionescu <ilg@livius.net> wrote:
> > On 25 Jul 2022, at 19:02, Peter Maydell <peter.maydell@linaro.org> wrote:
> > The one where SYS_HEAPINFO produces the bogus result putting the
> > heap at 0x04000000, that you mentioned in the original report with
> > the command line
> >
> > .../qemu-system-arm "--machine" "virt" "--cpu" "cortex-a15"
> > "--nographic" "-d" "unimp,guest_errors" "--semihosting-config"
> > "enable=on,target=native,arg=sample-test,arg=one,arg=two" -s -S
>
> ah, the bogus one... that's a bit more complicated, since it happened in the early tests, and I don't remember how I did it, it might be that I tried to load my code in flash and my data in ram, but I'm not sure.
>
> try to check the logic and avoid the cases when flash addresses are returned for heap, if possible.

I've looked at the code, and done some stepping about in the debugger,
and as far as I can tell it should never be possible for the heap
to be returned as being in flash memory: the flash device MemoryRegions
are created with 'mr->ram' false, and the code in common_semi_find_bases()
ignores regions where mr->ram is false.

Clearly something is going wrong with your test binary, but it's
not the straightforward "we weren't ignoring the flash memory blocks"
that I thought it was, and so without a reproducible test case I
can't really do anything more with this :-(

If you're able to figure out what your original failing binary
setup was and repro the problem again, please send me the executable.

thanks
-- PMM

Re: TARGET_SYS_HEAPINFO and Cortex-A15 memory map

[Liviu Ionescu]

> On 26 Jul 2022, at 13:56, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... If you're able to figure out what your original failing binary
> setup was and repro the problem again, please send me the executable.

Thank you for investigating. 

I'll try to revert my projects to running from flash, and, if still having problems, keep the binaries for further analysis.


Liviu