Discussion of Xenheap problems on AArch64

[Henry Wang <Henry.Wang@arm.com>]

Hi,

We are trying to implement the static memory allocation on AArch64. Part of
this feature is the reserved heap memory allocation, where a specific range of
memory is reserved only for heap. In the development process, we found a
pitfall in current AArch64 setup_xenheap_mappings() function.

According to a previous discussion in community
https://lore.kernel.org/xen-devel/20190216134456.10681-1-peng.fan@nxp.com/,
on AArch64, bootmem is initialized after setup_xenheap_mappings(),
setup_xenheap_mappings() may try to allocate memory before memory has been
handed over to the boot allocator. If the reserved heap memory allocation is
introduced, either of below 2 cases will trigger a crash:

1. If the reserved heap memory is at the end of the memory block list and the
gap between reserved and unreserved memory is bigger than 512GB, when we setup
mappings from the beginning of the memory block list, we will get OOM caused
by lack of pages in boot allocator. This is because the memory that is reserved
for heap has not been mapped and added to the boot allocator.

2. If we add the memory that is reserved for heap to boot allocator first, and
then setup mappings for banks in the memory block list, we may get a page which
has not been setup mapping, causing a data abort.

Also, according to Julien's reply in previous mailing list discussion, we are
meant to support up to 5TB of RAM (see
https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00881.html).
Therefore, we think that maybe it is the time to re-visit this problem and try
to find a proper way to address it. Any comments?

Kind regards,

Henry