Re: Discussion of Xenheap problems on AArch64

[Julien Grall <julien@xen.org>]

On 21/04/2021 10:32, Henry Wang wrote:
> Hi Julien,

Hi Henry,

>> -----Original Message-----
>> From: Julien Grall <julien@xen.org>
>> Sent: Wednesday, April 21, 2021 5:04 PM
>> To: Henry Wang <Henry.Wang@arm.com>; sstabellini@kernel.org; xen-
>> devel@lists.xenproject.org
>> Cc: Wei Chen <Wei.Chen@arm.com>; Penny Zheng
>> <Penny.Zheng@arm.com>; Bertrand Marquis <Bertrand.Marquis@arm.com>
>> Subject: Re: Discussion of Xenheap problems on AArch64
>>
>>
>>
>> On 21/04/2021 07:28, Henry Wang wrote:
>>> Hi,
>>
>> Hi Henry,
>>
>>>
>>> We are trying to implement the static memory allocation on AArch64. Part
>> of
>>> this feature is the reserved heap memory allocation, where a specific range
>> of
>>> memory is reserved only for heap. In the development process, we found a
>>> pitfall in current AArch64 setup_xenheap_mappings() function.
>>>
>>> According to a previous discussion in community
>>> https://lore.kernel.org/xen-devel/20190216134456.10681-1-
>> peng.fan@nxp.com/,
>>> on AArch64, bootmem is initialized after setup_xenheap_mappings(),
>>> setup_xenheap_mappings() may try to allocate memory before memory
>> has been
>>> handed over to the boot allocator. If the reserved heap memory allocation
>> is
>>> introduced, either of below 2 cases will trigger a crash:
>>>
>>> 1. If the reserved heap memory is at the end of the memory block list and
>> the
>>> gap between reserved and unreserved memory is bigger than 512GB, when
>> we setup
>>> mappings from the beginning of the memory block list, we will get OOM
>> caused
>>> by lack of pages in boot allocator. This is because the memory that is
>> reserved
>>> for heap has not been mapped and added to the boot allocator.
>>>
>>> 2. If we add the memory that is reserved for heap to boot allocator first,
>> and
>>> then setup mappings for banks in the memory block list, we may get a page
>> which
>>> has not been setup mapping, causing a data abort.
>>
>> There are a few issues with setup_xenheap_mappings(). I have been
>> reworking the code on my spare time and started to upstream bits of it.
>> A PoC can be found here:
>>
>> https://xenbits.xen.org/gitweb/?p=people/julieng/xen-
>> unstable.git;a=shortlog;h=refs/heads/pt/dev
>>
> 
> Really great news! Thanks you very much for the information and your hard
> work on the PoC :) I will start to go through your PoC code then.

I spent sometimes today to clean-up the PoC and sent a series on the ML 
(see [1]). This has been lightly tested so far.

Would you be able to give a try and let me know if it helps your problem?

For convenience, I have pushed a branch with the series applied here:

https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/pt/rfc-v2

Cheers,

[1] https://lore.kernel.org/xen-devel/20210425201318.15447-1-julien@xen.org/

-- 
Julien Grall