RE: Discussion of Xenheap problems on AArch64

From: Henry Wang <Henry.Wang@arm.com>
To: Julien Grall <julien@xen.org>,
	"sstabellini@kernel.org" <sstabellini@kernel.org>,
	"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>
Cc: Wei Chen <Wei.Chen@arm.com>, Penny Zheng <Penny.Zheng@arm.com>,
	Bertrand Marquis <Bertrand.Marquis@arm.com>
Subject: RE: Discussion of Xenheap problems on AArch64
Date: Mon, 17 May 2021 06:38:05 +0000	[thread overview]
Message-ID: <PA4PR08MB6253AB1B1286086E4EDE60A2922D9@PA4PR08MB6253.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <7247122c-127d-705c-78a5-7f9460f5821a@xen.org>

> From: Julien Grall <julien@xen.org>
Hi Julien,

> Hi Henry,
> 
> >>>> [...]
> 
> Ah yes, we don't have a variant with _. I thought compiled test before
> sending it :(.

No worries :)

> 
> >
> > (XEN) ----------banks=2--------
> > (XEN) ----------start=80000000--------
> > (XEN) ----------size=7F000000--------
> > (XEN) ----------start=F900000000--------
> > (XEN) ----------size=80000000--------
> > (XEN) Checking for initrd in /chosen
> > (XEN) RAM: 0000000080000000 - 00000000feffffff
> > (XEN) RAM: 000000f900000000 - 000000f97fffffff
> > (XEN)
> > (XEN) MODULE[0]: 0000000084000000 - 00000000841464c8 Xen
> > (XEN) MODULE[1]: 00000000841464c8 - 0000000084148c9b Device Tree
> > (XEN) MODULE[2]: 0000000080080000 - 0000000081080000 Kernel
> > (XEN)  RESVD[0]: 0000000080000000 - 0000000080010000
> > (XEN)
> > (XEN) Command line: noreboot dom0_mem=1024M console=dtuart
> > dtuart=serial0 bootscrub=0
> > (XEN) PFN compression on bits 21...22
> > (XEN) init_boot_pages: ps 0000000080010000 pe 0000000080080000
> 
> The size of this region is 448MB.
> 
> > (XEN) init_boot_pages: ps 0000000081080000 pe 0000000084000000
> 
> The size of this region is 47MB.
> 
> > (XEN) init_boot_pages: ps 0000000084149000 pe 00000000ff000000
> 
> The size of this region is 1966MB.
> 
> 
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) init_boot_pages: ps 000000f900000000 pe 000000f980000000
> 
> The size of this region is 2048MB.
> 
> > (XEN) alloc_boot_pages: nr_pfns 909312 pfn_align 8192
> 
> This is asking for 3552MB of contiguous memory which cannot be
> accommodated. In any case, this is quite a large region to ask.
> 
> Same...
> 
> > (XEN) Xen BUG at page_alloc.c:436
> >
> > To compare with the maximum start address (f800000000) of second part
> mem
> > where xen boots correctly, I also attached the log for your information:
> >
> > (XEN) ----------banks=2--------
> > (XEN) ----------start=80000000--------
> > (XEN) ----------size=7F000000--------
> > (XEN) ----------start=F800000000--------
> > (XEN) ----------size=80000000--------
> > (XEN) Checking for initrd in /chosen
> > (XEN) RAM: 0000000080000000 - 00000000feffffff
> > (XEN) RAM: 000000f800000000 - 000000f87fffffff
> > (XEN)
> > (XEN) MODULE[0]: 0000000084000000 - 00000000841464c8 Xen
> > (XEN) MODULE[1]: 00000000841464c8 - 0000000084148c9b Device Tree
> > (XEN) MODULE[2]: 0000000080080000 - 0000000081080000 Kernel
> > (XEN)  RESVD[0]: 0000000080000000 - 0000000080010000
> > (XEN)
> > (XEN) Command line: noreboot dom0_mem=1024M console=dtuart
> > dtuart=serial0 bootscrub=0
> > (XEN) PFN compression on bits 20...22
> > (XEN) init_boot_pages: ps 0000000080010000 pe 0000000080080000
> > (XEN) init_boot_pages: ps 0000000081080000 pe 0000000084000000
> > (XEN) init_boot_pages: ps 0000000084149000 pe 00000000ff000000
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) alloc_boot_pages: nr_pfns 1 pfn_align 1
> > (XEN) init_boot_pages: ps 000000f800000000 pe 000000f880000000
> > (XEN) alloc_boot_pages: nr_pfns 450560 pfn_align 8192
> 
> ... here. We are trying to allocate a 1.5GB frametable. You have only
> 4GB of memory so the frametable should be a lot smaller (few tens of MB).
> 
> This is happening because PDX is not able to find many bits to compress.
> I am not sure we can compress more with the current PDX algorithm. This
> may require some extensive improvement to reduce the footprint.

Yes you are right, then I don't have any more questions. Thanks very much
for the detailed explanation.

> 
> On a previous e-mail, you said you tweaked the FVP model to set those
> regions. Were you trying to mimick the memory layout of a real HW
> (either current or future)?

Not really, I was just trying to cover as many cases as possible and these
regions were just picked for testing your patchset in different scenarios.

As the issue is related to the PDX algorithm instead of the heap allocation,
and the "allocating a big heap or two heap banks with a big gap" is tested,
I think this patchset is perfect ^^ Thank you.

Kind regards,

Henry

> 
> Cheers,
> 
> --
> Julien Grall