Re: [PATCH DO NOT APPLY] docs: Document allocator properties and the rubric for using them

[George Dunlap <George.Dunlap@citrix.com>]

> On Mar 12, 2021, at 2:32 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> 
> 
> 
>> On Feb 16, 2021, at 3:29 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> 
>> On 16.02.2021 11:28, George Dunlap wrote:
>>> --- /dev/null
>>> +++ b/docs/hypervisor-guide/memory-allocation-functions.rst
>>> @@ -0,0 +1,118 @@
>>> +.. SPDX-License-Identifier: CC-BY-4.0
>>> +
>>> +Xenheap memory allocation functions
>>> +===================================
>>> +
>>> +In general Xen contains two pools (or "heaps") of memory: the *xen
>>> +heap* and the *dom heap*.  Please see the comment at the top of
>>> +``xen/common/page_alloc.c`` for the canonical explanation.
>>> +
>>> +This document describes the various functions available to allocate
>>> +memory from the xen heap: their properties and rules for when they should be
>>> +used.
>> 
>> Irrespective of your subsequent indication of you disliking the
>> proposal (which I understand only affects the guidelines further
>> down anyway) I'd like to point out that vmalloc() does not
>> allocate from the Xen heap. Therefore a benefit of always
>> recommending use of xvmalloc() would be that the function could
>> fall back to vmalloc() (and hence the larger domain heap) when
>> xmalloc() failed.
> 
> OK, that’s good to know.
> 
> So just trying to think this through: address space is limiting factor for how big the xenheap can be, right?  Presumably “vmap” space is also limited, and will be much smaller?  So in a sense the “fallback” is less about getting more memory, but about using up that extra little bit of virtual address space?
> 
> Another question that raises:  Are there times when it’s advantageous to specify which heap to allocate from?  If there are good reasons for allocations to be in the xenheap or in the domheap / vmap area, then the guidelines should probably say that as well.
> 
> And, of course, will the whole concept of the xenheap / domheap split go away if we ever get rid of the 1:1 map?
> 
>> 
>>> +TLDR guidelines
>>> +---------------
>>> +
>>> +* By default, ``xvmalloc`` (or its helper cognates) should be used
>>> +  unless you know you have specific properties that need to be met.
>>> +
>>> +* If you need memory which needs to be physically contiguous, and may
>>> +  be larger than ``PAGE_SIZE``...
>>> +  
>>> +  - ...and is order 2, use ``alloc_xenheap_pages``.
>>> +    
>>> +  - ...and is not order 2, use ``xmalloc`` (or its helper cognates)..
>> 
>> ITYM "an exact power of 2 number of pages"?
> 
> Yes, I’ll fix that.
> 
>> 
>>> +* If you don't need memory to be physically contiguous, and know the
>>> +  allocation will always be larger than ``PAGE_SIZE``, you may use
>>> +  ``vmalloc`` (or one of its helper cognates).
>>> +
>>> +* If you know that allocation will always be less than ``PAGE_SIZE``,
>>> +  you may use ``xmalloc``.
>> 
>> As per Julien's and your own replies, this wants to be "minimum
>> possible page size", which of course depends on where in the
>> tree the piece of code is to live. (It would be "maximum
>> possible page size" in the earlier paragraph.)
> 
> I’ll see if I can clarify this.

I think the only way to actually make this clear would be to set specific values for “minimum possible PAGE_SIZE” and “maximum possible PAGE_SIZE” — values past which the maintainers of the architecture are happy to do some sort of audit if PAGE_SIZE ever exceeds them.

 -George