Re: [PATCH] hugetlbfs: add hugepages_node kernel parameter

[Mike Kravetz <mike.kravetz@oracle.com>]

On 8/22/21 3:27 PM, Matthew Wilcox wrote:
> On Sun, Aug 22, 2021 at 03:19:52PM -0700, Andrew Morton wrote:
>> On Fri, 20 Aug 2021 11:05:36 +0800 yaozhenguo <yaozhenguo1@gmail.com> wrote:
>>
>>> We can specify the number of hugepages to allocate at boot. But the
>>> hugepages is balanced in all nodes at present. In some scenarios,
>>> we only need hugepags in one node. For example: DPDK needs hugepages
>>> which is in the same node as NIC. if DPDK needs four hugepags of 1G
>>> size in node1 and system has 16 numa nodes. We must reserve 64 hugepags
>>> in kernel cmdline. But, only four hugepages is used. The others should
>>> be free after boot.If the system memory is low(for example: 64G), it will
>>> be an impossible task. So, add hugepages_node kernel parameter to specify
>>> node number of hugepages to allocate at boot.
>>> For example add following parameter:
>>>
>>> hugepagesz=1G hugepages_node=1 hugepages=4
>>>
>>> It will allocate 4 hugepags in node1 at boot.
>>
>> If were going to do this, shouldn't we permit more than one node?
>>
>> 	hugepages_nodes=1,2,5
> 
> I'd think we'd be better off expanding the definition of hugepages.
> eg:
> 
> hugepagesz=1G hugepages=1:4,3:8,5:2
> 
> would say to allocate 4 pages from node 1, 8 pages from node 3 and 2
> pages from node 5.

Thanks Matthew and Andrew!

I was trying to wrap my head around the big issue before making any
suggestions.  It is true that the desired functionality of allocating
huge pages from a specific node is lacking today.

I like the idea of expanding the definition of hugepages so that nodes
can be specified.

One word of caution.  It is easy to make mistakes when taking data
directly from the user on the command line.  For example, in the follow
on patch I do not believe node is not checked against MAX_NUMNODES so
it may write beyond the end of array.  Also, we need to think about what
the behavior should be if part of the
'node1:count1,node2:count2,node3:count3' string is invalid?  Suppose
node2 is invalid.  Do we still allocate from node1 and node3, or do we
just discard the entire string?  During a recent rewrite of hugetlb
command line processing, I had a matrix of different command line
options, order, values and expected results.  We need to be as thorough
when adding this new option.

I'll take a closer look at the proposed patch in the next few days.
-- 
Mike Kravetz