Re: [PATCH v9 02/11] mm/hugetlb: Introduce a new config HUGETLB_PAGE_FREE_VMEMMAP

From: Mike Kravetz <mike.kravetz@oracle.com>
To: Muchun Song <songmuchun@bytedance.com>,
	corbet@lwn.net, tglx@linutronix.de, mingo@redhat.com,
	bp@alien8.de, x86@kernel.org, hpa@zytor.com,
	dave.hansen@linux.intel.com, luto@kernel.org,
	peterz@infradead.org, viro@zeniv.linux.org.uk,
	akpm@linux-foundation.org, paulmck@kernel.org,
	mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com,
	rdunlap@infradead.org, oneukum@suse.com,
	anshuman.khandual@arm.com, jroedel@suse.de,
	almasrymina@google.com, rientjes@google.com, willy@infradead.org,
	osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com,
	david@redhat.com
Cc: duanxiongchun@bytedance.com, linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH v9 02/11] mm/hugetlb: Introduce a new config HUGETLB_PAGE_FREE_VMEMMAP
Date: Tue, 15 Dec 2020 19:45:28 -0800	[thread overview]
Message-ID: <e9abb112-7654-6157-6782-9ccb4a9cd87e@oracle.com> (raw)
In-Reply-To: <7cfe44aa-3753-82d9-6630-194f1532e186@oracle.com>

On 12/15/20 5:03 PM, Mike Kravetz wrote:
> On 12/13/20 7:45 AM, Muchun Song wrote:
>> diff --git a/fs/Kconfig b/fs/Kconfig
>> index 976e8b9033c4..4c3a9c614983 100644
>> --- a/fs/Kconfig
>> +++ b/fs/Kconfig
>> @@ -245,6 +245,21 @@ config HUGETLBFS
>>  config HUGETLB_PAGE
>>  	def_bool HUGETLBFS
>>  
>> +config HUGETLB_PAGE_FREE_VMEMMAP
>> +	def_bool HUGETLB_PAGE
>> +	depends on X86_64
>> +	depends on SPARSEMEM_VMEMMAP
>> +	depends on HAVE_BOOTMEM_INFO_NODE
>> +	help
>> +	  When using HUGETLB_PAGE_FREE_VMEMMAP, the system can save up some
>> +	  memory from pre-allocated HugeTLB pages when they are not used.
>> +	  6 pages per HugeTLB page of the pmd level mapping and (PAGE_SIZE - 2)
>> +	  pages per HugeTLB page of the pud level mapping.
>> +
>> +	  When the pages are going to be used or freed up, the vmemmap array
>> +	  representing that range needs to be remapped again and the pages
>> +	  we discarded earlier need to be rellocated again.
> 
> I see the previous discussion with David about wording here.  How about
> leaving the functionality description general, and provide a specific
> example for x86_64?  As mentioned we can always update when new arch support
> is added.  Suggested text?
> 
> 	The option HUGETLB_PAGE_FREE_VMEMMAP allows for the freeing of
> 	some vmemmap pages associated with pre-allocated HugeTLB pages.
> 	For example, on X86_64 6 vmemmap pages of size 4KB each can be
> 	saved for each 2MB HugeTLB page.  4094 vmemmap pages of size 4KB
> 	each can be saved for each 1GB HugeTLB page.
> 
> 	When a HugeTLB page is allocated or freed, the vmemmap array
> 	representing the range associated with the page will need to be
> 	remapped.  When a page is allocated, vmemmap pages are freed
> 	after remapping.  When a page is freed, previously discarded
> 	vmemmap pages must be allocated before before remapping.

Sorry, I am slowly coming up to speed with discussions when I was away.

It appears vmemmap is not being mapped with huge pages if the boot option
hugetlb_free_vmemmap is on.   Is that correct?

If that is correct, we should document the trade off of increased page
table pages needed to map vmemmap vs the savings from freeing struct page
pages.  If a user/sysadmin only uses a small number of hugetlb pages (as
a percentage of system memory) they could end up using more memory with
hugetlb_free_vmemmap on as opposed to off.  Perhaps, it should be part of
the documentation for hugetlb_free_vmemmap?  If this is true, and people
think this should be documented, I can try to come up with something.

-- 
Mike Kravetz