From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 677E8C5519F for ; Fri, 20 Nov 2020 10:42:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AD9CF22255 for ; Fri, 20 Nov 2020 10:42:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="r5FHE4v2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD9CF22255 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 16D9D6B0036; Fri, 20 Nov 2020 05:42:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 11E986B005C; Fri, 20 Nov 2020 05:42:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F27F86B005D; Fri, 20 Nov 2020 05:42:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id B7E016B0036 for ; Fri, 20 Nov 2020 05:42:06 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4D0BD1EF1 for ; Fri, 20 Nov 2020 10:42:06 +0000 (UTC) X-FDA: 77504456652.11.frame89_5e0f9512734a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 2FE8D180F8B86 for ; Fri, 20 Nov 2020 10:42:06 +0000 (UTC) X-HE-Tag: frame89_5e0f9512734a X-Filterd-Recvd-Size: 10883 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Fri, 20 Nov 2020 10:42:05 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id l11so4669414plt.1 for ; Fri, 20 Nov 2020 02:42:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=rhg0TalW++AIGVdey4/dqRvHpgC5hAbWxadUO7j/mIU=; b=r5FHE4v24Sxrrbc9fxF9l8xtJhYKEd4JoQk3+U7q+3VcrO8KnwnZuD3xOJkCuBj9dx Tyfbmh8/76YsmPtn3BuNDyNTg7cKLaaA5Wgd/JoA6kJ/3JFkMtFnSmakT7jaZLUsr1ou H9ePsxd/XmuYwJlA9jLJsRmhCUTPhIiZcyq0pFwSgzSo216XSv/k6cpG5kVu7hiUOsZQ gWNC1bihiZYOfeh0aw/NoAC6X0Jcs799g6NpKy1SwIcIrLYnmd/XX15n+4l/z1Liw2PX vSxgSiyJeOuKGhLWEx5yTFredAnfVzvhwg4sB5OYFUO9LO4haNJY96fuFcyOhpcusGfW FQtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rhg0TalW++AIGVdey4/dqRvHpgC5hAbWxadUO7j/mIU=; b=lGrSPNzt3+3qwDMyNbTsXMY0poZWwTIkUTiTgDA2eq376DdAfEiEsXlPxzfPLvaZIP DciC+0e7sE6CVJdcx/BV5FtMP8/qmhtJAY4DFaQVUL3D/wnB/E+VlQgbEQ1k3F4GN5Lz Tmvqmso8x619zJreJh71VYpAZBz051W4FAc3xBmLbSWbF3tWZpdKm6T11UlVJqXRsACu RPMCtLX4kWo7YsFRJVdT/oy640WiMondL8Q7KtbJth6pdHwU3WXccspVeCxeV1N9khwv 2wymvhSb8GR/DsKVtte1Pi0A0kCHUtZ2BykiFDQLKAH5/rQOItMlN105B4tzMQU8Dy09 oDnw== X-Gm-Message-State: AOAM533NKlge6Cz+B6751Pjhq1Xj5Irpf7KKpHBfeHAjYO0OXV94FCo6 ep5EAiF6tWmmISbq/ZSzgA2Og24mmoAEWUWxe92JAQ== X-Google-Smtp-Source: ABdhPJwo70Vau0/TmWAHNwiYzcDzsWluXhtwVyPNqw28cOzHlwgb/GntNZfg3wCuFXDIabBvMCwFFnm5T/4nAtmjV2c= X-Received: by 2002:a17:902:bb92:b029:d9:e9bf:b775 with SMTP id m18-20020a170902bb92b02900d9e9bfb775mr1682739pls.24.1605868924468; Fri, 20 Nov 2020 02:42:04 -0800 (PST) MIME-Version: 1.0 References: <20201120064325.34492-1-songmuchun@bytedance.com> <20201120064325.34492-19-songmuchun@bytedance.com> <20201120082358.GH3200@dhcp22.suse.cz> In-Reply-To: <20201120082358.GH3200@dhcp22.suse.cz> From: Muchun Song Date: Fri, 20 Nov 2020 18:41:22 +0800 Message-ID: Subject: Re: [External] Re: [PATCH v5 18/21] mm/hugetlb: Merge pte to huge pmd only for gigantic page To: Michal Hocko Cc: Jonathan Corbet , Mike Kravetz , Thomas Gleixner , mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, Peter Zijlstra , viro@zeniv.linux.org.uk, Andrew Morton , paulmck@kernel.org, mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com, Randy Dunlap , oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de, Mina Almasry , David Rientjes , Matthew Wilcox , Oscar Salvador , "Song Bao Hua (Barry Song)" , Xiongchun duan , linux-doc@vger.kernel.org, LKML , Linux Memory Management List , linux-fsdevel Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Nov 20, 2020 at 4:24 PM Michal Hocko wrote: > > On Fri 20-11-20 14:43:22, Muchun Song wrote: > > Merge pte to huge pmd if it has ever been split. Now only support > > gigantic page which's vmemmap pages size is an integer multiple of > > PMD_SIZE. This is the simplest case to handle. > > I think it would be benefitial for anybody who plan to implement this > for normal PMDs to document challenges while you still have them fresh > in your mind. Yeah, I agree with you. I will document it. > > > Signed-off-by: Muchun Song > > --- > > arch/x86/include/asm/hugetlb.h | 8 +++ > > mm/hugetlb_vmemmap.c | 118 ++++++++++++++++++++++++++++++++++++++++- > > 2 files changed, 124 insertions(+), 2 deletions(-) > > > > diff --git a/arch/x86/include/asm/hugetlb.h b/arch/x86/include/asm/hugetlb.h > > index c601fe042832..1de1c519a84a 100644 > > --- a/arch/x86/include/asm/hugetlb.h > > +++ b/arch/x86/include/asm/hugetlb.h > > @@ -12,6 +12,14 @@ static inline bool vmemmap_pmd_huge(pmd_t *pmd) > > { > > return pmd_large(*pmd); > > } > > + > > +#define vmemmap_pmd_mkhuge vmemmap_pmd_mkhuge > > +static inline pmd_t vmemmap_pmd_mkhuge(struct page *page) > > +{ > > + pte_t entry = pfn_pte(page_to_pfn(page), PAGE_KERNEL_LARGE); > > + > > + return __pmd(pte_val(entry)); > > +} > > #endif > > > > #define hugepages_supported() boot_cpu_has(X86_FEATURE_PSE) > > diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c > > index c958699d1393..bf2b6b3e75af 100644 > > --- a/mm/hugetlb_vmemmap.c > > +++ b/mm/hugetlb_vmemmap.c > > @@ -144,6 +144,14 @@ static inline bool vmemmap_pmd_huge(pmd_t *pmd) > > } > > #endif > > > > +#ifndef vmemmap_pmd_mkhuge > > +#define vmemmap_pmd_mkhuge vmemmap_pmd_mkhuge > > +static inline pmd_t vmemmap_pmd_mkhuge(struct page *page) > > +{ > > + return pmd_mkhuge(mk_pmd(page, PAGE_KERNEL)); > > +} > > +#endif > > + > > static bool hugetlb_free_vmemmap_disabled __initdata; > > > > static int __init early_hugetlb_free_vmemmap_param(char *buf) > > @@ -422,6 +430,104 @@ static void __remap_huge_page_pte_vmemmap(struct page *reuse, pte_t *ptep, > > } > > } > > > > +static void __replace_huge_page_pte_vmemmap(pte_t *ptep, unsigned long start, > > + unsigned int nr, struct page *huge, > > + struct list_head *free_pages) > > +{ > > + unsigned long addr; > > + unsigned long end = start + (nr << PAGE_SHIFT); > > + pgprot_t pgprot = PAGE_KERNEL; > > + > > + for (addr = start; addr < end; addr += PAGE_SIZE, ptep++) { > > + struct page *page; > > + pte_t old = *ptep; > > + pte_t entry; > > + > > + prepare_vmemmap_page(huge); > > + > > + entry = mk_pte(huge++, pgprot); > > + VM_WARN_ON(!pte_present(old)); > > + page = pte_page(old); > > + list_add(&page->lru, free_pages); > > + > > + set_pte_at(&init_mm, addr, ptep, entry); > > + } > > +} > > + > > +static void replace_huge_page_pmd_vmemmap(pmd_t *pmd, unsigned long start, > > + struct page *huge, > > + struct list_head *free_pages) > > +{ > > + unsigned long end = start + VMEMMAP_HPAGE_SIZE; > > + > > + flush_cache_vunmap(start, end); > > + __replace_huge_page_pte_vmemmap(pte_offset_kernel(pmd, start), start, > > + VMEMMAP_HPAGE_NR, huge, free_pages); > > + flush_tlb_kernel_range(start, end); > > +} > > + > > +static pte_t *merge_vmemmap_pte(pmd_t *pmdp, unsigned long addr) > > +{ > > + pte_t *pte; > > + struct page *page; > > + > > + pte = pte_offset_kernel(pmdp, addr); > > + page = pte_page(*pte); > > + set_pmd(pmdp, vmemmap_pmd_mkhuge(page)); > > + > > + return pte; > > +} > > + > > +static void merge_huge_page_pmd_vmemmap(pmd_t *pmd, unsigned long start, > > + struct page *huge, > > + struct list_head *free_pages) > > +{ > > + replace_huge_page_pmd_vmemmap(pmd, start, huge, free_pages); > > + pte_free_kernel(&init_mm, merge_vmemmap_pte(pmd, start)); > > + flush_tlb_kernel_range(start, start + VMEMMAP_HPAGE_SIZE); > > +} > > + > > +static inline void dissolve_compound_page(struct page *page, unsigned int order) > > +{ > > + int i; > > + unsigned int nr_pages = 1 << order; > > + > > + for (i = 1; i < nr_pages; i++) > > + set_page_count(page + i, 1); > > +} > > + > > +static void merge_gigantic_page_vmemmap(struct hstate *h, struct page *head, > > + pmd_t *pmd) > > +{ > > + LIST_HEAD(free_pages); > > + unsigned long addr = (unsigned long)head; > > + unsigned long end = addr + vmemmap_pages_size_per_hpage(h); > > + > > + for (; addr < end; addr += VMEMMAP_HPAGE_SIZE) { > > + void *to; > > + struct page *page; > > + > > + page = alloc_pages(GFP_VMEMMAP_PAGE & ~__GFP_NOFAIL, > > + VMEMMAP_HPAGE_ORDER); > > + if (!page) > > + goto out; > > + > > + dissolve_compound_page(page, VMEMMAP_HPAGE_ORDER); > > + to = page_to_virt(page); > > + memcpy(to, (void *)addr, VMEMMAP_HPAGE_SIZE); > > + > > + /* > > + * Make sure that any data that writes to the > > + * @to is made visible to the physical page. > > + */ > > + flush_kernel_vmap_range(to, VMEMMAP_HPAGE_SIZE); > > + > > + merge_huge_page_pmd_vmemmap(pmd++, addr, page, &free_pages); > > + } > > +out: > > + free_vmemmap_page_list(&free_pages); > > +} > > + > > static inline void alloc_vmemmap_pages(struct hstate *h, struct list_head *list) > > { > > int i; > > @@ -454,10 +560,18 @@ void alloc_huge_page_vmemmap(struct hstate *h, struct page *head) > > __remap_huge_page_pte_vmemmap); > > if (!freed_vmemmap_hpage_dec(pmd_page(*pmd)) && pmd_split(pmd)) { > > /* > > - * Todo: > > - * Merge pte to huge pmd if it has ever been split. > > + * Merge pte to huge pmd if it has ever been split. Now only > > + * support gigantic page which's vmemmap pages size is an > > + * integer multiple of PMD_SIZE. This is the simplest case > > + * to handle. > > */ > > clear_pmd_split(pmd); > > + > > + if (IS_ALIGNED(vmemmap_pages_per_hpage(h), VMEMMAP_HPAGE_NR)) { > > + spin_unlock(ptl); > > + merge_gigantic_page_vmemmap(h, head, pmd); > > + return; > > + } > > } > > spin_unlock(ptl); > > } > > -- > > 2.11.0 > > -- > Michal Hocko > SUSE Labs -- Yours, Muchun