From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB7DBC433DB for ; Fri, 19 Mar 2021 13:20:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5359764EC0 for ; Fri, 19 Mar 2021 13:20:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5359764EC0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EB0278D0001; Fri, 19 Mar 2021 09:20:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5FC06B0075; Fri, 19 Mar 2021 09:20:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEC428D0001; Fri, 19 Mar 2021 09:20:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id B1CE26B0074 for ; Fri, 19 Mar 2021 09:20:40 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 74B7C181AC9CC for ; Fri, 19 Mar 2021 13:20:40 +0000 (UTC) X-FDA: 77936683440.22.0E9FFAD Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf11.hostedemail.com (Postfix) with ESMTP id D051420007C7 for ; Fri, 19 Mar 2021 13:20:31 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 92222AE72; Fri, 19 Mar 2021 13:20:20 +0000 (UTC) From: Oscar Salvador To: Andrew Morton Cc: Mike Kravetz , Vlastimil Babka , David Hildenbrand , Michal Hocko , Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador , Michal Hocko Subject: [PATCH v6 4/5] mm: Make alloc_contig_range handle in-use hugetlb pages Date: Fri, 19 Mar 2021 14:20:03 +0100 Message-Id: <20210319132004.4341-5-osalvador@suse.de> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20210319132004.4341-1-osalvador@suse.de> References: <20210319132004.4341-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D051420007C7 X-Stat-Signature: oii58ba58yqcnnwia7ks11u4jreh831q Received-SPF: none (suse.de>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1616160031-360769 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() will fail if it finds a HugeTLB page within the rang= e, without a chance to handle them. Since HugeTLB pages can be migrated as a= ny LRU or Movable page, it does not make sense to bail out without trying. Enable the interface to recognize in-use HugeTLB pages so we can migrate them, and have much better chances to succeed the call. Signed-off-by: Oscar Salvador Reviewed-by: Mike Kravetz Acked-by: Michal Hocko --- include/linux/hugetlb.h | 5 +++-- mm/compaction.c | 12 +++++++++++- mm/hugetlb.c | 22 ++++++++++++++++------ mm/vmscan.c | 5 +++-- 4 files changed, 33 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index bcff86ca616f..a37b4ce86e58 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -583,7 +583,7 @@ struct huge_bootmem_page { struct hstate *hstate; }; =20 -int isolate_or_dissolve_huge_page(struct page *page); +int isolate_or_dissolve_huge_page(struct page *page, struct list_head *l= ist); struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_ni= d, @@ -866,7 +866,8 @@ static inline void huge_ptep_modify_prot_commit(struc= t vm_area_struct *vma, #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; =20 -static inline int isolate_or_dissolve_huge_page(struct page *page) +static inline int isolate_or_dissolve_huge_page(struct page *page, + struct list_head *list) { return -ENOMEM; } diff --git a/mm/compaction.c b/mm/compaction.c index 3b26c5c6c931..4143d1984a21 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -909,7 +909,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, } =20 if (PageHuge(page) && cc->alloc_contig) { - ret =3D isolate_or_dissolve_huge_page(page); + ret =3D isolate_or_dissolve_huge_page(page, &cc->migratepages); =20 /* * Fail isolation in case isolate_or_dissolve_huge_page @@ -926,6 +926,15 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, goto isolate_fail; } =20 + if (PageHuge(page)) { + /* + * Hugepage was successfully isolated and placed + * on the cc->migratepages list. + */ + low_pfn +=3D compound_nr(page) - 1; + goto isolate_success_no_list; + } + /* * Ok, the hugepage was dissolved. Now these pages are * Buddy and cannot be re-allocated because they are @@ -1067,6 +1076,7 @@ isolate_migratepages_block(struct compact_control *= cc, unsigned long low_pfn, =20 isolate_success: list_add(&page->lru, &cc->migratepages); +isolate_success_no_list: cc->nr_migratepages +=3D compound_nr(page); nr_isolated +=3D compound_nr(page); =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3194c1bd9e32..85e4bedbd5ac 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2257,7 +2257,8 @@ static void restore_reserve_on_error(struct hstate = *h, * Returns 0 on success, otherwise negated error. */ =20 -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *o= ld_page) +static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *o= ld_page, + struct list_head *list) { gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; int nid =3D page_to_nid(old_page); @@ -2287,11 +2288,14 @@ static int alloc_and_dissolve_huge_page(struct hs= tate *h, struct page *old_page) goto unlock; } else if (page_count(old_page)) { /* - * Someone has grabbed the page, fail for now. + * Someone has grabbed the page, try to isolate it here. + * Fail with -EBUSY if not possible. */ - ret =3D -EBUSY; update_and_free_page(h, new_page); - goto unlock; + spin_unlock(&hugetlb_lock); + if (!isolate_huge_page(old_page, list)) + ret =3D -EBUSY; + return ret; } else if (!HPageFreed(old_page)) { /* * Page's refcount is 0 but it has not been enqueued in the @@ -2319,10 +2323,11 @@ static int alloc_and_dissolve_huge_page(struct hs= tate *h, struct page *old_page) return ret; } =20 -int isolate_or_dissolve_huge_page(struct page *page) +int isolate_or_dissolve_huge_page(struct page *page, struct list_head *l= ist) { struct hstate *h; struct page *head; + int ret =3D -EBUSY; =20 /* * The page might have been dissolved from under our feet, so make sure @@ -2347,7 +2352,12 @@ int isolate_or_dissolve_huge_page(struct page *pag= e) if (hstate_is_gigantic(h)) return -ENOMEM; =20 - return alloc_and_dissolve_huge_page(h, head); + if (page_count(head) && isolate_huge_page(head, list)) + ret =3D 0; + else if (!page_count(head)) + ret =3D alloc_and_dissolve_huge_page(h, head, list); + + return ret; } =20 struct page *alloc_huge_page(struct vm_area_struct *vma, diff --git a/mm/vmscan.c b/mm/vmscan.c index 562e87cbd7a1..42aaef30633e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1507,8 +1507,9 @@ unsigned int reclaim_clean_pages_from_list(struct z= one *zone, LIST_HEAD(clean_pages); =20 list_for_each_entry_safe(page, next, page_list, lru) { - if (page_is_file_lru(page) && !PageDirty(page) && - !__PageMovable(page) && !PageUnevictable(page)) { + if (!PageHuge(page) && page_is_file_lru(page) && + !PageDirty(page) && !__PageMovable(page) && + !PageUnevictable(page)) { ClearPageActive(page); list_move(&page->lru, &clean_pages); } --=20 2.16.3