From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71DF6C433FE for ; Fri, 4 Feb 2022 20:22:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 600808D0014; Fri, 4 Feb 2022 15:21:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 584A68D0007; Fri, 4 Feb 2022 15:21:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DAB98D0014; Fri, 4 Feb 2022 15:21:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id 2307C8D0007 for ; Fri, 4 Feb 2022 15:21:45 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DB6BB18274D01 for ; Fri, 4 Feb 2022 20:21:44 +0000 (UTC) X-FDA: 79106218128.25.286236C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 5D3CA20005 for ; Fri, 4 Feb 2022 20:21:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=izsXeAyMwji7NT2VLQq4XArrQ6mR4M3kQTuXbNI6dG8=; b=ZY7NHv98WBbMI0thwkgk2JliNb F4bBX4Iy5U3U+keYuogOFrOTFaedNF6gjXA2c2xaPRd60W1XKeasU1JtXBpKWbzkowtaG9y+WpQIX QMlBu+Tf/eiXytaFbl2VpAkcZd4X0O9XP+deDyVFl+qRjgUy6b5tu5H8tNDraGI1zTEN/8IhzGfu7 3Nc1Qafbq/NP7txRPKfXMlr5f92d1oWXlP3Kr4wROMmk5EaJkJYsObyNU9BLaGPo9t8R23TsgEXG2 ouHLri1WFzWKXvqqd3lowwM+V9Jh+rRNPEiRuRUCJLreTaMxJ3zZkbX7yvocb8eetvJBK2SuP8Irz O9uWr1aQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nG4jY-007LoC-PU; Fri, 04 Feb 2022 19:59:04 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org Subject: [PATCH 49/75] mm/mlock: Turn munlock_vma_page() into munlock_vma_folio() Date: Fri, 4 Feb 2022 19:58:26 +0000 Message-Id: <20220204195852.1751729-50-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220204195852.1751729-1-willy@infradead.org> References: <20220204195852.1751729-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: nil X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5D3CA20005 X-Stat-Signature: py9tnuebrx8u3qgrj4sywpbwn741fb5t Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZY7NHv98; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-HE-Tag: 1644006104-925271 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add back munlock_vma_page() as a wrapper function. Saves a few calls to compound_head() and an assertion that the page is not a tail page. Signed-off-by: Matthew Wilcox (Oracle) --- mm/folio-compat.c | 5 +++ mm/internal.h | 3 +- mm/mlock.c | 86 +++++++++++++++++++++++------------------------ 3 files changed, 50 insertions(+), 44 deletions(-) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 90f03187a5e3..3804fd8c1f20 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -176,6 +176,11 @@ void mlock_vma_page(struct page *page) mlock_vma_folio(page_folio(page)); } =20 +unsigned long munlock_vma_page(struct page *page) +{ + return munlock_vma_folio(page_folio(page)); +} + void page_mlock(struct page *page) { folio_mlock(page_folio(page)); diff --git a/mm/internal.h b/mm/internal.h index 18b024aa7e59..66645972cbd7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -413,7 +413,8 @@ static inline void munlock_vma_pages_all(struct vm_ar= ea_struct *vma) */ void mlock_vma_page(struct page *page); void mlock_vma_folio(struct folio *folio); -extern unsigned int munlock_vma_page(struct page *page); +unsigned long munlock_vma_page(struct page *page); +unsigned long munlock_vma_folio(struct folio *folio); =20 extern int mlock_future_check(struct mm_struct *mm, unsigned long flags, unsigned long len); diff --git a/mm/mlock.c b/mm/mlock.c index d998fd5c84bf..f188038ef48e 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -115,82 +115,81 @@ void mlock_vma_folio(struct folio *folio) /* * Finish munlock after successful page isolation * - * Page must be locked. This is a wrapper for page_mlock() - * and putback_lru_page() with munlock accounting. + * Folio must be locked. This is a wrapper for folio_mlock() + * and folio_putback_lru() with munlock accounting. */ -static void __munlock_isolated_page(struct page *page) +static void __munlock_isolated_folio(struct folio *folio) { /* * Optimization: if the page was mapped just once, that's our mapping * and we don't need to check all the other vmas. */ - if (page_mapcount(page) > 1) - page_mlock(page); + /* XXX: should be folio_mapcount(), surely? */ + if (page_mapcount(&folio->page) > 1) + folio_mlock(folio); =20 /* Did try_to_unlock() succeed or punt? */ - if (!PageMlocked(page)) - count_vm_events(UNEVICTABLE_PGMUNLOCKED, thp_nr_pages(page)); + if (!folio_test_mlocked(folio)) + count_vm_events(UNEVICTABLE_PGMUNLOCKED, folio_nr_pages(folio)); =20 - putback_lru_page(page); + folio_putback_lru(folio); } =20 /* - * Accounting for page isolation fail during munlock + * Accounting for folio isolation fail during munlock * - * Performs accounting when page isolation fails in munlock. There is no= thing - * else to do because it means some other task has already removed the p= age - * from the LRU. putback_lru_page() will take care of removing the page = from + * Performs accounting when folio isolation fails in munlock. There is n= othing + * else to do because it means some other task has already removed the f= olio + * from the LRU. folio_putback_lru() will take care of removing the foli= o from * the unevictable list, if necessary. vmscan [folio_referenced()] will = move - * the page back to the unevictable list if some other vma has it mlocke= d. + * the folio back to the unevictable list if some other vma has it mlock= ed. */ -static void __munlock_isolation_failed(struct page *page) +static void __munlock_isolation_failed(struct folio *folio) { - int nr_pages =3D thp_nr_pages(page); + long nr_pages =3D folio_nr_pages(folio); =20 - if (PageUnevictable(page)) + if (folio_test_unevictable(folio)) __count_vm_events(UNEVICTABLE_PGSTRANDED, nr_pages); else __count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages); } =20 /** - * munlock_vma_page - munlock a vma page - * @page: page to be unlocked, either a normal page or THP page head + * munlock_vma_folio() - munlock a vma folio. + * @folio: Folio to be unlocked. * - * returns the size of the page as a page mask (0 for normal page, - * HPAGE_PMD_NR - 1 for THP head page) - * - * called from munlock()/munmap() path with page supposedly on the LRU. - * When we munlock a page, because the vma where we found the page is be= ing + * called from munlock()/munmap() path with folio supposedly on the LRU. + * When we munlock a folio, because the vma where we found the folio is = being * munlock()ed or munmap()ed, we want to check whether other vmas hold t= he - * page locked so that we can leave it on the unevictable lru list and n= ot - * bother vmscan with it. However, to walk the page's rmap list in - * page_mlock() we must isolate the page from the LRU. If some other - * task has removed the page from the LRU, we won't be able to do that. - * So we clear the PageMlocked as we might not get another chance. If w= e - * can't isolate the page, we leave it for putback_lru_page() and vmscan + * folio locked so that we can leave it on the unevictable lru list and = not + * bother vmscan with it. However, to walk the folio's rmap list in + * folio_mlock() we must isolate the folio from the LRU. If some other + * task has removed the folio from the LRU, we won't be able to do that. + * So we clear the folio mlocked flag as we might not get another chance= . If + * we can't isolate the folio, we leave it for folio_putback_lru() and v= mscan * [folio_referenced()/try_to_unmap()] to deal with. + * + * Return: The size of the folio as a page mask (2^order - 1). */ -unsigned int munlock_vma_page(struct page *page) +unsigned long munlock_vma_folio(struct folio *folio) { - int nr_pages; + long nr_pages; =20 - /* For page_mlock() and to serialize with page migration */ - BUG_ON(!PageLocked(page)); - VM_BUG_ON_PAGE(PageTail(page), page); + /* For folio_mlock() and to serialize with page migration */ + BUG_ON(!folio_test_locked(folio)); =20 - if (!TestClearPageMlocked(page)) { - /* Potentially, PTE-mapped THP: do not skip the rest PTEs */ + if (!folio_test_clear_mlocked(folio)) { + /* Potentially, PTE-mapped folio: do not skip the other PTEs */ return 0; } =20 - nr_pages =3D thp_nr_pages(page); - mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); + nr_pages =3D folio_nr_pages(folio); + zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages); =20 - if (!isolate_lru_page(page)) - __munlock_isolated_page(page); + if (!folio_isolate_lru(folio)) + __munlock_isolated_folio(folio); else - __munlock_isolation_failed(page); + __munlock_isolation_failed(folio); =20 return nr_pages - 1; } @@ -289,7 +288,7 @@ static void __munlock_pagevec(struct pagevec *pvec, s= truct zone *zone) del_page_from_lru_list(page, lruvec); continue; } else - __munlock_isolation_failed(page); + __munlock_isolation_failed(folio); } else { delta_munlocked++; } @@ -318,6 +317,7 @@ static void __munlock_pagevec(struct pagevec *pvec, s= truct zone *zone) struct page *page =3D pvec->pages[i]; =20 if (page) { + struct folio *folio =3D page_folio(page); lock_page(page); if (!__putback_lru_fast_prepare(page, &pvec_putback, &pgrescued)) { @@ -326,7 +326,7 @@ static void __munlock_pagevec(struct pagevec *pvec, s= truct zone *zone) * pin before unlock_page() */ get_page(page); /* for putback_lru_page() */ - __munlock_isolated_page(page); + __munlock_isolated_folio(folio); unlock_page(page); put_page(page); /* from follow_page_mask() */ } --=20 2.34.1