From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F5F8C433EF for ; Fri, 4 Feb 2022 19:59:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 710386B0096; Fri, 4 Feb 2022 14:59:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E80BA6B007B; Fri, 4 Feb 2022 14:59:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E32F8D0009; Fri, 4 Feb 2022 14:59:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id 8F4E76B0093 for ; Fri, 4 Feb 2022 14:59:07 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3CBFA1815138D for ; Fri, 4 Feb 2022 19:59:07 +0000 (UTC) X-FDA: 79106161134.06.3399DD7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 8FF2D120003 for ; Fri, 4 Feb 2022 19:59:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=DYOK7Cv9XHLHWQbmW8cJx1Pne9VB46L3mBP+N/iRYRM=; b=hS9eGAQp5VnkH7lhSLNPJvAbwU w3ucXm7feVYCiGP7I1nY3Qn7Zc2IlB3dU+kJV/nyb1RqXNoaO0VEc7teXSNemYrvGdq5CN9D1rtrS dqbH9rv8xTHm8NFRDkR7fTkyD5y/QlQ6+5J0dCJAs8QMALhKxpAfAx7e/O35DqcL5LlxHuNYV3naF Q90rgGjvxIA92RZv/dKxBqIPxKLngXoHS8VJLdgTt7epm7wSdGxKdnxqILVPjorGtDAnFytK8GGOE cmgt2g4X3Kk5sXcPhHfIvWtr7pPhAZahtVYILGiYHSa1oVn23r218AtNon/VvYu04qSLWmmeceRaI oPkd97CA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nG4jZ-007LoY-3h; Fri, 04 Feb 2022 19:59:05 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org Subject: [PATCH 52/75] mm/rmap: Convert try_to_migrate() to folios Date: Fri, 4 Feb 2022 19:58:29 +0000 Message-Id: <20220204195852.1751729-53-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220204195852.1751729-1-willy@infradead.org> References: <20220204195852.1751729-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hS9eGAQp; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: nil X-Rspamd-Queue-Id: 8FF2D120003 X-Stat-Signature: gg7wa8notqu7qcfw9q1gikjt3rkr1pqf X-Rspamd-Server: rspam12 X-HE-Tag: 1644004746-164584 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the callers to pass a folio and the try_to_migrate_one() worker to use a folio throughout. Fixes an assumption that a folio must be <=3D PMD size. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 +- mm/huge_memory.c | 4 ++-- mm/migrate.c | 12 ++++++---- mm/rmap.c | 57 +++++++++++++++++++++++--------------------- 4 files changed, 41 insertions(+), 34 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 66407434c3b5..502439f20d88 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -192,7 +192,7 @@ static inline void page_dup_rmap(struct page *page, b= ool compound) int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags); =20 -void try_to_migrate(struct page *page, enum ttu_flags flags); +void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); =20 int make_device_exclusive_range(struct mm_struct *mm, unsigned long star= t, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4ea22b7319fd..21676a4afd07 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2294,8 +2294,8 @@ static void unmap_page(struct page *page) * pages can simply be left unmapped, then faulted back on demand. * If that is ever changed (perhaps for mlock), update remap_page(). */ - if (PageAnon(page)) - try_to_migrate(page, ttu_flags); + if (folio_test_anon(folio)) + try_to_migrate(folio, ttu_flags); else try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK); =20 diff --git a/mm/migrate.c b/mm/migrate.c index 766dc67874a1..5dcdd43d983d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -927,6 +927,7 @@ static int move_to_new_page(struct page *newpage, str= uct page *page, static int __unmap_and_move(struct page *page, struct page *newpage, int force, enum migrate_mode mode) { + struct folio *folio =3D page_folio(page); int rc =3D -EAGAIN; bool page_was_mapped =3D false; struct anon_vma *anon_vma =3D NULL; @@ -1030,7 +1031,7 @@ static int __unmap_and_move(struct page *page, stru= ct page *newpage, /* Establish migration ptes */ VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma, page); - try_to_migrate(page, 0); + try_to_migrate(folio, 0); page_was_mapped =3D true; } =20 @@ -1173,6 +1174,7 @@ static int unmap_and_move_huge_page(new_page_t get_= new_page, enum migrate_mode mode, int reason, struct list_head *ret) { + struct folio *src =3D page_folio(hpage); int rc =3D -EAGAIN; int page_was_mapped =3D 0; struct page *new_hpage; @@ -1249,7 +1251,7 @@ static int unmap_and_move_huge_page(new_page_t get_= new_page, ttu |=3D TTU_RMAP_LOCKED; } =20 - try_to_migrate(hpage, ttu); + try_to_migrate(src, ttu); page_was_mapped =3D 1; =20 if (mapping_locked) @@ -2449,6 +2451,7 @@ static void migrate_vma_unmap(struct migrate_vma *m= igrate) =20 for (i =3D 0; i < npages; i++) { struct page *page =3D migrate_pfn_to_page(migrate->src[i]); + struct folio *folio; =20 if (!page) continue; @@ -2472,8 +2475,9 @@ static void migrate_vma_unmap(struct migrate_vma *m= igrate) put_page(page); } =20 - if (page_mapped(page)) - try_to_migrate(page, 0); + folio =3D page_folio(page); + if (folio_mapped(folio)) + try_to_migrate(folio, 0); =20 if (page_mapped(page) || !migrate_vma_check_page(page)) { if (!is_zone_device_page(page)) { diff --git a/mm/rmap.c b/mm/rmap.c index c598fd667948..4cfac67e328c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1767,7 +1767,7 @@ static bool try_to_migrate_one(struct page *page, s= truct vm_area_struct *vma, range.end =3D vma_address_end(&pvmw); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, address, range.end); - if (PageHuge(page)) { + if (folio_test_hugetlb(folio)) { /* * If sharing is possible, start and end will be adjusted * accordingly. @@ -1781,21 +1781,24 @@ static bool try_to_migrate_one(struct page *page,= struct vm_area_struct *vma, #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION /* PMD-mapped THP migration entry */ if (!pvmw.pte) { - VM_BUG_ON_PAGE(PageHuge(page) || - !PageTransCompound(page), page); + subpage =3D folio_page(folio, + pmd_pfn(*pvmw.pmd) - folio_pfn(folio)); + VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || + !folio_test_pmd_mappable(folio), folio); =20 - set_pmd_migration_entry(&pvmw, page); + set_pmd_migration_entry(&pvmw, subpage); continue; } #endif =20 /* Unexpected PMD-mapped THP? */ - VM_BUG_ON_PAGE(!pvmw.pte, page); + VM_BUG_ON_FOLIO(!pvmw.pte, folio); =20 - subpage =3D page - page_to_pfn(page) + pte_pfn(*pvmw.pte); + subpage =3D folio_page(folio, + pte_pfn(*pvmw.pte) - folio_pfn(folio)); address =3D pvmw.address; =20 - if (PageHuge(page) && !PageAnon(page)) { + if (folio_test_hugetlb(folio) && !folio_test_anon(folio)) { /* * To call huge_pmd_unshare, i_mmap_rwsem must be * held in write mode. Caller needs to explicitly @@ -1833,15 +1836,15 @@ static bool try_to_migrate_one(struct page *page,= struct vm_area_struct *vma, flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); pteval =3D ptep_clear_flush(vma, address, pvmw.pte); =20 - /* Move the dirty bit to the page. Now the pte is gone. */ + /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pteval)) - set_page_dirty(page); + folio_mark_dirty(folio); =20 /* Update high watermark before we lower rss */ update_hiwater_rss(mm); =20 - if (is_zone_device_page(page)) { - unsigned long pfn =3D page_to_pfn(page); + if (folio_is_zone_device(folio)) { + unsigned long pfn =3D folio_pfn(folio); swp_entry_t entry; pte_t swp_pte; =20 @@ -1877,16 +1880,16 @@ static bool try_to_migrate_one(struct page *page,= struct vm_area_struct *vma, * changed when hugepage migrations to device private * memory are supported. */ - subpage =3D page; - } else if (PageHWPoison(page)) { + subpage =3D &folio->page; + } else if (PageHWPoison(subpage)) { pteval =3D swp_entry_to_pte(make_hwpoison_entry(subpage)); - if (PageHuge(page)) { - hugetlb_count_sub(compound_nr(page), mm); + if (folio_test_hugetlb(folio)) { + hugetlb_count_sub(folio_nr_pages(folio), mm); set_huge_swap_pte_at(mm, address, pvmw.pte, pteval, vma_mmu_pagesize(vma)); } else { - dec_mm_counter(mm, mm_counter(page)); + dec_mm_counter(mm, mm_counter(&folio->page)); set_pte_at(mm, address, pvmw.pte, pteval); } =20 @@ -1901,7 +1904,7 @@ static bool try_to_migrate_one(struct page *page, s= truct vm_area_struct *vma, * migration) will not expect userfaults on already * copied pages. */ - dec_mm_counter(mm, mm_counter(page)); + dec_mm_counter(mm, mm_counter(&folio->page)); /* We have to invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); @@ -1947,8 +1950,8 @@ static bool try_to_migrate_one(struct page *page, s= truct vm_area_struct *vma, * * See Documentation/vm/mmu_notifier.rst */ - page_remove_rmap(subpage, PageHuge(page)); - put_page(page); + page_remove_rmap(subpage, folio_test_hugetlb(folio)); + folio_put(folio); } =20 mmu_notifier_invalidate_range_end(&range); @@ -1958,13 +1961,13 @@ static bool try_to_migrate_one(struct page *page,= struct vm_area_struct *vma, =20 /** * try_to_migrate - try to replace all page table mappings with swap ent= ries - * @page: the page to replace page table entries for + * @folio: the folio to replace page table entries for * @flags: action and flags * - * Tries to remove all the page table entries which are mapping this pag= e and - * replace them with special swap entries. Caller must hold the page loc= k. + * Tries to remove all the page table entries which are mapping this fol= io and + * replace them with special swap entries. Caller must hold the folio lo= ck. */ -void try_to_migrate(struct page *page, enum ttu_flags flags) +void try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc =3D { .rmap_one =3D try_to_migrate_one, @@ -1981,7 +1984,7 @@ void try_to_migrate(struct page *page, enum ttu_fla= gs flags) TTU_SYNC))) return; =20 - if (is_zone_device_page(page) && !is_device_private_page(page)) + if (folio_is_zone_device(folio) && !folio_is_device_private(folio)) return; =20 /* @@ -1992,13 +1995,13 @@ void try_to_migrate(struct page *page, enum ttu_f= lags flags) * locking requirements of exec(), migration skips * temporary VMAs until after exec() completes. */ - if (!PageKsm(page) && PageAnon(page)) + if (!folio_test_ksm(folio) && folio_test_anon(folio)) rwc.invalid_vma =3D invalid_migration_vma; =20 if (flags & TTU_RMAP_LOCKED) - rmap_walk_locked(page, &rwc); + rmap_walk_locked(&folio->page, &rwc); else - rmap_walk(page, &rwc); + rmap_walk(&folio->page, &rwc); } =20 /* --=20 2.34.1