From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A0AAC433EF for ; Fri, 4 Feb 2022 19:59:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8717A8D0007; Fri, 4 Feb 2022 14:59:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F3028D0011; Fri, 4 Feb 2022 14:59:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF6586B0099; Fri, 4 Feb 2022 14:59:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0002.hostedemail.com [216.40.44.2]) by kanga.kvack.org (Postfix) with ESMTP id 7778D8D000C for ; Fri, 4 Feb 2022 14:59:08 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6C3ED96F2E for ; Fri, 4 Feb 2022 19:59:07 +0000 (UTC) X-FDA: 79106161134.16.8D3FB10 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id DAE921A0004 for ; Fri, 4 Feb 2022 19:59:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=upR0ccIncisj2dvsJYuu3dh0MkD7JYPSdEAM0Zbapz8=; b=C6qzo6CB9W6zST1Qhsd3lqRrFl OCesn6efWlnqF0SOdqWqb1n5FKyD8Pm0H+m5ggMZjMaUBM5sm7lbB+oTVAZty+mrKbp0slsSndu2C Zk6iivWxpM9UG8OuOYFiaEhFJLf2ZU/zGVG0m0dtZjouTRZg8Y9X+UF+sZxnrEfiuGcSsMO9xknqL ZqpJUCGxqDxUl0zVUtR11ooLl99PFy7I8bnhr/Grw9j4871cnWOo2tHivhCPMWzQ4u44/8lDJOoPk mUbPDaceheuOMWBrc89Z5jXqQgEANp5umafClAqowgNgC1LoPsUssZdEEcq7skDU1P9YXg3fwkqGu rf5kvqOQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nG4jZ-007Lp0-FF; Fri, 04 Feb 2022 19:59:05 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org Subject: [PATCH 54/75] mm/migrate: Convert remove_migration_ptes() to folios Date: Fri, 4 Feb 2022 19:58:31 +0000 Message-Id: <20220204195852.1751729-55-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220204195852.1751729-1-willy@infradead.org> References: <20220204195852.1751729-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DAE921A0004 X-Stat-Signature: c4gwcky4es8pok6qysjkyfz1wq6ae4ui Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C6qzo6CB; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: nil X-HE-Tag: 1644004746-549610 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the implementation and all callers. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 +- mm/huge_memory.c | 24 +++++++------- mm/migrate.c | 74 +++++++++++++++++++++++++------------------- 3 files changed, 56 insertions(+), 44 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 502439f20d88..85d17a38642c 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -263,7 +263,7 @@ int folio_mkclean(struct folio *); void page_mlock(struct page *page); void folio_mlock(struct folio *folio); =20 -void remove_migration_ptes(struct page *old, struct page *new, bool lock= ed); +void remove_migration_ptes(struct folio *src, struct folio *dst, bool lo= cked); =20 /* * Called by memory-failure.c to kill processes. diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 21676a4afd07..7a0f4aaf7838 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2302,18 +2302,19 @@ static void unmap_page(struct page *page) VM_WARN_ON_ONCE_PAGE(page_mapped(page), page); } =20 -static void remap_page(struct page *page, unsigned int nr) +static void remap_page(struct folio *folio, unsigned long nr) { - int i; + int i =3D 0; =20 /* If unmap_page() uses try_to_migrate() on file, remove this check */ - if (!PageAnon(page)) + if (!folio_test_anon(folio)) return; - if (PageTransHuge(page)) { - remove_migration_ptes(page, page, true); - } else { - for (i =3D 0; i < nr; i++) - remove_migration_ptes(page + i, page + i, true); + for (;;) { + remove_migration_ptes(folio, folio, true); + i +=3D folio_nr_pages(folio); + if (i >=3D nr) + break; + folio =3D folio_next(folio); } } =20 @@ -2470,7 +2471,7 @@ static void __split_huge_page(struct page *page, st= ruct list_head *list, } local_irq_enable(); =20 - remap_page(head, nr); + remap_page(folio, nr); =20 if (PageSwapCache(head)) { swp_entry_t entry =3D { .val =3D page_private(head) }; @@ -2579,7 +2580,8 @@ bool can_split_huge_page(struct page *page, int *pe= xtra_pins) */ int split_huge_page_to_list(struct page *page, struct list_head *list) { - struct page *head =3D compound_head(page); + struct folio *folio =3D page_folio(page); + struct page *head =3D &folio->page; struct deferred_split *ds_queue =3D get_deferred_split_queue(head); XA_STATE(xas, &head->mapping->i_pages, head->index); struct anon_vma *anon_vma =3D NULL; @@ -2696,7 +2698,7 @@ int split_huge_page_to_list(struct page *page, stru= ct list_head *list) if (mapping) xas_unlock(&xas); local_irq_enable(); - remap_page(head, thp_nr_pages(head)); + remap_page(folio, folio_nr_pages(folio)); ret =3D -EBUSY; } =20 diff --git a/mm/migrate.c b/mm/migrate.c index 5dcdd43d983d..4daa8298c79a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -176,34 +176,36 @@ void putback_movable_pages(struct list_head *l) static bool remove_migration_pte(struct page *page, struct vm_area_struc= t *vma, unsigned long addr, void *old) { + struct folio *folio =3D page_folio(page); struct page_vma_mapped_walk pvmw =3D { .vma =3D vma, .address =3D addr, .flags =3D PVMW_SYNC | PVMW_MIGRATION, }; - struct page *new; - pte_t pte; - swp_entry_t entry; =20 VM_BUG_ON_PAGE(PageTail(page), page); pvmw_set_page(&pvmw, old); while (page_vma_mapped_walk(&pvmw)) { - if (PageKsm(page)) - new =3D page; - else - new =3D page - pvmw.pgoff + - linear_page_index(vma, pvmw.address); + pte_t pte; + swp_entry_t entry; + struct page *new; + unsigned long idx =3D 0; + + if (!folio_test_ksm(folio)) + idx =3D linear_page_index(vma, pvmw.address) - pvmw.pgoff; + new =3D folio_page(folio, idx); =20 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION /* PMD-mapped THP migration entry */ if (!pvmw.pte) { - VM_BUG_ON_PAGE(PageHuge(page) || !PageTransCompound(page), page); + VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || + !folio_test_pmd_mappable(folio), folio); remove_migration_pmd(&pvmw, new); continue; } #endif =20 - get_page(new); + folio_get(folio); pte =3D pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); if (pte_swp_soft_dirty(*pvmw.pte)) pte =3D pte_mksoft_dirty(pte); @@ -232,12 +234,12 @@ static bool remove_migration_pte(struct page *page,= struct vm_area_struct *vma, } =20 #ifdef CONFIG_HUGETLB_PAGE - if (PageHuge(new)) { + if (folio_test_hugetlb(folio)) { unsigned int shift =3D huge_page_shift(hstate_vma(vma)); =20 pte =3D pte_mkhuge(pte); pte =3D arch_make_huge_pte(pte, shift, vma->vm_flags); - if (PageAnon(new)) + if (folio_test_anon(folio)) hugepage_add_anon_rmap(new, vma, pvmw.address); else page_dup_rmap(new, true); @@ -245,17 +247,17 @@ static bool remove_migration_pte(struct page *page,= struct vm_area_struct *vma, } else #endif { - if (PageAnon(new)) + if (folio_test_anon(folio)) page_add_anon_rmap(new, vma, pvmw.address, false); else page_add_file_rmap(new, false); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } - if (vma->vm_flags & VM_LOCKED && !PageTransCompound(new)) - mlock_vma_page(new); + if (vma->vm_flags & VM_LOCKED && !folio_test_large(folio)) + mlock_vma_folio(folio); =20 - if (PageTransHuge(page) && PageMlocked(page)) - clear_page_mlock(page); + if (folio_test_large(folio) && folio_test_mlocked(folio)) + folio_end_mlock(folio); =20 /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, pvmw.address, pvmw.pte); @@ -268,17 +270,17 @@ static bool remove_migration_pte(struct page *page,= struct vm_area_struct *vma, * Get rid of all migration entries and replace them by * references to the indicated page. */ -void remove_migration_ptes(struct page *old, struct page *new, bool lock= ed) +void remove_migration_ptes(struct folio *src, struct folio *dst, bool lo= cked) { struct rmap_walk_control rwc =3D { .rmap_one =3D remove_migration_pte, - .arg =3D old, + .arg =3D src, }; =20 if (locked) - rmap_walk_locked(new, &rwc); + rmap_walk_locked(&dst->page, &rwc); else - rmap_walk(new, &rwc); + rmap_walk(&dst->page, &rwc); } =20 /* @@ -771,6 +773,7 @@ int buffer_migrate_page_norefs(struct address_space *= mapping, */ static int writeout(struct address_space *mapping, struct page *page) { + struct folio *folio =3D page_folio(page); struct writeback_control wbc =3D { .sync_mode =3D WB_SYNC_NONE, .nr_to_write =3D 1, @@ -796,7 +799,7 @@ static int writeout(struct address_space *mapping, st= ruct page *page) * At this point we know that the migration attempt cannot * be successful. */ - remove_migration_ptes(page, page, false); + remove_migration_ptes(folio, folio, false); =20 rc =3D mapping->a_ops->writepage(page, &wbc); =20 @@ -928,6 +931,7 @@ static int __unmap_and_move(struct page *page, struct= page *newpage, int force, enum migrate_mode mode) { struct folio *folio =3D page_folio(page); + struct folio *dst =3D page_folio(newpage); int rc =3D -EAGAIN; bool page_was_mapped =3D false; struct anon_vma *anon_vma =3D NULL; @@ -1039,8 +1043,8 @@ static int __unmap_and_move(struct page *page, stru= ct page *newpage, rc =3D move_to_new_page(newpage, page, mode); =20 if (page_was_mapped) - remove_migration_ptes(page, - rc =3D=3D MIGRATEPAGE_SUCCESS ? newpage : page, false); + remove_migration_ptes(folio, + rc =3D=3D MIGRATEPAGE_SUCCESS ? dst : folio, false); =20 out_unlock_both: unlock_page(newpage); @@ -1174,7 +1178,7 @@ static int unmap_and_move_huge_page(new_page_t get_= new_page, enum migrate_mode mode, int reason, struct list_head *ret) { - struct folio *src =3D page_folio(hpage); + struct folio *dst, *src =3D page_folio(hpage); int rc =3D -EAGAIN; int page_was_mapped =3D 0; struct page *new_hpage; @@ -1202,6 +1206,7 @@ static int unmap_and_move_huge_page(new_page_t get_= new_page, new_hpage =3D get_new_page(hpage, private); if (!new_hpage) return -ENOMEM; + dst =3D page_folio(new_hpage); =20 if (!trylock_page(hpage)) { if (!force) @@ -1262,8 +1267,8 @@ static int unmap_and_move_huge_page(new_page_t get_= new_page, rc =3D move_to_new_page(new_hpage, hpage, mode); =20 if (page_was_mapped) - remove_migration_ptes(hpage, - rc =3D=3D MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false); + remove_migration_ptes(src, + rc =3D=3D MIGRATEPAGE_SUCCESS ? dst : src, false); =20 unlock_put_anon: unlock_page(new_hpage); @@ -2494,15 +2499,17 @@ static void migrate_vma_unmap(struct migrate_vma = *migrate) =20 for (i =3D 0; i < npages && restore; i++) { struct page *page =3D migrate_pfn_to_page(migrate->src[i]); + struct folio *folio; =20 if (!page || (migrate->src[i] & MIGRATE_PFN_MIGRATE)) continue; =20 - remove_migration_ptes(page, page, false); + folio =3D page_folio(page); + remove_migration_ptes(folio, folio, false); =20 migrate->src[i] =3D 0; - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); restore--; } } @@ -2851,6 +2858,7 @@ void migrate_vma_finalize(struct migrate_vma *migra= te) unsigned long i; =20 for (i =3D 0; i < npages; i++) { + struct folio *dst, *src; struct page *newpage =3D migrate_pfn_to_page(migrate->dst[i]); struct page *page =3D migrate_pfn_to_page(migrate->src[i]); =20 @@ -2870,8 +2878,10 @@ void migrate_vma_finalize(struct migrate_vma *migr= ate) newpage =3D page; } =20 - remove_migration_ptes(page, newpage, false); - unlock_page(page); + src =3D page_folio(page); + dst =3D page_folio(newpage); + remove_migration_ptes(src, dst, false); + folio_unlock(src); =20 if (is_zone_device_page(page)) put_page(page); --=20 2.34.1