From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
To: linux-arch@vger.kernel.org
Cc: Yin Fengwei <fengwei.yin@intel.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Matthew Wilcox <willy@infradead.org>
Subject: [PATCH v4 35/36] mm: Convert do_set_pte() to set_pte_range()
Date: Wed, 15 Mar 2023 05:14:43 +0000 [thread overview]
Message-ID: <20230315051444.3229621-36-willy@infradead.org> (raw)
In-Reply-To: <20230315051444.3229621-1-willy@infradead.org>
From: Yin Fengwei <fengwei.yin@intel.com>
set_pte_range() allows to setup page table entries for a specific
range. It takes advantage of batched rmap update for large folio.
It now takes care of calling update_mmu_cache_range().
Signed-off-by: Yin Fengwei <fengwei.yin@intel.com>
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
Documentation/filesystems/locking.rst | 2 +-
include/linux/mm.h | 3 ++-
mm/filemap.c | 3 +--
mm/memory.c | 27 +++++++++++++++------------
4 files changed, 19 insertions(+), 16 deletions(-)
diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst
index 7de7a7272a5e..922886fefb7f 100644
--- a/Documentation/filesystems/locking.rst
+++ b/Documentation/filesystems/locking.rst
@@ -663,7 +663,7 @@ locked. The VM will unlock the page.
Filesystem should find and map pages associated with offsets from "start_pgoff"
till "end_pgoff". ->map_pages() is called with page table locked and must
not block. If it's not possible to reach a page without blocking,
-filesystem should skip it. Filesystem should use do_set_pte() to setup
+filesystem should skip it. Filesystem should use set_pte_range() to setup
page table entry. Pointer to entry associated with the page is passed in
"pte" field in vm_fault structure. Pointers to entries for other offsets
should be calculated relative to "pte".
diff --git a/include/linux/mm.h b/include/linux/mm.h
index ee755bb4e1c1..81788c985a8c 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1299,7 +1299,8 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
}
vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page);
-void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr);
+void set_pte_range(struct vm_fault *vmf, struct folio *folio,
+ struct page *page, unsigned int nr, unsigned long addr);
vm_fault_t finish_fault(struct vm_fault *vmf);
vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);
diff --git a/mm/filemap.c b/mm/filemap.c
index 6e2b0778db45..e2317623dcbf 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3504,8 +3504,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,
ret = VM_FAULT_NOPAGE;
ref_count++;
- do_set_pte(vmf, page, addr);
- update_mmu_cache(vma, addr, vmf->pte);
+ set_pte_range(vmf, folio, page, 1, addr);
} while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages);
/* Restore the vmf->pte */
diff --git a/mm/memory.c b/mm/memory.c
index 6aa21e8f3753..9a654802f104 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4274,7 +4274,8 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
}
#endif
-void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr)
+void set_pte_range(struct vm_fault *vmf, struct folio *folio,
+ struct page *page, unsigned int nr, unsigned long addr)
{
struct vm_area_struct *vma = vmf->vma;
bool uffd_wp = vmf_orig_pte_uffd_wp(vmf);
@@ -4282,7 +4283,7 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr)
bool prefault = vmf->address != addr;
pte_t entry;
- flush_icache_page(vma, page);
+ flush_icache_pages(vma, page, nr);
entry = mk_pte(page, vma->vm_page_prot);
if (prefault && arch_wants_old_prefaulted_pte())
@@ -4296,14 +4297,18 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr)
entry = pte_mkuffd_wp(entry);
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
- inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, addr);
- lru_cache_add_inactive_or_unevictable(page, vma);
+ add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr);
+ VM_BUG_ON_FOLIO(nr != 1, folio);
+ folio_add_new_anon_rmap(folio, vma, addr);
+ folio_add_lru_vma(folio, vma);
} else {
- inc_mm_counter(vma->vm_mm, mm_counter_file(page));
- page_add_file_rmap(page, vma, false);
+ add_mm_counter(vma->vm_mm, mm_counter_file(page), nr);
+ folio_add_file_rmap_range(folio, page, nr, vma, false);
}
- set_pte_at(vma->vm_mm, addr, vmf->pte, entry);
+ set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr);
+
+ /* no need to invalidate: a not-present page won't be cached */
+ update_mmu_cache_range(vma, addr, vmf->pte, nr);
}
static bool vmf_pte_changed(struct vm_fault *vmf)
@@ -4376,11 +4381,9 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
/* Re-check under ptl */
if (likely(!vmf_pte_changed(vmf))) {
- do_set_pte(vmf, page, vmf->address);
-
- /* no need to invalidate: a not-present page won't be cached */
- update_mmu_cache(vma, vmf->address, vmf->pte);
+ struct folio *folio = page_folio(page);
+ set_pte_range(vmf, folio, page, 1, vmf->address);
ret = 0;
} else {
update_mmu_tlb(vma, vmf->address, vmf->pte);
--
2.39.2
next prev parent reply other threads:[~2023-03-15 5:16 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-15 5:14 [PATCH v4 00/36] New page table range API Matthew Wilcox (Oracle)
2023-03-15 5:14 ` [PATCH v4 01/36] mm: Convert page_table_check_pte_set() to page_table_check_ptes_set() Matthew Wilcox (Oracle)
2023-03-15 9:21 ` Mike Rapoport
2023-03-23 18:36 ` Pasha Tatashin
2023-05-25 2:16 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 02/36] mm: Add generic flush_icache_pages() and documentation Matthew Wilcox (Oracle)
2023-03-15 9:27 ` Mike Rapoport
2023-05-25 2:23 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 03/36] mm: Add folio_flush_mapping() Matthew Wilcox (Oracle)
2023-03-15 9:28 ` Mike Rapoport
2023-05-25 2:35 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 04/36] mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO Matthew Wilcox (Oracle)
2023-03-15 9:28 ` Mike Rapoport
2023-05-25 2:43 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 05/36] mm: Add default definition of set_ptes() Matthew Wilcox (Oracle)
2023-03-15 9:34 ` Mike Rapoport
2023-05-25 3:01 ` Anshuman Khandual
2023-05-25 4:06 ` Matthew Wilcox
2023-03-15 5:14 ` [PATCH v4 06/36] alpha: Implement the new page table range API Matthew Wilcox (Oracle)
2023-03-15 9:41 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 07/36] arc: " Matthew Wilcox (Oracle)
2023-03-15 9:44 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 08/36] arm: " Matthew Wilcox (Oracle)
2023-03-15 9:48 ` Mike Rapoport
2023-03-15 10:56 ` Russell King (Oracle)
2023-03-15 5:14 ` [PATCH v4 09/36] arm64: " Matthew Wilcox (Oracle)
2023-03-15 9:49 ` Mike Rapoport
2023-05-25 3:35 ` Anshuman Khandual
2023-05-25 4:05 ` Matthew Wilcox
2023-05-25 4:43 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 10/36] csky: " Matthew Wilcox (Oracle)
2023-03-15 9:50 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 11/36] hexagon: " Matthew Wilcox (Oracle)
2023-03-15 9:54 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 12/36] ia64: " Matthew Wilcox (Oracle)
2023-03-15 9:55 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 13/36] loongarch: " Matthew Wilcox (Oracle)
2023-03-15 10:07 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 14/36] m68k: " Matthew Wilcox (Oracle)
2023-03-15 7:43 ` Geert Uytterhoeven
2023-03-16 16:32 ` Geert Uytterhoeven
2023-03-15 10:07 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 15/36] microblaze: " Matthew Wilcox (Oracle)
2023-03-15 10:07 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 16/36] mips: " Matthew Wilcox (Oracle)
2023-03-15 10:08 ` Mike Rapoport
2023-03-15 10:50 ` Thomas Bogendoerfer
2023-03-15 20:33 ` Matthew Wilcox
2023-03-17 15:29 ` Thomas Bogendoerfer
2023-03-19 18:45 ` Thomas Bogendoerfer
2023-03-19 20:16 ` Matthew Wilcox
2023-03-21 11:30 ` Thomas Bogendoerfer
2023-03-15 5:14 ` [PATCH v4 17/36] nios2: " Matthew Wilcox (Oracle)
2023-03-15 10:08 ` Mike Rapoport
2023-06-13 22:45 ` Dinh Nguyen
2023-07-10 20:18 ` Matthew Wilcox
2023-07-10 23:10 ` Dinh Nguyen
2023-03-15 5:14 ` [PATCH v4 18/36] openrisc: " Matthew Wilcox (Oracle)
2023-03-15 10:09 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 19/36] parisc: " Matthew Wilcox (Oracle)
2023-03-15 10:09 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 20/36] powerpc: " Matthew Wilcox (Oracle)
2023-03-15 9:43 ` Christophe Leroy
2023-03-15 10:18 ` Christophe Leroy
2023-03-17 3:47 ` Matthew Wilcox
2023-03-18 9:19 ` Christophe Leroy
2023-07-10 20:24 ` Matthew Wilcox
2023-07-11 4:40 ` Christophe Leroy
2023-03-15 10:09 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 21/36] riscv: " Matthew Wilcox (Oracle)
2023-03-15 10:10 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 22/36] s390: " Matthew Wilcox (Oracle)
2023-03-15 10:10 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 23/36] superh: " Matthew Wilcox (Oracle)
2023-03-15 7:22 ` John Paul Adrian Glaubitz
2023-03-15 7:36 ` John Paul Adrian Glaubitz
2023-03-15 10:10 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 24/36] sparc32: " Matthew Wilcox (Oracle)
2023-03-15 10:11 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 25/36] sparc64: " Matthew Wilcox (Oracle)
2023-03-15 10:11 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 26/36] um: " Matthew Wilcox (Oracle)
2023-03-15 10:12 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 27/36] x86: " Matthew Wilcox (Oracle)
2023-03-15 10:12 ` Mike Rapoport
2023-03-15 10:34 ` Peter Zijlstra
2023-03-15 11:16 ` Mike Rapoport
2023-03-15 11:19 ` Peter Zijlstra
2023-03-15 16:12 ` Matthew Wilcox
2023-03-15 5:14 ` [PATCH v4 28/36] xtensa: " Matthew Wilcox (Oracle)
2023-03-15 10:12 ` Mike Rapoport
2023-03-15 5:14 ` [PATCH v4 29/36] mm: Remove page_mapping_file() Matthew Wilcox (Oracle)
2023-05-25 3:50 ` Anshuman Khandual
2023-05-25 4:03 ` Matthew Wilcox
2023-05-25 4:46 ` Anshuman Khandual
2023-05-25 5:37 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 30/36] mm: Rationalise flush_icache_pages() and flush_icache_page() Matthew Wilcox (Oracle)
2023-03-15 5:14 ` [PATCH v4 31/36] mm: Tidy up set_ptes definition Matthew Wilcox (Oracle)
2023-05-25 6:20 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 32/36] mm: Use flush_icache_pages() in do_set_pmd() Matthew Wilcox (Oracle)
2023-05-25 6:31 ` Anshuman Khandual
2023-03-15 5:14 ` [PATCH v4 33/36] filemap: Add filemap_map_folio_range() Matthew Wilcox (Oracle)
2023-03-15 5:14 ` [PATCH v4 34/36] rmap: add folio_add_file_rmap_range() Matthew Wilcox (Oracle)
2023-03-15 13:34 ` Ryan Roberts
2023-03-15 16:08 ` Ryan Roberts
2023-03-16 16:27 ` Yin, Fengwei
2023-03-16 16:34 ` Ryan Roberts
2023-03-17 8:23 ` Yin, Fengwei
2023-03-17 12:46 ` Ryan Roberts
2023-03-17 13:28 ` Yin, Fengwei
2023-03-15 5:14 ` Matthew Wilcox (Oracle) [this message]
2023-03-15 15:26 ` [PATCH v4 35/36] mm: Convert do_set_pte() to set_pte_range() Ryan Roberts
2023-03-16 16:23 ` Yin, Fengwei
2023-03-16 16:38 ` Ryan Roberts
2023-03-16 16:41 ` Yin, Fengwei
2023-03-16 16:50 ` Ryan Roberts
2023-03-16 17:52 ` Matthew Wilcox
2023-03-17 1:58 ` Yin, Fengwei
2023-03-17 3:44 ` Matthew Wilcox
2023-03-17 6:33 ` Yin, Fengwei
2023-03-17 8:00 ` Ryan Roberts
2023-03-17 8:19 ` Yin, Fengwei
2023-03-17 13:00 ` Ryan Roberts
2023-03-17 13:44 ` Yin, Fengwei
2023-03-24 14:58 ` Will Deacon
2023-03-24 15:11 ` Matthew Wilcox
2023-03-24 17:23 ` Will Deacon
2023-03-27 1:23 ` Yin Fengwei
2023-03-20 13:38 ` Yin, Fengwei
2023-03-20 14:08 ` Matthew Wilcox
2023-03-21 1:58 ` Yin, Fengwei
2023-03-21 5:13 ` Yin Fengwei
2023-05-30 8:07 ` [PATCH 0/4] New page table range API fixup patches Yin Fengwei
2023-05-30 8:07 ` [PATCH 1/4] filemap: avoid interfere with xas.xa_index Yin Fengwei
2023-05-30 8:07 ` [PATCH 2/4] rmap: fix typo in folio_add_file_rmap_range() Yin Fengwei
2023-05-30 8:07 ` [PATCH 3/4] mm: mark PTEs referencing the accessed folio young Yin Fengwei
2023-05-30 8:07 ` [PATCH 4/4] filemap: Check address range in filemap_map_folio_range() Yin Fengwei
2023-03-15 5:14 ` [PATCH v4 36/36] filemap: Batch PTE mappings Matthew Wilcox (Oracle)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230315051444.3229621-36-willy@infradead.org \
--to=willy@infradead.org \
--cc=fengwei.yin@intel.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).