From: Peter Xu <peterx@redhat.com>
To: linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: Kirill Shutemov <kirill@shutemov.name>,
Wei Zhang <wzam@amazon.com>,
Mike Rapoport <rppt@linux.vnet.ibm.com>,
Matthew Wilcox <willy@infradead.org>,
Miaohe Lin <linmiaohe@huawei.com>,
Andrea Arcangeli <aarcange@redhat.com>,
peterx@redhat.com, Gal Pressman <galpress@amazon.com>,
Jan Kara <jack@suse.cz>, Jann Horn <jannh@google.com>,
Andrew Morton <akpm@linux-foundation.org>,
Kirill Tkhai <ktkhai@virtuozzo.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Mike Kravetz <mike.kravetz@oracle.com>,
Jason Gunthorpe <jgg@ziepe.ca>,
David Gibson <david@gibson.dropbear.id.au>,
Christoph Hellwig <hch@lst.de>
Subject: [PATCH v2 3/4] mm: Introduce page_needs_cow_for_dma() for deciding whether cow
Date: Thu, 4 Feb 2021 09:50:32 -0500 [thread overview]
Message-ID: <20210204145033.136755-4-peterx@redhat.com> (raw)
In-Reply-To: <20210204145033.136755-1-peterx@redhat.com>
We've got quite a few places (pte, pmd, pud) that explicitly checked against
whether we should break the cow right now during fork(). It's easier to
provide a helper, especially before we work the same thing on hugetlbfs.
Since we'll reference is_cow_mapping() in mm.h, move it there too. Actually it
suites mm.h more since internal.h is mm/ only, but mm.h is exported to the
whole kernel. With that we should expect another patch to use is_cow_mapping()
whenever we can across the kernel since we do use it quite a lot but it's
always done with raw code against VM_* flags.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
include/linux/mm.h | 21 +++++++++++++++++++++
mm/huge_memory.c | 8 ++------
mm/internal.h | 5 -----
mm/memory.c | 7 +------
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index ecdf8a8cd6ae..6ea20721d349 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1291,6 +1291,27 @@ static inline bool page_maybe_dma_pinned(struct page *page)
GUP_PIN_COUNTING_BIAS;
}
+static inline bool is_cow_mapping(vm_flags_t flags)
+{
+ return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
+}
+
+/*
+ * This should most likely only be called during fork() to see whether we
+ * should break the cow immediately for a page on the src mm.
+ */
+static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,
+ struct page *page)
+{
+ if (!is_cow_mapping(vma->vm_flags))
+ return false;
+
+ if (!atomic_read(&vma->vm_mm->has_pinned))
+ return false;
+
+ return page_maybe_dma_pinned(page);
+}
+
#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
#define SECTION_IN_PAGE_FLAGS
#endif
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 9237976abe72..dbff6c7eda67 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1095,9 +1095,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* best effort that the pinned pages won't be replaced by another
* random page during the coming copy-on-write.
*/
- if (unlikely(is_cow_mapping(vma->vm_flags) &&
- atomic_read(&src_mm->has_pinned) &&
- page_maybe_dma_pinned(src_page))) {
+ if (unlikely(page_needs_cow_for_dma(vma, src_page))) {
pte_free(dst_mm, pgtable);
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
@@ -1209,9 +1207,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
}
/* Please refer to comments in copy_huge_pmd() */
- if (unlikely(is_cow_mapping(vma->vm_flags) &&
- atomic_read(&src_mm->has_pinned) &&
- page_maybe_dma_pinned(pud_page(pud)))) {
+ if (unlikely(page_needs_cow_for_dma(vma, pud_page(pud)))) {
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
__split_huge_pud(vma, src_pud, addr);
diff --git a/mm/internal.h b/mm/internal.h
index 25d2b2439f19..24eec93d0dac 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -296,11 +296,6 @@ static inline unsigned int buddy_order(struct page *page)
*/
#define buddy_order_unsafe(page) READ_ONCE(page_private(page))
-static inline bool is_cow_mapping(vm_flags_t flags)
-{
- return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
-}
-
/*
* These three helpers classifies VMAs for virtual memory accounting.
*/
diff --git a/mm/memory.c b/mm/memory.c
index feff48e1465a..b2849e1d4aab 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -800,9 +800,6 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
struct mm_struct *src_mm = src_vma->vm_mm;
struct page *new_page;
- if (!is_cow_mapping(src_vma->vm_flags))
- return 1;
-
/*
* What we want to do is to check whether this page may
* have been pinned by the parent process. If so,
@@ -816,9 +813,7 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
* the page count. That might give false positives for
* for pinning, but it will work correctly.
*/
- if (likely(!atomic_read(&src_mm->has_pinned)))
- return 1;
- if (likely(!page_maybe_dma_pinned(page)))
+ if (likely(!page_needs_cow_for_dma(src_vma, page)))
return 1;
new_page = *prealloc;
--
2.26.2
next prev parent reply other threads:[~2021-02-04 19:33 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-04 14:50 [PATCH v2 0/4] mm/hugetlb: Early cow on fork, and a few cleanups Peter Xu
2021-02-04 14:50 ` [PATCH v2 1/4] hugetlb: Dedup the code to add a new file_region Peter Xu
2021-02-04 14:50 ` [PATCH v2 2/4] hugetlg: Break earlier in add_reservation_in_range() when we can Peter Xu
2021-02-04 14:50 ` Peter Xu [this message]
2021-02-04 17:54 ` [PATCH v2 3/4] mm: Introduce page_needs_cow_for_dma() for deciding whether cow Linus Torvalds
2021-02-04 19:25 ` Peter Xu
2021-02-04 23:20 ` Jason Gunthorpe
2021-02-05 0:50 ` Peter Xu
2021-02-04 14:50 ` [PATCH v2 4/4] hugetlb: Do early cow when page pinned on src mm Peter Xu
2021-02-04 23:25 ` Mike Kravetz
2021-02-05 1:43 ` Peter Xu
2021-02-05 5:11 ` Mike Kravetz
2021-02-05 16:05 ` Peter Xu
[not found] ` <329ADC08-552E-423B-9230-99643B81C14A@amazon.com>
2021-02-05 15:51 ` Peter Xu
2021-02-07 9:09 ` Gal Pressman
2021-02-07 15:31 ` Peter Xu
2021-02-04 20:20 ` [PATCH v2 5/4] mm: Use is_cow_mapping() across tree where proper Peter Xu
2021-02-04 20:26 ` Linus Torvalds
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210204145033.136755-4-peterx@redhat.com \
--to=peterx@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=david@gibson.dropbear.id.au \
--cc=galpress@amazon.com \
--cc=hch@lst.de \
--cc=jack@suse.cz \
--cc=jannh@google.com \
--cc=jgg@ziepe.ca \
--cc=kirill@shutemov.name \
--cc=ktkhai@virtuozzo.com \
--cc=linmiaohe@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mike.kravetz@oracle.com \
--cc=rppt@linux.vnet.ibm.com \
--cc=torvalds@linux-foundation.org \
--cc=willy@infradead.org \
--cc=wzam@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).