archive mirror
 help / color / mirror / Atom feed
From: "Aneesh Kumar K.V" <>
Cc: "Aneesh Kumar K.V" <>,
	Hugh Dickins <>,
	Linus Torvalds <>,,,,
	"Kirill A . Shutemov" <>,,
	"Kirill A . Shutemov" <>
Subject: [PATCH 6/6] mm/mremap: hold the rmap lock in write mode when moving page table entries.
Date: Thu, 10 Jun 2021 14:05:49 +0530	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

To avoid a race between rmap walk and mremap, mremap does take_rmap_locks().
The lock was taken to ensure that rmap walk don't miss a page table entry due to
PTE moves via move_pagetables(). The kernel does further optimization of
this lock such that if we are going to find the newly added vma after the
old vma, the rmap lock is not taken. This is because rmap walk would find the
vmas in the same order and if we don't find the page table attached to
older vma we would find it with the new vma which we would iterate later.

As explained in commit eb66ae030829 ("mremap: properly flush TLB before releasing the page")
mremap is special in that it doesn't take ownership of the page. The
optimized version for PUD/PMD aligned mremap also doesn't hold the ptl lock.
This can result in stale TLB entries as show below.

This patch updates the rmap locking requirement in mremap to handle the race condition
explained below with optimized mremap::

Optmized PMD move

    CPU 1                           CPU 2                                   CPU 3

    mremap(old_addr, new_addr)      page_shrinker/try_to_unmap_one


                                    addr = old_addr
    pmd = *old_pmd

    *new_pmd = pmd
                                                                            *new_addr = 10; and fills
                                                                            TLB with new addr
                                                                            and old pfn

                                    old pfn is free.
                                                                            Stale TLB entry

Optimized PUD move also suffers from a similar race.
Both the above race condition can be fixed if we force mremap path to take rmap lock.

Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
Fixes: c49dd3401802 ("mm: speedup mremap on 1GB or larger regions")
Acked-by: Hugh Dickins <>
Acked-by: Kirill A. Shutemov <>
Signed-off-by: Aneesh Kumar K.V <>
 mm/mremap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/mremap.c b/mm/mremap.c
index dacfa9111ab1..b8eed7645cea 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -504,7 +504,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 		} else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) {
 			if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new_addr,
-					   old_pud, new_pud, need_rmap_locks))
+					   old_pud, new_pud, true))
@@ -531,7 +531,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 			 * moving at the PMD level if possible.
 			if (move_pgt_entry(NORMAL_PMD, vma, old_addr, new_addr,
-					   old_pmd, new_pmd, need_rmap_locks))
+					   old_pmd, new_pmd, true))

  parent reply	other threads:[~2021-06-10  8:39 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-10  8:35 [PATCH 0/6] mremap fixes Aneesh Kumar K.V
2021-06-10  8:35 ` [PATCH 1/6] selftest/mremap_test: Update the test to handle pagesize other than 4K Aneesh Kumar K.V
2021-06-10  8:35 ` [PATCH 2/6] selftest/mremap_test: Avoid crash with static build Aneesh Kumar K.V
2021-06-10  8:35 ` [PATCH 3/6] mm/mremap: Convert huge PUD move to separate helper Aneesh Kumar K.V
2021-06-10 22:03   ` Hugh Dickins
2021-06-10  8:35 ` [PATCH 4/6] mm/mremap: Don't enable optimized PUD move if page table levels is 2 Aneesh Kumar K.V
2021-06-10  8:35 ` [PATCH 5/6] mm/mremap: Use pmd/pud_poplulate to update page table entries Aneesh Kumar K.V
2021-06-10 18:16   ` Linus Torvalds
2021-06-13  9:06     ` Aneesh Kumar K.V
2021-06-13 10:50       ` Matthew Wilcox
2021-06-13 11:13         ` Aneesh Kumar K.V
2021-06-14  5:27           ` Christophe Leroy
2021-06-13 18:53       ` Linus Torvalds
2021-06-10  8:35 ` Aneesh Kumar K.V [this message]
2021-06-11  8:11   ` [PATCH 6/6] mm/mremap: hold the rmap lock in write mode when moving " Jann Horn

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH 6/6] mm/mremap: hold the rmap lock in write mode when moving page table entries.' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).