From: Sasha Levin <sashal@kernel.org>
To: stable@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: Andrea Arcangeli <aarcange@redhat.com>,
Jerome Glisse <jglisse@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Sasha Levin <sashal@kernel.org>,
linux-mm@kvack.org
Subject: [PATCH AUTOSEL 4.18 34/39] mm: thp: fix mmu_notifier in migrate_misplaced_transhuge_page()
Date: Tue, 13 Nov 2018 00:50:48 -0500 [thread overview]
Message-ID: <20181113055053.78352-34-sashal@kernel.org> (raw)
In-Reply-To: <20181113055053.78352-1-sashal@kernel.org>
From: Andrea Arcangeli <aarcange@redhat.com>
[ Upstream commit 7066f0f933a1fd707bb38781866657769cff7efc ]
change_huge_pmd() after arming the numa/protnone pmd doesn't flush the TLB
right away. do_huge_pmd_numa_page() flushes the TLB before calling
migrate_misplaced_transhuge_page(). By the time do_huge_pmd_numa_page()
runs some CPU could still access the page through the TLB.
change_huge_pmd() before arming the numa/protnone transhuge pmd calls
mmu_notifier_invalidate_range_start(). So there's no need of
mmu_notifier_invalidate_range_start()/mmu_notifier_invalidate_range_only_end()
sequence in migrate_misplaced_transhuge_page() too, because by the time
migrate_misplaced_transhuge_page() runs, the pmd mapping has already been
invalidated in the secondary MMUs. It has to or if a secondary MMU can
still write to the page, the migrate_page_copy() would lose data.
However an explicit mmu_notifier_invalidate_range() is needed before
migrate_misplaced_transhuge_page() starts copying the data of the
transhuge page or the below can happen for MMU notifier users sharing the
primary MMU pagetables and only implementing ->invalidate_range:
CPU0 CPU1 GPU sharing linux pagetables using
only ->invalidate_range
----------- ------------ ---------
GPU secondary MMU writes to the page
mapped by the transhuge pmd
change_pmd_range()
mmu..._range_start()
->invalidate_range_start() noop
change_huge_pmd()
set_pmd_at(numa/protnone)
pmd_unlock()
do_huge_pmd_numa_page()
CPU TLB flush globally (1)
CPU cannot write to page
migrate_misplaced_transhuge_page()
GPU writes to the page...
migrate_page_copy()
...GPU stops writing to the page
CPU TLB flush (2)
mmu..._range_end() (3)
->invalidate_range_stop() noop
->invalidate_range()
GPU secondary MMU is invalidated
and cannot write to the page anymore
(too late)
Just like we need a CPU TLB flush (1) because the TLB flush (2) arrives
too late, we also need a mmu_notifier_invalidate_range() before calling
migrate_misplaced_transhuge_page(), because the ->invalidate_range() in
(3) also arrives too late.
This requirement is the result of the lazy optimization in
change_huge_pmd() that releases the pmd_lock without first flushing the
TLB and without first calling mmu_notifier_invalidate_range().
Even converting the removed mmu_notifier_invalidate_range_only_end() into
a mmu_notifier_invalidate_range_end() would not have been enough to fix
this, because it run after migrate_page_copy().
After the hugepage data copy is done migrate_misplaced_transhuge_page()
can proceed and call set_pmd_at without having to flush the TLB nor any
secondary MMUs because the secondary MMU invalidate, just like the CPU TLB
flush, has to happen before the migrate_page_copy() is called or it would
be a bug in the first place (and it was for drivers using
->invalidate_range()).
KVM is unaffected because it doesn't implement ->invalidate_range().
The standard PAGE_SIZEd migrate_misplaced_page is less accelerated and
uses the generic migrate_pages which transitions the pte from
numa/protnone to a migration entry in try_to_unmap_one() and flushes TLBs
and all mmu notifiers there before copying the page.
Link: http://lkml.kernel.org/r/20181013002430.698-3-aarcange@redhat.com
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Acked-by: Mel Gorman <mgorman@suse.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reviewed-by: Aaron Tomlin <atomlin@redhat.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
mm/huge_memory.c | 14 +++++++++++++-
mm/migrate.c | 19 ++++++-------------
2 files changed, 19 insertions(+), 14 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 3238bb2d0c93..d8d46936e39e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1560,8 +1560,20 @@ int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
* We are not sure a pending tlb flush here is for a huge page
* mapping or not. Hence use the tlb range variant
*/
- if (mm_tlb_flush_pending(vma->vm_mm))
+ if (mm_tlb_flush_pending(vma->vm_mm)) {
flush_tlb_range(vma, haddr, haddr + HPAGE_PMD_SIZE);
+ /*
+ * change_huge_pmd() released the pmd lock before
+ * invalidating the secondary MMUs sharing the primary
+ * MMU pagetables (with ->invalidate_range()). The
+ * mmu_notifier_invalidate_range_end() (which
+ * internally calls ->invalidate_range()) in
+ * change_pmd_range() will run after us, so we can't
+ * rely on it here and we need an explicit invalidate.
+ */
+ mmu_notifier_invalidate_range(vma->vm_mm, haddr,
+ haddr + HPAGE_PMD_SIZE);
+ }
/*
* Migrate the THP to the requested node, returns with page unlocked
diff --git a/mm/migrate.c b/mm/migrate.c
index 38ad6365ed10..1d189fc0d206 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2022,8 +2022,8 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
int isolated = 0;
struct page *new_page = NULL;
int page_lru = page_is_file_cache(page);
- unsigned long mmun_start = address & HPAGE_PMD_MASK;
- unsigned long mmun_end = mmun_start + HPAGE_PMD_SIZE;
+ unsigned long start = address & HPAGE_PMD_MASK;
+ unsigned long end = start + HPAGE_PMD_SIZE;
/*
* Rate-limit the amount of data that is being migrated to a node.
@@ -2058,11 +2058,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
WARN_ON(PageLRU(new_page));
/* Recheck the target PMD */
- mmu_notifier_invalidate_range_start(mm, mmun_start, mmun_end);
ptl = pmd_lock(mm, pmd);
if (unlikely(!pmd_same(*pmd, entry) || !page_ref_freeze(page, 2))) {
spin_unlock(ptl);
- mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
/* Reverse changes made by migrate_page_copy() */
if (TestClearPageActive(new_page))
@@ -2093,8 +2091,8 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
* new page and page_add_new_anon_rmap guarantee the copy is
* visible before the pagetable update.
*/
- flush_cache_range(vma, mmun_start, mmun_end);
- page_add_anon_rmap(new_page, vma, mmun_start, true);
+ flush_cache_range(vma, start, end);
+ page_add_anon_rmap(new_page, vma, start, true);
/*
* At this point the pmd is numa/protnone (i.e. non present) and the TLB
* has already been flushed globally. So no TLB can be currently
@@ -2106,7 +2104,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
* MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this
* pmd.
*/
- set_pmd_at(mm, mmun_start, pmd, entry);
+ set_pmd_at(mm, start, pmd, entry);
update_mmu_cache_pmd(vma, address, &entry);
page_ref_unfreeze(page, 2);
@@ -2115,11 +2113,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
set_page_owner_migrate_reason(new_page, MR_NUMA_MISPLACED);
spin_unlock(ptl);
- /*
- * No need to double call mmu_notifier->invalidate_range() callback as
- * the above pmdp_huge_clear_flush_notify() did already call it.
- */
- mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
/* Take an "isolate" reference and put new page on the LRU. */
get_page(new_page);
@@ -2144,7 +2137,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
ptl = pmd_lock(mm, pmd);
if (pmd_same(*pmd, entry)) {
entry = pmd_modify(entry, vma->vm_page_prot);
- set_pmd_at(mm, mmun_start, pmd, entry);
+ set_pmd_at(mm, start, pmd, entry);
update_mmu_cache_pmd(vma, address, &entry);
}
spin_unlock(ptl);
--
2.17.1
next prev parent reply other threads:[~2018-11-13 5:51 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-13 5:50 [PATCH AUTOSEL 4.18 01/39] bfs: add sanity check at bfs_fill_super() Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 02/39] cifs: don't dereference smb_file_target before null check Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 03/39] cifs: fix return value for cifs_listxattr Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 04/39] arm64: kprobe: make page to RO mode when allocate it Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 05/39] nvme-pci: fix conflicting p2p resource adds Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 06/39] block: brd: associate with queue until adding disk Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 07/39] net: hns3: bugfix for rtnl_lock's range in the hclgevf_reset() Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 08/39] net: hns3: bugfix for rtnl_lock's range in the hclge_reset() Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 09/39] net: hns3: bugfix for the initialization of command queue's spin lock Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 10/39] ixgbe: fix MAC anti-spoofing filter after VFLR Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 11/39] mm: Fix warning in insert_pfn() Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 12/39] mm/memory_hotplug: make add_memory() take the device_hotplug_lock Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 13/39] reiserfs: propagate errors from fill_with_dentries() properly Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 14/39] hfs: prevent btree data loss on root split Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 15/39] hfsplus: " Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 16/39] mm/gup_benchmark.c: prevent integer overflow in ioctl Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 17/39] perf unwind: Take pgoff into account when reporting elf to libdwfl Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 18/39] um: Give start_idle_thread() a return code Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 19/39] drm/edid: Add 6 bpc quirk for BOE panel Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 20/39] afs: Handle EIO from delivery function Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 21/39] platform/x86: intel_telemetry: report debugfs failure Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 22/39] clk: fixed-rate: fix of_node_get-put imbalance Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 23/39] perf symbols: Set PLT entry/header sizes properly on Sparc Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 24/39] fs/exofs: fix potential memory leak in mount option parsing Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 25/39] clk: samsung: exynos5420: Enable PERIS clocks for suspend Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 26/39] apparmor: Fix uninitialized value in aa_split_fqname Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 27/39] x86/earlyprintk: Add a force option for pciserial device Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 28/39] platform/x86: acerhdf: Add BIOS entry for Gateway LT31 v1.3307 Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 29/39] arm64: percpu: Initialize ret in the default case Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 30/39] clk: meson: clk-pll: drop CLK_GET_RATE_NOCACHE where unnecessary Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 31/39] clk: sunxi-ng: sun50i: h6: Add 2x fixed post-divider to MMC module clocks Sasha Levin
2018-11-13 12:27 ` Icenowy Zheng
2018-11-22 19:35 ` Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 32/39] clk: ti: fix OF child-node lookup Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 33/39] mm: thp: fix MADV_DONTNEED vs migrate_misplaced_transhuge_page race condition Sasha Levin
2018-11-13 5:50 ` Sasha Levin [this message]
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 35/39] mm: calculate deferred pages after skipping mirrored memory Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 36/39] mm/vmstat.c: assert that vmstat_text is in sync with stat_items_size Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 37/39] userfaultfd: allow get_mempolicy(MPOL_F_NODE|MPOL_F_ADDR) to trigger userfaults Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 38/39] mm: don't miss the last page because of round-off error Sasha Levin
2018-11-13 5:50 ` [PATCH AUTOSEL 4.18 39/39] mm: don't warn about large allocations for slab Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181113055053.78352-34-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=jglisse@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=stable@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).