From: Nadav Amit <namit@vmware.com> To: <linux-mm@kvack.org> Cc: <nadav.amit@gmail.com>, <linux-kernel@vger.kernel.org>, <akpm@linux-foundation.org>, Minchan Kim <minchan@kernel.org>, Mel Gorman <mgorman@techsingularity.net>, Hugh Dickins <hughd@google.com>, Nadav Amit <namit@vmware.com> Subject: [PATCH v6 7/7] mm: fix KSM data corruption Date: Tue, 1 Aug 2017 17:08:18 -0700 [thread overview] Message-ID: <20170802000818.4760-8-namit@vmware.com> (raw) In-Reply-To: <20170802000818.4760-1-namit@vmware.com> From: Minchan Kim <minchan@kernel.org> Nadav reported KSM can corrupt the user data by the TLB batching race[1]. That means data user written can be lost. Quote from Nadav Amit " For this race we need 4 CPUs: CPU0: Caches a writable and dirty PTE entry, and uses the stale value for write later. CPU1: Runs madvise_free on the range that includes the PTE. It would clear the dirty-bit. It batches TLB flushes. CPU2: Writes 4 to /proc/PID/clear_refs , clearing the PTEs soft-dirty. We care about the fact that it clears the PTE write-bit, and of course, batches TLB flushes. CPU3: Runs KSM. Our purpose is to pass the following test in write_protect_page(): if (pte_write(*pvmw.pte) || pte_dirty(*pvmw.pte) || (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte))) Since it will avoid TLB flush. And we want to do it while the PTE is stale. Later, and before replacing the page, we would be able to change the page. Note that all the operations the CPU1-3 perform canhappen in parallel since they only acquire mmap_sem for read. We start with two identical pages. Everything below regards the same page/PTE. CPU0 CPU1 CPU2 CPU3 ---- ---- ---- ---- Write the same value on page [cache PTE as dirty in TLB] MADV_FREE pte_mkclean() 4 > clear_refs pte_wrprotect() write_protect_page() [ success, no flush ] pages_indentical() [ ok ] Write to page different value [Ok, using stale PTE] replace_page() Later, CPU1, CPU2 and CPU3 would flush the TLB, but that is too late. CPU0 already wrote on the page, but KSM ignored this write, and it got lost. " In above scenario, MADV_FREE is fixed by changing TLB batching API including [set|clear]_tlb_flush_pending. Remained thing is soft-dirty part. This patch changes soft-dirty uses TLB batching API instead of flush_tlb_mm and KSM checks pending TLB flush by using mm_tlb_flush_pending so that it will flush TLB to avoid data lost if there are other parallel threads pending TLB flush. [1] http://lkml.kernel.org/r/BD3A0EBE-ECF4-41D4-87FA-C755EA9AB6BD@gmail.com Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Hugh Dickins <hughd@google.com> Signed-off-by: Minchan Kim <minchan@kernel.org> Signed-off-by: Nadav Amit <namit@vmware.com> Reported-by: Nadav Amit <namit@vmware.com> Reviewed-by: Andrea Arcangeli <aarcange@redhat.com> Tested-by: Nadav Amit <namit@vmware.com> --- fs/proc/task_mmu.c | 7 +++++-- mm/ksm.c | 3 ++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 520802da059c..aa20da220973 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -16,9 +16,10 @@ #include <linux/mmu_notifier.h> #include <linux/page_idle.h> #include <linux/shmem_fs.h> +#include <linux/uaccess.h> #include <asm/elf.h> -#include <linux/uaccess.h> +#include <asm/tlb.h> #include <asm/tlbflush.h> #include "internal.h" @@ -1009,6 +1010,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, struct mm_struct *mm; struct vm_area_struct *vma; enum clear_refs_types type; + struct mmu_gather tlb; int itype; int rv; @@ -1055,6 +1057,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, } down_read(&mm->mmap_sem); + tlb_gather_mmu(&tlb, mm, 0, -1); if (type == CLEAR_REFS_SOFT_DIRTY) { for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) @@ -1076,7 +1079,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, walk_page_range(0, mm->highest_vm_end, &clear_refs_walk); if (type == CLEAR_REFS_SOFT_DIRTY) mmu_notifier_invalidate_range_end(mm, 0, -1); - flush_tlb_mm(mm); + tlb_finish_mmu(&tlb, 0, -1); up_read(&mm->mmap_sem); out_mm: mmput(mm); diff --git a/mm/ksm.c b/mm/ksm.c index 216184af0e19..e5bf02e39752 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -883,7 +883,8 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; if (pte_write(*pvmw.pte) || pte_dirty(*pvmw.pte) || - (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte))) { + (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte)) || + mm_tlb_flush_pending(mm)) { pte_t entry; swapped = PageSwapCache(page); -- 2.11.0
WARNING: multiple messages have this Message-ID (diff)
From: Nadav Amit <namit@vmware.com> To: linux-mm@kvack.org Cc: nadav.amit@gmail.com, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, Minchan Kim <minchan@kernel.org>, Mel Gorman <mgorman@techsingularity.net>, Hugh Dickins <hughd@google.com>, Nadav Amit <namit@vmware.com> Subject: [PATCH v6 7/7] mm: fix KSM data corruption Date: Tue, 1 Aug 2017 17:08:18 -0700 [thread overview] Message-ID: <20170802000818.4760-8-namit@vmware.com> (raw) In-Reply-To: <20170802000818.4760-1-namit@vmware.com> From: Minchan Kim <minchan@kernel.org> Nadav reported KSM can corrupt the user data by the TLB batching race[1]. That means data user written can be lost. Quote from Nadav Amit " For this race we need 4 CPUs: CPU0: Caches a writable and dirty PTE entry, and uses the stale value for write later. CPU1: Runs madvise_free on the range that includes the PTE. It would clear the dirty-bit. It batches TLB flushes. CPU2: Writes 4 to /proc/PID/clear_refs , clearing the PTEs soft-dirty. We care about the fact that it clears the PTE write-bit, and of course, batches TLB flushes. CPU3: Runs KSM. Our purpose is to pass the following test in write_protect_page(): if (pte_write(*pvmw.pte) || pte_dirty(*pvmw.pte) || (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte))) Since it will avoid TLB flush. And we want to do it while the PTE is stale. Later, and before replacing the page, we would be able to change the page. Note that all the operations the CPU1-3 perform canhappen in parallel since they only acquire mmap_sem for read. We start with two identical pages. Everything below regards the same page/PTE. CPU0 CPU1 CPU2 CPU3 ---- ---- ---- ---- Write the same value on page [cache PTE as dirty in TLB] MADV_FREE pte_mkclean() 4 > clear_refs pte_wrprotect() write_protect_page() [ success, no flush ] pages_indentical() [ ok ] Write to page different value [Ok, using stale PTE] replace_page() Later, CPU1, CPU2 and CPU3 would flush the TLB, but that is too late. CPU0 already wrote on the page, but KSM ignored this write, and it got lost. " In above scenario, MADV_FREE is fixed by changing TLB batching API including [set|clear]_tlb_flush_pending. Remained thing is soft-dirty part. This patch changes soft-dirty uses TLB batching API instead of flush_tlb_mm and KSM checks pending TLB flush by using mm_tlb_flush_pending so that it will flush TLB to avoid data lost if there are other parallel threads pending TLB flush. [1] http://lkml.kernel.org/r/BD3A0EBE-ECF4-41D4-87FA-C755EA9AB6BD@gmail.com Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Hugh Dickins <hughd@google.com> Signed-off-by: Minchan Kim <minchan@kernel.org> Signed-off-by: Nadav Amit <namit@vmware.com> Reported-by: Nadav Amit <namit@vmware.com> Reviewed-by: Andrea Arcangeli <aarcange@redhat.com> Tested-by: Nadav Amit <namit@vmware.com> --- fs/proc/task_mmu.c | 7 +++++-- mm/ksm.c | 3 ++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 520802da059c..aa20da220973 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -16,9 +16,10 @@ #include <linux/mmu_notifier.h> #include <linux/page_idle.h> #include <linux/shmem_fs.h> +#include <linux/uaccess.h> #include <asm/elf.h> -#include <linux/uaccess.h> +#include <asm/tlb.h> #include <asm/tlbflush.h> #include "internal.h" @@ -1009,6 +1010,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, struct mm_struct *mm; struct vm_area_struct *vma; enum clear_refs_types type; + struct mmu_gather tlb; int itype; int rv; @@ -1055,6 +1057,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, } down_read(&mm->mmap_sem); + tlb_gather_mmu(&tlb, mm, 0, -1); if (type == CLEAR_REFS_SOFT_DIRTY) { for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) @@ -1076,7 +1079,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, walk_page_range(0, mm->highest_vm_end, &clear_refs_walk); if (type == CLEAR_REFS_SOFT_DIRTY) mmu_notifier_invalidate_range_end(mm, 0, -1); - flush_tlb_mm(mm); + tlb_finish_mmu(&tlb, 0, -1); up_read(&mm->mmap_sem); out_mm: mmput(mm); diff --git a/mm/ksm.c b/mm/ksm.c index 216184af0e19..e5bf02e39752 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -883,7 +883,8 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; if (pte_write(*pvmw.pte) || pte_dirty(*pvmw.pte) || - (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte))) { + (pte_protnone(*pvmw.pte) && pte_savedwrite(*pvmw.pte)) || + mm_tlb_flush_pending(mm)) { pte_t entry; swapped = PageSwapCache(page); -- 2.11.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-08-02 7:34 UTC|newest] Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-08-02 0:08 [PATCH v6 0/7] fixes of TLB batching races Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` [PATCH v6 1/7] mm: migrate: prevent racy access to tlb_flush_pending Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` [PATCH v6 2/7] mm: migrate: fix barriers around tlb_flush_pending Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` [PATCH v6 3/7] Revert "mm: numa: defer TLB flush for THP migration as long as possible" Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-11 10:50 ` Peter Zijlstra 2017-08-11 10:50 ` Peter Zijlstra 2017-08-02 0:08 ` [PATCH v6 4/7] mm: refactoring TLB gathering API Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-11 9:23 ` Peter Zijlstra 2017-08-11 9:23 ` Peter Zijlstra 2017-08-11 17:12 ` Nadav Amit 2017-08-11 17:12 ` Nadav Amit 2017-08-14 0:49 ` Minchan Kim 2017-08-14 0:49 ` Minchan Kim 2017-08-02 0:08 ` [PATCH v6 5/7] mm: make tlb_flush_pending global Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 14:28 ` kbuild test robot 2017-08-02 14:28 ` kbuild test robot 2017-08-02 23:23 ` Minchan Kim 2017-08-02 23:23 ` Minchan Kim 2017-08-02 23:27 ` Andrew Morton 2017-08-02 23:27 ` Andrew Morton 2017-08-02 23:34 ` Minchan Kim 2017-08-02 23:34 ` Minchan Kim 2017-08-03 16:40 ` kbuild test robot 2017-08-03 16:40 ` kbuild test robot 2017-08-02 0:08 ` [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-08 1:19 ` [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 2:28 ` Minchan Kim 2017-08-08 2:28 ` Minchan Kim 2017-08-08 2:28 ` Minchan Kim 2017-08-08 4:23 ` Nadav Amit 2017-08-08 4:23 ` Nadav Amit 2017-08-08 4:23 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:16 ` Nadav Amit 2017-08-08 8:16 ` Nadav Amit 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:20 ` Minchan Kim 2017-08-10 4:20 ` Minchan Kim 2017-08-10 4:20 ` Minchan Kim 2017-08-11 13:30 ` [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem Peter Zijlstra 2017-08-11 13:30 ` Peter Zijlstra 2017-08-13 6:14 ` Nadav Amit 2017-08-13 12:08 ` Peter Zijlstra 2017-08-13 12:08 ` Peter Zijlstra 2017-08-13 12:08 ` Peter Zijlstra 2017-08-14 1:26 ` Minchan Kim 2017-08-14 1:26 ` Minchan Kim 2017-08-14 1:26 ` Minchan Kim 2017-08-02 0:08 ` Nadav Amit [this message] 2017-08-02 0:08 ` [PATCH v6 7/7] mm: fix KSM data corruption Nadav Amit 2017-08-02 23:26 ` [PATCH v6 0/7] fixes of TLB batching races Minchan Kim 2017-08-02 23:26 ` Minchan Kim
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170802000818.4760-8-namit@vmware.com \ --to=namit@vmware.com \ --cc=akpm@linux-foundation.org \ --cc=hughd@google.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mgorman@techsingularity.net \ --cc=minchan@kernel.org \ --cc=nadav.amit@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.