From: Nadav Amit <namit@vmware.com> To: <linux-mm@kvack.org> Cc: <sergey.senozhatsky@gmail.com>, <minchan@kernel.org>, <nadav.amit@gmail.com>, <mgorman@suse.de>, <riel@redhat.com>, <luto@kernel.org>, <stable@vger.kernel.org>, Nadav Amit <namit@vmware.com> Subject: [PATCH v3 1/2] mm: migrate: prevent racy access to tlb_flush_pending Date: Thu, 27 Jul 2017 04:40:14 -0700 [thread overview] Message-ID: <20170727114015.3452-2-namit@vmware.com> (raw) In-Reply-To: <20170727114015.3452-1-namit@vmware.com> From: Nadav Amit <nadav.amit@gmail.com> Setting and clearing mm->tlb_flush_pending can be performed by multiple threads, since mmap_sem may only be acquired for read in task_numa_work(). If this happens, tlb_flush_pending might be cleared while one of the threads still changes PTEs and batches TLB flushes. This can lead to the same race between migration and change_protection_range() that led to the introduction of tlb_flush_pending. The result of this race was data corruption, which means that this patch also addresses a theoretically possible data corruption. An actual data corruption was not observed, yet the race was was confirmed by adding assertion to check tlb_flush_pending is not set by two threads, adding artificial latency in change_protection_range() and using sysctl to reduce kernel.numa_balancing_scan_delay_ms. Fixes: 20841405940e ("mm: fix TLB flush race between migration, and change_protection_range") Cc: stable@vger.kernel.org Signed-off-by: Nadav Amit <namit@vmware.com> Acked-by: Mel Gorman <mgorman@suse.de> --- include/linux/mm_types.h | 8 ++++---- kernel/fork.c | 4 +++- mm/debug.c | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 45cdb27791a3..36f4ec589544 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -493,7 +493,7 @@ struct mm_struct { * can move process memory needs to flush the TLB when moving a * PROT_NONE or PROT_NUMA mapped page. */ - bool tlb_flush_pending; + atomic_t tlb_flush_pending; #endif struct uprobes_state uprobes_state; #ifdef CONFIG_HUGETLB_PAGE @@ -528,11 +528,11 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) static inline bool mm_tlb_flush_pending(struct mm_struct *mm) { barrier(); - return mm->tlb_flush_pending; + return atomic_read(&mm->tlb_flush_pending) > 0; } static inline void set_tlb_flush_pending(struct mm_struct *mm) { - mm->tlb_flush_pending = true; + atomic_inc(&mm->tlb_flush_pending); /* * Guarantee that the tlb_flush_pending store does not leak into the @@ -544,7 +544,7 @@ static inline void set_tlb_flush_pending(struct mm_struct *mm) static inline void clear_tlb_flush_pending(struct mm_struct *mm) { barrier(); - mm->tlb_flush_pending = false; + atomic_dec(&mm->tlb_flush_pending); } #else static inline bool mm_tlb_flush_pending(struct mm_struct *mm) diff --git a/kernel/fork.c b/kernel/fork.c index e53770d2bf95..d6bf35b1cf31 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -809,7 +809,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm_init_aio(mm); mm_init_owner(mm, p); mmu_notifier_mm_init(mm); - clear_tlb_flush_pending(mm); +#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_COMPACTION) + atomic_set(&mm->tlb_flush_pending, 0); +#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte = NULL; #endif diff --git a/mm/debug.c b/mm/debug.c index db1cd26d8752..d70103bb4731 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -159,7 +159,7 @@ void dump_mm(const struct mm_struct *mm) mm->numa_next_scan, mm->numa_scan_offset, mm->numa_scan_seq, #endif #if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_COMPACTION) - mm->tlb_flush_pending, + atomic_read(&mm->tlb_flush_pending), #endif mm->def_flags, &mm->def_flags ); -- 2.11.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Nadav Amit <namit@vmware.com> To: linux-mm@kvack.org Cc: sergey.senozhatsky@gmail.com, minchan@kernel.org, nadav.amit@gmail.com, mgorman@suse.de, riel@redhat.com, luto@kernel.org, stable@vger.kernel.org, Nadav Amit <namit@vmware.com> Subject: [PATCH v3 1/2] mm: migrate: prevent racy access to tlb_flush_pending Date: Thu, 27 Jul 2017 04:40:14 -0700 [thread overview] Message-ID: <20170727114015.3452-2-namit@vmware.com> (raw) In-Reply-To: <20170727114015.3452-1-namit@vmware.com> From: Nadav Amit <nadav.amit@gmail.com> Setting and clearing mm->tlb_flush_pending can be performed by multiple threads, since mmap_sem may only be acquired for read in task_numa_work(). If this happens, tlb_flush_pending might be cleared while one of the threads still changes PTEs and batches TLB flushes. This can lead to the same race between migration and change_protection_range() that led to the introduction of tlb_flush_pending. The result of this race was data corruption, which means that this patch also addresses a theoretically possible data corruption. An actual data corruption was not observed, yet the race was was confirmed by adding assertion to check tlb_flush_pending is not set by two threads, adding artificial latency in change_protection_range() and using sysctl to reduce kernel.numa_balancing_scan_delay_ms. Fixes: 20841405940e ("mm: fix TLB flush race between migration, and change_protection_range") Cc: stable@vger.kernel.org Signed-off-by: Nadav Amit <namit@vmware.com> Acked-by: Mel Gorman <mgorman@suse.de> --- include/linux/mm_types.h | 8 ++++---- kernel/fork.c | 4 +++- mm/debug.c | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 45cdb27791a3..36f4ec589544 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -493,7 +493,7 @@ struct mm_struct { * can move process memory needs to flush the TLB when moving a * PROT_NONE or PROT_NUMA mapped page. */ - bool tlb_flush_pending; + atomic_t tlb_flush_pending; #endif struct uprobes_state uprobes_state; #ifdef CONFIG_HUGETLB_PAGE @@ -528,11 +528,11 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) static inline bool mm_tlb_flush_pending(struct mm_struct *mm) { barrier(); - return mm->tlb_flush_pending; + return atomic_read(&mm->tlb_flush_pending) > 0; } static inline void set_tlb_flush_pending(struct mm_struct *mm) { - mm->tlb_flush_pending = true; + atomic_inc(&mm->tlb_flush_pending); /* * Guarantee that the tlb_flush_pending store does not leak into the @@ -544,7 +544,7 @@ static inline void set_tlb_flush_pending(struct mm_struct *mm) static inline void clear_tlb_flush_pending(struct mm_struct *mm) { barrier(); - mm->tlb_flush_pending = false; + atomic_dec(&mm->tlb_flush_pending); } #else static inline bool mm_tlb_flush_pending(struct mm_struct *mm) diff --git a/kernel/fork.c b/kernel/fork.c index e53770d2bf95..d6bf35b1cf31 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -809,7 +809,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm_init_aio(mm); mm_init_owner(mm, p); mmu_notifier_mm_init(mm); - clear_tlb_flush_pending(mm); +#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_COMPACTION) + atomic_set(&mm->tlb_flush_pending, 0); +#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte = NULL; #endif diff --git a/mm/debug.c b/mm/debug.c index db1cd26d8752..d70103bb4731 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -159,7 +159,7 @@ void dump_mm(const struct mm_struct *mm) mm->numa_next_scan, mm->numa_scan_offset, mm->numa_scan_seq, #endif #if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_COMPACTION) - mm->tlb_flush_pending, + atomic_read(&mm->tlb_flush_pending), #endif mm->def_flags, &mm->def_flags ); -- 2.11.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-07-27 11:40 UTC|newest] Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-07-27 11:40 [PATCH v3 0/2] mm: fixes of tlb_flush_pending races Nadav Amit 2017-07-27 11:40 ` Nadav Amit [this message] 2017-07-27 11:40 ` [PATCH v3 1/2] mm: migrate: prevent racy access to tlb_flush_pending Nadav Amit 2017-07-28 1:34 ` Sergey Senozhatsky 2017-07-28 1:44 ` Nadav Amit 2017-07-28 1:42 ` Sergey Senozhatsky 2017-07-28 2:28 ` Rik van Riel 2017-07-27 11:40 ` [PATCH v3 2/2] mm: migrate: fix barriers around tlb_flush_pending Nadav Amit 2017-07-28 7:42 ` Mel Gorman 2017-07-28 16:40 ` Nadav Amit
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170727114015.3452-2-namit@vmware.com \ --to=namit@vmware.com \ --cc=linux-mm@kvack.org \ --cc=luto@kernel.org \ --cc=mgorman@suse.de \ --cc=minchan@kernel.org \ --cc=nadav.amit@gmail.com \ --cc=riel@redhat.com \ --cc=sergey.senozhatsky@gmail.com \ --cc=stable@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.