From: Nadav Amit <namit@vmware.com> To: <linux-mm@kvack.org> Cc: <nadav.amit@gmail.com>, <linux-kernel@vger.kernel.org>, <akpm@linux-foundation.org>, Nadav Amit <namit@vmware.com>, Minchan Kim <minchan@kernel.org>, Sergey Senozhatsky <sergey.senozhatsky@gmail.com>, Andy Lutomirski <luto@kernel.org>, Mel Gorman <mgorman@suse.de> Subject: [PATCH v6 2/7] mm: migrate: fix barriers around tlb_flush_pending Date: Tue, 1 Aug 2017 17:08:13 -0700 [thread overview] Message-ID: <20170802000818.4760-3-namit@vmware.com> (raw) In-Reply-To: <20170802000818.4760-1-namit@vmware.com> Reading tlb_flush_pending while the page-table lock is taken does not require a barrier, since the lock/unlock already acts as a barrier. Removing the barrier in mm_tlb_flush_pending() to address this issue. However, migrate_misplaced_transhuge_page() calls mm_tlb_flush_pending() while the page-table lock is already released, which may present a problem on architectures with weak memory model (PPC). To deal with this case, a new parameter is added to mm_tlb_flush_pending() to indicate if it is read without the page-table lock taken, and calling smp_mb__after_unlock_lock() in this case. Cc: Minchan Kim <minchan@kernel.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Mel Gorman <mgorman@suse.de> Signed-off-by: Nadav Amit <namit@vmware.com> Acked-by: Rik van Riel <riel@redhat.com> --- include/linux/mm_types.h | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index f5263dd0f1bc..2956513619a7 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -522,12 +522,12 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) /* * Memory barriers to keep this state in sync are graciously provided by * the page table locks, outside of which no page table modifications happen. - * The barriers below prevent the compiler from re-ordering the instructions - * around the memory barriers that are already present in the code. + * The barriers are used to ensure the order between tlb_flush_pending updates, + * which happen while the lock is not taken, and the PTE updates, which happen + * while the lock is taken, are serialized. */ static inline bool mm_tlb_flush_pending(struct mm_struct *mm) { - barrier(); return atomic_read(&mm->tlb_flush_pending) > 0; } @@ -550,7 +550,13 @@ static inline void inc_tlb_flush_pending(struct mm_struct *mm) /* Clearing is done after a TLB flush, which also provides a barrier. */ static inline void dec_tlb_flush_pending(struct mm_struct *mm) { - barrier(); + /* + * Guarantee that the tlb_flush_pending does not not leak into the + * critical section, since we must order the PTE change and changes to + * the pending TLB flush indication. We could have relied on TLB flush + * as a memory barrier, but this behavior is not clearly documented. + */ + smp_mb__before_atomic(); atomic_dec(&mm->tlb_flush_pending); } #else -- 2.11.0
WARNING: multiple messages have this Message-ID (diff)
From: Nadav Amit <namit@vmware.com> To: linux-mm@kvack.org Cc: nadav.amit@gmail.com, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, Nadav Amit <namit@vmware.com>, Minchan Kim <minchan@kernel.org>, Sergey Senozhatsky <sergey.senozhatsky@gmail.com>, Andy Lutomirski <luto@kernel.org>, Mel Gorman <mgorman@suse.de> Subject: [PATCH v6 2/7] mm: migrate: fix barriers around tlb_flush_pending Date: Tue, 1 Aug 2017 17:08:13 -0700 [thread overview] Message-ID: <20170802000818.4760-3-namit@vmware.com> (raw) In-Reply-To: <20170802000818.4760-1-namit@vmware.com> Reading tlb_flush_pending while the page-table lock is taken does not require a barrier, since the lock/unlock already acts as a barrier. Removing the barrier in mm_tlb_flush_pending() to address this issue. However, migrate_misplaced_transhuge_page() calls mm_tlb_flush_pending() while the page-table lock is already released, which may present a problem on architectures with weak memory model (PPC). To deal with this case, a new parameter is added to mm_tlb_flush_pending() to indicate if it is read without the page-table lock taken, and calling smp_mb__after_unlock_lock() in this case. Cc: Minchan Kim <minchan@kernel.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Mel Gorman <mgorman@suse.de> Signed-off-by: Nadav Amit <namit@vmware.com> Acked-by: Rik van Riel <riel@redhat.com> --- include/linux/mm_types.h | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index f5263dd0f1bc..2956513619a7 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -522,12 +522,12 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) /* * Memory barriers to keep this state in sync are graciously provided by * the page table locks, outside of which no page table modifications happen. - * The barriers below prevent the compiler from re-ordering the instructions - * around the memory barriers that are already present in the code. + * The barriers are used to ensure the order between tlb_flush_pending updates, + * which happen while the lock is not taken, and the PTE updates, which happen + * while the lock is taken, are serialized. */ static inline bool mm_tlb_flush_pending(struct mm_struct *mm) { - barrier(); return atomic_read(&mm->tlb_flush_pending) > 0; } @@ -550,7 +550,13 @@ static inline void inc_tlb_flush_pending(struct mm_struct *mm) /* Clearing is done after a TLB flush, which also provides a barrier. */ static inline void dec_tlb_flush_pending(struct mm_struct *mm) { - barrier(); + /* + * Guarantee that the tlb_flush_pending does not not leak into the + * critical section, since we must order the PTE change and changes to + * the pending TLB flush indication. We could have relied on TLB flush + * as a memory barrier, but this behavior is not clearly documented. + */ + smp_mb__before_atomic(); atomic_dec(&mm->tlb_flush_pending); } #else -- 2.11.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-08-02 7:33 UTC|newest] Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-08-02 0:08 [PATCH v6 0/7] fixes of TLB batching races Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` [PATCH v6 1/7] mm: migrate: prevent racy access to tlb_flush_pending Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` Nadav Amit [this message] 2017-08-02 0:08 ` [PATCH v6 2/7] mm: migrate: fix barriers around tlb_flush_pending Nadav Amit 2017-08-02 0:08 ` [PATCH v6 3/7] Revert "mm: numa: defer TLB flush for THP migration as long as possible" Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-11 10:50 ` Peter Zijlstra 2017-08-11 10:50 ` Peter Zijlstra 2017-08-02 0:08 ` [PATCH v6 4/7] mm: refactoring TLB gathering API Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-11 9:23 ` Peter Zijlstra 2017-08-11 9:23 ` Peter Zijlstra 2017-08-11 17:12 ` Nadav Amit 2017-08-11 17:12 ` Nadav Amit 2017-08-14 0:49 ` Minchan Kim 2017-08-14 0:49 ` Minchan Kim 2017-08-02 0:08 ` [PATCH v6 5/7] mm: make tlb_flush_pending global Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 14:28 ` kbuild test robot 2017-08-02 14:28 ` kbuild test robot 2017-08-02 23:23 ` Minchan Kim 2017-08-02 23:23 ` Minchan Kim 2017-08-02 23:27 ` Andrew Morton 2017-08-02 23:27 ` Andrew Morton 2017-08-02 23:34 ` Minchan Kim 2017-08-02 23:34 ` Minchan Kim 2017-08-03 16:40 ` kbuild test robot 2017-08-03 16:40 ` kbuild test robot 2017-08-02 0:08 ` [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-08 1:19 ` [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 1:19 ` kernel test robot 2017-08-08 2:28 ` Minchan Kim 2017-08-08 2:28 ` Minchan Kim 2017-08-08 2:28 ` Minchan Kim 2017-08-08 4:23 ` Nadav Amit 2017-08-08 4:23 ` Nadav Amit 2017-08-08 4:23 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 5:51 ` Nadav Amit 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:08 ` Minchan Kim 2017-08-08 8:16 ` Nadav Amit 2017-08-08 8:16 ` Nadav Amit 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 1:25 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-09 2:59 ` Ye Xiaolong 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:13 ` Minchan Kim 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:14 ` Nadav Amit 2017-08-10 4:20 ` Minchan Kim 2017-08-10 4:20 ` Minchan Kim 2017-08-10 4:20 ` Minchan Kim 2017-08-11 13:30 ` [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem Peter Zijlstra 2017-08-11 13:30 ` Peter Zijlstra 2017-08-13 6:14 ` Nadav Amit 2017-08-13 12:08 ` Peter Zijlstra 2017-08-13 12:08 ` Peter Zijlstra 2017-08-13 12:08 ` Peter Zijlstra 2017-08-14 1:26 ` Minchan Kim 2017-08-14 1:26 ` Minchan Kim 2017-08-14 1:26 ` Minchan Kim 2017-08-02 0:08 ` [PATCH v6 7/7] mm: fix KSM data corruption Nadav Amit 2017-08-02 0:08 ` Nadav Amit 2017-08-02 23:26 ` [PATCH v6 0/7] fixes of TLB batching races Minchan Kim 2017-08-02 23:26 ` Minchan Kim
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170802000818.4760-3-namit@vmware.com \ --to=namit@vmware.com \ --cc=akpm@linux-foundation.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=luto@kernel.org \ --cc=mgorman@suse.de \ --cc=minchan@kernel.org \ --cc=nadav.amit@gmail.com \ --cc=sergey.senozhatsky@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.