From: Jason Gunthorpe <jgg@ziepe.ca>
To: Jerome Glisse <jglisse@redhat.com>,
Ralph Campbell <rcampbell@nvidia.com>,
John Hubbard <jhubbard@nvidia.com>,
Felix.Kuehling@amd.com
Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org,
Andrea Arcangeli <aarcange@redhat.com>,
dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
Ben Skeggs <bskeggs@redhat.com>,
Jason Gunthorpe <jgg@mellanox.com>,
Ira Weiny <ira.weiny@intel.com>,
Philip Yang <Philip.Yang@amd.com>
Subject: [PATCH v3 hmm 04/12] mm/hmm: Simplify hmm_get_or_create and make it reliable
Date: Thu, 13 Jun 2019 21:44:42 -0300 [thread overview]
Message-ID: <20190614004450.20252-5-jgg@ziepe.ca> (raw)
In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca>
From: Jason Gunthorpe <jgg@mellanox.com>
As coded this function can false-fail in various racy situations. Make it
reliable and simpler by running under the write side of the mmap_sem and
avoiding the false-failing compare/exchange pattern. Due to the mmap_sem
this no longer has to avoid racing with a 2nd parallel
hmm_get_or_create().
Unfortunately this still has to use the page_table_lock as the
non-sleeping lock protecting mm->hmm, since the contexts where we free the
hmm are incompatible with mmap_sem.
Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Ralph Campbell <rcampbell@nvidia.com>
Reviewed-by: Ira Weiny <ira.weiny@intel.com>
Tested-by: Philip Yang <Philip.Yang@amd.com>
---
v2:
- Fix error unwind of mmgrab (Jerome)
- Use hmm local instead of 2nd container_of (Jerome)
v3:
- Can't use mmap_sem in the SRCU callback, keep using the
page_table_lock (Philip)
---
mm/hmm.c | 84 ++++++++++++++++++++++++--------------------------------
1 file changed, 36 insertions(+), 48 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 080b17a2e87e2d..4c64d4c32f4825 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -31,16 +31,6 @@
#if IS_ENABLED(CONFIG_HMM_MIRROR)
static const struct mmu_notifier_ops hmm_mmu_notifier_ops;
-static inline struct hmm *mm_get_hmm(struct mm_struct *mm)
-{
- struct hmm *hmm = READ_ONCE(mm->hmm);
-
- if (hmm && kref_get_unless_zero(&hmm->kref))
- return hmm;
-
- return NULL;
-}
-
/**
* hmm_get_or_create - register HMM against an mm (HMM internal)
*
@@ -55,11 +45,19 @@ static inline struct hmm *mm_get_hmm(struct mm_struct *mm)
*/
static struct hmm *hmm_get_or_create(struct mm_struct *mm)
{
- struct hmm *hmm = mm_get_hmm(mm);
- bool cleanup = false;
+ struct hmm *hmm;
- if (hmm)
- return hmm;
+ lockdep_assert_held_exclusive(&mm->mmap_sem);
+
+ /* Abuse the page_table_lock to also protect mm->hmm. */
+ spin_lock(&mm->page_table_lock);
+ if (mm->hmm) {
+ if (kref_get_unless_zero(&mm->hmm->kref)) {
+ spin_unlock(&mm->page_table_lock);
+ return mm->hmm;
+ }
+ }
+ spin_unlock(&mm->page_table_lock);
hmm = kmalloc(sizeof(*hmm), GFP_KERNEL);
if (!hmm)
@@ -74,57 +72,47 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm)
hmm->notifiers = 0;
hmm->dead = false;
hmm->mm = mm;
- mmgrab(hmm->mm);
- spin_lock(&mm->page_table_lock);
- if (!mm->hmm)
- mm->hmm = hmm;
- else
- cleanup = true;
- spin_unlock(&mm->page_table_lock);
+ hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops;
+ if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) {
+ kfree(hmm);
+ return NULL;
+ }
- if (cleanup)
- goto error;
+ mmgrab(hmm->mm);
/*
- * We should only get here if hold the mmap_sem in write mode ie on
- * registration of first mirror through hmm_mirror_register()
+ * We hold the exclusive mmap_sem here so we know that mm->hmm is
+ * still NULL or 0 kref, and is safe to update.
*/
- hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops;
- if (__mmu_notifier_register(&hmm->mmu_notifier, mm))
- goto error_mm;
-
- return hmm;
-
-error_mm:
spin_lock(&mm->page_table_lock);
- if (mm->hmm == hmm)
- mm->hmm = NULL;
+ mm->hmm = hmm;
spin_unlock(&mm->page_table_lock);
-error:
- mmdrop(hmm->mm);
- kfree(hmm);
- return NULL;
+ return hmm;
}
static void hmm_free_rcu(struct rcu_head *rcu)
{
- kfree(container_of(rcu, struct hmm, rcu));
+ struct hmm *hmm = container_of(rcu, struct hmm, rcu);
+
+ /*
+ * The mm->hmm pointer is kept valid while notifier ops can be running
+ * so they don't have to deal with a NULL mm->hmm value
+ */
+ spin_lock(&hmm->mm->page_table_lock);
+ if (hmm->mm->hmm == hmm)
+ hmm->mm->hmm = NULL;
+ spin_unlock(&hmm->mm->page_table_lock);
+ mmdrop(hmm->mm);
+
+ kfree(hmm);
}
static void hmm_free(struct kref *kref)
{
struct hmm *hmm = container_of(kref, struct hmm, kref);
- struct mm_struct *mm = hmm->mm;
-
- mmu_notifier_unregister_no_release(&hmm->mmu_notifier, mm);
- spin_lock(&mm->page_table_lock);
- if (mm->hmm == hmm)
- mm->hmm = NULL;
- spin_unlock(&mm->page_table_lock);
-
- mmdrop(hmm->mm);
+ mmu_notifier_unregister_no_release(&hmm->mmu_notifier, hmm->mm);
mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu);
}
--
2.21.0
next prev parent reply other threads:[~2019-06-14 0:45 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-14 0:44 [PATCH v3 hmm 00/12] mm/hmm: Various revisions from a locking/code review Jason Gunthorpe
2019-06-14 0:44 ` [PATCH v3 hmm 01/12] mm/hmm: fix use after free with struct hmm in the mmu notifiers Jason Gunthorpe
2019-06-15 13:56 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 02/12] mm/hmm: Use hmm_mirror not mm as an argument for hmm_range_register Jason Gunthorpe
2019-06-15 13:59 ` Christoph Hellwig
2019-06-18 13:05 ` Jason Gunthorpe
2019-06-19 8:14 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 03/12] mm/hmm: Hold a mmgrab from hmm to mm Jason Gunthorpe
2019-06-15 13:59 ` Christoph Hellwig
2019-06-14 0:44 ` Jason Gunthorpe [this message]
2019-06-15 14:12 ` [PATCH v3 hmm 04/12] mm/hmm: Simplify hmm_get_or_create and make it reliable Christoph Hellwig
2019-06-18 0:36 ` Jason Gunthorpe
2019-06-18 18:55 ` Jason Gunthorpe
2019-06-14 0:44 ` [PATCH v3 hmm 05/12] mm/hmm: Remove duplicate condition test before wait_event_timeout Jason Gunthorpe
2019-06-15 14:12 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 06/12] mm/hmm: Hold on to the mmget for the lifetime of the range Jason Gunthorpe
2019-06-15 14:14 ` Christoph Hellwig
2019-06-18 15:11 ` Jason Gunthorpe
2019-06-19 8:18 ` Christoph Hellwig
2019-06-19 11:34 ` Jason Gunthorpe
2019-06-19 11:54 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 07/12] mm/hmm: Use lockdep instead of comments Jason Gunthorpe
2019-06-15 14:14 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 08/12] mm/hmm: Remove racy protection against double-unregistration Jason Gunthorpe
2019-06-15 14:16 ` Christoph Hellwig
2019-06-18 13:13 ` Jason Gunthorpe
2019-06-18 13:27 ` Christoph Hellwig
2019-06-18 18:57 ` Jason Gunthorpe
2019-06-19 8:19 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 09/12] mm/hmm: Poison hmm_range during unregister Jason Gunthorpe
2019-06-15 14:17 ` Christoph Hellwig
2019-06-18 18:04 ` Jason Gunthorpe
2019-06-14 0:44 ` [PATCH v3 hmm 10/12] mm/hmm: Do not use list*_rcu() for hmm->ranges Jason Gunthorpe
2019-06-15 14:18 ` Christoph Hellwig
2019-06-18 0:38 ` Jason Gunthorpe
2019-06-14 0:44 ` [PATCH v3 hmm 11/12] mm/hmm: Remove confusing comment and logic from hmm_release Jason Gunthorpe
2019-06-15 14:21 ` Christoph Hellwig
2019-06-18 0:45 ` Jason Gunthorpe
2019-06-18 5:37 ` Christoph Hellwig
2019-06-19 0:53 ` Kuehling, Felix
2019-06-19 8:07 ` Christoph Hellwig
2019-06-19 11:56 ` Jason Gunthorpe
2019-06-19 12:03 ` Christoph Hellwig
2019-06-14 0:44 ` [PATCH v3 hmm 12/12] mm/hmm: Fix error flows in hmm_invalidate_range_start Jason Gunthorpe
2019-06-15 14:25 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190614004450.20252-5-jgg@ziepe.ca \
--to=jgg@ziepe.ca \
--cc=Felix.Kuehling@amd.com \
--cc=Philip.Yang@amd.com \
--cc=aarcange@redhat.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=bskeggs@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=ira.weiny@intel.com \
--cc=jgg@mellanox.com \
--cc=jglisse@redhat.com \
--cc=jhubbard@nvidia.com \
--cc=linux-mm@kvack.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rcampbell@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).