From: Lokesh Gidra <lokeshgidra@google.com>
To: akpm@linux-foundation.org
Cc: lokeshgidra@google.com, linux-fsdevel@vger.kernel.org,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
selinux@vger.kernel.org, surenb@google.com,
kernel-team@android.com, aarcange@redhat.com, peterx@redhat.com,
david@redhat.com, axelrasmussen@google.com, bgeffon@google.com,
willy@infradead.org, jannh@google.com, kaleshsingh@google.com,
ngeoffray@google.com, timmurray@google.com, rppt@kernel.org,
Liam.Howlett@oracle.com
Subject: [PATCH v5 0/3] per-vma locks in userfaultfd
Date: Mon, 12 Feb 2024 16:19:17 -0800 [thread overview]
Message-ID: <20240213001920.3551772-1-lokeshgidra@google.com> (raw)
Performing userfaultfd operations (like copy/move etc.) in critical
section of mmap_lock (read-mode) causes significant contention on the
lock when operations requiring the lock in write-mode are taking place
concurrently. We can use per-vma locks instead to significantly reduce
the contention issue.
Android runtime's Garbage Collector uses userfaultfd for concurrent
compaction. mmap-lock contention during compaction potentially causes
jittery experience for the user. During one such reproducible scenario,
we observed the following improvements with this patch-set:
- Wall clock time of compaction phase came down from ~3s to <500ms
- Uninterruptible sleep time (across all threads in the process) was
~10ms (none in mmap_lock) during compaction, instead of >20s
Changes since v4 [4]:
- Fix possible deadlock in find_and_lock_vmas() which may arise if
lock_vma() is used for both src and dst vmas.
- Ensure we lock vma only once if src and dst vmas are same.
- Fix error handling in move_pages() after successfully locking vmas.
- Introduce helper function for finding dst vma and preparing its
anon_vma when done in mmap_lock critical section, per Liam Howlett.
- Introduce helper function for finding dst and src vmas when done in
mmap_lock critical section.
Changes since v3 [3]:
- Rename function names to clearly reflect which lock is being taken,
per Liam Howlett.
- Have separate functions and abstractions in mm/userfaultfd.c to avoid
confusion around which lock is being acquired/released, per Liam Howlett.
- Prepare anon_vma for all private vmas, anonymous or file-backed,
per Jann Horn.
Changes since v2 [2]:
- Implement and use lock_vma() which uses mmap_lock critical section
to lock the VMA using per-vma lock if lock_vma_under_rcu() fails,
per Liam R. Howlett. This helps simplify the code and also avoids
performing the entire userfaultfd operation under mmap_lock.
Changes since v1 [1]:
- rebase patches on 'mm-unstable' branch
[1] https://lore.kernel.org/all/20240126182647.2748949-1-lokeshgidra@google.com/
[2] https://lore.kernel.org/all/20240129193512.123145-1-lokeshgidra@google.com/
[3] https://lore.kernel.org/all/20240206010919.1109005-1-lokeshgidra@google.com/
[4] https://lore.kernel.org/all/20240208212204.2043140-1-lokeshgidra@google.com/
Lokesh Gidra (3):
userfaultfd: move userfaultfd_ctx struct to header file
userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx
userfaultfd: use per-vma locks in userfaultfd operations
fs/userfaultfd.c | 86 ++-----
include/linux/userfaultfd_k.h | 75 ++++--
mm/userfaultfd.c | 450 ++++++++++++++++++++++++++--------
3 files changed, 421 insertions(+), 190 deletions(-)
--
2.43.0.687.g38aa6559b0-goog
next reply other threads:[~2024-02-13 0:19 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-13 0:19 Lokesh Gidra [this message]
2024-02-13 0:19 ` [PATCH v5 1/3] userfaultfd: move userfaultfd_ctx struct to header file Lokesh Gidra
2024-02-13 0:19 ` [PATCH v5 2/3] userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx Lokesh Gidra
2024-02-13 0:19 ` [PATCH v5 3/3] userfaultfd: use per-vma locks in userfaultfd operations Lokesh Gidra
2024-02-13 3:33 ` Liam R. Howlett
2024-02-13 11:24 ` Lokesh Gidra
2024-02-13 17:06 ` Liam R. Howlett
2024-02-13 18:14 ` Lokesh Gidra
2024-02-13 18:25 ` Suren Baghdasaryan
2024-02-13 18:49 ` Liam R. Howlett
2024-02-13 18:56 ` Suren Baghdasaryan
2024-02-13 19:18 ` Lokesh Gidra
2024-02-13 19:27 ` Liam R. Howlett
2024-02-13 19:30 ` Suren Baghdasaryan
2024-02-13 19:36 ` Lokesh Gidra
2024-02-13 19:51 ` Liam R. Howlett
2024-02-13 19:55 ` Lokesh Gidra
2024-02-13 20:40 ` Liam R. Howlett
2024-02-13 19:21 ` Liam R. Howlett
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240213001920.3551772-1-lokeshgidra@google.com \
--to=lokeshgidra@google.com \
--cc=Liam.Howlett@oracle.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=bgeffon@google.com \
--cc=david@redhat.com \
--cc=jannh@google.com \
--cc=kaleshsingh@google.com \
--cc=kernel-team@android.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ngeoffray@google.com \
--cc=peterx@redhat.com \
--cc=rppt@kernel.org \
--cc=selinux@vger.kernel.org \
--cc=surenb@google.com \
--cc=timmurray@google.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.