From: Yang Shi <yang.shi@linux.alibaba.com>
To: mhocko@kernel.org, willy@infradead.org,
ldufour@linux.vnet.ibm.com, akpm@linux-foundation.org,
peterz@infradead.org, mingo@redhat.com, acme@kernel.org,
alexander.shishkin@linux.intel.com, jolsa@redhat.com,
namhyung@kernel.org
Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Subject: [RFC v2 PATCH 2/2] mm: mmap: zap pages with read mmap_sem for large mapping
Date: Tue, 19 Jun 2018 07:34:16 +0800 [thread overview]
Message-ID: <1529364856-49589-3-git-send-email-yang.shi@linux.alibaba.com> (raw)
In-Reply-To: <1529364856-49589-1-git-send-email-yang.shi@linux.alibaba.com>
When running some mmap/munmap scalability tests with large memory (i.e.
> 300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: G E 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D 0 14018 1 0x00000004
ffff885582f84000 ffff885e8682f000 ffff880972943000 ffff885ebf499bc0
ffff8828ee120000 ffffc900349bfca8 ffffffff817154d0 0000000000000040
00ffffff812f872a ffff885ebf499bc0 024000d000948300 ffff880972943000
Call Trace:
[<ffffffff817154d0>] ? __schedule+0x250/0x730
[<ffffffff817159e6>] schedule+0x36/0x80
[<ffffffff81718560>] rwsem_down_read_failed+0xf0/0x150
[<ffffffff81390a28>] call_rwsem_down_read_failed+0x18/0x30
[<ffffffff81717db0>] down_read+0x20/0x40
[<ffffffff812b9439>] proc_pid_cmdline_read+0xd9/0x4e0
[<ffffffff81253c95>] ? do_filp_open+0xa5/0x100
[<ffffffff81241d87>] __vfs_read+0x37/0x150
[<ffffffff812f824b>] ? security_file_permission+0x9b/0xc0
[<ffffffff81242266>] vfs_read+0x96/0x130
[<ffffffff812437b5>] SyS_read+0x55/0xc0
[<ffffffff8171a6da>] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem from very beginning to all the way
down to the end, and doesn't release it in the middle. When unmapping
large mapping, it may take long time (take ~18 seconds to unmap 320GB
mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hock [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to manipulate vmas.
Define large mapping size thresh as PUD size or 1GB, just zap pages with
read mmap_sem for mappings which are >= thresh value.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, then just
fallback to regular path since unmapping those mappings need acquire
write mmap_sem.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites remain intact since it sounds the
complexity to handle race conditions outpace the benefits.
The below is some regression and performance data collected on a machine
with 32 cores of E5-2680 @ 2.70GHz and 384GB memory.
With the patched kernel, write mmap_sem hold time is dropped to us level
from second.
Throughput of page faults (#/s) with the below stress-ng test:
stress-ng --mmap 0 --mmap-bytes 80G --mmap-file --metrics --perf
--timeout 600s
pristine patched delta
89.41K/sec 97.29K/sec +8.8%
[1] https://lwn.net/Articles/753269/
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Yang Shi <yang.shi@linux.alibaba.com>
---
mm/mmap.c | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 147 insertions(+), 1 deletion(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index fc41c05..e84f80c 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2686,6 +2686,141 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
return __split_vma(mm, vma, addr, new_below);
}
+/* Consider PUD size or 1GB mapping as large mapping */
+#ifdef HPAGE_PUD_SIZE
+#define LARGE_MAP_THRESH HPAGE_PUD_SIZE
+#else
+#define LARGE_MAP_THRESH (1 * 1024 * 1024 * 1024)
+#endif
+
+/* Unmap large mapping early with acquiring read mmap_sem */
+static int do_munmap_zap_early(struct mm_struct *mm, unsigned long start,
+ size_t len, struct list_head *uf)
+{
+ unsigned long end = 0;
+ struct vm_area_struct *vma = NULL, *prev, *last, *tmp;
+ bool success = false;
+ int ret = 0;
+
+ if ((offset_in_page(start)) || start > TASK_SIZE || len > TASK_SIZE - start)
+ return -EINVAL;
+
+ len = (PAGE_ALIGN(len));
+ if (len == 0)
+ return -EINVAL;
+
+ /* Just deal with uf in regular path */
+ if (unlikely(uf))
+ goto regular_path;
+
+ if (len >= LARGE_MAP_THRESH) {
+ down_read(&mm->mmap_sem);
+ vma = find_vma(mm, start);
+ if (!vma) {
+ up_read(&mm->mmap_sem);
+ return 0;
+ }
+
+ prev = vma->vm_prev;
+
+ end = start + len;
+ if (vma->vm_start > end) {
+ up_read(&mm->mmap_sem);
+ return 0;
+ }
+
+ if (start > vma->vm_start) {
+ int error;
+
+ if (end < vma->vm_end &&
+ mm->map_count > sysctl_max_map_count) {
+ up_read(&mm->mmap_sem);
+ return -ENOMEM;
+ }
+
+ error = __split_vma(mm, vma, start, 0);
+ if (error) {
+ up_read(&mm->mmap_sem);
+ return error;
+ }
+ prev = vma;
+ }
+
+ last = find_vma(mm, end);
+ if (last && end > last->vm_start) {
+ int error = __split_vma(mm, last, end, 1);
+
+ if (error) {
+ up_read(&mm->mmap_sem);
+ return error;
+ }
+ }
+ vma = prev ? prev->vm_next : mm->mmap;
+
+ /*
+ * Unmapping vmas, which has VM_LOCKED|VM_HUGETLB|VM_PFNMAP
+ * flag set or has uprobes set, need acquire write map_sem,
+ * so skip them in early zap. Just deal with such mapping in
+ * regular path.
+ * Borrow can_madv_dontneed_vma() to check the conditions.
+ */
+ tmp = vma;
+ while (tmp && tmp->vm_start < end) {
+ if (!can_madv_dontneed_vma(tmp) ||
+ vma_has_uprobes(tmp, start, end))
+ goto sem_drop;
+ tmp = tmp->vm_next;
+ }
+
+ unmap_region(mm, vma, prev, start, end);
+ /* indicates early zap is success */
+ success = true;
+
+sem_drop:
+ up_read(&mm->mmap_sem);
+ }
+
+regular_path:
+ /* hold write mmap_sem for vma manipulation or regular path */
+ if (down_write_killable(&mm->mmap_sem))
+ return -EINTR;
+ if (success) {
+ /* vmas have been zapped, here just deal with loose end */
+ detach_vmas_to_be_unmapped(mm, vma, prev, end);
+ arch_unmap(mm, vma, start, end);
+ remove_vma_list(mm, vma);
+ } else {
+ /* vma is VM_LOCKED|VM_HUGETLB|VM_PFNMAP or has uprobe */
+ if (vma) {
+ if (unlikely(uf)) {
+ int ret = userfaultfd_unmap_prep(vma, start,
+ end, uf);
+ if (ret)
+ goto out;
+ }
+ if (mm->locked_vm) {
+ tmp = vma;
+ while (tmp && tmp->vm_start < end) {
+ if (tmp->vm_flags & VM_LOCKED) {
+ mm->locked_vm -= vma_pages(tmp);
+ munlock_vma_pages_all(tmp);
+ }
+ tmp = tmp->vm_next;
+ }
+ }
+ detach_vmas_to_be_unmapped(mm, vma, prev, end);
+ unmap_region(mm, vma, prev, start, end);
+ remove_vma_list(mm, vma);
+ } else
+ /* When mapping size < LARGE_MAP_THRESH */
+ ret = do_munmap(mm, start, len, uf);
+ }
+
+out:
+ up_write(&mm->mmap_sem);
+ return ret;
+}
+
/* Munmap is split into 2 main parts -- this part which finds
* what needs doing, and the areas themselves, which do the
* work. This now handles partial unmappings.
@@ -2792,6 +2927,17 @@ int do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
return 0;
}
+static int vm_munmap_zap_early(unsigned long start, size_t len)
+{
+ int ret;
+ struct mm_struct *mm = current->mm;
+ LIST_HEAD(uf);
+
+ ret = do_munmap_zap_early(mm, start, len, &uf);
+ userfaultfd_unmap_complete(mm, &uf);
+ return ret;
+}
+
int vm_munmap(unsigned long start, size_t len)
{
int ret;
@@ -2811,7 +2957,7 @@ int vm_munmap(unsigned long start, size_t len)
SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len)
{
profile_munmap(addr);
- return vm_munmap(addr, len);
+ return vm_munmap_zap_early(addr, len);
}
--
1.8.3.1
next prev parent reply other threads:[~2018-06-18 23:34 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-18 23:34 [RFC v2 0/2] mm: zap pages with read mmap_sem in munmap for large mapping Yang Shi
2018-06-18 23:34 ` [RFC v2 PATCH 1/2] uprobes: make vma_has_uprobes non-static Yang Shi
2018-06-18 23:34 ` Yang Shi [this message]
2018-06-19 10:02 ` [RFC v2 PATCH 2/2] mm: mmap: zap pages with read mmap_sem for large mapping Peter Zijlstra
2018-06-19 21:13 ` Yang Shi
2018-06-20 7:17 ` Michal Hocko
2018-06-20 16:23 ` Yang Shi
2018-06-19 22:17 ` Nadav Amit
[not found] ` <158a4e4c-d290-77c4-a595-71332ede392b@linux.alibaba.com>
2018-06-20 0:31 ` Nadav Amit
2018-06-20 7:18 ` Michal Hocko
2018-06-20 17:12 ` Nadav Amit
2018-06-20 18:42 ` Yang Shi
2018-06-23 1:01 ` Yang Shi
2018-06-25 9:14 ` Michal Hocko
2018-06-26 0:06 ` Yang Shi
2018-06-26 7:43 ` Peter Zijlstra
2018-06-27 1:03 ` Yang Shi
2018-06-27 7:24 ` Michal Hocko
2018-06-27 17:23 ` Yang Shi
2018-06-28 11:51 ` Michal Hocko
2018-06-28 19:10 ` Yang Shi
2018-06-29 0:59 ` Yang Shi
2018-06-29 11:39 ` Michal Hocko
2018-06-29 16:50 ` Yang Shi
2018-06-29 11:34 ` Michal Hocko
2018-06-29 16:45 ` Yang Shi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1529364856-49589-3-git-send-email-yang.shi@linux.alibaba.com \
--to=yang.shi@linux.alibaba.com \
--cc=acme@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=jolsa@redhat.com \
--cc=ldufour@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).