From: Peter Xu <peterx@redhat.com>
To: linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: Jan Kara <jack@suse.cz>, John Hubbard <jhubbard@nvidia.com>,
peterx@redhat.com, Linus Torvalds <torvalds@linux-foundation.org>,
Michal Hocko <mhocko@suse.com>,
Kirill Tkhai <ktkhai@virtuozzo.com>,
Kirill Shutemov <kirill@shutemov.name>,
Oleg Nesterov <oleg@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Jann Horn <jannh@google.com>,
Andrea Arcangeli <aarcange@redhat.com>,
Jason Gunthorpe <jgg@nvidia.com>,
Matthew Wilcox <willy@infradead.org>,
Hugh Dickins <hughd@google.com>
Subject: [PATCH v2 2/3] mm: gup: allow FOLL_PIN to scale in SMP
Date: Fri, 7 May 2021 11:05:52 -0400 [thread overview]
Message-ID: <20210507150553.208763-3-peterx@redhat.com> (raw)
In-Reply-To: <20210507150553.208763-1-peterx@redhat.com>
From: Andrea Arcangeli <aarcange@redhat.com>
has_pinned cannot be written by each pin-fast or it won't scale in
SMP. This isn't "false sharing" strictly speaking (it's more like
"true non-sharing"), but it creates the same SMP scalability
bottleneck of "false sharing".
To verify the improvement, below test is done on 40 cpus host with Intel(R)
Xeon(R) CPU E5-2630 v4 @ 2.20GHz (must be with CONFIG_GUP_TEST=y):
$ sudo chrt -f 1 ./gup_test -a -m 512 -j 40
Where we can get (average value for 40 threads):
Old kernel: 477729.97 (+- 3.79%)
New kernel: 89144.65 (+-11.76%)
On a similar condition with 256 cpus, this commits increases the SMP
scalability of pin_user_pages_fast() executed by different threads of the same
process by more than 4000%.
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
[peterx: rewrite commit message, add parentheses against "(A & B)"]
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
mm/gup.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 63a079e361a3d..9933bc5c2eff2 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1292,7 +1292,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
BUG_ON(*locked != 1);
}
- if (flags & FOLL_PIN)
+ if ((flags & FOLL_PIN) && !atomic_read(&mm->has_pinned))
atomic_set(&mm->has_pinned, 1);
/*
@@ -2617,7 +2617,7 @@ static int internal_get_user_pages_fast(unsigned long start,
FOLL_FAST_ONLY)))
return -EINVAL;
- if (gup_flags & FOLL_PIN)
+ if ((gup_flags & FOLL_PIN) && !atomic_read(¤t->mm->has_pinned))
atomic_set(¤t->mm->has_pinned, 1);
if (!(gup_flags & FOLL_FAST_ONLY))
--
2.31.1
next prev parent reply other threads:[~2021-05-07 15:06 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-07 15:05 [PATCH v2 0/3] mm/gup: Fix pin page write cache bouncing on has_pinned Peter Xu
2021-05-07 15:05 ` [PATCH v2 1/3] mm/gup_benchmark: Support threading Peter Xu
2021-05-07 15:05 ` Peter Xu [this message]
2021-05-07 15:05 ` [PATCH v2 3/3] mm: gup: pack has_pinned in MMF_HAS_PINNED Peter Xu
2021-05-08 1:12 ` John Hubbard
2021-05-12 9:49 ` Geert Uytterhoeven
2021-05-12 12:34 ` Peter Xu
2021-05-12 9:49 ` Naresh Kamboju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210507150553.208763-3-peterx@redhat.com \
--to=peterx@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=hughd@google.com \
--cc=jack@suse.cz \
--cc=jannh@google.com \
--cc=jgg@nvidia.com \
--cc=jhubbard@nvidia.com \
--cc=kirill@shutemov.name \
--cc=ktkhai@virtuozzo.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=oleg@redhat.com \
--cc=torvalds@linux-foundation.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).