From: Alexander Potapenko <glider@google.com>
To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org
Cc: kernel-hardening@lists.openwall.com,
Masahiro Yamada <yamada.masahiro@socionext.com>,
James Morris <jmorris@namei.org>,
"Serge E. Hallyn" <serge@hallyn.com>,
Nick Desaulniers <ndesaulniers@google.com>,
Kostya Serebryany <kcc@google.com>,
Dmitry Vyukov <dvyukov@google.com>,
Sandeep Patil <sspatil@android.com>,
Laura Abbott <labbott@redhat.com>,
Randy Dunlap <rdunlap@infradead.org>,
Jann Horn <jannh@google.com>,
Mark Rutland <mark.rutland@arm.com>,
linux-mm@kvack.org, linux-security-module@vger.kernel.org
Subject: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
Date: Tue, 14 May 2019 16:35:37 +0200 [thread overview]
Message-ID: <20190514143537.10435-5-glider@google.com> (raw)
In-Reply-To: <20190514143537.10435-1-glider@google.com>
Add sock_alloc_send_pskb_noinit(), which is similar to
sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT.
This helps reduce the slowdown on hackbench in the init_on_alloc mode
from 6.84% to 3.45%.
Slowdown for the initialization features compared to init_on_free=0,
init_on_alloc=0:
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%)
hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
Linux build with -j12, init_on_free=1: +8.34% wall time (st.err 0.39%)
Linux build with -j12, init_on_free=1: +24.13% sys time (st.err 0.47%)
Linux build with -j12, init_on_alloc=1: -0.04% wall time (st.err 0.46%)
Linux build with -j12, init_on_alloc=1: +0.50% sys time (st.err 0.45%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the
baseline is within the standard error.
Signed-off-by: Alexander Potapenko <glider@google.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: Christoph Lameter <cl@linux.com>
To: Kees Cook <keescook@chromium.org>
Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: James Morris <jmorris@namei.org>
Cc: "Serge E. Hallyn" <serge@hallyn.com>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Kostya Serebryany <kcc@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Sandeep Patil <sspatil@android.com>
Cc: Laura Abbott <labbott@redhat.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Jann Horn <jannh@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-mm@kvack.org
Cc: linux-security-module@vger.kernel.org
Cc: kernel-hardening@lists.openwall.com
---
v2:
- changed __GFP_NOINIT to __GFP_NO_AUTOINIT
---
include/net/sock.h | 5 +++++
net/core/sock.c | 29 +++++++++++++++++++++++++----
net/unix/af_unix.c | 13 +++++++------
3 files changed, 37 insertions(+), 10 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index 4d208c0f9c14..0dcb90a0c14d 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1626,6 +1626,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
unsigned long data_len, int noblock,
int *errcode, int max_page_order);
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+ unsigned long header_len,
+ unsigned long data_len,
+ int noblock, int *errcode,
+ int max_page_order);
void *sock_kmalloc(struct sock *sk, int size, gfp_t priority);
void sock_kfree_s(struct sock *sk, void *mem, int size);
void sock_kzfree_s(struct sock *sk, void *mem, int size);
diff --git a/net/core/sock.c b/net/core/sock.c
index 9ceb90c875bc..7c24b70b7069 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2192,9 +2192,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
* Generic send/receive buffer handlers
*/
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
- unsigned long data_len, int noblock,
- int *errcode, int max_page_order)
+struct sk_buff *sock_alloc_send_pskb_internal(struct sock *sk,
+ unsigned long header_len,
+ unsigned long data_len,
+ int noblock, int *errcode,
+ int max_page_order, gfp_t gfp)
{
struct sk_buff *skb;
long timeo;
@@ -2223,7 +2225,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
timeo = sock_wait_for_wmem(sk, timeo);
}
skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
- errcode, sk->sk_allocation);
+ errcode, sk->sk_allocation | gfp);
if (skb)
skb_set_owner_w(skb, sk);
return skb;
@@ -2234,8 +2236,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
*errcode = err;
return NULL;
}
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order)
+{
+ return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+ noblock, errcode, max_page_order, /*gfp*/0);
+}
EXPORT_SYMBOL(sock_alloc_send_pskb);
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+ unsigned long header_len,
+ unsigned long data_len,
+ int noblock, int *errcode,
+ int max_page_order)
+{
+ return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+ noblock, errcode, max_page_order, /*gfp*/__GFP_NO_AUTOINIT);
+}
+EXPORT_SYMBOL(sock_alloc_send_pskb_noinit);
+
struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
int noblock, int *errcode)
{
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index e68d7454f2e3..a4c15620b66d 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg,
BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE);
}
- skb = sock_alloc_send_pskb(sk, len - data_len, data_len,
- msg->msg_flags & MSG_DONTWAIT, &err,
- PAGE_ALLOC_COSTLY_ORDER);
+ skb = sock_alloc_send_pskb_noinit(sk, len - data_len, data_len,
+ msg->msg_flags & MSG_DONTWAIT, &err,
+ PAGE_ALLOC_COSTLY_ORDER);
if (skb == NULL)
goto out;
@@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg,
data_len = min_t(size_t, size, PAGE_ALIGN(data_len));
- skb = sock_alloc_send_pskb(sk, size - data_len, data_len,
- msg->msg_flags & MSG_DONTWAIT, &err,
- get_order(UNIX_SKB_FRAGS_SZ));
+ skb = sock_alloc_send_pskb_noinit(sk, size - data_len, data_len,
+ msg->msg_flags & MSG_DONTWAIT,
+ &err,
+ get_order(UNIX_SKB_FRAGS_SZ));
if (!skb)
goto out_err;
--
2.21.0.1020.gf2820cf01a-goog
next prev parent reply other threads:[~2019-05-14 14:36 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20190514143537.10435-1-glider@google.com>
2019-05-14 14:35 ` [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options Alexander Potapenko
2019-05-16 16:19 ` Kees Cook
2019-05-16 16:42 ` Alexander Potapenko
2019-05-16 17:03 ` Kees Cook
2019-05-17 1:26 ` Kees Cook
2019-05-17 14:38 ` Alexander Potapenko
2019-05-17 14:04 ` Michal Hocko
2019-05-17 14:11 ` Alexander Potapenko
2019-05-17 14:20 ` Michal Hocko
2019-05-17 16:36 ` Kees Cook
2019-05-17 17:11 ` Michal Hocko
2019-05-14 14:35 ` [PATCH v2 2/4] lib: introduce test_meminit module Alexander Potapenko
2019-05-16 1:02 ` Kees Cook
2019-05-17 15:51 ` Alexander Potapenko
2019-05-17 16:37 ` Kees Cook
2019-05-14 14:35 ` [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT Alexander Potapenko
2019-05-17 12:59 ` Michal Hocko
2019-05-17 13:18 ` Alexander Potapenko
2019-05-17 13:25 ` Michal Hocko
2019-05-17 13:37 ` Alexander Potapenko
2019-05-17 14:01 ` Michal Hocko
2019-05-17 16:27 ` Kees Cook
2019-05-17 17:11 ` Michal Hocko
2019-05-21 14:18 ` Alexander Potapenko
2019-05-21 14:25 ` Michal Hocko
2019-05-14 14:35 ` Alexander Potapenko [this message]
2019-05-16 16:53 ` [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations Kees Cook
2019-05-17 0:26 ` Kees Cook
2019-05-17 8:49 ` Alexander Potapenko
2019-05-17 13:50 ` Alexander Potapenko
2019-05-17 16:13 ` Kees Cook
2019-05-17 0:50 ` [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches Kees Cook
2019-05-17 8:34 ` Alexander Potapenko
2019-05-17 15:59 ` Kees Cook
2019-05-20 6:10 ` Mathias Krause
2019-05-20 16:12 ` Kees Cook
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190514143537.10435-5-glider@google.com \
--to=glider@google.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=dvyukov@google.com \
--cc=jannh@google.com \
--cc=jmorris@namei.org \
--cc=kcc@google.com \
--cc=keescook@chromium.org \
--cc=kernel-hardening@lists.openwall.com \
--cc=labbott@redhat.com \
--cc=linux-mm@kvack.org \
--cc=linux-security-module@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=ndesaulniers@google.com \
--cc=rdunlap@infradead.org \
--cc=serge@hallyn.com \
--cc=sspatil@android.com \
--cc=yamada.masahiro@socionext.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).