All of lore.kernel.org
 help / color / mirror / Atom feed
From: Brendan Jackman <jackmanb@google.com>
To: bpf@vger.kernel.org
Cc: Alexei Starovoitov <ast@kernel.org>, Yonghong Song <yhs@fb.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	KP Singh <kpsingh@chromium.org>,
	Florent Revest <revest@chromium.org>,
	linux-kernel@vger.kernel.org, Jann Horn <jannh@google.com>,
	Brendan Jackman <jackmanb@google.com>
Subject: [PATCH v2 bpf-next 09/13] bpf: Pull out a macro for interpreting atomic ALU operations
Date: Fri, 27 Nov 2020 17:57:34 +0000	[thread overview]
Message-ID: <20201127175738.1085417-10-jackmanb@google.com> (raw)
In-Reply-To: <20201127175738.1085417-1-jackmanb@google.com>

Since the atomic operations that are added in subsequent commits are
all isomorphic with BPF_ADD, pull out a macro to avoid the
interpreter becoming dominated by lines of atomic-related code.

Note that this sacrificies interpreter performance (combining
STX_ATOMIC_W and STX_ATOMIC_DW into single switch case means that we
need an extra conditional branch to differentiate them) in favour of
compact and (relatively!) simple C code.

Signed-off-by: Brendan Jackman <jackmanb@google.com>
---
 kernel/bpf/core.c | 79 +++++++++++++++++++++++------------------------
 1 file changed, 38 insertions(+), 41 deletions(-)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 05350a8f87c0..20a5351d1dc2 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1626,55 +1626,52 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)
 	LDX_PROBE(DW, 8)
 #undef LDX_PROBE
 
-	STX_ATOMIC_W:
-		switch (IMM) {
-		case BPF_ADD:
-			/* lock xadd *(u32 *)(dst_reg + off16) += src_reg */
-			atomic_add((u32) SRC, (atomic_t *)(unsigned long)
-				   (DST + insn->off));
-			break;
-		case BPF_ADD | BPF_FETCH:
-			SRC = (u32) atomic_fetch_add(
-				(u32) SRC,
-				(atomic_t *)(unsigned long) (DST + insn->off));
-			break;
-		case BPF_XCHG:
-			SRC = (u32) atomic_xchg(
-				(atomic_t *)(unsigned long) (DST + insn->off),
-				(u32) SRC);
-			break;
-		case BPF_CMPXCHG:
-			BPF_R0 = (u32) atomic_cmpxchg(
-				(atomic_t *)(unsigned long) (DST + insn->off),
-				(u32) BPF_R0, (u32) SRC);
+#define ATOMIC(BOP, KOP)						\
+		case BOP:						\
+			if (BPF_SIZE(insn->code) == BPF_W)		\
+				atomic_##KOP((u32) SRC, (atomic_t *)(unsigned long) \
+					     (DST + insn->off));	\
+			else						\
+				atomic64_##KOP((u64) SRC, (atomic64_t *)(unsigned long) \
+					       (DST + insn->off));	\
+			break;						\
+		case BOP | BPF_FETCH:					\
+			if (BPF_SIZE(insn->code) == BPF_W)		\
+				SRC = (u32) atomic_fetch_##KOP(		\
+					(u32) SRC,			\
+					(atomic_t *)(unsigned long) (DST + insn->off)); \
+			else						\
+				SRC = (u64) atomic64_fetch_##KOP(	\
+					(u64) SRC,			\
+					(atomic64_t *)(s64) (DST + insn->off)); \
 			break;
-		default:
-			goto default_label;
-		}
-		CONT;
 
 	STX_ATOMIC_DW:
+	STX_ATOMIC_W:
 		switch (IMM) {
-		case BPF_ADD:
-			/* lock xadd *(u64 *)(dst_reg + off16) += src_reg */
-			atomic64_add((u64) SRC, (atomic64_t *)(unsigned long)
-				     (DST + insn->off));
-			break;
-		case BPF_ADD | BPF_FETCH:
-			SRC = (u64) atomic64_fetch_add(
-				(u64) SRC,
-				(atomic64_t *)(s64) (DST + insn->off));
-			break;
+		ATOMIC(BPF_ADD, add)
+
 		case BPF_XCHG:
-			SRC = (u64) atomic64_xchg(
-				(atomic64_t *)(u64) (DST + insn->off),
-				(u64) SRC);
+			if (BPF_SIZE(insn->code) == BPF_W)
+				SRC = (u32) atomic_xchg(
+					(atomic_t *)(unsigned long) (DST + insn->off),
+					(u32) SRC);
+			else
+				SRC = (u64) atomic64_xchg(
+					(atomic64_t *)(u64) (DST + insn->off),
+					(u64) SRC);
 			break;
 		case BPF_CMPXCHG:
-			BPF_R0 = (u64) atomic64_cmpxchg(
-				(atomic64_t *)(u64) (DST + insn->off),
-				(u64) BPF_R0, (u64) SRC);
+			if (BPF_SIZE(insn->code) == BPF_W)
+				BPF_R0 = (u32) atomic_cmpxchg(
+					(atomic_t *)(unsigned long) (DST + insn->off),
+					(u32) BPF_R0, (u32) SRC);
+			else
+				BPF_R0 = (u64) atomic64_cmpxchg(
+					(atomic64_t *)(u64) (DST + insn->off),
+					(u64) BPF_R0, (u64) SRC);
 			break;
+
 		default:
 			goto default_label;
 		}
-- 
2.29.2.454.gaff20da3a2-goog


  parent reply	other threads:[~2020-11-27 17:58 UTC|newest]

Thread overview: 59+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-27 17:57 [PATCH v2 bpf-next 00/13] Atomics for eBPF Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 01/13] bpf: x86: Factor out emission of ModR/M for *(reg + off) Brendan Jackman
2020-11-29  1:15   ` Alexei Starovoitov
2020-12-01 12:14     ` Brendan Jackman
2020-12-02  5:50       ` Alexei Starovoitov
2020-12-02 10:52         ` Brendan Jackman
2020-12-02 17:35           ` Alexei Starovoitov
2020-11-27 17:57 ` [PATCH v2 bpf-next 02/13] bpf: x86: Factor out emission of REX byte Brendan Jackman
2020-11-29  1:14   ` Alexei Starovoitov
2020-12-01 12:12     ` Brendan Jackman
2020-12-02  5:48       ` Alexei Starovoitov
2020-12-02 10:54         ` Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 03/13] bpf: x86: Factor out function to emit NEG Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 04/13] bpf: x86: Factor out a lookup table for some ALU opcodes Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 05/13] bpf: Rename BPF_XADD and prepare to encode other atomics in .imm Brendan Jackman
2020-11-28  3:43   ` Yonghong Song
2020-12-01 12:17     ` Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 06/13] bpf: Move BPF_STX reserved field check into BPF_STX verifier code Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 07/13] bpf: Add BPF_FETCH field / create atomic_fetch_add instruction Brendan Jackman
2020-11-28  4:15   ` Yonghong Song
2020-12-01 12:22     ` Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 08/13] bpf: Add instructions for atomic_[cmp]xchg Brendan Jackman
2020-11-28  5:25   ` Yonghong Song
2020-12-01 12:27     ` Brendan Jackman
2020-11-29  1:27   ` Alexei Starovoitov
2020-12-01 12:32     ` Brendan Jackman
2020-11-27 17:57 ` Brendan Jackman [this message]
2020-11-27 17:57 ` [PATCH v2 bpf-next 10/13] bpf: Add instructions for atomic[64]_[fetch_]sub Brendan Jackman
2020-11-27 21:39   ` kernel test robot
2020-11-27 21:39     ` kernel test robot
2020-11-27 21:39   ` [RFC PATCH] bpf: bpf_atomic_alu_string[] can be static kernel test robot
2020-11-27 21:39     ` kernel test robot
2020-11-28  5:35   ` [PATCH v2 bpf-next 10/13] bpf: Add instructions for atomic[64]_[fetch_]sub Yonghong Song
2020-11-29  1:34     ` Alexei Starovoitov
2020-11-30 17:18       ` Yonghong Song
2020-12-01 12:38         ` Brendan Jackman
2020-12-02  5:55           ` Alexei Starovoitov
2020-12-02 11:19             ` Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 11/13] bpf: Add bitwise atomic instructions Brendan Jackman
2020-11-28  5:39   ` Yonghong Song
2020-11-29  1:36     ` Alexei Starovoitov
2020-11-30 17:20       ` Yonghong Song
2020-11-27 17:57 ` [PATCH v2 bpf-next 12/13] bpf: Add tests for new BPF atomic operations Brendan Jackman
2020-12-01  3:55   ` Yonghong Song
2020-12-01 12:56     ` Brendan Jackman
2020-12-01 17:24       ` Yonghong Song
2020-12-02  2:22   ` Andrii Nakryiko
2020-12-02 12:26     ` Brendan Jackman
2020-11-27 17:57 ` [PATCH v2 bpf-next 13/13] bpf: Document new atomic instructions Brendan Jackman
2020-11-28  5:53 ` [PATCH v2 bpf-next 00/13] Atomics for eBPF Yonghong Song
2020-11-29  1:40   ` Alexei Starovoitov
2020-11-30 17:22     ` Yonghong Song
2020-12-01  3:48       ` Yonghong Song
2020-12-02  2:00         ` Andrii Nakryiko
2020-12-02  5:05           ` Yonghong Song
2020-12-02  5:53             ` John Fastabend
2020-12-02  5:59               ` Andrii Nakryiko
2020-12-02  6:27                 ` John Fastabend
2020-12-02  8:03             ` Yonghong Song

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201127175738.1085417-10-jackmanb@google.com \
    --to=jackmanb@google.com \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=jannh@google.com \
    --cc=kpsingh@chromium.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=revest@chromium.org \
    --cc=yhs@fb.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.