All of lore.kernel.org
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions
@ 2020-05-18 15:52 Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
                   ` (5 more replies)
  0 siblings, 6 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev

To fill the gap with linux kernel eBPF implementation,
add support for two non-generic instructions:
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
which are used to access packet data.
Make necessary changes in BPF verifier, interpreter and x86 JIT code.

Konstantin Ananyev (5):
  test/bpf: fix few small issues
  bpf: fix add/sub min/max estimations
  bpf: add support for packet data load instructions
  test/bpf: add new test cases for mbuf load instructions
  bpf: x86 JIT support for packet data load instructions

 app/test/test_bpf.c               | 504 ++++++++++++++++++++++++++++--
 doc/guides/prog_guide/bpf_lib.rst |  30 +-
 lib/librte_bpf/bpf_exec.c         |  57 ++++
 lib/librte_bpf/bpf_jit_x86.c      | 181 +++++++++++
 lib/librte_bpf/bpf_validate.c     | 104 +++++-
 5 files changed, 843 insertions(+), 33 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
@ 2020-05-18 15:52 ` Konstantin Ananyev
  2020-06-24 21:33   ` [dpdk-dev] [dpdk-stable] " Thomas Monjalon
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev, stable

Address for few small issues:
 - unreachable return statement
 - failed test-case can finish with 'success' status

Also use unified cmp_res() function to check return value.

Fixes: a9de470cc7c0 ("test: move to app directory")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/test_bpf.c | 41 +++++++++++++++--------------------------
 1 file changed, 15 insertions(+), 26 deletions(-)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index ee534687a..4a61a7d7c 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -1797,13 +1797,6 @@ test_call1_check(uint64_t rc, const void *arg)
 	dummy_func1(arg, &v32, &v64);
 	v64 += v32;
 
-	if (v64 != rc) {
-		printf("%s@%d: invalid return value "
-			"expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n",
-			__func__, __LINE__, v64, rc);
-		return -1;
-	}
-	return 0;
 	return cmp_res(__func__, v64, rc, dv, dv, sizeof(*dv));
 }
 
@@ -1934,13 +1927,7 @@ test_call2_check(uint64_t rc, const void *arg)
 	dummy_func2(&a, &b);
 	v = a.u64 + a.u32 + b.u16 + b.u8;
 
-	if (v != rc) {
-		printf("%s@%d: invalid return value "
-			"expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n",
-			__func__, __LINE__, v, rc);
-		return -1;
-	}
-	return 0;
+	return cmp_res(__func__, v, rc, arg, arg, 0);
 }
 
 static const struct rte_bpf_xsym test_call2_xsym[] = {
@@ -2429,7 +2416,6 @@ test_call5_check(uint64_t rc, const void *arg)
 	v = 0;
 
 fail:
-
 	return cmp_res(__func__, v, rc, &v, &rc, sizeof(v));
 }
 
@@ -2458,6 +2444,7 @@ static const struct rte_bpf_xsym test_call5_xsym[] = {
 	},
 };
 
+/* all bpf test cases */
 static const struct bpf_test tests[] = {
 	{
 		.name = "test_store1",
@@ -2738,7 +2725,6 @@ run_test(const struct bpf_test *tst)
 	}
 
 	tst->prepare(tbuf);
-
 	rc = rte_bpf_exec(bpf, tbuf);
 	ret = tst->check_result(rc, tbuf);
 	if (ret != 0) {
@@ -2746,17 +2732,20 @@ run_test(const struct bpf_test *tst)
 			__func__, __LINE__, tst->name, ret, strerror(ret));
 	}
 
+	/* repeat the same test with jit, when possible */
 	rte_bpf_get_jit(bpf, &jit);
-	if (jit.func == NULL)
-		return 0;
-
-	tst->prepare(tbuf);
-	rc = jit.func(tbuf);
-	rv = tst->check_result(rc, tbuf);
-	ret |= rv;
-	if (rv != 0) {
-		printf("%s@%d: check_result(%s) failed, error: %d(%s);\n",
-			__func__, __LINE__, tst->name, rv, strerror(ret));
+	if (jit.func != NULL) {
+
+		tst->prepare(tbuf);
+		rc = jit.func(tbuf);
+		rv = tst->check_result(rc, tbuf);
+		ret |= rv;
+		if (rv != 0) {
+			printf("%s@%d: check_result(%s) failed, "
+				"error: %d(%s);\n",
+				__func__, __LINE__, tst->name,
+				rv, strerror(ret));
+		}
 	}
 
 	rte_bpf_destroy(bpf);
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
@ 2020-05-18 15:52 ` Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev, stable

eval_add()/eval_sub() not always correctly estimate
minimum and maximum possible values of add/sub operations.

Fixes: 8021917293d0 ("bpf: add extra validation for input BPF program")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_bpf/bpf_validate.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index 6bd6f78e9..80d21fabb 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -226,7 +226,7 @@ eval_add(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk)
 	struct bpf_reg_val rv;
 
 	rv.u.min = (rd->u.min + rs->u.min) & msk;
-	rv.u.max = (rd->u.min + rs->u.max) & msk;
+	rv.u.max = (rd->u.max + rs->u.max) & msk;
 	rv.s.min = (rd->s.min + rs->s.min) & msk;
 	rv.s.max = (rd->s.max + rs->s.max) & msk;
 
@@ -254,10 +254,10 @@ eval_sub(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk)
 {
 	struct bpf_reg_val rv;
 
-	rv.u.min = (rd->u.min - rs->u.min) & msk;
-	rv.u.max = (rd->u.min - rs->u.max) & msk;
-	rv.s.min = (rd->s.min - rs->s.min) & msk;
-	rv.s.max = (rd->s.max - rs->s.max) & msk;
+	rv.u.min = (rd->u.min - rs->u.max) & msk;
+	rv.u.max = (rd->u.max - rs->u.min) & msk;
+	rv.s.min = (rd->s.min - rs->s.max) & msk;
+	rv.s.max = (rd->s.max - rs->s.min) & msk;
 
 	/*
 	 * if at least one of the operands is not constant,
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
@ 2020-05-18 15:52 ` Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev

To fill the gap with linux kernel eBPF implementation,
add support for two non-generic instructions:
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
which are used to access packet data.
These instructions can only be used when BPF context is a pointer
to 'struct rte_mbuf' (i.e: RTE_BPF_ARG_PTR_MBUF type).

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/prog_guide/bpf_lib.rst | 30 +++++++++++-
 lib/librte_bpf/bpf_exec.c         | 57 +++++++++++++++++++++++
 lib/librte_bpf/bpf_validate.c     | 77 +++++++++++++++++++++++++++++++
 3 files changed, 163 insertions(+), 1 deletion(-)

diff --git a/doc/guides/prog_guide/bpf_lib.rst b/doc/guides/prog_guide/bpf_lib.rst
index 9c728da7b..1feb7734a 100644
--- a/doc/guides/prog_guide/bpf_lib.rst
+++ b/doc/guides/prog_guide/bpf_lib.rst
@@ -27,6 +27,35 @@ The library API provides the following basic operations:
 
 *   Load BPF program from the ELF file and install callback to execute it on given ethdev port/queue.
 
+Packet data load instructions
+-----------------------------
+
+DPDK supports two non-generic instructions: ``(BPF_ABS | size | BPF_LD)``
+and ``(BPF_IND | size | BPF_LD)`` which are used to access packet data.
+These instructions can only be used when execution context is a pointer to
+``struct rte_mbuf`` and have seven implicit operands.
+Register ``R6`` is an implicit input that must contain pointer to ``rte_mbuf``.
+Register ``R0`` is an implicit output which contains the data fetched from the
+packet. Registers ``R1-R5`` are scratch registers
+and must not be used to store the data across these instructions.
+These instructions have implicit program exit condition as well. When
+eBPF program is trying to access the data beyond the packet boundary,
+the interpreter will abort the execution of the program. JIT compilers
+therefore must preserve this property. ``src_reg`` and ``imm32`` fields are
+explicit inputs to these instructions.
+For example, ``(BPF_IND | BPF_W | BPF_LD)`` means:
+
+.. code-block:: c
+
+    uint32_t tmp;
+    R0 = rte_pktmbuf_read((const struct rte_mbuf *)R6,  src_reg + imm32,
+	sizeof(tmp), &tmp);
+    if (R0 == NULL) return FAILED;
+    R0 = ntohl(*(uint32_t *)R0);
+
+and ``R1-R5`` were scratched.
+
+
 Not currently supported eBPF features
 -------------------------------------
 
@@ -34,5 +63,4 @@ Not currently supported eBPF features
  - cBPF
  - tail-pointer call
  - eBPF MAP
- - skb
  - external function calls for 32-bit platforms
diff --git a/lib/librte_bpf/bpf_exec.c b/lib/librte_bpf/bpf_exec.c
index 1bb226643..b921112fe 100644
--- a/lib/librte_bpf/bpf_exec.c
+++ b/lib/librte_bpf/bpf_exec.c
@@ -74,6 +74,26 @@
 		(uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \
 		reg[ins->src_reg]))
 
+/* BPF_LD | BPF_ABS/BPF_IND */
+
+#define	NOP(x)	(x)
+
+#define BPF_LD_ABS(bpf, reg, ins, type, op) do { \
+	const type *p = bpf_ld_mbuf(bpf, reg, ins, (ins)->imm, sizeof(type)); \
+	if (p == NULL)  \
+		return 0; \
+	reg[EBPF_REG_0] = op(p[0]); \
+} while (0)
+
+#define BPF_LD_IND(bpf, reg, ins, type, op) do { \
+	uint32_t ofs = reg[ins->src_reg] + (ins)->imm; \
+	const type *p = bpf_ld_mbuf(bpf, reg, ins, ofs, sizeof(type)); \
+	if (p == NULL)  \
+		return 0; \
+	reg[EBPF_REG_0] = op(p[0]); \
+} while (0)
+
+
 static inline void
 bpf_alu_be(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins)
 {
@@ -112,6 +132,23 @@ bpf_alu_le(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins)
 	}
 }
 
+static inline const void *
+bpf_ld_mbuf(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM],
+	const struct ebpf_insn *ins, uint32_t off, uint32_t len)
+{
+	const struct rte_mbuf *mb;
+	const void *p;
+
+	mb = (const struct rte_mbuf *)(uintptr_t)reg[EBPF_REG_6];
+	p = rte_pktmbuf_read(mb, off, len, reg + EBPF_REG_0);
+	if (p == NULL)
+		RTE_BPF_LOG(DEBUG, "%s(bpf=%p, mbuf=%p, ofs=%u, len=%u): "
+			"load beyond packet boundary at pc: %#zx;\n",
+			__func__, bpf, mb, off, len,
+			(uintptr_t)(ins) - (uintptr_t)(bpf)->prm.ins);
+	return p;
+}
+
 static inline uint64_t
 bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM])
 {
@@ -296,6 +333,26 @@ bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM])
 				(uint64_t)(uint32_t)ins[1].imm << 32;
 			ins++;
 			break;
+		/* load absolute instructions */
+		case (BPF_LD | BPF_ABS | BPF_B):
+			BPF_LD_ABS(bpf, reg, ins, uint8_t, NOP);
+			break;
+		case (BPF_LD | BPF_ABS | BPF_H):
+			BPF_LD_ABS(bpf, reg, ins, uint16_t, rte_be_to_cpu_16);
+			break;
+		case (BPF_LD | BPF_ABS | BPF_W):
+			BPF_LD_ABS(bpf, reg, ins, uint32_t, rte_be_to_cpu_32);
+			break;
+		/* load indirect instructions */
+		case (BPF_LD | BPF_IND | BPF_B):
+			BPF_LD_IND(bpf, reg, ins, uint8_t, NOP);
+			break;
+		case (BPF_LD | BPF_IND | BPF_H):
+			BPF_LD_IND(bpf, reg, ins, uint16_t, rte_be_to_cpu_16);
+			break;
+		case (BPF_LD | BPF_IND | BPF_W):
+			BPF_LD_IND(bpf, reg, ins, uint32_t, rte_be_to_cpu_32);
+			break;
 		/* store instructions */
 		case (BPF_STX | BPF_MEM | BPF_B):
 			BPF_ST_REG(reg, ins, uint8_t);
diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index 80d21fabb..fecdda0e1 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -102,6 +102,9 @@ struct bpf_ins_check {
 #define	WRT_REGS	RTE_LEN2MASK(EBPF_REG_10, uint16_t)
 #define	ZERO_REG	RTE_LEN2MASK(EBPF_REG_1, uint16_t)
 
+/* For LD_IND R6 is an implicit CTX register. */
+#define	IND_SRC_REGS	(WRT_REGS ^ 1 << EBPF_REG_6)
+
 /*
  * check and evaluate functions for particular instruction types.
  */
@@ -580,6 +583,42 @@ eval_neg(struct bpf_reg_val *rd, size_t opsz, uint64_t msk)
 	rd->s.min = RTE_MIN(sx, sy);
 }
 
+static const char *
+eval_ld_mbuf(struct bpf_verifier *bvf, const struct ebpf_insn *ins)
+{
+	uint32_t i, mode;
+	struct bpf_reg_val *rv, ri, rs;
+
+	mode = BPF_MODE(ins->code);
+
+	/* R6 is an implicit input that must contain pointer to mbuf */
+	if (bvf->evst->rv[EBPF_REG_6].v.type != RTE_BPF_ARG_PTR_MBUF)
+		return "invalid type for implicit ctx register";
+
+	if (mode == BPF_IND) {
+		rs = bvf->evst->rv[ins->src_reg];
+		if (rs.v.type != RTE_BPF_ARG_RAW)
+			return "unexpected type for src register";
+
+		eval_fill_imm(&ri, UINT64_MAX, ins->imm);
+		eval_add(&rs, &ri, UINT64_MAX);
+
+		if (rs.s.max < 0 || rs.u.min > UINT32_MAX)
+			return "mbuf boundary violation";
+	}
+
+	/* R1-R5 scratch registers */
+	for (i = EBPF_REG_1; i != EBPF_REG_6; i++)
+		bvf->evst->rv[i].v.type = RTE_BPF_ARG_UNDEF;
+
+	/* R0 is an implicit output, contains data fetched from the packet */
+	rv = bvf->evst->rv + EBPF_REG_0;
+	rv->v.size = bpf_size(BPF_SIZE(ins->code));
+	eval_fill_max_bound(rv, RTE_LEN2MASK(rv->v.size * CHAR_BIT, uint64_t));
+
+	return NULL;
+}
+
 /*
  * check that destination and source operand are in defined state.
  */
@@ -1425,6 +1464,44 @@ static const struct bpf_ins_check ins_chk[UINT8_MAX + 1] = {
 		.imm = { .min = 0, .max = UINT32_MAX},
 		.eval = eval_ld_imm64,
 	},
+	/* load absolute instructions */
+	[(BPF_LD | BPF_ABS | BPF_B)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_ABS | BPF_H)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_ABS | BPF_W)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	/* load indirect instructions */
+	[(BPF_LD | BPF_IND | BPF_B)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_IND | BPF_H)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_IND | BPF_W)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
 	/* store REG instructions */
 	[(BPF_STX | BPF_MEM | BPF_B)] = {
 		.mask = { .dreg = ALL_REGS, .sreg = ALL_REGS},
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf load instructions
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                   ` (2 preceding siblings ...)
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
@ 2020-05-18 15:52 ` Konstantin Ananyev
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev

Add new test-cases for BPF_ABS/BPF_IND load instructions.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/test_bpf.c | 463 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 463 insertions(+)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 4a61a7d7c..7c3de96c6 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -43,6 +43,14 @@ struct dummy_net {
 	struct rte_ipv4_hdr ip_hdr;
 };
 
+#define	DUMMY_MBUF_NUM	2
+
+/* first mbuf in the packet, should always be at offset 0 */
+struct dummy_mbuf {
+	struct rte_mbuf mb[DUMMY_MBUF_NUM];
+	uint8_t buf[DUMMY_MBUF_NUM][RTE_MBUF_DEFAULT_BUF_SIZE];
+};
+
 #define	TEST_FILL_1	0xDEADBEEF
 
 #define	TEST_MUL_1	21
@@ -2444,6 +2452,413 @@ static const struct rte_bpf_xsym test_call5_xsym[] = {
 	},
 };
 
+/* load mbuf (BPF_ABS/BPF_IND) test-cases */
+static const struct ebpf_insn test_ld_mbuf1_prog[] = {
+
+	/* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_6,
+		.src_reg = EBPF_REG_1,
+	},
+	/* load IPv4 version and IHL */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_B),
+		.imm = offsetof(struct rte_ipv4_hdr, version_ihl),
+	},
+	/* check IP version */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_2,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = 0xf0,
+	},
+	{
+		.code = (BPF_JMP | BPF_JEQ | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = IPVERSION << 4,
+		.off = 2,
+	},
+	/* invalid IP version, return 0 */
+	{
+		.code = (EBPF_ALU64 | BPF_XOR | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+	/* load 3-rd byte of IP data */
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = RTE_IPV4_HDR_IHL_MASK,
+	},
+	{
+		.code = (BPF_ALU | BPF_LSH | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = 2,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_B),
+		.src_reg = EBPF_REG_0,
+		.imm = 3,
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load IPv4 src addr */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_W),
+		.imm = offsetof(struct rte_ipv4_hdr, src_addr),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load IPv4 total length */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_H),
+		.imm = offsetof(struct rte_ipv4_hdr, total_length),
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_8,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load last 4 bytes of IP data */
+	{
+		.code = (BPF_LD | BPF_IND | BPF_W),
+		.src_reg = EBPF_REG_8,
+		.imm = -(int32_t)sizeof(uint32_t),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load 2 bytes from the middle of IP data */
+	{
+		.code = (EBPF_ALU64 | BPF_RSH | BPF_K),
+		.dst_reg = EBPF_REG_8,
+		.imm = 1,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_H),
+		.src_reg = EBPF_REG_8,
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_7,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+};
+
+static void
+dummy_mbuf_prep(struct rte_mbuf *mb, uint8_t buf[], uint32_t buf_len,
+	uint32_t data_len)
+{
+	uint32_t i;
+	uint8_t *db;
+
+	mb->buf_addr = buf;
+	mb->buf_iova = (uintptr_t)buf;
+	mb->buf_len = buf_len;
+	rte_mbuf_refcnt_set(mb, 1);
+
+	/* set pool pointer to dummy value, test doesn't use it */
+	mb->pool = (void *)buf;
+
+	rte_pktmbuf_reset(mb);
+	db = (uint8_t *)rte_pktmbuf_append(mb, data_len);
+
+	for (i = 0; i != data_len; i++)
+		db[i] = i;
+}
+
+static void
+test_ld_mbuf1_prepare(void *arg)
+{
+	struct dummy_mbuf *dm;
+	struct rte_ipv4_hdr *ph;
+
+	const uint32_t plen = 400;
+	const struct rte_ipv4_hdr iph = {
+		.version_ihl = RTE_IPV4_VHL_DEF,
+		.total_length = rte_cpu_to_be_16(plen),
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_RAW,
+		.src_addr = rte_cpu_to_be_32(RTE_IPV4_LOOPBACK),
+		.dst_addr = rte_cpu_to_be_32(RTE_IPV4_BROADCAST),
+	};
+
+	dm = arg;
+	memset(dm, 0, sizeof(*dm));
+
+	dummy_mbuf_prep(&dm->mb[0], dm->buf[0], sizeof(dm->buf[0]),
+		plen / 2 + 1);
+	dummy_mbuf_prep(&dm->mb[1], dm->buf[1], sizeof(dm->buf[0]),
+		plen / 2 - 1);
+
+	rte_pktmbuf_chain(&dm->mb[0], &dm->mb[1]);
+
+	ph = rte_pktmbuf_mtod(dm->mb, typeof(ph));
+	memcpy(ph, &iph, sizeof(iph));
+}
+
+static uint64_t
+test_ld_mbuf1(const struct rte_mbuf *pkt)
+{
+	uint64_t n, v;
+	const uint8_t *p8;
+	const uint16_t *p16;
+	const uint32_t *p32;
+	struct dummy_offset dof;
+
+	/* load IPv4 version and IHL */
+	p8 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, version_ihl), sizeof(*p8),
+		&dof);
+	if (p8 == NULL)
+		return 0;
+
+	/* check IP version */
+	if ((p8[0] & 0xf0) != IPVERSION << 4)
+		return 0;
+
+	n = (p8[0] & RTE_IPV4_HDR_IHL_MASK) * RTE_IPV4_IHL_MULTIPLIER;
+
+	/* load 3-rd byte of IP data */
+	p8 = rte_pktmbuf_read(pkt, n + 3, sizeof(*p8), &dof);
+	if (p8 == NULL)
+		return 0;
+
+	v = p8[0];
+
+	/* load IPv4 src addr */
+	p32 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, src_addr), sizeof(*p32),
+		&dof);
+	if (p32 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_32(p32[0]);
+
+	/* load IPv4 total length */
+	p16 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, total_length), sizeof(*p16),
+		&dof);
+	if (p16 == NULL)
+		return 0;
+
+	n = rte_be_to_cpu_16(p16[0]);
+
+	/* load last 4 bytes of IP data */
+	p32 = rte_pktmbuf_read(pkt, n - sizeof(*p32), sizeof(*p32), &dof);
+	if (p32 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_32(p32[0]);
+
+	/* load 2 bytes from the middle of IP data */
+	p16 = rte_pktmbuf_read(pkt, n / 2, sizeof(*p16), &dof);
+	if (p16 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_16(p16[0]);
+	return v;
+}
+
+static int
+test_ld_mbuf1_check(uint64_t rc, const void *arg)
+{
+	const struct dummy_mbuf *dm;
+	uint64_t v;
+
+	dm = arg;
+	v = test_ld_mbuf1(dm->mb);
+	return cmp_res(__func__, v, rc, arg, arg, 0);
+}
+
+/*
+ * same as ld_mbuf1, but then trancate the mbuf by 1B,
+ * so load of last 4B fail.
+ */
+static void
+test_ld_mbuf2_prepare(void *arg)
+{
+	struct dummy_mbuf *dm;
+
+	test_ld_mbuf1_prepare(arg);
+	dm = arg;
+	rte_pktmbuf_trim(dm->mb, 1);
+}
+
+static int
+test_ld_mbuf2_check(uint64_t rc, const void *arg)
+{
+	return cmp_res(__func__, 0, rc, arg, arg, 0);
+}
+
+/* same as test_ld_mbuf1, but now store intermediate results on the stack */
+static const struct ebpf_insn test_ld_mbuf3_prog[] = {
+
+	/* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_6,
+		.src_reg = EBPF_REG_1,
+	},
+	/* load IPv4 version and IHL */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_B),
+		.imm = offsetof(struct rte_ipv4_hdr, version_ihl),
+	},
+	/* check IP version */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_2,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = 0xf0,
+	},
+	{
+		.code = (BPF_JMP | BPF_JEQ | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = IPVERSION << 4,
+		.off = 2,
+	},
+	/* invalid IP version, return 0 */
+	{
+		.code = (EBPF_ALU64 | BPF_XOR | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+	/* load 3-rd byte of IP data */
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = RTE_IPV4_HDR_IHL_MASK,
+	},
+	{
+		.code = (BPF_ALU | BPF_LSH | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = 2,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_B),
+		.src_reg = EBPF_REG_0,
+		.imm = 3,
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | BPF_B),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u8) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load IPv4 src addr */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_W),
+		.imm = offsetof(struct rte_ipv4_hdr, src_addr),
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | BPF_W),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u32) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load IPv4 total length */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_H),
+		.imm = offsetof(struct rte_ipv4_hdr, total_length),
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_8,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load last 4 bytes of IP data */
+	{
+		.code = (BPF_LD | BPF_IND | BPF_W),
+		.src_reg = EBPF_REG_8,
+		.imm = -(int32_t)sizeof(uint32_t),
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | EBPF_DW),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u64) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load 2 bytes from the middle of IP data */
+	{
+		.code = (EBPF_ALU64 | BPF_RSH | BPF_K),
+		.dst_reg = EBPF_REG_8,
+		.imm = 1,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_H),
+		.src_reg = EBPF_REG_8,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | EBPF_DW),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u64) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | BPF_W),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u32) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | BPF_B),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u8) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+};
+
 /* all bpf test cases */
 static const struct bpf_test tests[] = {
 	{
@@ -2704,6 +3119,54 @@ static const struct bpf_test tests[] = {
 		/* for now don't support function calls on 32 bit platform */
 		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
 	},
+	{
+		.name = "test_ld_mbuf1",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf1_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf1_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf1_prepare,
+		.check_result = test_ld_mbuf1_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
+	{
+		.name = "test_ld_mbuf2",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf1_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf1_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf2_prepare,
+		.check_result = test_ld_mbuf2_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
+	{
+		.name = "test_ld_mbuf3",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf3_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf3_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf1_prepare,
+		.check_result = test_ld_mbuf1_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
 };
 
 static int
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data load instructions
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                   ` (3 preceding siblings ...)
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
@ 2020-05-18 15:52 ` Konstantin Ananyev
  2020-05-24 13:37   ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  5 siblings, 1 reply; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
  To: dev; +Cc: stephen, jerinj, Konstantin Ananyev

Make x86 JIT to generate native code for
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
instructions.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_bpf/bpf_jit_x86.c  | 181 ++++++++++++++++++++++++++++++++++
 lib/librte_bpf/bpf_validate.c |  17 +++-
 2 files changed, 197 insertions(+), 1 deletion(-)

diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c
index f70cd6be5..8a7ad45b3 100644
--- a/lib/librte_bpf/bpf_jit_x86.c
+++ b/lib/librte_bpf/bpf_jit_x86.c
@@ -87,6 +87,14 @@ enum {
 	REG_TMP1 = R10,
 };
 
+/* LD_ABS/LD_IMM offsets */
+enum {
+	LDMB_FSP_OFS, /* fast-path */
+	LDMB_SLP_OFS, /* slow-path */
+	LDMB_FIN_OFS, /* final part */
+	LDMB_OFS_NUM
+};
+
 /*
  * callee saved registers list.
  * keep RBP as the last one.
@@ -100,6 +108,9 @@ struct bpf_jit_state {
 		uint32_t num;
 		int32_t off;
 	} exit;
+	struct {
+		uint32_t stack_ofs;
+	} ldmb;
 	uint32_t reguse;
 	int32_t *off;
 	uint8_t *ins;
@@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t dreg,
 		emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX);
 }
 
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates code for 'fast_path':
+ * calculate load offset and check is it inside first packet segment.
+ */
+static void
+emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7],
+	uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm,
+	const int32_t ofs[LDMB_OFS_NUM])
+{
+	/* make R2 contain *off* value */
+
+	if (sreg != rg[EBPF_REG_2]) {
+		emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K,
+			rg[EBPF_REG_2], imm);
+		if (mode == BPF_IND)
+			emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+				sreg, rg[EBPF_REG_2]);
+	} else
+		/* BPF_IND with sreg == R2 */
+		emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K,
+			rg[EBPF_REG_2], imm);
+
+	/* R3 = mbuf->data_len */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
+		rg[EBPF_REG_6], rg[EBPF_REG_3],
+		offsetof(struct rte_mbuf, data_len));
+
+	/* R3 = R3 - R2 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X,
+		rg[EBPF_REG_2], rg[EBPF_REG_3]);
+
+	/* JSLE R3, <sz> <slow_path> */
+	emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz);
+	emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]);
+
+	/* R3 = mbuf->data_off */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
+		rg[EBPF_REG_6], rg[EBPF_REG_3],
+		offsetof(struct rte_mbuf, data_off));
+
+	/* R0 = mbuf->buf_addr */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW,
+		rg[EBPF_REG_6], rg[EBPF_REG_0],
+		offsetof(struct rte_mbuf, buf_addr));
+
+	/* R0 = R0 + R3 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+		rg[EBPF_REG_3], rg[EBPF_REG_0]);
+
+	/* R0 = R0 + R2 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+		rg[EBPF_REG_2], rg[EBPF_REG_0]);
+
+	/* JMP <fin_part> */
+	emit_abs_jmp(st, ofs[LDMB_FIN_OFS]);
+}
+
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates code for 'slow_path':
+ * call __rte_pktmbuf_read() and check return value.
+ */
+static void
+emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7],
+	uint32_t sz)
+{
+	/* make R3 contain *len* value (1/2/4) */
+
+	emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz);
+
+	/* make R4 contain (RBP - ldmb.stack_ofs) */
+
+	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]);
+	emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4],
+		st->ldmb.stack_ofs);
+
+	/* make R1 contain mbuf ptr */
+
+	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X,
+		rg[EBPF_REG_6], rg[EBPF_REG_1]);
+
+	/* call rte_pktmbuf_read */
+	emit_call(st, (uintptr_t)__rte_pktmbuf_read);
+
+	/* check that return value (R0) is not zero */
+	emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]);
+	emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off);
+}
+
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates final part of code for BPF_ABS/BPF_IND load:
+ * perform data load and endianness conversion.
+ * expects dreg to contain valid data pointer.
+ */
+static void
+emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz,
+	uint32_t sz)
+{
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0);
+	if (sz != sizeof(uint8_t))
+		emit_be2le(st, dreg, sz * CHAR_BIT);
+}
+
+/*
+ * emit code for BPF_ABS/BPF_IND load.
+ * generates the following construction:
+ * fast_path:
+ *   off = ins->sreg + ins->imm
+ *   if (off + ins->opsz < mbuf->data_len)
+ *      goto slow_path;
+ *   ptr = mbuf->buf_addr + bmf->data_off + off;
+ *   goto fin_part;
+ * slow_path:
+ *   typeof(ins->opsz) buf; //allocate space on the stack
+ *   ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf);
+ *   if (ptr == NULL)
+ *      goto exit_label;
+ * fin_part:
+ *   res = *(typeof(ins->opsz))ptr;
+ *   res = bswap(res);
+ */
+static void
+emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t imm)
+{
+	uint32_t i, mode, opsz, sz;
+	uint32_t rg[EBPF_REG_7];
+	int32_t ofs[LDMB_OFS_NUM];
+
+	mode = BPF_MODE(op);
+	opsz = BPF_SIZE(op);
+	sz = bpf_size(opsz);
+
+	for (i = 0; i != RTE_DIM(rg); i++)
+		rg[i] = ebpf2x86[i];
+
+	/* fill with fake offsets */
+	for (i = 0; i != RTE_DIM(ofs); i++)
+		ofs[i] = st->sz + INT8_MAX;
+
+	/* dry run first to calculate jump offsets */
+
+	ofs[LDMB_FSP_OFS] = st->sz;
+	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
+	ofs[LDMB_SLP_OFS] = st->sz;
+	emit_ldmb_slow_path(st, rg, sz);
+	ofs[LDMB_FIN_OFS] = st->sz;
+	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
+
+	RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX);
+
+	/* reset dry-run code and do a proper run */
+
+	st->sz = ofs[LDMB_FSP_OFS];
+	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
+	emit_ldmb_slow_path(st, rg, sz);
+	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
+}
+
 static void
 emit_prolog(struct bpf_jit_state *st, int32_t stack_size)
 {
@@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf)
 	/* reset state fields */
 	st->sz = 0;
 	st->exit.num = 0;
+	st->ldmb.stack_ofs = bpf->stack_sz;
 
 	emit_prolog(st, bpf->stack_sz);
 
@@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf)
 			emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm);
 			i++;
 			break;
+		/* load absolute/indirect instructions */
+		case (BPF_LD | BPF_ABS | BPF_B):
+		case (BPF_LD | BPF_ABS | BPF_H):
+		case (BPF_LD | BPF_ABS | BPF_W):
+		case (BPF_LD | BPF_IND | BPF_B):
+		case (BPF_LD | BPF_IND | BPF_H):
+		case (BPF_LD | BPF_IND | BPF_W):
+			emit_ld_mbuf(st, op, sr, ins->imm);
+			break;
 		/* store instructions */
 		case (BPF_STX | BPF_MEM | BPF_B):
 		case (BPF_STX | BPF_MEM | BPF_H):
diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index fecdda0e1..9214f1503 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -70,6 +70,7 @@ struct bpf_verifier {
 	uint64_t stack_sz;
 	uint32_t nb_nodes;
 	uint32_t nb_jcc_nodes;
+	uint32_t nb_ldmb_nodes;
 	uint32_t node_colour[MAX_NODE_COLOUR];
 	uint32_t edge_type[MAX_EDGE_TYPE];
 	struct bpf_eval_state *evst;
@@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf)
 			rc |= add_edge(bvf, node, i + 2);
 			i++;
 			break;
+		case (BPF_LD | BPF_ABS | BPF_B):
+		case (BPF_LD | BPF_ABS | BPF_H):
+		case (BPF_LD | BPF_ABS | BPF_W):
+		case (BPF_LD | BPF_IND | BPF_B):
+		case (BPF_LD | BPF_IND | BPF_H):
+		case (BPF_LD | BPF_IND | BPF_W):
+			bvf->nb_ldmb_nodes++;
+			/* fallthrough */
 		default:
 			rc |= add_edge(bvf, node, i + 1);
 			break;
@@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf)
 	free(bvf.in);
 
 	/* copy collected info */
-	if (rc == 0)
+	if (rc == 0) {
 		bpf->stack_sz = bvf.stack_sz;
 
+		/* for LD_ABS/LD_IND, we'll need extra space on the stack */
+		if (bvf.nb_ldmb_nodes != 0)
+			bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz +
+				sizeof(uint64_t), sizeof(uint64_t));
+	}
+
 	return rc;
 }
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
@ 2020-05-24 13:37   ` Morten Brørup
  2020-05-24 23:08     ` Ananyev, Konstantin
  0 siblings, 1 reply; 17+ messages in thread
From: Morten Brørup @ 2020-05-24 13:37 UTC (permalink / raw)
  To: Konstantin Ananyev, dev; +Cc: stephen, jerinj

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Konstantin Ananyev
> Sent: Monday, May 18, 2020 5:53 PM
> 
> Make x86 JIT to generate native code for
> (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
> instructions.
> 
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
>  lib/librte_bpf/bpf_jit_x86.c  | 181 ++++++++++++++++++++++++++++++++++
>  lib/librte_bpf/bpf_validate.c |  17 +++-
>  2 files changed, 197 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c
> index f70cd6be5..8a7ad45b3 100644
> --- a/lib/librte_bpf/bpf_jit_x86.c
> +++ b/lib/librte_bpf/bpf_jit_x86.c
> @@ -87,6 +87,14 @@ enum {
>  	REG_TMP1 = R10,
>  };
> 
> +/* LD_ABS/LD_IMM offsets */
> +enum {
> +	LDMB_FSP_OFS, /* fast-path */
> +	LDMB_SLP_OFS, /* slow-path */
> +	LDMB_FIN_OFS, /* final part */
> +	LDMB_OFS_NUM
> +};
> +
>  /*
>   * callee saved registers list.
>   * keep RBP as the last one.
> @@ -100,6 +108,9 @@ struct bpf_jit_state {
>  		uint32_t num;
>  		int32_t off;
>  	} exit;
> +	struct {
> +		uint32_t stack_ofs;
> +	} ldmb;
>  	uint32_t reguse;
>  	int32_t *off;
>  	uint8_t *ins;
> @@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op,
> uint32_t sreg, uint32_t dreg,
>  		emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX);
>  }
> 
> +/*
> + * helper function, used by emit_ld_mbuf().
> + * generates code for 'fast_path':
> + * calculate load offset and check is it inside first packet segment.
> + */
> +static void
> +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t
> rg[EBPF_REG_7],
> +	uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm,
> +	const int32_t ofs[LDMB_OFS_NUM])
> +{
> +	/* make R2 contain *off* value */
> +
> +	if (sreg != rg[EBPF_REG_2]) {
> +		emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K,
> +			rg[EBPF_REG_2], imm);
> +		if (mode == BPF_IND)
> +			emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> +				sreg, rg[EBPF_REG_2]);
> +	} else
> +		/* BPF_IND with sreg == R2 */
> +		emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K,
> +			rg[EBPF_REG_2], imm);
> +
> +	/* R3 = mbuf->data_len */
> +	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
> +		rg[EBPF_REG_6], rg[EBPF_REG_3],
> +		offsetof(struct rte_mbuf, data_len));
> +
> +	/* R3 = R3 - R2 */
> +	emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X,
> +		rg[EBPF_REG_2], rg[EBPF_REG_3]);
> +
> +	/* JSLE R3, <sz> <slow_path> */
> +	emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz);
> +	emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]);
> +
> +	/* R3 = mbuf->data_off */
> +	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
> +		rg[EBPF_REG_6], rg[EBPF_REG_3],
> +		offsetof(struct rte_mbuf, data_off));
> +
> +	/* R0 = mbuf->buf_addr */
> +	emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW,
> +		rg[EBPF_REG_6], rg[EBPF_REG_0],
> +		offsetof(struct rte_mbuf, buf_addr));
> +
> +	/* R0 = R0 + R3 */
> +	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> +		rg[EBPF_REG_3], rg[EBPF_REG_0]);
> +
> +	/* R0 = R0 + R2 */
> +	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> +		rg[EBPF_REG_2], rg[EBPF_REG_0]);
> +
> +	/* JMP <fin_part> */
> +	emit_abs_jmp(st, ofs[LDMB_FIN_OFS]);
> +}
> +
> +/*
> + * helper function, used by emit_ld_mbuf().
> + * generates code for 'slow_path':
> + * call __rte_pktmbuf_read() and check return value.
> + */
> +static void
> +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t
> rg[EBPF_REG_7],
> +	uint32_t sz)
> +{
> +	/* make R3 contain *len* value (1/2/4) */
> +
> +	emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz);
> +
> +	/* make R4 contain (RBP - ldmb.stack_ofs) */
> +
> +	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]);
> +	emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4],
> +		st->ldmb.stack_ofs);
> +
> +	/* make R1 contain mbuf ptr */
> +
> +	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X,
> +		rg[EBPF_REG_6], rg[EBPF_REG_1]);
> +
> +	/* call rte_pktmbuf_read */
> +	emit_call(st, (uintptr_t)__rte_pktmbuf_read);
> +
> +	/* check that return value (R0) is not zero */
> +	emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]);
> +	emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off);
> +}
> +
> +/*
> + * helper function, used by emit_ld_mbuf().
> + * generates final part of code for BPF_ABS/BPF_IND load:
> + * perform data load and endianness conversion.
> + * expects dreg to contain valid data pointer.
> + */
> +static void
> +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz,
> +	uint32_t sz)
> +{
> +	emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0);
> +	if (sz != sizeof(uint8_t))
> +		emit_be2le(st, dreg, sz * CHAR_BIT);
> +}
> +
> +/*
> + * emit code for BPF_ABS/BPF_IND load.
> + * generates the following construction:
> + * fast_path:
> + *   off = ins->sreg + ins->imm
> + *   if (off + ins->opsz < mbuf->data_len)
> + *      goto slow_path;

I am not an eBPF expert, but I am not sure this is correct.

I think it should be > instead of <. Also, it looks like you actually emit:
if (mbuf->data_len - off <= ins->opsz)
   goto slow_path;

Could you please double check that both the comment and the emitted code has the comparison turning the correct way.

> + *   ptr = mbuf->buf_addr + bmf->data_off + off;
> + *   goto fin_part;
> + * slow_path:
> + *   typeof(ins->opsz) buf; //allocate space on the stack
> + *   ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf);
> + *   if (ptr == NULL)
> + *      goto exit_label;
> + * fin_part:
> + *   res = *(typeof(ins->opsz))ptr;
> + *   res = bswap(res);
> + */
> +static void
> +emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg,
> uint32_t imm)
> +{
> +	uint32_t i, mode, opsz, sz;
> +	uint32_t rg[EBPF_REG_7];
> +	int32_t ofs[LDMB_OFS_NUM];
> +
> +	mode = BPF_MODE(op);
> +	opsz = BPF_SIZE(op);
> +	sz = bpf_size(opsz);
> +
> +	for (i = 0; i != RTE_DIM(rg); i++)
> +		rg[i] = ebpf2x86[i];
> +
> +	/* fill with fake offsets */
> +	for (i = 0; i != RTE_DIM(ofs); i++)
> +		ofs[i] = st->sz + INT8_MAX;
> +
> +	/* dry run first to calculate jump offsets */
> +
> +	ofs[LDMB_FSP_OFS] = st->sz;
> +	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
> +	ofs[LDMB_SLP_OFS] = st->sz;
> +	emit_ldmb_slow_path(st, rg, sz);
> +	ofs[LDMB_FIN_OFS] = st->sz;
> +	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
> +
> +	RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX);
> +
> +	/* reset dry-run code and do a proper run */
> +
> +	st->sz = ofs[LDMB_FSP_OFS];
> +	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
> +	emit_ldmb_slow_path(st, rg, sz);
> +	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
> +}
> +
>  static void
>  emit_prolog(struct bpf_jit_state *st, int32_t stack_size)
>  {
> @@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf
> *bpf)
>  	/* reset state fields */
>  	st->sz = 0;
>  	st->exit.num = 0;
> +	st->ldmb.stack_ofs = bpf->stack_sz;
> 
>  	emit_prolog(st, bpf->stack_sz);
> 
> @@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf
> *bpf)
>  			emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm);
>  			i++;
>  			break;
> +		/* load absolute/indirect instructions */
> +		case (BPF_LD | BPF_ABS | BPF_B):
> +		case (BPF_LD | BPF_ABS | BPF_H):
> +		case (BPF_LD | BPF_ABS | BPF_W):
> +		case (BPF_LD | BPF_IND | BPF_B):
> +		case (BPF_LD | BPF_IND | BPF_H):
> +		case (BPF_LD | BPF_IND | BPF_W):
> +			emit_ld_mbuf(st, op, sr, ins->imm);
> +			break;
>  		/* store instructions */
>  		case (BPF_STX | BPF_MEM | BPF_B):
>  		case (BPF_STX | BPF_MEM | BPF_H):
> diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
> index fecdda0e1..9214f1503 100644
> --- a/lib/librte_bpf/bpf_validate.c
> +++ b/lib/librte_bpf/bpf_validate.c
> @@ -70,6 +70,7 @@ struct bpf_verifier {
>  	uint64_t stack_sz;
>  	uint32_t nb_nodes;
>  	uint32_t nb_jcc_nodes;
> +	uint32_t nb_ldmb_nodes;
>  	uint32_t node_colour[MAX_NODE_COLOUR];
>  	uint32_t edge_type[MAX_EDGE_TYPE];
>  	struct bpf_eval_state *evst;
> @@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf)
>  			rc |= add_edge(bvf, node, i + 2);
>  			i++;
>  			break;
> +		case (BPF_LD | BPF_ABS | BPF_B):
> +		case (BPF_LD | BPF_ABS | BPF_H):
> +		case (BPF_LD | BPF_ABS | BPF_W):
> +		case (BPF_LD | BPF_IND | BPF_B):
> +		case (BPF_LD | BPF_IND | BPF_H):
> +		case (BPF_LD | BPF_IND | BPF_W):
> +			bvf->nb_ldmb_nodes++;
> +			/* fallthrough */
>  		default:
>  			rc |= add_edge(bvf, node, i + 1);
>  			break;
> @@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf)
>  	free(bvf.in);
> 
>  	/* copy collected info */
> -	if (rc == 0)
> +	if (rc == 0) {
>  		bpf->stack_sz = bvf.stack_sz;
> 
> +		/* for LD_ABS/LD_IND, we'll need extra space on the stack */
> +		if (bvf.nb_ldmb_nodes != 0)
> +			bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz +
> +				sizeof(uint64_t), sizeof(uint64_t));
> +	}
> +
>  	return rc;
>  }
> --
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions
  2020-05-24 13:37   ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup
@ 2020-05-24 23:08     ` Ananyev, Konstantin
  0 siblings, 0 replies; 17+ messages in thread
From: Ananyev, Konstantin @ 2020-05-24 23:08 UTC (permalink / raw)
  To: Morten Brørup, dev; +Cc: stephen, jerinj

> >
> > +/*
> > + * helper function, used by emit_ld_mbuf().
> > + * generates code for 'fast_path':
> > + * calculate load offset and check is it inside first packet segment.
> > + */
> > +static void
> > +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t
> > rg[EBPF_REG_7],
> > +	uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm,
> > +	const int32_t ofs[LDMB_OFS_NUM])
> > +{
> > +	/* make R2 contain *off* value */
> > +
> > +	if (sreg != rg[EBPF_REG_2]) {
> > +		emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K,
> > +			rg[EBPF_REG_2], imm);
> > +		if (mode == BPF_IND)
> > +			emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> > +				sreg, rg[EBPF_REG_2]);
> > +	} else
> > +		/* BPF_IND with sreg == R2 */
> > +		emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K,
> > +			rg[EBPF_REG_2], imm);
> > +
> > +	/* R3 = mbuf->data_len */
> > +	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
> > +		rg[EBPF_REG_6], rg[EBPF_REG_3],
> > +		offsetof(struct rte_mbuf, data_len));
> > +
> > +	/* R3 = R3 - R2 */
> > +	emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X,
> > +		rg[EBPF_REG_2], rg[EBPF_REG_3]);
> > +
> > +	/* JSLE R3, <sz> <slow_path> */
> > +	emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz);
> > +	emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]);
> > +
> > +	/* R3 = mbuf->data_off */
> > +	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
> > +		rg[EBPF_REG_6], rg[EBPF_REG_3],
> > +		offsetof(struct rte_mbuf, data_off));
> > +
> > +	/* R0 = mbuf->buf_addr */
> > +	emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW,
> > +		rg[EBPF_REG_6], rg[EBPF_REG_0],
> > +		offsetof(struct rte_mbuf, buf_addr));
> > +
> > +	/* R0 = R0 + R3 */
> > +	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> > +		rg[EBPF_REG_3], rg[EBPF_REG_0]);
> > +
> > +	/* R0 = R0 + R2 */
> > +	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> > +		rg[EBPF_REG_2], rg[EBPF_REG_0]);
> > +
> > +	/* JMP <fin_part> */
> > +	emit_abs_jmp(st, ofs[LDMB_FIN_OFS]);
> > +}
> > +
> > +/*
> > + * helper function, used by emit_ld_mbuf().
> > + * generates code for 'slow_path':
> > + * call __rte_pktmbuf_read() and check return value.
> > + */
> > +static void
> > +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t
> > rg[EBPF_REG_7],
> > +	uint32_t sz)
> > +{
> > +	/* make R3 contain *len* value (1/2/4) */
> > +
> > +	emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz);
> > +
> > +	/* make R4 contain (RBP - ldmb.stack_ofs) */
> > +
> > +	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]);
> > +	emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4],
> > +		st->ldmb.stack_ofs);
> > +
> > +	/* make R1 contain mbuf ptr */
> > +
> > +	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X,
> > +		rg[EBPF_REG_6], rg[EBPF_REG_1]);
> > +
> > +	/* call rte_pktmbuf_read */
> > +	emit_call(st, (uintptr_t)__rte_pktmbuf_read);
> > +
> > +	/* check that return value (R0) is not zero */
> > +	emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]);
> > +	emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off);
> > +}
> > +
> > +/*
> > + * helper function, used by emit_ld_mbuf().
> > + * generates final part of code for BPF_ABS/BPF_IND load:
> > + * perform data load and endianness conversion.
> > + * expects dreg to contain valid data pointer.
> > + */
> > +static void
> > +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz,
> > +	uint32_t sz)
> > +{
> > +	emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0);
> > +	if (sz != sizeof(uint8_t))
> > +		emit_be2le(st, dreg, sz * CHAR_BIT);
> > +}
> > +
> > +/*
> > + * emit code for BPF_ABS/BPF_IND load.
> > + * generates the following construction:
> > + * fast_path:
> > + *   off = ins->sreg + ins->imm
> > + *   if (off + ins->opsz < mbuf->data_len)
> > + *      goto slow_path;
> 
> I am not an eBPF expert, but I am not sure this is correct.
> 
> I think it should be > instead of <. Also, it looks like you actually emit:
> if (mbuf->data_len - off <= ins->opsz)
>    goto slow_path;
> 
> Could you please double check that both the comment and the emitted code has the comparison turning the correct way.

Ack, should be:
if (mbuf->data_len - off < ins->opsz)
Will send v2.



^ permalink raw reply	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions
  2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                   ` (4 preceding siblings ...)
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
@ 2020-05-27 14:16 ` Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev
                     ` (5 more replies)
  5 siblings, 6 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev

To fill the gap with linux kernel eBPF implementation,
add support for two non-generic instructions:
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
which are used to access packet data in a safe manner.
Make necessary changes in BPF verifier, interpreter and x86 JIT code.

Konstantin Ananyev (5):
  test/bpf: fix few small issues
  bpf: fix add/sub min/max estimations
  bpf: add support for packet data load instructions
  test/bpf: add new test cases for mbuf load instructions
  bpf: x86 JIT support for packet data load instructions

 app/test/test_bpf.c                    | 504 +++++++++++++++++++++++--
 doc/guides/prog_guide/bpf_lib.rst      |  30 +-
 doc/guides/rel_notes/release_20_08.rst |   7 +
 lib/librte_bpf/bpf_exec.c              |  57 +++
 lib/librte_bpf/bpf_jit_x86.c           | 181 +++++++++
 lib/librte_bpf/bpf_validate.c          | 104 ++++-
 6 files changed, 850 insertions(+), 33 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
@ 2020-05-27 14:16   ` Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
                     ` (4 subsequent siblings)
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev, stable

Address for few small issues:
 - unreachable return statement
 - failed test-case can finish with 'success' status

Also use unified cmp_res() function to check return value.

Fixes: a9de470cc7c0 ("test: move to app directory")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/test_bpf.c | 41 +++++++++++++++--------------------------
 1 file changed, 15 insertions(+), 26 deletions(-)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index ee534687a..4a61a7d7c 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -1797,13 +1797,6 @@ test_call1_check(uint64_t rc, const void *arg)
 	dummy_func1(arg, &v32, &v64);
 	v64 += v32;
 
-	if (v64 != rc) {
-		printf("%s@%d: invalid return value "
-			"expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n",
-			__func__, __LINE__, v64, rc);
-		return -1;
-	}
-	return 0;
 	return cmp_res(__func__, v64, rc, dv, dv, sizeof(*dv));
 }
 
@@ -1934,13 +1927,7 @@ test_call2_check(uint64_t rc, const void *arg)
 	dummy_func2(&a, &b);
 	v = a.u64 + a.u32 + b.u16 + b.u8;
 
-	if (v != rc) {
-		printf("%s@%d: invalid return value "
-			"expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n",
-			__func__, __LINE__, v, rc);
-		return -1;
-	}
-	return 0;
+	return cmp_res(__func__, v, rc, arg, arg, 0);
 }
 
 static const struct rte_bpf_xsym test_call2_xsym[] = {
@@ -2429,7 +2416,6 @@ test_call5_check(uint64_t rc, const void *arg)
 	v = 0;
 
 fail:
-
 	return cmp_res(__func__, v, rc, &v, &rc, sizeof(v));
 }
 
@@ -2458,6 +2444,7 @@ static const struct rte_bpf_xsym test_call5_xsym[] = {
 	},
 };
 
+/* all bpf test cases */
 static const struct bpf_test tests[] = {
 	{
 		.name = "test_store1",
@@ -2738,7 +2725,6 @@ run_test(const struct bpf_test *tst)
 	}
 
 	tst->prepare(tbuf);
-
 	rc = rte_bpf_exec(bpf, tbuf);
 	ret = tst->check_result(rc, tbuf);
 	if (ret != 0) {
@@ -2746,17 +2732,20 @@ run_test(const struct bpf_test *tst)
 			__func__, __LINE__, tst->name, ret, strerror(ret));
 	}
 
+	/* repeat the same test with jit, when possible */
 	rte_bpf_get_jit(bpf, &jit);
-	if (jit.func == NULL)
-		return 0;
-
-	tst->prepare(tbuf);
-	rc = jit.func(tbuf);
-	rv = tst->check_result(rc, tbuf);
-	ret |= rv;
-	if (rv != 0) {
-		printf("%s@%d: check_result(%s) failed, error: %d(%s);\n",
-			__func__, __LINE__, tst->name, rv, strerror(ret));
+	if (jit.func != NULL) {
+
+		tst->prepare(tbuf);
+		rc = jit.func(tbuf);
+		rv = tst->check_result(rc, tbuf);
+		ret |= rv;
+		if (rv != 0) {
+			printf("%s@%d: check_result(%s) failed, "
+				"error: %d(%s);\n",
+				__func__, __LINE__, tst->name,
+				rv, strerror(ret));
+		}
 	}
 
 	rte_bpf_destroy(bpf);
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev
@ 2020-05-27 14:16   ` Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
                     ` (3 subsequent siblings)
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev, stable

eval_add()/eval_sub() not always correctly estimate
minimum and maximum possible values of add/sub operations.

Fixes: 8021917293d0 ("bpf: add extra validation for input BPF program")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_bpf/bpf_validate.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index 6bd6f78e9..80d21fabb 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -226,7 +226,7 @@ eval_add(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk)
 	struct bpf_reg_val rv;
 
 	rv.u.min = (rd->u.min + rs->u.min) & msk;
-	rv.u.max = (rd->u.min + rs->u.max) & msk;
+	rv.u.max = (rd->u.max + rs->u.max) & msk;
 	rv.s.min = (rd->s.min + rs->s.min) & msk;
 	rv.s.max = (rd->s.max + rs->s.max) & msk;
 
@@ -254,10 +254,10 @@ eval_sub(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk)
 {
 	struct bpf_reg_val rv;
 
-	rv.u.min = (rd->u.min - rs->u.min) & msk;
-	rv.u.max = (rd->u.min - rs->u.max) & msk;
-	rv.s.min = (rd->s.min - rs->s.min) & msk;
-	rv.s.max = (rd->s.max - rs->s.max) & msk;
+	rv.u.min = (rd->u.min - rs->u.max) & msk;
+	rv.u.max = (rd->u.max - rs->u.min) & msk;
+	rv.s.min = (rd->s.min - rs->s.max) & msk;
+	rv.s.max = (rd->s.max - rs->s.min) & msk;
 
 	/*
 	 * if at least one of the operands is not constant,
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
@ 2020-05-27 14:16   ` Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
                     ` (2 subsequent siblings)
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev

To fill the gap with linux kernel eBPF implementation,
add support for two non-generic instructions:
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
which are used to access packet data.
These instructions can only be used when BPF context is a pointer
to 'struct rte_mbuf' (i.e: RTE_BPF_ARG_PTR_MBUF type).

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/prog_guide/bpf_lib.rst      | 30 +++++++++-
 doc/guides/rel_notes/release_20_08.rst |  7 +++
 lib/librte_bpf/bpf_exec.c              | 57 +++++++++++++++++++
 lib/librte_bpf/bpf_validate.c          | 77 ++++++++++++++++++++++++++
 4 files changed, 170 insertions(+), 1 deletion(-)

diff --git a/doc/guides/prog_guide/bpf_lib.rst b/doc/guides/prog_guide/bpf_lib.rst
index 9c728da7b..1feb7734a 100644
--- a/doc/guides/prog_guide/bpf_lib.rst
+++ b/doc/guides/prog_guide/bpf_lib.rst
@@ -27,6 +27,35 @@ The library API provides the following basic operations:
 
 *   Load BPF program from the ELF file and install callback to execute it on given ethdev port/queue.
 
+Packet data load instructions
+-----------------------------
+
+DPDK supports two non-generic instructions: ``(BPF_ABS | size | BPF_LD)``
+and ``(BPF_IND | size | BPF_LD)`` which are used to access packet data.
+These instructions can only be used when execution context is a pointer to
+``struct rte_mbuf`` and have seven implicit operands.
+Register ``R6`` is an implicit input that must contain pointer to ``rte_mbuf``.
+Register ``R0`` is an implicit output which contains the data fetched from the
+packet. Registers ``R1-R5`` are scratch registers
+and must not be used to store the data across these instructions.
+These instructions have implicit program exit condition as well. When
+eBPF program is trying to access the data beyond the packet boundary,
+the interpreter will abort the execution of the program. JIT compilers
+therefore must preserve this property. ``src_reg`` and ``imm32`` fields are
+explicit inputs to these instructions.
+For example, ``(BPF_IND | BPF_W | BPF_LD)`` means:
+
+.. code-block:: c
+
+    uint32_t tmp;
+    R0 = rte_pktmbuf_read((const struct rte_mbuf *)R6,  src_reg + imm32,
+	sizeof(tmp), &tmp);
+    if (R0 == NULL) return FAILED;
+    R0 = ntohl(*(uint32_t *)R0);
+
+and ``R1-R5`` were scratched.
+
+
 Not currently supported eBPF features
 -------------------------------------
 
@@ -34,5 +63,4 @@ Not currently supported eBPF features
  - cBPF
  - tail-pointer call
  - eBPF MAP
- - skb
  - external function calls for 32-bit platforms
diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst
index 39064afbe..3c0824616 100644
--- a/doc/guides/rel_notes/release_20_08.rst
+++ b/doc/guides/rel_notes/release_20_08.rst
@@ -56,6 +56,13 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added support for BPF_ABS/BPF_IND load instructions.**
+
+  Added support for two BPF non-generic instructions:
+  ``(BPF_ABS | <size> | BPF_LD)`` and ``(BPF_IND | <size> | BPF_LD)``
+  which are used to access packet data in a safe manner. Currently JIT support
+  for these instructions is implemented for x86 only.
+
 
 Removed Items
 -------------
diff --git a/lib/librte_bpf/bpf_exec.c b/lib/librte_bpf/bpf_exec.c
index 1bb226643..b921112fe 100644
--- a/lib/librte_bpf/bpf_exec.c
+++ b/lib/librte_bpf/bpf_exec.c
@@ -74,6 +74,26 @@
 		(uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \
 		reg[ins->src_reg]))
 
+/* BPF_LD | BPF_ABS/BPF_IND */
+
+#define	NOP(x)	(x)
+
+#define BPF_LD_ABS(bpf, reg, ins, type, op) do { \
+	const type *p = bpf_ld_mbuf(bpf, reg, ins, (ins)->imm, sizeof(type)); \
+	if (p == NULL)  \
+		return 0; \
+	reg[EBPF_REG_0] = op(p[0]); \
+} while (0)
+
+#define BPF_LD_IND(bpf, reg, ins, type, op) do { \
+	uint32_t ofs = reg[ins->src_reg] + (ins)->imm; \
+	const type *p = bpf_ld_mbuf(bpf, reg, ins, ofs, sizeof(type)); \
+	if (p == NULL)  \
+		return 0; \
+	reg[EBPF_REG_0] = op(p[0]); \
+} while (0)
+
+
 static inline void
 bpf_alu_be(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins)
 {
@@ -112,6 +132,23 @@ bpf_alu_le(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins)
 	}
 }
 
+static inline const void *
+bpf_ld_mbuf(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM],
+	const struct ebpf_insn *ins, uint32_t off, uint32_t len)
+{
+	const struct rte_mbuf *mb;
+	const void *p;
+
+	mb = (const struct rte_mbuf *)(uintptr_t)reg[EBPF_REG_6];
+	p = rte_pktmbuf_read(mb, off, len, reg + EBPF_REG_0);
+	if (p == NULL)
+		RTE_BPF_LOG(DEBUG, "%s(bpf=%p, mbuf=%p, ofs=%u, len=%u): "
+			"load beyond packet boundary at pc: %#zx;\n",
+			__func__, bpf, mb, off, len,
+			(uintptr_t)(ins) - (uintptr_t)(bpf)->prm.ins);
+	return p;
+}
+
 static inline uint64_t
 bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM])
 {
@@ -296,6 +333,26 @@ bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM])
 				(uint64_t)(uint32_t)ins[1].imm << 32;
 			ins++;
 			break;
+		/* load absolute instructions */
+		case (BPF_LD | BPF_ABS | BPF_B):
+			BPF_LD_ABS(bpf, reg, ins, uint8_t, NOP);
+			break;
+		case (BPF_LD | BPF_ABS | BPF_H):
+			BPF_LD_ABS(bpf, reg, ins, uint16_t, rte_be_to_cpu_16);
+			break;
+		case (BPF_LD | BPF_ABS | BPF_W):
+			BPF_LD_ABS(bpf, reg, ins, uint32_t, rte_be_to_cpu_32);
+			break;
+		/* load indirect instructions */
+		case (BPF_LD | BPF_IND | BPF_B):
+			BPF_LD_IND(bpf, reg, ins, uint8_t, NOP);
+			break;
+		case (BPF_LD | BPF_IND | BPF_H):
+			BPF_LD_IND(bpf, reg, ins, uint16_t, rte_be_to_cpu_16);
+			break;
+		case (BPF_LD | BPF_IND | BPF_W):
+			BPF_LD_IND(bpf, reg, ins, uint32_t, rte_be_to_cpu_32);
+			break;
 		/* store instructions */
 		case (BPF_STX | BPF_MEM | BPF_B):
 			BPF_ST_REG(reg, ins, uint8_t);
diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index 80d21fabb..fecdda0e1 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -102,6 +102,9 @@ struct bpf_ins_check {
 #define	WRT_REGS	RTE_LEN2MASK(EBPF_REG_10, uint16_t)
 #define	ZERO_REG	RTE_LEN2MASK(EBPF_REG_1, uint16_t)
 
+/* For LD_IND R6 is an implicit CTX register. */
+#define	IND_SRC_REGS	(WRT_REGS ^ 1 << EBPF_REG_6)
+
 /*
  * check and evaluate functions for particular instruction types.
  */
@@ -580,6 +583,42 @@ eval_neg(struct bpf_reg_val *rd, size_t opsz, uint64_t msk)
 	rd->s.min = RTE_MIN(sx, sy);
 }
 
+static const char *
+eval_ld_mbuf(struct bpf_verifier *bvf, const struct ebpf_insn *ins)
+{
+	uint32_t i, mode;
+	struct bpf_reg_val *rv, ri, rs;
+
+	mode = BPF_MODE(ins->code);
+
+	/* R6 is an implicit input that must contain pointer to mbuf */
+	if (bvf->evst->rv[EBPF_REG_6].v.type != RTE_BPF_ARG_PTR_MBUF)
+		return "invalid type for implicit ctx register";
+
+	if (mode == BPF_IND) {
+		rs = bvf->evst->rv[ins->src_reg];
+		if (rs.v.type != RTE_BPF_ARG_RAW)
+			return "unexpected type for src register";
+
+		eval_fill_imm(&ri, UINT64_MAX, ins->imm);
+		eval_add(&rs, &ri, UINT64_MAX);
+
+		if (rs.s.max < 0 || rs.u.min > UINT32_MAX)
+			return "mbuf boundary violation";
+	}
+
+	/* R1-R5 scratch registers */
+	for (i = EBPF_REG_1; i != EBPF_REG_6; i++)
+		bvf->evst->rv[i].v.type = RTE_BPF_ARG_UNDEF;
+
+	/* R0 is an implicit output, contains data fetched from the packet */
+	rv = bvf->evst->rv + EBPF_REG_0;
+	rv->v.size = bpf_size(BPF_SIZE(ins->code));
+	eval_fill_max_bound(rv, RTE_LEN2MASK(rv->v.size * CHAR_BIT, uint64_t));
+
+	return NULL;
+}
+
 /*
  * check that destination and source operand are in defined state.
  */
@@ -1425,6 +1464,44 @@ static const struct bpf_ins_check ins_chk[UINT8_MAX + 1] = {
 		.imm = { .min = 0, .max = UINT32_MAX},
 		.eval = eval_ld_imm64,
 	},
+	/* load absolute instructions */
+	[(BPF_LD | BPF_ABS | BPF_B)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_ABS | BPF_H)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_ABS | BPF_W)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = ZERO_REG},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = INT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	/* load indirect instructions */
+	[(BPF_LD | BPF_IND | BPF_B)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_IND | BPF_H)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
+	[(BPF_LD | BPF_IND | BPF_W)] = {
+		.mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS},
+		.off = { .min = 0, .max = 0},
+		.imm = { .min = 0, .max = UINT32_MAX},
+		.eval = eval_ld_mbuf,
+	},
 	/* store REG instructions */
 	[(BPF_STX | BPF_MEM | BPF_B)] = {
 		.mask = { .dreg = ALL_REGS, .sreg = ALL_REGS},
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf load instructions
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                     ` (2 preceding siblings ...)
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
@ 2020-05-27 14:16   ` Konstantin Ananyev
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
  2020-06-24 21:43   ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon
  5 siblings, 0 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev

Add new test-cases for BPF_ABS/BPF_IND load instructions.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/test_bpf.c | 463 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 463 insertions(+)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 4a61a7d7c..7c3de96c6 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -43,6 +43,14 @@ struct dummy_net {
 	struct rte_ipv4_hdr ip_hdr;
 };
 
+#define	DUMMY_MBUF_NUM	2
+
+/* first mbuf in the packet, should always be at offset 0 */
+struct dummy_mbuf {
+	struct rte_mbuf mb[DUMMY_MBUF_NUM];
+	uint8_t buf[DUMMY_MBUF_NUM][RTE_MBUF_DEFAULT_BUF_SIZE];
+};
+
 #define	TEST_FILL_1	0xDEADBEEF
 
 #define	TEST_MUL_1	21
@@ -2444,6 +2452,413 @@ static const struct rte_bpf_xsym test_call5_xsym[] = {
 	},
 };
 
+/* load mbuf (BPF_ABS/BPF_IND) test-cases */
+static const struct ebpf_insn test_ld_mbuf1_prog[] = {
+
+	/* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_6,
+		.src_reg = EBPF_REG_1,
+	},
+	/* load IPv4 version and IHL */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_B),
+		.imm = offsetof(struct rte_ipv4_hdr, version_ihl),
+	},
+	/* check IP version */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_2,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = 0xf0,
+	},
+	{
+		.code = (BPF_JMP | BPF_JEQ | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = IPVERSION << 4,
+		.off = 2,
+	},
+	/* invalid IP version, return 0 */
+	{
+		.code = (EBPF_ALU64 | BPF_XOR | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+	/* load 3-rd byte of IP data */
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = RTE_IPV4_HDR_IHL_MASK,
+	},
+	{
+		.code = (BPF_ALU | BPF_LSH | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = 2,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_B),
+		.src_reg = EBPF_REG_0,
+		.imm = 3,
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load IPv4 src addr */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_W),
+		.imm = offsetof(struct rte_ipv4_hdr, src_addr),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load IPv4 total length */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_H),
+		.imm = offsetof(struct rte_ipv4_hdr, total_length),
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_8,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load last 4 bytes of IP data */
+	{
+		.code = (BPF_LD | BPF_IND | BPF_W),
+		.src_reg = EBPF_REG_8,
+		.imm = -(int32_t)sizeof(uint32_t),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_7,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load 2 bytes from the middle of IP data */
+	{
+		.code = (EBPF_ALU64 | BPF_RSH | BPF_K),
+		.dst_reg = EBPF_REG_8,
+		.imm = 1,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_H),
+		.src_reg = EBPF_REG_8,
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_7,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+};
+
+static void
+dummy_mbuf_prep(struct rte_mbuf *mb, uint8_t buf[], uint32_t buf_len,
+	uint32_t data_len)
+{
+	uint32_t i;
+	uint8_t *db;
+
+	mb->buf_addr = buf;
+	mb->buf_iova = (uintptr_t)buf;
+	mb->buf_len = buf_len;
+	rte_mbuf_refcnt_set(mb, 1);
+
+	/* set pool pointer to dummy value, test doesn't use it */
+	mb->pool = (void *)buf;
+
+	rte_pktmbuf_reset(mb);
+	db = (uint8_t *)rte_pktmbuf_append(mb, data_len);
+
+	for (i = 0; i != data_len; i++)
+		db[i] = i;
+}
+
+static void
+test_ld_mbuf1_prepare(void *arg)
+{
+	struct dummy_mbuf *dm;
+	struct rte_ipv4_hdr *ph;
+
+	const uint32_t plen = 400;
+	const struct rte_ipv4_hdr iph = {
+		.version_ihl = RTE_IPV4_VHL_DEF,
+		.total_length = rte_cpu_to_be_16(plen),
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_RAW,
+		.src_addr = rte_cpu_to_be_32(RTE_IPV4_LOOPBACK),
+		.dst_addr = rte_cpu_to_be_32(RTE_IPV4_BROADCAST),
+	};
+
+	dm = arg;
+	memset(dm, 0, sizeof(*dm));
+
+	dummy_mbuf_prep(&dm->mb[0], dm->buf[0], sizeof(dm->buf[0]),
+		plen / 2 + 1);
+	dummy_mbuf_prep(&dm->mb[1], dm->buf[1], sizeof(dm->buf[0]),
+		plen / 2 - 1);
+
+	rte_pktmbuf_chain(&dm->mb[0], &dm->mb[1]);
+
+	ph = rte_pktmbuf_mtod(dm->mb, typeof(ph));
+	memcpy(ph, &iph, sizeof(iph));
+}
+
+static uint64_t
+test_ld_mbuf1(const struct rte_mbuf *pkt)
+{
+	uint64_t n, v;
+	const uint8_t *p8;
+	const uint16_t *p16;
+	const uint32_t *p32;
+	struct dummy_offset dof;
+
+	/* load IPv4 version and IHL */
+	p8 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, version_ihl), sizeof(*p8),
+		&dof);
+	if (p8 == NULL)
+		return 0;
+
+	/* check IP version */
+	if ((p8[0] & 0xf0) != IPVERSION << 4)
+		return 0;
+
+	n = (p8[0] & RTE_IPV4_HDR_IHL_MASK) * RTE_IPV4_IHL_MULTIPLIER;
+
+	/* load 3-rd byte of IP data */
+	p8 = rte_pktmbuf_read(pkt, n + 3, sizeof(*p8), &dof);
+	if (p8 == NULL)
+		return 0;
+
+	v = p8[0];
+
+	/* load IPv4 src addr */
+	p32 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, src_addr), sizeof(*p32),
+		&dof);
+	if (p32 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_32(p32[0]);
+
+	/* load IPv4 total length */
+	p16 = rte_pktmbuf_read(pkt,
+		offsetof(struct rte_ipv4_hdr, total_length), sizeof(*p16),
+		&dof);
+	if (p16 == NULL)
+		return 0;
+
+	n = rte_be_to_cpu_16(p16[0]);
+
+	/* load last 4 bytes of IP data */
+	p32 = rte_pktmbuf_read(pkt, n - sizeof(*p32), sizeof(*p32), &dof);
+	if (p32 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_32(p32[0]);
+
+	/* load 2 bytes from the middle of IP data */
+	p16 = rte_pktmbuf_read(pkt, n / 2, sizeof(*p16), &dof);
+	if (p16 == NULL)
+		return 0;
+
+	v += rte_be_to_cpu_16(p16[0]);
+	return v;
+}
+
+static int
+test_ld_mbuf1_check(uint64_t rc, const void *arg)
+{
+	const struct dummy_mbuf *dm;
+	uint64_t v;
+
+	dm = arg;
+	v = test_ld_mbuf1(dm->mb);
+	return cmp_res(__func__, v, rc, arg, arg, 0);
+}
+
+/*
+ * same as ld_mbuf1, but then trancate the mbuf by 1B,
+ * so load of last 4B fail.
+ */
+static void
+test_ld_mbuf2_prepare(void *arg)
+{
+	struct dummy_mbuf *dm;
+
+	test_ld_mbuf1_prepare(arg);
+	dm = arg;
+	rte_pktmbuf_trim(dm->mb, 1);
+}
+
+static int
+test_ld_mbuf2_check(uint64_t rc, const void *arg)
+{
+	return cmp_res(__func__, 0, rc, arg, arg, 0);
+}
+
+/* same as test_ld_mbuf1, but now store intermediate results on the stack */
+static const struct ebpf_insn test_ld_mbuf3_prog[] = {
+
+	/* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_6,
+		.src_reg = EBPF_REG_1,
+	},
+	/* load IPv4 version and IHL */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_B),
+		.imm = offsetof(struct rte_ipv4_hdr, version_ihl),
+	},
+	/* check IP version */
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_2,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = 0xf0,
+	},
+	{
+		.code = (BPF_JMP | BPF_JEQ | BPF_K),
+		.dst_reg = EBPF_REG_2,
+		.imm = IPVERSION << 4,
+		.off = 2,
+	},
+	/* invalid IP version, return 0 */
+	{
+		.code = (EBPF_ALU64 | BPF_XOR | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_0,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+	/* load 3-rd byte of IP data */
+	{
+		.code = (BPF_ALU | BPF_AND | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = RTE_IPV4_HDR_IHL_MASK,
+	},
+	{
+		.code = (BPF_ALU | BPF_LSH | BPF_K),
+		.dst_reg = EBPF_REG_0,
+		.imm = 2,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_B),
+		.src_reg = EBPF_REG_0,
+		.imm = 3,
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | BPF_B),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u8) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load IPv4 src addr */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_W),
+		.imm = offsetof(struct rte_ipv4_hdr, src_addr),
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | BPF_W),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u32) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load IPv4 total length */
+	{
+		.code = (BPF_LD | BPF_ABS | BPF_H),
+		.imm = offsetof(struct rte_ipv4_hdr, total_length),
+	},
+	{
+		.code = (EBPF_ALU64 | EBPF_MOV | BPF_X),
+		.dst_reg = EBPF_REG_8,
+		.src_reg = EBPF_REG_0,
+	},
+	/* load last 4 bytes of IP data */
+	{
+		.code = (BPF_LD | BPF_IND | BPF_W),
+		.src_reg = EBPF_REG_8,
+		.imm = -(int32_t)sizeof(uint32_t),
+	},
+	{
+		.code = (BPF_STX | BPF_MEM | EBPF_DW),
+		.dst_reg = EBPF_REG_10,
+		.src_reg = EBPF_REG_0,
+		.off = (int16_t)(offsetof(struct dummy_offset, u64) -
+			sizeof(struct dummy_offset)),
+	},
+	/* load 2 bytes from the middle of IP data */
+	{
+		.code = (EBPF_ALU64 | BPF_RSH | BPF_K),
+		.dst_reg = EBPF_REG_8,
+		.imm = 1,
+	},
+	{
+		.code = (BPF_LD | BPF_IND | BPF_H),
+		.src_reg = EBPF_REG_8,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | EBPF_DW),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u64) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | BPF_W),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u32) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_LDX | BPF_MEM | BPF_B),
+		.dst_reg = EBPF_REG_1,
+		.src_reg = EBPF_REG_10,
+		.off = (int16_t)(offsetof(struct dummy_offset, u8) -
+			sizeof(struct dummy_offset)),
+	},
+	{
+		.code = (EBPF_ALU64 | BPF_ADD | BPF_X),
+		.dst_reg = EBPF_REG_0,
+		.src_reg = EBPF_REG_1,
+	},
+	{
+		.code = (BPF_JMP | EBPF_EXIT),
+	},
+};
+
 /* all bpf test cases */
 static const struct bpf_test tests[] = {
 	{
@@ -2704,6 +3119,54 @@ static const struct bpf_test tests[] = {
 		/* for now don't support function calls on 32 bit platform */
 		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
 	},
+	{
+		.name = "test_ld_mbuf1",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf1_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf1_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf1_prepare,
+		.check_result = test_ld_mbuf1_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
+	{
+		.name = "test_ld_mbuf2",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf1_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf1_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf2_prepare,
+		.check_result = test_ld_mbuf2_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
+	{
+		.name = "test_ld_mbuf3",
+		.arg_sz = sizeof(struct dummy_mbuf),
+		.prm = {
+			.ins = test_ld_mbuf3_prog,
+			.nb_ins = RTE_DIM(test_ld_mbuf3_prog),
+			.prog_arg = {
+				.type = RTE_BPF_ARG_PTR_MBUF,
+				.buf_size = sizeof(struct dummy_mbuf),
+			},
+		},
+		.prepare = test_ld_mbuf1_prepare,
+		.check_result = test_ld_mbuf1_check,
+		/* mbuf as input argument is not supported on 32 bit platform */
+		.allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)),
+	},
 };
 
 static int
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data load instructions
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                     ` (3 preceding siblings ...)
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
@ 2020-05-27 14:16   ` Konstantin Ananyev
  2020-05-28  8:50     ` Morten Brørup
  2020-06-24 21:43   ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon
  5 siblings, 1 reply; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw)
  To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev

Make x86 JIT to generate native code for
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
instructions.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_bpf/bpf_jit_x86.c  | 181 ++++++++++++++++++++++++++++++++++
 lib/librte_bpf/bpf_validate.c |  17 +++-
 2 files changed, 197 insertions(+), 1 deletion(-)

diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c
index f70cd6be5..aa22ea78a 100644
--- a/lib/librte_bpf/bpf_jit_x86.c
+++ b/lib/librte_bpf/bpf_jit_x86.c
@@ -87,6 +87,14 @@ enum {
 	REG_TMP1 = R10,
 };
 
+/* LD_ABS/LD_IMM offsets */
+enum {
+	LDMB_FSP_OFS, /* fast-path */
+	LDMB_SLP_OFS, /* slow-path */
+	LDMB_FIN_OFS, /* final part */
+	LDMB_OFS_NUM
+};
+
 /*
  * callee saved registers list.
  * keep RBP as the last one.
@@ -100,6 +108,9 @@ struct bpf_jit_state {
 		uint32_t num;
 		int32_t off;
 	} exit;
+	struct {
+		uint32_t stack_ofs;
+	} ldmb;
 	uint32_t reguse;
 	int32_t *off;
 	uint8_t *ins;
@@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t dreg,
 		emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX);
 }
 
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates code for 'fast_path':
+ * calculate load offset and check is it inside first packet segment.
+ */
+static void
+emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7],
+	uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm,
+	const int32_t ofs[LDMB_OFS_NUM])
+{
+	/* make R2 contain *off* value */
+
+	if (sreg != rg[EBPF_REG_2]) {
+		emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K,
+			rg[EBPF_REG_2], imm);
+		if (mode == BPF_IND)
+			emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+				sreg, rg[EBPF_REG_2]);
+	} else
+		/* BPF_IND with sreg == R2 */
+		emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K,
+			rg[EBPF_REG_2], imm);
+
+	/* R3 = mbuf->data_len */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
+		rg[EBPF_REG_6], rg[EBPF_REG_3],
+		offsetof(struct rte_mbuf, data_len));
+
+	/* R3 = R3 - R2 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X,
+		rg[EBPF_REG_2], rg[EBPF_REG_3]);
+
+	/* JSLT R3, <sz> <slow_path> */
+	emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz);
+	emit_abs_jcc(st, BPF_JMP | EBPF_JSLT | BPF_K, ofs[LDMB_SLP_OFS]);
+
+	/* R3 = mbuf->data_off */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
+		rg[EBPF_REG_6], rg[EBPF_REG_3],
+		offsetof(struct rte_mbuf, data_off));
+
+	/* R0 = mbuf->buf_addr */
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW,
+		rg[EBPF_REG_6], rg[EBPF_REG_0],
+		offsetof(struct rte_mbuf, buf_addr));
+
+	/* R0 = R0 + R3 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+		rg[EBPF_REG_3], rg[EBPF_REG_0]);
+
+	/* R0 = R0 + R2 */
+	emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
+		rg[EBPF_REG_2], rg[EBPF_REG_0]);
+
+	/* JMP <fin_part> */
+	emit_abs_jmp(st, ofs[LDMB_FIN_OFS]);
+}
+
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates code for 'slow_path':
+ * call __rte_pktmbuf_read() and check return value.
+ */
+static void
+emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7],
+	uint32_t sz)
+{
+	/* make R3 contain *len* value (1/2/4) */
+
+	emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz);
+
+	/* make R4 contain (RBP - ldmb.stack_ofs) */
+
+	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]);
+	emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4],
+		st->ldmb.stack_ofs);
+
+	/* make R1 contain mbuf ptr */
+
+	emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X,
+		rg[EBPF_REG_6], rg[EBPF_REG_1]);
+
+	/* call rte_pktmbuf_read */
+	emit_call(st, (uintptr_t)__rte_pktmbuf_read);
+
+	/* check that return value (R0) is not zero */
+	emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]);
+	emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off);
+}
+
+/*
+ * helper function, used by emit_ld_mbuf().
+ * generates final part of code for BPF_ABS/BPF_IND load:
+ * perform data load and endianness conversion.
+ * expects dreg to contain valid data pointer.
+ */
+static void
+emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz,
+	uint32_t sz)
+{
+	emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0);
+	if (sz != sizeof(uint8_t))
+		emit_be2le(st, dreg, sz * CHAR_BIT);
+}
+
+/*
+ * emit code for BPF_ABS/BPF_IND load.
+ * generates the following construction:
+ * fast_path:
+ *   off = ins->sreg + ins->imm
+ *   if (mbuf->data_len - off < ins->opsz)
+ *      goto slow_path;
+ *   ptr = mbuf->buf_addr + mbuf->data_off + off;
+ *   goto fin_part;
+ * slow_path:
+ *   typeof(ins->opsz) buf; //allocate space on the stack
+ *   ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf);
+ *   if (ptr == NULL)
+ *      goto exit_label;
+ * fin_part:
+ *   res = *(typeof(ins->opsz))ptr;
+ *   res = bswap(res);
+ */
+static void
+emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t imm)
+{
+	uint32_t i, mode, opsz, sz;
+	uint32_t rg[EBPF_REG_7];
+	int32_t ofs[LDMB_OFS_NUM];
+
+	mode = BPF_MODE(op);
+	opsz = BPF_SIZE(op);
+	sz = bpf_size(opsz);
+
+	for (i = 0; i != RTE_DIM(rg); i++)
+		rg[i] = ebpf2x86[i];
+
+	/* fill with fake offsets */
+	for (i = 0; i != RTE_DIM(ofs); i++)
+		ofs[i] = st->sz + INT8_MAX;
+
+	/* dry run first to calculate jump offsets */
+
+	ofs[LDMB_FSP_OFS] = st->sz;
+	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
+	ofs[LDMB_SLP_OFS] = st->sz;
+	emit_ldmb_slow_path(st, rg, sz);
+	ofs[LDMB_FIN_OFS] = st->sz;
+	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
+
+	RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX);
+
+	/* reset dry-run code and do a proper run */
+
+	st->sz = ofs[LDMB_FSP_OFS];
+	emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs);
+	emit_ldmb_slow_path(st, rg, sz);
+	emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz);
+}
+
 static void
 emit_prolog(struct bpf_jit_state *st, int32_t stack_size)
 {
@@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf)
 	/* reset state fields */
 	st->sz = 0;
 	st->exit.num = 0;
+	st->ldmb.stack_ofs = bpf->stack_sz;
 
 	emit_prolog(st, bpf->stack_sz);
 
@@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf)
 			emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm);
 			i++;
 			break;
+		/* load absolute/indirect instructions */
+		case (BPF_LD | BPF_ABS | BPF_B):
+		case (BPF_LD | BPF_ABS | BPF_H):
+		case (BPF_LD | BPF_ABS | BPF_W):
+		case (BPF_LD | BPF_IND | BPF_B):
+		case (BPF_LD | BPF_IND | BPF_H):
+		case (BPF_LD | BPF_IND | BPF_W):
+			emit_ld_mbuf(st, op, sr, ins->imm);
+			break;
 		/* store instructions */
 		case (BPF_STX | BPF_MEM | BPF_B):
 		case (BPF_STX | BPF_MEM | BPF_H):
diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c
index fecdda0e1..9214f1503 100644
--- a/lib/librte_bpf/bpf_validate.c
+++ b/lib/librte_bpf/bpf_validate.c
@@ -70,6 +70,7 @@ struct bpf_verifier {
 	uint64_t stack_sz;
 	uint32_t nb_nodes;
 	uint32_t nb_jcc_nodes;
+	uint32_t nb_ldmb_nodes;
 	uint32_t node_colour[MAX_NODE_COLOUR];
 	uint32_t edge_type[MAX_EDGE_TYPE];
 	struct bpf_eval_state *evst;
@@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf)
 			rc |= add_edge(bvf, node, i + 2);
 			i++;
 			break;
+		case (BPF_LD | BPF_ABS | BPF_B):
+		case (BPF_LD | BPF_ABS | BPF_H):
+		case (BPF_LD | BPF_ABS | BPF_W):
+		case (BPF_LD | BPF_IND | BPF_B):
+		case (BPF_LD | BPF_IND | BPF_H):
+		case (BPF_LD | BPF_IND | BPF_W):
+			bvf->nb_ldmb_nodes++;
+			/* fallthrough */
 		default:
 			rc |= add_edge(bvf, node, i + 1);
 			break;
@@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf)
 	free(bvf.in);
 
 	/* copy collected info */
-	if (rc == 0)
+	if (rc == 0) {
 		bpf->stack_sz = bvf.stack_sz;
 
+		/* for LD_ABS/LD_IND, we'll need extra space on the stack */
+		if (bvf.nb_ldmb_nodes != 0)
+			bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz +
+				sizeof(uint64_t), sizeof(uint64_t));
+	}
+
 	return rc;
 }
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data load instructions
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
@ 2020-05-28  8:50     ` Morten Brørup
  0 siblings, 0 replies; 17+ messages in thread
From: Morten Brørup @ 2020-05-28  8:50 UTC (permalink / raw)
  To: Konstantin Ananyev, dev; +Cc: jerinj, stephen

> From: Konstantin Ananyev [mailto:konstantin.ananyev@intel.com]
> Sent: Wednesday, May 27, 2020 4:17 PM
> 
> Make x86 JIT to generate native code for
> (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
> instructions.
> 
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---

[...]

> +/*
> + * helper function, used by emit_ld_mbuf().
> + * generates code for 'fast_path':
> + * calculate load offset and check is it inside first packet segment.
> + */
> +static void
> +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t
> rg[EBPF_REG_7],
> +	uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm,
> +	const int32_t ofs[LDMB_OFS_NUM])
> +{
> +	/* make R2 contain *off* value */
> +
> +	if (sreg != rg[EBPF_REG_2]) {
> +		emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K,
> +			rg[EBPF_REG_2], imm);
> +		if (mode == BPF_IND)
> +			emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X,
> +				sreg, rg[EBPF_REG_2]);
> +	} else
> +		/* BPF_IND with sreg == R2 */
> +		emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K,
> +			rg[EBPF_REG_2], imm);
> +
> +	/* R3 = mbuf->data_len */
> +	emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H,
> +		rg[EBPF_REG_6], rg[EBPF_REG_3],
> +		offsetof(struct rte_mbuf, data_len));
> +
> +	/* R3 = R3 - R2 */
> +	emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X,
> +		rg[EBPF_REG_2], rg[EBPF_REG_3]);
> +
> +	/* JSLT R3, <sz> <slow_path> */
> +	emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz);
> +	emit_abs_jcc(st, BPF_JMP | EBPF_JSLT | BPF_K, ofs[LDMB_SLP_OFS]);
> +

[...]

> +
> +/*
> + * emit code for BPF_ABS/BPF_IND load.
> + * generates the following construction:
> + * fast_path:
> + *   off = ins->sreg + ins->imm
> + *   if (mbuf->data_len - off < ins->opsz)
> + *      goto slow_path;
> + *   ptr = mbuf->buf_addr + mbuf->data_off + off;
> + *   goto fin_part;
> + * slow_path:
> + *   typeof(ins->opsz) buf; //allocate space on the stack
> + *   ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf);
> + *   if (ptr == NULL)
> + *      goto exit_label;
> + * fin_part:
> + *   res = *(typeof(ins->opsz))ptr;
> + *   res = bswap(res);
> + */

[...] 

The comparison for jumping to the slow path looks correct now.

I haven't reviewed it all in depth, but it certainly deserves an:

Acked-by: Morten Brørup <mb@smartsharesystems.com>


^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [dpdk-stable] [PATCH 1/5] test/bpf: fix few small issues
  2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
@ 2020-06-24 21:33   ` Thomas Monjalon
  0 siblings, 0 replies; 17+ messages in thread
From: Thomas Monjalon @ 2020-06-24 21:33 UTC (permalink / raw)
  To: Konstantin Ananyev; +Cc: dev, stable, stephen, jerinj

18/05/2020 17:52, Konstantin Ananyev:
> Address for few small issues:
>  - unreachable return statement
>  - failed test-case can finish with 'success' status
> 
> Also use unified cmp_res() function to check return value.
> 
> Fixes: a9de470cc7c0 ("test: move to app directory")

Looks like you don't want to track the original commits.
In this case, better to not add any fake reference.

> Cc: stable@dpdk.org
> 
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>




^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions
  2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
                     ` (4 preceding siblings ...)
  2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
@ 2020-06-24 21:43   ` Thomas Monjalon
  5 siblings, 0 replies; 17+ messages in thread
From: Thomas Monjalon @ 2020-06-24 21:43 UTC (permalink / raw)
  To: Konstantin Ananyev; +Cc: dev, jerinj, stephen, mb

27/05/2020 16:16, Konstantin Ananyev:
> To fill the gap with linux kernel eBPF implementation,
> add support for two non-generic instructions:
> (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
> which are used to access packet data in a safe manner.
> Make necessary changes in BPF verifier, interpreter and x86 JIT code.
> 
> Konstantin Ananyev (5):
>   test/bpf: fix few small issues
>   bpf: fix add/sub min/max estimations
>   bpf: add support for packet data load instructions
>   test/bpf: add new test cases for mbuf load instructions
>   bpf: x86 JIT support for packet data load instructions

Applied, thanks



^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-06-24 21:43 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
2020-06-24 21:33   ` [dpdk-dev] [dpdk-stable] " Thomas Monjalon
2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
2020-05-24 13:37   ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup
2020-05-24 23:08     ` Ananyev, Konstantin
2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev
2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev
2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev
2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev
2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev
2020-05-27 14:16   ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev
2020-05-28  8:50     ` Morten Brørup
2020-06-24 21:43   ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.