[PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[Joe Stringer]

Update tools/lib/bpf to provide more functionality and improve interoperation
with other tools that generate and use eBPF code.

The kernel uapi headers are a bit newer than the version in the tools/
directory; synchronize those.

samples/bpf/libbpf* has a bit more functionality than tools/lib/bpf, so extend
tools/lib/bpf/bpf* with these functions to bring them into parity.

tools/lib/bpf cannot read ELFs that tc can read, and vice versa. Update the
map definition to be the same as in tc so the ELFs may be interchangeable
(at least for now; I don't have a long-term plan in mind to ensure this always
works).

Joe Stringer (3):
  tools lib bpf: Sync {tools,}/include/uapi/linux/bpf.h
  tools lib bpf: Sync with samples/bpf/libbpf
  tools lib bpf: Sync bpf_map_def with tc

 tools/include/uapi/linux/bpf.h |  51 ++++++++++
 tools/lib/bpf/bpf.c            | 139 ++++++++++++++++++++++-----
 tools/lib/bpf/bpf.h            | 208 +++++++++++++++++++++++++++++++++++++++--
 tools/lib/bpf/libbpf.c         |   3 +-
 tools/lib/bpf/libbpf.h         |  11 ++-
 5 files changed, 375 insertions(+), 37 deletions(-)

-- 
2.9.3

[PATCH net-next 1/3] tools lib bpf: Sync {tools,}/include/uapi/linux/bpf.h

[Joe Stringer]

The tools version of this header is out of date; update it to the latest
version from the kernel headers.

Signed-off-by: Joe Stringer <joe@ovn.org>
---
 tools/include/uapi/linux/bpf.h | 51 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 9e5fc168c8a3..f09c70b97eca 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -95,6 +95,7 @@ enum bpf_prog_type {
 	BPF_PROG_TYPE_SCHED_ACT,
 	BPF_PROG_TYPE_TRACEPOINT,
 	BPF_PROG_TYPE_XDP,
+	BPF_PROG_TYPE_PERF_EVENT,
 };
 
 #define BPF_PSEUDO_MAP_FD	1
@@ -375,6 +376,56 @@ enum bpf_func_id {
 	 */
 	BPF_FUNC_probe_write_user,
 
+	/**
+	 * bpf_current_task_under_cgroup(map, index) - Check cgroup2 membership of current task
+	 * @map: pointer to bpf_map in BPF_MAP_TYPE_CGROUP_ARRAY type
+	 * @index: index of the cgroup in the bpf_map
+	 * Return:
+	 *   == 0 current failed the cgroup2 descendant test
+	 *   == 1 current succeeded the cgroup2 descendant test
+	 *    < 0 error
+	 */
+	BPF_FUNC_current_task_under_cgroup,
+
+	/**
+	 * bpf_skb_change_tail(skb, len, flags)
+	 * The helper will resize the skb to the given new size,
+	 * to be used f.e. with control messages.
+	 * @skb: pointer to skb
+	 * @len: new skb length
+	 * @flags: reserved
+	 * Return: 0 on success or negative error
+	 */
+	BPF_FUNC_skb_change_tail,
+
+	/**
+	 * bpf_skb_pull_data(skb, len)
+	 * The helper will pull in non-linear data in case the
+	 * skb is non-linear and not all of len are part of the
+	 * linear section. Only needed for read/write with direct
+	 * packet access.
+	 * @skb: pointer to skb
+	 * @len: len to make read/writeable
+	 * Return: 0 on success or negative error
+	 */
+	BPF_FUNC_skb_pull_data,
+
+	/**
+	 * bpf_csum_update(skb, csum)
+	 * Adds csum into skb->csum in case of CHECKSUM_COMPLETE.
+	 * @skb: pointer to skb
+	 * @csum: csum to add
+	 * Return: csum on success or negative error
+	 */
+	BPF_FUNC_csum_update,
+
+	/**
+	 * bpf_set_hash_invalid(skb)
+	 * Invalidate current skb>hash.
+	 * @skb: pointer to skb
+	 */
+	BPF_FUNC_set_hash_invalid,
+
 	__BPF_FUNC_MAX_ID,
 };
 
-- 
2.9.3

[PATCH net-next 2/3] tools lib bpf: Sync with samples/bpf/libbpf

[Joe Stringer]

Extend the tools/ version of libbpf to include all of the functionality
provided in the samples/bpf version.

Signed-off-by: Joe Stringer <joe@ovn.org>
---
 tools/lib/bpf/bpf.c    | 139 +++++++++++++++++++++++++++------
 tools/lib/bpf/bpf.h    | 208 +++++++++++++++++++++++++++++++++++++++++++++++--
 tools/lib/bpf/libbpf.c |   3 +-
 3 files changed, 317 insertions(+), 33 deletions(-)

diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index 4212ed62235b..34b0511b3baa 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -20,10 +20,17 @@
  */
 
 #include <stdlib.h>
-#include <memory.h>
+#include <stdio.h>
 #include <unistd.h>
 #include <asm/unistd.h>
+#include <string.h>
+#include <linux/netlink.h>
 #include <linux/bpf.h>
+#include <errno.h>
+#include <net/ethernet.h>
+#include <net/if.h>
+#include <linux/if_packet.h>
+#include <arpa/inet.h>
 #include "bpf.h"
 
 /*
@@ -53,24 +60,71 @@ static int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
 	return syscall(__NR_bpf, cmd, attr, size);
 }
 
-int bpf_create_map(enum bpf_map_type map_type, int key_size,
-		   int value_size, int max_entries)
+int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
+		   int max_entries, int map_flags)
 {
-	union bpf_attr attr;
+	union bpf_attr attr = {
+		.map_type = map_type,
+		.key_size = key_size,
+		.value_size = value_size,
+		.max_entries = max_entries,
+		.map_flags = map_flags,
+	};
 
-	memset(&attr, '\0', sizeof(attr));
+	return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+}
 
-	attr.map_type = map_type;
-	attr.key_size = key_size;
-	attr.value_size = value_size;
-	attr.max_entries = max_entries;
+int bpf_update_elem(int fd, void *key, void *value, unsigned long long flags)
+{
+	union bpf_attr attr = {
+		.map_fd = fd,
+		.key = ptr_to_u64(key),
+		.value = ptr_to_u64(value),
+		.flags = flags,
+	};
 
-	return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+	return sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
+}
+
+int bpf_lookup_elem(int fd, void *key, void *value)
+{
+	union bpf_attr attr = {
+		.map_fd = fd,
+		.key = ptr_to_u64(key),
+		.value = ptr_to_u64(value),
+	};
+
+	return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+}
+
+int bpf_delete_elem(int fd, void *key)
+{
+	union bpf_attr attr = {
+		.map_fd = fd,
+		.key = ptr_to_u64(key),
+	};
+
+	return sys_bpf(BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
 }
 
-int bpf_load_program(enum bpf_prog_type type, struct bpf_insn *insns,
-		     size_t insns_cnt, char *license,
-		     u32 kern_version, char *log_buf, size_t log_buf_sz)
+int bpf_get_next_key(int fd, void *key, void *next_key)
+{
+	union bpf_attr attr = {
+		.map_fd = fd,
+		.key = ptr_to_u64(key),
+		.next_key = ptr_to_u64(next_key),
+	};
+
+	return sys_bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
+}
+
+#define ROUND_UP(x, n) (((x) + (n) - 1u) & ~((n) - 1u))
+
+
+int bpf_load_program(enum bpf_prog_type type,
+		     const struct bpf_insn *insns, int insns_cnt,
+		     const char *license, int kern_version,
+		     char *log_buf, size_t log_buf_sz)
 {
 	int fd;
 	union bpf_attr attr;
@@ -78,8 +132,8 @@ int bpf_load_program(enum bpf_prog_type type, struct bpf_insn *insns,
 	bzero(&attr, sizeof(attr));
 	attr.prog_type = type;
 	attr.insn_cnt = (__u32)insns_cnt;
-	attr.insns = ptr_to_u64(insns);
-	attr.license = ptr_to_u64(license);
+	attr.insns = ptr_to_u64((void *)insns);
+	attr.license = ptr_to_u64((void *)license);
 	attr.log_buf = ptr_to_u64(NULL);
 	attr.log_size = 0;
 	attr.log_level = 0;
@@ -97,16 +151,53 @@ int bpf_load_program(enum bpf_prog_type type, struct bpf_insn *insns,
 	return sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
 }
 
-int bpf_map_update_elem(int fd, void *key, void *value,
-			u64 flags)
+int bpf_obj_pin(int fd, const char *pathname)
 {
-	union bpf_attr attr;
+	union bpf_attr attr = {
+		.pathname	= ptr_to_u64((void *)pathname),
+		.bpf_fd		= fd,
+	};
 
-	bzero(&attr, sizeof(attr));
-	attr.map_fd = fd;
-	attr.key = ptr_to_u64(key);
-	attr.value = ptr_to_u64(value);
-	attr.flags = flags;
+	return sys_bpf(BPF_OBJ_PIN, &attr, sizeof(attr));
+}
 
-	return sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
+int bpf_obj_get(const char *pathname)
+{
+	union bpf_attr attr = {
+		.pathname	= ptr_to_u64((void *)pathname),
+	};
+
+	return sys_bpf(BPF_OBJ_GET, &attr, sizeof(attr));
+}
+
+int open_raw_sock(const char *name)
+{
+	struct sockaddr_ll sll;
+	int sock;
+
+	sock = socket(PF_PACKET, SOCK_RAW | SOCK_NONBLOCK | SOCK_CLOEXEC,
+		      htons(ETH_P_ALL));
+	if (sock < 0) {
+		printf("cannot create raw socket\n");
+		return -1;
+	}
+
+	memset(&sll, 0, sizeof(sll));
+	sll.sll_family = AF_PACKET;
+	sll.sll_ifindex = if_nametoindex(name);
+	sll.sll_protocol = htons(ETH_P_ALL);
+	if (bind(sock, (struct sockaddr *)&sll, sizeof(sll)) < 0) {
+		printf("bind to %s: %s\n", name, strerror(errno));
+		close(sock);
+		return -1;
+	}
+
+	return sock;
+}
+
+int perf_event_open(struct perf_event_attr *attr, int pid, int cpu,
+		    int group_fd, unsigned long flags)
+{
+	return syscall(__NR_perf_event_open, attr, pid, cpu,
+		       group_fd, flags);
 }
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index e8ba54087497..227edb23c022 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -23,16 +23,208 @@
 
 #include <linux/bpf.h>
 
+struct bpf_insn;
+
 int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
-		   int max_entries);
+		   int max_entries, int map_flags);
+int bpf_update_elem(int fd, void *key, void *value, unsigned long long flags);
+int bpf_lookup_elem(int fd, void *key, void *value);
+int bpf_delete_elem(int fd, void *key);
+int bpf_get_next_key(int fd, void *key, void *next_key);
+
+int bpf_load_program(enum bpf_prog_type prog_type,
+		     const struct bpf_insn *insns, int insn_len,
+		     const char *license, int kern_version,
+		     char *log_buf, size_t log_buf_sz);
+
+int bpf_obj_pin(int fd, const char *pathname);
+int bpf_obj_get(const char *pathname);
 
-/* Recommend log buffer size */
 #define BPF_LOG_BUF_SIZE 65536
-int bpf_load_program(enum bpf_prog_type type, struct bpf_insn *insns,
-		     size_t insns_cnt, char *license,
-		     u32 kern_version, char *log_buf,
-		     size_t log_buf_sz);
 
-int bpf_map_update_elem(int fd, void *key, void *value,
-			u64 flags);
+/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */
+
+#define BPF_ALU64_REG(OP, DST, SRC)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_OP(OP) | BPF_X,	\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = 0,					\
+		.imm   = 0 })
+
+#define BPF_ALU32_REG(OP, DST, SRC)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU | BPF_OP(OP) | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = 0,					\
+		.imm   = 0 })
+
+/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */
+
+#define BPF_ALU64_IMM(OP, DST, IMM)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_OP(OP) | BPF_K,	\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = IMM })
+
+#define BPF_ALU32_IMM(OP, DST, IMM)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU | BPF_OP(OP) | BPF_K,		\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = IMM })
+
+/* Short form of mov, dst_reg = src_reg */
+
+#define BPF_MOV64_REG(DST, SRC)					\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_MOV | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = 0,					\
+		.imm   = 0 })
+
+#define BPF_MOV32_REG(DST, SRC)					\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU | BPF_MOV | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = 0,					\
+		.imm   = 0 })
+
+/* Short form of mov, dst_reg = imm32 */
+
+#define BPF_MOV64_IMM(DST, IMM)					\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_MOV | BPF_K,		\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = IMM })
+
+#define BPF_MOV32_IMM(DST, IMM)					\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU | BPF_MOV | BPF_K,		\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = IMM })
+
+/* BPF_LD_IMM64 macro encodes single 'load 64-bit immediate' insn */
+#define BPF_LD_IMM64(DST, IMM)					\
+	BPF_LD_IMM64_RAW(DST, 0, IMM)
+
+#define BPF_LD_IMM64_RAW(DST, SRC, IMM)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_LD | BPF_DW | BPF_IMM,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = 0,					\
+		.imm   = (__u32) (IMM) }),			\
+	((struct bpf_insn) {					\
+		.code  = 0, /* zero is reserved opcode */	\
+		.dst_reg = 0,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = ((__u64) (IMM)) >> 32 })
+
+#ifndef BPF_PSEUDO_MAP_FD
+# define BPF_PSEUDO_MAP_FD	1
+#endif
+
+/* pseudo BPF_LD_IMM64 insn used to refer to process-local map_fd */
+#define BPF_LD_MAP_FD(DST, MAP_FD)				\
+	BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
+
+
+/* Direct packet access, R0 = *(uint *) (skb->data + imm32) */
+
+#define BPF_LD_ABS(SIZE, IMM)					\
+	((struct bpf_insn) {					\
+		.code  = BPF_LD | BPF_SIZE(SIZE) | BPF_ABS,	\
+		.dst_reg = 0,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = IMM })
+
+/* Memory load, dst_reg = *(uint *) (src_reg + off16) */
+
+#define BPF_LDX_MEM(SIZE, DST, SRC, OFF)			\
+	((struct bpf_insn) {					\
+		.code  = BPF_LDX | BPF_SIZE(SIZE) | BPF_MEM,	\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = OFF,					\
+		.imm   = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = src_reg */
+
+#define BPF_STX_MEM(SIZE, DST, SRC, OFF)			\
+	((struct bpf_insn) {					\
+		.code  = BPF_STX | BPF_SIZE(SIZE) | BPF_MEM,	\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = OFF,					\
+		.imm   = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = imm32 */
+
+#define BPF_ST_MEM(SIZE, DST, OFF, IMM)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ST | BPF_SIZE(SIZE) | BPF_MEM,	\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = OFF,					\
+		.imm   = IMM })
+
+/* Conditional jumps against registers, if (dst_reg 'op' src_reg) goto pc + off16 */
+
+#define BPF_JMP_REG(OP, DST, SRC, OFF)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_JMP | BPF_OP(OP) | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = OFF,					\
+		.imm   = 0 })
+
+/* Conditional jumps against immediates, if (dst_reg 'op' imm32) goto pc + off16 */
+
+#define BPF_JMP_IMM(OP, DST, IMM, OFF)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_JMP | BPF_OP(OP) | BPF_K,		\
+		.dst_reg = DST,					\
+		.src_reg = 0,					\
+		.off   = OFF,					\
+		.imm   = IMM })
+
+/* Raw code statement block */
+
+#define BPF_RAW_INSN(CODE, DST, SRC, OFF, IMM)			\
+	((struct bpf_insn) {					\
+		.code  = CODE,					\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = OFF,					\
+		.imm   = IMM })
+
+/* Program exit */
+
+#define BPF_EXIT_INSN()						\
+	((struct bpf_insn) {					\
+		.code  = BPF_JMP | BPF_EXIT,			\
+		.dst_reg = 0,					\
+		.src_reg = 0,					\
+		.off   = 0,					\
+		.imm   = 0 })
+
+/* create RAW socket and bind to interface 'name' */
+int open_raw_sock(const char *name);
+
+struct perf_event_attr;
+int perf_event_open(struct perf_event_attr *attr, int pid, int cpu,
+		    int group_fd, unsigned long flags);
 #endif
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index f3c667417a13..19d70e9ee280 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -796,7 +796,8 @@ bpf_object__create_maps(struct bpf_object *obj)
 		*pfd = bpf_create_map(def->type,
 				      def->key_size,
 				      def->value_size,
-				      def->max_entries);
+				      def->max_entries,
+				      0);
 		if (*pfd < 0) {
 			size_t j;
 			int err = *pfd;
-- 
2.9.3

[PATCH net-next 3/3] tools lib bpf: Sync bpf_map_def with tc

[Joe Stringer]

TC uses a slightly different map layout in its ELFs. Update libbpf to
use the same definition so that ELFs may be built using libbpf and
loaded using tc.

Signed-off-by: Joe Stringer <joe@ovn.org>
---
 tools/lib/bpf/libbpf.h | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index dd7a513efb10..ea70c2744f8c 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -181,10 +181,13 @@ bool bpf_program__is_kprobe(struct bpf_program *prog);
  * and will be treated as an error due to -Werror.
  */
 struct bpf_map_def {
-	unsigned int type;
-	unsigned int key_size;
-	unsigned int value_size;
-	unsigned int max_entries;
+	uint32_t type;
+	uint32_t key_size;
+	uint32_t value_size;
+	uint32_t max_entries;
+	uint32_t flags;
+	uint32_t id;
+	uint32_t pinning;
 };
 
 /*
-- 
2.9.3

Re: [PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[David Miller]

From: Joe Stringer <joe@ovn.org>
Date: Mon, 31 Oct 2016 11:39:14 -0700

> Update tools/lib/bpf to provide more functionality and improve interoperation
> with other tools that generate and use eBPF code.
> 
> The kernel uapi headers are a bit newer than the version in the tools/
> directory; synchronize those.
> 
> samples/bpf/libbpf* has a bit more functionality than tools/lib/bpf, so extend
> tools/lib/bpf/bpf* with these functions to bring them into parity.
> 
> tools/lib/bpf cannot read ELFs that tc can read, and vice versa. Update the
> map definition to be the same as in tc so the ELFs may be interchangeable
> (at least for now; I don't have a long-term plan in mind to ensure this always
> works).

Alexei and Daniel, please review.

Re: [PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[David Ahern]

On 10/31/16 12:39 PM, Joe Stringer wrote:
> Update tools/lib/bpf to provide more functionality and improve interoperation
> with other tools that generate and use eBPF code.
> 
> The kernel uapi headers are a bit newer than the version in the tools/
> directory; synchronize those.
> 
> samples/bpf/libbpf* has a bit more functionality than tools/lib/bpf, so extend
> tools/lib/bpf/bpf* with these functions to bring them into parity.
> 
> tools/lib/bpf cannot read ELFs that tc can read, and vice versa. Update the
> map definition to be the same as in tc so the ELFs may be interchangeable
> (at least for now; I don't have a long-term plan in mind to ensure this always
> works).

can samples/bpf be converted to use tools/lib/bpf/libbpf.a? 

Re: [PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[Joe Stringer]

On 1 November 2016 at 13:51, David Ahern <dsa@cumulusnetworks.com> wrote:
> On 10/31/16 12:39 PM, Joe Stringer wrote:
>> Update tools/lib/bpf to provide more functionality and improve interoperation
>> with other tools that generate and use eBPF code.
>>
>> The kernel uapi headers are a bit newer than the version in the tools/
>> directory; synchronize those.
>>
>> samples/bpf/libbpf* has a bit more functionality than tools/lib/bpf, so extend
>> tools/lib/bpf/bpf* with these functions to bring them into parity.
>>
>> tools/lib/bpf cannot read ELFs that tc can read, and vice versa. Update the
>> map definition to be the same as in tc so the ELFs may be interchangeable
>> (at least for now; I don't have a long-term plan in mind to ensure this always
>> works).
>
> can samples/bpf be converted to use tools/lib/bpf/libbpf.a?

I have a few other patches sitting around that need this series,
including an attempt at this.

Re: [PATCH net-next 0/3] tools lib bpf: Synchronize implementations

[Daniel Borkmann]

On 11/01/2016 11:17 PM, Joe Stringer wrote:
> On 1 November 2016 at 13:51, David Ahern <dsa@cumulusnetworks.com> wrote:
>> On 10/31/16 12:39 PM, Joe Stringer wrote:
>>> Update tools/lib/bpf to provide more functionality and improve interoperation
>>> with other tools that generate and use eBPF code.
>>>
>>> The kernel uapi headers are a bit newer than the version in the tools/
>>> directory; synchronize those.
>>>
>>> samples/bpf/libbpf* has a bit more functionality than tools/lib/bpf, so extend
>>> tools/lib/bpf/bpf* with these functions to bring them into parity.
>>>
>>> tools/lib/bpf cannot read ELFs that tc can read, and vice versa. Update the
>>> map definition to be the same as in tc so the ELFs may be interchangeable
>>> (at least for now; I don't have a long-term plan in mind to ensure this always
>>> works).
>>
>> can samples/bpf be converted to use tools/lib/bpf/libbpf.a?
>
> I have a few other patches sitting around that need this series,
> including an attempt at this.

That sounds great, looking forward!

Re: [PATCH net-next 2/3] tools lib bpf: Sync with samples/bpf/libbpf

[Daniel Borkmann]

On 10/31/2016 07:39 PM, Joe Stringer wrote:
> Extend the tools/ version of libbpf to include all of the functionality
> provided in the samples/bpf version.
>
> Signed-off-by: Joe Stringer <joe@ovn.org>
> ---
>   tools/lib/bpf/bpf.c    | 139 +++++++++++++++++++++++++++------
>   tools/lib/bpf/bpf.h    | 208 +++++++++++++++++++++++++++++++++++++++++++++++--
>   tools/lib/bpf/libbpf.c |   3 +-
>   3 files changed, 317 insertions(+), 33 deletions(-)
>
[...]
> +int open_raw_sock(const char *name)
> +{
> +	struct sockaddr_ll sll;
> +	int sock;
> +
> +	sock = socket(PF_PACKET, SOCK_RAW | SOCK_NONBLOCK | SOCK_CLOEXEC,
> +		      htons(ETH_P_ALL));
> +	if (sock < 0) {
> +		printf("cannot create raw socket\n");
> +		return -1;
> +	}
> +
> +	memset(&sll, 0, sizeof(sll));
> +	sll.sll_family = AF_PACKET;
> +	sll.sll_ifindex = if_nametoindex(name);
> +	sll.sll_protocol = htons(ETH_P_ALL);
> +	if (bind(sock, (struct sockaddr *)&sll, sizeof(sll)) < 0) {
> +		printf("bind to %s: %s\n", name, strerror(errno));
> +		close(sock);
> +		return -1;
> +	}
> +
> +	return sock;
> +}
> +
> +int perf_event_open(struct perf_event_attr *attr, int pid, int cpu,
> +		    int group_fd, unsigned long flags)
> +{
> +	return syscall(__NR_perf_event_open, attr, pid, cpu,
> +		       group_fd, flags);
>   }

I'm actually wondering, above bits are not really libbpf related. Maybe
these should go elsewhere into some other misc header file just for the
samples to use?

> diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
> index e8ba54087497..227edb23c022 100644
> --- a/tools/lib/bpf/bpf.h
> +++ b/tools/lib/bpf/bpf.h
> @@ -23,16 +23,208 @@
>
>   #include <linux/bpf.h>
>
> +struct bpf_insn;
> +

Isn't that already defined in the above uapi bpf.h anyway?

>   int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
> -		   int max_entries);
> +		   int max_entries, int map_flags);
> +int bpf_update_elem(int fd, void *key, void *value, unsigned long long flags);
> +int bpf_lookup_elem(int fd, void *key, void *value);
> +int bpf_delete_elem(int fd, void *key);
> +int bpf_get_next_key(int fd, void *key, void *next_key);
> +
> +int bpf_load_program(enum bpf_prog_type prog_type,
> +		     const struct bpf_insn *insns, int insn_len,
> +		     const char *license, int kern_version,
> +		     char *log_buf, size_t log_buf_sz);
> +
> +int bpf_obj_pin(int fd, const char *pathname);
> +int bpf_obj_get(const char *pathname);

Re: [PATCH net-next 3/3] tools lib bpf: Sync bpf_map_def with tc

[Daniel Borkmann]

On 10/31/2016 07:39 PM, Joe Stringer wrote:
> TC uses a slightly different map layout in its ELFs. Update libbpf to
> use the same definition so that ELFs may be built using libbpf and
> loaded using tc.
>
> Signed-off-by: Joe Stringer <joe@ovn.org>
> ---
>   tools/lib/bpf/libbpf.h | 11 +++++++----
>   1 file changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
> index dd7a513efb10..ea70c2744f8c 100644
> --- a/tools/lib/bpf/libbpf.h
> +++ b/tools/lib/bpf/libbpf.h
> @@ -181,10 +181,13 @@ bool bpf_program__is_kprobe(struct bpf_program *prog);
>    * and will be treated as an error due to -Werror.
>    */
>   struct bpf_map_def {
> -	unsigned int type;
> -	unsigned int key_size;
> -	unsigned int value_size;
> -	unsigned int max_entries;
> +	uint32_t type;
> +	uint32_t key_size;
> +	uint32_t value_size;
> +	uint32_t max_entries;
> +	uint32_t flags;
> +	uint32_t id;
> +	uint32_t pinning;
>   };
>
>   /*

I think the problem is that this would break existing obj files that have
been compiled with the current struct bpf_map_def (besides libbpf not having
a use for the last two members right now).

For tc, we have a refactoring of the tc_bpf.c bits that generalizes them so
we can move these bits into iproute2 lib part and add new BPF types really
easily. What I did along with that is to implement a map compat mode, where
it detects the size of struct bpf_elf_map (or however you want to name it)
from the obj file and fixes up the missing members with some reasonable default,
so these programs can still be loaded. Thus, the sample code using the current
struct bpf_map_def will then work with tc as well. (I'll post the iproute2
patch early next week.)

Re: [PATCH net-next 2/3] tools lib bpf: Sync with samples/bpf/libbpf

[Joe Stringer]

On 1 November 2016 at 19:52, Daniel Borkmann <daniel@iogearbox.net> wrote:
> On 10/31/2016 07:39 PM, Joe Stringer wrote:
>>
>> Extend the tools/ version of libbpf to include all of the functionality
>> provided in the samples/bpf version.
>>
>> Signed-off-by: Joe Stringer <joe@ovn.org>
>> ---
>>   tools/lib/bpf/bpf.c    | 139 +++++++++++++++++++++++++++------
>>   tools/lib/bpf/bpf.h    | 208
>> +++++++++++++++++++++++++++++++++++++++++++++++--
>>   tools/lib/bpf/libbpf.c |   3 +-
>>   3 files changed, 317 insertions(+), 33 deletions(-)
>>
> [...]
>
>> +int open_raw_sock(const char *name)
>> +{
>> +       struct sockaddr_ll sll;
>> +       int sock;
>> +
>> +       sock = socket(PF_PACKET, SOCK_RAW | SOCK_NONBLOCK | SOCK_CLOEXEC,
>> +                     htons(ETH_P_ALL));
>> +       if (sock < 0) {
>> +               printf("cannot create raw socket\n");
>> +               return -1;
>> +       }
>> +
>> +       memset(&sll, 0, sizeof(sll));
>> +       sll.sll_family = AF_PACKET;
>> +       sll.sll_ifindex = if_nametoindex(name);
>> +       sll.sll_protocol = htons(ETH_P_ALL);
>> +       if (bind(sock, (struct sockaddr *)&sll, sizeof(sll)) < 0) {
>> +               printf("bind to %s: %s\n", name, strerror(errno));
>> +               close(sock);
>> +               return -1;
>> +       }
>> +
>> +       return sock;
>> +}
>> +
>> +int perf_event_open(struct perf_event_attr *attr, int pid, int cpu,
>> +                   int group_fd, unsigned long flags)
>> +{
>> +       return syscall(__NR_perf_event_open, attr, pid, cpu,
>> +                      group_fd, flags);
>>   }
>
>
> I'm actually wondering, above bits are not really libbpf related. Maybe
> these should go elsewhere into some other misc header file just for the
> samples to use?

True, I started this patch by just copying the whole file over to
tools/lib/bpf so this came with it. I can bring it back to the samples
directory in v2.

>> diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
>> index e8ba54087497..227edb23c022 100644
>> --- a/tools/lib/bpf/bpf.h
>> +++ b/tools/lib/bpf/bpf.h
>> @@ -23,16 +23,208 @@
>>
>>   #include <linux/bpf.h>
>>
>> +struct bpf_insn;
>> +
>
>
> Isn't that already defined in the above uapi bpf.h anyway?

Yup, I'll drop this bit.

Re: [PATCH net-next 3/3] tools lib bpf: Sync bpf_map_def with tc

[Joe Stringer]

On 1 November 2016 at 20:09, Daniel Borkmann <daniel@iogearbox.net> wrote:
> On 10/31/2016 07:39 PM, Joe Stringer wrote:
>>
>> TC uses a slightly different map layout in its ELFs. Update libbpf to
>> use the same definition so that ELFs may be built using libbpf and
>> loaded using tc.
>>
>> Signed-off-by: Joe Stringer <joe@ovn.org>
>> ---
>>   tools/lib/bpf/libbpf.h | 11 +++++++----
>>   1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
>> index dd7a513efb10..ea70c2744f8c 100644
>> --- a/tools/lib/bpf/libbpf.h
>> +++ b/tools/lib/bpf/libbpf.h
>> @@ -181,10 +181,13 @@ bool bpf_program__is_kprobe(struct bpf_program
>> *prog);
>>    * and will be treated as an error due to -Werror.
>>    */
>>   struct bpf_map_def {
>> -       unsigned int type;
>> -       unsigned int key_size;
>> -       unsigned int value_size;
>> -       unsigned int max_entries;
>> +       uint32_t type;
>> +       uint32_t key_size;
>> +       uint32_t value_size;
>> +       uint32_t max_entries;
>> +       uint32_t flags;
>> +       uint32_t id;
>> +       uint32_t pinning;
>>   };
>>
>>   /*
>
>
> I think the problem is that this would break existing obj files that have
> been compiled with the current struct bpf_map_def (besides libbpf not having
> a use for the last two members right now).

Right, this is a problem. I wasn't sure whether libbpf was yet at a
stage where it tries to retain compatibility with binaries compiled
against older kernels.

> For tc, we have a refactoring of the tc_bpf.c bits that generalizes them so
> we can move these bits into iproute2 lib part and add new BPF types really
> easily. What I did along with that is to implement a map compat mode, where
> it detects the size of struct bpf_elf_map (or however you want to name it)
> from the obj file and fixes up the missing members with some reasonable
> default,
> so these programs can still be loaded. Thus, the sample code using the
> current
> struct bpf_map_def will then work with tc as well. (I'll post the iproute2
> patch early next week.)

Are you encoding the number of maps into the start of the maps section
in the ELF then using that to divide out and determine the size?

I look forward to your patches. Maybe if TC is more tolerant of other
map definition sizes then this patch is less relevant.

Re: [PATCH net-next 3/3] tools lib bpf: Sync bpf_map_def with tc

[Daniel Borkmann]

On 11/02/2016 05:09 AM, Joe Stringer wrote:
> On 1 November 2016 at 20:09, Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 10/31/2016 07:39 PM, Joe Stringer wrote:
>>>
>>> TC uses a slightly different map layout in its ELFs. Update libbpf to
>>> use the same definition so that ELFs may be built using libbpf and
>>> loaded using tc.
>>>
>>> Signed-off-by: Joe Stringer <joe@ovn.org>
>>> ---
>>>    tools/lib/bpf/libbpf.h | 11 +++++++----
>>>    1 file changed, 7 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
>>> index dd7a513efb10..ea70c2744f8c 100644
>>> --- a/tools/lib/bpf/libbpf.h
>>> +++ b/tools/lib/bpf/libbpf.h
>>> @@ -181,10 +181,13 @@ bool bpf_program__is_kprobe(struct bpf_program
>>> *prog);
>>>     * and will be treated as an error due to -Werror.
>>>     */
>>>    struct bpf_map_def {
>>> -       unsigned int type;
>>> -       unsigned int key_size;
>>> -       unsigned int value_size;
>>> -       unsigned int max_entries;
>>> +       uint32_t type;
>>> +       uint32_t key_size;
>>> +       uint32_t value_size;
>>> +       uint32_t max_entries;
>>> +       uint32_t flags;
>>> +       uint32_t id;
>>> +       uint32_t pinning;
>>>    };
>>>
>>>    /*
>>
>> I think the problem is that this would break existing obj files that have
>> been compiled with the current struct bpf_map_def (besides libbpf not having
>> a use for the last two members right now).
>
> Right, this is a problem. I wasn't sure whether libbpf was yet at a
> stage where it tries to retain compatibility with binaries compiled
> against older kernels.
>
>> For tc, we have a refactoring of the tc_bpf.c bits that generalizes them so
>> we can move these bits into iproute2 lib part and add new BPF types really
>> easily. What I did along with that is to implement a map compat mode, where
>> it detects the size of struct bpf_elf_map (or however you want to name it)
>> from the obj file and fixes up the missing members with some reasonable
>> default,
>> so these programs can still be loaded. Thus, the sample code using the
>> current
>> struct bpf_map_def will then work with tc as well. (I'll post the iproute2
>> patch early next week.)
>
> Are you encoding the number of maps into the start of the maps section
> in the ELF then using that to divide out and determine the size?

No. It's walking the symbol table and simply counts the number of symbols
that are present in the maps section with correct st_info attribution. It
works because that section name is fixed ABI for all cases and really per
definition only map structs are present there. The minimum attributes which
are allowed to be loaded are type, key_size, value_size and max_entries.

> I look forward to your patches. Maybe if TC is more tolerant of other
> map definition sizes then this patch is less relevant.

Re: [PATCH net-next 3/3] tools lib bpf: Sync bpf_map_def with tc

[Daniel Borkmann]

On 11/02/2016 03:12 PM, Daniel Borkmann wrote:
> On 11/02/2016 05:09 AM, Joe Stringer wrote:
>> On 1 November 2016 at 20:09, Daniel Borkmann <daniel@iogearbox.net> wrote:
>>> On 10/31/2016 07:39 PM, Joe Stringer wrote:
>>>>
>>>> TC uses a slightly different map layout in its ELFs. Update libbpf to
>>>> use the same definition so that ELFs may be built using libbpf and
>>>> loaded using tc.
>>>>
>>>> Signed-off-by: Joe Stringer <joe@ovn.org>
>>>> ---
>>>>    tools/lib/bpf/libbpf.h | 11 +++++++----
>>>>    1 file changed, 7 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
>>>> index dd7a513efb10..ea70c2744f8c 100644
>>>> --- a/tools/lib/bpf/libbpf.h
>>>> +++ b/tools/lib/bpf/libbpf.h
>>>> @@ -181,10 +181,13 @@ bool bpf_program__is_kprobe(struct bpf_program
>>>> *prog);
>>>>     * and will be treated as an error due to -Werror.
>>>>     */
>>>>    struct bpf_map_def {
>>>> -       unsigned int type;
>>>> -       unsigned int key_size;
>>>> -       unsigned int value_size;
>>>> -       unsigned int max_entries;
>>>> +       uint32_t type;
>>>> +       uint32_t key_size;
>>>> +       uint32_t value_size;
>>>> +       uint32_t max_entries;
>>>> +       uint32_t flags;
>>>> +       uint32_t id;
>>>> +       uint32_t pinning;
>>>>    };
>>>>
>>>>    /*
>>>
>>> I think the problem is that this would break existing obj files that have
>>> been compiled with the current struct bpf_map_def (besides libbpf not having
>>> a use for the last two members right now).
>>
>> Right, this is a problem. I wasn't sure whether libbpf was yet at a
>> stage where it tries to retain compatibility with binaries compiled
>> against older kernels.
>>
>>> For tc, we have a refactoring of the tc_bpf.c bits that generalizes them so
>>> we can move these bits into iproute2 lib part and add new BPF types really
>>> easily. What I did along with that is to implement a map compat mode, where
>>> it detects the size of struct bpf_elf_map (or however you want to name it)
>>> from the obj file and fixes up the missing members with some reasonable
>>> default,
>>> so these programs can still be loaded. Thus, the sample code using the
>>> current
>>> struct bpf_map_def will then work with tc as well. (I'll post the iproute2
>>> patch early next week.)
>>
>> Are you encoding the number of maps into the start of the maps section
>> in the ELF then using that to divide out and determine the size?
>
> No. It's walking the symbol table and simply counts the number of symbols
> that are present in the maps section with correct st_info attribution. It
> works because that section name is fixed ABI for all cases and really per
> definition only map structs are present there. The minimum attributes which
> are allowed to be loaded are type, key_size, value_size and max_entries.
>
>> I look forward to your patches. Maybe if TC is more tolerant of other
>> map definition sizes then this patch is less relevant.

Just a thought (not really related to your set though): Perhaps it makes sense
for the bpflib, to also add an option where the user passes a callback to parse
the map section itself if something else than struct bpf_map_def is expected in
the obj, and then have a way to access the meta data along with the fd via lib
api later on. Perhaps that would also include another callback passed to the
lib that would take care to invoke bpf(2) for map creation, which could be useful
if some custom interaction with bpffs is desired.