* [PATCH bpf-next 0/3] Introduce xdp_call.h and the BPF dispatcher @ 2019-11-19 16:07 Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Björn Töpel @ 2019-11-19 16:07 UTC (permalink / raw) To: netdev, ast, daniel Cc: Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko Overview ======== This series introduces the BPF dispatcher and a wrapper, xdp_call.h, which are a mechanism to avoid indirect calls when retpolines are enabled. The BPF dispatcher is a multi-way branch code generator, mainly targeted for XDP programs. When an XDP program is executed via the bpf_prog_run_xdp(), it is invoked via an indirect call. With retpolines enabled, the indirect call has a substantial performance impact. The dispatcher is a mechanism that transform multiple indirect calls to direct calls, and therefore avoids the retpoline. The dispatcher is generated using the BPF JIT, and relies on text poking provided by bpf_arch_text_poke(). The dispatcher hijacks a trampoline function it via the __fentry__ nop of the trampoline. One dispatcher instance currently supports up to 16 dispatch points. This can be extended in the future. An example: A module/driver allocates a dispatcher. The dispatcher is shared for all netdevs. Each unique XDP program has a slot in the dispatcher, registered by a netdev. The netdev then uses the dispatcher to call the correct program with a direct call. The xdp_call.h header wraps a more user-friendly API around the BPF dispatcher. A user adds a trampoline/XDP caller using the DEFINE_XDP_CALL macro, and updates the BPF dispatcher via xdp_call_update(). The actual dispatch is done via xdp_call(). This series relies on Daniel's text poking parts of the "Optimize BPF tail calls for direct jumps" work [0]. [0] https://patchwork.ozlabs.org/cover/1197087/ Generated code, x86-64 ====================== The dispatcher currently has a maximum of 16 entries, where one entry is a unique BPF program. Multiple users of a dispatcher instance using the same BPF program will share that entry. The program/slot lookup is performed by a binary search, O(log n). Let's have a look at the generated code. The trampoline function has the following signature: unsigned int tramp(const void *xdp_ctx, const struct bpf_insn *insnsi, unsigned int (*bpf_func)(const void *, const struct bpf_insn *)) On Intel x86-64 this means that rdx will contain the bpf_func. To, make it easier to read, I've let the BPF programs have the following range: 0xffffffffffffffff (-1) to 0xfffffffffffffff0 (-16). 0xffffffff81c00f10 is the retpoline thunk, in this case __x86_indirect_thunk_rdx. The minimal dispatcher will then look like this: ffffffffc0002000: cmp rdx,0xffffffffffffffff ffffffffc0002007: je 0xffffffffffffffff ; -1 ffffffffc000200d: jmp 0xffffffff81c00f10 The largest dispatcher looks like this: ffffffffc0020000: cmp rdx,0xfffffffffffffff7 ; -9 ffffffffc0020007: jg 0xffffffffc0020130 ffffffffc002000d: cmp rdx,0xfffffffffffffff3 ; -13 ffffffffc0020014: jg 0xffffffffc00200a0 ffffffffc002001a: cmp rdx,0xfffffffffffffff1 ; -15 ffffffffc0020021: jg 0xffffffffc0020060 ffffffffc0020023: cmp rdx,0xfffffffffffffff0 ; -16 ffffffffc002002a: jg 0xffffffffc0020040 ffffffffc002002c: cmp rdx,0xfffffffffffffff0 ; -16 ffffffffc0020033: je 0xfffffffffffffff0 ; -16 ffffffffc0020039: jmp 0xffffffff81c00f10 ffffffffc002003e: xchg ax,ax ffffffffc0020040: cmp rdx,0xfffffffffffffff1 ; -15 ffffffffc0020047: je 0xfffffffffffffff1 ; -15 ffffffffc002004d: jmp 0xffffffff81c00f10 ffffffffc0020052: nop DWORD PTR [rax+rax*1+0x0] ffffffffc002005a: nop WORD PTR [rax+rax*1+0x0] ffffffffc0020060: cmp rdx,0xfffffffffffffff2 ; -14 ffffffffc0020067: jg 0xffffffffc0020080 ffffffffc0020069: cmp rdx,0xfffffffffffffff2 ; -14 ffffffffc0020070: je 0xfffffffffffffff2 ; -14 ffffffffc0020076: jmp 0xffffffff81c00f10 ffffffffc002007b: nop DWORD PTR [rax+rax*1+0x0] ffffffffc0020080: cmp rdx,0xfffffffffffffff3 ; -13 ffffffffc0020087: je 0xfffffffffffffff3 ; -13 ffffffffc002008d: jmp 0xffffffff81c00f10 ffffffffc0020092: nop DWORD PTR [rax+rax*1+0x0] ffffffffc002009a: nop WORD PTR [rax+rax*1+0x0] ffffffffc00200a0: cmp rdx,0xfffffffffffffff5 ; -11 ffffffffc00200a7: jg 0xffffffffc00200f0 ffffffffc00200a9: cmp rdx,0xfffffffffffffff4 ; -12 ffffffffc00200b0: jg 0xffffffffc00200d0 ffffffffc00200b2: cmp rdx,0xfffffffffffffff4 ; -12 ffffffffc00200b9: je 0xfffffffffffffff4 ; -12 ffffffffc00200bf: jmp 0xffffffff81c00f10 ffffffffc00200c4: nop DWORD PTR [rax+rax*1+0x0] ffffffffc00200cc: nop DWORD PTR [rax+0x0] ffffffffc00200d0: cmp rdx,0xfffffffffffffff5 ; -11 ffffffffc00200d7: je 0xfffffffffffffff5 ; -11 ffffffffc00200dd: jmp 0xffffffff81c00f10 ffffffffc00200e2: nop DWORD PTR [rax+rax*1+0x0] ffffffffc00200ea: nop WORD PTR [rax+rax*1+0x0] ffffffffc00200f0: cmp rdx,0xfffffffffffffff6 ; -10 ffffffffc00200f7: jg 0xffffffffc0020110 ffffffffc00200f9: cmp rdx,0xfffffffffffffff6 ; -10 ffffffffc0020100: je 0xfffffffffffffff6 ; -10 ffffffffc0020106: jmp 0xffffffff81c00f10 ffffffffc002010b: nop DWORD PTR [rax+rax*1+0x0] ffffffffc0020110: cmp rdx,0xfffffffffffffff7 ; -9 ffffffffc0020117: je 0xfffffffffffffff7 ; -9 ffffffffc002011d: jmp 0xffffffff81c00f10 ffffffffc0020122: nop DWORD PTR [rax+rax*1+0x0] ffffffffc002012a: nop WORD PTR [rax+rax*1+0x0] ffffffffc0020130: cmp rdx,0xfffffffffffffffb ; -5 ffffffffc0020137: jg 0xffffffffc00201d0 ffffffffc002013d: cmp rdx,0xfffffffffffffff9 ; -7 ffffffffc0020144: jg 0xffffffffc0020190 ffffffffc0020146: cmp rdx,0xfffffffffffffff8 ; -8 ffffffffc002014d: jg 0xffffffffc0020170 ffffffffc002014f: cmp rdx,0xfffffffffffffff8 ; -8 ffffffffc0020156: je 0xfffffffffffffff8 ; -8 ffffffffc002015c: jmp 0xffffffff81c00f10 ffffffffc0020161: nop DWORD PTR [rax+rax*1+0x0] ffffffffc0020169: nop DWORD PTR [rax+0x0] ffffffffc0020170: cmp rdx,0xfffffffffffffff9 ; -7 ffffffffc0020177: je 0xfffffffffffffff9 ; -7 ffffffffc002017d: jmp 0xffffffff81c00f10 ffffffffc0020182: nop DWORD PTR [rax+rax*1+0x0] ffffffffc002018a: nop WORD PTR [rax+rax*1+0x0] ffffffffc0020190: cmp rdx,0xfffffffffffffffa ; -6 ffffffffc0020197: jg 0xffffffffc00201b0 ffffffffc0020199: cmp rdx,0xfffffffffffffffa ; -6 ffffffffc00201a0: je 0xfffffffffffffffa ; -6 ffffffffc00201a6: jmp 0xffffffff81c00f10 ffffffffc00201ab: nop DWORD PTR [rax+rax*1+0x0] ffffffffc00201b0: cmp rdx,0xfffffffffffffffb ; -5 ffffffffc00201b7: je 0xfffffffffffffffb ; -5 ffffffffc00201bd: jmp 0xffffffff81c00f10 ffffffffc00201c2: nop DWORD PTR [rax+rax*1+0x0] ffffffffc00201ca: nop WORD PTR [rax+rax*1+0x0] ffffffffc00201d0: cmp rdx,0xfffffffffffffffd ; -3 ffffffffc00201d7: jg 0xffffffffc0020220 ffffffffc00201d9: cmp rdx,0xfffffffffffffffc ; -4 ffffffffc00201e0: jg 0xffffffffc0020200 ffffffffc00201e2: cmp rdx,0xfffffffffffffffc ; -4 ffffffffc00201e9: je 0xfffffffffffffffc ; -4 ffffffffc00201ef: jmp 0xffffffff81c00f10 ffffffffc00201f4: nop DWORD PTR [rax+rax*1+0x0] ffffffffc00201fc: nop DWORD PTR [rax+0x0] ffffffffc0020200: cmp rdx,0xfffffffffffffffd ; -3 ffffffffc0020207: je 0xfffffffffffffffd ; -3 ffffffffc002020d: jmp 0xffffffff81c00f10 ffffffffc0020212: nop DWORD PTR [rax+rax*1+0x0] ffffffffc002021a: nop WORD PTR [rax+rax*1+0x0] ffffffffc0020220: cmp rdx,0xfffffffffffffffe ; -2 ffffffffc0020227: jg 0xffffffffc0020240 ffffffffc0020229: cmp rdx,0xfffffffffffffffe ; -2 ffffffffc0020230: je 0xfffffffffffffffe ; -2 ffffffffc0020236: jmp 0xffffffff81c00f10 ffffffffc002023b: nop DWORD PTR [rax+rax*1+0x0] ffffffffc0020240: cmp rdx,0xffffffffffffffff ; -1 ffffffffc0020247: je 0xffffffffffffffff ; -1 ffffffffc002024d: jmp 0xffffffff81c00f10 The nops are there to align jump targets to 16B. Performance =========== The tests were performed using the xdp_rxq_info sample program with the following command-line: # xdp_rxq_info --dev eth0 --action XDP_DROP 64B UDP packets at linerate (~59 Mpps) from a packet generator to a 40GbE i40e NIC attached to a 3GHz Intel Skylake machine. 1. Baseline w/o dispatcher: 22.7 Mpps 2. Dispatcher, 1 entry: 31.7 Mpps (+40%) 3. Dispatcher, 2 entries: 32.2 Mpps (+42%) 4. Dispatcher, 3 entries: 31.3 Mpps (+38%) 5. Dispatcher, 4 entries: 32.0 Mpps (+41%) 6. Dispatcher, 5 entries: 31.2 Mpps (+37%) 7. Dispatcher, 6 entries: 31.2 Mpps (+37%) 8. Dispatcher, 7 entries: 30.2 Mpps (+33%) 9. Dispatcher, 8 entries: 31.3 Mpps (+39%) 10. Dispatcher, 9 entries: 30.1 Mpps (+32%) 11. Dispatcher, 10 entries: 31.6 Mpps (+39%) 12. Dispatcher, 11 entries: 31.1 Mpps (+37%) 13. Dispatcher, 12 entries: 30.9 Mpps (+36%) 14. Dispatcher, 13 entries: 30.4 Mpps (+34%) 15. Dispatcher, 14 entries: 31.2 Mpps (+37%) 16. Dispatcher, 15 entries: 30.9 Mpps (+36%) 17. Dispatcher, 16 entries: 32.1 Mpps (+41%) 18. Dispatcher, full: 22.4 Mpps (- 1%) Test 18 is to show-case the cost of walking the a full dispatcher, and then fallback to an indirect call. As the results show, it is hard to see any difference between 1 to 16 entries, other than small variations between runs. Revisions ========= RFC->v1: [1] * Improved error handling (Edward and Andrii) * Explicit cleanup (Andrii) * Use 32B with sext cmp (Alexei) * Align jump targets to 16B (Alexei) * 4 to 16 entries (Toke) * Added stats to xdp_call_run() [1] https://lore.kernel.org/bpf/20191113204737.31623-1-bjorn.topel@gmail.com/ Thanks! Björn Björn Töpel (3): bpf: introduce BPF dispatcher xdp: introduce xdp_call i40e: start using xdp_call.h arch/x86/net/bpf_jit_comp.c | 135 +++++++++++++ drivers/net/ethernet/intel/i40e/i40e_main.c | 5 + drivers/net/ethernet/intel/i40e/i40e_txrx.c | 5 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 5 +- include/linux/xdp_call.h | 66 +++++++ kernel/bpf/Makefile | 1 + kernel/bpf/dispatcher.c | 208 ++++++++++++++++++++ 7 files changed, 423 insertions(+), 2 deletions(-) create mode 100644 include/linux/xdp_call.h create mode 100644 kernel/bpf/dispatcher.c -- 2.20.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH bpf-next 1/3] bpf: introduce BPF dispatcher 2019-11-19 16:07 [PATCH bpf-next 0/3] Introduce xdp_call.h and the BPF dispatcher Björn Töpel @ 2019-11-19 16:07 ` Björn Töpel 2019-11-21 7:27 ` kbuild test robot ` (2 more replies) 2019-11-19 16:07 ` [PATCH bpf-next 2/3] xdp: introduce xdp_call Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 3/3] i40e: start using xdp_call.h Björn Töpel 2 siblings, 3 replies; 15+ messages in thread From: Björn Töpel @ 2019-11-19 16:07 UTC (permalink / raw) To: netdev, ast, daniel Cc: Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko From: Björn Töpel <bjorn.topel@intel.com> The BPF dispatcher is a multiway branch code generator, mainly targeted for XDP programs. When an XDP program is executed via the bpf_prog_run_xdp(), it is invoked via an indirect call. With retpolines enabled, the indirect call has a substantial performance impact. The dispatcher is a mechanism that transform multiple indirect calls to direct calls, and therefore avoids the retpoline. The dispatcher is generated using the BPF JIT, and relies on text poking provided by bpf_arch_text_poke(). The dispatcher hijacks a trampoline function it via the __fentry__ nop of the trampoline. One dispatcher instance currently supports up to 16 dispatch points. This can be extended in the future. An example: A module/driver allocates a dispatcher. The dispatcher is shared for all netdevs. Each unique XDP program has a slot in the dispatcher, registered by a netdev. The netdev then uses the dispatcher to call the correct program with a direct call. Signed-off-by: Björn Töpel <bjorn.topel@intel.com> --- arch/x86/net/bpf_jit_comp.c | 135 +++++++++++++++++++++++ kernel/bpf/Makefile | 1 + kernel/bpf/dispatcher.c | 208 ++++++++++++++++++++++++++++++++++++ 3 files changed, 344 insertions(+) create mode 100644 kernel/bpf/dispatcher.c diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 8704b33ad2e6..579ecb40d32e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -10,10 +10,12 @@ #include <linux/if_vlan.h> #include <linux/bpf.h> #include <linux/memory.h> +#include <linux/sort.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/text-patching.h> +#include <asm/asm-prototypes.h> static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { @@ -1551,6 +1553,139 @@ int arch_prepare_bpf_trampoline(void *image, struct btf_func_model *m, u32 flags return 0; } +#if defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) + +static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) +{ + u8 *prog = *pprog; + int cnt = 0; + s64 offset; + + offset = func - (ip + 2 + 4); + if (!is_simm32(offset)) { + pr_err("Target %p is out of range\n", func); + return -EINVAL; + } + EMIT2_off32(0x0F, jmp_cond + 0x10, offset); + *pprog = prog; + return 0; +} + +static void emit_nops(u8 **pprog, unsigned int len) +{ + unsigned int i, noplen; + u8 *prog = *pprog; + int cnt = 0; + + while (len > 0) { + noplen = len; + + if (noplen > ASM_NOP_MAX) + noplen = ASM_NOP_MAX; + + for (i = 0; i < noplen; i++) + EMIT1(ideal_nops[noplen][i]); + len -= noplen; + } + + *pprog = prog; +} + +static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs) +{ + u8 *jg_reloc, *jg_target, *prog = *pprog; + int pivot, err, jg_bytes = 1, cnt = 0; + s64 jg_offset; + + if (a == b) { + /* Leaf node of recursion, i.e. not a range of indices + * anymore. + */ + EMIT1(add_1mod(0x48, BPF_REG_3)); /* cmp rdx,func */ + if (!is_simm32(progs[a])) + return -1; + EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3), + progs[a]); + err = emit_cond_near_jump(&prog, /* je func */ + (void *)progs[a], prog, + X86_JE); + if (err) + return err; + + err = emit_jump(&prog, /* jmp thunk */ + __x86_indirect_thunk_rdx, prog); + if (err) + return err; + + *pprog = prog; + return 0; + } + + /* Not a leaf node, so we pivot, and recursively descend into + * the lower and upper ranges. + */ + pivot = (b - a) / 2; + EMIT1(add_1mod(0x48, BPF_REG_3)); /* cmp rdx,func */ + if (!is_simm32(progs[a + pivot])) + return -1; + EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3), progs[a + pivot]); + + if (pivot > 2) { /* jg upper_part */ + /* Require near jump. */ + jg_bytes = 4; + EMIT2_off32(0x0F, X86_JG + 0x10, 0); + } else { + EMIT2(X86_JG, 0); + } + jg_reloc = prog; + + err = emit_bpf_dispatcher(&prog, a, a + pivot, /* emit lower_part */ + progs); + if (err) + return err; + + /* Intel 64 and IA-32 ArchitecturesOptimization Reference + * Manual, 3.4.1.5 Code Alignment Assembly/Compiler Coding + * Rule 12. (M impact, H generality) All branch targets should + * be 16-byte aligned. + */ + jg_target = PTR_ALIGN(prog, 16); + if (jg_target != prog) + emit_nops(&prog, jg_target - prog); + jg_offset = prog - jg_reloc; + emit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes); + + err = emit_bpf_dispatcher(&prog, a + pivot + 1, /* emit upper_part */ + b, progs); + if (err) + return err; + + *pprog = prog; + return 0; +} + +static int cmp_ips(const void *a, const void *b) +{ + const s64 *ipa = a; + const s64 *ipb = b; + + if (*ipa > *ipb) + return 1; + if (*ipa < *ipb) + return -1; + return 0; +} + +int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs) +{ + u8 *prog = image; + + sort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL); + return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs); +} + +#endif + struct x64_jit_data { struct bpf_binary_header *header; int *addrs; diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 3f671bf617e8..d4f330351f87 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o obj-$(CONFIG_BPF_JIT) += trampoline.o obj-$(CONFIG_BPF_SYSCALL) += btf.o +obj-$(CONFIG_BPF_JIT) += dispatcher.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o obj-$(CONFIG_BPF_SYSCALL) += cpumap.o diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c new file mode 100644 index 000000000000..59a565107fd1 --- /dev/null +++ b/kernel/bpf/dispatcher.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2019 Intel Corporation. */ + +#ifdef CONFIG_RETPOLINE + +#include <linux/hash.h> +#include <linux/bpf.h> +#include <linux/filter.h> + +/* The BPF dispatcher is a multiway branch code generator. The + * dispatcher is a mechanism to avoid the performance penalty of an + * indirect call when retpolines are enabled. A dispatch client tries + * to register a BPF program into the dispatcher, and if there is + * available room in the dispatcher a direct call to the BPF program + * will be generated. All calls to the BPF programs called via the + * dispatcher will then be a direct call, instead of an indirect. The + * dispatcher hijacks a trampoline function it via the __fentry__ of + * the trampoline. The trampoline function has the following + * signature: + * unsigned int trampoline( + * const void *xdp_ctx, + * const struct bpf_insn *insnsi, + * unsigned int (*bpf_func)(const void *, + * const struct bpf_insn *)); + * + * Direct use of the dispatcher is discouraged, and instead a wrapper + * such as xdp_call.h should be used. + */ + +#define DISPATCHER_HASH_BITS 10 +#define DISPATCHER_TABLE_SIZE (1 << DISPATCHER_HASH_BITS) + +static struct hlist_head dispatcher_table[DISPATCHER_TABLE_SIZE]; + +#define BPF_DISPATCHER_MAX 16 + +struct bpf_dispatcher { + struct hlist_node hlist; + void *func; + struct bpf_prog *progs[BPF_DISPATCHER_MAX]; + int num_progs; + void *image; + u64 selector; +}; + +static DEFINE_MUTEX(dispatcher_mutex); + +struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +{ + struct bpf_dispatcher *d; + struct hlist_head *head; + void *image; + + head = &dispatcher_table[hash_ptr(func, DISPATCHER_HASH_BITS)]; + hlist_for_each_entry(d, head, hlist) { + if (d->func == func) + return d; + } + d = kzalloc(sizeof(*d), GFP_KERNEL); + if (!d) + return NULL; + + image = bpf_jit_alloc_exec(PAGE_SIZE); + if (!image) { + kfree(d); + return NULL; + } + + d->func = func; + INIT_HLIST_NODE(&d->hlist); + hlist_add_head(&d->hlist, head); + + set_vm_flush_reset_perms(image); + set_memory_x((long)image, 1); + d->image = image; + return d; +} + +static void bpf_dispatcher_free(struct bpf_dispatcher *d) +{ + bpf_jit_free_exec(d->image); + hlist_del(&d->hlist); + kfree(d); +} + +static bool bpf_dispatcher_add_prog(struct bpf_dispatcher *d, + struct bpf_prog *prog) +{ + struct bpf_prog **entry = NULL; + int i; + + if (!prog) + return false; + + if (d->num_progs == BPF_DISPATCHER_MAX) + return false; + + for (i = 0; i < BPF_DISPATCHER_MAX; i++) { + if (!entry && !d->progs[i]) + entry = &d->progs[i]; + if (d->progs[i] == prog) + return false; + } + + bpf_prog_inc(prog); + *entry = prog; + d->num_progs++; + return true; +} + +static bool bpf_dispatcher_remove_prog(struct bpf_dispatcher *d, + struct bpf_prog *prog) +{ + int i; + + if (!prog) + return false; + + for (i = 0; i < BPF_DISPATCHER_MAX; i++) { + if (d->progs[i] == prog) { + bpf_prog_put(prog); + d->progs[i] = NULL; + d->num_progs--; + return true; + } + } + return false; +} + +int __weak arch_prepare_bpf_dispatcher(void *image, s64 *funcs, + int num_funcs) +{ + return -ENOTSUPP; +} + +static void bpf_dispatcher_update(struct bpf_dispatcher *d) +{ + void *old_image = d->image + ((d->selector + 1) & 1) * PAGE_SIZE / 2; + void *new_image = d->image + (d->selector & 1) * PAGE_SIZE / 2; + s64 ips[BPF_DISPATCHER_MAX] = {}, *ipsp = &ips[0]; + int i, err; + + if (!d->num_progs) { + bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_NOP, + old_image, NULL); + return; + } + + for (i = 0; i < BPF_DISPATCHER_MAX; i++) { + if (d->progs[i]) + *ipsp++ = (s64)d->progs[i]->bpf_func; + } + err = arch_prepare_bpf_dispatcher(new_image, &ips[0], d->num_progs); + if (err) + return; + + if (d->selector) { + /* progs already running at this address */ + err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_JUMP, + old_image, new_image); + } else { + /* first time registering */ + err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JUMP, + NULL, new_image); + } + if (err) + return; + d->selector++; +} + +void bpf_dispatcher_change_prog(void *func, struct bpf_prog *from, + struct bpf_prog *to) +{ + struct bpf_dispatcher *d; + bool changed = false; + + if (from == to) + return; + + mutex_lock(&dispatcher_mutex); + d = bpf_dispatcher_lookup(func); + if (!d) + goto out; + + changed |= bpf_dispatcher_remove_prog(d, from); + changed |= bpf_dispatcher_add_prog(d, to); + + if (!changed) + goto out; + + bpf_dispatcher_update(d); + if (!d->num_progs) + bpf_dispatcher_free(d); +out: + mutex_unlock(&dispatcher_mutex); +} + +static int __init init_dispatchers(void) +{ + int i; + + for (i = 0; i < DISPATCHER_TABLE_SIZE; i++) + INIT_HLIST_HEAD(&dispatcher_table[i]); + return 0; +} +late_initcall(init_dispatchers); + +#endif -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH bpf-next 1/3] bpf: introduce BPF dispatcher 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel @ 2019-11-21 7:27 ` kbuild test robot 2019-11-23 0:25 ` kbuild test robot 2019-11-23 14:03 ` kbuild test robot 2 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-21 7:27 UTC (permalink / raw) To: Björn Töpel Cc: kbuild-all, netdev, ast, daniel, Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko [-- Attachment #1: Type: text/plain, Size: 3604 bytes --] Hi "Björn, I love your patch! Yet something to improve: [auto build test ERROR on bpf-next/master] [also build test ERROR on next-20191120] [cannot apply to v5.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Bj-rn-T-pel/Introduce-xdp_call-h-and-the-BPF-dispatcher/20191120-074435 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: i386-randconfig-g003-20191121 (attached as .config) compiler: gcc-7 (Debian 7.4.0-14) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=i386 If you fix the issue, kindly add following tag Reported-by: kbuild test robot <lkp@intel.com> All error/warnings (new ones prefixed by >>): kernel/bpf/dispatcher.c: In function 'bpf_dispatcher_update': >> kernel/bpf/dispatcher.c:144:31: error: 'BPF_MOD_JUMP_TO_NOP' undeclared (first use in this function); did you mean 'BPF_MOD_CALL_TO_NOP'? bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_NOP, ^~~~~~~~~~~~~~~~~~~ BPF_MOD_CALL_TO_NOP kernel/bpf/dispatcher.c:144:31: note: each undeclared identifier is reported only once for each function it appears in >> kernel/bpf/dispatcher.c:151:14: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] *ipsp++ = (s64)d->progs[i]->bpf_func; ^ >> kernel/bpf/dispatcher.c:159:37: error: 'BPF_MOD_JUMP_TO_JUMP' undeclared (first use in this function); did you mean 'BPF_MOD_JUMP_TO_NOP'? err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_JUMP, ^~~~~~~~~~~~~~~~~~~~ BPF_MOD_JUMP_TO_NOP >> kernel/bpf/dispatcher.c:163:37: error: 'BPF_MOD_NOP_TO_JUMP' undeclared (first use in this function); did you mean 'BPF_MOD_JUMP_TO_JUMP'? err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JUMP, ^~~~~~~~~~~~~~~~~~~ BPF_MOD_JUMP_TO_JUMP vim +144 kernel/bpf/dispatcher.c 135 136 static void bpf_dispatcher_update(struct bpf_dispatcher *d) 137 { 138 void *old_image = d->image + ((d->selector + 1) & 1) * PAGE_SIZE / 2; 139 void *new_image = d->image + (d->selector & 1) * PAGE_SIZE / 2; 140 s64 ips[BPF_DISPATCHER_MAX] = {}, *ipsp = &ips[0]; 141 int i, err; 142 143 if (!d->num_progs) { > 144 bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_NOP, 145 old_image, NULL); 146 return; 147 } 148 149 for (i = 0; i < BPF_DISPATCHER_MAX; i++) { 150 if (d->progs[i]) > 151 *ipsp++ = (s64)d->progs[i]->bpf_func; 152 } 153 err = arch_prepare_bpf_dispatcher(new_image, &ips[0], d->num_progs); 154 if (err) 155 return; 156 157 if (d->selector) { 158 /* progs already running at this address */ > 159 err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_JUMP, 160 old_image, new_image); 161 } else { 162 /* first time registering */ > 163 err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JUMP, 164 NULL, new_image); 165 } 166 if (err) 167 return; 168 d->selector++; 169 } 170 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org Intel Corporation [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 35785 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH bpf-next 1/3] bpf: introduce BPF dispatcher @ 2019-11-21 7:27 ` kbuild test robot 0 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-21 7:27 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 3691 bytes --] Hi "Björn, I love your patch! Yet something to improve: [auto build test ERROR on bpf-next/master] [also build test ERROR on next-20191120] [cannot apply to v5.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Bj-rn-T-pel/Introduce-xdp_call-h-and-the-BPF-dispatcher/20191120-074435 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: i386-randconfig-g003-20191121 (attached as .config) compiler: gcc-7 (Debian 7.4.0-14) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=i386 If you fix the issue, kindly add following tag Reported-by: kbuild test robot <lkp@intel.com> All error/warnings (new ones prefixed by >>): kernel/bpf/dispatcher.c: In function 'bpf_dispatcher_update': >> kernel/bpf/dispatcher.c:144:31: error: 'BPF_MOD_JUMP_TO_NOP' undeclared (first use in this function); did you mean 'BPF_MOD_CALL_TO_NOP'? bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_NOP, ^~~~~~~~~~~~~~~~~~~ BPF_MOD_CALL_TO_NOP kernel/bpf/dispatcher.c:144:31: note: each undeclared identifier is reported only once for each function it appears in >> kernel/bpf/dispatcher.c:151:14: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] *ipsp++ = (s64)d->progs[i]->bpf_func; ^ >> kernel/bpf/dispatcher.c:159:37: error: 'BPF_MOD_JUMP_TO_JUMP' undeclared (first use in this function); did you mean 'BPF_MOD_JUMP_TO_NOP'? err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_JUMP, ^~~~~~~~~~~~~~~~~~~~ BPF_MOD_JUMP_TO_NOP >> kernel/bpf/dispatcher.c:163:37: error: 'BPF_MOD_NOP_TO_JUMP' undeclared (first use in this function); did you mean 'BPF_MOD_JUMP_TO_JUMP'? err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JUMP, ^~~~~~~~~~~~~~~~~~~ BPF_MOD_JUMP_TO_JUMP vim +144 kernel/bpf/dispatcher.c 135 136 static void bpf_dispatcher_update(struct bpf_dispatcher *d) 137 { 138 void *old_image = d->image + ((d->selector + 1) & 1) * PAGE_SIZE / 2; 139 void *new_image = d->image + (d->selector & 1) * PAGE_SIZE / 2; 140 s64 ips[BPF_DISPATCHER_MAX] = {}, *ipsp = &ips[0]; 141 int i, err; 142 143 if (!d->num_progs) { > 144 bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_NOP, 145 old_image, NULL); 146 return; 147 } 148 149 for (i = 0; i < BPF_DISPATCHER_MAX; i++) { 150 if (d->progs[i]) > 151 *ipsp++ = (s64)d->progs[i]->bpf_func; 152 } 153 err = arch_prepare_bpf_dispatcher(new_image, &ips[0], d->num_progs); 154 if (err) 155 return; 156 157 if (d->selector) { 158 /* progs already running at this address */ > 159 err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP_TO_JUMP, 160 old_image, new_image); 161 } else { 162 /* first time registering */ > 163 err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JUMP, 164 NULL, new_image); 165 } 166 if (err) 167 return; 168 d->selector++; 169 } 170 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation [-- Attachment #2: config.gz --] [-- Type: application/gzip, Size: 35785 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH bpf-next 1/3] bpf: introduce BPF dispatcher 2019-11-21 7:27 ` kbuild test robot (?) @ 2019-11-21 9:23 ` Björn Töpel -1 siblings, 0 replies; 15+ messages in thread From: Björn Töpel @ 2019-11-21 9:23 UTC (permalink / raw) To: kbuild test robot Cc: kbuild-all, Netdev, Alexei Starovoitov, Daniel Borkmann, Björn Töpel, bpf, Magnus Karlsson, Karlsson, Magnus, Jonathan Lemon, Edward Cree, Toke Høiland-Jørgensen, Andrii Nakryiko On Thu, 21 Nov 2019 at 08:27, kbuild test robot <lkp@intel.com> wrote: > [...] > >> kernel/bpf/dispatcher.c:151:14: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] > *ipsp++ = (s64)d->progs[i]->bpf_func; > ^ I'll fix the cast warning on i386 in the v2. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC PATCH] bpf: bpf_dispatcher_lookup() can be static 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel @ 2019-11-23 0:25 ` kbuild test robot 2019-11-23 0:25 ` kbuild test robot 2019-11-23 14:03 ` kbuild test robot 2 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 0:25 UTC (permalink / raw) To: Björn Töpel Cc: kbuild-all, netdev, ast, daniel, Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko Fixes: 316d60dee82c ("bpf: introduce BPF dispatcher") Signed-off-by: kbuild test robot <lkp@intel.com> --- dispatcher.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c index 59a565107fd1d..30c964f94a173 100644 --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -45,7 +45,7 @@ struct bpf_dispatcher { static DEFINE_MUTEX(dispatcher_mutex); -struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +static struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) { struct bpf_dispatcher *d; struct hlist_head *head; ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC PATCH] bpf: bpf_dispatcher_lookup() can be static @ 2019-11-23 0:25 ` kbuild test robot 0 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 0:25 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 630 bytes --] Fixes: 316d60dee82c ("bpf: introduce BPF dispatcher") Signed-off-by: kbuild test robot <lkp@intel.com> --- dispatcher.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c index 59a565107fd1d..30c964f94a173 100644 --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -45,7 +45,7 @@ struct bpf_dispatcher { static DEFINE_MUTEX(dispatcher_mutex); -struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +static struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) { struct bpf_dispatcher *d; struct hlist_head *head; ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC PATCH] bpf: bpf_dispatcher_lookup() can be static 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel @ 2019-11-23 14:03 ` kbuild test robot 2019-11-23 0:25 ` kbuild test robot 2019-11-23 14:03 ` kbuild test robot 2 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 14:03 UTC (permalink / raw) To: Björn Töpel Cc: kbuild-all, netdev, ast, daniel, Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko Fixes: 316d60dee82c ("bpf: introduce BPF dispatcher") Signed-off-by: kbuild test robot <lkp@intel.com> --- dispatcher.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c index 59a565107fd1d..30c964f94a173 100644 --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -45,7 +45,7 @@ struct bpf_dispatcher { static DEFINE_MUTEX(dispatcher_mutex); -struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +static struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) { struct bpf_dispatcher *d; struct hlist_head *head; ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC PATCH] bpf: bpf_dispatcher_lookup() can be static @ 2019-11-23 14:03 ` kbuild test robot 0 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 14:03 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 630 bytes --] Fixes: 316d60dee82c ("bpf: introduce BPF dispatcher") Signed-off-by: kbuild test robot <lkp@intel.com> --- dispatcher.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c index 59a565107fd1d..30c964f94a173 100644 --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -45,7 +45,7 @@ struct bpf_dispatcher { static DEFINE_MUTEX(dispatcher_mutex); -struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +static struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) { struct bpf_dispatcher *d; struct hlist_head *head; ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH bpf-next 2/3] xdp: introduce xdp_call 2019-11-19 16:07 [PATCH bpf-next 0/3] Introduce xdp_call.h and the BPF dispatcher Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel @ 2019-11-19 16:07 ` Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 3/3] i40e: start using xdp_call.h Björn Töpel 2 siblings, 0 replies; 15+ messages in thread From: Björn Töpel @ 2019-11-19 16:07 UTC (permalink / raw) To: netdev, ast, daniel Cc: Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko From: Björn Töpel <bjorn.topel@intel.com> The xdp_call.h header wraps a more user-friendly API around the BPF dispatcher. A user adds a trampoline/XDP caller using the DEFINE_XDP_CALL macro, and updates the BPF dispatcher via xdp_call_update(). The actual dispatch is done via xdp_call(). The next patch will show-case how the i40e driver uses xdp_call. Signed-off-by: Björn Töpel <bjorn.topel@intel.com> --- include/linux/xdp_call.h | 66 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 include/linux/xdp_call.h diff --git a/include/linux/xdp_call.h b/include/linux/xdp_call.h new file mode 100644 index 000000000000..3591a834d4dd --- /dev/null +++ b/include/linux/xdp_call.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2019 Intel Corporation. */ +#ifndef _LINUX_XDP_CALL_H +#define _LINUX_XDP_CALL_H + +#include <linux/filter.h> + +#if defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) + +void bpf_dispatcher_change_prog(void *func, struct bpf_prog *from, + struct bpf_prog *to); + +#define XDP_CALL_TRAMP(name) ____xdp_call_##name##_tramp + +#define DEFINE_XDP_CALL(name) \ + unsigned int XDP_CALL_TRAMP(name)( \ + const void *xdp_ctx, \ + const struct bpf_insn *insnsi, \ + unsigned int (*bpf_func)(const void *, \ + const struct bpf_insn *)) \ + { \ + return bpf_func(xdp_ctx, insnsi); \ + } + +#define DECLARE_XDP_CALL(name) \ + unsigned int XDP_CALL_TRAMP(name)( \ + const void *xdp_ctx, \ + const struct bpf_insn *insnsi, \ + unsigned int (*bpf_func)(const void *, \ + const struct bpf_insn *)) + +#define xdp_call_run(name, prog, ctx) ({ \ + u32 ret; \ + cant_sleep(); \ + if (static_branch_unlikely(&bpf_stats_enabled_key)) { \ + struct bpf_prog_stats *stats; \ + u64 start = sched_clock(); \ + ret = XDP_CALL_TRAMP(name)(ctx, \ + (prog)->insnsi, \ + (prog)->bpf_func); \ + stats = this_cpu_ptr((prog)->aux->stats); \ + u64_stats_update_begin(&stats->syncp); \ + stats->cnt++; \ + stats->nsecs += sched_clock() - start; \ + u64_stats_update_end(&stats->syncp); \ + } else { \ + ret = XDP_CALL_TRAMP(name)(ctx, \ + (prog)->insnsi, \ + (prog)->bpf_func); \ + } \ + ret; }) + +#define xdp_call_update(name, from_xdp_prog, to_xdp_prog) \ + bpf_dispatcher_change_prog(&XDP_CALL_TRAMP(name), \ + from_xdp_prog, \ + to_xdp_prog) + +#else /* !defined(CONFIG_BPF_JIT) || !defined(CONFIG_RETPOLINE) */ + +#define DEFINE_XDP_CALL(name) +#define DECLARE_XDP_CALL(name) +#define xdp_call_run(name, xdp_prog, xdp) bpf_prog_run_xdp(xdp_prog, xdp) +#define xdp_call_update(name, from_xdp_prog, to_xdp_prog) + +#endif /* defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) */ +#endif /* _LINUX_XDP_CALL_H */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH bpf-next 3/3] i40e: start using xdp_call.h 2019-11-19 16:07 [PATCH bpf-next 0/3] Introduce xdp_call.h and the BPF dispatcher Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 2/3] xdp: introduce xdp_call Björn Töpel @ 2019-11-19 16:07 ` Björn Töpel 2019-11-23 15:52 ` kbuild test robot 2019-11-23 15:52 ` kbuild test robot 2 siblings, 2 replies; 15+ messages in thread From: Björn Töpel @ 2019-11-19 16:07 UTC (permalink / raw) To: netdev, ast, daniel Cc: Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko From: Björn Töpel <bjorn.topel@intel.com> This commit starts using xdp_call.h and the BPF dispatcher to avoid the retpoline overhead. Signed-off-by: Björn Töpel <bjorn.topel@intel.com> --- drivers/net/ethernet/intel/i40e/i40e_main.c | 5 +++++ drivers/net/ethernet/intel/i40e/i40e_txrx.c | 5 ++++- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 5 ++++- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index b3d7edbb1389..59b530e4198f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -5,6 +5,7 @@ #include <linux/of_net.h> #include <linux/pci.h> #include <linux/bpf.h> +#include <linux/xdp_call.h> /* Local includes */ #include "i40e.h" @@ -12517,6 +12518,8 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb, return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK); } +DEFINE_XDP_CALL(i40e_xdp_call); + /** * i40e_xdp_setup - add/remove an XDP program * @vsi: VSI to changed @@ -12552,6 +12555,8 @@ static int i40e_xdp_setup(struct i40e_vsi *vsi, for (i = 0; i < vsi->num_queue_pairs; i++) WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog); + xdp_call_update(i40e_xdp_call, old_prog, prog); + if (old_prog) bpf_prog_put(old_prog); diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index b8496037ef7f..34d7b15897a1 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -3,6 +3,7 @@ #include <linux/prefetch.h> #include <linux/bpf_trace.h> +#include <linux/xdp_call.h> #include <net/xdp.h> #include "i40e.h" #include "i40e_trace.h" @@ -2188,6 +2189,8 @@ int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring) return i40e_xmit_xdp_ring(xdpf, xdp_ring); } +DECLARE_XDP_CALL(i40e_xdp_call); + /** * i40e_run_xdp - run an XDP program * @rx_ring: Rx ring being processed @@ -2209,7 +2212,7 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, prefetchw(xdp->data_hard_start); /* xdp_frame write */ - act = bpf_prog_run_xdp(xdp_prog, xdp); + act = xdp_call_run(i40e_xdp_call, xdp_prog, xdp); switch (act) { case XDP_PASS: break; diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index a05dfecdd9b4..c623eeaeb625 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -2,6 +2,7 @@ /* Copyright(c) 2018 Intel Corporation. */ #include <linux/bpf_trace.h> +#include <linux/xdp_call.h> #include <net/xdp_sock.h> #include <net/xdp.h> @@ -179,6 +180,8 @@ int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem, i40e_xsk_umem_disable(vsi, qid); } +DECLARE_XDP_CALL(i40e_xdp_call); + /** * i40e_run_xdp_zc - Executes an XDP program on an xdp_buff * @rx_ring: Rx ring @@ -202,7 +205,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) * this path is enabled by setting an XDP program. */ xdp_prog = READ_ONCE(rx_ring->xdp_prog); - act = bpf_prog_run_xdp(xdp_prog, xdp); + act = xdp_call_run(i40e_xdp_call, xdp_prog, xdp); offset = xdp->data - xdp->data_hard_start; xdp->handle = xsk_umem_adjust_offset(umem, xdp->handle, offset); -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH bpf-next 3/3] i40e: start using xdp_call.h 2019-11-19 16:07 ` [PATCH bpf-next 3/3] i40e: start using xdp_call.h Björn Töpel @ 2019-11-23 15:52 ` kbuild test robot 2019-11-23 15:52 ` kbuild test robot 1 sibling, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 15:52 UTC (permalink / raw) To: Björn Töpel Cc: kbuild-all, netdev, ast, daniel, Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko Hi "Björn, I love your patch! Perhaps something to improve: [auto build test WARNING on bpf-next/master] [also build test WARNING on next-20191122] [cannot apply to v5.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Bj-rn-T-pel/Introduce-xdp_call-h-and-the-BPF-dispatcher/20191120-074435 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master reproduce: # apt-get install sparse # sparse version: v0.6.1-36-g9305d48-dirty make ARCH=x86_64 allmodconfig make C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' If you fix the issue, kindly add following tag Reported-by: kbuild test robot <lkp@intel.com> sparse warnings: (new ones prefixed by >>) drivers/net/ethernet/intel/i40e/i40e_main.c:5904:32: sparse: sparse: cast from restricted __le16 drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned short [usertype] stat_counter_idx @@ got resunsigned short [usertype] stat_counter_idx @@ drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: expected unsigned short [usertype] stat_counter_idx drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: got restricted __le16 [usertype] drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned int [assigned] [usertype] ipa @@ got ed int [assigned] [usertype] ipa @@ drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: expected unsigned int [assigned] [usertype] ipa drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: got restricted __le32 [usertype] drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned int [assigned] [usertype] ipa @@ got ed int [assigned] [usertype] ipa @@ drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: expected unsigned int [assigned] [usertype] ipa drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: got restricted __le32 [usertype] >> drivers/net/ethernet/intel/i40e/i40e_main.c:12521:1: sparse: sparse: symbol '____xdp_call_i40e_xdp_call_tramp' was not declared. Should it be static? arch/x86/include/asm/bitops.h:77:37: sparse: sparse: cast truncates bits from constant value (ffffff7f becomes 7f) arch/x86/include/asm/bitops.h:77:37: sparse: sparse: cast truncates bits from constant value (ffffff7f becomes 7f) Please review and possibly fold the followup patch. --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org Intel Corporation ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH bpf-next 3/3] i40e: start using xdp_call.h @ 2019-11-23 15:52 ` kbuild test robot 0 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 15:52 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 3040 bytes --] Hi "Björn, I love your patch! Perhaps something to improve: [auto build test WARNING on bpf-next/master] [also build test WARNING on next-20191122] [cannot apply to v5.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Bj-rn-T-pel/Introduce-xdp_call-h-and-the-BPF-dispatcher/20191120-074435 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master reproduce: # apt-get install sparse # sparse version: v0.6.1-36-g9305d48-dirty make ARCH=x86_64 allmodconfig make C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' If you fix the issue, kindly add following tag Reported-by: kbuild test robot <lkp@intel.com> sparse warnings: (new ones prefixed by >>) drivers/net/ethernet/intel/i40e/i40e_main.c:5904:32: sparse: sparse: cast from restricted __le16 drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned short [usertype] stat_counter_idx @@ got resunsigned short [usertype] stat_counter_idx @@ drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: expected unsigned short [usertype] stat_counter_idx drivers/net/ethernet/intel/i40e/i40e_main.c:5904:30: sparse: got restricted __le16 [usertype] drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned int [assigned] [usertype] ipa @@ got ed int [assigned] [usertype] ipa @@ drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: expected unsigned int [assigned] [usertype] ipa drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: got restricted __le32 [usertype] drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned int [assigned] [usertype] ipa @@ got ed int [assigned] [usertype] ipa @@ drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: expected unsigned int [assigned] [usertype] ipa drivers/net/ethernet/intel/i40e/i40e_main.c:7564:29: sparse: got restricted __le32 [usertype] >> drivers/net/ethernet/intel/i40e/i40e_main.c:12521:1: sparse: sparse: symbol '____xdp_call_i40e_xdp_call_tramp' was not declared. Should it be static? arch/x86/include/asm/bitops.h:77:37: sparse: sparse: cast truncates bits from constant value (ffffff7f becomes 7f) arch/x86/include/asm/bitops.h:77:37: sparse: sparse: cast truncates bits from constant value (ffffff7f becomes 7f) Please review and possibly fold the followup patch. --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation ^ permalink raw reply [flat|nested] 15+ messages in thread
* [RFC PATCH] i40e: ____xdp_call_i40e_xdp_call_tramp can be static 2019-11-19 16:07 ` [PATCH bpf-next 3/3] i40e: start using xdp_call.h Björn Töpel @ 2019-11-23 15:52 ` kbuild test robot 2019-11-23 15:52 ` kbuild test robot 1 sibling, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 15:52 UTC (permalink / raw) To: Björn Töpel Cc: kbuild-all, netdev, ast, daniel, Björn Töpel, bpf, magnus.karlsson, magnus.karlsson, jonathan.lemon, ecree, thoiland, andrii.nakryiko Fixes: cfbcb56c2871 ("i40e: start using xdp_call.h") Signed-off-by: kbuild test robot <lkp@intel.com> --- i40e_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 59b530e4198f2..6771d77871152 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -12518,7 +12518,7 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb, return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK); } -DEFINE_XDP_CALL(i40e_xdp_call); +static DEFINE_XDP_CALL(i40e_xdp_call); /** * i40e_xdp_setup - add/remove an XDP program ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [RFC PATCH] i40e: ____xdp_call_i40e_xdp_call_tramp can be static @ 2019-11-23 15:52 ` kbuild test robot 0 siblings, 0 replies; 15+ messages in thread From: kbuild test robot @ 2019-11-23 15:52 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 724 bytes --] Fixes: cfbcb56c2871 ("i40e: start using xdp_call.h") Signed-off-by: kbuild test robot <lkp@intel.com> --- i40e_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 59b530e4198f2..6771d77871152 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -12518,7 +12518,7 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb, return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK); } -DEFINE_XDP_CALL(i40e_xdp_call); +static DEFINE_XDP_CALL(i40e_xdp_call); /** * i40e_xdp_setup - add/remove an XDP program ^ permalink raw reply related [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-11-23 15:53 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-19 16:07 [PATCH bpf-next 0/3] Introduce xdp_call.h and the BPF dispatcher Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 1/3] bpf: introduce " Björn Töpel 2019-11-21 7:27 ` kbuild test robot 2019-11-21 7:27 ` kbuild test robot 2019-11-21 9:23 ` Björn Töpel 2019-11-23 0:25 ` [RFC PATCH] bpf: bpf_dispatcher_lookup() can be static kbuild test robot 2019-11-23 0:25 ` kbuild test robot 2019-11-23 14:03 ` kbuild test robot 2019-11-23 14:03 ` kbuild test robot 2019-11-19 16:07 ` [PATCH bpf-next 2/3] xdp: introduce xdp_call Björn Töpel 2019-11-19 16:07 ` [PATCH bpf-next 3/3] i40e: start using xdp_call.h Björn Töpel 2019-11-23 15:52 ` kbuild test robot 2019-11-23 15:52 ` kbuild test robot 2019-11-23 15:52 ` [RFC PATCH] i40e: ____xdp_call_i40e_xdp_call_tramp can be static kbuild test robot 2019-11-23 15:52 ` kbuild test robot
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.