From: Song Liu <song@kernel.org>
To: <bpf@vger.kernel.org>, <netdev@vger.kernel.org>, <linux-mm@kvack.org>
Cc: <ast@kernel.org>, <daniel@iogearbox.net>, <andrii@kernel.org>,
<kernel-team@fb.com>, <akpm@linux-foundation.org>,
<rick.p.edgecombe@intel.com>, <hch@infradead.org>,
<imbrenda@linux.ibm.com>, Song Liu <song@kernel.org>
Subject: [PATCH bpf 2/2] bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack
Date: Fri, 8 Apr 2022 15:34:43 -0700 [thread overview]
Message-ID: <20220408223443.3303509-3-song@kernel.org> (raw)
In-Reply-To: <20220408223443.3303509-1-song@kernel.org>
Use __vmalloc_node_range with VM_ALLOW_HUGE_VMAP for bpf_prog_pack so that
BPF programs sit on PMD_SIZE pages. This benefits system performance by
reducing iTLB miss rate.
Signed-off-by: Song Liu <song@kernel.org>
---
kernel/bpf/core.c | 24 ++++++++++++++++++++----
1 file changed, 20 insertions(+), 4 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 13e9dbeeedf3..04214f4e64f1 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -851,13 +851,28 @@ static LIST_HEAD(pack_list);
#define BPF_HPAGE_MASK PAGE_MASK
#endif
+static void *bpf_prog_pack_vmalloc(unsigned long size)
+{
+#if defined(MODULES_VADDR)
+ unsigned long start = MODULES_VADDR;
+ unsigned long end = MODULES_END;
+#else
+ unsigned long start = VMALLOC_START;
+ unsigned long end = VMALLOC_END;
+#endif
+
+ return __vmalloc_node_range(size, PAGE_SIZE, start, end, GFP_KERNEL, PAGE_KERNEL,
+ VM_DEFER_KMEMLEAK | VM_ALLOW_HUGE_VMAP,
+ NUMA_NO_NODE, __builtin_return_address(0));
+}
+
static size_t select_bpf_prog_pack_size(void)
{
size_t size;
void *ptr;
size = BPF_HPAGE_SIZE * num_online_nodes();
- ptr = module_alloc(size);
+ ptr = bpf_prog_pack_vmalloc(size);
/* Test whether we can get huge pages. If not just use PAGE_SIZE
* packs.
@@ -881,7 +896,7 @@ static struct bpf_prog_pack *alloc_new_pack(void)
GFP_KERNEL);
if (!pack)
return NULL;
- pack->ptr = module_alloc(bpf_prog_pack_size);
+ pack->ptr = bpf_prog_pack_vmalloc(bpf_prog_pack_size);
if (!pack->ptr) {
kfree(pack);
return NULL;
@@ -889,7 +904,6 @@ static struct bpf_prog_pack *alloc_new_pack(void)
bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE);
list_add_tail(&pack->list, &pack_list);
- set_vm_flush_reset_perms(pack->ptr);
set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
return pack;
@@ -970,7 +984,9 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr)
if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0,
bpf_prog_chunk_count(), 0) == 0) {
list_del(&pack->list);
- module_memfree(pack->ptr);
+ set_memory_nx((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
+ set_memory_rw((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_SIZE);
+ vfree(pack->ptr);
kfree(pack);
}
out:
--
2.30.2
next prev parent reply other threads:[~2022-04-08 22:40 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-08 22:34 [PATCH bpf 0/2] vmalloc: bpf: introduce VM_ALLOW_HUGE_VMAP Song Liu
2022-04-08 22:34 ` [PATCH bpf 1/2] vmalloc: replace VM_NO_HUGE_VMAP with VM_ALLOW_HUGE_VMAP Song Liu
2022-04-09 5:28 ` Christoph Hellwig
2022-04-10 1:25 ` Song Liu
2022-04-08 22:34 ` Song Liu [this message]
2022-04-09 5:29 ` [PATCH bpf 2/2] bpf: use vmalloc with VM_ALLOW_HUGE_VMAP for bpf_prog_pack Christoph Hellwig
2022-04-10 1:34 ` Song Liu
2022-04-11 6:56 ` Christoph Hellwig
2022-04-11 22:18 ` Song Liu
2022-04-09 11:43 ` [PATCH bpf 0/2] vmalloc: bpf: introduce VM_ALLOW_HUGE_VMAP Thorsten Leemhuis
2022-04-10 1:36 ` Song Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220408223443.3303509-3-song@kernel.org \
--to=song@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=hch@infradead.org \
--cc=imbrenda@linux.ibm.com \
--cc=kernel-team@fb.com \
--cc=linux-mm@kvack.org \
--cc=netdev@vger.kernel.org \
--cc=rick.p.edgecombe@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).