From: Jisheng Zhang <jszhang3@mail.ustc.edu.cn>
To: Alex Ghiti <alex@ghiti.fr>
Cc: "Andreas Schwab" <schwab@linux-m68k.org>,
"Paul Walmsley" <paul.walmsley@sifive.com>,
"Palmer Dabbelt" <palmer@dabbelt.com>,
"Albert Ou" <aou@eecs.berkeley.edu>,
"Andrey Ryabinin" <ryabinin.a.a@gmail.com>,
"Alexander Potapenko" <glider@google.com>,
"Andrey Konovalov" <andreyknvl@gmail.com>,
"Dmitry Vyukov" <dvyukov@google.com>,
"Björn Töpel" <bjorn@kernel.org>,
"Alexei Starovoitov" <ast@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Martin KaFai Lau" <kafai@fb.com>,
"Song Liu" <songliubraving@fb.com>, "Yonghong Song" <yhs@fb.com>,
"John Fastabend" <john.fastabend@gmail.com>,
"KP Singh" <kpsingh@kernel.org>,
"Luke Nelson" <luke.r.nels@gmail.com>,
"Xi Wang" <xi.wang@gmail.com>,
linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
kasan-dev@googlegroups.com, netdev@vger.kernel.org,
bpf@vger.kernel.org
Subject: Re: [PATCH] riscv: Ensure BPF_JIT_REGION_START aligned with PMD size
Date: Fri, 18 Jun 2021 01:17:12 +0800 [thread overview]
Message-ID: <20210618011712.2bbacb27@xhacker> (raw)
In-Reply-To: <4cdb1261-6474-8ae6-7a92-a3be81ce8cb5@ghiti.fr>
On Thu, 17 Jun 2021 09:23:04 +0200
Alex Ghiti <alex@ghiti.fr> wrote:
> Le 16/06/2021 à 02:03, Jisheng Zhang a écrit :
> > On Tue, 15 Jun 2021 20:54:19 +0200
> > Alex Ghiti <alex@ghiti.fr> wrote:
> >
> >> Hi Jisheng,
> >
> > Hi Alex,
> >
> >>
> >> Le 14/06/2021 à 18:49, Jisheng Zhang a écrit :
> >>> From: Jisheng Zhang <jszhang@kernel.org>
> >>>
> >>> Andreas reported commit fc8504765ec5 ("riscv: bpf: Avoid breaking W^X")
> >>> breaks booting with one kind of config file, I reproduced a kernel panic
> >>> with the config:
> >>>
> >>> [ 0.138553] Unable to handle kernel paging request at virtual address ffffffff81201220
> >>> [ 0.139159] Oops [#1]
> >>> [ 0.139303] Modules linked in:
> >>> [ 0.139601] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.13.0-rc5-default+ #1
> >>> [ 0.139934] Hardware name: riscv-virtio,qemu (DT)
> >>> [ 0.140193] epc : __memset+0xc4/0xfc
> >>> [ 0.140416] ra : skb_flow_dissector_init+0x1e/0x82
> >>> [ 0.140609] epc : ffffffff8029806c ra : ffffffff8033be78 sp : ffffffe001647da0
> >>> [ 0.140878] gp : ffffffff81134b08 tp : ffffffe001654380 t0 : ffffffff81201158
> >>> [ 0.141156] t1 : 0000000000000002 t2 : 0000000000000154 s0 : ffffffe001647dd0
> >>> [ 0.141424] s1 : ffffffff80a43250 a0 : ffffffff81201220 a1 : 0000000000000000
> >>> [ 0.141654] a2 : 000000000000003c a3 : ffffffff81201258 a4 : 0000000000000064
> >>> [ 0.141893] a5 : ffffffff8029806c a6 : 0000000000000040 a7 : ffffffffffffffff
> >>> [ 0.142126] s2 : ffffffff81201220 s3 : 0000000000000009 s4 : ffffffff81135088
> >>> [ 0.142353] s5 : ffffffff81135038 s6 : ffffffff8080ce80 s7 : ffffffff80800438
> >>> [ 0.142584] s8 : ffffffff80bc6578 s9 : 0000000000000008 s10: ffffffff806000ac
> >>> [ 0.142810] s11: 0000000000000000 t3 : fffffffffffffffc t4 : 0000000000000000
> >>> [ 0.143042] t5 : 0000000000000155 t6 : 00000000000003ff
> >>> [ 0.143220] status: 0000000000000120 badaddr: ffffffff81201220 cause: 000000000000000f
> >>> [ 0.143560] [<ffffffff8029806c>] __memset+0xc4/0xfc
> >>> [ 0.143859] [<ffffffff8061e984>] init_default_flow_dissectors+0x22/0x60
> >>> [ 0.144092] [<ffffffff800010fc>] do_one_initcall+0x3e/0x168
> >>> [ 0.144278] [<ffffffff80600df0>] kernel_init_freeable+0x1c8/0x224
> >>> [ 0.144479] [<ffffffff804868a8>] kernel_init+0x12/0x110
> >>> [ 0.144658] [<ffffffff800022de>] ret_from_exception+0x0/0xc
> >>> [ 0.145124] ---[ end trace f1e9643daa46d591 ]---
> >>>
> >>> After some investigation, I think I found the root cause: commit
> >>> 2bfc6cd81bd ("move kernel mapping outside of linear mapping") moves
> >>> BPF JIT region after the kernel:
> >>>
> >>> The &_end is unlikely aligned with PMD size, so the front bpf jit
> >>> region sits with part of kernel .data section in one PMD size mapping.
> >>> But kernel is mapped in PMD SIZE, when bpf_jit_binary_lock_ro() is
> >>> called to make the first bpf jit prog ROX, we will make part of kernel
> >>> .data section RO too, so when we write to, for example memset the
> >>> .data section, MMU will trigger a store page fault.
> >>
> >> Good catch, we make sure no physical allocation happens between _end and
> >> the next PMD aligned address, but I missed this one.
> >>
> >>>
> >>> To fix the issue, we need to ensure the BPF JIT region is PMD size
> >>> aligned. This patch acchieve this goal by restoring the BPF JIT region
> >>> to original position, I.E the 128MB before kernel .text section.
> >>
> >> But I disagree with your solution: I made sure modules and BPF programs
> >> get their own virtual regions to avoid worst case scenario where one
> >> could allocate all the space and leave nothing to the other (we are
> >> limited to +- 2GB offset). Why don't just align BPF_JIT_REGION_START to
> >> the next PMD aligned address?
> >
> > Originally, I planed to fix the issue by aligning BPF_JIT_REGION_START, but
> > IIRC, BPF experts are adding (or have added) "Calling kernel functions from BPF"
> > feature, there's a risk that BPF JIT region is beyond the 2GB of module region:
> >
> > ------
> > module
> > ------
> > kernel
> > ------
> > BPF_JIT
> >
> > So I made this patch finally. In this patch, we let BPF JIT region sit
> > between module and kernel.
> >
>
> From what I read in the lwn article, I'm not sure BPF programs can call
> module functions, can someone tell us if it is possible? Or planned?
What about module call BPF program? this case also wants the 2GB address limit.
>
> > To address "make sure modules and BPF programs get their own virtual regions",
> > what about something as below (applied against this patch)?
> >
> > diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> > index 380cd3a7e548..da1158f10b09 100644
> > --- a/arch/riscv/include/asm/pgtable.h
> > +++ b/arch/riscv/include/asm/pgtable.h
> > @@ -31,7 +31,7 @@
> > #define BPF_JIT_REGION_SIZE (SZ_128M)
> > #ifdef CONFIG_64BIT
> > #define BPF_JIT_REGION_START (BPF_JIT_REGION_END - BPF_JIT_REGION_SIZE)
> > -#define BPF_JIT_REGION_END (MODULES_END)
> > +#define BPF_JIT_REGION_END (PFN_ALIGN((unsigned long)&_start))
> > #else
> > #define BPF_JIT_REGION_START (PAGE_OFFSET - BPF_JIT_REGION_SIZE)
> > #define BPF_JIT_REGION_END (VMALLOC_END)
> > @@ -40,7 +40,7 @@
> > /* Modules always live before the kernel */
> > #ifdef CONFIG_64BIT
> > #define MODULES_VADDR (PFN_ALIGN((unsigned long)&_end) - SZ_2G)
> > -#define MODULES_END (PFN_ALIGN((unsigned long)&_start))
> > +#define MODULES_END (BPF_JIT_REGION_END)
> > #endif
> >
> >
>
> In case it is possible, I would let the vmalloc allocator handle the
> case where modules steal room from BPF: I would then not implement the
> above but rather your first patch.
>
> And do not forget to modify Documentation/riscv/vm-layout.rst
> accordingly and remove the comment "/* KASLR should leave at least 128MB
> for BPF after the kernel */"
Thanks for the comments
next prev parent reply other threads:[~2021-06-17 17:24 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-29 18:21 [PATCH 0/9] riscv: improve self-protection Jisheng Zhang
2021-03-29 18:22 ` [PATCH 1/9] riscv: add __init section marker to some functions Jisheng Zhang
2021-03-29 18:22 ` [PATCH 2/9] riscv: Mark some global variables __ro_after_init Jisheng Zhang
2021-03-29 18:23 ` [PATCH 3/9] riscv: Constify sys_call_table Jisheng Zhang
2021-03-29 18:23 ` [PATCH 4/9] riscv: Constify sbi_ipi_ops Jisheng Zhang
2021-03-29 18:24 ` [PATCH 5/9] riscv: kprobes: Implement alloc_insn_page() Jisheng Zhang
2021-03-29 18:24 ` [PATCH 6/9] riscv: bpf: Move bpf_jit_alloc_exec() and bpf_jit_free_exec() to core Jisheng Zhang
2021-03-29 20:41 ` Luke Nelson
2021-03-29 18:25 ` [PATCH 7/9] riscv: bpf: Avoid breaking W^X Jisheng Zhang
2021-06-11 14:10 ` Andreas Schwab
2021-06-11 16:23 ` Jisheng Zhang
2021-06-11 16:41 ` Andreas Schwab
2021-06-13 17:05 ` Jisheng Zhang
2021-06-13 19:50 ` Andreas Schwab
2021-06-14 16:49 ` [PATCH] riscv: Ensure BPF_JIT_REGION_START aligned with PMD size Jisheng Zhang
2021-06-15 12:29 ` Daniel Borkmann
2021-06-15 18:54 ` Alex Ghiti
2021-06-16 0:03 ` Jisheng Zhang
2021-06-17 7:23 ` Alex Ghiti
2021-06-17 17:17 ` Jisheng Zhang [this message]
2021-06-17 7:30 ` Palmer Dabbelt
2021-06-17 8:09 ` Alex Ghiti
2021-06-17 14:18 ` Alex Ghiti
2021-06-17 17:27 ` Jisheng Zhang
2021-06-17 17:46 ` Jisheng Zhang
2021-06-17 18:10 ` Jisheng Zhang
2021-06-17 18:15 ` [PATCH v2] " Jisheng Zhang
2021-06-18 6:48 ` Alex Ghiti
2021-03-29 18:25 ` [PATCH 8/9] riscv: module: Create module allocations without exec permissions Jisheng Zhang
2021-03-29 18:26 ` [PATCH 9/9] riscv: Set ARCH_HAS_STRICT_MODULE_RWX if MMU Jisheng Zhang
2021-04-23 1:48 ` [PATCH 0/9] riscv: improve self-protection Palmer Dabbelt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210618011712.2bbacb27@xhacker \
--to=jszhang3@mail.ustc.edu.cn \
--cc=alex@ghiti.fr \
--cc=andreyknvl@gmail.com \
--cc=andrii@kernel.org \
--cc=aou@eecs.berkeley.edu \
--cc=ast@kernel.org \
--cc=bjorn@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dvyukov@google.com \
--cc=glider@google.com \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kasan-dev@googlegroups.com \
--cc=kpsingh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=luke.r.nels@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=palmer@dabbelt.com \
--cc=paul.walmsley@sifive.com \
--cc=ryabinin.a.a@gmail.com \
--cc=schwab@linux-m68k.org \
--cc=songliubraving@fb.com \
--cc=xi.wang@gmail.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).