* kernel panic: Attempted to kill init!
@ 2022-12-22 4:35 Hao Sun
2022-12-28 6:35 ` Yonghong Song
0 siblings, 1 reply; 8+ messages in thread
From: Hao Sun @ 2022-12-22 4:35 UTC (permalink / raw)
To: bpf
Cc: ast, daniel, john.fastabend, andrii, martin.lau, song, yhs,
kpsingh, sdf, haoluo, jolsa, davem, linux-kernel, Hao Sun
Hi,
This crash can be triggered by executing the C reproducer for
multiple times, which just keep loading the following prog as
raw tracepoint into kmem_cache_free().
The prog send SIGSEGV to current via bpf_send_signal_thread(),
after load this, whoever tries to free mem would trigger this,
kernel crashed when this happens to init.
Seems we should filter init out in bpf_send_signal_common() by
is_global_init(current), or maybe we should check this in the
verifier?
This can be reproduced on:
HEAD commit: 59fe41b5255f selftests/bpf: Add verifier test exercising jit PROBE_MEM logic
git tree: bpf-next
console output: https://pastebin.com/raw/FMgyvEnH
kernel config : https://pastebin.com/raw/XeF6jU43
C reproducer : https://pastebin.com/raw/Tag5N893
func#0 @0
0: R1=ctx(off=0,imm=0) R10=fp0
0: (18) r0 = 0x0 ; R0_w=0
2: (18) r6 = 0x0 ; R6_w=0
4: (18) r7 = 0x0 ; R7_w=0
6: (18) r8 = 0x0 ; R8_w=0
8: (18) r9 = 0x0 ; R9_w=0
10: (2d) if r0 > r0 goto pc+2
last_idx 10 first_idx 0
regs=1 stack=0 before 8: (18) r9 = 0x0
regs=1 stack=0 before 6: (18) r8 = 0x0
regs=1 stack=0 before 4: (18) r7 = 0x0
regs=1 stack=0 before 2: (18) r6 = 0x0
regs=1 stack=0 before 0: (18) r0 = 0x0
last_idx 10 first_idx 0
regs=1 stack=0 before 8: (18) r9 = 0x0
regs=1 stack=0 before 6: (18) r8 = 0x0
regs=1 stack=0 before 4: (18) r7 = 0x0
regs=1 stack=0 before 2: (18) r6 = 0x0
regs=1 stack=0 before 0: (18) r0 = 0x0
11: R0_w=0
11: (b7) r1 = 11 ; R1_w=11
12: (85) call bpf_send_signal_thread#117 ; R0=scalar()
13: (95) exit
processed 9 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
CPU: 3 PID: 1 Comm: systemd Not tainted 6.1.0-09652-g59fe41b5255f #148
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x100/0x178 lib/dump_stack.c:106
panic+0x2c4/0x60f kernel/panic.c:275
do_exit.cold+0x63/0xe4 kernel/exit.c:789
do_group_exit+0xd4/0x2a0 kernel/exit.c:950
get_signal+0x2460/0x2600 kernel/signal.c:2858
arch_do_signal_or_restart+0x78/0x5d0 arch/x86/kernel/signal.c:306
exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:203
__syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:296
do_syscall_64+0x44/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x55e738964df0
Code: 00 31 f6 89 ef 4c 8d 05 be 1b 0d 00 48 8d 15 b0 85 0c 00 31 c0 e8 f0 c3 ff ff e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 <41> 57 41 56 41 55 41 54 41 89 fc 55 53 48 81 ec 48 01 00 00 64 48
RSP: 002b:00007ffeb8e87bb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000f7
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f29dc8a6bc1
RDX: 00007ffeb8e87bc0 RSI: 00007ffeb8e87cf0 RDI: 000000000000000b
RBP: 00007ffeb90b73c0 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000004 R11: 0000000000000246 R12: 00007f29dc3f76c8
R13: 000000000000294d R14: 0000000000000000 R15: 00007ffeb9686870
</TASK>
Kernel Offset: disabled
Rebooting in 86400 seconds..
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2022-12-22 4:35 kernel panic: Attempted to kill init! Hao Sun
@ 2022-12-28 6:35 ` Yonghong Song
2022-12-30 9:54 ` Hao Sun
0 siblings, 1 reply; 8+ messages in thread
From: Yonghong Song @ 2022-12-28 6:35 UTC (permalink / raw)
To: Hao Sun, bpf
Cc: ast, daniel, john.fastabend, andrii, martin.lau, song, yhs,
kpsingh, sdf, haoluo, jolsa, davem, linux-kernel
On 12/21/22 8:35 PM, Hao Sun wrote:
> Hi,
>
> This crash can be triggered by executing the C reproducer for
> multiple times, which just keep loading the following prog as
> raw tracepoint into kmem_cache_free().
>
> The prog send SIGSEGV to current via bpf_send_signal_thread(),
> after load this, whoever tries to free mem would trigger this,
> kernel crashed when this happens to init.
>
> Seems we should filter init out in bpf_send_signal_common() by
> is_global_init(current), or maybe we should check this in the
> verifier?
The helper is just to send a particular signal to *current*
thread. In typical use case, it is never a good idea to send
the signal to a *random* thread. In certain cases, maybe user
indeed wants to send the signal to init thread to observe
something. Note that such destructive side effect already
exists in the bpf land. For example, for a xdp program,
it could drop all packets to make machine not responsive
to ssh etc. Therefore, I recommend to keep the existing
bpf_send_signal_common() helper behavior.
>
> This can be reproduced on:
>
> HEAD commit: 59fe41b5255f selftests/bpf: Add verifier test exercising jit PROBE_MEM logic
> git tree: bpf-next
> console output: https://pastebin.com/raw/FMgyvEnH
> kernel config : https://pastebin.com/raw/XeF6jU43
> C reproducer : https://pastebin.com/raw/Tag5N893
>
> func#0 @0
> 0: R1=ctx(off=0,imm=0) R10=fp0
> 0: (18) r0 = 0x0 ; R0_w=0
> 2: (18) r6 = 0x0 ; R6_w=0
> 4: (18) r7 = 0x0 ; R7_w=0
> 6: (18) r8 = 0x0 ; R8_w=0
> 8: (18) r9 = 0x0 ; R9_w=0
> 10: (2d) if r0 > r0 goto pc+2
> last_idx 10 first_idx 0
> regs=1 stack=0 before 8: (18) r9 = 0x0
> regs=1 stack=0 before 6: (18) r8 = 0x0
> regs=1 stack=0 before 4: (18) r7 = 0x0
> regs=1 stack=0 before 2: (18) r6 = 0x0
> regs=1 stack=0 before 0: (18) r0 = 0x0
> last_idx 10 first_idx 0
> regs=1 stack=0 before 8: (18) r9 = 0x0
> regs=1 stack=0 before 6: (18) r8 = 0x0
> regs=1 stack=0 before 4: (18) r7 = 0x0
> regs=1 stack=0 before 2: (18) r6 = 0x0
> regs=1 stack=0 before 0: (18) r0 = 0x0
> 11: R0_w=0
> 11: (b7) r1 = 11 ; R1_w=11
> 12: (85) call bpf_send_signal_thread#117 ; R0=scalar()
> 13: (95) exit
> processed 9 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
>
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> CPU: 3 PID: 1 Comm: systemd Not tainted 6.1.0-09652-g59fe41b5255f #148
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x100/0x178 lib/dump_stack.c:106
> panic+0x2c4/0x60f kernel/panic.c:275
> do_exit.cold+0x63/0xe4 kernel/exit.c:789
> do_group_exit+0xd4/0x2a0 kernel/exit.c:950
> get_signal+0x2460/0x2600 kernel/signal.c:2858
> arch_do_signal_or_restart+0x78/0x5d0 arch/x86/kernel/signal.c:306
> exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
> exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:203
> __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:296
> do_syscall_64+0x44/0xb0 arch/x86/entry/common.c:86
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
> RIP: 0033:0x55e738964df0
> Code: 00 31 f6 89 ef 4c 8d 05 be 1b 0d 00 48 8d 15 b0 85 0c 00 31 c0 e8 f0 c3 ff ff e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 <41> 57 41 56 41 55 41 54 41 89 fc 55 53 48 81 ec 48 01 00 00 64 48
> RSP: 002b:00007ffeb8e87bb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000f7
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f29dc8a6bc1
> RDX: 00007ffeb8e87bc0 RSI: 00007ffeb8e87cf0 RDI: 000000000000000b
> RBP: 00007ffeb90b73c0 R08: 0000000000000000 R09: 0000000000000002
> R10: 0000000000000004 R11: 0000000000000246 R12: 00007f29dc3f76c8
> R13: 000000000000294d R14: 0000000000000000 R15: 00007ffeb9686870
> </TASK>
> Kernel Offset: disabled
> Rebooting in 86400 seconds..
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2022-12-28 6:35 ` Yonghong Song
@ 2022-12-30 9:54 ` Hao Sun
2022-12-30 16:55 ` Alexei Starovoitov
0 siblings, 1 reply; 8+ messages in thread
From: Hao Sun @ 2022-12-30 9:54 UTC (permalink / raw)
To: Yonghong Song
Cc: bpf, Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Song Liu, Yonghong Song,
KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa, David Miller,
Linux Kernel Mailing List
> On 28 Dec 2022, at 2:35 PM, Yonghong Song <yhs@meta.com> wrote:
>
>
>
> On 12/21/22 8:35 PM, Hao Sun wrote:
>> Hi,
>> This crash can be triggered by executing the C reproducer for
>> multiple times, which just keep loading the following prog as
>> raw tracepoint into kmem_cache_free().
>> The prog send SIGSEGV to current via bpf_send_signal_thread(),
>> after load this, whoever tries to free mem would trigger this,
>> kernel crashed when this happens to init.
>> Seems we should filter init out in bpf_send_signal_common() by
>> is_global_init(current), or maybe we should check this in the
>> verifier?
>
> The helper is just to send a particular signal to *current*
> thread. In typical use case, it is never a good idea to send
> the signal to a *random* thread. In certain cases, maybe user
> indeed wants to send the signal to init thread to observe
> something. Note that such destructive side effect already
> exists in the bpf land. For example, for a xdp program,
> it could drop all packets to make machine not responsive
> to ssh etc. Therefore, I recommend to keep the existing
> bpf_send_signal_common() helper behavior.
Sound the two are different cases. Not responsive in XDP seems like
an intended behaviour, panic caused by killing init is buggy. If the
last thread of global init was killed, kernel panic immediately.
>
>> This can be reproduced on:
>> HEAD commit: 59fe41b5255f selftests/bpf: Add verifier test exercising jit PROBE_MEM logic
>> git tree: bpf-next
>> console output: https://pastebin.com/raw/FMgyvEnH
>> kernel config : https://pastebin.com/raw/XeF6jU43
>> C reproducer : https://pastebin.com/raw/Tag5N893
>> func#0 @0
>> 0: R1=ctx(off=0,imm=0) R10=fp0
>> 0: (18) r0 = 0x0 ; R0_w=0
>> 2: (18) r6 = 0x0 ; R6_w=0
>> 4: (18) r7 = 0x0 ; R7_w=0
>> 6: (18) r8 = 0x0 ; R8_w=0
>> 8: (18) r9 = 0x0 ; R9_w=0
>> 10: (2d) if r0 > r0 goto pc+2
>> last_idx 10 first_idx 0
>> regs=1 stack=0 before 8: (18) r9 = 0x0
>> regs=1 stack=0 before 6: (18) r8 = 0x0
>> regs=1 stack=0 before 4: (18) r7 = 0x0
>> regs=1 stack=0 before 2: (18) r6 = 0x0
>> regs=1 stack=0 before 0: (18) r0 = 0x0
>> last_idx 10 first_idx 0
>> regs=1 stack=0 before 8: (18) r9 = 0x0
>> regs=1 stack=0 before 6: (18) r8 = 0x0
>> regs=1 stack=0 before 4: (18) r7 = 0x0
>> regs=1 stack=0 before 2: (18) r6 = 0x0
>> regs=1 stack=0 before 0: (18) r0 = 0x0
>> 11: R0_w=0
>> 11: (b7) r1 = 11 ; R1_w=11
>> 12: (85) call bpf_send_signal_thread#117 ; R0=scalar()
>> 13: (95) exit
>> processed 9 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
>> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>> CPU: 3 PID: 1 Comm: systemd Not tainted 6.1.0-09652-g59fe41b5255f #148
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
>> Call Trace:
>> <TASK>
>> __dump_stack lib/dump_stack.c:88 [inline]
>> dump_stack_lvl+0x100/0x178 lib/dump_stack.c:106
>> panic+0x2c4/0x60f kernel/panic.c:275
>> do_exit.cold+0x63/0xe4 kernel/exit.c:789
>> do_group_exit+0xd4/0x2a0 kernel/exit.c:950
>> get_signal+0x2460/0x2600 kernel/signal.c:2858
>> arch_do_signal_or_restart+0x78/0x5d0 arch/x86/kernel/signal.c:306
>> exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
>> exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:203
>> __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
>> syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:296
>> do_syscall_64+0x44/0xb0 arch/x86/entry/common.c:86
>> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>> RIP: 0033:0x55e738964df0
>> Code: 00 31 f6 89 ef 4c 8d 05 be 1b 0d 00 48 8d 15 b0 85 0c 00 31 c0 e8 f0 c3 ff ff e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 <41> 57 41 56 41 55 41 54 41 89 fc 55 53 48 81 ec 48 01 00 00 64 48
>> RSP: 002b:00007ffeb8e87bb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000f7
>> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f29dc8a6bc1
>> RDX: 00007ffeb8e87bc0 RSI: 00007ffeb8e87cf0 RDI: 000000000000000b
>> RBP: 00007ffeb90b73c0 R08: 0000000000000000 R09: 0000000000000002
>> R10: 0000000000000004 R11: 0000000000000246 R12: 00007f29dc3f76c8
>> R13: 000000000000294d R14: 0000000000000000 R15: 00007ffeb9686870
>> </TASK>
>> Kernel Offset: disabled
>> Rebooting in 86400 seconds..
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2022-12-30 9:54 ` Hao Sun
@ 2022-12-30 16:55 ` Alexei Starovoitov
2023-01-03 12:46 ` Hao Sun
0 siblings, 1 reply; 8+ messages in thread
From: Alexei Starovoitov @ 2022-12-30 16:55 UTC (permalink / raw)
To: Hao Sun
Cc: Yonghong Song, bpf, Alexei Starovoitov, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Miller, Linux Kernel Mailing List
On Fri, Dec 30, 2022 at 1:54 AM Hao Sun <sunhao.th@gmail.com> wrote:
>
>
>
> > On 28 Dec 2022, at 2:35 PM, Yonghong Song <yhs@meta.com> wrote:
> >
> >
> >
> > On 12/21/22 8:35 PM, Hao Sun wrote:
> >> Hi,
> >> This crash can be triggered by executing the C reproducer for
> >> multiple times, which just keep loading the following prog as
> >> raw tracepoint into kmem_cache_free().
> >> The prog send SIGSEGV to current via bpf_send_signal_thread(),
> >> after load this, whoever tries to free mem would trigger this,
> >> kernel crashed when this happens to init.
> >> Seems we should filter init out in bpf_send_signal_common() by
> >> is_global_init(current), or maybe we should check this in the
> >> verifier?
> >
> > The helper is just to send a particular signal to *current*
> > thread. In typical use case, it is never a good idea to send
> > the signal to a *random* thread. In certain cases, maybe user
> > indeed wants to send the signal to init thread to observe
> > something. Note that such destructive side effect already
> > exists in the bpf land. For example, for a xdp program,
> > it could drop all packets to make machine not responsive
> > to ssh etc. Therefore, I recommend to keep the existing
> > bpf_send_signal_common() helper behavior.
>
> Sound the two are different cases. Not responsive in XDP seems like
> an intended behaviour, panic caused by killing init is buggy. If the
> last thread of global init was killed, kernel panic immediately.
I don't get it. How was it possible that this prog was
executed with current == pid 1 ?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2022-12-30 16:55 ` Alexei Starovoitov
@ 2023-01-03 12:46 ` Hao Sun
2023-01-03 18:33 ` Alexei Starovoitov
0 siblings, 1 reply; 8+ messages in thread
From: Hao Sun @ 2023-01-03 12:46 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Yonghong Song, bpf, Alexei Starovoitov, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Miller, Linux Kernel Mailing List
> On 31 Dec 2022, at 12:55 AM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Dec 30, 2022 at 1:54 AM Hao Sun <sunhao.th@gmail.com> wrote:
>>
>>
>>
>>> On 28 Dec 2022, at 2:35 PM, Yonghong Song <yhs@meta.com> wrote:
>>>
>>>
>>>
>>> On 12/21/22 8:35 PM, Hao Sun wrote:
>>>> Hi,
>>>> This crash can be triggered by executing the C reproducer for
>>>> multiple times, which just keep loading the following prog as
>>>> raw tracepoint into kmem_cache_free().
>>>> The prog send SIGSEGV to current via bpf_send_signal_thread(),
>>>> after load this, whoever tries to free mem would trigger this,
>>>> kernel crashed when this happens to init.
>>>> Seems we should filter init out in bpf_send_signal_common() by
>>>> is_global_init(current), or maybe we should check this in the
>>>> verifier?
>>>
>>> The helper is just to send a particular signal to *current*
>>> thread. In typical use case, it is never a good idea to send
>>> the signal to a *random* thread. In certain cases, maybe user
>>> indeed wants to send the signal to init thread to observe
>>> something. Note that such destructive side effect already
>>> exists in the bpf land. For example, for a xdp program,
>>> it could drop all packets to make machine not responsive
>>> to ssh etc. Therefore, I recommend to keep the existing
>>> bpf_send_signal_common() helper behavior.
>>
>> Sound the two are different cases. Not responsive in XDP seems like
>> an intended behaviour, panic caused by killing init is buggy. If the
>> last thread of global init was killed, kernel panic immediately.
>
> I don't get it. How was it possible that this prog was
> executed with current == pid 1 ?
The prog is raw trace point and is attached to ‘kmem_cache_free’ event.
When init triggered the event, the prog would be executed with pid 1.
But, the reason of this crash is not very clear to me, because it’s
really hard to debug with original C reproducer.
The following is the corresponding Syz prog:
# {Threaded:true Repeat:true RepeatTimes:0 Procs:1 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}
r0 = bpf$BPF_PROG_RAW_TRACEPOINT_LOAD(0x5, &(0x7f0000000000)={0x11, 0xe, &(0x7f0000000400)=ANY=[@ANYBLOB="18000000000000000000000000000000180600000000000000000000000000001807000000000000000000000000000018080000000000000000000000000000180900000000000000000000000000002d00020000000000b70100000b000000850000007500000095"], &(0x7f00000000c0)}, 0x80)
bpf$BPF_RAW_TRACEPOINT_OPEN(0x11, &(0x7f0000000100)={&(0x7f0000000080)='kmem_cache_free\x00', r0}, 0x10)
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2023-01-03 12:46 ` Hao Sun
@ 2023-01-03 18:33 ` Alexei Starovoitov
2023-01-05 9:00 ` Hao Sun
0 siblings, 1 reply; 8+ messages in thread
From: Alexei Starovoitov @ 2023-01-03 18:33 UTC (permalink / raw)
To: Hao Sun
Cc: Yonghong Song, bpf, Alexei Starovoitov, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Miller, Linux Kernel Mailing List
On Tue, Jan 3, 2023 at 4:46 AM Hao Sun <sunhao.th@gmail.com> wrote:
>
>
>
> > On 31 Dec 2022, at 12:55 AM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Dec 30, 2022 at 1:54 AM Hao Sun <sunhao.th@gmail.com> wrote:
> >>
> >>
> >>
> >>> On 28 Dec 2022, at 2:35 PM, Yonghong Song <yhs@meta.com> wrote:
> >>>
> >>>
> >>>
> >>> On 12/21/22 8:35 PM, Hao Sun wrote:
> >>>> Hi,
> >>>> This crash can be triggered by executing the C reproducer for
> >>>> multiple times, which just keep loading the following prog as
> >>>> raw tracepoint into kmem_cache_free().
> >>>> The prog send SIGSEGV to current via bpf_send_signal_thread(),
> >>>> after load this, whoever tries to free mem would trigger this,
> >>>> kernel crashed when this happens to init.
> >>>> Seems we should filter init out in bpf_send_signal_common() by
> >>>> is_global_init(current), or maybe we should check this in the
> >>>> verifier?
> >>>
> >>> The helper is just to send a particular signal to *current*
> >>> thread. In typical use case, it is never a good idea to send
> >>> the signal to a *random* thread. In certain cases, maybe user
> >>> indeed wants to send the signal to init thread to observe
> >>> something. Note that such destructive side effect already
> >>> exists in the bpf land. For example, for a xdp program,
> >>> it could drop all packets to make machine not responsive
> >>> to ssh etc. Therefore, I recommend to keep the existing
> >>> bpf_send_signal_common() helper behavior.
> >>
> >> Sound the two are different cases. Not responsive in XDP seems like
> >> an intended behaviour, panic caused by killing init is buggy. If the
> >> last thread of global init was killed, kernel panic immediately.
> >
> > I don't get it. How was it possible that this prog was
> > executed with current == pid 1 ?
>
> The prog is raw trace point and is attached to ‘kmem_cache_free’ event.
> When init triggered the event, the prog would be executed with pid 1.
> But, the reason of this crash is not very clear to me, because it’s
> really hard to debug with original C reproducer.
>
> The following is the corresponding Syz prog:
>
> # {Threaded:true Repeat:true RepeatTimes:0 Procs:1 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}
> r0 = bpf$BPF_PROG_RAW_TRACEPOINT_LOAD(0x5, &(0x7f0000000000)={0x11, 0xe, &(0x7f0000000400)=ANY=[@ANYBLOB="18000000000000000000000000000000180600000000000000000000000000001807000000000000000000000000000018080000000000000000000000000000180900000000000000000000000000002d00020000000000b70100000b000000850000007500000095"], &(0x7f00000000c0)}, 0x80)
> bpf$BPF_RAW_TRACEPOINT_OPEN(0x11, &(0x7f0000000100)={&(0x7f0000000080)='kmem_cache_free\x00', r0}, 0x10)
Does syzbot running without any user space?
Is syzbot itself a pid=1 ? and the only process ?
If so, the error would makes sense.
I guess we can add a safety check to bpf_send_signal_common
to prevent syzbot from killing itself.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2023-01-03 18:33 ` Alexei Starovoitov
@ 2023-01-05 9:00 ` Hao Sun
2023-01-06 3:01 ` Alexei Starovoitov
0 siblings, 1 reply; 8+ messages in thread
From: Hao Sun @ 2023-01-05 9:00 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Yonghong Song, bpf, Alexei Starovoitov, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Miller, Linux Kernel Mailing List
> On 4 Jan 2023, at 2:33 AM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Jan 3, 2023 at 4:46 AM Hao Sun <sunhao.th@gmail.com> wrote:
>>
>>
>>
>>> On 31 Dec 2022, at 12:55 AM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>>
>>> On Fri, Dec 30, 2022 at 1:54 AM Hao Sun <sunhao.th@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>> On 28 Dec 2022, at 2:35 PM, Yonghong Song <yhs@meta.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 12/21/22 8:35 PM, Hao Sun wrote:
>>>>>> Hi,
>>>>>> This crash can be triggered by executing the C reproducer for
>>>>>> multiple times, which just keep loading the following prog as
>>>>>> raw tracepoint into kmem_cache_free().
>>>>>> The prog send SIGSEGV to current via bpf_send_signal_thread(),
>>>>>> after load this, whoever tries to free mem would trigger this,
>>>>>> kernel crashed when this happens to init.
>>>>>> Seems we should filter init out in bpf_send_signal_common() by
>>>>>> is_global_init(current), or maybe we should check this in the
>>>>>> verifier?
>>>>>
>>>>> The helper is just to send a particular signal to *current*
>>>>> thread. In typical use case, it is never a good idea to send
>>>>> the signal to a *random* thread. In certain cases, maybe user
>>>>> indeed wants to send the signal to init thread to observe
>>>>> something. Note that such destructive side effect already
>>>>> exists in the bpf land. For example, for a xdp program,
>>>>> it could drop all packets to make machine not responsive
>>>>> to ssh etc. Therefore, I recommend to keep the existing
>>>>> bpf_send_signal_common() helper behavior.
>>>>
>>>> Sound the two are different cases. Not responsive in XDP seems like
>>>> an intended behaviour, panic caused by killing init is buggy. If the
>>>> last thread of global init was killed, kernel panic immediately.
>>>
>>> I don't get it. How was it possible that this prog was
>>> executed with current == pid 1 ?
>>
>> The prog is raw trace point and is attached to ‘kmem_cache_free’ event.
>> When init triggered the event, the prog would be executed with pid 1.
>> But, the reason of this crash is not very clear to me, because it’s
>> really hard to debug with original C reproducer.
>>
>> The following is the corresponding Syz prog:
>>
>> # {Threaded:true Repeat:true RepeatTimes:0 Procs:1 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}
>> r0 = bpf$BPF_PROG_RAW_TRACEPOINT_LOAD(0x5, &(0x7f0000000000)={0x11, 0xe, &(0x7f0000000400)=ANY=[@ANYBLOB="18000000000000000000000000000000180600000000000000000000000000001807000000000000000000000000000018080000000000000000000000000000180900000000000000000000000000002d00020000000000b70100000b000000850000007500000095"], &(0x7f00000000c0)}, 0x80)
>> bpf$BPF_RAW_TRACEPOINT_OPEN(0x11, &(0x7f0000000100)={&(0x7f0000000080)='kmem_cache_free\x00', r0}, 0x10)
>
> Does syzbot running without any user space?
> Is syzbot itself a pid=1 ? and the only process ?
> If so, the error would makes sense.
Yes, after read the C reproducer again, noticed that after a
bunch of sandbox setup, the pid of the reproducer process at
runtime is 1.
> I guess we can add a safety check to bpf_send_signal_common
> to prevent syzbot from killing itself.
Maybe something like this? This can avoid the panic, but won’t
allow task with pid=1 to send signal with prog.
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 23ce498bca97..94d2af2ce433 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -844,6 +844,8 @@ static int bpf_send_signal_common(u32 sig, enum pid_type type)
*/
if (unlikely(current->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
+ if (unlikely(is_global_init(current)))
+ return -EPERM;
if (unlikely(!nmi_uaccess_okay()))
return -EPERM;
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: kernel panic: Attempted to kill init!
2023-01-05 9:00 ` Hao Sun
@ 2023-01-06 3:01 ` Alexei Starovoitov
0 siblings, 0 replies; 8+ messages in thread
From: Alexei Starovoitov @ 2023-01-06 3:01 UTC (permalink / raw)
To: Hao Sun
Cc: Yonghong Song, bpf, Alexei Starovoitov, Daniel Borkmann,
John Fastabend, Andrii Nakryiko, Martin KaFai Lau, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Miller, Linux Kernel Mailing List
On Thu, Jan 5, 2023 at 1:00 AM Hao Sun <sunhao.th@gmail.com> wrote:
> >
> > Does syzbot running without any user space?
> > Is syzbot itself a pid=1 ? and the only process ?
> > If so, the error would makes sense.
>
> Yes, after read the C reproducer again, noticed that after a
> bunch of sandbox setup, the pid of the reproducer process at
> runtime is 1.
>
> > I guess we can add a safety check to bpf_send_signal_common
> > to prevent syzbot from killing itself.
>
> Maybe something like this? This can avoid the panic, but won’t
> allow task with pid=1 to send signal with prog.
>
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index 23ce498bca97..94d2af2ce433 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -844,6 +844,8 @@ static int bpf_send_signal_common(u32 sig, enum pid_type type)
> */
> if (unlikely(current->flags & (PF_KTHREAD | PF_EXITING)))
> return -EPERM;
> + if (unlikely(is_global_init(current)))
> + return -EPERM;
> if (unlikely(!nmi_uaccess_okay()))
> return -EPERM;
Yep. Good idea. Pls send an official patch.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-01-06 3:03 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-22 4:35 kernel panic: Attempted to kill init! Hao Sun
2022-12-28 6:35 ` Yonghong Song
2022-12-30 9:54 ` Hao Sun
2022-12-30 16:55 ` Alexei Starovoitov
2023-01-03 12:46 ` Hao Sun
2023-01-03 18:33 ` Alexei Starovoitov
2023-01-05 9:00 ` Hao Sun
2023-01-06 3:01 ` Alexei Starovoitov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).