preemptirq_delay_test module can trigger crash on arm64 vm

preemptirq_delay_test module can trigger crash on arm64 vm

[Xiao Yang]

Hi Steve, Joel

Sorry to bother you.

On my slow arm64 vm, inserting and removing preemptirq_delay_test module
in loops triggers kernel panic sometimes.

Reproduce steps:
Do the following steps in loops(more than 10 times):
[root@CentOS81-aarch64 ]# modprobe preemptirq_delay_test
test_mode=preempt delay=500000; rmmod preemptirq_delay_test

Call Trace:
-------------------------------------------------------------------
[172413.750294] Unable to handle kernel paging request at virtual
address fffffe0009010280
[172413.790511] Mem abort info:
[172413.791129]   ESR = 0x86000007
[172413.791656]   EC = 0x21: IABT (current EL), IL = 32 bits
[172413.792322]   SET = 0, FnV = 0
[172413.792793]   EA = 0, S1PTW = 0
[172413.794082] swapper pgtable: 64k pages, 42-bit VAs,
pgdp=0000000040ef0000
[172413.794556] [fffffe0009010280] pgd=00000000bfff0003,
pud=00000000bfff0003, pmd=00000000bfff0003, pte=0000000000000000
[172413.795690] Internal error: Oops: 86000007 [#1] PREEMPT SMP
[172413.796285] Modules linked in: sunrpc vfat fat ext4 mbcache jbd2
crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce ip_tables xfs
libcrc32c virtio_net net_failover failover virtio_blk virtio_mmio [last
unloaded: preemptirq_delay_test]
[172413.797797] CPU: 1 PID: 15750 Comm: preempt_test Not tainted
5.6.0-rc7 #17
[172413.798275] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0
02/06/2015
[172413.799049] pstate: 40000005 (nZcv daif -PAN -UAO)
[172413.799934] pc : 0xfffffe0009010280
[172413.800230] lr : 0xfffffe0009010280
[172413.800497] sp : fffffe001390fe00
[172413.800799] x29: fffffe001390fe00 x28: 0000000000000000
[172413.801263] x27: 0000000000000000 x26: fffffe0009010000
[172413.801726] x25: 0000000000000000 x24: fffffe00114a3670
[172413.802164] x23: fffffe0010d1d150 x22: fffffe0016b2f9e8
[172413.802635] x21: 0000000000000000 x20: fffffe0009030008
[172413.803191] x19: fffffe0009030000 x18: 0000000000000000
[172413.803672] x17: 0000000000000000 x16: 0000000000000000
[172413.804118] x15: 0000000000000000 x14: 0000000000000000
[172413.804558] x13: 0000000000000000 x12: 0000000000000000
[172413.805013] x11: 0000000000000000 x10: 0000000000000d50
[172413.805441] x9 : fffffe001390fd10 x8 : fffffc00762930b0
[172413.805944] x7 : 00000000fa83b2da x6 : 0000000000000000
[172413.806455] x5 : fffffe001127c000 x4 : fffffe006ed00000
[172413.806874] x3 : fffffe001127cf50 x2 : 0000000000000002
[172413.807312] x1 : 0000000000000000 x0 : 0000000000000228
[172413.807869] Call trace:
[172413.808146]  0xfffffe0009010280
[172413.808436]  0xfffffe0009010294
[172413.808723]  0xfffffe0009010050
[172413.809519]  kthread+0x110/0x118
[172413.809734]  ret_from_fork+0x10/0x18
[172413.810338] Code: bad PC value
[172413.811203] ---[ end trace 69c688402fdc8bb1 ]---
[172413.811670] Kernel panic - not syncing: Fatal exception
[172413.812108] SMP: stopping secondary CPUs
[172413.812733] Kernel Offset: disabled
[172413.813043] CPU features: 0x10002,20006082
[172413.813387] Memory Limit: none
[172413.813849] ---[ end Kernel panic - not syncing: Fatal exception]---
-------------------------------------------------------------------

Do you have any suggestion about the issue.

Thanks,
Xiao Yang

Re: preemptirq_delay_test module can trigger crash on arm64 vm

[Steven Rostedt]

On Fri, 24 Apr 2020 17:26:32 +0800
Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:

> Hi Steve, Joel
> 
> Sorry to bother you.
> 
> On my slow arm64 vm, inserting and removing preemptirq_delay_test module
> in loops triggers kernel panic sometimes.
> 
> Reproduce steps:
> Do the following steps in loops(more than 10 times):
> [root@CentOS81-aarch64 ]# modprobe preemptirq_delay_test
> test_mode=preempt delay=500000; rmmod preemptirq_delay_test
> 
>

Joel,

I never did really look at that module, but doing a quick scan, I see you
never clean up the task you start.

Which means, you can remove the module and the task still exists, and when
it runs, it will execute code that has been freed.

The module exit should still do a kthread_stop() on it. If anything, it
will prevent the task existing after the module is removed.

-- Steve

Re: preemptirq_delay_test module can trigger crash on arm64 vm

[Joel Fernandes]

On Fri, Apr 24, 2020 at 10:01:46AM -0400, Steven Rostedt wrote:
> On Fri, 24 Apr 2020 17:26:32 +0800
> Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:
> 
> > Hi Steve, Joel
> > 
> > Sorry to bother you.
> > 
> > On my slow arm64 vm, inserting and removing preemptirq_delay_test module
> > in loops triggers kernel panic sometimes.
> > 
> > Reproduce steps:
> > Do the following steps in loops(more than 10 times):
> > [root@CentOS81-aarch64 ]# modprobe preemptirq_delay_test
> > test_mode=preempt delay=500000; rmmod preemptirq_delay_test
> > 
> >
> 
> Joel,
> 
> I never did really look at that module, but doing a quick scan, I see you
> never clean up the task you start.
> 
> Which means, you can remove the module and the task still exists, and when
> it runs, it will execute code that has been freed.
> 
> The module exit should still do a kthread_stop() on it. If anything, it
> will prevent the task existing after the module is removed.

Thanks Steve for taking a look, I will submit a fix for it ASAP and send it to you
for -rc cycle. Thanks Xiao for the report.

 - Joel