* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
@ 2020-04-17 2:40 Qian Cai
2020-04-17 2:46 ` Russell Currey
0 siblings, 1 reply; 12+ messages in thread
From: Qian Cai @ 2020-04-17 2:40 UTC (permalink / raw)
To: Russell Currey
Cc: Steven Rostedt, Michael Ellerman, Nicholas Piggin, linuxppc-dev, LKML
> On Apr 16, 2020, at 10:27 PM, Russell Currey <ruscur@russell.cc> wrote:
>
> Reverting the patch with the given config will have the same effect as
> STRICT_KERNEL_RWX=n. Not discounting that it could be a bug on the
> powerpc side (i.e. relocatable kernels with strict RWX on haven't been
> exhaustively tested yet), but we should definitely figure out what's
> going on with this bad access first.
BTW, this bad access only happened once. The overwhelming rest of crashes are with NULL pointer NIP like below. How can you explain that STRICT_KERNEL_RWX=n would also make those NULL NIP disappear if STRICT_KERNEL_RWX is just a messenger?
[ 215.281666][T16896] LTP: starting chown04_16
[ 215.424203][T18297] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[ 215.424289][T18297] Faulting instruction address: 0x00000000
[ 215.424313][T18297] Oops: Kernel access of bad area, sig: 11 [#1]
[ 215.424341][T18297] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256 DEBUG_PAGEALLOC NUMA PowerNV
[ 215.424383][T18297] Modules linked in: loop kvm_hv kvm ip_tables x_tables xfs sd_mod bnx2x mdio tg3 ahci libahci libphy libata firmware_class dm_mirror dm_region_hash dm_log dm_mod
[ 215.424459][T18297] CPU: 85 PID: 18297 Comm: chown04_16 Tainted: G W 5.6.0-next-20200405+ #3
[ 215.424489][T18297] NIP: 0000000000000000 LR: c00800000fbc0408 CTR: 0000000000000000
[ 215.424530][T18297] REGS: c000200b8606f990 TRAP: 0400 Tainted: G W (5.6.0-next-20200405+)
[ 215.424570][T18297] MSR: 9000000040009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
[ 215.424619][T18297] CFAR: c00800000fbc64f4 IRQMASK: 0
[ 215.424619][T18297] GPR00: c0000000006c2238 c000200b8606fc20 c00000000165ce00 0000000000000000
[ 215.424619][T18297] GPR04: c000201a58106400 c000200b8606fcc0 000000005f037e7d ffffffff00013bfb
[ 215.424619][T18297] GPR08: c000201a58106400 0000000000000000 0000000000000000 c000000001652ee0
[ 215.424619][T18297] GPR12: 0000000000000000 c000201fff69a600 0000000000000000 0000000000000000
[ 215.424619][T18297] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 215.424619][T18297] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000007
[ 215.424619][T18297] GPR24: 0000000000000000 0000000000000000 c00800000fbc8688 c000200b8606fcc0
[ 215.424619][T18297] GPR28: 0000000000000000 000000007fffffff c00800000fbc0400 c00020068b8c0e70
[ 215.424914][T18297] NIP [0000000000000000] 0x0
[ 215.424953][T18297] LR [c00800000fbc0408] find_free_cb+0x8/0x30 [loop]
find_free_cb at drivers/block/loop.c:2129
[ 215.424997][T18297] Call Trace:
[ 215.425036][T18297] [c000200b8606fc20] [c0000000006c2290] idr_for_each+0xf0/0x170 (unreliable)
[ 215.425073][T18297] [c000200b8606fca0] [c00800000fbc2744] loop_lookup.part.2+0x4c/0xb0 [loop]
loop_lookup at drivers/block/loop.c:2144
[ 215.425105][T18297] [c000200b8606fce0] [c00800000fbc3558] loop_control_ioctl+0x120/0x1d0 [loop]
[ 215.425149][T18297] [c000200b8606fd40] [c0000000004eb688] ksys_ioctl+0xd8/0x130
[ 215.425190][T18297] [c000200b8606fd90] [c0000000004eb708] sys_ioctl+0x28/0x40
[ 215.425233][T18297] [c000200b8606fdb0] [c00000000003cc30] system_call_exception+0x110/0x1e0
[ 215.425274][T18297] [c000200b8606fe20] [c00000000000c9f0] system_call_common+0xf0/0x278
[ 215.425314][T18297] Instruction dump:
[ 215.425338][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[ 215.425374][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[ 215.425422][T18297] ---[ end trace ebed248fad431966 ]---
[ 215.642114][T18297]
[ 216.642220][T18297] Kernel panic - not syncing: Fatal exception
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 2:40 POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...) Qian Cai
@ 2020-04-17 2:46 ` Russell Currey
2020-04-17 3:16 ` Qian Cai
2020-04-17 11:49 ` Qian Cai
0 siblings, 2 replies; 12+ messages in thread
From: Russell Currey @ 2020-04-17 2:46 UTC (permalink / raw)
To: Qian Cai
Cc: Steven Rostedt, Michael Ellerman, Nicholas Piggin, linuxppc-dev, LKML
On Thu, 2020-04-16 at 22:40 -0400, Qian Cai wrote:
> > On Apr 16, 2020, at 10:27 PM, Russell Currey <ruscur@russell.cc>
> > wrote:
> >
> > Reverting the patch with the given config will have the same effect
> > as
> > STRICT_KERNEL_RWX=n. Not discounting that it could be a bug on the
> > powerpc side (i.e. relocatable kernels with strict RWX on haven't
> > been
> > exhaustively tested yet), but we should definitely figure out
> > what's
> > going on with this bad access first.
>
> BTW, this bad access only happened once. The overwhelming rest of
> crashes are with NULL pointer NIP like below. How can you explain
> that STRICT_KERNEL_RWX=n would also make those NULL NIP disappear if
> STRICT_KERNEL_RWX is just a messenger?
What happens if you test with STRICT_KERNEL_RWX=y and RELOCATABLE=n,
reverting my patch? This would give us an idea of whether it's
something broken recently or if there's something else going on.
>
> [ 215.281666][T16896] LTP: starting chown04_16
> [ 215.424203][T18297] BUG: Unable to handle kernel instruction fetch
> (NULL pointer?)
> [ 215.424289][T18297] Faulting instruction address: 0x00000000
> [ 215.424313][T18297] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 215.424341][T18297] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256
> DEBUG_PAGEALLOC NUMA PowerNV
> [ 215.424383][T18297] Modules linked in: loop kvm_hv kvm ip_tables
> x_tables xfs sd_mod bnx2x mdio tg3 ahci libahci libphy libata
> firmware_class dm_mirror dm_region_hash dm_log dm_mod
> [ 215.424459][T18297] CPU: 85 PID: 18297 Comm: chown04_16 Tainted:
> G W 5.6.0-next-20200405+ #3
> [ 215.424489][T18297] NIP: 0000000000000000 LR: c00800000fbc0408
> CTR: 0000000000000000
> [ 215.424530][T18297] REGS: c000200b8606f990 TRAP: 0400 Tainted:
> G W (5.6.0-next-20200405+)
> [ 215.424570][T18297] MSR: 9000000040009033
> <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
> [ 215.424619][T18297] CFAR: c00800000fbc64f4 IRQMASK: 0
> [ 215.424619][T18297] GPR00: c0000000006c2238 c000200b8606fc20
> c00000000165ce00 0000000000000000
> [ 215.424619][T18297] GPR04: c000201a58106400 c000200b8606fcc0
> 000000005f037e7d ffffffff00013bfb
> [ 215.424619][T18297] GPR08: c000201a58106400 0000000000000000
> 0000000000000000 c000000001652ee0
> [ 215.424619][T18297] GPR12: 0000000000000000 c000201fff69a600
> 0000000000000000 0000000000000000
> [ 215.424619][T18297] GPR16: 0000000000000000 0000000000000000
> 0000000000000000 0000000000000000
> [ 215.424619][T18297] GPR20: 0000000000000000 0000000000000000
> 0000000000000000 0000000000000007
> [ 215.424619][T18297] GPR24: 0000000000000000 0000000000000000
> c00800000fbc8688 c000200b8606fcc0
> [ 215.424619][T18297] GPR28: 0000000000000000 000000007fffffff
> c00800000fbc0400 c00020068b8c0e70
> [ 215.424914][T18297] NIP [0000000000000000] 0x0
> [ 215.424953][T18297] LR [c00800000fbc0408] find_free_cb+0x8/0x30
> [loop]
> find_free_cb at drivers/block/loop.c:2129
> [ 215.424997][T18297] Call Trace:
> [ 215.425036][T18297] [c000200b8606fc20] [c0000000006c2290]
> idr_for_each+0xf0/0x170 (unreliable)
> [ 215.425073][T18297] [c000200b8606fca0] [c00800000fbc2744]
> loop_lookup.part.2+0x4c/0xb0 [loop]
> loop_lookup at drivers/block/loop.c:2144
> [ 215.425105][T18297] [c000200b8606fce0] [c00800000fbc3558]
> loop_control_ioctl+0x120/0x1d0 [loop]
> [ 215.425149][T18297] [c000200b8606fd40] [c0000000004eb688]
> ksys_ioctl+0xd8/0x130
> [ 215.425190][T18297] [c000200b8606fd90] [c0000000004eb708]
> sys_ioctl+0x28/0x40
> [ 215.425233][T18297] [c000200b8606fdb0] [c00000000003cc30]
> system_call_exception+0x110/0x1e0
> [ 215.425274][T18297] [c000200b8606fe20] [c00000000000c9f0]
> system_call_common+0xf0/0x278
> [ 215.425314][T18297] Instruction dump:
> [ 215.425338][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> XXXXXXXX XXXXXXXX XXXXXXXX
> [ 215.425374][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> XXXXXXXX XXXXXXXX XXXXXXXX
> [ 215.425422][T18297] ---[ end trace ebed248fad431966 ]---
> [ 215.642114][T18297]
> [ 216.642220][T18297] Kernel panic - not syncing: Fatal exception
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 2:46 ` Russell Currey
@ 2020-04-17 3:16 ` Qian Cai
2020-04-17 11:49 ` Qian Cai
1 sibling, 0 replies; 12+ messages in thread
From: Qian Cai @ 2020-04-17 3:16 UTC (permalink / raw)
To: Russell Currey
Cc: Steven Rostedt, Michael Ellerman, Nicholas Piggin, linuxppc-dev, LKML
> On Apr 16, 2020, at 10:46 PM, Russell Currey <ruscur@russell.cc> wrote:
>
> On Thu, 2020-04-16 at 22:40 -0400, Qian Cai wrote:
>>> On Apr 16, 2020, at 10:27 PM, Russell Currey <ruscur@russell.cc>
>>> wrote:
>>>
>>> Reverting the patch with the given config will have the same effect
>>> as
>>> STRICT_KERNEL_RWX=n. Not discounting that it could be a bug on the
>>> powerpc side (i.e. relocatable kernels with strict RWX on haven't
>>> been
>>> exhaustively tested yet), but we should definitely figure out
>>> what's
>>> going on with this bad access first.
>>
>> BTW, this bad access only happened once. The overwhelming rest of
>> crashes are with NULL pointer NIP like below. How can you explain
>> that STRICT_KERNEL_RWX=n would also make those NULL NIP disappear if
>> STRICT_KERNEL_RWX is just a messenger?
>
> What happens if you test with STRICT_KERNEL_RWX=y and RELOCATABLE=n,
> reverting my patch? This would give us an idea of whether it's
> something broken recently or if there's something else going on.
I don’t know what did you mean by reverting your patch because that combination
can be tested as-is. Anyway, it could take a long time to reproduce, so I’ll keep it
running for up to 12-hour to confirm it could not really crash.
>
>>
>> [ 215.281666][T16896] LTP: starting chown04_16
>> [ 215.424203][T18297] BUG: Unable to handle kernel instruction fetch
>> (NULL pointer?)
>> [ 215.424289][T18297] Faulting instruction address: 0x00000000
>> [ 215.424313][T18297] Oops: Kernel access of bad area, sig: 11 [#1]
>> [ 215.424341][T18297] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256
>> DEBUG_PAGEALLOC NUMA PowerNV
>> [ 215.424383][T18297] Modules linked in: loop kvm_hv kvm ip_tables
>> x_tables xfs sd_mod bnx2x mdio tg3 ahci libahci libphy libata
>> firmware_class dm_mirror dm_region_hash dm_log dm_mod
>> [ 215.424459][T18297] CPU: 85 PID: 18297 Comm: chown04_16 Tainted:
>> G W 5.6.0-next-20200405+ #3
>> [ 215.424489][T18297] NIP: 0000000000000000 LR: c00800000fbc0408
>> CTR: 0000000000000000
>> [ 215.424530][T18297] REGS: c000200b8606f990 TRAP: 0400 Tainted:
>> G W (5.6.0-next-20200405+)
>> [ 215.424570][T18297] MSR: 9000000040009033
>> <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
>> [ 215.424619][T18297] CFAR: c00800000fbc64f4 IRQMASK: 0
>> [ 215.424619][T18297] GPR00: c0000000006c2238 c000200b8606fc20
>> c00000000165ce00 0000000000000000
>> [ 215.424619][T18297] GPR04: c000201a58106400 c000200b8606fcc0
>> 000000005f037e7d ffffffff00013bfb
>> [ 215.424619][T18297] GPR08: c000201a58106400 0000000000000000
>> 0000000000000000 c000000001652ee0
>> [ 215.424619][T18297] GPR12: 0000000000000000 c000201fff69a600
>> 0000000000000000 0000000000000000
>> [ 215.424619][T18297] GPR16: 0000000000000000 0000000000000000
>> 0000000000000000 0000000000000000
>> [ 215.424619][T18297] GPR20: 0000000000000000 0000000000000000
>> 0000000000000000 0000000000000007
>> [ 215.424619][T18297] GPR24: 0000000000000000 0000000000000000
>> c00800000fbc8688 c000200b8606fcc0
>> [ 215.424619][T18297] GPR28: 0000000000000000 000000007fffffff
>> c00800000fbc0400 c00020068b8c0e70
>> [ 215.424914][T18297] NIP [0000000000000000] 0x0
>> [ 215.424953][T18297] LR [c00800000fbc0408] find_free_cb+0x8/0x30
>> [loop]
>> find_free_cb at drivers/block/loop.c:2129
>> [ 215.424997][T18297] Call Trace:
>> [ 215.425036][T18297] [c000200b8606fc20] [c0000000006c2290]
>> idr_for_each+0xf0/0x170 (unreliable)
>> [ 215.425073][T18297] [c000200b8606fca0] [c00800000fbc2744]
>> loop_lookup.part.2+0x4c/0xb0 [loop]
>> loop_lookup at drivers/block/loop.c:2144
>> [ 215.425105][T18297] [c000200b8606fce0] [c00800000fbc3558]
>> loop_control_ioctl+0x120/0x1d0 [loop]
>> [ 215.425149][T18297] [c000200b8606fd40] [c0000000004eb688]
>> ksys_ioctl+0xd8/0x130
>> [ 215.425190][T18297] [c000200b8606fd90] [c0000000004eb708]
>> sys_ioctl+0x28/0x40
>> [ 215.425233][T18297] [c000200b8606fdb0] [c00000000003cc30]
>> system_call_exception+0x110/0x1e0
>> [ 215.425274][T18297] [c000200b8606fe20] [c00000000000c9f0]
>> system_call_common+0xf0/0x278
>> [ 215.425314][T18297] Instruction dump:
>> [ 215.425338][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
>> XXXXXXXX XXXXXXXX XXXXXXXX
>> [ 215.425374][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
>> XXXXXXXX XXXXXXXX XXXXXXXX
>> [ 215.425422][T18297] ---[ end trace ebed248fad431966 ]---
>> [ 215.642114][T18297]
>> [ 216.642220][T18297] Kernel panic - not syncing: Fatal exception
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 2:46 ` Russell Currey
2020-04-17 3:16 ` Qian Cai
@ 2020-04-17 11:49 ` Qian Cai
1 sibling, 0 replies; 12+ messages in thread
From: Qian Cai @ 2020-04-17 11:49 UTC (permalink / raw)
To: Russell Currey
Cc: Steven Rostedt, Michael Ellerman, Nicholas Piggin, linuxppc-dev, LKML
> On Apr 16, 2020, at 10:46 PM, Russell Currey <ruscur@russell.cc> wrote:
>
> On Thu, 2020-04-16 at 22:40 -0400, Qian Cai wrote:
>>> On Apr 16, 2020, at 10:27 PM, Russell Currey <ruscur@russell.cc>
>>> wrote:
>>>
>>> Reverting the patch with the given config will have the same effect
>>> as
>>> STRICT_KERNEL_RWX=n. Not discounting that it could be a bug on the
>>> powerpc side (i.e. relocatable kernels with strict RWX on haven't
>>> been
>>> exhaustively tested yet), but we should definitely figure out
>>> what's
>>> going on with this bad access first.
>>
>> BTW, this bad access only happened once. The overwhelming rest of
>> crashes are with NULL pointer NIP like below. How can you explain
>> that STRICT_KERNEL_RWX=n would also make those NULL NIP disappear if
>> STRICT_KERNEL_RWX is just a messenger?
>
> What happens if you test with STRICT_KERNEL_RWX=y and RELOCATABLE=n,
> reverting my patch? This would give us an idea of whether it's
> something broken recently or if there's something else going on.
That combination will crash as well. I don’t think it is broken recently though due to
the crash could happen back in 5.6-rc1 when your commit first introduced.
>
>>
>> [ 215.281666][T16896] LTP: starting chown04_16
>> [ 215.424203][T18297] BUG: Unable to handle kernel instruction fetch
>> (NULL pointer?)
>> [ 215.424289][T18297] Faulting instruction address: 0x00000000
>> [ 215.424313][T18297] Oops: Kernel access of bad area, sig: 11 [#1]
>> [ 215.424341][T18297] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256
>> DEBUG_PAGEALLOC NUMA PowerNV
>> [ 215.424383][T18297] Modules linked in: loop kvm_hv kvm ip_tables
>> x_tables xfs sd_mod bnx2x mdio tg3 ahci libahci libphy libata
>> firmware_class dm_mirror dm_region_hash dm_log dm_mod
>> [ 215.424459][T18297] CPU: 85 PID: 18297 Comm: chown04_16 Tainted:
>> G W 5.6.0-next-20200405+ #3
>> [ 215.424489][T18297] NIP: 0000000000000000 LR: c00800000fbc0408
>> CTR: 0000000000000000
>> [ 215.424530][T18297] REGS: c000200b8606f990 TRAP: 0400 Tainted:
>> G W (5.6.0-next-20200405+)
>> [ 215.424570][T18297] MSR: 9000000040009033
>> <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
>> [ 215.424619][T18297] CFAR: c00800000fbc64f4 IRQMASK: 0
>> [ 215.424619][T18297] GPR00: c0000000006c2238 c000200b8606fc20
>> c00000000165ce00 0000000000000000
>> [ 215.424619][T18297] GPR04: c000201a58106400 c000200b8606fcc0
>> 000000005f037e7d ffffffff00013bfb
>> [ 215.424619][T18297] GPR08: c000201a58106400 0000000000000000
>> 0000000000000000 c000000001652ee0
>> [ 215.424619][T18297] GPR12: 0000000000000000 c000201fff69a600
>> 0000000000000000 0000000000000000
>> [ 215.424619][T18297] GPR16: 0000000000000000 0000000000000000
>> 0000000000000000 0000000000000000
>> [ 215.424619][T18297] GPR20: 0000000000000000 0000000000000000
>> 0000000000000000 0000000000000007
>> [ 215.424619][T18297] GPR24: 0000000000000000 0000000000000000
>> c00800000fbc8688 c000200b8606fcc0
>> [ 215.424619][T18297] GPR28: 0000000000000000 000000007fffffff
>> c00800000fbc0400 c00020068b8c0e70
>> [ 215.424914][T18297] NIP [0000000000000000] 0x0
>> [ 215.424953][T18297] LR [c00800000fbc0408] find_free_cb+0x8/0x30
>> [loop]
>> find_free_cb at drivers/block/loop.c:2129
>> [ 215.424997][T18297] Call Trace:
>> [ 215.425036][T18297] [c000200b8606fc20] [c0000000006c2290]
>> idr_for_each+0xf0/0x170 (unreliable)
>> [ 215.425073][T18297] [c000200b8606fca0] [c00800000fbc2744]
>> loop_lookup.part.2+0x4c/0xb0 [loop]
>> loop_lookup at drivers/block/loop.c:2144
>> [ 215.425105][T18297] [c000200b8606fce0] [c00800000fbc3558]
>> loop_control_ioctl+0x120/0x1d0 [loop]
>> [ 215.425149][T18297] [c000200b8606fd40] [c0000000004eb688]
>> ksys_ioctl+0xd8/0x130
>> [ 215.425190][T18297] [c000200b8606fd90] [c0000000004eb708]
>> sys_ioctl+0x28/0x40
>> [ 215.425233][T18297] [c000200b8606fdb0] [c00000000003cc30]
>> system_call_exception+0x110/0x1e0
>> [ 215.425274][T18297] [c000200b8606fe20] [c00000000000c9f0]
>> system_call_common+0xf0/0x278
>> [ 215.425314][T18297] Instruction dump:
>> [ 215.425338][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
>> XXXXXXXX XXXXXXXX XXXXXXXX
>> [ 215.425374][T18297] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
>> XXXXXXXX XXXXXXXX XXXXXXXX
>> [ 215.425422][T18297] ---[ end trace ebed248fad431966 ]---
>> [ 215.642114][T18297]
>> [ 216.642220][T18297] Kernel panic - not syncing: Fatal exception
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 12:00 ` Qian Cai
@ 2020-04-17 15:18 ` Naveen N. Rao
0 siblings, 0 replies; 12+ messages in thread
From: Naveen N. Rao @ 2020-04-17 15:18 UTC (permalink / raw)
To: Qian Cai, Russell Currey
Cc: LKML, linuxppc-dev, Michael Ellerman, Nicholas Piggin, Steven Rostedt
Qian Cai wrote:
>
>
>> On Apr 17, 2020, at 3:01 AM, Naveen N. Rao <naveen.n.rao@linux.ibm.com> wrote:
>>
>> Hi Qian,
>>
>> Qian Cai wrote:
>>> OK, reverted the commit,
>>> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>>> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
>>> https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
>>
>> Do you see any errors logged in dmesg when you see the crash? STRICT_KERNEL_RWX changes how patch_instruction() works, so it would be interesting to see if there are any ftrace-related errors thrown before the crash.
>
> Yes, looks like there is a warning right after,
>
> echo function > /sys/kernel/debug/tracing/current_tracer
> echo nop > /sys/kernel/debug/tracing/current_tracer
>
> and just before the crash,
>
> [ T3454] ftrace-powerpc: Unexpected call sequence at 00000000de85f044: 48003d1d 7c0802a6
> [ 56.870472][ T3454] ------------[ cut here ]------------
> [ 56.870500][ T3454] WARNING: CPU: 52 PID: 3454 at kernel/trace/ftrace.c:2026 ftrace_bug+0x104/0x310
> [ 56.870527][ T3454] Modules linked in: kvm_hv kvm ses enclosure scsi_transport_sas ip_tables x_tables xfs sd_mod i40e firmware_class aacraid dm_mirror dm_region_hash dm_log dm_mod
> [ 56.870592][ T3454] CPU: 52 PID: 3454 Comm: nip.sh Not tainted 5.7.0-rc1-next-20200416 #4
> [ 56.870627][ T3454] NIP: c0000000002a3ae4 LR: c0000000002a47fc CTR: c0000000002436f0
> [ 56.870661][ T3454] REGS: c00000069a9ef710 TRAP: 0700 Not tainted (5.7.0-rc1-next-20200416)
> [ 56.870697][ T3454] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28228222 XER: 00000000
> [ 56.870748][ T3454] CFAR: c0000000002a3a2c IRQMASK: 0
> [ 56.870748][ T3454] GPR00: c0000000002a47fc c00000069a9ef9a0 c0000000012f9000 ffffffffffffffea
> [ 56.870748][ T3454] GPR04: c0002004e2160438 c0000007fedf0ad8 00000000614ca19d 0000000000000007
> [ 56.870748][ T3454] GPR08: 0000000000000003 0000000000000000 0000000000000000 0000000000000002
> [ 56.870748][ T3454] GPR12: 0000000000004000 c0000007fffd5600 0000000040000000 0000000139ae9798
> [ 56.870748][ T3454] GPR16: 0000000139ae9724 0000000139a86968 0000000139a1f230 0000000139aed568
> [ 56.870748][ T3454] GPR20: 00000001402af8b0 0000000000000009 0000000139a996e8 00007fffc9186d94
> [ 56.870748][ T3454] GPR24: 0000000000000000 c00000069a9efc00 c00000000132cd00 c00000069a9efc40
> [ 56.870748][ T3454] GPR28: c0000000011c29e8 0000000000000001 c0002004e2160438 c008000009321a64
> [ 56.870969][ T3454] NIP [c0000000002a3ae4] ftrace_bug+0x104/0x310
> ftrace_bug at kernel/trace/ftrace.c:2026
> [ 56.870995][ T3454] LR [c0000000002a47fc] ftrace_modify_all_code+0x16c/0x210
> ftrace_modify_all_code at kernel/trace/ftrace.c:2672
> [ 56.871034][ T3454] Call Trace:
> [ 56.871057][ T3454] [c00000069a9ef9a0] [4bffff899a9efa00] 0x4bffff899a9efa00 (unreliable)
> [ 56.871086][ T3454] [c00000069a9efa20] [c0000000002a47fc] ftrace_modify_all_code+0x16c/0x210
> [ 56.871125][ T3454] [c00000069a9efa50] [c000000000061b68] arch_ftrace_update_code+0x18/0x30
> [ 56.871162][ T3454] [c00000069a9efa70] [c0000000002a49c4] ftrace_run_update_code+0x44/0xc0
> [ 56.871199][ T3454] [c00000069a9efaa0] [c0000000002aa3c8] ftrace_startup+0xe8/0x1b0
> [ 56.871236][ T3454] [c00000069a9efae0] [c0000000002aa4e0] register_ftrace_function+0x50/0xc0
> [ 56.871275][ T3454] [c00000069a9efb10] [c0000000002d0468] function_trace_init+0x98/0xd0
> [ 56.871312][ T3454] [c00000069a9efb40] [c0000000002c75c0] tracing_set_tracer+0x350/0x640
> [ 56.871349][ T3454] [c00000069a9efbe0] [c0000000002c7a90] tracing_set_trace_write+0x1e0/0x370
> [ 56.871388][ T3454] [c00000069a9efd00] [c00000000052094c] __vfs_write+0x3c/0x70
> [ 56.871424][ T3454] [c00000069a9efd20] [c000000000523d4c] vfs_write+0xcc/0x200
> [ 56.871461][ T3454] [c00000069a9efd70] [c0000000005240ec] ksys_write+0x7c/0x140
> [ 56.871498][ T3454] [c00000069a9efdc0] [c000000000038a94] system_call_exception+0x114/0x1e0
> [ 56.871535][ T3454] [c00000069a9efe20] [c00000000000c870] system_call_common+0xf0/0x278
> [ 56.871570][ T3454] Instruction dump:
> [ 56.871592][ T3454] 7d908120 4e800020 60000000 2b890001 409effd4 3c62ff8b 38631958 4bf4491d
> [ 56.871639][ T3454] 60000000 4bffffc0 60000000 fba10068 <0fe00000> 39000001 3ce20003 3d22fed7
> [ 56.871685][ T3454] irq event stamp: 95388
> [ 56.871708][ T3454] hardirqs last enabled at (95387): [<c0000000001e4f94>] console_unlock+0x6a4/0x950
> [ 56.871746][ T3454] hardirqs last disabled at (95388): [<c00000000000960c>] program_check_common_virt+0x2bc/0x310
> [ 56.871785][ T3454] softirqs last enabled at (91222): [<c000000000a213c8>] __do_softirq+0x658/0x8d8
> [ 56.871823][ T3454] softirqs last disabled at (91215): [<c00000000011b40c>] irq_exit+0x16c/0x1d0
> [ 56.871859][ T3454] ---[ end trace 48f8445450a4e206 ]---
> [ 56.871907][ T3454] ftrace failed to modify
> [ 56.871913][ T3454] [<c008000009321a64>] show_sas_rphy_phy_identifier+0xc/0x60 [scsi_transport_sas]
> show_sas_rphy_phy_identifier at drivers/scsi/scsi_transport_sas.c:1221
> [ 56.871969][ T3454] actual: 1d:3d:00:48
> [ 56.871996][ T3454] Setting ftrace call site to call ftrace function
> [ 56.872020][ T3454] ftrace record flags: 80000001
> [ 56.872054][ T3454] (1)
> [ 56.872054][ T3454] expected tramp: c000000000061fac
Thanks. That confirms the problem.
In this particular case, we are unable to convert a module function to
call into ftrace_caller() since the _mcount() entry has not been nop-ed
out during module load. I suspect there might be another error/warning
earlier on indicating that failure. There are a few scenarios where we
are not propagating errors from patch_instruction() properly, so it is
possible that no errors were thrown previously. I will send a separate
set of patches to address that.
The reason for the original crash is a bit more subtle. On module load,
we patch _mcount() call sites of all the module functions with a nop.
However, with STRICT_KERNEL_RWX, one of these is failing. When that
happens, we end up continuing to call into _mcount(), which uses the
default module relocation stub, and not the special ftrace stub. The
default stub uses r2, which won't always be loaded with the module TOC
pointer with -mprofile-kernel. Instead, r2 likely points to the kernel
TOC, so we end up loading incorrect entries from the kernel TOC to
branch to. In all the traces we've seen so far, a kernel function has
called into a module function, wherein the module function does not set
up its own TOC (ext4_iomap_end(), find_free_cb()).
The core of the problem though is that patch_instruction() is failing in
certain scenarios. We need to find out why that is, and address that.
Russell,
You mentioned in the past that you identified an issue with
patch_instruction() during early boot:
http://lkml.kernel.org/r/c336400d5b7765eb72b3090cd9f8a3c57761d0b6.camel@russell.cc
Does that failure happen without STRICT_MODULE_RWX, with just
STRICT_KERNEL_RWX? That might be relevant here.
- Naveen
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 7:01 ` Naveen N. Rao
2020-04-17 11:49 ` Michael Ellerman
@ 2020-04-17 12:00 ` Qian Cai
2020-04-17 15:18 ` Naveen N. Rao
1 sibling, 1 reply; 12+ messages in thread
From: Qian Cai @ 2020-04-17 12:00 UTC (permalink / raw)
To: Naveen N. Rao
Cc: Michael Ellerman, Russell Currey, LKML, linuxppc-dev,
Nicholas Piggin, Steven Rostedt
> On Apr 17, 2020, at 3:01 AM, Naveen N. Rao <naveen.n.rao@linux.ibm.com> wrote:
>
> Hi Qian,
>
> Qian Cai wrote:
>> OK, reverted the commit,
>> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
>> https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
>
> Do you see any errors logged in dmesg when you see the crash? STRICT_KERNEL_RWX changes how patch_instruction() works, so it would be interesting to see if there are any ftrace-related errors thrown before the crash.
Yes, looks like there is a warning right after,
echo function > /sys/kernel/debug/tracing/current_tracer
echo nop > /sys/kernel/debug/tracing/current_tracer
and just before the crash,
[ T3454] ftrace-powerpc: Unexpected call sequence at 00000000de85f044: 48003d1d 7c0802a6
[ 56.870472][ T3454] ------------[ cut here ]------------
[ 56.870500][ T3454] WARNING: CPU: 52 PID: 3454 at kernel/trace/ftrace.c:2026 ftrace_bug+0x104/0x310
[ 56.870527][ T3454] Modules linked in: kvm_hv kvm ses enclosure scsi_transport_sas ip_tables x_tables xfs sd_mod i40e firmware_class aacraid dm_mirror dm_region_hash dm_log dm_mod
[ 56.870592][ T3454] CPU: 52 PID: 3454 Comm: nip.sh Not tainted 5.7.0-rc1-next-20200416 #4
[ 56.870627][ T3454] NIP: c0000000002a3ae4 LR: c0000000002a47fc CTR: c0000000002436f0
[ 56.870661][ T3454] REGS: c00000069a9ef710 TRAP: 0700 Not tainted (5.7.0-rc1-next-20200416)
[ 56.870697][ T3454] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28228222 XER: 00000000
[ 56.870748][ T3454] CFAR: c0000000002a3a2c IRQMASK: 0
[ 56.870748][ T3454] GPR00: c0000000002a47fc c00000069a9ef9a0 c0000000012f9000 ffffffffffffffea
[ 56.870748][ T3454] GPR04: c0002004e2160438 c0000007fedf0ad8 00000000614ca19d 0000000000000007
[ 56.870748][ T3454] GPR08: 0000000000000003 0000000000000000 0000000000000000 0000000000000002
[ 56.870748][ T3454] GPR12: 0000000000004000 c0000007fffd5600 0000000040000000 0000000139ae9798
[ 56.870748][ T3454] GPR16: 0000000139ae9724 0000000139a86968 0000000139a1f230 0000000139aed568
[ 56.870748][ T3454] GPR20: 00000001402af8b0 0000000000000009 0000000139a996e8 00007fffc9186d94
[ 56.870748][ T3454] GPR24: 0000000000000000 c00000069a9efc00 c00000000132cd00 c00000069a9efc40
[ 56.870748][ T3454] GPR28: c0000000011c29e8 0000000000000001 c0002004e2160438 c008000009321a64
[ 56.870969][ T3454] NIP [c0000000002a3ae4] ftrace_bug+0x104/0x310
ftrace_bug at kernel/trace/ftrace.c:2026
[ 56.870995][ T3454] LR [c0000000002a47fc] ftrace_modify_all_code+0x16c/0x210
ftrace_modify_all_code at kernel/trace/ftrace.c:2672
[ 56.871034][ T3454] Call Trace:
[ 56.871057][ T3454] [c00000069a9ef9a0] [4bffff899a9efa00] 0x4bffff899a9efa00 (unreliable)
[ 56.871086][ T3454] [c00000069a9efa20] [c0000000002a47fc] ftrace_modify_all_code+0x16c/0x210
[ 56.871125][ T3454] [c00000069a9efa50] [c000000000061b68] arch_ftrace_update_code+0x18/0x30
[ 56.871162][ T3454] [c00000069a9efa70] [c0000000002a49c4] ftrace_run_update_code+0x44/0xc0
[ 56.871199][ T3454] [c00000069a9efaa0] [c0000000002aa3c8] ftrace_startup+0xe8/0x1b0
[ 56.871236][ T3454] [c00000069a9efae0] [c0000000002aa4e0] register_ftrace_function+0x50/0xc0
[ 56.871275][ T3454] [c00000069a9efb10] [c0000000002d0468] function_trace_init+0x98/0xd0
[ 56.871312][ T3454] [c00000069a9efb40] [c0000000002c75c0] tracing_set_tracer+0x350/0x640
[ 56.871349][ T3454] [c00000069a9efbe0] [c0000000002c7a90] tracing_set_trace_write+0x1e0/0x370
[ 56.871388][ T3454] [c00000069a9efd00] [c00000000052094c] __vfs_write+0x3c/0x70
[ 56.871424][ T3454] [c00000069a9efd20] [c000000000523d4c] vfs_write+0xcc/0x200
[ 56.871461][ T3454] [c00000069a9efd70] [c0000000005240ec] ksys_write+0x7c/0x140
[ 56.871498][ T3454] [c00000069a9efdc0] [c000000000038a94] system_call_exception+0x114/0x1e0
[ 56.871535][ T3454] [c00000069a9efe20] [c00000000000c870] system_call_common+0xf0/0x278
[ 56.871570][ T3454] Instruction dump:
[ 56.871592][ T3454] 7d908120 4e800020 60000000 2b890001 409effd4 3c62ff8b 38631958 4bf4491d
[ 56.871639][ T3454] 60000000 4bffffc0 60000000 fba10068 <0fe00000> 39000001 3ce20003 3d22fed7
[ 56.871685][ T3454] irq event stamp: 95388
[ 56.871708][ T3454] hardirqs last enabled at (95387): [<c0000000001e4f94>] console_unlock+0x6a4/0x950
[ 56.871746][ T3454] hardirqs last disabled at (95388): [<c00000000000960c>] program_check_common_virt+0x2bc/0x310
[ 56.871785][ T3454] softirqs last enabled at (91222): [<c000000000a213c8>] __do_softirq+0x658/0x8d8
[ 56.871823][ T3454] softirqs last disabled at (91215): [<c00000000011b40c>] irq_exit+0x16c/0x1d0
[ 56.871859][ T3454] ---[ end trace 48f8445450a4e206 ]---
[ 56.871907][ T3454] ftrace failed to modify
[ 56.871913][ T3454] [<c008000009321a64>] show_sas_rphy_phy_identifier+0xc/0x60 [scsi_transport_sas]
show_sas_rphy_phy_identifier at drivers/scsi/scsi_transport_sas.c:1221
[ 56.871969][ T3454] actual: 1d:3d:00:48
[ 56.871996][ T3454] Setting ftrace call site to call ftrace function
[ 56.872020][ T3454] ftrace record flags: 80000001
[ 56.872054][ T3454] (1)
[ 56.872054][ T3454] expected tramp: c000000000061fac
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 7:01 ` Naveen N. Rao
@ 2020-04-17 11:49 ` Michael Ellerman
2020-04-17 12:00 ` Qian Cai
1 sibling, 0 replies; 12+ messages in thread
From: Michael Ellerman @ 2020-04-17 11:49 UTC (permalink / raw)
To: Naveen N. Rao, Qian Cai, Russell Currey
Cc: LKML, linuxppc-dev, Nicholas Piggin, Steven Rostedt
"Naveen N. Rao" <naveen.n.rao@linux.ibm.com> writes:
> Hi Qian,
>
> Qian Cai wrote:
>> OK, reverted the commit,
>>
>> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>>
>> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
>>
>> https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
>
> Do you see any errors logged in dmesg when you see the crash?
> STRICT_KERNEL_RWX changes how patch_instruction() works, so it would be
> interesting to see if there are any ftrace-related errors thrown before
> the crash.
I've been able to reproduce with STRICT_KERNEL_RWX=y and concurrently
running:
# while true; do echo function > /sys/kernel/debug/tracing/current_tracer ; echo nop > /sys/kernel/debug/tracing/current_tracer ; done
and:
# while true; do find /lib/modules/$(uname -r) -name '*.ko' -printf "%f\n" | sed -e "s/\.ko//" | xargs -i modprobe -va {}; lsmod | awk '{print $1}' | xargs -i modprobe -vr {}; done
ie. stressing module loading/unloading and ftrace at the same time.
It's not 100% but it usually reproduces within 10-20 minutes.
It looks like sometimes our __patch_instruction() fails, and then that
somehow leads to things getting further messed up. Presumably we have
some bad error handling somewhere.
cheers
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 2:17 ` Steven Rostedt
2020-04-17 2:27 ` Russell Currey
@ 2020-04-17 11:45 ` Michael Ellerman
1 sibling, 0 replies; 12+ messages in thread
From: Michael Ellerman @ 2020-04-17 11:45 UTC (permalink / raw)
To: Steven Rostedt, Qian Cai
Cc: Russell Currey, Nicholas Piggin, linuxppc-dev, LKML
Steven Rostedt <rostedt@goodmis.org> writes:
> On Thu, 16 Apr 2020 21:19:10 -0400
> Qian Cai <cai@lca.pw> wrote:
>
>> OK, reverted the commit,
>>
>> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>>
>> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
>
> This may be a symptom and not a cure.
I think it is a cure.
But we still have a bug, which is that when STRICT_KERNEL_RWX is enabled
we have some sort of corruption going on.
Enabling STRICT_KERNEL_RWX changes our implementation of
patch_instruction() which is used by ftrace, so I suspect this is a
powerpc bug.
>> [ 148.110969][T13115] LTP: starting chown04_16
>> [ 148.255048][T13380] kernel tried to execute exec-protected page (c0000000016804ac) - exploit attempt? (uid: 0)
>> [ 148.255099][T13380] BUG: Unable to handle kernel instruction fetch
>> [ 148.255122][T13380] Faulting instruction address: 0xc0000000016804ac
>> [ 148.255136][T13380] Oops: Kernel access of bad area, sig: 11 [#1]
>> [ 148.255157][T13380] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256 DEBUG_PAGEALLOC NUMA PowerNV
>> [ 148.255171][T13380] Modules linked in: loop kvm_hv kvm xfs sd_mod bnx2x mdio ahci tg3 libahci libphy libata firmware_class dm_mirror dm_region_hash dm_log dm_mod
>> [ 148.255213][T13380] CPU: 45 PID: 13380 Comm: chown04_16 Tainted: G W 5.6.0+ #7
>> [ 148.255236][T13380] NIP: c0000000016804ac LR: c00800000fa60408 CTR: c0000000016804ac
>> [ 148.255250][T13380] REGS: c0000010a6fafa00 TRAP: 0400 Tainted: G W (5.6.0+)
>> [ 148.255281][T13380] MSR: 9000000010009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
>> [ 148.255310][T13380] CFAR: c00800000fa66534 IRQMASK: 0
>> [ 148.255310][T13380] GPR00: c000000000973268 c0000010a6fafc90 c000000001648200 0000000000000000
>> [ 148.255310][T13380] GPR04: c000000d8a22dc00 c0000010a6fafd30 00000000b5e98331 ffffffff00012c9f
>> [ 148.255310][T13380] GPR08: c000000d8a22dc00 0000000000000000 0000000000000000 c00000000163c520
>> [ 148.255310][T13380] GPR12: c0000000016804ac c000001ffffdad80 0000000000000000 0000000000000000
>> [ 148.255310][T13380] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 148.255310][T13380] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 148.255310][T13380] GPR24: 00007fff8f5e2e48 0000000000000000 c00800000fa6a488 c0000010a6fafd30
>> [ 148.255310][T13380] GPR28: 0000000000000000 000000007fffffff c00800000fa60400 c000000efd0c6780
>> [ 148.255494][T13380] NIP [c0000000016804ac] sysctl_net_busy_read+0x0/0x4
>
> The instruction pointer is on sysctl_net_busy_read? Isn't that data and
> not code?
Yes.
But we're corrupting the text, or data, somewhere, so we can jump
anywhere.
I have another trace where vhost_init() appears to call into
proc_dointvec() before crashing. vhost_init() is an empty function.
cheers
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 1:19 ` POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...) Qian Cai
2020-04-17 2:17 ` Steven Rostedt
@ 2020-04-17 7:01 ` Naveen N. Rao
2020-04-17 11:49 ` Michael Ellerman
2020-04-17 12:00 ` Qian Cai
1 sibling, 2 replies; 12+ messages in thread
From: Naveen N. Rao @ 2020-04-17 7:01 UTC (permalink / raw)
To: Qian Cai, Michael Ellerman, Russell Currey
Cc: LKML, linuxppc-dev, Nicholas Piggin, Steven Rostedt
Hi Qian,
Qian Cai wrote:
> OK, reverted the commit,
>
> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>
> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
>
> https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
Do you see any errors logged in dmesg when you see the crash?
STRICT_KERNEL_RWX changes how patch_instruction() works, so it would be
interesting to see if there are any ftrace-related errors thrown before
the crash.
- Naveen
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 2:17 ` Steven Rostedt
@ 2020-04-17 2:27 ` Russell Currey
2020-04-17 11:45 ` Michael Ellerman
1 sibling, 0 replies; 12+ messages in thread
From: Russell Currey @ 2020-04-17 2:27 UTC (permalink / raw)
To: Steven Rostedt, Qian Cai
Cc: Michael Ellerman, Nicholas Piggin, linuxppc-dev, LKML
On Thu, 2020-04-16 at 22:17 -0400, Steven Rostedt wrote:
> On Thu, 16 Apr 2020 21:19:10 -0400
> Qian Cai <cai@lca.pw> wrote:
>
> > OK, reverted the commit,
> >
> > c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility
> > with RELOCATABLE”)
> >
> > or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned
> > in this thread,
>
> This may be a symptom and not a cure.
Reverting the patch with the given config will have the same effect as
STRICT_KERNEL_RWX=n. Not discounting that it could be a bug on the
powerpc side (i.e. relocatable kernels with strict RWX on haven't been
exhaustively tested yet), but we should definitely figure out what's
going on with this bad access first.
>
> > https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
> >
> > [ 148.110969][T13115] LTP: starting chown04_16
> > [ 148.255048][T13380] kernel tried to execute exec-protected page
> > (c0000000016804ac) - exploit attempt? (uid: 0)
> > [ 148.255099][T13380] BUG: Unable to handle kernel instruction
> > fetch
> > [ 148.255122][T13380] Faulting instruction address:
> > 0xc0000000016804ac
> > [ 148.255136][T13380] Oops: Kernel access of bad area, sig: 11
> > [#1]
> > [ 148.255157][T13380] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256
> > DEBUG_PAGEALLOC NUMA PowerNV
> > [ 148.255171][T13380] Modules linked in: loop kvm_hv kvm xfs
> > sd_mod bnx2x mdio ahci tg3 libahci libphy libata firmware_class
> > dm_mirror dm_region_hash dm_log dm_mod
> > [ 148.255213][T13380] CPU: 45 PID: 13380 Comm: chown04_16 Tainted:
> > G W 5.6.0+ #7
> > [ 148.255236][T13380] NIP: c0000000016804ac LR: c00800000fa60408
> > CTR: c0000000016804ac
> > [ 148.255250][T13380] REGS: c0000010a6fafa00 TRAP: 0400 Tainted:
> > G W (5.6.0+)
> > [ 148.255281][T13380] MSR: 9000000010009033
> > <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
> > [ 148.255310][T13380] CFAR: c00800000fa66534 IRQMASK: 0
> > [ 148.255310][T13380] GPR00: c000000000973268 c0000010a6fafc90
> > c000000001648200 0000000000000000
> > [ 148.255310][T13380] GPR04: c000000d8a22dc00 c0000010a6fafd30
> > 00000000b5e98331 ffffffff00012c9f
> > [ 148.255310][T13380] GPR08: c000000d8a22dc00 0000000000000000
> > 0000000000000000 c00000000163c520
> > [ 148.255310][T13380] GPR12: c0000000016804ac c000001ffffdad80
> > 0000000000000000 0000000000000000
> > [ 148.255310][T13380] GPR16: 0000000000000000 0000000000000000
> > 0000000000000000 0000000000000000
> > [ 148.255310][T13380] GPR20: 0000000000000000 0000000000000000
> > 0000000000000000 0000000000000000
> > [ 148.255310][T13380] GPR24: 00007fff8f5e2e48 0000000000000000
> > c00800000fa6a488 c0000010a6fafd30
> > [ 148.255310][T13380] GPR28: 0000000000000000 000000007fffffff
> > c00800000fa60400 c000000efd0c6780
> > [ 148.255494][T13380] NIP [c0000000016804ac]
> > sysctl_net_busy_read+0x0/0x4
>
> The instruction pointer is on sysctl_net_busy_read? Isn't that data
> and
> not code?
>
> In net/socket.c:
>
> #ifdef CONFIG_NET_RX_BUSY_POLL
> unsigned int sysctl_net_busy_read __read_mostly;
> unsigned int sysctl_net_busy_poll __read_mostly;
> #endif
>
> -- Steve
>
>
> > [ 148.255516][T13380] LR [c00800000fa60408] find_free_cb+0x8/0x30
> > [loop]
> > [ 148.255528][T13380] Call Trace:
> > [ 148.255538][T13380] [c0000010a6fafc90] [c0000000009732c0]
> > idr_for_each+0xf0/0x170 (unreliable)
> > [ 148.255572][T13380] [c0000010a6fafd10] [c00800000fa626c4]
> > loop_lookup.part.1+0x4c/0xb0 [loop]
> > [ 148.255597][T13380] [c0000010a6fafd50] [c00800000fa634d8]
> > loop_control_ioctl+0x120/0x1d0 [loop]
> > [ 148.255623][T13380] [c0000010a6fafdb0] [c0000000004ddc08]
> > ksys_ioctl+0xd8/0x130
> > [ 148.255636][T13380] [c0000010a6fafe00] [c0000000004ddc88]
> > sys_ioctl+0x28/0x40
> > [ 148.255669][T13380] [c0000010a6fafe20] [c00000000000b378]
> > system_call+0x5c/0x68
> > [ 148.255699][T13380] Instruction dump:
> > [ 148.255718][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> > XXXXXXXX XXXXXXXX XXXXXXXX
> > [ 148.255744][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> > XXXXXXXX XXXXXXXX XXXXXXXX
> > [ 148.255772][T13380] ---[ end trace a5894a74208c22ec ]---
> > [ 148.576663][T13380]
> > [ 149.576765][T13380] Kernel panic - not syncing: Fatal exception
> >
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-17 1:19 ` POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...) Qian Cai
@ 2020-04-17 2:17 ` Steven Rostedt
2020-04-17 2:27 ` Russell Currey
2020-04-17 11:45 ` Michael Ellerman
2020-04-17 7:01 ` Naveen N. Rao
1 sibling, 2 replies; 12+ messages in thread
From: Steven Rostedt @ 2020-04-17 2:17 UTC (permalink / raw)
To: Qian Cai
Cc: Michael Ellerman, Russell Currey, Nicholas Piggin, linuxppc-dev, LKML
On Thu, 16 Apr 2020 21:19:10 -0400
Qian Cai <cai@lca.pw> wrote:
> OK, reverted the commit,
>
> c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
>
> or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
This may be a symptom and not a cure.
>
> https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
>
> [ 148.110969][T13115] LTP: starting chown04_16
> [ 148.255048][T13380] kernel tried to execute exec-protected page (c0000000016804ac) - exploit attempt? (uid: 0)
> [ 148.255099][T13380] BUG: Unable to handle kernel instruction fetch
> [ 148.255122][T13380] Faulting instruction address: 0xc0000000016804ac
> [ 148.255136][T13380] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 148.255157][T13380] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256 DEBUG_PAGEALLOC NUMA PowerNV
> [ 148.255171][T13380] Modules linked in: loop kvm_hv kvm xfs sd_mod bnx2x mdio ahci tg3 libahci libphy libata firmware_class dm_mirror dm_region_hash dm_log dm_mod
> [ 148.255213][T13380] CPU: 45 PID: 13380 Comm: chown04_16 Tainted: G W 5.6.0+ #7
> [ 148.255236][T13380] NIP: c0000000016804ac LR: c00800000fa60408 CTR: c0000000016804ac
> [ 148.255250][T13380] REGS: c0000010a6fafa00 TRAP: 0400 Tainted: G W (5.6.0+)
> [ 148.255281][T13380] MSR: 9000000010009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
> [ 148.255310][T13380] CFAR: c00800000fa66534 IRQMASK: 0
> [ 148.255310][T13380] GPR00: c000000000973268 c0000010a6fafc90 c000000001648200 0000000000000000
> [ 148.255310][T13380] GPR04: c000000d8a22dc00 c0000010a6fafd30 00000000b5e98331 ffffffff00012c9f
> [ 148.255310][T13380] GPR08: c000000d8a22dc00 0000000000000000 0000000000000000 c00000000163c520
> [ 148.255310][T13380] GPR12: c0000000016804ac c000001ffffdad80 0000000000000000 0000000000000000
> [ 148.255310][T13380] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 148.255310][T13380] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 148.255310][T13380] GPR24: 00007fff8f5e2e48 0000000000000000 c00800000fa6a488 c0000010a6fafd30
> [ 148.255310][T13380] GPR28: 0000000000000000 000000007fffffff c00800000fa60400 c000000efd0c6780
> [ 148.255494][T13380] NIP [c0000000016804ac] sysctl_net_busy_read+0x0/0x4
The instruction pointer is on sysctl_net_busy_read? Isn't that data and
not code?
In net/socket.c:
#ifdef CONFIG_NET_RX_BUSY_POLL
unsigned int sysctl_net_busy_read __read_mostly;
unsigned int sysctl_net_busy_poll __read_mostly;
#endif
-- Steve
> [ 148.255516][T13380] LR [c00800000fa60408] find_free_cb+0x8/0x30 [loop]
> [ 148.255528][T13380] Call Trace:
> [ 148.255538][T13380] [c0000010a6fafc90] [c0000000009732c0] idr_for_each+0xf0/0x170 (unreliable)
> [ 148.255572][T13380] [c0000010a6fafd10] [c00800000fa626c4] loop_lookup.part.1+0x4c/0xb0 [loop]
> [ 148.255597][T13380] [c0000010a6fafd50] [c00800000fa634d8] loop_control_ioctl+0x120/0x1d0 [loop]
> [ 148.255623][T13380] [c0000010a6fafdb0] [c0000000004ddc08] ksys_ioctl+0xd8/0x130
> [ 148.255636][T13380] [c0000010a6fafe00] [c0000000004ddc88] sys_ioctl+0x28/0x40
> [ 148.255669][T13380] [c0000010a6fafe20] [c00000000000b378] system_call+0x5c/0x68
> [ 148.255699][T13380] Instruction dump:
> [ 148.255718][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> [ 148.255744][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> [ 148.255772][T13380] ---[ end trace a5894a74208c22ec ]---
> [ 148.576663][T13380]
> [ 149.576765][T13380] Kernel panic - not syncing: Fatal exception
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...)
2020-04-15 15:57 ` Qian Cai
@ 2020-04-17 1:19 ` Qian Cai
2020-04-17 2:17 ` Steven Rostedt
2020-04-17 7:01 ` Naveen N. Rao
0 siblings, 2 replies; 12+ messages in thread
From: Qian Cai @ 2020-04-17 1:19 UTC (permalink / raw)
To: Michael Ellerman, Russell Currey
Cc: Nicholas Piggin, linuxppc-dev, LKML, Steven Rostedt
OK, reverted the commit,
c55d7b5e6426 (“powerpc: Remove STRICT_KERNEL_RWX incompatibility with RELOCATABLE”)
or set STRICT_KERNEL_RWX=n fixed the crash below and also mentioned in this thread,
https://lore.kernel.org/lkml/15AC5B0E-A221-4B8C-9039-FA96B8EF7C88@lca.pw/
[ 148.110969][T13115] LTP: starting chown04_16
[ 148.255048][T13380] kernel tried to execute exec-protected page (c0000000016804ac) - exploit attempt? (uid: 0)
[ 148.255099][T13380] BUG: Unable to handle kernel instruction fetch
[ 148.255122][T13380] Faulting instruction address: 0xc0000000016804ac
[ 148.255136][T13380] Oops: Kernel access of bad area, sig: 11 [#1]
[ 148.255157][T13380] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=256 DEBUG_PAGEALLOC NUMA PowerNV
[ 148.255171][T13380] Modules linked in: loop kvm_hv kvm xfs sd_mod bnx2x mdio ahci tg3 libahci libphy libata firmware_class dm_mirror dm_region_hash dm_log dm_mod
[ 148.255213][T13380] CPU: 45 PID: 13380 Comm: chown04_16 Tainted: G W 5.6.0+ #7
[ 148.255236][T13380] NIP: c0000000016804ac LR: c00800000fa60408 CTR: c0000000016804ac
[ 148.255250][T13380] REGS: c0000010a6fafa00 TRAP: 0400 Tainted: G W (5.6.0+)
[ 148.255281][T13380] MSR: 9000000010009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 84000248 XER: 20040000
[ 148.255310][T13380] CFAR: c00800000fa66534 IRQMASK: 0
[ 148.255310][T13380] GPR00: c000000000973268 c0000010a6fafc90 c000000001648200 0000000000000000
[ 148.255310][T13380] GPR04: c000000d8a22dc00 c0000010a6fafd30 00000000b5e98331 ffffffff00012c9f
[ 148.255310][T13380] GPR08: c000000d8a22dc00 0000000000000000 0000000000000000 c00000000163c520
[ 148.255310][T13380] GPR12: c0000000016804ac c000001ffffdad80 0000000000000000 0000000000000000
[ 148.255310][T13380] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 148.255310][T13380] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 148.255310][T13380] GPR24: 00007fff8f5e2e48 0000000000000000 c00800000fa6a488 c0000010a6fafd30
[ 148.255310][T13380] GPR28: 0000000000000000 000000007fffffff c00800000fa60400 c000000efd0c6780
[ 148.255494][T13380] NIP [c0000000016804ac] sysctl_net_busy_read+0x0/0x4
[ 148.255516][T13380] LR [c00800000fa60408] find_free_cb+0x8/0x30 [loop]
[ 148.255528][T13380] Call Trace:
[ 148.255538][T13380] [c0000010a6fafc90] [c0000000009732c0] idr_for_each+0xf0/0x170 (unreliable)
[ 148.255572][T13380] [c0000010a6fafd10] [c00800000fa626c4] loop_lookup.part.1+0x4c/0xb0 [loop]
[ 148.255597][T13380] [c0000010a6fafd50] [c00800000fa634d8] loop_control_ioctl+0x120/0x1d0 [loop]
[ 148.255623][T13380] [c0000010a6fafdb0] [c0000000004ddc08] ksys_ioctl+0xd8/0x130
[ 148.255636][T13380] [c0000010a6fafe00] [c0000000004ddc88] sys_ioctl+0x28/0x40
[ 148.255669][T13380] [c0000010a6fafe20] [c00000000000b378] system_call+0x5c/0x68
[ 148.255699][T13380] Instruction dump:
[ 148.255718][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[ 148.255744][T13380] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[ 148.255772][T13380] ---[ end trace a5894a74208c22ec ]---
[ 148.576663][T13380]
[ 149.576765][T13380] Kernel panic - not syncing: Fatal exception
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-04-17 15:18 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-17 2:40 POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...) Qian Cai
2020-04-17 2:46 ` Russell Currey
2020-04-17 3:16 ` Qian Cai
2020-04-17 11:49 ` Qian Cai
-- strict thread matches above, loose matches on Subject: below --
2020-04-07 2:28 Linux-next POWER9 NULL pointer NIP since 1st Apr Qian Cai
2020-04-07 12:42 ` Michael Ellerman
2020-04-07 13:01 ` Qian Cai
2020-04-07 13:30 ` Steven Rostedt
2020-04-09 10:06 ` Qian Cai
2020-04-09 14:14 ` Steven Rostedt
2020-04-10 19:20 ` Qian Cai
2020-04-15 15:57 ` Qian Cai
2020-04-17 1:19 ` POWER9 crash due to STRICT_KERNEL_RWX (WAS: Re: Linux-next POWER9 NULL pointer NIP...) Qian Cai
2020-04-17 2:17 ` Steven Rostedt
2020-04-17 2:27 ` Russell Currey
2020-04-17 11:45 ` Michael Ellerman
2020-04-17 7:01 ` Naveen N. Rao
2020-04-17 11:49 ` Michael Ellerman
2020-04-17 12:00 ` Qian Cai
2020-04-17 15:18 ` Naveen N. Rao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).