[PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Jisheng Zhang]

From: Jisheng Zhang <jszhang@kernel.org>

Current riscv's kprobe handlers are run with both preemption and
interrupt enabled, this violates kprobe requirements. Fix this issue
by keeping interrupts disabled for BREAKPOINT exception.

Fixes: c22b0bcb1dd0 ("riscv: Add kprobes supported")
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
---
 arch/riscv/kernel/entry.S | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index 744f3209c48d..4114b65698ec 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -130,6 +130,8 @@ skip_context_tracking:
 	 */
 	andi t0, s1, SR_PIE
 	beqz t0, 1f
+	li t0, EXC_BREAKPOINT
+	beq s4, t0, 1f
 #ifdef CONFIG_TRACE_IRQFLAGS
 	call trace_hardirqs_on
 #endif
-- 
2.31.0

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Masami Hiramatsu]

Hi Jisheng,

On Tue, 30 Mar 2021 02:16:24 +0800
Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:

> From: Jisheng Zhang <jszhang@kernel.org>
> 
> Current riscv's kprobe handlers are run with both preemption and
> interrupt enabled, this violates kprobe requirements. Fix this issue
> by keeping interrupts disabled for BREAKPOINT exception.

Not only while the breakpoint exception but also until the end of
the single step (maybe you are using __BUG_INSN_32 ??) need to be
disable interrupts. Can this do that?

Thank you,

> 
> Fixes: c22b0bcb1dd0 ("riscv: Add kprobes supported")
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> ---
>  arch/riscv/kernel/entry.S | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
> index 744f3209c48d..4114b65698ec 100644
> --- a/arch/riscv/kernel/entry.S
> +++ b/arch/riscv/kernel/entry.S
> @@ -130,6 +130,8 @@ skip_context_tracking:
>  	 */
>  	andi t0, s1, SR_PIE
>  	beqz t0, 1f
> +	li t0, EXC_BREAKPOINT
> +	beq s4, t0, 1f
>  #ifdef CONFIG_TRACE_IRQFLAGS
>  	call trace_hardirqs_on
>  #endif
> -- 
> 2.31.0
> 
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Jisheng Zhang]

On Tue, 30 Mar 2021 18:33:16 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> Hi Jisheng,

Hi Masami,

> 
> On Tue, 30 Mar 2021 02:16:24 +0800
> Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:
> 
> > From: Jisheng Zhang <jszhang@kernel.org>
> > 
> > Current riscv's kprobe handlers are run with both preemption and
> > interrupt enabled, this violates kprobe requirements. Fix this issue
> > by keeping interrupts disabled for BREAKPOINT exception.  
> 
> Not only while the breakpoint exception but also until the end of
> the single step (maybe you are using __BUG_INSN_32 ??) need to be
> disable interrupts. Can this do that?
> 

interrupt is disabled during "single step" by kprobes_save_local_irqflag()
and kprobes_restore_local_irqflag(). The code flow looks like: 

do_trap_break()   // for bp
  kprobe_breakpoint_handler()
    setup_singlestep()
      kprobes_restore_local_irqflag()

do_trap_break()  // for ss
  kprobe_single_step_handler()
    kprobes_restore_local_irqflag()

Thanks

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Masami Hiramatsu]

Hi,

On Wed, 31 Mar 2021 22:22:44 +0800
Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:

> On Tue, 30 Mar 2021 18:33:16 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > Hi Jisheng,
> 
> Hi Masami,
> 
> > 
> > On Tue, 30 Mar 2021 02:16:24 +0800
> > Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:
> > 
> > > From: Jisheng Zhang <jszhang@kernel.org>
> > > 
> > > Current riscv's kprobe handlers are run with both preemption and
> > > interrupt enabled, this violates kprobe requirements. Fix this issue
> > > by keeping interrupts disabled for BREAKPOINT exception.  
> > 
> > Not only while the breakpoint exception but also until the end of
> > the single step (maybe you are using __BUG_INSN_32 ??) need to be
> > disable interrupts. Can this do that?
> > 
> 
> interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> and kprobes_restore_local_irqflag(). The code flow looks like: 
> 
> do_trap_break()   // for bp
>   kprobe_breakpoint_handler()
>     setup_singlestep()
>       kprobes_restore_local_irqflag()
> 
> do_trap_break()  // for ss
>   kprobe_single_step_handler()
>     kprobes_restore_local_irqflag()

OK, thanks for the confirmation!

Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org>

Thanks,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Maciej W. Rozycki]

On Thu, 1 Apr 2021, Masami Hiramatsu wrote:

> > > > Current riscv's kprobe handlers are run with both preemption and
> > > > interrupt enabled, this violates kprobe requirements. Fix this issue
> > > > by keeping interrupts disabled for BREAKPOINT exception.  
> > > 
> > > Not only while the breakpoint exception but also until the end of
> > > the single step (maybe you are using __BUG_INSN_32 ??) need to be
> > > disable interrupts. Can this do that?
> > > 
> > 
> > interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> > and kprobes_restore_local_irqflag(). The code flow looks like: 
> > 
> > do_trap_break()   // for bp
> >   kprobe_breakpoint_handler()
> >     setup_singlestep()
> >       kprobes_restore_local_irqflag()
> > 
> > do_trap_break()  // for ss
> >   kprobe_single_step_handler()
> >     kprobes_restore_local_irqflag()
> 
> OK, thanks for the confirmation!

 Is this approach guaranteed to keep interrupt handling latency low enough 
for the system not to be negatively affected, e.g. for the purpose of NTP 
timekeeping?

  Maciej

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Jisheng Zhang]

On Sat, 3 Apr 2021 20:30:53 +0200 (CEST)
"Maciej W. Rozycki" <macro@orcam.me.uk> wrote:

> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> On Thu, 1 Apr 2021, Masami Hiramatsu wrote:
> 
> > > > > Current riscv's kprobe handlers are run with both preemption and
> > > > > interrupt enabled, this violates kprobe requirements. Fix this issue
> > > > > by keeping interrupts disabled for BREAKPOINT exception.  
> > > >
> > > > Not only while the breakpoint exception but also until the end of
> > > > the single step (maybe you are using __BUG_INSN_32 ??) need to be
> > > > disable interrupts. Can this do that?
> > > >  
> > >
> > > interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> > > and kprobes_restore_local_irqflag(). The code flow looks like:
> > >
> > > do_trap_break()   // for bp
> > >   kprobe_breakpoint_handler()
> > >     setup_singlestep()
> > >       kprobes_restore_local_irqflag()
> > >
> > > do_trap_break()  // for ss
> > >   kprobe_single_step_handler()
> > >     kprobes_restore_local_irqflag()  
> >
> > OK, thanks for the confirmation!  
> 
>  Is this approach guaranteed to keep interrupt handling latency low enough
> for the system not to be negatively affected, e.g. for the purpose of NTP
> timekeeping?

IMHO, interrupt latency can't be ensured if kprobes is triggered.

thanks

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Masami Hiramatsu]

On Thu, 8 Apr 2021 19:23:48 +0800
Jisheng Zhang <Jisheng.Zhang@synaptics.com> wrote:

> On Sat, 3 Apr 2021 20:30:53 +0200 (CEST)
> "Maciej W. Rozycki" <macro@orcam.me.uk> wrote:
> 
> > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> > 
> > 
> > On Thu, 1 Apr 2021, Masami Hiramatsu wrote:
> > 
> > > > > > Current riscv's kprobe handlers are run with both preemption and
> > > > > > interrupt enabled, this violates kprobe requirements. Fix this issue
> > > > > > by keeping interrupts disabled for BREAKPOINT exception.  
> > > > >
> > > > > Not only while the breakpoint exception but also until the end of
> > > > > the single step (maybe you are using __BUG_INSN_32 ??) need to be
> > > > > disable interrupts. Can this do that?
> > > > >  
> > > >
> > > > interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> > > > and kprobes_restore_local_irqflag(). The code flow looks like:
> > > >
> > > > do_trap_break()   // for bp
> > > >   kprobe_breakpoint_handler()
> > > >     setup_singlestep()
> > > >       kprobes_restore_local_irqflag()
> > > >
> > > > do_trap_break()  // for ss
> > > >   kprobe_single_step_handler()
> > > >     kprobes_restore_local_irqflag()  
> > >
> > > OK, thanks for the confirmation!  
> > 
> >  Is this approach guaranteed to keep interrupt handling latency low enough
> > for the system not to be negatively affected, e.g. for the purpose of NTP
> > timekeeping?
> 
> IMHO, interrupt latency can't be ensured if kprobes is triggered.

Indeed. The latency depends on what the kprobe user handler does. Of course
it must be as minimal as possible... On x86, it is less than a few microseconds.
Thus it may be a jitter on hard realtime system, but not a big issue on
usual system like NTP timekeeping.

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Maciej W. Rozycki]

On Fri, 9 Apr 2021, Masami Hiramatsu wrote:

> > >  Is this approach guaranteed to keep interrupt handling latency low enough
> > > for the system not to be negatively affected, e.g. for the purpose of NTP
> > > timekeeping?
> > 
> > IMHO, interrupt latency can't be ensured if kprobes is triggered.
> 
> Indeed. The latency depends on what the kprobe user handler does. Of course
> it must be as minimal as possible... On x86, it is less than a few microseconds.
> Thus it may be a jitter on hard realtime system, but not a big issue on
> usual system like NTP timekeeping.

 Ack.  Assuming that the breakpoint exception will only disable interrupts 
if kprobes are in use it seems reasonable to me.

 Thanks for double-checking.

  Maciej

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[liaochang (A)]

Hi Jisheng,

在 2021/3/31 22:22, Jisheng Zhang 写道:
> On Tue, 30 Mar 2021 18:33:16 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
>> Hi Jisheng,
> 
> Hi Masami,
> 
>>
>> On Tue, 30 Mar 2021 02:16:24 +0800
>> Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:
>>
>>> From: Jisheng Zhang <jszhang@kernel.org>
>>>
>>> Current riscv's kprobe handlers are run with both preemption and
>>> interrupt enabled, this violates kprobe requirements. Fix this issue
>>> by keeping interrupts disabled for BREAKPOINT exception.  
>>
>> Not only while the breakpoint exception but also until the end of
>> the single step (maybe you are using __BUG_INSN_32 ??) need to be
>> disable interrupts. Can this do that?
>>
> 
> interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> and kprobes_restore_local_irqflag(). The code flow looks like: 
> 
> do_trap_break()   // for bp
>   kprobe_breakpoint_handler()
>     setup_singlestep()
>       kprobes_restore_local_irqflag()
> 
> do_trap_break()  // for ss
>   kprobe_single_step_handler()
>     kprobes_restore_local_irqflag()

Recently, kernel hit BUG_ON() on QEMU after I install a probe at "sys_read" via kprobe,
accoriding to my debugging and analysis it looks like caused by the "irq disable" operation for single-stepping.

I present a detailed description about this problem in the mail with title "[PATCH] riscv/kprobe: fix kernel panic when invoking sys_read traced by kprobe".
Looking forward to some feedback,Thanks.

BR,
Liao Chang
> 
> Thanks
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
> .
> 

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Jisheng Zhang]

On Thu, 1 Apr 2021 16:49:47 +0800
"liaochang (A)" <liaochang1@huawei.com> wrote:

> Hi Jisheng,

Hi,

> 
> 在 2021/3/31 22:22, Jisheng Zhang 写道:
> > On Tue, 30 Mar 2021 18:33:16 +0900
> > Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >   
> >> Hi Jisheng,  
> > 
> > Hi Masami,
> >   
> >>
> >> On Tue, 30 Mar 2021 02:16:24 +0800
> >> Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:
> >>  
> >>> From: Jisheng Zhang <jszhang@kernel.org>
> >>>
> >>> Current riscv's kprobe handlers are run with both preemption and
> >>> interrupt enabled, this violates kprobe requirements. Fix this issue
> >>> by keeping interrupts disabled for BREAKPOINT exception.    
> >>
> >> Not only while the breakpoint exception but also until the end of
> >> the single step (maybe you are using __BUG_INSN_32 ??) need to be
> >> disable interrupts. Can this do that?
> >>  
> > 
> > interrupt is disabled during "single step" by kprobes_save_local_irqflag()
> > and kprobes_restore_local_irqflag(). The code flow looks like: 
> > 
> > do_trap_break()   // for bp
> >   kprobe_breakpoint_handler()
> >     setup_singlestep()
> >       kprobes_restore_local_irqflag()
> > 
> > do_trap_break()  // for ss
> >   kprobe_single_step_handler()
> >     kprobes_restore_local_irqflag()  
> 
> Recently, kernel hit BUG_ON() on QEMU after I install a probe at "sys_read" via kprobe,

TIPS: Each line should not exceed 80 chars

> accoriding to my debugging and analysis it looks like caused by the "irq disable" operation for single-stepping.
> 
> I present a detailed description about this problem in the mail with title "[PATCH] riscv/kprobe: fix kernel panic when invoking sys_read traced by kprobe".
> Looking forward to some feedback,Thanks.
> 

I will comment that patch.

thanks

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[liaochang (A)]

Hi Jisheng,

在 2021/4/2 21:32, Jisheng Zhang 写道:
> On Thu, 1 Apr 2021 16:49:47 +0800
> "liaochang (A)" <liaochang1@huawei.com> wrote:
> 
>> Hi Jisheng,
> 
> Hi,
> 
>>
>> 在 2021/3/31 22:22, Jisheng Zhang 写道:
>>> On Tue, 30 Mar 2021 18:33:16 +0900
>>> Masami Hiramatsu <mhiramat@kernel.org> wrote:
>>>   
>>>> Hi Jisheng,  
>>>
>>> Hi Masami,
>>>   
>>>>
>>>> On Tue, 30 Mar 2021 02:16:24 +0800
>>>> Jisheng Zhang <jszhang3@mail.ustc.edu.cn> wrote:
>>>>  
>>>>> From: Jisheng Zhang <jszhang@kernel.org>
>>>>>
>>>>> Current riscv's kprobe handlers are run with both preemption and
>>>>> interrupt enabled, this violates kprobe requirements. Fix this issue
>>>>> by keeping interrupts disabled for BREAKPOINT exception.    
>>>>
>>>> Not only while the breakpoint exception but also until the end of
>>>> the single step (maybe you are using __BUG_INSN_32 ??) need to be
>>>> disable interrupts. Can this do that?
>>>>  
>>>
>>> interrupt is disabled during "single step" by kprobes_save_local_irqflag()
>>> and kprobes_restore_local_irqflag(). The code flow looks like: 
>>>
>>> do_trap_break()   // for bp
>>>   kprobe_breakpoint_handler()
>>>     setup_singlestep()
>>>       kprobes_restore_local_irqflag()
>>>
>>> do_trap_break()  // for ss
>>>   kprobe_single_step_handler()
>>>     kprobes_restore_local_irqflag()  
>>
>> Recently, kernel hit BUG_ON() on QEMU after I install a probe at "sys_read" via kprobe,
> 
> TIPS: Each line should not exceed 80 chars
> 
>> accoriding to my debugging and analysis it looks like caused by the "irq disable" operation for single-stepping.
>>
>> I present a detailed description about this problem in the mail with title "[PATCH] riscv/kprobe: fix kernel panic when invoking sys_read traced by kprobe".
>> Looking forward to some feedback,Thanks.
>>
> 
> I will comment that patch.

Thanks for your reminder.

> 
> thanks
> 
> .
> 

Re: [PATCH] riscv: keep interrupts disabled for BREAKPOINT exception

[Palmer Dabbelt]

On Mon, 29 Mar 2021 11:16:24 PDT (-0700), jszhang3@mail.ustc.edu.cn wrote:
> From: Jisheng Zhang <jszhang@kernel.org>
>
> Current riscv's kprobe handlers are run with both preemption and
> interrupt enabled, this violates kprobe requirements. Fix this issue
> by keeping interrupts disabled for BREAKPOINT exception.
>
> Fixes: c22b0bcb1dd0 ("riscv: Add kprobes supported")
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> ---
>  arch/riscv/kernel/entry.S | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
> index 744f3209c48d..4114b65698ec 100644
> --- a/arch/riscv/kernel/entry.S
> +++ b/arch/riscv/kernel/entry.S
> @@ -130,6 +130,8 @@ skip_context_tracking:
>  	 */
>  	andi t0, s1, SR_PIE
>  	beqz t0, 1f
> +	li t0, EXC_BREAKPOINT
> +	beq s4, t0, 1f
>  #ifdef CONFIG_TRACE_IRQFLAGS
>  	call trace_hardirqs_on
>  #endif

This is on fixes, with a comment as otherwise I'm just going to forget 
why.