print active stack on watchdog pre timeout for separate irq stack

print active stack on watchdog pre timeout for separate irq stack

[Mehrtens, Hauke]

Hi,

We would like to print the stack of the currently active kernel thread from the interrupt handler when the watchdog pre timeout interrupt for our watchdog is triggered, currently we have a WARN_ONCE() in the code of the interrupt handler, but this only prints the interrupt stack, which is pretty boring. On MIPS the interrupts are handled on a separate stack and not on top of the stack of the current active kernel thread to avoid stack overflows. Is there some function which would print the stack trace of the current active kernel thread in addition or instead of the interrupt stack inside of an interrupt?

The kernel also has these pre timeout handlers, but they also seem to be affected by this problem:
https://elixir.bootlin.com/linux/v5.0-rc6/source/drivers/watchdog/pretimeout_panic.c

This was seen on kernel 4.9.109, but I am not aware of any changes in this area in the last few years. 

Hauke

Re: print active stack on watchdog pre timeout for separate irq stack

[Guenter Roeck]

On Thu, Feb 14, 2019 at 09:27:17AM +0000, Mehrtens, Hauke wrote:
> Hi,
> 
> We would like to print the stack of the currently active kernel thread from the interrupt handler when the watchdog pre timeout interrupt for our watchdog is triggered, currently we have a WARN_ONCE() in the code of the interrupt handler, but this only prints the interrupt stack, which is pretty boring. On MIPS the interrupts are handled on a separate stack and not on top of the stack of the current active kernel thread to avoid stack overflows. Is there some function which would print the stack trace of the current active kernel thread in addition or instead of the interrupt stack inside of an interrupt?
> 
> The kernel also has these pre timeout handlers, but they also seem to be affected by this problem:
> https://elixir.bootlin.com/linux/v5.0-rc6/source/drivers/watchdog/pretimeout_panic.c
> 
Not sure I understand. Are you saying that panic() doesn't help either ?
If that is correct, I have no idea what else could be done.

Guenter

Re: print active stack on watchdog pre timeout for separate irq stack

[Paul Burton]

Hi Hauke,

On Thu, Feb 14, 2019 at 09:27:17AM +0000, Mehrtens, Hauke wrote:
> We would like to print the stack of the currently active kernel thread
> from the interrupt handler when the watchdog pre timeout interrupt for
> our watchdog is triggered, currently we have a WARN_ONCE() in the code
> of the interrupt handler, but this only prints the interrupt stack,
> which is pretty boring. On MIPS the interrupts are handled on a
> separate stack and not on top of the stack of the current active
> kernel thread to avoid stack overflows. Is there some function which
> would print the stack trace of the current active kernel thread in
> addition or instead of the interrupt stack inside of an interrupt?
>%
> This was seen on kernel 4.9.109, but I am not aware of any changes in
> this area in the last few years.

What's meant to happen is that we unwind the interrupt stack, then once
we reach the end of that we jump over to the task stack & continue
unwinding there to obtain a complete stack trace.

That was added in v4.11 by commit db8466c581cc ("MIPS: IRQ Stack: Unwind
IRQ stack onto task stack"), and it looks like it was backported to v4.9
starting with v4.9.54 so you should have it already.

Could you take a look at the "if (unlikely(*sp == irq_stack_high))"
check in unwind_stack_by_address() & see whether you hit it?

Thanks,
    Paul