Re: watchdog: how to enable?

[Muni Sekhar <munisekharrms@gmail.com>]

On Mon, Nov 18, 2019 at 7:40 PM Guenter Roeck <linux@roeck-us.net> wrote:
>
> On 11/18/19 1:52 AM, Muni Sekhar wrote:
> > On Sun, Nov 17, 2019 at 3:12 AM Guenter Roeck <linux@roeck-us.net> wrote:
> >>
> >> On 11/16/19 10:34 AM, Muni Sekhar wrote:
> >>> On Sat, Nov 16, 2019 at 9:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>>>
> >>>> On 11/15/19 7:03 PM, Muni Sekhar wrote:
> >>>> [ ... ]
> >>>>>>
> >>>>>> Another possibility, of course, might be to enable a hardware watchdog
> >>>>>> in your system (assuming it supports one). I personally would not trust
> >>>>>> the NMI watchdog because to detect a system hang, after all, there are
> >>>>>> situations where even NMIs no longer work.
> >>>>>
> >>>>> >From dmesg , Is it possible to know whether my system supports
> >>>>> hardware watchdog or not?
> >>>>> I assume that my system supports the hardware watchdog , then how to
> >>>>> enable the hardware watchdog to debug the system freeze issues?
> >>>>>
> >>>>
> >>>> Hardware watchdog support really depends on the board type. Most PC
> >>>> mainboards support a watchdog in the Super-IO chip, but on some it is
> >>>> not wired correctly. On embedded boards it is often built into the SoC.
> >>>> The easiest way to see if you have a watchdog would be to check for the
> >>>> existence of /dev/watchdog. However, on a PC that would most likely
> >>>> not be there because the necessary module is not auto-loaded.
> >>>> If you tell us your board type, or better the Super-IO chip on the board,
> >>>> we might be able to help.
> >>>
> >>> I’m having two same configuration systems, in one system I installed
> >>> the Vanilla kernel and I see the /dev/watchdog and /dev/watchdog0
> >>> nodes. In other system I’m running with ubuntu distribution kernel,
> >>> but I don’t see any watchdog device node. So it looks like I need to
> >>> manually load the kernel module in distro kernel. Is there a way to
> >>> know what is the corresponding kernel module for  /dev/watchdog node?
> >>>
> >>> # ls -l /dev/watchdog*
> >>> crw------- 1 root root  10, 130 Nov 15 17:15 /dev/watchdog
> >>> crw------- 1 root root 248,   0 Nov 15 17:15 /dev/watchdog0
> >>>
> >>> # ps -ax | grep watchdog
> >>>     678 ?        S      0:00 [watchdogd]
> >>>
> >>> Regarding Super-IO chip, how to find out the Super-IO chip model?
> >>>
> >> You could try to run sensors-detect (from the "sensors" package).
> >>
> >> If you can boot a system with /dev/watchdog0, you should see the type
> >> in /sys/class/watchdog/watchdog0/identity.
> > I could not find the /sys/class/watchdog/watchdog0/identity and
> > /sys/class/watchdog/watchdog0/timeout files.
> > $ ls -l /sys/class/watchdog/watchdog0/
> > total 0
> > -r--r--r-- 1 root root 4096 Nov 18 15:12 dev
> > lrwxrwxrwx 1 root root    0 Nov 18 15:12 device -> ../../../iTCO_wdt.0.auto
> > drwxr-xr-x 2 root root    0 Nov 18 15:12 power
> > lrwxrwxrwx 1 root root    0 Nov 18 14:53 subsystem ->
> > ../../../../../../class/watchdog
> > -rw-r--r-- 1 root root 4096 Nov 18 14:53 uevent
> >
>
> Presumably CONFIG_WATCHDOG_SYSFS is not enabled in your configuration.
>
> >>
> >> Also, you can test if the watchdog works with "sudo cat /dev/watchdog",
> >> assuming the watchdog daemon is not running. The watchdog works if the
> >> system reboots after the watchdog times out (/sys/class/watchdog/watchdog0/timeout
> >> is the timeout in seconds).
> > sudo cat /dev/watchdog perfectly rebooted my system. I don't see
> > timeout node, how do I configure the timeout value?
>
> sudo apt-get install watchdog
> man watchdog
>
> should tell you. Alternatively, enable CONFIG_WATCHDOG_SYSFS.
>
> >>
> >>>>
> >>>> Note though that this won't help to debug the problem. A hardware
> >>>> watchdog resets the system. It helps to recover, but it is not intended
> >>>> to help with debugging.
> >>> How do I use the hardware watchdog to reset my system when system is
> >>> frozen? It helps me to collect the crashdump and finally helps me to
> >>> find the root cause for the system frozen issue.
> >>>
> >> There won't be a crashdump. It just hard-resets the system.
> > So is there any other solution to capture the crashdump or trigger
> > soft reboot once kernel is lockedup?
>
> Not that I know of. I suspect, though, that you either have a hard lockup
> where even NMI is non-operational, or NMI doesn't work in your system
> to start with.
>
> If you have nmi_watchdog=1 in your kernel command line, /proc/interrupts
> should show a non-zero number of NMI interrupts. Do you see that in your system ?

Yes, I see non-zero number. When it(NMI interrupt count) supposed to change?

$ cat /proc/interrupts | grep NMI
 NMI:       4129       4153       4192        183   Non-maskable interrupts

$ dmesg | grep NMI
[    0.402175] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.402199] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.402220] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.402242] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    4.636467] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    4.658289] | NMI testsuite:
[   13.863284] INFO: NMI handler (kgdb_nmi_handler) took too long to
run: 9.744 msecs

Also I enabled pstore\ramoops. While testing the hardware watchdog by
running 'sudo cat /dev/watchdog', I see that console dump updates
between next boot. I see the same behavior consistently.

$ cat /sys/fs/pstore/console-ramoops-0
[  293.462623] printk: console [pstore-1] enabled
[  293.471026] pstore: Registered ramoops as persistent store backend
[  293.477800] ramoops: using 0x100000@0x3ff00000, ecc: 16
[  315.461263] systemd-journald[1665]: Sent WATCHDOG=1 notification.
[  317.447791] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[  317.456616] watchdog: watchdog0: watchdog did not stop!
No errors detected

Now I installed the watchdog daemon and started that service before
the kernel locks up. On triggering few tests kernel locked up and
hardware watchdog triggered the reset, but in this case I don't see
console-ramoops-0 file. Only difference is , this time 'watchdog'
daemon triggered the hardware watchdog. Not sure why console dump not
updated in this scenario?


>
> Guenter



-- 
Thanks,
Sekhar