Re: bisected: ttyS panic on pa-risc

[Greg Kroah-Hartman <gregkh@linuxfoundation.org>]

[-- Attachment #1: Type: text/plain, Size: 4356 bytes --]

On Thu, Jan 10, 2019 at 05:54:50PM +0200, Meelis Roos wrote:
> My HP 9000 A500 (pa-risc architecture) paniced in 5.0-rc1. It happened after printing dmesg lines about ttyS and before moving on to scsi printk-s.
> I bisected it and the panic symptoms changed during that (some had backtrace, some had just panic).
> 
> This is one of the crashes I got:
> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> serial 0000:00:04.0: enabling device (0146 -> 0147)
> printk: console [ttyS0] disabled
> 
> 0000:00:04.0: ttyS0 at MMIO 0xfffffffff8000000 (irq = 21, base_baud = 115200) is a 16550A
> printk: console [ttyS0] enabled
> printk: console [ttyS0] enabled
> printk: bootconsole [ttyB0] disabled
> printk: bootconsole [ttyB0] disabled
> 0000:00:04.0: ttyS1 at MMIO 0xfffffffff8000008 (irq = 21, base_baud = 115200) is a 16550A
> 0000:00:04.0: ttyS2 at MMIO 0xfffffffff8000010 (irq = 21, base_baud = 115200) is a 16550A
> serial 0000:00:05.0: enabling device (0140 -> 0143)
> 0000:00:05.0: ttyS3 at MMIO 0xfffffffff8005000 (irq = 22, base_baud = 115200) is a 16550A
> Backtrace:
>  [<0000000040502268>] pciserial_init_ports+0x128/0x240
>  [<00000000405040b8>] pciserial_init_one+0x1e0/0x2f0
>  [<00000000404b2b8c>] pci_device_probe+0xfc/0x180
>  [<0000000040513958>] really_probe+0x268/0x3d0
>  [<0000000040513d28>] driver_probe_device+0xf8/0x100
>  [<0000000040513e54>] __driver_attach+0x124/0x130
>  [<0000000040510dc4>] bus_for_each_dev+0x9c/0xe8
>  [<0000000040513040>] driver_attach+0x28/0x38
>  [<00000000405128c0>] bus_a
> 
> Normal dmesg excerpt from working kernel before the problem:
> 
> [    6.746131] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> [    6.771772] serial 0000:00:04.0: enabling device (0146 -> 0147)
> [    6.792657] printk: console [ttyS0] disabled
> [    6.829825] 0000:00:04.0: ttyS0 at MMIO 0xfffffffff8000000 (irq = 21, base_baud = 115200) is a 16550A
> [    6.837151] printk: console [ttyS0] enabled
> [    6.877768] printk: bootconsole [ttyB0] disabled
> [    6.904352] 0000:00:04.0: ttyS1 at MMIO 0xfffffffff8000008 (irq = 21, base_baud = 115200) is a 16550A
> [    6.961051] 0000:00:04.0: ttyS2 at MMIO 0xfffffffff8000010 (irq = 21, base_baud = 115200) is a 16550A
> [    6.969881] serial 0000:00:05.0: enabling device (0000 -> 0003)
> [    7.004160] serial 0000:00:05.0: enabling SERR and PARITY (0003 -> 0143)
> [    7.030298] 0000:00:05.0: ttyS3 at MMIO 0xfffffffff8005000 (irq = 22, base_baud = 115200) is a 16550A
> [    7.041663] serial 0000:00:05.0: Couldn't register serial port 0, irq 22, type 2, error -28
> [    7.145456] sym53c8xx 0000:00:01.0: enabling device (0000 -> 0003)
> 
> 
> Bisection leads to this commit:
> 
> 6d7f677a2afa1c82d7fc7af7f9159cbffd5dc010 is the first bad commit
> commit 6d7f677a2afa1c82d7fc7af7f9159cbffd5dc010
> Author: Darwin Dingel <darwin.dingel@alliedtelesis.co.nz>
> Date:   Mon Dec 10 11:29:09 2018 +1300
> 
>     serial: 8250: Rate limit serial port rx interrupts during input overruns
> 
>     When a serial port gets faulty or gets flooded with inputs, its interrupt
>     handler starts to work double time to get the characters to the workqueue
>     for the tty layer to handle them. When this busy time on the serial/tty
>     subsystem happens during boot, where it is also busy on the userspace
>     trying to initialise, some processes can continuously get preempted
>     and will be on hold until the interrupts subside.
> 
>     The fix is to backoff on processing received characters for a specified
>     amount of time when an input overrun is seen (received a new character
>     before the previous one is processed). This only stops receive and will
>     continue to transmit characters to serial port. After the backoff period
>     is done, it receive will be re-enabled. This is optional and will only
>     be enabled by setting 'overrun-throttle-ms' in the dts.
> 
>     Signed-off-by: Darwin Dingel <darwin.dingel@alliedtelesis.co.nz>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> :040000 040000 4ea6cd68ededa0c9ffaa218668ffeb35557070a5 a011db1916fbf5cfdcfff836a81e4fb5ee737003 M      drivers
> :040000 040000 b1b1dc977965eb2db6b2cc79939446a1cf2f684d 41322ab1c199f504cfcc5b2ca211b4638d41351c M      include

The patch below was just applied to my tree, hopefully ti fixes this
issue.

thanks,

greg k-h

[-- Attachment #2: 0002-serial-8250-Fix-serial8250-initialization-crash.patch --]
[-- Type: text/plain, Size: 2660 bytes --]

From 352c4cf40c4a7d439fa5d30aa2160f54b394da82 Mon Sep 17 00:00:00 2001
From: He Zhe <zhe.he@windriver.com>
Date: Thu, 17 Jan 2019 17:00:19 +0800
Subject: serial: 8250: Fix serial8250 initialization crash

The initialization code of interrupt backoff work might reference NULL
pointer and cause the following crash, if no port was found.

[   10.017727] CPU 0 Unable to handle kernel paging request at virtual address 000001b0, epc == 807088e0, ra == 8070863c
---- snip ----
[   11.704470] [<807088e0>] serial8250_register_8250_port+0x318/0x4ac
[   11.747251] [<80708d74>] serial8250_probe+0x148/0x1c0
[   11.789301] [<80728450>] platform_drv_probe+0x40/0x94
[   11.830515] [<807264f8>] really_probe+0xf8/0x318
[   11.870876] [<80726b7c>] __driver_attach+0x110/0x12c
[   11.910960] [<80724374>] bus_for_each_dev+0x78/0xcc
[   11.951134] [<80725958>] bus_add_driver+0x200/0x234
[   11.989756] [<807273d8>] driver_register+0x84/0x148
[   12.029832] [<80d72f84>] serial8250_init+0x138/0x198
[   12.070447] [<80100e6c>] do_one_initcall+0x5c/0x2a0
[   12.110104] [<80d3a208>] kernel_init_freeable+0x370/0x484
[   12.150722] [<80a49420>] kernel_init+0x10/0xf8
[   12.191517] [<8010756c>] ret_from_kernel_thread+0x14/0x1c

This patch makes sure the initialization code can be reached only if a port
is found.

Fixes: 6d7f677a2afa ("serial: 8250: Rate limit serial port rx interrupts during input overruns")
Signed-off-by: He Zhe <zhe.he@windriver.com>
Reviewed-by: Darwin Dingel <darwin.dingel@alliedtelesis.co.nz>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 drivers/tty/serial/8250/8250_core.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c
index 189ab1212d9a..e441221e04b9 100644
--- a/drivers/tty/serial/8250/8250_core.c
+++ b/drivers/tty/serial/8250/8250_core.c
@@ -1070,15 +1070,16 @@ int serial8250_register_8250_port(struct uart_8250_port *up)
 
 			ret = 0;
 		}
-	}
 
-	/* Initialise interrupt backoff work if required */
-	if (up->overrun_backoff_time_ms > 0) {
-		uart->overrun_backoff_time_ms = up->overrun_backoff_time_ms;
-		INIT_DELAYED_WORK(&uart->overrun_backoff,
-				  serial_8250_overrun_backoff_work);
-	} else {
-		uart->overrun_backoff_time_ms = 0;
+		/* Initialise interrupt backoff work if required */
+		if (up->overrun_backoff_time_ms > 0) {
+			uart->overrun_backoff_time_ms =
+				up->overrun_backoff_time_ms;
+			INIT_DELAYED_WORK(&uart->overrun_backoff,
+					serial_8250_overrun_backoff_work);
+		} else {
+			uart->overrun_backoff_time_ms = 0;
+		}
 	}
 
 	mutex_unlock(&serial_mutex);
-- 
2.20.1