All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
@ 2021-09-30  3:11 Guanghui Feng
  2021-09-30  5:38 ` Greg KH
  0 siblings, 1 reply; 9+ messages in thread
From: Guanghui Feng @ 2021-09-30  3:11 UTC (permalink / raw)
  To: gregkh, jirislaby; +Cc: baolin.wang, linux-kernel

When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
which look like this one:

  watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
  CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
  Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
  Workqueue: events_unbound flush_to_ldisc
  pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
  pc : slip_unesc+0x80/0x214 [slip]
  lr : slip_receive_buf+0x84/0x100 [slip]
  sp : ffff80005274bce0
  x29: ffff80005274bce0 x28: 0000000000000000
  x27: ffff00525626fcc8 x26: ffff800011921078
  x25: 0000000000000000 x24: 0000000000000004
  x23: ffff00402b4059c0 x22: ffff00402b405940
  x21: ffff205d87b81e21 x20: ffff205d87b81b9b
  x19: 0000000000000000 x18: 0000000000000000
  x17: 0000000000000000 x16: 0000000000000000
  x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
  x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
  x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
  x9 : ffff8000097d7628 x8 : ffff205d87b85e20
  x7 : 0000000000000000 x6 : 0000000000000001
  x5 : ffff8000097dc008 x4 : ffff8000097d75a4
  x3 : ffff205d87b81e1f x2 : 0000000000000005
  x1 : 000000000000005f x0 : ffff00402b405940
  Call trace:
   slip_unesc+0x80/0x214 [slip]
   tty_ldisc_receive_buf+0x64/0x80
   tty_port_default_receive_buf+0x50/0x90
   flush_to_ldisc+0xbc/0x110
   process_one_work+0x1d4/0x4b0
   worker_thread+0x180/0x430
   kthread+0x11c/0x120
  Kernel panic - not syncing: softlockup: hung tasks
  CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
  Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
  Workqueue: events_unbound flush_to_ldisc
  Call trace:
   dump_backtrace+0x0/0x1ec
   show_stack+0x24/0x30
   dump_stack+0xd0/0x128
   panic+0x15c/0x374
   watchdog_timer_fn+0x2b8/0x304
   __run_hrtimer+0x88/0x2c0
   __hrtimer_run_queues+0xa4/0x120
   hrtimer_interrupt+0xfc/0x270
   arch_timer_handler_phys+0x40/0x50
   handle_percpu_devid_irq+0x94/0x220
   __handle_domain_irq+0x88/0xf0
   gic_handle_irq+0x84/0xfc
   el1_irq+0xc8/0x180
   slip_unesc+0x80/0x214 [slip]
   tty_ldisc_receive_buf+0x64/0x80
   tty_port_default_receive_buf+0x50/0x90
   flush_to_ldisc+0xbc/0x110
   process_one_work+0x1d4/0x4b0
   worker_thread+0x180/0x430
   kthread+0x11c/0x120
  SMP: stopping secondary CPUs

In the testcase pty04, there are multple processes and we only pay close attention to the
first three actually. The first process call the write syscall to send data to the pty master
with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
The third process will wait a moment in which the first two processes will do there work and then
it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
master's buffhead to line discipline in a loop until there is no more data left without any on one's
own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
in the flush_to_ldisc while loop to avoid it.

Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
---
 drivers/tty/tty_buffer.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
index bd2d915..77b92f9 100644
--- a/drivers/tty/tty_buffer.c
+++ b/drivers/tty/tty_buffer.c
@@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
 		if (!count)
 			break;
 		head->read += count;
+		cond_resched();
 	}
 
 	mutex_unlock(&buf->lock);
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-09-30  3:11 [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc Guanghui Feng
@ 2021-09-30  5:38 ` Greg KH
  2021-09-30  6:02   ` Baolin Wang
  2021-10-08  7:50   ` guanghui.fgh
  0 siblings, 2 replies; 9+ messages in thread
From: Greg KH @ 2021-09-30  5:38 UTC (permalink / raw)
  To: Guanghui Feng; +Cc: jirislaby, baolin.wang, linux-kernel

On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> which look like this one:
> 
>   watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
>   CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
>   Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>   Workqueue: events_unbound flush_to_ldisc
>   pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
>   pc : slip_unesc+0x80/0x214 [slip]
>   lr : slip_receive_buf+0x84/0x100 [slip]
>   sp : ffff80005274bce0
>   x29: ffff80005274bce0 x28: 0000000000000000
>   x27: ffff00525626fcc8 x26: ffff800011921078
>   x25: 0000000000000000 x24: 0000000000000004
>   x23: ffff00402b4059c0 x22: ffff00402b405940
>   x21: ffff205d87b81e21 x20: ffff205d87b81b9b
>   x19: 0000000000000000 x18: 0000000000000000
>   x17: 0000000000000000 x16: 0000000000000000
>   x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
>   x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
>   x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
>   x9 : ffff8000097d7628 x8 : ffff205d87b85e20
>   x7 : 0000000000000000 x6 : 0000000000000001
>   x5 : ffff8000097dc008 x4 : ffff8000097d75a4
>   x3 : ffff205d87b81e1f x2 : 0000000000000005
>   x1 : 000000000000005f x0 : ffff00402b405940
>   Call trace:
>    slip_unesc+0x80/0x214 [slip]
>    tty_ldisc_receive_buf+0x64/0x80
>    tty_port_default_receive_buf+0x50/0x90
>    flush_to_ldisc+0xbc/0x110
>    process_one_work+0x1d4/0x4b0
>    worker_thread+0x180/0x430
>    kthread+0x11c/0x120
>   Kernel panic - not syncing: softlockup: hung tasks
>   CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
>   Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>   Workqueue: events_unbound flush_to_ldisc
>   Call trace:
>    dump_backtrace+0x0/0x1ec
>    show_stack+0x24/0x30
>    dump_stack+0xd0/0x128
>    panic+0x15c/0x374
>    watchdog_timer_fn+0x2b8/0x304
>    __run_hrtimer+0x88/0x2c0
>    __hrtimer_run_queues+0xa4/0x120
>    hrtimer_interrupt+0xfc/0x270
>    arch_timer_handler_phys+0x40/0x50
>    handle_percpu_devid_irq+0x94/0x220
>    __handle_domain_irq+0x88/0xf0
>    gic_handle_irq+0x84/0xfc
>    el1_irq+0xc8/0x180
>    slip_unesc+0x80/0x214 [slip]
>    tty_ldisc_receive_buf+0x64/0x80
>    tty_port_default_receive_buf+0x50/0x90
>    flush_to_ldisc+0xbc/0x110
>    process_one_work+0x1d4/0x4b0
>    worker_thread+0x180/0x430
>    kthread+0x11c/0x120
>   SMP: stopping secondary CPUs
> 
> In the testcase pty04, there are multple processes and we only pay close attention to the
> first three actually. The first process call the write syscall to send data to the pty master
> with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
> The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
> The third process will wait a moment in which the first two processes will do there work and then
> it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
> Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
> time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
> master's buffhead to line discipline in a loop until there is no more data left without any on one's
> own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
> without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
> in the flush_to_ldisc while loop to avoid it.

Please properly wrap your changelog text at 72 columns.


> 
> Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
> ---
>  drivers/tty/tty_buffer.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
> index bd2d915..77b92f9 100644
> --- a/drivers/tty/tty_buffer.c
> +++ b/drivers/tty/tty_buffer.c
> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>  		if (!count)
>  			break;
>  		head->read += count;
> +		cond_resched();

This is almost never the correct solution for fixing a problem in the
kernel anymore.

And if it is, it needs to be documented really really well.  I think you
just slowed down the overall throughput of a tty device by adding this
call, so are you sure you didn't break something?

And why are you not running with a preempt kernel here?  What prevents
that from being enabled to solve issues like this?

Also, having only one CPU burning through a network workload like this
seems correct to me, why would you want the CPU to stop handling the
data being sent to it like this?  You have at least 40 other ones to do
other things here :)

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-09-30  5:38 ` Greg KH
@ 2021-09-30  6:02   ` Baolin Wang
  2021-10-08  7:50   ` guanghui.fgh
  1 sibling, 0 replies; 9+ messages in thread
From: Baolin Wang @ 2021-09-30  6:02 UTC (permalink / raw)
  To: Greg KH, Guanghui Feng; +Cc: jirislaby, linux-kernel

Hi Greg,

On 2021/9/30 13:38, Greg KH wrote:
> On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>> which look like this one:
>>
>>    watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
>>    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
>>    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>    Workqueue: events_unbound flush_to_ldisc
>>    pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
>>    pc : slip_unesc+0x80/0x214 [slip]
>>    lr : slip_receive_buf+0x84/0x100 [slip]
>>    sp : ffff80005274bce0
>>    x29: ffff80005274bce0 x28: 0000000000000000
>>    x27: ffff00525626fcc8 x26: ffff800011921078
>>    x25: 0000000000000000 x24: 0000000000000004
>>    x23: ffff00402b4059c0 x22: ffff00402b405940
>>    x21: ffff205d87b81e21 x20: ffff205d87b81b9b
>>    x19: 0000000000000000 x18: 0000000000000000
>>    x17: 0000000000000000 x16: 0000000000000000
>>    x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
>>    x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
>>    x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
>>    x9 : ffff8000097d7628 x8 : ffff205d87b85e20
>>    x7 : 0000000000000000 x6 : 0000000000000001
>>    x5 : ffff8000097dc008 x4 : ffff8000097d75a4
>>    x3 : ffff205d87b81e1f x2 : 0000000000000005
>>    x1 : 000000000000005f x0 : ffff00402b405940
>>    Call trace:
>>     slip_unesc+0x80/0x214 [slip]
>>     tty_ldisc_receive_buf+0x64/0x80
>>     tty_port_default_receive_buf+0x50/0x90
>>     flush_to_ldisc+0xbc/0x110
>>     process_one_work+0x1d4/0x4b0
>>     worker_thread+0x180/0x430
>>     kthread+0x11c/0x120
>>    Kernel panic - not syncing: softlockup: hung tasks
>>    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
>>    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>    Workqueue: events_unbound flush_to_ldisc
>>    Call trace:
>>     dump_backtrace+0x0/0x1ec
>>     show_stack+0x24/0x30
>>     dump_stack+0xd0/0x128
>>     panic+0x15c/0x374
>>     watchdog_timer_fn+0x2b8/0x304
>>     __run_hrtimer+0x88/0x2c0
>>     __hrtimer_run_queues+0xa4/0x120
>>     hrtimer_interrupt+0xfc/0x270
>>     arch_timer_handler_phys+0x40/0x50
>>     handle_percpu_devid_irq+0x94/0x220
>>     __handle_domain_irq+0x88/0xf0
>>     gic_handle_irq+0x84/0xfc
>>     el1_irq+0xc8/0x180
>>     slip_unesc+0x80/0x214 [slip]
>>     tty_ldisc_receive_buf+0x64/0x80
>>     tty_port_default_receive_buf+0x50/0x90
>>     flush_to_ldisc+0xbc/0x110
>>     process_one_work+0x1d4/0x4b0
>>     worker_thread+0x180/0x430
>>     kthread+0x11c/0x120
>>    SMP: stopping secondary CPUs
>>
>> In the testcase pty04, there are multple processes and we only pay close attention to the
>> first three actually. The first process call the write syscall to send data to the pty master
>> with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
>> The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
>> The third process will wait a moment in which the first two processes will do there work and then
>> it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
>> Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
>> time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
>> master's buffhead to line discipline in a loop until there is no more data left without any on one's
>> own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
>> without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
>> in the flush_to_ldisc while loop to avoid it.
> 
> Please properly wrap your changelog text at 72 columns.
> 
> 
>>
>> Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
>> ---
>>   drivers/tty/tty_buffer.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
>> index bd2d915..77b92f9 100644
>> --- a/drivers/tty/tty_buffer.c
>> +++ b/drivers/tty/tty_buffer.c
>> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>>   		if (!count)
>>   			break;
>>   		head->read += count;
>> +		cond_resched();
> 
> This is almost never the correct solution for fixing a problem in the
> kernel anymore.
> 
> And if it is, it needs to be documented really really well.  I think you
> just slowed down the overall throughput of a tty device by adding this
> call, so are you sure you didn't break something?
> 
> And why are you not running with a preempt kernel here?  What prevents
> that from being enabled to solve issues like this?

For server scenario, we usually select CONFIG_PREEMPT_VOLUNTARY instead 
of selecting CONFIG_PREEMPT. So for this stress test case, lots of data 
need to receive in the loop in the flush_to_ldisc(), which will block 
other threads to be scheduled for a long time on this CPU if no 
scheduling check.

For the throughput concern, I think we can change to below code to avoid 
impacting the throughput, which means only need rescheduling if 
necessary, and also fix the softlockup issue.

+ if (need_resched())
+	cond_resched();

> 
> Also, having only one CPU burning through a network workload like this
> seems correct to me, why would you want the CPU to stop handling the
> data being sent to it like this?  You have at least 40 other ones to do
> other things here :)
> 
> thanks,
> 
> greg k-h
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-09-30  5:38 ` Greg KH
  2021-09-30  6:02   ` Baolin Wang
@ 2021-10-08  7:50   ` guanghui.fgh
  2021-10-10 13:18     ` Greg KH
  1 sibling, 1 reply; 9+ messages in thread
From: guanghui.fgh @ 2021-10-08  7:50 UTC (permalink / raw)
  To: Greg KH; +Cc: jirislaby, baolin.wang, linux-kernel


在 2021/9/30 13:38, Greg KH 写道:
> On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>> which look like this one:
>>
>>    watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
>>    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
>>    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>    Workqueue: events_unbound flush_to_ldisc
>>    pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
>>    pc : slip_unesc+0x80/0x214 [slip]
>>    lr : slip_receive_buf+0x84/0x100 [slip]
>>    sp : ffff80005274bce0
>>    x29: ffff80005274bce0 x28: 0000000000000000
>>    x27: ffff00525626fcc8 x26: ffff800011921078
>>    x25: 0000000000000000 x24: 0000000000000004
>>    x23: ffff00402b4059c0 x22: ffff00402b405940
>>    x21: ffff205d87b81e21 x20: ffff205d87b81b9b
>>    x19: 0000000000000000 x18: 0000000000000000
>>    x17: 0000000000000000 x16: 0000000000000000
>>    x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
>>    x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
>>    x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
>>    x9 : ffff8000097d7628 x8 : ffff205d87b85e20
>>    x7 : 0000000000000000 x6 : 0000000000000001
>>    x5 : ffff8000097dc008 x4 : ffff8000097d75a4
>>    x3 : ffff205d87b81e1f x2 : 0000000000000005
>>    x1 : 000000000000005f x0 : ffff00402b405940
>>    Call trace:
>>     slip_unesc+0x80/0x214 [slip]
>>     tty_ldisc_receive_buf+0x64/0x80
>>     tty_port_default_receive_buf+0x50/0x90
>>     flush_to_ldisc+0xbc/0x110
>>     process_one_work+0x1d4/0x4b0
>>     worker_thread+0x180/0x430
>>     kthread+0x11c/0x120
>>    Kernel panic - not syncing: softlockup: hung tasks
>>    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
>>    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>    Workqueue: events_unbound flush_to_ldisc
>>    Call trace:
>>     dump_backtrace+0x0/0x1ec
>>     show_stack+0x24/0x30
>>     dump_stack+0xd0/0x128
>>     panic+0x15c/0x374
>>     watchdog_timer_fn+0x2b8/0x304
>>     __run_hrtimer+0x88/0x2c0
>>     __hrtimer_run_queues+0xa4/0x120
>>     hrtimer_interrupt+0xfc/0x270
>>     arch_timer_handler_phys+0x40/0x50
>>     handle_percpu_devid_irq+0x94/0x220
>>     __handle_domain_irq+0x88/0xf0
>>     gic_handle_irq+0x84/0xfc
>>     el1_irq+0xc8/0x180
>>     slip_unesc+0x80/0x214 [slip]
>>     tty_ldisc_receive_buf+0x64/0x80
>>     tty_port_default_receive_buf+0x50/0x90
>>     flush_to_ldisc+0xbc/0x110
>>     process_one_work+0x1d4/0x4b0
>>     worker_thread+0x180/0x430
>>     kthread+0x11c/0x120
>>    SMP: stopping secondary CPUs
>>
>> In the testcase pty04, there are multple processes and we only pay close attention to the
>> first three actually. The first process call the write syscall to send data to the pty master
>> with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
>> The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
>> The third process will wait a moment in which the first two processes will do there work and then
>> it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
>> Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
>> time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
>> master's buffhead to line discipline in a loop until there is no more data left without any on one's
>> own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
>> without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
>> in the flush_to_ldisc while loop to avoid it.
> Please properly wrap your changelog text at 72 columns.

When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
which look like this one:
Call trace:
    dump_backtrace+0x0/0x1ec
    show_stack+0x24/0x30
    dump_stack+0xd0/0x128
    panic+0x15c/0x374
    watchdog_timer_fn+0x2b8/0x304
    __run_hrtimer+0x88/0x2c0
    __hrtimer_run_queues+0xa4/0x120
    hrtimer_interrupt+0xfc/0x270
    arch_timer_handler_phys+0x40/0x50
    handle_percpu_devid_irq+0x94/0x220
    __handle_domain_irq+0x88/0xf0
    gic_handle_irq+0x84/0xfc
    el1_irq+0xc8/0x180
    slip_unesc+0x80/0x214 [slip]
    tty_ldisc_receive_buf+0x64/0x80
    tty_port_default_receive_buf+0x50/0x90
    flush_to_ldisc+0xbc/0x110
    process_one_work+0x1d4/0x4b0
    worker_thread+0x180/0x430
    kthread+0x11c/0x120

In the testcase pty04, The first process call the write syscall to send data to the pty master.
At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.

>
>
>> Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
>> ---
>>   drivers/tty/tty_buffer.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
>> index bd2d915..77b92f9 100644
>> --- a/drivers/tty/tty_buffer.c
>> +++ b/drivers/tty/tty_buffer.c
>> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>>   		if (!count)
>>   			break;
>>   		head->read += count;
>> +		cond_resched();
> This is almost never the correct solution for fixing a problem in the
> kernel anymore.
>
> And if it is, it needs to be documented really really well.  I think you
> just slowed down the overall throughput of a tty device by adding this
> call, so are you sure you didn't break something?

OK, it should be:

diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
index bd2d915..77b92f9 100644
--- a/drivers/tty/tty_buffer.c
+++ b/drivers/tty/tty_buffer.c
@@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
  		if (!count)
  			break;
  		head->read += count;
+		if (need_resched())
+			cond_resched();

> And why are you not running with a preempt kernel here?  What prevents
> that from being enabled to solve issues like this?
In server mode, we usually running without preempt kernel for 
performance(with less scheduling)
> Also, having only one CPU burning through a network workload like this
> seems correct to me, why would you want the CPU to stop handling the
> data being sent to it like this?  You have at least 40 other ones to do
> other things here :)
>
> thanks,
>
> greg k-h

When only using one core, the pty data sending and workqueue can't do 
work simultaneously. When the sender and workqueue

running in different core, the workqueue will do the flush_to_ldisc in a 
loop until there is no more data left which will result in

occuring softlockup when the sender sends data fastly in full time. So I 
add need_resched check and cond_resched in the

flush_to_ldisc while loop to avoid it(without preempt kernel).

thanks

Guanghui Feng


^ permalink raw reply related	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-10-08  7:50   ` guanghui.fgh
@ 2021-10-10 13:18     ` Greg KH
  2021-10-11  7:42       ` guanghui.fgh
  0 siblings, 1 reply; 9+ messages in thread
From: Greg KH @ 2021-10-10 13:18 UTC (permalink / raw)
  To: guanghui.fgh; +Cc: jirislaby, baolin.wang, linux-kernel

On Fri, Oct 08, 2021 at 03:50:15PM +0800, guanghui.fgh wrote:
> 
> 在 2021/9/30 13:38, Greg KH 写道:
> > On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
> > > When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> > > which look like this one:
> > > 
> > >    watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
> > >    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
> > >    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > >    Workqueue: events_unbound flush_to_ldisc
> > >    pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
> > >    pc : slip_unesc+0x80/0x214 [slip]
> > >    lr : slip_receive_buf+0x84/0x100 [slip]
> > >    sp : ffff80005274bce0
> > >    x29: ffff80005274bce0 x28: 0000000000000000
> > >    x27: ffff00525626fcc8 x26: ffff800011921078
> > >    x25: 0000000000000000 x24: 0000000000000004
> > >    x23: ffff00402b4059c0 x22: ffff00402b405940
> > >    x21: ffff205d87b81e21 x20: ffff205d87b81b9b
> > >    x19: 0000000000000000 x18: 0000000000000000
> > >    x17: 0000000000000000 x16: 0000000000000000
> > >    x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
> > >    x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
> > >    x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
> > >    x9 : ffff8000097d7628 x8 : ffff205d87b85e20
> > >    x7 : 0000000000000000 x6 : 0000000000000001
> > >    x5 : ffff8000097dc008 x4 : ffff8000097d75a4
> > >    x3 : ffff205d87b81e1f x2 : 0000000000000005
> > >    x1 : 000000000000005f x0 : ffff00402b405940
> > >    Call trace:
> > >     slip_unesc+0x80/0x214 [slip]
> > >     tty_ldisc_receive_buf+0x64/0x80
> > >     tty_port_default_receive_buf+0x50/0x90
> > >     flush_to_ldisc+0xbc/0x110
> > >     process_one_work+0x1d4/0x4b0
> > >     worker_thread+0x180/0x430
> > >     kthread+0x11c/0x120
> > >    Kernel panic - not syncing: softlockup: hung tasks
> > >    CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
> > >    Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > >    Workqueue: events_unbound flush_to_ldisc
> > >    Call trace:
> > >     dump_backtrace+0x0/0x1ec
> > >     show_stack+0x24/0x30
> > >     dump_stack+0xd0/0x128
> > >     panic+0x15c/0x374
> > >     watchdog_timer_fn+0x2b8/0x304
> > >     __run_hrtimer+0x88/0x2c0
> > >     __hrtimer_run_queues+0xa4/0x120
> > >     hrtimer_interrupt+0xfc/0x270
> > >     arch_timer_handler_phys+0x40/0x50
> > >     handle_percpu_devid_irq+0x94/0x220
> > >     __handle_domain_irq+0x88/0xf0
> > >     gic_handle_irq+0x84/0xfc
> > >     el1_irq+0xc8/0x180
> > >     slip_unesc+0x80/0x214 [slip]
> > >     tty_ldisc_receive_buf+0x64/0x80
> > >     tty_port_default_receive_buf+0x50/0x90
> > >     flush_to_ldisc+0xbc/0x110
> > >     process_one_work+0x1d4/0x4b0
> > >     worker_thread+0x180/0x430
> > >     kthread+0x11c/0x120
> > >    SMP: stopping secondary CPUs
> > > 
> > > In the testcase pty04, there are multple processes and we only pay close attention to the
> > > first three actually. The first process call the write syscall to send data to the pty master
> > > with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
> > > The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
> > > The third process will wait a moment in which the first two processes will do there work and then
> > > it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
> > > Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
> > > time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
> > > master's buffhead to line discipline in a loop until there is no more data left without any on one's
> > > own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
> > > without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
> > > in the flush_to_ldisc while loop to avoid it.
> > Please properly wrap your changelog text at 72 columns.
> 
> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> which look like this one:
> Call trace:
>    dump_backtrace+0x0/0x1ec
>    show_stack+0x24/0x30
>    dump_stack+0xd0/0x128
>    panic+0x15c/0x374
>    watchdog_timer_fn+0x2b8/0x304
>    __run_hrtimer+0x88/0x2c0
>    __hrtimer_run_queues+0xa4/0x120
>    hrtimer_interrupt+0xfc/0x270
>    arch_timer_handler_phys+0x40/0x50
>    handle_percpu_devid_irq+0x94/0x220
>    __handle_domain_irq+0x88/0xf0
>    gic_handle_irq+0x84/0xfc
>    el1_irq+0xc8/0x180
>    slip_unesc+0x80/0x214 [slip]
>    tty_ldisc_receive_buf+0x64/0x80
>    tty_port_default_receive_buf+0x50/0x90
>    flush_to_ldisc+0xbc/0x110
>    process_one_work+0x1d4/0x4b0
>    worker_thread+0x180/0x430
>    kthread+0x11c/0x120
> 
> In the testcase pty04, The first process call the write syscall to send data to the pty master.
> At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
> in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
> long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.

Is this a "real" test for something that you have seen in a normal
workload?  ltp is known for having buggy/confusing tests in it in the
past, you might wish to consult with the authors of that test.

> 
> > 
> > 
> > > Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
> > > ---
> > >   drivers/tty/tty_buffer.c | 1 +
> > >   1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
> > > index bd2d915..77b92f9 100644
> > > --- a/drivers/tty/tty_buffer.c
> > > +++ b/drivers/tty/tty_buffer.c
> > > @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
> > >   		if (!count)
> > >   			break;
> > >   		head->read += count;
> > > +		cond_resched();
> > This is almost never the correct solution for fixing a problem in the
> > kernel anymore.
> > 
> > And if it is, it needs to be documented really really well.  I think you
> > just slowed down the overall throughput of a tty device by adding this
> > call, so are you sure you didn't break something?
> 
> OK, it should be:
> 
> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
> index bd2d915..77b92f9 100644
> --- a/drivers/tty/tty_buffer.c
> +++ b/drivers/tty/tty_buffer.c
> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>  		if (!count)
>  			break;
>  		head->read += count;
> +		if (need_resched())
> +			cond_resched();

Still feels really wrong, we do not sprinkle this all around the kernel
if we do not have to.

> > And why are you not running with a preempt kernel here?  What prevents
> > that from being enabled to solve issues like this?
> In server mode, we usually running without preempt kernel for
> performance(with less scheduling)

You are trading off throughput for this very reason, you are sending
data faster than you could normally have, so why are you wanting to stop
that?

> > Also, having only one CPU burning through a network workload like this
> > seems correct to me, why would you want the CPU to stop handling the
> > data being sent to it like this?  You have at least 40 other ones to do
> > other things here :)
> > 
> > thanks,
> > 
> > greg k-h
> 
> When only using one core, the pty data sending and workqueue can't do work
> simultaneously. When the sender and workqueue
> 
> running in different core, the workqueue will do the flush_to_ldisc in a
> loop until there is no more data left which will result in
> 
> occuring softlockup when the sender sends data fastly in full time. So I add
> need_resched check and cond_resched in the
> 
> flush_to_ldisc while loop to avoid it(without preempt kernel).

Why not just switch to preempt kernel then if this specific workload
really is important to you?

Again, is this a real workload, or just a contrived test that is trying
to get as much throughput as possible for a single pty device?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-10-10 13:18     ` Greg KH
@ 2021-10-11  7:42       ` guanghui.fgh
  2021-10-11  7:50         ` Greg KH
  0 siblings, 1 reply; 9+ messages in thread
From: guanghui.fgh @ 2021-10-11  7:42 UTC (permalink / raw)
  To: Greg KH; +Cc: jirislaby, baolin.wang, linux-kernel, zhuo.song, zhangliguang


在 2021/10/10 21:18, Greg KH 写道:
> On Fri, Oct 08, 2021 at 03:50:15PM +0800, guanghui.fgh wrote:
>> 在 2021/9/30 13:38, Greg KH 写道:
>>> On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
>>>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>>>> which look like this one:
>>>>
>>>>     watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
>>>>     CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
>>>>     Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>>>     Workqueue: events_unbound flush_to_ldisc
>>>>     pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
>>>>     pc : slip_unesc+0x80/0x214 [slip]
>>>>     lr : slip_receive_buf+0x84/0x100 [slip]
>>>>     sp : ffff80005274bce0
>>>>     x29: ffff80005274bce0 x28: 0000000000000000
>>>>     x27: ffff00525626fcc8 x26: ffff800011921078
>>>>     x25: 0000000000000000 x24: 0000000000000004
>>>>     x23: ffff00402b4059c0 x22: ffff00402b405940
>>>>     x21: ffff205d87b81e21 x20: ffff205d87b81b9b
>>>>     x19: 0000000000000000 x18: 0000000000000000
>>>>     x17: 0000000000000000 x16: 0000000000000000
>>>>     x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
>>>>     x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
>>>>     x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
>>>>     x9 : ffff8000097d7628 x8 : ffff205d87b85e20
>>>>     x7 : 0000000000000000 x6 : 0000000000000001
>>>>     x5 : ffff8000097dc008 x4 : ffff8000097d75a4
>>>>     x3 : ffff205d87b81e1f x2 : 0000000000000005
>>>>     x1 : 000000000000005f x0 : ffff00402b405940
>>>>     Call trace:
>>>>      slip_unesc+0x80/0x214 [slip]
>>>>      tty_ldisc_receive_buf+0x64/0x80
>>>>      tty_port_default_receive_buf+0x50/0x90
>>>>      flush_to_ldisc+0xbc/0x110
>>>>      process_one_work+0x1d4/0x4b0
>>>>      worker_thread+0x180/0x430
>>>>      kthread+0x11c/0x120
>>>>     Kernel panic - not syncing: softlockup: hung tasks
>>>>     CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
>>>>     Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>>>     Workqueue: events_unbound flush_to_ldisc
>>>>     Call trace:
>>>>      dump_backtrace+0x0/0x1ec
>>>>      show_stack+0x24/0x30
>>>>      dump_stack+0xd0/0x128
>>>>      panic+0x15c/0x374
>>>>      watchdog_timer_fn+0x2b8/0x304
>>>>      __run_hrtimer+0x88/0x2c0
>>>>      __hrtimer_run_queues+0xa4/0x120
>>>>      hrtimer_interrupt+0xfc/0x270
>>>>      arch_timer_handler_phys+0x40/0x50
>>>>      handle_percpu_devid_irq+0x94/0x220
>>>>      __handle_domain_irq+0x88/0xf0
>>>>      gic_handle_irq+0x84/0xfc
>>>>      el1_irq+0xc8/0x180
>>>>      slip_unesc+0x80/0x214 [slip]
>>>>      tty_ldisc_receive_buf+0x64/0x80
>>>>      tty_port_default_receive_buf+0x50/0x90
>>>>      flush_to_ldisc+0xbc/0x110
>>>>      process_one_work+0x1d4/0x4b0
>>>>      worker_thread+0x180/0x430
>>>>      kthread+0x11c/0x120
>>>>     SMP: stopping secondary CPUs
>>>>
>>>> In the testcase pty04, there are multple processes and we only pay close attention to the
>>>> first three actually. The first process call the write syscall to send data to the pty master
>>>> with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
>>>> The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
>>>> The third process will wait a moment in which the first two processes will do there work and then
>>>> it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
>>>> Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
>>>> time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
>>>> master's buffhead to line discipline in a loop until there is no more data left without any on one's
>>>> own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
>>>> without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
>>>> in the flush_to_ldisc while loop to avoid it.
>>> Please properly wrap your changelog text at 72 columns.
>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>> which look like this one:
>> Call trace:
>>     dump_backtrace+0x0/0x1ec
>>     show_stack+0x24/0x30
>>     dump_stack+0xd0/0x128
>>     panic+0x15c/0x374
>>     watchdog_timer_fn+0x2b8/0x304
>>     __run_hrtimer+0x88/0x2c0
>>     __hrtimer_run_queues+0xa4/0x120
>>     hrtimer_interrupt+0xfc/0x270
>>     arch_timer_handler_phys+0x40/0x50
>>     handle_percpu_devid_irq+0x94/0x220
>>     __handle_domain_irq+0x88/0xf0
>>     gic_handle_irq+0x84/0xfc
>>     el1_irq+0xc8/0x180
>>     slip_unesc+0x80/0x214 [slip]
>>     tty_ldisc_receive_buf+0x64/0x80
>>     tty_port_default_receive_buf+0x50/0x90
>>     flush_to_ldisc+0xbc/0x110
>>     process_one_work+0x1d4/0x4b0
>>     worker_thread+0x180/0x430
>>     kthread+0x11c/0x120
>>
>> In the testcase pty04, The first process call the write syscall to send data to the pty master.
>> At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
>> in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
>> long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.
> Is this a "real" test for something that you have seen in a normal
> workload?  ltp is known for having buggy/confusing tests in it in the
> past, you might wish to consult with the authors of that test.

Firstly, thanks for your response.

I have check the ltp pty testcase. At the same time, I find the pty 
softlockup in arm64, and it is similar to others.

https://github.com/victronenergy/venus/issues/350

https://groups.google.com/g/syzkaller-lts-bugs/c/SpkH8yH26js/m/3aifBl_GAwAJ

>>>
>>>> Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
>>>> ---
>>>>    drivers/tty/tty_buffer.c | 1 +
>>>>    1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
>>>> index bd2d915..77b92f9 100644
>>>> --- a/drivers/tty/tty_buffer.c
>>>> +++ b/drivers/tty/tty_buffer.c
>>>> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>>>>    		if (!count)
>>>>    			break;
>>>>    		head->read += count;
>>>> +		cond_resched();
>>> This is almost never the correct solution for fixing a problem in the
>>> kernel anymore.
>>>
>>> And if it is, it needs to be documented really really well.  I think you
>>> just slowed down the overall throughput of a tty device by adding this
>>> call, so are you sure you didn't break something?
>> OK, it should be:
>>
>> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
>> index bd2d915..77b92f9 100644
>> --- a/drivers/tty/tty_buffer.c
>> +++ b/drivers/tty/tty_buffer.c
>> @@ -534,6 +534,7 @@ static void flush_to_ldisc(struct work_struct *work)
>>   		if (!count)
>>   			break;
>>   		head->read += count;
>> +		if (need_resched())
>> +			cond_resched();
> Still feels really wrong, we do not sprinkle this all around the kernel
> if we do not have to.
>
>>> And why are you not running with a preempt kernel here?  What prevents
>>> that from being enabled to solve issues like this?
>> In server mode, we usually running without preempt kernel for
>> performance(with less scheduling)
> You are trading off throughput for this very reason, you are sending
> data faster than you could normally have, so why are you wanting to stop
> that?
>
>>> Also, having only one CPU burning through a network workload like this
>>> seems correct to me, why would you want the CPU to stop handling the
>>> data being sent to it like this?  You have at least 40 other ones to do
>>> other things here :)
>>>
>>> thanks,
>>>
>>> greg k-h
>> When only using one core, the pty data sending and workqueue can't do work
>> simultaneously. When the sender and workqueue
>>
>> running in different core, the workqueue will do the flush_to_ldisc in a
>> loop until there is no more data left which will result in
>>
>> occuring softlockup when the sender sends data fastly in full time. So I add
>> need_resched check and cond_resched in the
>>
>> flush_to_ldisc while loop to avoid it(without preempt kernel).
> Why not just switch to preempt kernel then if this specific workload
> really is important to you?
>
> Again, is this a real workload, or just a contrived test that is trying
> to get as much throughput as possible for a single pty device?
>
> thanks,
>
> greg k-h

Because there are many enviroment working right for a long time, and 
normally only doing bug fix,

and the consumers don't be willing to change the kernel configues.

thanks

Guanghui Feng


^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-10-11  7:42       ` guanghui.fgh
@ 2021-10-11  7:50         ` Greg KH
  2021-10-11 10:18           ` guanghui.fgh
  0 siblings, 1 reply; 9+ messages in thread
From: Greg KH @ 2021-10-11  7:50 UTC (permalink / raw)
  To: guanghui.fgh
  Cc: jirislaby, baolin.wang, linux-kernel, zhuo.song, zhangliguang

On Mon, Oct 11, 2021 at 03:42:38PM +0800, guanghui.fgh wrote:
> 
> 在 2021/10/10 21:18, Greg KH 写道:
> > On Fri, Oct 08, 2021 at 03:50:15PM +0800, guanghui.fgh wrote:
> > > 在 2021/9/30 13:38, Greg KH 写道:
> > > > On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
> > > > > When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> > > > > which look like this one:
> > > > > 
> > > > >     watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
> > > > >     CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
> > > > >     Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > > > >     Workqueue: events_unbound flush_to_ldisc
> > > > >     pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
> > > > >     pc : slip_unesc+0x80/0x214 [slip]
> > > > >     lr : slip_receive_buf+0x84/0x100 [slip]
> > > > >     sp : ffff80005274bce0
> > > > >     x29: ffff80005274bce0 x28: 0000000000000000
> > > > >     x27: ffff00525626fcc8 x26: ffff800011921078
> > > > >     x25: 0000000000000000 x24: 0000000000000004
> > > > >     x23: ffff00402b4059c0 x22: ffff00402b405940
> > > > >     x21: ffff205d87b81e21 x20: ffff205d87b81b9b
> > > > >     x19: 0000000000000000 x18: 0000000000000000
> > > > >     x17: 0000000000000000 x16: 0000000000000000
> > > > >     x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
> > > > >     x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
> > > > >     x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
> > > > >     x9 : ffff8000097d7628 x8 : ffff205d87b85e20
> > > > >     x7 : 0000000000000000 x6 : 0000000000000001
> > > > >     x5 : ffff8000097dc008 x4 : ffff8000097d75a4
> > > > >     x3 : ffff205d87b81e1f x2 : 0000000000000005
> > > > >     x1 : 000000000000005f x0 : ffff00402b405940
> > > > >     Call trace:
> > > > >      slip_unesc+0x80/0x214 [slip]
> > > > >      tty_ldisc_receive_buf+0x64/0x80
> > > > >      tty_port_default_receive_buf+0x50/0x90
> > > > >      flush_to_ldisc+0xbc/0x110
> > > > >      process_one_work+0x1d4/0x4b0
> > > > >      worker_thread+0x180/0x430
> > > > >      kthread+0x11c/0x120
> > > > >     Kernel panic - not syncing: softlockup: hung tasks
> > > > >     CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
> > > > >     Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > > > >     Workqueue: events_unbound flush_to_ldisc
> > > > >     Call trace:
> > > > >      dump_backtrace+0x0/0x1ec
> > > > >      show_stack+0x24/0x30
> > > > >      dump_stack+0xd0/0x128
> > > > >      panic+0x15c/0x374
> > > > >      watchdog_timer_fn+0x2b8/0x304
> > > > >      __run_hrtimer+0x88/0x2c0
> > > > >      __hrtimer_run_queues+0xa4/0x120
> > > > >      hrtimer_interrupt+0xfc/0x270
> > > > >      arch_timer_handler_phys+0x40/0x50
> > > > >      handle_percpu_devid_irq+0x94/0x220
> > > > >      __handle_domain_irq+0x88/0xf0
> > > > >      gic_handle_irq+0x84/0xfc
> > > > >      el1_irq+0xc8/0x180
> > > > >      slip_unesc+0x80/0x214 [slip]
> > > > >      tty_ldisc_receive_buf+0x64/0x80
> > > > >      tty_port_default_receive_buf+0x50/0x90
> > > > >      flush_to_ldisc+0xbc/0x110
> > > > >      process_one_work+0x1d4/0x4b0
> > > > >      worker_thread+0x180/0x430
> > > > >      kthread+0x11c/0x120
> > > > >     SMP: stopping secondary CPUs
> > > > > 
> > > > > In the testcase pty04, there are multple processes and we only pay close attention to the
> > > > > first three actually. The first process call the write syscall to send data to the pty master
> > > > > with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
> > > > > The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
> > > > > The third process will wait a moment in which the first two processes will do there work and then
> > > > > it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
> > > > > Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
> > > > > time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
> > > > > master's buffhead to line discipline in a loop until there is no more data left without any on one's
> > > > > own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
> > > > > without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
> > > > > in the flush_to_ldisc while loop to avoid it.
> > > > Please properly wrap your changelog text at 72 columns.
> > > When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> > > which look like this one:
> > > Call trace:
> > >     dump_backtrace+0x0/0x1ec
> > >     show_stack+0x24/0x30
> > >     dump_stack+0xd0/0x128
> > >     panic+0x15c/0x374
> > >     watchdog_timer_fn+0x2b8/0x304
> > >     __run_hrtimer+0x88/0x2c0
> > >     __hrtimer_run_queues+0xa4/0x120
> > >     hrtimer_interrupt+0xfc/0x270
> > >     arch_timer_handler_phys+0x40/0x50
> > >     handle_percpu_devid_irq+0x94/0x220
> > >     __handle_domain_irq+0x88/0xf0
> > >     gic_handle_irq+0x84/0xfc
> > >     el1_irq+0xc8/0x180
> > >     slip_unesc+0x80/0x214 [slip]
> > >     tty_ldisc_receive_buf+0x64/0x80
> > >     tty_port_default_receive_buf+0x50/0x90
> > >     flush_to_ldisc+0xbc/0x110
> > >     process_one_work+0x1d4/0x4b0
> > >     worker_thread+0x180/0x430
> > >     kthread+0x11c/0x120
> > > 
> > > In the testcase pty04, The first process call the write syscall to send data to the pty master.
> > > At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
> > > in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
> > > long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.
> > Is this a "real" test for something that you have seen in a normal
> > workload?  ltp is known for having buggy/confusing tests in it in the
> > past, you might wish to consult with the authors of that test.
> 
> Firstly, thanks for your response.
> 
> I have check the ltp pty testcase. At the same time, I find the pty
> softlockup in arm64, and it is similar to others.
> 
> https://github.com/victronenergy/venus/issues/350
> 
> https://groups.google.com/g/syzkaller-lts-bugs/c/SpkH8yH26js/m/3aifBl_GAwAJ

Hm, ok, can you please resubmit this based on the changes discussed in
thread and I will re-review it again.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-10-11  7:50         ` Greg KH
@ 2021-10-11 10:18           ` guanghui.fgh
  2021-10-11 10:54             ` Greg KH
  0 siblings, 1 reply; 9+ messages in thread
From: guanghui.fgh @ 2021-10-11 10:18 UTC (permalink / raw)
  To: Greg KH; +Cc: jirislaby, baolin.wang, linux-kernel, zhuo.song, zhangliguang


在 2021/10/11 15:50, Greg KH 写道:
> On Mon, Oct 11, 2021 at 03:42:38PM +0800, guanghui.fgh wrote:
>> 在 2021/10/10 21:18, Greg KH 写道:
>>> On Fri, Oct 08, 2021 at 03:50:15PM +0800, guanghui.fgh wrote:
>>>> 在 2021/9/30 13:38, Greg KH 写道:
>>>>> On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
>>>>>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>>>>>> which look like this one:
>>>>>>
>>>>>>      watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
>>>>>>      CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
>>>>>>      Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>>>>>      Workqueue: events_unbound flush_to_ldisc
>>>>>>      pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
>>>>>>      pc : slip_unesc+0x80/0x214 [slip]
>>>>>>      lr : slip_receive_buf+0x84/0x100 [slip]
>>>>>>      sp : ffff80005274bce0
>>>>>>      x29: ffff80005274bce0 x28: 0000000000000000
>>>>>>      x27: ffff00525626fcc8 x26: ffff800011921078
>>>>>>      x25: 0000000000000000 x24: 0000000000000004
>>>>>>      x23: ffff00402b4059c0 x22: ffff00402b405940
>>>>>>      x21: ffff205d87b81e21 x20: ffff205d87b81b9b
>>>>>>      x19: 0000000000000000 x18: 0000000000000000
>>>>>>      x17: 0000000000000000 x16: 0000000000000000
>>>>>>      x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
>>>>>>      x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
>>>>>>      x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
>>>>>>      x9 : ffff8000097d7628 x8 : ffff205d87b85e20
>>>>>>      x7 : 0000000000000000 x6 : 0000000000000001
>>>>>>      x5 : ffff8000097dc008 x4 : ffff8000097d75a4
>>>>>>      x3 : ffff205d87b81e1f x2 : 0000000000000005
>>>>>>      x1 : 000000000000005f x0 : ffff00402b405940
>>>>>>      Call trace:
>>>>>>       slip_unesc+0x80/0x214 [slip]
>>>>>>       tty_ldisc_receive_buf+0x64/0x80
>>>>>>       tty_port_default_receive_buf+0x50/0x90
>>>>>>       flush_to_ldisc+0xbc/0x110
>>>>>>       process_one_work+0x1d4/0x4b0
>>>>>>       worker_thread+0x180/0x430
>>>>>>       kthread+0x11c/0x120
>>>>>>      Kernel panic - not syncing: softlockup: hung tasks
>>>>>>      CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
>>>>>>      Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
>>>>>>      Workqueue: events_unbound flush_to_ldisc
>>>>>>      Call trace:
>>>>>>       dump_backtrace+0x0/0x1ec
>>>>>>       show_stack+0x24/0x30
>>>>>>       dump_stack+0xd0/0x128
>>>>>>       panic+0x15c/0x374
>>>>>>       watchdog_timer_fn+0x2b8/0x304
>>>>>>       __run_hrtimer+0x88/0x2c0
>>>>>>       __hrtimer_run_queues+0xa4/0x120
>>>>>>       hrtimer_interrupt+0xfc/0x270
>>>>>>       arch_timer_handler_phys+0x40/0x50
>>>>>>       handle_percpu_devid_irq+0x94/0x220
>>>>>>       __handle_domain_irq+0x88/0xf0
>>>>>>       gic_handle_irq+0x84/0xfc
>>>>>>       el1_irq+0xc8/0x180
>>>>>>       slip_unesc+0x80/0x214 [slip]
>>>>>>       tty_ldisc_receive_buf+0x64/0x80
>>>>>>       tty_port_default_receive_buf+0x50/0x90
>>>>>>       flush_to_ldisc+0xbc/0x110
>>>>>>       process_one_work+0x1d4/0x4b0
>>>>>>       worker_thread+0x180/0x430
>>>>>>       kthread+0x11c/0x120
>>>>>>      SMP: stopping secondary CPUs
>>>>>>
>>>>>> In the testcase pty04, there are multple processes and we only pay close attention to the
>>>>>> first three actually. The first process call the write syscall to send data to the pty master
>>>>>> with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
>>>>>> The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
>>>>>> The third process will wait a moment in which the first two processes will do there work and then
>>>>>> it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
>>>>>> Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
>>>>>> time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
>>>>>> master's buffhead to line discipline in a loop until there is no more data left without any on one's
>>>>>> own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
>>>>>> without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
>>>>>> in the flush_to_ldisc while loop to avoid it.
>>>>> Please properly wrap your changelog text at 72 columns.
>>>> When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
>>>> which look like this one:
>>>> Call trace:
>>>>      dump_backtrace+0x0/0x1ec
>>>>      show_stack+0x24/0x30
>>>>      dump_stack+0xd0/0x128
>>>>      panic+0x15c/0x374
>>>>      watchdog_timer_fn+0x2b8/0x304
>>>>      __run_hrtimer+0x88/0x2c0
>>>>      __hrtimer_run_queues+0xa4/0x120
>>>>      hrtimer_interrupt+0xfc/0x270
>>>>      arch_timer_handler_phys+0x40/0x50
>>>>      handle_percpu_devid_irq+0x94/0x220
>>>>      __handle_domain_irq+0x88/0xf0
>>>>      gic_handle_irq+0x84/0xfc
>>>>      el1_irq+0xc8/0x180
>>>>      slip_unesc+0x80/0x214 [slip]
>>>>      tty_ldisc_receive_buf+0x64/0x80
>>>>      tty_port_default_receive_buf+0x50/0x90
>>>>      flush_to_ldisc+0xbc/0x110
>>>>      process_one_work+0x1d4/0x4b0
>>>>      worker_thread+0x180/0x430
>>>>      kthread+0x11c/0x120
>>>>
>>>> In the testcase pty04, The first process call the write syscall to send data to the pty master.
>>>> At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
>>>> in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
>>>> long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.
>>> Is this a "real" test for something that you have seen in a normal
>>> workload?  ltp is known for having buggy/confusing tests in it in the
>>> past, you might wish to consult with the authors of that test.
>> Firstly, thanks for your response.
>>
>> I have check the ltp pty testcase. At the same time, I find the pty
>> softlockup in arm64, and it is similar to others.
>>
>> https://github.com/victronenergy/venus/issues/350
>>
>> https://groups.google.com/g/syzkaller-lts-bugs/c/SpkH8yH26js/m/3aifBl_GAwAJ
> Hm, ok, can you please resubmit this based on the changes discussed in
> thread and I will re-review it again.
>
> thanks,
>
> greg k-h

When I run the ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
which look like this one:
Call trace:
     dump_backtrace+0x0/0x1ec
     show_stack+0x24/0x30
     dump_stack+0xd0/0x128
     panic+0x15c/0x374
     watchdog_timer_fn+0x2b8/0x304
     __run_hrtimer+0x88/0x2c0
     __hrtimer_run_queues+0xa4/0x120
     hrtimer_interrupt+0xfc/0x270
     arch_timer_handler_phys+0x40/0x50
     handle_percpu_devid_irq+0x94/0x220
     __handle_domain_irq+0x88/0xf0
     gic_handle_irq+0x84/0xfc
     el1_irq+0xc8/0x180
     slip_unesc+0x80/0x214 [slip]
     tty_ldisc_receive_buf+0x64/0x80
     tty_port_default_receive_buf+0x50/0x90
     flush_to_ldisc+0xbc/0x110
     process_one_work+0x1d4/0x4b0
     worker_thread+0x180/0x430
     kthread+0x11c/0x120

In the testcase pty04, The first process call the write syscall to send data to the pty master.
At the same time, the workqueue will do the flush_to_ldisc to pop data in a loop until there is
no more data left. When the sender and workqueue running in different core, the sender sends data
fastly in full time which will result in workqueue doing work in loop for a long time and occuring
softlockup in flush_to_ldisc with kernel configured without CONFIG_PREEMPT. So I add need_resched
check and cond_resched in the flush_to_ldisc loop to avoid it.

Signed-off-by: Guanghui Feng<guanghuifeng@linux.alibaba.com>


diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
index bd2d915..77b92f9 100644
--- a/drivers/tty/tty_buffer.c
+++ b/drivers/tty/tty_buffer.c
@@ -534,6 +534,8 @@ static void flush_to_ldisc(struct work_struct *work)
  		if (!count)
  			break;
  		head->read += count;
+		if (need_resched())
+			cond_resched();
  	}
  
  	mutex_unlock(&buf->lock);
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread
* Re: [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
  2021-10-11 10:18           ` guanghui.fgh
@ 2021-10-11 10:54             ` Greg KH
  0 siblings, 0 replies; 9+ messages in thread
From: Greg KH @ 2021-10-11 10:54 UTC (permalink / raw)
  To: guanghui.fgh
  Cc: jirislaby, baolin.wang, linux-kernel, zhuo.song, zhangliguang

On Mon, Oct 11, 2021 at 06:18:41PM +0800, guanghui.fgh wrote:
> 
> 在 2021/10/11 15:50, Greg KH 写道:
> > On Mon, Oct 11, 2021 at 03:42:38PM +0800, guanghui.fgh wrote:
> > > 在 2021/10/10 21:18, Greg KH 写道:
> > > > On Fri, Oct 08, 2021 at 03:50:15PM +0800, guanghui.fgh wrote:
> > > > > 在 2021/9/30 13:38, Greg KH 写道:
> > > > > > On Thu, Sep 30, 2021 at 11:11:38AM +0800, Guanghui Feng wrote:
> > > > > > > When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> > > > > > > which look like this one:
> > > > > > > 
> > > > > > >      watchdog: BUG: soft lockup - CPU#41 stuck for 67s! [kworker/u192:2:106867]
> > > > > > >      CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OE     5.10.23 #1
> > > > > > >      Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > > > > > >      Workqueue: events_unbound flush_to_ldisc
> > > > > > >      pstate: 00c00009 (nzcv daif +PAN +UAO -TCO BTYPE=--)
> > > > > > >      pc : slip_unesc+0x80/0x214 [slip]
> > > > > > >      lr : slip_receive_buf+0x84/0x100 [slip]
> > > > > > >      sp : ffff80005274bce0
> > > > > > >      x29: ffff80005274bce0 x28: 0000000000000000
> > > > > > >      x27: ffff00525626fcc8 x26: ffff800011921078
> > > > > > >      x25: 0000000000000000 x24: 0000000000000004
> > > > > > >      x23: ffff00402b4059c0 x22: ffff00402b405940
> > > > > > >      x21: ffff205d87b81e21 x20: ffff205d87b81b9b
> > > > > > >      x19: 0000000000000000 x18: 0000000000000000
> > > > > > >      x17: 0000000000000000 x16: 0000000000000000
> > > > > > >      x15: 0000000000000000 x14: 5f5f5f5f5f5f5f5f
> > > > > > >      x13: 5f5f5f5f5f5f5f5f x12: 5f5f5f5f5f5f5f5f
> > > > > > >      x11: 5f5f5f5f5f5f5f5f x10: 5f5f5f5f5f5f5f5f
> > > > > > >      x9 : ffff8000097d7628 x8 : ffff205d87b85e20
> > > > > > >      x7 : 0000000000000000 x6 : 0000000000000001
> > > > > > >      x5 : ffff8000097dc008 x4 : ffff8000097d75a4
> > > > > > >      x3 : ffff205d87b81e1f x2 : 0000000000000005
> > > > > > >      x1 : 000000000000005f x0 : ffff00402b405940
> > > > > > >      Call trace:
> > > > > > >       slip_unesc+0x80/0x214 [slip]
> > > > > > >       tty_ldisc_receive_buf+0x64/0x80
> > > > > > >       tty_port_default_receive_buf+0x50/0x90
> > > > > > >       flush_to_ldisc+0xbc/0x110
> > > > > > >       process_one_work+0x1d4/0x4b0
> > > > > > >       worker_thread+0x180/0x430
> > > > > > >       kthread+0x11c/0x120
> > > > > > >      Kernel panic - not syncing: softlockup: hung tasks
> > > > > > >      CPU: 41 PID: 106867 Comm: kworker/u192:2 Kdump: loaded Tainted: G           OEL    5.10.23 #1
> > > > > > >      Hardware name: H3C R4960 G3/BC82AMDDA, BIOS 1.70 01/07/2021
> > > > > > >      Workqueue: events_unbound flush_to_ldisc
> > > > > > >      Call trace:
> > > > > > >       dump_backtrace+0x0/0x1ec
> > > > > > >       show_stack+0x24/0x30
> > > > > > >       dump_stack+0xd0/0x128
> > > > > > >       panic+0x15c/0x374
> > > > > > >       watchdog_timer_fn+0x2b8/0x304
> > > > > > >       __run_hrtimer+0x88/0x2c0
> > > > > > >       __hrtimer_run_queues+0xa4/0x120
> > > > > > >       hrtimer_interrupt+0xfc/0x270
> > > > > > >       arch_timer_handler_phys+0x40/0x50
> > > > > > >       handle_percpu_devid_irq+0x94/0x220
> > > > > > >       __handle_domain_irq+0x88/0xf0
> > > > > > >       gic_handle_irq+0x84/0xfc
> > > > > > >       el1_irq+0xc8/0x180
> > > > > > >       slip_unesc+0x80/0x214 [slip]
> > > > > > >       tty_ldisc_receive_buf+0x64/0x80
> > > > > > >       tty_port_default_receive_buf+0x50/0x90
> > > > > > >       flush_to_ldisc+0xbc/0x110
> > > > > > >       process_one_work+0x1d4/0x4b0
> > > > > > >       worker_thread+0x180/0x430
> > > > > > >       kthread+0x11c/0x120
> > > > > > >      SMP: stopping secondary CPUs
> > > > > > > 
> > > > > > > In the testcase pty04, there are multple processes and we only pay close attention to the
> > > > > > > first three actually. The first process call the write syscall to send data to the pty master
> > > > > > > with all one's strength(tty_write->file_tty_write->do_tty_write->n_tty_write call chain).
> > > > > > > The second process call the read syscall to receive data by the pty slave(with PF_PACKET socket).
> > > > > > > The third process will wait a moment in which the first two processes will do there work and then
> > > > > > > it call ioctl to hangup the pty pair which will cease the first two process read/write to the pty.
> > > > > > > Before hangup the pty, the first process send data to the pty buffhead with high speed. At the same
> > > > > > > time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data from pty
> > > > > > > master's buffhead to line discipline in a loop until there is no more data left without any on one's
> > > > > > > own schedule which will result in doing work in flush_to_ldisc for a long time. As kernel configured
> > > > > > > without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc. So I add cond_resched
> > > > > > > in the flush_to_ldisc while loop to avoid it.
> > > > > > Please properly wrap your changelog text at 72 columns.
> > > > > When I run ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> > > > > which look like this one:
> > > > > Call trace:
> > > > >      dump_backtrace+0x0/0x1ec
> > > > >      show_stack+0x24/0x30
> > > > >      dump_stack+0xd0/0x128
> > > > >      panic+0x15c/0x374
> > > > >      watchdog_timer_fn+0x2b8/0x304
> > > > >      __run_hrtimer+0x88/0x2c0
> > > > >      __hrtimer_run_queues+0xa4/0x120
> > > > >      hrtimer_interrupt+0xfc/0x270
> > > > >      arch_timer_handler_phys+0x40/0x50
> > > > >      handle_percpu_devid_irq+0x94/0x220
> > > > >      __handle_domain_irq+0x88/0xf0
> > > > >      gic_handle_irq+0x84/0xfc
> > > > >      el1_irq+0xc8/0x180
> > > > >      slip_unesc+0x80/0x214 [slip]
> > > > >      tty_ldisc_receive_buf+0x64/0x80
> > > > >      tty_port_default_receive_buf+0x50/0x90
> > > > >      flush_to_ldisc+0xbc/0x110
> > > > >      process_one_work+0x1d4/0x4b0
> > > > >      worker_thread+0x180/0x430
> > > > >      kthread+0x11c/0x120
> > > > > 
> > > > > In the testcase pty04, The first process call the write syscall to send data to the pty master.
> > > > > At the same time if the workqueue is waken up, the workqueue will do the flush_to_ldisc to pop data
> > > > > in a loop until there is no more data left which will result in doing work in flush_to_ldisc for a
> > > > > long time. As kernel configured without CONFIG_PREEMPT, there maybe occurs softlockup in the flush_to_ldisc.
> > > > Is this a "real" test for something that you have seen in a normal
> > > > workload?  ltp is known for having buggy/confusing tests in it in the
> > > > past, you might wish to consult with the authors of that test.
> > > Firstly, thanks for your response.
> > > 
> > > I have check the ltp pty testcase. At the same time, I find the pty
> > > softlockup in arm64, and it is similar to others.
> > > 
> > > https://github.com/victronenergy/venus/issues/350
> > > 
> > > https://groups.google.com/g/syzkaller-lts-bugs/c/SpkH8yH26js/m/3aifBl_GAwAJ
> > Hm, ok, can you please resubmit this based on the changes discussed in
> > thread and I will re-review it again.
> > 
> > thanks,
> > 
> > greg k-h
> 
> When I run the ltp testcase(ltp/testcases/kernel/pty/pty04.c) with arm64, there is a soft lockup,
> which look like this one:
> Call trace:
>     dump_backtrace+0x0/0x1ec
>     show_stack+0x24/0x30
>     dump_stack+0xd0/0x128
>     panic+0x15c/0x374
>     watchdog_timer_fn+0x2b8/0x304
>     __run_hrtimer+0x88/0x2c0
>     __hrtimer_run_queues+0xa4/0x120
>     hrtimer_interrupt+0xfc/0x270
>     arch_timer_handler_phys+0x40/0x50
>     handle_percpu_devid_irq+0x94/0x220
>     __handle_domain_irq+0x88/0xf0
>     gic_handle_irq+0x84/0xfc
>     el1_irq+0xc8/0x180
>     slip_unesc+0x80/0x214 [slip]
>     tty_ldisc_receive_buf+0x64/0x80
>     tty_port_default_receive_buf+0x50/0x90
>     flush_to_ldisc+0xbc/0x110
>     process_one_work+0x1d4/0x4b0
>     worker_thread+0x180/0x430
>     kthread+0x11c/0x120
> 
> In the testcase pty04, The first process call the write syscall to send data to the pty master.
> At the same time, the workqueue will do the flush_to_ldisc to pop data in a loop until there is
> no more data left. When the sender and workqueue running in different core, the sender sends data
> fastly in full time which will result in workqueue doing work in loop for a long time and occuring
> softlockup in flush_to_ldisc with kernel configured without CONFIG_PREEMPT. So I add need_resched
> check and cond_resched in the flush_to_ldisc loop to avoid it.
> 
> Signed-off-by: Guanghui Feng<guanghuifeng@linux.alibaba.com>
> 
> 
> diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
> index bd2d915..77b92f9 100644
> --- a/drivers/tty/tty_buffer.c
> +++ b/drivers/tty/tty_buffer.c
> @@ -534,6 +534,8 @@ static void flush_to_ldisc(struct work_struct *work)
>  		if (!count)
>  			break;
>  		head->read += count;
> +		if (need_resched())
> +			cond_resched();
>  	}
>  	mutex_unlock(&buf->lock);
> -- 
> 1.8.3.1
> 

This is in no format that I can apply it in.  You ignored my request to
properly wrap your changelog text lines, and your signed-off-by needs a
space after the name, and it's in response to another patch and is not
stand alone, and it is a v2 patch with no list of changes from the
previous one.

Please fix up.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-11 10:54 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-30  3:11 [PATCH] tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc Guanghui Feng
2021-09-30  5:38 ` Greg KH
2021-09-30  6:02   ` Baolin Wang
2021-10-08  7:50   ` guanghui.fgh
2021-10-10 13:18     ` Greg KH
2021-10-11  7:42       ` guanghui.fgh
2021-10-11  7:50         ` Greg KH
2021-10-11 10:18           ` guanghui.fgh
2021-10-11 10:54             ` Greg KH

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.