[PATCH v2] serial: port: Don't suspend if the port is still busy

[PATCH v2] serial: port: Don't suspend if the port is still busy

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

We accidently met the issue that the bash prompt is not shown after the
previous command done and until the next input if there's only one CPU
(In our issue other CPUs are isolated by isolcpus=). Further analysis
shows it's because the port entering runtime suspend even if there's
still pending chars in the buffer and the pending chars will only be
processed in next device resuming. We are using amba-pl011 and the
problematic flow is like below:

Bash                                         kworker
tty_write()
  file_tty_write()
    n_tty_write()
      uart_write()
        __uart_start()
          pm_runtime_get() // wakeup waker
            queue_work()
                                             pm_runtime_work()
                                               rpm_resume()
                                                status = RPM_RESUMING
                                                serial_port_runtime_resume()
                                                  port->ops->start_tx()
                                                    pl011_tx_chars()
                                                      uart_write_wakeup()
        […]
        __uart_start()
          pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
                               // later data are not commit to the port driver
                                                status = RPM_ACTIVE
                                                rpm_idle() -> rpm_suspend()

This patch tries to fix this by checking the port busy before entering
runtime suspending. A runtime_suspend callback is added for the port
driver. When entering runtime suspend the callback is invoked, if there's
still pending chars in the buffer then flush the buffer.

Cc: Tony Lindgren <tony@atomide.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
Change since v1:
- Use port lock wrapper per John
- Flush the pending chars and return -EBUSY per Tony.
Thanks.
Link: https://lore.kernel.org/all/20240204031957.58176-1-yangyicong@huawei.com/

 drivers/tty/serial/serial_port.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/tty/serial/serial_port.c b/drivers/tty/serial/serial_port.c
index 88975a4df306..0617d5158235 100644
--- a/drivers/tty/serial/serial_port.c
+++ b/drivers/tty/serial/serial_port.c
@@ -46,8 +46,32 @@ static int serial_port_runtime_resume(struct device *dev)
 	return 0;
 }
 
+static int serial_port_runtime_suspend(struct device *dev)
+{
+	struct serial_port_device *port_dev = to_serial_base_port_device(dev);
+	struct uart_port *port;
+	unsigned long flags;
+	int ret = 0;
+
+	port = port_dev->port;
+
+	if (port->flags & UPF_DEAD)
+		return ret;
+
+	uart_port_lock_irqsave(port, &flags);
+	if (__serial_port_busy(port)) {
+		port->ops->start_tx(port);
+		pm_runtime_mark_last_busy(dev);
+		ret = -EBUSY;
+	}
+	uart_port_unlock_irqrestore(port, flags);
+
+	return ret;
+}
+
 static DEFINE_RUNTIME_DEV_PM_OPS(serial_port_pm,
-				 NULL, serial_port_runtime_resume, NULL);
+				 serial_port_runtime_suspend,
+				 serial_port_runtime_resume, NULL);
 
 static int serial_port_probe(struct device *dev)
 {
-- 
2.24.0

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Tony Lindgren]

* Yicong Yang <yangyicong@huawei.com> [240206 09:37]:
> Change since v1:
> - Use port lock wrapper per John
> - Flush the pending chars and return -EBUSY per Tony.

Looks good to me thanks:

Reviewed-by: Tony Lindgren <tony@atomide.com>

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Greg KH]

On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:
> From: Yicong Yang <yangyicong@hisilicon.com>
> 
> We accidently met the issue that the bash prompt is not shown after the
> previous command done and until the next input if there's only one CPU
> (In our issue other CPUs are isolated by isolcpus=). Further analysis
> shows it's because the port entering runtime suspend even if there's
> still pending chars in the buffer and the pending chars will only be
> processed in next device resuming. We are using amba-pl011 and the
> problematic flow is like below:
> 
> Bash                                         kworker
> tty_write()
>   file_tty_write()
>     n_tty_write()
>       uart_write()
>         __uart_start()
>           pm_runtime_get() // wakeup waker
>             queue_work()
>                                              pm_runtime_work()
>                                                rpm_resume()
>                                                 status = RPM_RESUMING
>                                                 serial_port_runtime_resume()
>                                                   port->ops->start_tx()
>                                                     pl011_tx_chars()
>                                                       uart_write_wakeup()
>         […]
>         __uart_start()
>           pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
>                                // later data are not commit to the port driver
>                                                 status = RPM_ACTIVE
>                                                 rpm_idle() -> rpm_suspend()
> 
> This patch tries to fix this by checking the port busy before entering
> runtime suspending. A runtime_suspend callback is added for the port
> driver. When entering runtime suspend the callback is invoked, if there's
> still pending chars in the buffer then flush the buffer.
> 
> Cc: Tony Lindgren <tony@atomide.com>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>

Is this a regression that was caused by the port code?  If so, what
commit id does this fix?  Should it be backported to older kernels?

thanks,

greg k-h

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Yicong Yang]

On 2024/2/6 17:44, Greg KH wrote:
> On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:
>> From: Yicong Yang <yangyicong@hisilicon.com>
>>
>> We accidently met the issue that the bash prompt is not shown after the
>> previous command done and until the next input if there's only one CPU
>> (In our issue other CPUs are isolated by isolcpus=). Further analysis
>> shows it's because the port entering runtime suspend even if there's
>> still pending chars in the buffer and the pending chars will only be
>> processed in next device resuming. We are using amba-pl011 and the
>> problematic flow is like below:
>>
>> Bash                                         kworker
>> tty_write()
>>   file_tty_write()
>>     n_tty_write()
>>       uart_write()
>>         __uart_start()
>>           pm_runtime_get() // wakeup waker
>>             queue_work()
>>                                              pm_runtime_work()
>>                                                rpm_resume()
>>                                                 status = RPM_RESUMING
>>                                                 serial_port_runtime_resume()
>>                                                   port->ops->start_tx()
>>                                                     pl011_tx_chars()
>>                                                       uart_write_wakeup()
>>         […]
>>         __uart_start()
>>           pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
>>                                // later data are not commit to the port driver
>>                                                 status = RPM_ACTIVE
>>                                                 rpm_idle() -> rpm_suspend()
>>
>> This patch tries to fix this by checking the port busy before entering
>> runtime suspending. A runtime_suspend callback is added for the port
>> driver. When entering runtime suspend the callback is invoked, if there's
>> still pending chars in the buffer then flush the buffer.
>>
>> Cc: Tony Lindgren <tony@atomide.com>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> 
> Is this a regression that was caused by the port code?  If so, what
> commit id does this fix?  Should it be backported to older kernels?
> 

Sorry for missing it. The fix tag should be:

Fixes: 84a9582fd203 ("serial: core: Start managing serial controllers to enable runtime PM")

Thanks.

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Andy Shevchenko]

On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:
> From: Yicong Yang <yangyicong@hisilicon.com>
> 
> We accidently met the issue that the bash prompt is not shown after the
> previous command done and until the next input if there's only one CPU
> (In our issue other CPUs are isolated by isolcpus=). Further analysis
> shows it's because the port entering runtime suspend even if there's
> still pending chars in the buffer and the pending chars will only be
> processed in next device resuming. We are using amba-pl011 and the
> problematic flow is like below:
> 
> Bash                                         kworker
> tty_write()
>   file_tty_write()
>     n_tty_write()
>       uart_write()
>         __uart_start()
>           pm_runtime_get() // wakeup waker
>             queue_work()
>                                              pm_runtime_work()
>                                                rpm_resume()
>                                                 status = RPM_RESUMING
>                                                 serial_port_runtime_resume()
>                                                   port->ops->start_tx()
>                                                     pl011_tx_chars()
>                                                       uart_write_wakeup()
>         […]
>         __uart_start()
>           pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
>                                // later data are not commit to the port driver
>                                                 status = RPM_ACTIVE
>                                                 rpm_idle() -> rpm_suspend()
> 
> This patch tries to fix this by checking the port busy before entering
> runtime suspending. A runtime_suspend callback is added for the port
> driver. When entering runtime suspend the callback is invoked, if there's
> still pending chars in the buffer then flush the buffer.

...

> +static int serial_port_runtime_suspend(struct device *dev)
> +{
> +	struct serial_port_device *port_dev = to_serial_base_port_device(dev);
> +	struct uart_port *port;
> +	unsigned long flags;
> +	int ret = 0;
> +
> +	port = port_dev->port;
> +
> +	if (port->flags & UPF_DEAD)
> +		return ret;
> +
> +	uart_port_lock_irqsave(port, &flags);
> +	if (__serial_port_busy(port)) {
> +		port->ops->start_tx(port);

> +		pm_runtime_mark_last_busy(dev);

Do you think we need to call this under a lock?

> +		ret = -EBUSY;
> +	}
> +	uart_port_unlock_irqrestore(port, flags);
> +
> +	return ret;
> +}

With the above I would rather write it as

static int __serial_port_busy(struct uart_port *port)
{
	if (uart_tx_stopped(port))
		return 0;

	if (uart_circ_chars_pending(&port->state->xmit)
		return -EBUSY;

	return 0;
}

static int serial_port_runtime_suspend(struct device *dev)
{
	int ret;
	...
	uart_port_lock_irqsave(port, &flags);
	ret = __serial_port_busy(port);
	if (ret)
		port->ops->start_tx(port);
	uart_port_unlock_irqrestore(port, flags);

	if (ret)
		pm_runtime_mark_last_busy(dev);

	return ret;
}

It also seems aligned with the resume implementation above.

...

For the consistency's sake the resume can be refactored as

static int serial_port_runtime_resume(struct device *dev)
{
	...
	int ret;
	...
	ret = __serial_port_busy(port);
	if (ret)
	...
}

but this can be done later.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Andy Shevchenko]

On Tue, Feb 06, 2024 at 03:09:32PM +0200, Andy Shevchenko wrote:
> On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:
> > From: Yicong Yang <yangyicong@hisilicon.com>
> > 
> > We accidently met the issue that the bash prompt is not shown after the
> > previous command done and until the next input if there's only one CPU
> > (In our issue other CPUs are isolated by isolcpus=). Further analysis
> > shows it's because the port entering runtime suspend even if there's
> > still pending chars in the buffer and the pending chars will only be
> > processed in next device resuming. We are using amba-pl011 and the
> > problematic flow is like below:
> > 
> > Bash                                         kworker
> > tty_write()
> >   file_tty_write()
> >     n_tty_write()
> >       uart_write()
> >         __uart_start()
> >           pm_runtime_get() // wakeup waker
> >             queue_work()
> >                                              pm_runtime_work()
> >                                                rpm_resume()
> >                                                 status = RPM_RESUMING
> >                                                 serial_port_runtime_resume()
> >                                                   port->ops->start_tx()
> >                                                     pl011_tx_chars()
> >                                                       uart_write_wakeup()
> >         […]
> >         __uart_start()
> >           pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
> >                                // later data are not commit to the port driver
> >                                                 status = RPM_ACTIVE
> >                                                 rpm_idle() -> rpm_suspend()
> > 
> > This patch tries to fix this by checking the port busy before entering
> > runtime suspending. A runtime_suspend callback is added for the port
> > driver. When entering runtime suspend the callback is invoked, if there's
> > still pending chars in the buffer then flush the buffer.

...

> > +static int serial_port_runtime_suspend(struct device *dev)
> > +{
> > +	struct serial_port_device *port_dev = to_serial_base_port_device(dev);
> > +	struct uart_port *port;
> > +	unsigned long flags;
> > +	int ret = 0;
> > +
> > +	port = port_dev->port;
> > +
> > +	if (port->flags & UPF_DEAD)
> > +		return ret;
> > +
> > +	uart_port_lock_irqsave(port, &flags);
> > +	if (__serial_port_busy(port)) {
> > +		port->ops->start_tx(port);
> 
> > +		pm_runtime_mark_last_busy(dev);
> 
> Do you think we need to call this under a lock?
> 
> > +		ret = -EBUSY;
> > +	}
> > +	uart_port_unlock_irqrestore(port, flags);
> > +
> > +	return ret;
> > +}
> 
> With the above I would rather write it as
> 
> static int __serial_port_busy(struct uart_port *port)
> {
> 	if (uart_tx_stopped(port))
> 		return 0;
> 
> 	if (uart_circ_chars_pending(&port->state->xmit)
> 		return -EBUSY;
> 
> 	return 0;
> }
> 
> static int serial_port_runtime_suspend(struct device *dev)
> {
> 	int ret;
> 	...
> 	uart_port_lock_irqsave(port, &flags);
> 	ret = __serial_port_busy(port);
> 	if (ret)
> 		port->ops->start_tx(port);
> 	uart_port_unlock_irqrestore(port, flags);

> 	if (ret)
> 		pm_runtime_mark_last_busy(dev);

And obvious question here: why in case of 0 we can't mark this as busy as well?
I.o.w. why do we need to mark it only when error is set?

> 	return ret;
> }
> 
> It also seems aligned with the resume implementation above.
> 
> ...
> 
> For the consistency's sake the resume can be refactored as
> 
> static int serial_port_runtime_resume(struct device *dev)
> {
> 	...
> 	int ret;
> 	...
> 	ret = __serial_port_busy(port);
> 	if (ret)
> 	...
> }
> 
> but this can be done later.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Tony Lindgren]

* Andy Shevchenko <andriy.shevchenko@linux.intel.com> [240206 13:12]:
> > static int serial_port_runtime_suspend(struct device *dev)
> > {
> > 	int ret;
> > 	...
> > 	uart_port_lock_irqsave(port, &flags);
> > 	ret = __serial_port_busy(port);
> > 	if (ret)
> > 		port->ops->start_tx(port);
> > 	uart_port_unlock_irqrestore(port, flags);
> 
> > 	if (ret)
> > 		pm_runtime_mark_last_busy(dev);
> 
> And obvious question here: why in case of 0 we can't mark this as busy as well?
> I.o.w. why do we need to mark it only when error is set?

No need to call in the 0 case. The last time driver was busy
was when pm_runtime_mark_last_busy() was called, and in the 0 case
we just runtime suspend based on the autosuspend timeout value.

Regards,

Tony

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Yicong Yang]

Hi Andy,

On 2024/2/6 21:09, Andy Shevchenko wrote:
> On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:
>> From: Yicong Yang <yangyicong@hisilicon.com>
>>
>> We accidently met the issue that the bash prompt is not shown after the
>> previous command done and until the next input if there's only one CPU
>> (In our issue other CPUs are isolated by isolcpus=). Further analysis
>> shows it's because the port entering runtime suspend even if there's
>> still pending chars in the buffer and the pending chars will only be
>> processed in next device resuming. We are using amba-pl011 and the
>> problematic flow is like below:
>>
>> Bash                                         kworker
>> tty_write()
>>   file_tty_write()
>>     n_tty_write()
>>       uart_write()
>>         __uart_start()
>>           pm_runtime_get() // wakeup waker
>>             queue_work()
>>                                              pm_runtime_work()
>>                                                rpm_resume()
>>                                                 status = RPM_RESUMING
>>                                                 serial_port_runtime_resume()
>>                                                   port->ops->start_tx()
>>                                                     pl011_tx_chars()
>>                                                       uart_write_wakeup()
>>         […]
>>         __uart_start()
>>           pm_runtime_get() < 0 // because runtime status = RPM_RESUMING
>>                                // later data are not commit to the port driver
>>                                                 status = RPM_ACTIVE
>>                                                 rpm_idle() -> rpm_suspend()
>>
>> This patch tries to fix this by checking the port busy before entering
>> runtime suspending. A runtime_suspend callback is added for the port
>> driver. When entering runtime suspend the callback is invoked, if there's
>> still pending chars in the buffer then flush the buffer.
> 
> ...
> 
>> +static int serial_port_runtime_suspend(struct device *dev)
>> +{
>> +	struct serial_port_device *port_dev = to_serial_base_port_device(dev);
>> +	struct uart_port *port;
>> +	unsigned long flags;
>> +	int ret = 0;
>> +
>> +	port = port_dev->port;
>> +
>> +	if (port->flags & UPF_DEAD)
>> +		return ret;
>> +
>> +	uart_port_lock_irqsave(port, &flags);
>> +	if (__serial_port_busy(port)) {
>> +		port->ops->start_tx(port);
> 
>> +		pm_runtime_mark_last_busy(dev);
> 
> Do you think we need to call this under a lock?
> 

I just put this close to the ops->start_tx() where I used the device. Yes I have no
strong reason to put it in/with the lock region, but pm_runtime_mark_last_busy()
should be no costy and safe enough to put it in the spinlock region.

Any thoughts?

>> +		ret = -EBUSY;
>> +	}
>> +	uart_port_unlock_irqrestore(port, flags);
>> +
>> +	return ret;
>> +}
> 
> With the above I would rather write it as
> 
> static int __serial_port_busy(struct uart_port *port)
> {
> 	if (uart_tx_stopped(port))
> 		return 0;
> 
> 	if (uart_circ_chars_pending(&port->state->xmit)
> 		return -EBUSY;

I'm not sure but EBUSY seems not quite match here. EBUSY for
"Device or resource busy" so the device probably cannot be used
but we're testing whether the port is busy here. Hope I understand it
correctly.

> 
> 	return 0;
> }
> 
> static int serial_port_runtime_suspend(struct device *dev)
> {
> 	int ret;
> 	...
> 	uart_port_lock_irqsave(port, &flags);
> 	ret = __serial_port_busy(port);
> 	if (ret)
> 		port->ops->start_tx(port);
> 	uart_port_unlock_irqrestore(port, flags);
> 
> 	if (ret)
> 		pm_runtime_mark_last_busy(dev);
> 
> 	return ret;
> }
> 
> It also seems aligned with the resume implementation above.
> 
> ...
> 
> For the consistency's sake the resume can be refactored as
> 
> static int serial_port_runtime_resume(struct device *dev)
> {
> 	...
> 	int ret;
> 	...
> 	ret = __serial_port_busy(port);
> 	if (ret)
> 	...
> }
> 
> but this can be done later.
> 

I agree the refactoring should go to a separate patch. But it doesn't seem
to be more simpler or readable comparing to the current implementation? Just
want to narrowing the spinlock region?

Thanks.

Re: [PATCH v2] serial: port: Don't suspend if the port is still busy

[Andy Shevchenko]

On Wed, Feb 07, 2024 at 03:22:17PM +0800, Yicong Yang wrote:
> On 2024/2/6 21:09, Andy Shevchenko wrote:
> > On Tue, Feb 06, 2024 at 03:33:22PM +0800, Yicong Yang wrote:

...

> >> +		pm_runtime_mark_last_busy(dev);
> > 
> > Do you think we need to call this under a lock?
> 
> I just put this close to the ops->start_tx() where I used the device. Yes I have no
> strong reason to put it in/with the lock region, but pm_runtime_mark_last_busy()
> should be no costy and safe enough to put it in the spinlock region.
> 
> Any thoughts?

As I mentioned before, moving it out makes it similar to the resume
counterpart implementation.

...

> > With the above I would rather write it as
> > 
> > static int __serial_port_busy(struct uart_port *port)
> > {
> > 	if (uart_tx_stopped(port))
> > 		return 0;
> > 
> > 	if (uart_circ_chars_pending(&port->state->xmit)
> > 		return -EBUSY;
> 
> I'm not sure but EBUSY seems not quite match here. EBUSY for
> "Device or resource busy" so the device probably cannot be used
> but we're testing whether the port is busy here. Hope I understand it
> correctly.

Port is also "device" in the broader meaning. I don't see how this is
problematic. Prototype is originally int (while returning boolean).
I assume it was an idea behind similar (if not the same) as mine at
some point, but then vanished. Yet, the function itself can be renamed
to reflect these changes, like
__serial_port_get_status() // 0 - idling, -EBUSY - busy

> > 	return 0;
> > }
> > 
> > static int serial_port_runtime_suspend(struct device *dev)
> > {
> > 	int ret;
> > 	...
> > 	uart_port_lock_irqsave(port, &flags);
> > 	ret = __serial_port_busy(port);
> > 	if (ret)
> > 		port->ops->start_tx(port);
> > 	uart_port_unlock_irqrestore(port, flags);
> > 
> > 	if (ret)
> > 		pm_runtime_mark_last_busy(dev);
> > 
> > 	return ret;
> > }
> > 
> > It also seems aligned with the resume implementation above.
> > 
> > ...
> > 
> > For the consistency's sake the resume can be refactored as
> > 
> > static int serial_port_runtime_resume(struct device *dev)
> > {
> > 	...
> > 	int ret;
> > 	...
> > 	ret = __serial_port_busy(port);
> > 	if (ret)
> > 	...
> > }
> > 
> > but this can be done later.
> > 
> 
> I agree the refactoring should go to a separate patch. But it doesn't seem
> to be more simpler or readable comparing to the current implementation? Just
> want to narrowing the spinlock region?

Yes, at bare minimum I would expect the PM call be moved out of a lock.

As this seems a fix (and hence subject to backport) I would also minimize
invasion.

-- 
With Best Regards,
Andy Shevchenko