acm_port_shutdown hangs for 30 seconds

acm_port_shutdown hangs for 30 seconds

[Jookia]

Hello there,

I've been banging my head against this issue over the years but sat down
and started to debug it today.

When I try to quit GNU screen, minicom or another serial program it
sometimes hangs for around 30 seconds.

To reproduce I do this:

1. Connect an Arduino Micro with stock LED blink firmware
2. Run 'screen /dev/ttyACM1 9600'
3. Type some letters (no response from the board is given)
4. Quit in some way

If I skip step 3 (typing letters), the hang does not happen.

In userspace the hang happens at a call to close() on the TTY, and using
perf and ftrace it looks to be spending a lot of time poisoning urbs in
acm_port_shutdown.
This kind of makes sense to me as there is some in flight data to be
sent and there's no acknowledgement coming from the chip.

I've had this issue with embedded Linux boards too that for some reason
or another have stopped functioning.
The current solution I have is to forcefully unplug the board, but
that's not ideal.

My questions are:

- Is this a bug?
- Can I reduce the timeout somehow?

Thanks for your time,
Jookia.

Re: acm_port_shutdown hangs for 30 seconds

[Greg KH]

On Sun, Mar 13, 2022 at 12:58:28AM +1100, Jookia wrote:
> Hello there,
> 
> I've been banging my head against this issue over the years but sat down
> and started to debug it today.
> 
> When I try to quit GNU screen, minicom or another serial program it
> sometimes hangs for around 30 seconds.
> 
> To reproduce I do this:
> 
> 1. Connect an Arduino Micro with stock LED blink firmware
> 2. Run 'screen /dev/ttyACM1 9600'
> 3. Type some letters (no response from the board is given)
> 4. Quit in some way
> 
> If I skip step 3 (typing letters), the hang does not happen.
> 
> In userspace the hang happens at a call to close() on the TTY, and using
> perf and ftrace it looks to be spending a lot of time poisoning urbs in
> acm_port_shutdown.
> This kind of makes sense to me as there is some in flight data to be
> sent and there's no acknowledgement coming from the chip.
> 
> I've had this issue with embedded Linux boards too that for some reason
> or another have stopped functioning.
> The current solution I have is to forcefully unplug the board, but
> that's not ideal.
> 
> My questions are:
> 
> - Is this a bug?

It's a bug in your hardware, yes.

> - Can I reduce the timeout somehow?

You could, but then you risk the problem of hardware that needs delays
to not work properly.

What host controller driver is this showing up on?

thanks,

greg k-h

Re: acm_port_shutdown hangs for 30 seconds

[Jookia]

Hi Greg,

Thanks for getting back to this!

On Mon, Jun 27, 2022 at 02:57:17PM +0200, Greg KH wrote:
> What host controller driver is this showing up on?

I have the following modules loaded:

xhci_pci_renesas
xhci_pci
xhci_hcd

This isn't an ARM board or anything, this is a standard x86 desktop
(B450 plus motherboard, Ryzen 3700x chipset)- but I believe I've had the
same error when using serial cables on a years older computer and in
general when doing embedded development.

I believe this issue is related to buggy serial devices, in particular
ones run by a microcontroller like an ATmega32u4 and perhaps Linux USB
gadget where the device is abruptly reset but the serial session isn't
terminated.

The host device USB system has to time out then, which is fine, but in
userspace this seems to hang close() instead of doing this
asynchronously.

> thanks,
> 
> greg k-h

Jookia.

Re: acm_port_shutdown hangs for 30 seconds

[Greg KH]

On Tue, Jun 28, 2022 at 07:21:48PM +1000, Jookia wrote:
> Hi Greg,
> 
> Thanks for getting back to this!
> 
> On Mon, Jun 27, 2022 at 02:57:17PM +0200, Greg KH wrote:
> > What host controller driver is this showing up on?
> 
> I have the following modules loaded:
> 
> xhci_pci_renesas
> xhci_pci
> xhci_hcd
> 
> This isn't an ARM board or anything, this is a standard x86 desktop
> (B450 plus motherboard, Ryzen 3700x chipset)- but I believe I've had the
> same error when using serial cables on a years older computer and in
> general when doing embedded development.
> 
> I believe this issue is related to buggy serial devices, in particular
> ones run by a microcontroller like an ATmega32u4 and perhaps Linux USB
> gadget where the device is abruptly reset but the serial session isn't
> terminated.
> 
> The host device USB system has to time out then, which is fine, but in
> userspace this seems to hang close() instead of doing this
> asynchronously.

The close is timing out as it tries to flush the data to the device
which is no longer responding.  So this kind of is expected behavior,
sorry.

greg k-h

Re: acm_port_shutdown hangs for 30 seconds

[Jookia]

On Wed, Jun 29, 2022 at 10:37:55AM +0200, Greg KH wrote:
> The close is timing out as it tries to flush the data to the device
> which is no longer responding.  So this kind of is expected behavior,
> sorry.

That makes sense. I'm guessing that changing this behaviour to be
asynchronous would break applications that rely on the return code from
close() to say EIO if it does that.

> 
> greg k-h

Jookia.

Re: acm_port_shutdown hangs for 30 seconds

[Oliver Neukum]

On 12.03.22 14:58, Jookia wrote:
> Hello there,
> 
> I've been banging my head against this issue over the years but sat down
> and started to debug it today.
> 
> When I try to quit GNU screen, minicom or another serial program it
> sometimes hangs for around 30 seconds.
> 
> To reproduce I do this:
> 
> 1. Connect an Arduino Micro with stock LED blink firmware
> 2. Run 'screen /dev/ttyACM1 9600'
> 3. Type some letters (no response from the board is given)
> 4. Quit in some way
> 
> If I skip step 3 (typing letters), the hang does not happen.

You have something in the buffer, which the tty layer will try to send.

> In userspace the hang happens at a call to close() on the TTY, and using

The tty layer is supposed to wait a defined amount of time
if a tty needs to drain.
This amount (among many other things) is traditionally
set with the TIOCSSERIAL ioctl()

> perf and ftrace it looks to be spending a lot of time poisoning urbs in
> acm_port_shutdown.

That is a bit odd.

> - Is this a bug?

Maybe, but unlikely. If and only it takes much more time than the
termios say.

> - Can I reduce the timeout somehow?

ioctl() TIOCSSERIAL. If that does not work,
it's a bug.

	HTH
		Oliver

Re: acm_port_shutdown hangs for 30 seconds

[Jookia]

On Wed, Jun 29, 2022 at 02:01:18PM +0200, Oliver Neukum wrote:
> You have something in the buffer, which the tty layer will try to send.
> 
> > In userspace the hang happens at a call to close() on the TTY, and using
> 
> The tty layer is supposed to wait a defined amount of time
> if a tty needs to drain.
> This amount (among many other things) is traditionally
> set with the TIOCSSERIAL ioctl()
> 
> > perf and ftrace it looks to be spending a lot of time poisoning urbs in
> > acm_port_shutdown.
> 
> That is a bit odd.
> 
> > - Is this a bug?
> 
> Maybe, but unlikely. If and only it takes much more time than the
> termios say.
> 
> > - Can I reduce the timeout somehow?
> 
> ioctl() TIOCSSERIAL. If that does not work,
> it's a bug.

This sounds like exactly what's happening. Next time I hit this issue
I'll see if changing closing_wait fixes this.

> 	HTH
> 		Oliver