Re: Bugs: usb serial crash when close second comport

Re: Bugs: usb serial crash when close second comport

[Johan Hovold]

On Mon, Feb 01, 2021 at 08:31:13AM +0000, Pham Van Thanh Tung (GAM.VN.DAP) wrote:

> ***********************************************************************
> IMPORTANT NOTICE
> This email may contain confidential and/ or privileged information that belongs to FPT Software. If you are not the intended recipient or might have received this email by accident from an unreliable source, please notify the sender from FPT Software immediately and destroy this email. Keep in mind that any unauthorized copying, editing, disclosure or distribution of the material in this email is strictly forbidden, plus against the law by which FPT Software and involved clients abide.

Please resend without this footer, otherwise there's not much we can do
to help.

Johan

Re: Bugs: usb serial crash when close second comport

[Johan Hovold]

Hi Tung,

On Mon, Feb 01, 2021 at 10:02:18AM +0000, Tung Pham wrote:
> Dear MR Johan Hovold.
> 
> usb serial crash when close second comport
> Environment: Raspberry Pi 4 installed Ubuntu 20.10 (with the latest
> updates) ( kernel version: 5.8.0-1011)

Can you reproduce this with a mainline kernel?

Otherwise, you may need to bring this up with Ubuntu.

> -------------------------------------------
> I uses Raspberry Pi 4 to connect to HB-UM43 hub, then connect cp2108
> EK USB to serial board through the hub.
> After opening connections to both serial devices use screen command,
> the cp210x driver intermittently crashes on closing the second
> connection. The problem does not duplicate if opening only one of the
> connections. Repeatedly opening/closing just the first device works
> fine. Repeatedly opening/closing just the second device works fine.
> Opening both, and then trying to close both can trigger the issue. It
> doesn't matter what ordered they are opened or closed; closing down
> the second one can trigger the issue.
> I has also realized the same test with our cp2108 boards and the
> results are:
> 
> If I connects the Pi directly to the cp2108 EK board, the problem does
> not happen.
> If he uses an Raspberry Pi 4 connected to cp2108 boards through the
> HB-UM43 hub, the problem DOES happen.
> 
> I has also tried modifying cp210x driver downloaded from silabs web
> site, and i thinks the problem is rooted in cp210x_close() function as
> i explains
> "This is the cp210x_close() call from the cp210x driver:
> 
> static void cp210x_close(struct usb_serial_port *port)
> {
> struct cp210x_port_private *port_priv = usb_get_serial_port_data(port);
> usb_serial_generic_close(port); /* Clear both queues; cp2108 needs this to avoid an occasional hang */
> cp210x_write_u16_reg(port, CP210X_PURGE, PURGE_ALL);
> cp210x_write_u16_reg(port, CP210X_IFC_ENABLE, UART_DISABLE); /* Disabling the interface disables event-insertion mode. */
> port_priv->event_mode = false;
> }
> 
> It does basically four things: cancel any pending transactions on the
> BULK OUT and IN endpoints plus clear the OUT fifo
> (usb_serial_generic_close() call), send a request to clear the
> transmit and receive queues (CP210X_PURGE, PURGE_ALL), send a request
> to disable the UART (CP210X_IFC_ENABLE, UART_DISABLE), and then flag
> that the event_mode is off.
> 
> The issue occurs during the usb_serial_generic_close() call. Note that
> this call doesn't seem to actually close the port handle, but just
> does some cleanup in preparation for the close. The actual binary for
> this call comes from the OS; it is linked into the driver.
> 
> The driver also recovers if you unbind then bind the driver to the
> device. So it appears this is a software issue and not the silicon
> getting hung."
> 
> Question: Does the removal of this line usb_serial_generic_close() of
> code have any side effect? (e.g: make driver unstable, etc?), or is it
> an issue of the kernel, how i can fix this?.

This sounds like you could have a problem with the host-controller
driver. What controller is the RPi4 using?

As you write above, the usb_serial_generic_close() is needed to cancel
the outstanding URBs during close so you can't remove that.

But my guess is that something breaks in the HCD when unlinking those
URBs when connected through the hub. Do you have a stack trace from an
oops? Or do things just lock up?

Johan

RE: Bugs: usb serial crash when close second comport

[Tung Pham]

Dear mr Johan Hovold

Can you reproduce this with a mainline kernel?

Otherwise, you may need to bring this up with Ubuntu.

-> this bug doesn't occur on Ubuntu running on pc, it only occur on raspberry.
One additional thing: the bug doesn't occur with HUB: Orico, model W6ph4-u3-v1, Manufacture: VIA Labs, inc.
It only occur with hub: HB-UM43 hub, Genesys Logic, Inc.

This sounds like you could have a problem with the host-controller driver. What controller is the RPi4 using?

-> For the Pi 4, a fully-featured host controller drives the downstream USB ports. Downstream USB is provided by a Via Labs VL805 chip - that supports two USB 2.0 ports and two USB 3.0 ports

As you write above, the usb_serial_generic_close() is needed to cancel the outstanding URBs during close so you can't remove that.

But my guess is that something breaks in the HCD when unlinking those URBs when connected through the hub. Do you have a stack trace from an oops? Or do things just lock up?
-> I don't know how to capture stack trace?

Tung pham.

Re: Bugs: usb serial crash when close second comport

[Johan Hovold]

On Tue, Feb 02, 2021 at 03:50:50AM +0000, Tung Pham wrote:

> > Can you reproduce this with a mainline kernel?
> > 
> > Otherwise, you may need to bring this up with Ubuntu.
> 
> this bug doesn't occur on Ubuntu running on pc, it only occur on
> raspberry.

I understand, but if you can only reproduce this using a Ubuntu kernel
on the RPi it could be to something that they've put in their kernel's
(e.g. some out-of-tree driver) which doesn't exist in mainline. That's
why you need to be able to reproduce it on a mainline kernel.

> One additional thing: the bug doesn't occur with HUB: Orico, model
> W6ph4-u3-v1, Manufacture: VIA Labs, inc.
> It only occur with hub: HB-UM43 hub, Genesys Logic, Inc.

Ok, so depending on if the kernel crashes or not, it could still be a
bug in the hub firmware.

> > This sounds like you could have a problem with the host-controller
> > driver. What controller is the RPi4 using?
>  
> For the Pi 4, a fully-featured host controller drives the downstream
> USB ports. Downstream USB is provided by a Via Labs VL805 chip - that
> supports two USB 2.0 ports and two USB 3.0 ports

Which driver does it use? I believe some of the earlier RPi used an
out-of-tree host-controller driver, but perhaps that's no longer the
case.

> > As you write above, the usb_serial_generic_close() is needed to
> > cancel the outstanding URBs during close so you can't remove that.
> > 
> > But my guess is that something breaks in the HCD when unlinking
> > those URBs when connected through the hub. Do you have a stack trace
> > from an oops? Or do things just lock up?
>
> I don't know how to capture stack trace?

If the kernel oopses/crashes (e.g. due to a bug in the driver) you
should find a stack trace in the kernel log (e.g. run dmesg).

But if the USB bus just locks up (until you disconnect the hub or
device, or reload the USB serial driver), then this could also be a
firmware bug in the host-controller or hub.

Johan

RE: Bugs: usb serial crash when close second comport

[Tung Pham]

Dear mr Johan Hovold <johan@kernel.org>
After the bug occur, when open or close device it show on log of system:

failed set request 0x12 status: -110
failed set request 0x00 status: -110
failed set request 0x00 status: -110

0x00 = enable or disable serial device USB request.
0x12 = flush serial device USB request.
-110 = time out.

it means driver is not crashes but it can't open or close device any more.
We may find mainline kernel of raspberry if have, and find the driver of USB they use in next days.
Thanks.

-----Original Message-----
From: Johan Hovold <johan@kernel.org> 
Sent: Tuesday, February 2, 2021 3:32 PM
To: Tung Pham <Tung.Pham@silabs.com>
Cc: Hung Nguyen <Hung.Nguyen@silabs.com>; linux-usb@vger.kernel.org; Pho Tran <Pho.Tran@silabs.com>
Subject: Re: Bugs: usb serial crash when close second comport

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


On Tue, Feb 02, 2021 at 03:50:50AM +0000, Tung Pham wrote:

> > Can you reproduce this with a mainline kernel?
> >
> > Otherwise, you may need to bring this up with Ubuntu.
>
> this bug doesn't occur on Ubuntu running on pc, it only occur on 
> raspberry.

I understand, but if you can only reproduce this using a Ubuntu kernel on the RPi it could be to something that they've put in their kernel's (e.g. some out-of-tree driver) which doesn't exist in mainline. That's why you need to be able to reproduce it on a mainline kernel.

> One additional thing: the bug doesn't occur with HUB: Orico, model 
> W6ph4-u3-v1, Manufacture: VIA Labs, inc.
> It only occur with hub: HB-UM43 hub, Genesys Logic, Inc.

Ok, so depending on if the kernel crashes or not, it could still be a bug in the hub firmware.

> > This sounds like you could have a problem with the host-controller 
> > driver. What controller is the RPi4 using?
>
> For the Pi 4, a fully-featured host controller drives the downstream 
> USB ports. Downstream USB is provided by a Via Labs VL805 chip - that 
> supports two USB 2.0 ports and two USB 3.0 ports

Which driver does it use? I believe some of the earlier RPi used an out-of-tree host-controller driver, but perhaps that's no longer the case.

> > As you write above, the usb_serial_generic_close() is needed to 
> > cancel the outstanding URBs during close so you can't remove that.
> >
> > But my guess is that something breaks in the HCD when unlinking 
> > those URBs when connected through the hub. Do you have a stack trace 
> > from an oops? Or do things just lock up?
>
> I don't know how to capture stack trace?

If the kernel oopses/crashes (e.g. due to a bug in the driver) you should find a stack trace in the kernel log (e.g. run dmesg).

But if the USB bus just locks up (until you disconnect the hub or device, or reload the USB serial driver), then this could also be a firmware bug in the host-controller or hub.

Johan

Re: Bugs: usb serial crash when close second comport

[Johan Hovold]

On Tue, Feb 02, 2021 at 09:48:01AM +0000, Tung Pham wrote:
> After the bug occur, when open or close device it show on log of system:
> 
> failed set request 0x12 status: -110
> failed set request 0x00 status: -110
> failed set request 0x00 status: -110
> 
> 0x00 = enable or disable serial device USB request.
> 0x12 = flush serial device USB request.
> -110 = time out.
> 
> it means driver is not crashes but it can't open or close device any more.

Right, so the device just fails to respond. Can you communicate with
other devices connected to the same hub when this happens?

I found this note about "Interoperability between the Raspberry Pi and
USB 3.0 hubs" which appears to be related:

	https://www.raspberrypi.org/documentation/hardware/raspberrypi/usb/README.md

Does connecting the device through an intermediary USB 2.0 hub make the
problem go away?

> We may find mainline kernel of raspberry if have, and find the driver
> of USB they use in next days.

Sounds good (even if this is starting to sound like a firmware issue).

Johan