data throttling under load when serial to usb adapter is used

data throttling under load when serial to usb adapter is used

[Filip Žaludek]

[-- Attachment #1: Type: text/plain, Size: 3156 bytes --]




Hi,
  under load I am experiencing data throttling when serial to usb adapter is used,
but only in direction from wires to usb. Opposite direction seems to be robust.
  To reproduce plug both ends of usb to serial adapter into same hardware and run
attached script. You should see data flowing in both directions, then saturate cpu
`stress --cpu #cpus_plus_two`.
  Data will be corrupted only in serial to usb direction, usually from beginning,
often within first urb. Not seems to be due to flow control.
  Even though chopped bytes are received, rx/tx counters are correctly incremented.
Script implements data sending in two ways, first utilizing socat and second standard
cat command. It is adjusted for RPi4 with miniUART ttyS0, for multiport hardware
and/or ttyAMA0 you will need to tailor accordingly.

Rough failure rates observed (former trials with ~32k dmesg):
ttyAMA0: 'cat' 20% vs. 'socat'  2%
ttyS0  : 'cat' 90% vs. 'socat' 40%

Why to plug both adapter ends into same hardware?
- I believe this is sub-issue of two connected devices, but it is hard to reproduce
- serial port programming




Regards,
Filip Zaludek




Tested chipsets:

ftdi_sio [TTL], Bus 004 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
pl2303 [TTL], Bus 001 Device 005: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port / Mobile Action MA-8910P
ch341-uart [RS232], Bus 001 Device 006: ID 1a86:7523 QinHeng Electronics CH340 serial converter



Tested chipset|platform|OS|kernel:

ftdi_sio|Raspberry Pi 4 Model B Rev 1.4|Oracle Linux Server 8.5|kernel-5.16.0-rc5
* errors only in s->u direction, often corrupted without load

ftdi_sio|Raspberry Pi 4 Model B Rev 1.4|Oracle Linux Server 8.5|kernel-uek-5.15-2148.0.1.el8uek
* errors only in s->u direction

ftdi_sio|BananaPi-M64|openSUSE Tumbleweed 20210829|5.13.12-2-default
ftdi_sio|BananaPi-M64|openSUSE Tumbleweed 20211202|5.15.5-1-default
* errors only in s->u direction, often corrupted without load

ftdi_sio|Raspberry Pi 4 Model B Rev 1.4|Raspbian GNU/Linux 10 (buster)|5.10.63-v7l+
* errors only in s->u direction

ftdi_sio|Raspberry Pi 4 Model B Rev 1.4|Ubuntu 21.10|5.13.0-1011-raspi
* errors only in s->u direction

ftdi_sio|Raspberry Pi 4 Model B Rev 1.4|openSUSE Tumbleweed 20211202|5.15.5-1-default
* errors only in s->u direction

pl2303|Raspberry Pi 4 Model B Rev 1.4|Oracle Linux Server 8.5|kernel-uek-5.4.17-2136.300.7.el8uek
pl2303|Raspberry Pi 4 Model B Rev 1.4|Oracle Linux Server 8.5|kernel-uek-5.15-2148.0.1.el8uek
* errors only in s->u direction

pl2303|Raspberry Pi 4 Model B Rev 1.4|openSUSE Tumbleweed 20211202|5.15.5-1-default
* errors only in s->u direction, often corrupted without load

pl2303|Raspberry Pi 4 Model B Rev 1.4|Ubuntu 21.10|5.13.0-1011-raspi
* errors only in s->u direction, for miniUART & socat errors seems to be really rare under load

pl2303|Raspberry Pi 4 Model B Rev 1.4|Raspbian GNU/Linux 10 (buster)|5.10.63-v7l+
* errors only in s->u direction

ch341-uart|x86_64|Ubuntu 18.04.5 LTS|4.15.0-154-generic
* I was able only once to run this setup, kernel seemed to be resistant under load. Chipset is unreliable.



[-- Attachment #2: usb_uart.sh --]
[-- Type: application/x-shellscript, Size: 1697 bytes --]

Re: data throttling under load when serial to usb adapter is used

[Greg KH]

On Mon, Dec 20, 2021 at 09:11:06AM +0100, Filip Žaludek wrote:
> 
> 
> 
> Hi,
>  under load I am experiencing data throttling when serial to usb adapter is used,
> but only in direction from wires to usb. Opposite direction seems to be robust.
>  To reproduce plug both ends of usb to serial adapter into same hardware and run
> attached script. You should see data flowing in both directions, then saturate cpu
> `stress --cpu #cpus_plus_two`.
>  Data will be corrupted only in serial to usb direction, usually from beginning,
> often within first urb. Not seems to be due to flow control.
>  Even though chopped bytes are received, rx/tx counters are correctly incremented.
> Script implements data sending in two ways, first utilizing socat and second standard
> cat command. It is adjusted for RPi4 with miniUART ttyS0, for multiport hardware
> and/or ttyAMA0 you will need to tailor accordingly.
> 
> Rough failure rates observed (former trials with ~32k dmesg):
> ttyAMA0: 'cat' 20% vs. 'socat'  2%
> ttyS0  : 'cat' 90% vs. 'socat' 40%
> 
> Why to plug both adapter ends into same hardware?
> - I believe this is sub-issue of two connected devices, but it is hard to reproduce
> - serial port programming
> 
> 
> 
> 
> Regards,
> Filip Zaludek
> 
> 
> 
> 
> Tested chipsets:
> 
> ftdi_sio [TTL], Bus 004 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
> pl2303 [TTL], Bus 001 Device 005: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port / Mobile Action MA-8910P
> ch341-uart [RS232], Bus 001 Device 006: ID 1a86:7523 QinHeng Electronics CH340 serial converter

Using 'cat' is not a good way to ever use a serial port.  Please use a
tool that can properly detect and use the serial port flow control
settings, which should prevent the issues you are seeing here.

Is socat using flow control?  If so, which settings, hardware or
software flow control?

Also please note that the usb-serial devices you are using here are
_very_ cheap and not good for huge amounts of data like you seem to want
to use here.  I would recommend a much more robust usb-serial device if
you need high data rates and good hardware flow control handling.  Most
of the time, the devices you have here do not even have any hardware
flow control support as the manufacturers do not hook up those lines.
So be careful.

thanks,

greg k-h

Re: [External] : Re: data throttling under load when serial to usb adapter is used

[Filip Žaludek]

> 
> Using 'cat' is not a good way to ever use a serial port.  Please use a
> tool that can properly detect and use the serial port flow control
> settings, which should prevent the issues you are seeing here.
> 
> Is socat using flow control?  If so, which settings, hardware or
> software flow control?
> 
> Also please note that the usb-serial devices you are using here are
> _very_ cheap and not good for huge amounts of data like you seem to want
> to use here.  I would recommend a much more robust usb-serial device if
> you need high data rates and good hardware flow control handling.  Most
> of the time, the devices you have here do not even have any hardware
> flow control support as the manufacturers do not hook up those lines.
> So be careful.
> 
> thanks,
> 
> greg k-h
> 



  Yes, you are right! Such gadget hardware is without hw flow control. I am using
serial line only to capture console logs. I've tested both commands also with
sw flow control ixon, ixoff with similar results.

  Hopefully somebody sitting on scarce big iron with both usb port and serial port
with proper hw flow control from this list gets attracted to kindly re-test.

Interesting part for me is that only serial to usb direction is failing..



Thanks,
Filip Zaludek

Re: [External] : Re: data throttling under load when serial to usb adapter is used

[Greg KH]

On Mon, Dec 20, 2021 at 11:19:03AM +0100, Filip Žaludek wrote:
> 
> > 
> > Using 'cat' is not a good way to ever use a serial port.  Please use a
> > tool that can properly detect and use the serial port flow control
> > settings, which should prevent the issues you are seeing here.
> > 
> > Is socat using flow control?  If so, which settings, hardware or
> > software flow control?
> > 
> > Also please note that the usb-serial devices you are using here are
> > _very_ cheap and not good for huge amounts of data like you seem to want
> > to use here.  I would recommend a much more robust usb-serial device if
> > you need high data rates and good hardware flow control handling.  Most
> > of the time, the devices you have here do not even have any hardware
> > flow control support as the manufacturers do not hook up those lines.
> > So be careful.
> > 
> > thanks,
> > 
> > greg k-h
> > 
> 
> 
> 
>  Yes, you are right! Such gadget hardware is without hw flow control. I am using
> serial line only to capture console logs. I've tested both commands also with
> sw flow control ixon, ixoff with similar results.
> 
>  Hopefully somebody sitting on scarce big iron with both usb port and serial port
> with proper hw flow control from this list gets attracted to kindly re-test.
> 
> Interesting part for me is that only serial to usb direction is failing..

Makes sense, these devices are not designed for lots of high-speed data
being sent to them.  Please use a device that is designed for that if
you wish to do this.

thanks,

greg k-h

Re: [External] : Re: data throttling under load when serial to usb adapter is used

[Johan Hovold]

On Mon, Dec 20, 2021 at 11:19:03AM +0100, Filip Žaludek wrote:

>   Yes, you are right! Such gadget hardware is without hw flow control.
>   I am using serial line only to capture console logs. I've tested
>   both commands also with sw flow control ixon, ixoff with similar
>   results.

The FTDI and PL2303 chips typically support automatic hardware flow
control, but you need to make sure the signals are wired up.

>   Hopefully somebody sitting on scarce big iron with both usb port and
>   serial port with proper hw flow control from this list gets
>   attracted to kindly re-test.

No need to retest anything. Just enable hardware flow control otherwise
there cannot be any guarantees.

> Interesting part for me is that only serial to usb direction is
> failing..

But in this case, it looks like your test script is just broken. You
kill the consumer before it has read all the data:

	uart_send__cat()
	{
	    cat < "$1" > /tmp/d16 &
	    RX_PID=$!
	    cat /tmp/d16_0 > "$2"
	    stty sane                           # better results
	    kill $RX_PID
	    wait $RX_PID
	    uart_info "$3"
	}

	...

	stty -F /dev/$SER 115200 raw -echo
	stty -F /dev/$USB 115200 raw -echo

	while true
	do
	    uart_send__cat /dev/$SER /dev/$USB "u->s"
	    uart_send__cat /dev/$USB /dev/$SER "s->u"
	done

Johan