serial: imx: hardware flow control failure

serial: imx: hardware flow control failure

[Tomasz Moń]

Hello,

I am reliably able to trigger hardware flow control failure on imx7d
based system under following conditions:
  * two (or more) UARTs receive data simultaneously,
  * 1.5M baud, 8N1,
  * raw mode, CRTSCTS enabled,
  * "uart-has-rtscts" property present in devicetree,
  * DMA is disabled (no "dmas" nor "dma-names" in devicetree).

By hardware flow control failure I mean userspace application reading
from serial port device does not get all characters received by UART.

The sender is fully respecting RTS output (CTS_B) signal level. When
the failure occurs, the system appears frozen and unfreezes only when
the sender stops sending data.

Debugging led me to believe this is a starvation issue. I am not sure
what is the proper way to fix it.

Line discipline workqueue handler flush_to_ldisc() defined in
drivers/tty/tty_buffer.c is responsible for tty throttling. The call
stack when throttling is as follows:
  uart_throttle                  drivers/tty/serial/serial_core.c:681
  tty_throttle_safe              drivers/tty/tty_ioctl.c:147
  n_tty_check_throttle           drivers/tty/n_tty.c:264
  n_tty_receive_buf_common       drivers/tty/n_tty.c:1693
  n_tty_receive_buf2             drivers/tty/n_tty.c:1709
  tty_ldisc_receive_buf          drivers/tty/tty_buffer.c:471
  tty_port_default_receive_buf   drivers/tty/tty_port.c:39
  receive_buf                    drivers/tty/tty_buffer.c:491
  flush_to_ldisc                 drivers/tty/tty_buffer.c:543
  process_one_work               kernel/workqueue.c:2298
  worker_thread                  kernel/workqueue.c:2445

Unthrottling happens in read syscall handler:
  uart_unthrottle                drivers/tty/serial/serial_core.c:710
  tty_unthrottle_safe            drivers/tty/tty_ioctl.c:178
  n_tty_check_unthrottle         drivers/tty/n_tty.c:295
  n_tty_read                     drivers/tty/n_tty.c:2095
  iterate_tty_read               drivers/tty/tty_io.c:868
  tty_read                       drivers/tty/tty_io.c:944
  new_sync_read                  fs/read_write.c:400
  vfs_read                       fs/read_write.c:481
  ksys_read                      fs/read_write.c:619

The receive interrupt handler call stack is as follows:
  __imx_uart_rxint               drivers/tty/serial/imx.c:785
  imx_uart_int                   drivers/tty/serial/imx.c:961
  __handle_irq_event_percpu      kernel/irq/handle.c:158
  handle_irq_event_percpu        kernel/irq/handle.c:198
  handle_irq_event               kernel/irq/handle.c:215
  handle_fasteoi_irq             kernel/irq/chip.c:717
  handle_irq_desc                kernel/irq/irqdesc.c:646
  generic_handle_domain_irq      kernel/irq/irqdesc.c:680
  gic_handle_irq                 drivers/irqchip/irq-gic.c:372

While the imx UART has autoRTS feature, it is not effectively used by
Linux driver. That is, the feature itself (UCR2_CTSC bit) is enabled
when unthrottled, but the actual observable RTS output (CTS_B) changes
are the direct result of tty throttling/unthrottling.

During the failure, CPU is busy with receiving data from RxFIFO. Once
the flip buffer cannot store any more characters, the handler starts
dropping data (incrementing sport->port.icount.buf_overrun counter).
The CPU is fast enough to keep the RxFIFO level below the autoRTS
threshold (CTSTL in UCR4 register). To take the advantage of autoRTS,
all the CPU needs to do is stop reading from RxFIFO.

While the idea seems clear, converting it to code is not. I think the
tty flip buffer would need to expose mechanism to indicate if it can
store at least one more character *before* the rxint handler retrieves
it from the RxFIFO. If flip buffer cannot store any more characters,
the rxint handler would have to disable the receive interrupt to stop
the high CPU usage. But who should re-enable the receive interrupt?

When hardware flow control is achieved via RTS gpio, simply ceasing to
read from the RxFIFO is not enough. How to ensure that throttle will
happen as soon as possible? Would moving the rxint handler to tasklet
solve the issue?

What to do when the hardware flow control is not available? Should the
driver continue operating as-is, or should there be some mechanism to
prevent the rxint handler from effectively taking up all the CPU?

Best Regards,
Tomasz Mon