Re: rt_dev_send() stalls periodic task

[Jan Kiszka <jan.kiszka@siemens.com>]

On 26.04.19 02:59, C Smith wrote:
> On Thu, Apr 25, 2019 at 1:23 AM Jan Kiszka <jan.kiszka@siemens.com 
> <mailto:jan.kiszka@siemens.com>> wrote:
> 
>     On 25.04.19 09:15, C Smith wrote:
>      > Hi Jan,
>      >
>      > Your patch worked somewhat but not completely. It prevents my app from
>     stalling
>      > forever, but I caugh the serial transmission itself stalling on the
>     oscilloscope
>      > for quite a long time. My 72 byte TX packet from the xenomai periodic
>     task gets
>      > cut in half and there is no transmission for 7msec, then the transmission
>      > resumes. (I'll send you a screenshot)
> 
>     What is driver and application state during that phase? Who is waiting on what?
>     This will be the key to resolve that issue as I'm not yet seeing another
>     mistake
>     in the driver.
> 
> 
> I don't think there is a bug in the serial driver, per se, but my strange UART 
> requires more from a driver to prevent stalls.
> This is a BCM corp 'BCM87Q' industrial motherboard. They are still sold, not yet 
> EOL.
> 
> We do know a lot about the state the serial driver is in: It is just waiting, 
> thinking it doesn't have any more bytes to transmit. Remember in previous tests 
> the IIR indicated no pending bytes in the THR. I've demonstrated how to get past 
> this state with my TX "polling patch".  I ran my latest test for 12+ hours where 
> I was using your patch plus my polling patch and there were no stalls whatsoever 
> of the serial driver, as verified by an Oscilloscope which triggers on a TX 
> stall. The maximum inter-packet jitter of my TX packet was also fairly low, at 
> <= 450us. In my polling patch, during a RX interrupt, the code redundantly 
> checks the high level transmit buffer to see if rt_16550_tx_fill() should be 
> called. Sure, this workaround only helps when you have full-duplex 
> communications, it would not help during simplex communications.
> 
> Since a device driver can't be reliably polled, I'd prefer some self-correcting 
> mechanism in the driver which set a callback when it thinks it has transmitted 
> the last byte, and wakes up and checks one more time about 100us later to see if 
> it needs to transmit anything else.

I'd prefer to install any watchdog for potential hardware issues until
we really know they aren't software races.

If there a chance to either break a trace or record the full run when
the issue happens? Then you could try this instrumentation, together with ftrace (trace-cmd record -e "cobalt*"):

diff --git a/kernel/drivers/serial/16550A.c b/kernel/drivers/serial/16550A.c
index 81acc6344e..504d85ccbe 100644
--- a/kernel/drivers/serial/16550A.c
+++ b/kernel/drivers/serial/16550A.c
@@ -197,6 +197,7 @@ static void rt_16550_tx_fill(struct rt_16550_context *ctx)
 	unsigned long base = ctx->base_addr;
 	int mode = rt_16550_io_mode_from_ctx(ctx);
 
+	trace_printk("tx_fill, out_npend: %ld", ctx->out_npend);
 /*	if (uart->modem & MSR_CTS)*/
 	{
 		for (count = ctx->tx_fifo;
@@ -239,6 +240,7 @@ static int rt_16550_interrupt(rtdm_irq_t * irq_context)
 
 	while (1) {
 		iir = rt_16550_reg_in(mode, base, IIR) & IIR_MASK;
+		trace_printk("IIR: 0x%x", iir);
 		if (iir & IIR_PIRQ)
 			break;
 
@@ -284,6 +286,7 @@ static int rt_16550_interrupt(rtdm_irq_t * irq_context)
 	}
 
 	if ((ctx->ier_status & IER_TX) && (ctx->out_npend == 0)) {
+		trace_printk("IER_TX off");
 		/* mask transmitter empty interrupt */
 		ctx->ier_status &= ~IER_TX;
 
@@ -1030,10 +1033,12 @@ ssize_t rt_16550_write(struct rtdm_fd *fd, const void *buf, size_t nbyte)
 
 			lsr = rt_16550_reg_in(rt_16550_io_mode_from_ctx(ctx),
 					      ctx->base_addr, LSR);
+			trace_printk("LSR: 0x%x", lsr);
 			if (lsr & RTSER_LSR_THR_EMTPY)
 				rt_16550_tx_fill(ctx);
 
 			if (ctx->out_npend > 0 && !(ctx->ier_status & IER_TX)) {
+				trace_printk("IER_TX on, out_npend: %lu", ctx->out_npend);
 				/* unmask tx interrupt */
 				ctx->ier_status |= IER_TX;
 				rt_16550_reg_out(rt_16550_io_mode_from_ctx(ctx),


Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux