* Serial console and interrupts latency. @ 2020-03-24 9:04 Sergey Organov 2020-03-27 13:13 ` Jiri Slaby 0 siblings, 1 reply; 10+ messages in thread From: Sergey Organov @ 2020-03-24 9:04 UTC (permalink / raw) To: linux-serial; +Cc: Russell King, Greg Kroah-Hartman, Jiri Slaby Hello, [Extended CC list to try to get some attention] I was investigating random serial overruns on my embedded board and figured it strongly correlates with serial output (to another serial port) from kernel printk() calls, that forced me to dig into the kernel sources, and now I'm very confused. I'm reading drivers/tty/serial/8250/8250_port.c, and serial8250_console_write() function in particular (being on tty-next branch). What I see is that it locks interrupts 3141: spin_lock_irqsave(&port->lock, flags); and then calls wait_for_xmitr() both indirectly here: 3159: uart_console_write(port, s, count, serial8250_console_putchar); and then directly as well: 3165: wait_for_xmitr(up, BOTH_EMPTY); before re-enabling interrupts at: 3179: spin_unlock_irqrestore(&port->lock, flags); Now, wait_for_xmitr(), even according to comments, could busy-wait for up to 10+1000 milliseconds, and in this case this huge delay will happen at interrupts disabled? Does it mean any serial console output out of printk() could cause 10 milliseconds or even 1 second interrupts latency? Somehow I can't believe it. What do I miss? Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-24 9:04 Serial console and interrupts latency Sergey Organov @ 2020-03-27 13:13 ` Jiri Slaby 2020-03-27 13:38 ` Russell King - ARM Linux admin 2020-03-27 13:58 ` Sergey Organov 0 siblings, 2 replies; 10+ messages in thread From: Jiri Slaby @ 2020-03-27 13:13 UTC (permalink / raw) To: Sergey Organov, linux-serial; +Cc: Russell King, Greg Kroah-Hartman On 24. 03. 20, 10:04, Sergey Organov wrote: > Hello, > > [Extended CC list to try to get some attention] > > I was investigating random serial overruns on my embedded board and > figured it strongly correlates with serial output (to another serial > port) from kernel printk() calls, that forced me to dig into the kernel > sources, and now I'm very confused. > > I'm reading drivers/tty/serial/8250/8250_port.c, and > serial8250_console_write() function in particular (being on tty-next > branch). > > What I see is that it locks interrupts > > 3141: spin_lock_irqsave(&port->lock, flags); > > and then calls wait_for_xmitr() both indirectly here: > > 3159: uart_console_write(port, s, count, serial8250_console_putchar); > > and then directly as well: > > 3165: wait_for_xmitr(up, BOTH_EMPTY); > > before re-enabling interrupts at: > > 3179: spin_unlock_irqrestore(&port->lock, flags); > > Now, wait_for_xmitr(), even according to comments, could busy-wait for > up to 10+1000 milliseconds, and in this case this huge delay will happen > at interrupts disabled? > > Does it mean any serial console output out of printk() could cause 10 > milliseconds or even 1 second interrupts latency? Somehow I can't > believe it. > > What do I miss? 1 second _timeout_ is for flow-control-enabled consoles. 10 ms is _timeout_ for a character. With slow 9600 baud console, sending one character takes 0.8 ms. With 115200, it is 70 us. If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. So yes, serial consoles can slow down the boot and add latency. Use faster speeds or faster devices for consoles, if you mind. And do not enable flow control. Serial is serial. You can also try to eliminate the interrupts disablement, but that would be a bit tough task, IMO. thanks, -- js suse labs ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-27 13:13 ` Jiri Slaby @ 2020-03-27 13:38 ` Russell King - ARM Linux admin 2020-03-27 14:03 ` Sergey Organov 2020-03-27 13:58 ` Sergey Organov 1 sibling, 1 reply; 10+ messages in thread From: Russell King - ARM Linux admin @ 2020-03-27 13:38 UTC (permalink / raw) To: Jiri Slaby; +Cc: Sergey Organov, linux-serial, Greg Kroah-Hartman On Fri, Mar 27, 2020 at 02:13:12PM +0100, Jiri Slaby wrote: > On 24. 03. 20, 10:04, Sergey Organov wrote: > > Hello, > > > > [Extended CC list to try to get some attention] > > > > I was investigating random serial overruns on my embedded board and > > figured it strongly correlates with serial output (to another serial > > port) from kernel printk() calls, that forced me to dig into the kernel > > sources, and now I'm very confused. > > > > I'm reading drivers/tty/serial/8250/8250_port.c, and > > serial8250_console_write() function in particular (being on tty-next > > branch). > > > > What I see is that it locks interrupts > > > > 3141: spin_lock_irqsave(&port->lock, flags); > > > > and then calls wait_for_xmitr() both indirectly here: > > > > 3159: uart_console_write(port, s, count, serial8250_console_putchar); > > > > and then directly as well: > > > > 3165: wait_for_xmitr(up, BOTH_EMPTY); > > > > before re-enabling interrupts at: > > > > 3179: spin_unlock_irqrestore(&port->lock, flags); > > > > Now, wait_for_xmitr(), even according to comments, could busy-wait for > > up to 10+1000 milliseconds, and in this case this huge delay will happen > > at interrupts disabled? > > > > Does it mean any serial console output out of printk() could cause 10 > > milliseconds or even 1 second interrupts latency? Somehow I can't > > believe it. > > > > What do I miss? > > 1 second _timeout_ is for flow-control-enabled consoles. > > 10 ms is _timeout_ for a character. With slow 9600 baud console, sending > one character takes 0.8 ms. With 115200, it is 70 us. > > If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. > > So yes, serial consoles can slow down the boot and add latency. Use > faster speeds or faster devices for consoles, if you mind. And do not > enable flow control. Serial is serial. > > You can also try to eliminate the interrupts disablement, but that would > be a bit tough task, IMO. ... and likely difficult to implement, since the spinlock is there to prevent coincident usage from userspace and kernel space to the port. If userspace is outputting a message in unison with a kernel console message, there must be exclusivity, otherwise the two messages will either be intermingled, or completely corrupted due to TX FIFO overflow, making the console quite useless. Higher latency is something that IMHO has to be accepted as a result of using serial console. As Jiri says, the way around that is to use as fast a baud rate as possible on the serial console. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-27 13:38 ` Russell King - ARM Linux admin @ 2020-03-27 14:03 ` Sergey Organov 0 siblings, 0 replies; 10+ messages in thread From: Sergey Organov @ 2020-03-27 14:03 UTC (permalink / raw) To: Russell King - ARM Linux admin Cc: Jiri Slaby, linux-serial, Greg Kroah-Hartman Russell King - ARM Linux admin <linux@armlinux.org.uk> writes: > On Fri, Mar 27, 2020 at 02:13:12PM +0100, Jiri Slaby wrote: >> On 24. 03. 20, 10:04, Sergey Organov wrote: >> > Hello, >> > >> > [Extended CC list to try to get some attention] >> > >> > I was investigating random serial overruns on my embedded board and >> > figured it strongly correlates with serial output (to another serial >> > port) from kernel printk() calls, that forced me to dig into the kernel >> > sources, and now I'm very confused. >> > >> > I'm reading drivers/tty/serial/8250/8250_port.c, and >> > serial8250_console_write() function in particular (being on tty-next >> > branch). >> > >> > What I see is that it locks interrupts >> > >> > 3141: spin_lock_irqsave(&port->lock, flags); >> > >> > and then calls wait_for_xmitr() both indirectly here: >> > >> > 3159: uart_console_write(port, s, count, serial8250_console_putchar); >> > >> > and then directly as well: >> > >> > 3165: wait_for_xmitr(up, BOTH_EMPTY); >> > >> > before re-enabling interrupts at: >> > >> > 3179: spin_unlock_irqrestore(&port->lock, flags); >> > >> > Now, wait_for_xmitr(), even according to comments, could busy-wait for >> > up to 10+1000 milliseconds, and in this case this huge delay will happen >> > at interrupts disabled? >> > >> > Does it mean any serial console output out of printk() could cause 10 >> > milliseconds or even 1 second interrupts latency? Somehow I can't >> > believe it. >> > >> > What do I miss? >> >> 1 second _timeout_ is for flow-control-enabled consoles. >> >> 10 ms is _timeout_ for a character. With slow 9600 baud console, sending >> one character takes 0.8 ms. With 115200, it is 70 us. >> >> If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. >> >> So yes, serial consoles can slow down the boot and add latency. Use >> faster speeds or faster devices for consoles, if you mind. And do not >> enable flow control. Serial is serial. >> >> You can also try to eliminate the interrupts disablement, but that would >> be a bit tough task, IMO. > > ... and likely difficult to implement, since the spinlock is there to > prevent coincident usage from userspace and kernel space to the port. > If userspace is outputting a message in unison with a kernel console > message, there must be exclusivity, otherwise the two messages will > either be intermingled, or completely corrupted due to TX FIFO > overflow, making the console quite useless. > > Higher latency is something that IMHO has to be accepted as a result > of using serial console. This (local) interrupts disabling might be not a big problem on SMP, I dunno, as another core could still handle IRQs, but for non-SMP case this is too bad as it causes missed data on other ports. Can we disable this locking on non-SMP configs as a work-around? > As Jiri says, the way around that is to use as fast a baud rate as > possible on the serial console. Unfortunately, this is entirely impractical for an application that utilizes maximum baud rates on other serial ports. Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-27 13:13 ` Jiri Slaby 2020-03-27 13:38 ` Russell King - ARM Linux admin @ 2020-03-27 13:58 ` Sergey Organov 2020-03-27 23:24 ` Russell King - ARM Linux admin 1 sibling, 1 reply; 10+ messages in thread From: Sergey Organov @ 2020-03-27 13:58 UTC (permalink / raw) To: Jiri Slaby; +Cc: linux-serial, Russell King, Greg Kroah-Hartman Jiri Slaby <jslaby@suse.cz> writes: > On 24. 03. 20, 10:04, Sergey Organov wrote: >> Hello, >> >> [Extended CC list to try to get some attention] >> >> I was investigating random serial overruns on my embedded board and >> figured it strongly correlates with serial output (to another serial >> port) from kernel printk() calls, that forced me to dig into the kernel >> sources, and now I'm very confused. >> >> I'm reading drivers/tty/serial/8250/8250_port.c, and >> serial8250_console_write() function in particular (being on tty-next >> branch). >> >> What I see is that it locks interrupts >> >> 3141: spin_lock_irqsave(&port->lock, flags); >> >> and then calls wait_for_xmitr() both indirectly here: >> >> 3159: uart_console_write(port, s, count, serial8250_console_putchar); >> >> and then directly as well: >> >> 3165: wait_for_xmitr(up, BOTH_EMPTY); >> >> before re-enabling interrupts at: >> >> 3179: spin_unlock_irqrestore(&port->lock, flags); >> >> Now, wait_for_xmitr(), even according to comments, could busy-wait for >> up to 10+1000 milliseconds, and in this case this huge delay will happen >> at interrupts disabled? >> >> Does it mean any serial console output out of printk() could cause 10 >> milliseconds or even 1 second interrupts latency? Somehow I can't >> believe it. >> >> What do I miss? > > 1 second _timeout_ is for flow-control-enabled consoles. Yeah, sure. So it does mean interrupts could be disabled for up to 1 second, on already up and running system. Too bad. Actually, I use 8250 only as a reference implementation, my actual chip is handled by imx.c, and the latter even has no timeouts on this path, so apparently may block (the entire kernel) indefinitely! > 10 ms is _timeout_ for a character. With slow 9600 baud console, sending > one character takes 0.8 ms. With 115200, it is 70 us. 70us of disabled interrupts is a huge number, and for FIFO-enabled chips the estimate should be multiplied by FIFO size (say, x16) that brings us close to 1ms even on 115200, right? Anyway, it must cause receiving overruns on another port running at higher or the same baud rate and no DMA, sooner or later, as it does for me. > > If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. > > So yes, serial consoles can slow down the boot and add latency. Use > faster speeds or faster devices for consoles, if you mind. And do not > enable flow control. Serial is serial. I don't care about slowing-down boot. I care about huge interrupt latency on up and running system, causing loss of characters (overruns) on other serial ports. To be sure, it is this code that works on already running system as well, not only on boot-time, right? Or is my system somehow misconfigured? I'm confused as this seems to be a major issue and nobody but me seems to care or to suffer from it, and I can't figure why. For reference, I figured this locking was introduced by: commit d8a5a8d7cc32e4474326e0ecc1b959063490efc9 Author: Russell King <rmk@dyn-67.arm.linux.org.uk> Date: Tue May 2 16:04:29 2006 +0100 [SERIAL] 8250: add locking to console write function x86 SMP breaks as a result of the previous change, we have no real option other than to add locking to the 8250 console write function. If an oops is in progress, try to acquire the lock. If we fail to do so, continue anyway. > You can also try to eliminate the interrupts disablement, but that would > be a bit tough task, IMO. It seems like I need to, and yeah, it'd be a somewhat tough task indeed, but then there is one simple question: why isn't console output handled through usual buffer/ISR paths? I have an idea to investigate the netconsole module to see how it does similar job for networks. Hopefully it has no such problems and will give some useful tips. Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-27 13:58 ` Sergey Organov @ 2020-03-27 23:24 ` Russell King - ARM Linux admin 2020-04-14 9:50 ` Sergey Organov 0 siblings, 1 reply; 10+ messages in thread From: Russell King - ARM Linux admin @ 2020-03-27 23:24 UTC (permalink / raw) To: Sergey Organov; +Cc: Jiri Slaby, linux-serial, Greg Kroah-Hartman On Fri, Mar 27, 2020 at 04:58:33PM +0300, Sergey Organov wrote: > Jiri Slaby <jslaby@suse.cz> writes: > > > On 24. 03. 20, 10:04, Sergey Organov wrote: > >> Hello, > >> > >> [Extended CC list to try to get some attention] > >> > >> I was investigating random serial overruns on my embedded board and > >> figured it strongly correlates with serial output (to another serial > >> port) from kernel printk() calls, that forced me to dig into the kernel > >> sources, and now I'm very confused. > >> > >> I'm reading drivers/tty/serial/8250/8250_port.c, and > >> serial8250_console_write() function in particular (being on tty-next > >> branch). > >> > >> What I see is that it locks interrupts > >> > >> 3141: spin_lock_irqsave(&port->lock, flags); > >> > >> and then calls wait_for_xmitr() both indirectly here: > >> > >> 3159: uart_console_write(port, s, count, serial8250_console_putchar); > >> > >> and then directly as well: > >> > >> 3165: wait_for_xmitr(up, BOTH_EMPTY); > >> > >> before re-enabling interrupts at: > >> > >> 3179: spin_unlock_irqrestore(&port->lock, flags); > >> > >> Now, wait_for_xmitr(), even according to comments, could busy-wait for > >> up to 10+1000 milliseconds, and in this case this huge delay will happen > >> at interrupts disabled? > >> > >> Does it mean any serial console output out of printk() could cause 10 > >> milliseconds or even 1 second interrupts latency? Somehow I can't > >> believe it. > >> > >> What do I miss? > > > > 1 second _timeout_ is for flow-control-enabled consoles. > > Yeah, sure. So it does mean interrupts could be disabled for up to 1 > second, on already up and running system. Too bad. > > Actually, I use 8250 only as a reference implementation, my actual > chip is handled by imx.c, and the latter even has no timeouts on this > path, so apparently may block (the entire kernel) indefinitely! > > > 10 ms is _timeout_ for a character. With slow 9600 baud console, sending > > one character takes 0.8 ms. With 115200, it is 70 us. > > 70us of disabled interrupts is a huge number, and for FIFO-enabled chips > the estimate should be multiplied by FIFO size (say, x16) that brings us > close to 1ms even on 115200, right? > > Anyway, it must cause receiving overruns on another port running at > higher or the same baud rate and no DMA, sooner or later, as it does > for me. So, don't use serial console then, it's unsuitable for your use case. > > If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. > > > > So yes, serial consoles can slow down the boot and add latency. Use > > faster speeds or faster devices for consoles, if you mind. And do not > > enable flow control. Serial is serial. > > I don't care about slowing-down boot. I care about huge interrupt > latency on up and running system, causing loss of characters (overruns) > on other serial ports. > > To be sure, it is this code that works on already running system as > well, not only on boot-time, right? Or is my system somehow > misconfigured? > > I'm confused as this seems to be a major issue and nobody but me seems > to care or to suffer from it, and I can't figure why. > > For reference, I figured this locking was introduced by: > > commit d8a5a8d7cc32e4474326e0ecc1b959063490efc9 > Author: Russell King <rmk@dyn-67.arm.linux.org.uk> > Date: Tue May 2 16:04:29 2006 +0100 > > [SERIAL] 8250: add locking to console write function > > x86 SMP breaks as a result of the previous change, we have no real > option other than to add locking to the 8250 console write function. > If an oops is in progress, try to acquire the lock. If we fail to > do so, continue anyway. Correct, and what I said back then still applies - and more. > It seems like I need to, and yeah, it'd be a somewhat tough task indeed, > but then there is one simple question: why isn't console output handled > through usual buffer/ISR paths? The "usual" paths may not be active, and, in the case of an oops, we want to see the output, which we wouldn't be able to if the oops occurred in interrupt context. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-03-27 23:24 ` Russell King - ARM Linux admin @ 2020-04-14 9:50 ` Sergey Organov 2020-04-14 9:56 ` Russell King - ARM Linux admin 0 siblings, 1 reply; 10+ messages in thread From: Sergey Organov @ 2020-04-14 9:50 UTC (permalink / raw) To: Russell King - ARM Linux admin Cc: Jiri Slaby, linux-serial, Greg Kroah-Hartman Russell King - ARM Linux admin <linux@armlinux.org.uk>: > > On Fri, Mar 27, 2020 at 04:58:33PM +0300, Sergey Organov wrote: > > Jiri Slaby <jslaby@suse.cz> writes: > > > > > On 24. 03. 20, 10:04, Sergey Organov wrote: > > >> Hello, > > >> > > >> [Extended CC list to try to get some attention] > > >> > > >> I was investigating random serial overruns on my embedded board and > > >> figured it strongly correlates with serial output (to another serial > > >> port) from kernel printk() calls, that forced me to dig into the kernel > > >> sources, and now I'm very confused. > > >> > > >> I'm reading drivers/tty/serial/8250/8250_port.c, and > > >> serial8250_console_write() function in particular (being on tty-next > > >> branch). > > >> > > >> What I see is that it locks interrupts > > >> > > >> 3141: spin_lock_irqsave(&port->lock, flags); > > >> > > >> and then calls wait_for_xmitr() both indirectly here: > > >> > > >> 3159: uart_console_write(port, s, count, serial8250_console_putchar); > > >> > > >> and then directly as well: > > >> > > >> 3165: wait_for_xmitr(up, BOTH_EMPTY); > > >> > > >> before re-enabling interrupts at: > > >> > > >> 3179: spin_unlock_irqrestore(&port->lock, flags); > > >> > > >> Now, wait_for_xmitr(), even according to comments, could busy-wait for > > >> up to 10+1000 milliseconds, and in this case this huge delay will happen > > >> at interrupts disabled? > > >> > > >> Does it mean any serial console output out of printk() could cause 10 > > >> milliseconds or even 1 second interrupts latency? Somehow I can't > > >> believe it. > > >> > > >> What do I miss? > > > > > > 1 second _timeout_ is for flow-control-enabled consoles. > > > > Yeah, sure. So it does mean interrupts could be disabled for up to 1 > > second, on already up and running system. Too bad. > > > > Actually, I use 8250 only as a reference implementation, my actual > > chip is handled by imx.c, and the latter even has no timeouts on this > > path, so apparently may block (the entire kernel) indefinitely! > > > > > 10 ms is _timeout_ for a character. With slow 9600 baud console, sending > > > one character takes 0.8 ms. With 115200, it is 70 us. > > > > 70us of disabled interrupts is a huge number, and for FIFO-enabled chips > > the estimate should be multiplied by FIFO size (say, x16) that brings us > > close to 1ms even on 115200, right? > > > > Anyway, it must cause receiving overruns on another port running at > > higher or the same baud rate and no DMA, sooner or later, as it does > > for me. > > So, don't use serial console then, it's unsuitable for your use case. Well, I'd rather fix it, as serial console is otherwise very suitable for my needs. If nobody else interested, I'll simply disable the lock you've added for non-smp builds in my kernel version, rather than trying to fix the issue in general. That said, finding generic solution would be an interesting quest. > > > > If you send one line (80 chars), it is really 66 and 5.5 ms, respectively. > > > > > > So yes, serial consoles can slow down the boot and add latency. Use > > > faster speeds or faster devices for consoles, if you mind. And do not > > > enable flow control. Serial is serial. > > > > I don't care about slowing-down boot. I care about huge interrupt > > latency on up and running system, causing loss of characters (overruns) > > on other serial ports. > > > > To be sure, it is this code that works on already running system as > > well, not only on boot-time, right? Or is my system somehow > > misconfigured? > > > > I'm confused as this seems to be a major issue and nobody but me seems > > to care or to suffer from it, and I can't figure why. > > > > For reference, I figured this locking was introduced by: > > > > commit d8a5a8d7cc32e4474326e0ecc1b959063490efc9 > > Author: Russell King <rmk@dyn-67.arm.linux.org.uk> > > Date: Tue May 2 16:04:29 2006 +0100 > > > > [SERIAL] 8250: add locking to console write function > > > > x86 SMP breaks as a result of the previous change, we have no real > > option other than to add locking to the 8250 console write function. > > If an oops is in progress, try to acquire the lock. If we fail to > > do so, continue anyway. > > Correct, and what I said back then still applies - and more. What bothers me is "we have no real option..." part of this, as it's rarely happens to be the case. > > > It seems like I need to, and yeah, it'd be a somewhat tough task indeed, > > but then there is one simple question: why isn't console output handled > > through usual buffer/ISR paths? > > The "usual" paths may not be active, and, in the case of an oops, we > want to see the output, which we wouldn't be able to if the oops > occurred in interrupt context. The oops part is already special-cased and could be left special-cased, one way or another. What is important is not to keep interrupts disabled for long during normal system operation. If it could be achieved at all, it probably should then be implemented at the upper level, to save low-level drivers from these complexities. Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-04-14 9:50 ` Sergey Organov @ 2020-04-14 9:56 ` Russell King - ARM Linux admin 2020-04-14 12:14 ` Sergey Organov 0 siblings, 1 reply; 10+ messages in thread From: Russell King - ARM Linux admin @ 2020-04-14 9:56 UTC (permalink / raw) To: Sergey Organov; +Cc: Jiri Slaby, linux-serial, Greg Kroah-Hartman On Tue, Apr 14, 2020 at 12:50:49PM +0300, Sergey Organov wrote: > Russell King - ARM Linux admin <linux@armlinux.org.uk>: > > Correct, and what I said back then still applies - and more. > > What bothers me is "we have no real option..." part of this, as it's rarely > happens to be the case. I don't see you coming forward with a solution beyond "let's revert the commit" or "let's comment out the lock" - neither of which are an option for mainline kernels. Until *you* do, since you obviously have better ideas, "we have no real option". But, as I've said, one of the *important* characteristics of serial console is that it is synchronous with the kernel, so it can be relied upon to get complete messages out if the kernel crashes after a printk() has been executed, and that must not be lost. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Serial console and interrupts latency. 2020-04-14 9:56 ` Russell King - ARM Linux admin @ 2020-04-14 12:14 ` Sergey Organov 0 siblings, 0 replies; 10+ messages in thread From: Sergey Organov @ 2020-04-14 12:14 UTC (permalink / raw) To: Russell King - ARM Linux admin Cc: Jiri Slaby, linux-serial, Greg Kroah-Hartman Russell King - ARM Linux admin <linux@armlinux.org.uk> writes: > On Tue, Apr 14, 2020 at 12:50:49PM +0300, Sergey Organov wrote: >> Russell King - ARM Linux admin <linux@armlinux.org.uk>: >> > Correct, and what I said back then still applies - and more. >> >> What bothers me is "we have no real option..." part of this, as it's rarely >> happens to be the case. > > I don't see you coming forward with a solution beyond "let's revert > the commit" or "let's comment out the lock" - neither of which are > an option for mainline kernels. I didn't suggest to revert the commit, as it obviously solves real problem. I asked if and why the lock is needed on non-SMP targets, but either I didn't get a reply or missed it, sorry. I mean everything I said was to get some help rather than to spread useless critics or even insults. > > Until *you* do, since you obviously have better ideas, "we have no > real option". I'm in the process of making myself familiar with the internals of printk machinery to find generic solution, but if nobody else actually cares, I'll stop bothering you guys with my questions. Do I get it right that you still think there are no sensible options but to disable interrupts for ages? If so, it would greatly reduce my drive to find one... or maybe the other way round, but it'd be nice to know either way. > But, as I've said, one of the *important* characteristics of serial > console is that it is synchronous with the kernel, so it can be > relied upon to get complete messages out if the kernel crashes after > a printk() has been executed, and that must not be lost. Do I get it right that it's covered by the 'oops' case, or are there hidden pitfalls as well? Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
* Serial console and interrupts latency. @ 2020-03-20 15:15 Sergey Organov 0 siblings, 0 replies; 10+ messages in thread From: Sergey Organov @ 2020-03-20 15:15 UTC (permalink / raw) To: linux-serial Hello, I was investigating random serial overruns on my embedded board and figured it strongly correlates with serial output (to another serial port) from kernel printk() calls, that forced me to dig into the kernel sources, and now I'm very confused. I'm reading drivers/tty/serial/8250/8250_port.c, and serial8250_console_write() function in particular (being on tty-next branch). What I see is that it locks interrupts 3141: spin_lock_irqsave(&port->lock, flags); and then calls wait_for_xmitr() both indirectly here: 3159: uart_console_write(port, s, count, serial8250_console_putchar); and then directly as well: 3165: wait_for_xmitr(up, BOTH_EMPTY); before re-enabling interrupts at: 3179: spin_unlock_irqrestore(&port->lock, flags); Now, wait_for_xmitr(), even according to comments, could busy-wait for up to 10+1000 milliseconds, and in this case this huge delay will happen at interrupts disabled? Does it mean any serial console output out of printk() could cause 10 milliseconds or even 1 second interrupts latency? Somehow I can't believe it. What do I miss? Thanks, -- Sergey ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-04-14 12:14 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-24 9:04 Serial console and interrupts latency Sergey Organov 2020-03-27 13:13 ` Jiri Slaby 2020-03-27 13:38 ` Russell King - ARM Linux admin 2020-03-27 14:03 ` Sergey Organov 2020-03-27 13:58 ` Sergey Organov 2020-03-27 23:24 ` Russell King - ARM Linux admin 2020-04-14 9:50 ` Sergey Organov 2020-04-14 9:56 ` Russell King - ARM Linux admin 2020-04-14 12:14 ` Sergey Organov -- strict thread matches above, loose matches on Subject: below -- 2020-03-20 15:15 Sergey Organov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.