linux-serial.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: John Ogness <john.ogness@linutronix.de>
To: Eugeniu Rosca <erosca@de.adit-jv.com>
Cc: <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Geert Uytterhoeven <geert+renesas@glider.be>,
	Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>,
	Andrew Gabbasov <andrew_gabbasov@mentor.com>,
	Sanjeev Chugh <sanjeev_chugh@mentor.com>,
	Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Daniel Wang <wonderfly@google.com>,
	Dean Jenkins <dean_jenkins@mentor.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Dirk Behme <dirk.behme@de.bosch.com>,
	Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
	Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
	<linux-serial@vger.kernel.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Eugeniu Rosca <roscaeugeniu@gmail.com>
Subject: Re: [RFC PATCH v1 00/25] printk: new implementation
Date: Wed, 22 Jan 2020 00:56:48 +0100	[thread overview]
Message-ID: <87v9p4mkhr.fsf@linutronix.de> (raw)
In-Reply-To: <20200120230522.GA23636@lxhi-065.adit-jv.com> (Eugeniu Rosca's message of "Tue, 21 Jan 2020 00:05:22 +0100")

Hello Eugeniu,

On 2020-01-21, Eugeniu Rosca <erosca@de.adit-jv.com> wrote:
> This [1] is a fairly old thread, but I only recently stumbled upon it,
> while co-investigating below audio distortions [2] on R-Car3 ARM64
> boards, which can be reproduced by stressing [3] the serial console.
>
> The investigation started a few months ago, when users reported audio
> drops during the first seconds of system startup. Only after a few
> weeks it became clear (thanks to some people in Cc) that the
> distortions were contributed by the above-average serial console load
> during the early boot. Once understood, we were able to come up with a
> synthetic test [2-3].
>
> I thought it would be interesting to share below reproduction matrix,
> in order to contrast vanilla to linux-rt-devel [4], as well as to
> compare various preemption models.
>  
>                            | Ser.console  Ser.console
>                            | stressed     at rest or disabled
>       --------------------------------------------
>       v5.5-rc6 (PREEMPT=y) | distorted    clean
>     v5.4.5-rt3 (PREEMPT=y) | distorted    clean
>  v5.4.5-rt3 (PREEMPT_RT=y) | clean        clean
>
> My feeling is that the results probably do not surprise linux-rt
> people.
>
> My first question is, should there be any improvement in the case of
> v5.4.5-rt3 (PREEMPT=y), which I do not sense? I would expect so, based
> on the cover letter of this series (pointing out the advantages of the
> redesigned printk mechanism).

The problem you are reporting is not the problem that the printk rework
is trying to solve.

In your chart, v5.4.5-rt3 (PREEMPT_RT=y) is the only configuration that
is _not_ disabling hardware interrupts during UART activity. I would
guess your problem is due to interrupts being disabled for unacceptable
lengths of time. You need a low-latency system, so PREEMPT_RT=y _is_ the
correct (and only) solution if a verbose serial console is a must.

The printk rework focusses on making printk non-interfering by
decoupling console printing from printk() callers. However, the console
printing itself will still do just as much interrupt disabling as
before. That is driver-related, not printk-related.

> And the other question is, how would you, generally speaking, tackle
> the problem, given that backporting the linux-rt patches is *not* an
> option and enabling serial console is a must?

The linux-rt patches (which include this printk rework) *are* being
ported to mainline now. My recommendation is to continue using the
linux-rt patches (with PREEMPT_RT=y) until PREEMPT_RT is available
mainline.

John Ogness

> [1] https://lore.kernel.org/lkml/20190212143003.48446-1-john.ogness@linutronix.de/
> [2] H3ULCB> speaker-test -f24_LE -c2 -t wav -Dplughw:rcarsound -b 4000
>     https://vocaroo.com/9NV98mMgdjX
> [3] https://github.com/erosca/linux/tree/stress-serial
> [4] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/

  reply	other threads:[~2020-01-21 23:57 UTC|newest]

Thread overview: 147+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-12 14:29 [RFC PATCH v1 00/25] printk: new implementation John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 01/25] printk-rb: add printk ring buffer documentation John Ogness
2019-02-12 14:45   ` Greg Kroah-Hartman
2019-02-12 14:29 ` [RFC PATCH v1 02/25] printk-rb: add prb locking functions John Ogness
2019-02-13 15:45   ` Petr Mladek
2019-02-13 21:39     ` John Ogness
2019-02-14 10:33       ` Petr Mladek
2019-02-14 12:10         ` John Ogness
2019-02-15 10:26           ` Petr Mladek
2019-02-15 10:56             ` John Ogness
2019-03-07  2:12   ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 03/25] printk-rb: define ring buffer struct and initializer John Ogness
2019-02-12 14:46   ` Greg Kroah-Hartman
2019-02-14 12:46     ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 04/25] printk-rb: add writer interface John Ogness
2019-02-14 15:16   ` Petr Mladek
2019-02-14 23:36     ` John Ogness
2019-02-15  1:19       ` John Ogness
2019-02-15 13:47       ` Petr Mladek
2019-02-17  1:32         ` John Ogness
2019-02-21 13:51           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 05/25] printk-rb: add basic non-blocking reading interface John Ogness
2019-02-18 12:54   ` Petr Mladek
2019-02-19 21:44     ` John Ogness
2019-02-21 16:22       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 06/25] printk-rb: add blocking reader support John Ogness
2019-02-18 14:05   ` Petr Mladek
2019-02-19 21:47     ` John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 07/25] printk-rb: add functionality required by printk John Ogness
2019-02-12 17:15   ` Linus Torvalds
2019-02-13  9:20     ` John Ogness
2019-02-18 15:59   ` Petr Mladek
2019-02-19 22:08     ` John Ogness
2019-02-22  9:58       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 08/25] printk: add ring buffer and kthread John Ogness
2019-02-12 15:47   ` Sergey Senozhatsky
2019-02-19 13:54   ` Petr Mladek
2019-03-04  7:38   ` Sergey Senozhatsky
2019-03-04 10:00     ` Sergey Senozhatsky
2019-03-04 11:07       ` Sergey Senozhatsky
2019-03-05 21:00         ` John Ogness
2019-03-06 15:57           ` Petr Mladek
2019-03-06 21:17             ` John Ogness
2019-03-06 22:22               ` John Ogness
2019-03-07  6:41                 ` Sergey Senozhatsky
2019-03-07  6:51                   ` Sergey Senozhatsky
2019-03-07 12:50               ` Petr Mladek
2019-03-07  5:15           ` Sergey Senozhatsky
2019-03-11 10:51             ` John Ogness
2019-03-12  9:58               ` Sergey Senozhatsky
2019-03-12 10:30               ` Petr Mladek
2019-03-07 12:06     ` John Ogness
2019-03-08  1:31       ` Sergey Senozhatsky
2019-03-08 10:04         ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 09/25] printk: remove exclusive console hack John Ogness
2019-02-19 14:03   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 10/25] printk: redirect emit/store to new ringbuffer John Ogness
2019-02-20  9:01   ` Petr Mladek
2019-02-20 21:25     ` John Ogness
2019-02-22 14:43       ` Petr Mladek
2019-02-22 15:06         ` John Ogness
2019-02-22 15:25           ` Petr Mladek
2019-02-25 12:11       ` Petr Mladek
2019-02-25 16:41         ` John Ogness
2019-02-26  9:45           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 11/25] printk_safe: remove printk safe code John Ogness
2019-02-22 10:37   ` Petr Mladek
2019-02-22 13:38     ` John Ogness
2019-02-22 15:15       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 12/25] printk: minimize console locking implementation John Ogness
2019-02-25 13:44   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 13/25] printk: track seq per console John Ogness
2019-02-25 14:59   ` Petr Mladek
2019-02-26  8:45     ` John Ogness
2019-02-26 13:11       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 14/25] printk: do boot_delay_msec inside printk_delay John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 15/25] printk: print history for new consoles John Ogness
2019-02-26 14:58   ` Petr Mladek
2019-02-26 15:22     ` John Ogness
2019-02-27  9:02       ` Petr Mladek
2019-02-27 10:02         ` John Ogness
2019-02-27 13:12           ` Petr Mladek
2019-03-04  9:24       ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 16/25] printk: implement CON_PRINTBUFFER John Ogness
2019-02-26 15:38   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 17/25] printk: add processor number to output John Ogness
2019-02-13 22:29   ` John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 18/25] console: add write_atomic interface John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 19/25] printk: introduce emergency messages John Ogness
2019-03-07  7:30   ` Sergey Senozhatsky
2019-03-08 10:31     ` Petr Mladek
2019-03-11 12:04       ` John Ogness
2019-03-12  2:51         ` Sergey Senozhatsky
2019-03-12  2:58       ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 20/25] serial: 8250: implement write_atomic John Ogness
2019-02-27  9:46   ` Petr Mladek
2019-02-27 10:32     ` John Ogness
2019-02-27 13:55       ` Petr Mladek
2019-03-08  4:05         ` John Ogness
2019-03-08  4:17           ` John Ogness
2019-03-08 10:28           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 21/25] printk: implement KERN_CONT John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 22/25] printk: implement /dev/kmsg John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 23/25] printk: implement syslog John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 24/25] printk: implement kmsg_dump John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 25/25] printk: remove unused code John Ogness
2019-03-08 14:02   ` Sebastian Andrzej Siewior
2019-03-11  2:46     ` Sergey Senozhatsky
2019-03-11  8:18       ` Sebastian Andrzej Siewior
2019-03-12  9:38         ` Petr Mladek
2019-02-13  1:31 ` [RFC PATCH v1 00/25] printk: new implementation Sergey Senozhatsky
2019-02-13 13:43   ` John Ogness
2019-03-04  6:39     ` Sergey Senozhatsky
2019-02-13  1:41 ` Sergey Senozhatsky
2019-02-13 14:15   ` John Ogness
2019-03-04  5:31     ` Sergey Senozhatsky
2019-02-13  2:55 ` Sergey Senozhatsky
2019-02-13 14:43   ` John Ogness
2019-03-04  5:23     ` Sergey Senozhatsky
2019-03-07  9:53       ` John Ogness
2019-03-08 10:00         ` Petr Mladek
2019-03-11 10:54         ` Sergey Senozhatsky
2019-03-12 12:38           ` Petr Mladek
2019-03-12 15:15             ` John Ogness
2019-03-13  2:15               ` Sergey Senozhatsky
2019-03-13  8:19                 ` John Ogness
2019-03-13  8:40                   ` Sebastian Siewior
2019-03-13  9:27                     ` Sergey Senozhatsky
2019-03-13 10:06                       ` Sergey Senozhatsky
2019-03-14  9:27                       ` Petr Mladek
2019-03-13  8:46                   ` Sergey Senozhatsky
2019-03-14  9:14               ` Petr Mladek
2019-03-14  9:35                 ` John Ogness
2019-03-13  2:00             ` Sergey Senozhatsky
2019-02-13 16:54 ` David Laight
2019-02-13 22:20   ` John Ogness
2020-01-20 23:05 ` Eugeniu Rosca
2020-01-21 23:56   ` John Ogness [this message]
2020-01-22  2:34     ` Eugeniu Rosca
2020-01-22  7:31       ` Geert Uytterhoeven
2020-01-22 16:58         ` Eugeniu Rosca
2020-01-22 19:48           ` Geert Uytterhoeven
2020-01-24 16:09             ` Eugeniu Rosca
2020-01-27 12:32               ` Petr Mladek
2020-01-27 13:45                 ` Eugeniu Rosca
2020-01-22 10:33       ` John Ogness
2020-01-24 12:13         ` Eugeniu Rosca

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87v9p4mkhr.fsf@linutronix.de \
    --to=john.ogness@linutronix.de \
    --cc=akpm@linux-foundation.org \
    --cc=andrew_gabbasov@mentor.com \
    --cc=dean_jenkins@mentor.com \
    --cc=dirk.behme@de.bosch.com \
    --cc=erosca@de.adit-jv.com \
    --cc=geert+renesas@glider.be \
    --cc=gnomes@lxorguk.ukuu.org.uk \
    --cc=gregkh@linuxfoundation.org \
    --cc=jslaby@suse.com \
    --cc=kuninori.morimoto.gx@renesas.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-serial@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=pfeiner@google.com \
    --cc=pmladek@suse.com \
    --cc=roscaeugeniu@gmail.com \
    --cc=rostedt@goodmis.org \
    --cc=sanjeev_chugh@mentor.com \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --cc=wonderfly@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).