Re: [RFC PATCH v1 00/25] printk: new implementation

From: Eugeniu Rosca <erosca@de.adit-jv.com>
To: John Ogness <john.ogness@linutronix.de>
Cc: <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Geert Uytterhoeven <geert+renesas@glider.be>,
	Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>,
	Andrew Gabbasov <andrew_gabbasov@mentor.com>,
	Sanjeev Chugh <sanjeev_chugh@mentor.com>,
	Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Daniel Wang <wonderfly@google.com>,
	Dean Jenkins <dean_jenkins@mentor.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Dirk Behme <dirk.behme@de.bosch.com>,
	Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
	Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
	<linux-serial@vger.kernel.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Eugeniu Rosca <erosca@de.adit-jv.com>,
	Eugeniu Rosca <roscaeugeniu@gmail.com>
Subject: Re: [RFC PATCH v1 00/25] printk: new implementation
Date: Tue, 21 Jan 2020 00:05:22 +0100	[thread overview]
Message-ID: <20200120230522.GA23636@lxhi-065.adit-jv.com> (raw)
In-Reply-To: <20190212143003.48446-1-john.ogness@linutronix.de>

Hello John, all,

Cc: Geert, Morimoto-san,

On Tue, Feb 12, 2019 at 03:29:38PM +0100, John Ogness wrote:
> Hello,
> 
> As probably many of you are aware, the current printk implementation
> has some issues. This series (against 5.0-rc6) makes some fundamental
> changes in an attempt to address these issues. The particular issues I
> am referring to:
> 
> 1. The printk buffer is protected by a global raw spinlock for readers
>    and writers. This restricts the contexts that are allowed to
>    access the buffer.
> 
> 2. Because of #1, NMI and recursive contexts are handled by deferring
>    logging/printing to a spinlock-safe context. This means that
>    messages will not be visible if (for example) the kernel dies in
>    NMI context and the irq_work mechanism does not survive.
> 
> 3. Because of #1, when *not* using features such as PREEMPT_RT, large
>    latencies exist when printing to slow consoles.

This [1] is a fairly old thread, but I only recently stumbled upon it,
while co-investigating below audio distortions [2] on R-Car3 ARM64
boards, which can be reproduced by stressing [3] the serial console.

The investigation started a few months ago, when users reported
audio drops during the first seconds of system startup. Only after
a few weeks it became clear (thanks to some people in Cc) that the
distortions were contributed by the above-average serial console load
during the early boot. Once understood, we were able to come up with
a synthetic test [2-3].

I thought it would be interesting to share below reproduction matrix,
in order to contrast vanilla to linux-rt-devel [4], as well as to
compare various preemption models.

                           | Ser.console  Ser.console
                           | stressed     at rest or disabled
      --------------------------------------------
      v5.5-rc6 (PREEMPT=y) | distorted    clean
    v5.4.5-rt3 (PREEMPT=y) | distorted    clean
 v5.4.5-rt3 (PREEMPT_RT=y) | clean        clean

My feeling is that the results probably do not surprise linux-rt people.

My first question is, should there be any improvement in the case of
v5.4.5-rt3 (PREEMPT=y), which I do not sense? I would expect so, based
on the cover letter of this series (pointing out the advantages of the
redesigned printk mechanism).

And the other question is, how would you, generally speaking, tackle
the problem, given that backporting the linux-rt patches is *not* an
option and enabling serial console is a must?

[1] https://lore.kernel.org/lkml/20190212143003.48446-1-john.ogness@linutronix.de/
[2] H3ULCB> speaker-test -f24_LE -c2 -t wav -Dplughw:rcarsound -b 4000
    https://vocaroo.com/9NV98mMgdjX
[3] https://github.com/erosca/linux/tree/stress-serial
[4] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/

-- 
Best Regards,
Eugeniu