Linux-Serial Archive on lore.kernel.org
 help / color / Atom feed
From: Eugeniu Rosca <erosca@de.adit-jv.com>
To: John Ogness <john.ogness@linutronix.de>
Cc: <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Geert Uytterhoeven <geert+renesas@glider.be>,
	Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>,
	Andrew Gabbasov <andrew_gabbasov@mentor.com>,
	Sanjeev Chugh <sanjeev_chugh@mentor.com>,
	Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Daniel Wang <wonderfly@google.com>,
	Dean Jenkins <dean_jenkins@mentor.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Dirk Behme <dirk.behme@de.bosch.com>,
	Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
	Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
	<linux-serial@vger.kernel.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Eugeniu Rosca <erosca@de.adit-jv.com>,
	Eugeniu Rosca <roscaeugeniu@gmail.com>
Subject: Re: [RFC PATCH v1 00/25] printk: new implementation
Date: Tue, 21 Jan 2020 00:05:22 +0100
Message-ID: <20200120230522.GA23636@lxhi-065.adit-jv.com> (raw)
In-Reply-To: <20190212143003.48446-1-john.ogness@linutronix.de>

Hello John, all,

Cc: Geert, Morimoto-san,

On Tue, Feb 12, 2019 at 03:29:38PM +0100, John Ogness wrote:
> Hello,
> 
> As probably many of you are aware, the current printk implementation
> has some issues. This series (against 5.0-rc6) makes some fundamental
> changes in an attempt to address these issues. The particular issues I
> am referring to:
> 
> 1. The printk buffer is protected by a global raw spinlock for readers
>    and writers. This restricts the contexts that are allowed to
>    access the buffer.
> 
> 2. Because of #1, NMI and recursive contexts are handled by deferring
>    logging/printing to a spinlock-safe context. This means that
>    messages will not be visible if (for example) the kernel dies in
>    NMI context and the irq_work mechanism does not survive.
> 
> 3. Because of #1, when *not* using features such as PREEMPT_RT, large
>    latencies exist when printing to slow consoles.

This [1] is a fairly old thread, but I only recently stumbled upon it,
while co-investigating below audio distortions [2] on R-Car3 ARM64
boards, which can be reproduced by stressing [3] the serial console.

The investigation started a few months ago, when users reported
audio drops during the first seconds of system startup. Only after
a few weeks it became clear (thanks to some people in Cc) that the
distortions were contributed by the above-average serial console load
during the early boot. Once understood, we were able to come up with
a synthetic test [2-3].

I thought it would be interesting to share below reproduction matrix,
in order to contrast vanilla to linux-rt-devel [4], as well as to
compare various preemption models.
 
                           | Ser.console  Ser.console
                           | stressed     at rest or disabled
      --------------------------------------------
      v5.5-rc6 (PREEMPT=y) | distorted    clean
    v5.4.5-rt3 (PREEMPT=y) | distorted    clean
 v5.4.5-rt3 (PREEMPT_RT=y) | clean        clean

My feeling is that the results probably do not surprise linux-rt people.

My first question is, should there be any improvement in the case of
v5.4.5-rt3 (PREEMPT=y), which I do not sense? I would expect so, based
on the cover letter of this series (pointing out the advantages of the
redesigned printk mechanism).

And the other question is, how would you, generally speaking, tackle
the problem, given that backporting the linux-rt patches is *not* an
option and enabling serial console is a must?

[1] https://lore.kernel.org/lkml/20190212143003.48446-1-john.ogness@linutronix.de/
[2] H3ULCB> speaker-test -f24_LE -c2 -t wav -Dplughw:rcarsound -b 4000
    https://vocaroo.com/9NV98mMgdjX
[3] https://github.com/erosca/linux/tree/stress-serial
[4] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/

-- 
Best Regards,
Eugeniu

  parent reply index

Thread overview: 147+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-12 14:29 John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 01/25] printk-rb: add printk ring buffer documentation John Ogness
2019-02-12 14:45   ` Greg Kroah-Hartman
2019-02-12 14:29 ` [RFC PATCH v1 02/25] printk-rb: add prb locking functions John Ogness
2019-02-13 15:45   ` Petr Mladek
2019-02-13 21:39     ` John Ogness
2019-02-14 10:33       ` Petr Mladek
2019-02-14 12:10         ` John Ogness
2019-02-15 10:26           ` Petr Mladek
2019-02-15 10:56             ` John Ogness
2019-03-07  2:12   ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 03/25] printk-rb: define ring buffer struct and initializer John Ogness
2019-02-12 14:46   ` Greg Kroah-Hartman
2019-02-14 12:46     ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 04/25] printk-rb: add writer interface John Ogness
2019-02-14 15:16   ` Petr Mladek
2019-02-14 23:36     ` John Ogness
2019-02-15  1:19       ` John Ogness
2019-02-15 13:47       ` Petr Mladek
2019-02-17  1:32         ` John Ogness
2019-02-21 13:51           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 05/25] printk-rb: add basic non-blocking reading interface John Ogness
2019-02-18 12:54   ` Petr Mladek
2019-02-19 21:44     ` John Ogness
2019-02-21 16:22       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 06/25] printk-rb: add blocking reader support John Ogness
2019-02-18 14:05   ` Petr Mladek
2019-02-19 21:47     ` John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 07/25] printk-rb: add functionality required by printk John Ogness
2019-02-12 17:15   ` Linus Torvalds
2019-02-13  9:20     ` John Ogness
2019-02-18 15:59   ` Petr Mladek
2019-02-19 22:08     ` John Ogness
2019-02-22  9:58       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 08/25] printk: add ring buffer and kthread John Ogness
2019-02-12 15:47   ` Sergey Senozhatsky
2019-02-19 13:54   ` Petr Mladek
2019-03-04  7:38   ` Sergey Senozhatsky
2019-03-04 10:00     ` Sergey Senozhatsky
2019-03-04 11:07       ` Sergey Senozhatsky
2019-03-05 21:00         ` John Ogness
2019-03-06 15:57           ` Petr Mladek
2019-03-06 21:17             ` John Ogness
2019-03-06 22:22               ` John Ogness
2019-03-07  6:41                 ` Sergey Senozhatsky
2019-03-07  6:51                   ` Sergey Senozhatsky
2019-03-07 12:50               ` Petr Mladek
2019-03-07  5:15           ` Sergey Senozhatsky
2019-03-11 10:51             ` John Ogness
2019-03-12  9:58               ` Sergey Senozhatsky
2019-03-12 10:30               ` Petr Mladek
2019-03-07 12:06     ` John Ogness
2019-03-08  1:31       ` Sergey Senozhatsky
2019-03-08 10:04         ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 09/25] printk: remove exclusive console hack John Ogness
2019-02-19 14:03   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 10/25] printk: redirect emit/store to new ringbuffer John Ogness
2019-02-20  9:01   ` Petr Mladek
2019-02-20 21:25     ` John Ogness
2019-02-22 14:43       ` Petr Mladek
2019-02-22 15:06         ` John Ogness
2019-02-22 15:25           ` Petr Mladek
2019-02-25 12:11       ` Petr Mladek
2019-02-25 16:41         ` John Ogness
2019-02-26  9:45           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 11/25] printk_safe: remove printk safe code John Ogness
2019-02-22 10:37   ` Petr Mladek
2019-02-22 13:38     ` John Ogness
2019-02-22 15:15       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 12/25] printk: minimize console locking implementation John Ogness
2019-02-25 13:44   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 13/25] printk: track seq per console John Ogness
2019-02-25 14:59   ` Petr Mladek
2019-02-26  8:45     ` John Ogness
2019-02-26 13:11       ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 14/25] printk: do boot_delay_msec inside printk_delay John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 15/25] printk: print history for new consoles John Ogness
2019-02-26 14:58   ` Petr Mladek
2019-02-26 15:22     ` John Ogness
2019-02-27  9:02       ` Petr Mladek
2019-02-27 10:02         ` John Ogness
2019-02-27 13:12           ` Petr Mladek
2019-03-04  9:24       ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 16/25] printk: implement CON_PRINTBUFFER John Ogness
2019-02-26 15:38   ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 17/25] printk: add processor number to output John Ogness
2019-02-13 22:29   ` John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 18/25] console: add write_atomic interface John Ogness
2019-02-12 14:29 ` [RFC PATCH v1 19/25] printk: introduce emergency messages John Ogness
2019-03-07  7:30   ` Sergey Senozhatsky
2019-03-08 10:31     ` Petr Mladek
2019-03-11 12:04       ` John Ogness
2019-03-12  2:51         ` Sergey Senozhatsky
2019-03-12  2:58       ` Sergey Senozhatsky
2019-02-12 14:29 ` [RFC PATCH v1 20/25] serial: 8250: implement write_atomic John Ogness
2019-02-27  9:46   ` Petr Mladek
2019-02-27 10:32     ` John Ogness
2019-02-27 13:55       ` Petr Mladek
2019-03-08  4:05         ` John Ogness
2019-03-08  4:17           ` John Ogness
2019-03-08 10:28           ` Petr Mladek
2019-02-12 14:29 ` [RFC PATCH v1 21/25] printk: implement KERN_CONT John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 22/25] printk: implement /dev/kmsg John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 23/25] printk: implement syslog John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 24/25] printk: implement kmsg_dump John Ogness
2019-02-12 14:30 ` [RFC PATCH v1 25/25] printk: remove unused code John Ogness
2019-03-08 14:02   ` Sebastian Andrzej Siewior
2019-03-11  2:46     ` Sergey Senozhatsky
2019-03-11  8:18       ` Sebastian Andrzej Siewior
2019-03-12  9:38         ` Petr Mladek
2019-02-13  1:31 ` [RFC PATCH v1 00/25] printk: new implementation Sergey Senozhatsky
2019-02-13 13:43   ` John Ogness
2019-03-04  6:39     ` Sergey Senozhatsky
2019-02-13  1:41 ` Sergey Senozhatsky
2019-02-13 14:15   ` John Ogness
2019-03-04  5:31     ` Sergey Senozhatsky
2019-02-13  2:55 ` Sergey Senozhatsky
2019-02-13 14:43   ` John Ogness
2019-03-04  5:23     ` Sergey Senozhatsky
2019-03-07  9:53       ` John Ogness
2019-03-08 10:00         ` Petr Mladek
2019-03-11 10:54         ` Sergey Senozhatsky
2019-03-12 12:38           ` Petr Mladek
2019-03-12 15:15             ` John Ogness
2019-03-13  2:15               ` Sergey Senozhatsky
2019-03-13  8:19                 ` John Ogness
2019-03-13  8:40                   ` Sebastian Siewior
2019-03-13  9:27                     ` Sergey Senozhatsky
2019-03-13 10:06                       ` Sergey Senozhatsky
2019-03-14  9:27                       ` Petr Mladek
2019-03-13  8:46                   ` Sergey Senozhatsky
2019-03-14  9:14               ` Petr Mladek
2019-03-14  9:35                 ` John Ogness
2019-03-13  2:00             ` Sergey Senozhatsky
2019-02-13 16:54 ` David Laight
2019-02-13 22:20   ` John Ogness
2020-01-20 23:05 ` Eugeniu Rosca [this message]
2020-01-21 23:56   ` John Ogness
2020-01-22  2:34     ` Eugeniu Rosca
2020-01-22  7:31       ` Geert Uytterhoeven
2020-01-22 16:58         ` Eugeniu Rosca
2020-01-22 19:48           ` Geert Uytterhoeven
2020-01-24 16:09             ` Eugeniu Rosca
2020-01-27 12:32               ` Petr Mladek
2020-01-27 13:45                 ` Eugeniu Rosca
2020-01-22 10:33       ` John Ogness
2020-01-24 12:13         ` Eugeniu Rosca

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200120230522.GA23636@lxhi-065.adit-jv.com \
    --to=erosca@de.adit-jv.com \
    --cc=akpm@linux-foundation.org \
    --cc=andrew_gabbasov@mentor.com \
    --cc=dean_jenkins@mentor.com \
    --cc=dirk.behme@de.bosch.com \
    --cc=geert+renesas@glider.be \
    --cc=gnomes@lxorguk.ukuu.org.uk \
    --cc=gregkh@linuxfoundation.org \
    --cc=john.ogness@linutronix.de \
    --cc=jslaby@suse.com \
    --cc=kuninori.morimoto.gx@renesas.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-serial@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=pfeiner@google.com \
    --cc=pmladek@suse.com \
    --cc=roscaeugeniu@gmail.com \
    --cc=rostedt@goodmis.org \
    --cc=sanjeev_chugh@mentor.com \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --cc=wonderfly@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Serial Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-serial/0 linux-serial/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-serial linux-serial/ https://lore.kernel.org/linux-serial \
		linux-serial@vger.kernel.org
	public-inbox-index linux-serial

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-serial


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git