[GIT PULL] printk for 5.20

[GIT PULL] printk for 5.20

[Petr Mladek]

Hi Linus,

please pull the latest printk changes from

  git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git tags/printk-for-5.20

=====================================

- Completely disable printing on consoles with CONFIG_RT. They break
  RT guarantees. They are too slow to be called with raw spinlocks.
  They can't be called with normal spinlocks in an atomic context because
  RT spinlocks might sleep. It is going to get fixed by the kthreads and
  atomic consoles. In the meantime, the debugging of CONFIG_RT kernel
  will be possible only by the user space kernel log readers.

- Allow to read kernel log in gdb even on 32 bits systems.

- More granular check of the buffer usage in printf selftest.

- Clang warning fix.

----------------------------------------------------------------
Antonio Borneo (1):
      scripts/gdb: fix 'lx-dmesg' on 32 bits arch

Justin Stitt (1):
      lib/test_printf.c: fix clang -Wformat warnings

Petr Mladek (2):
      Merge branch 'for-5.20' into for-linus
      Merge branch 'rework/kthreads' into for-linus

Rasmus Villemoes (1):
      lib/test_printf.c: split write-beyond-buffer check in two

Sebastian Andrzej Siewior (1):
      printk: Skip console drivers on PREEMPT_RT.

 kernel/printk/printk.c     | 10 ++++++++++
 lib/test_printf.c          | 21 +++++++++++++++++----
 scripts/gdb/linux/dmesg.py |  9 +++------
 scripts/gdb/linux/utils.py | 14 ++++++++++++--
 4 files changed, 42 insertions(+), 12 deletions(-)

RE: [GIT PULL] printk for 5.20

[David Laight]

> - Completely disable printing on consoles with CONFIG_RT. They break
>   RT guarantees. They are too slow to be called with raw spinlocks.
>   They can't be called with normal spinlocks in an atomic context because
>   RT spinlocks might sleep. It is going to get fixed by the kthreads and
>   atomic consoles. In the meantime, the debugging of CONFIG_RT kernel
>   will be possible only by the user space kernel log readers.

SYSV used to be like that if you were running syslogd.
Used to make debugging some issues completely impossible.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [GIT PULL] printk for 5.20

[Petr Mladek]

On Mon 2022-08-01 15:45:13, David Laight wrote:
> > - Completely disable printing on consoles with CONFIG_RT. They break
> >   RT guarantees. They are too slow to be called with raw spinlocks.
> >   They can't be called with normal spinlocks in an atomic context because
> >   RT spinlocks might sleep. It is going to get fixed by the kthreads and
> >   atomic consoles. In the meantime, the debugging of CONFIG_RT kernel
> >   will be possible only by the user space kernel log readers.
> 
> SYSV used to be like that if you were running syslogd.
> Used to make debugging some issues completely impossible.

Sure. My understanding is that this is how RT guys want it in
4.19. There is no quick magic solution. Either printk() would break
RT guarantees or messages will not be seen on consoles. The blank
consoles were chosen as the least bad solution. Of course, we
are still working on the printk kthreads and atomic consoles
that would allow debugging using consoles on RT again.

Best Regards,
Petr

Re: [GIT PULL] printk for 5.20

[Linus Torvalds]

On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote:
>
> - Completely disable printing on consoles with CONFIG_RT.

I don't think this is acceptable.

We don't suddenly change behavior just because some random developer
has decided "this is the RightThing(tm) to do".

Users matter.

For all we know, there may be random users who are playing around with
PREEMPT_RT. They don't *have* to, but they want to.

Just saying "you get no console because you wanted to try it out" is
simply not acceptable.

It's also NOT SANE.

Seriously, even if you have strict RT requirements, you may also have
strict debugging requirements, and if something goes wrong, you want
to KNOW ABOUT IT. At that point, your RT rules may well fly out the
window, because you have more serious problems.

End result: no way will I accept this kind of completely arbitrary and
frankly not very intelligent patch.

If people want to disable console printing, that's THEIR CHOICE. It
could be a new config variable where you ASK people about what they
want. Not this kind of idiotic tying together of things.

And guys, I want to make it really clear how disappointed I am with
the printk tree lately. There seems to be some kind of hardline
religious fervor having taken over to make these kinds of "this is how
it has to be done, screw any sanity or common sense".

There is exactly one thing you should hold sacred: don't break
people's setups. All the rest is just engineering, and a HUGE part of
"engineering" is to realize that everything is a trade-off.

Linux kernel development is a pragmatic thing where existing users and
existing code matters, and you don't get to just throw it all away
because you have some odd personal hangup.

And printing messages to a console is not some "oh, we'll just stop
doing that because you asked for PREEMPT_RT".

Put another way: not only am I not pulling this, I'm concerned that I
will not be pulling printk patches in the future either because of
where these pull requests seem to be trending.

               Linus

Re: [GIT PULL] printk for 5.20

[Sebastian Andrzej Siewior]

On 2022-08-02 20:19:34 [-0700], Linus Torvalds wrote:
> Seriously, even if you have strict RT requirements, you may also have
> strict debugging requirements, and if something goes wrong, you want
> to KNOW ABOUT IT. At that point, your RT rules may well fly out the
> window, because you have more serious problems.
> 
> End result: no way will I accept this kind of completely arbitrary and
> frankly not very intelligent patch.
> 
> If people want to disable console printing, that's THEIR CHOICE. It
> could be a new config variable where you ASK people about what they
> want. Not this kind of idiotic tying together of things.

I don't argue about the loss in debugging with disabled consoles and I
spare you the time explaining how it got there.

Petr, do you revert it yourself or do you want me to send a revert?

>                Linus

Sebastian

RE: [GIT PULL] printk for 5.20

[David Laight]

> On 2022-08-02 20:19:34 [-0700], Linus Torvalds wrote:
> > Seriously, even if you have strict RT requirements, you may also have
> > strict debugging requirements, and if something goes wrong, you want
> > to KNOW ABOUT IT. At that point, your RT rules may well fly out the
> > window, because you have more serious problems.
> >
> > End result: no way will I accept this kind of completely arbitrary and
> > frankly not very intelligent patch.
> >
> > If people want to disable console printing, that's THEIR CHOICE. It
> > could be a new config variable where you ASK people about what they
> > want. Not this kind of idiotic tying together of things.

There is already the sysctl kernel.printk
That can be used to limit the effects of console logging.

I don't actually remember the 'threaded console' stuff being
enabled/disabled in much the same way.

I end up setting '6 4 1 7' to avoid serial port delays.
But (in some sense) it might be useful to have the supressed
'info' messages output by a thread.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [GIT PULL] printk for 5.20

[Petr Mladek]

On Tue 2022-08-02 20:19:34, Linus Torvalds wrote:
> On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote:
> >
> > - Completely disable printing on consoles with CONFIG_RT.
> 
> I don't think this is acceptable.
> 
> We don't suddenly change behavior just because some random developer
> has decided "this is the RightThing(tm) to do".
> 
> Users matter.

I fully agree.


> For all we know, there may be random users who are playing around with
> PREEMPT_RT. They don't *have* to, but they want to.
>
> Just saying "you get no console because you wanted to try it out" is
> simply not acceptable.

This is where I probably made a mistake. I know that PREEMPT_RT is not
production ready in upstream. And I am not sure what people playing
with it expect.

My first reaction was that the patch was a joke. I tried to
formulate the concerns somehow, see
https://lore.kernel.org/all/Yt6MzEEFfpyTBIIj@alley/

Hmm, I ignored my intuition and let people familiar with PREEMPT_RT
decide. I knew that John was working on the proper solution so
it was not supposed to be final one.


> Seriously, even if you have strict RT requirements, you may also have
> strict debugging requirements, and if something goes wrong, you want
> to KNOW ABOUT IT. At that point, your RT rules may well fly out the
> window, because you have more serious problems.
>
> End result: no way will I accept this kind of completely arbitrary and
> frankly not very intelligent patch.
> 
> If people want to disable console printing, that's THEIR CHOICE. It
> could be a new config variable where you ASK people about what they
> want. Not this kind of idiotic tying together of things.
> 
> And guys, I want to make it really clear how disappointed I am with
> the printk tree lately. There seems to be some kind of hardline
> religious fervor having taken over to make these kinds of "this is how
> it has to be done, screw any sanity or common sense".
>
> There is exactly one thing you should hold sacred: don't break
> people's setups. All the rest is just engineering, and a HUGE part of
> "engineering" is to realize that everything is a trade-off.
>
> Linux kernel development is a pragmatic thing where existing users and
> existing code matters, and you don't get to just throw it all away
> because you have some odd personal hangup.
>
> And printing messages to a console is not some "oh, we'll just stop
> doing that because you asked for PREEMPT_RT".

My thinking was that PREEMPT_RT was used only by some rather small
community that was very well aware of the upstream status. I kind of
though that this was their choice.

I think that I underestimated political and human influences.


> Put another way: not only am I not pulling this, I'm concerned that I
> will not be pulling printk patches in the future either because of
> where these pull requests seem to be trending.

I admit that _I did a big mistake_ by this "disable consoles on RT"
patch. It broke many principles and it was a real hack.

On the positive side. My intuition told me that it was very controversial.
This is why I clearly described the effect. And it was the very first
sentence in the commit message. I think that I made it _very visible_.

The previous merge window was different. We tried to get into mainline
a feature that many people wanted for years (since 2012). We though
that it was ready but it wasn't and we took it back in time.


Otherwise, I think that I am quite demanding maintainer. I focus on
that the change must make sense, must not break existing behavior,
any user interface must be sane, the code must be readable and
maintainable.

I do mistakes. But I have learned big lessons last and this merge
window. I am going to believe more into my intuition and be more
strict.

I am going to take a break and think twice before sending any
further pull request.

Best Regards,
Petr

Re: [GIT PULL] printk for 5.20

[Steven Rostedt]

On Wed, 3 Aug 2022 17:43:12 +0200
Petr Mladek <pmladek@suse.com> wrote:

> On the positive side. My intuition told me that it was very controversial.
> This is why I clearly described the effect. And it was the very first
> sentence in the commit message. I think that I made it _very visible_.

Right. I do not see you doing anything to lose Linus's trust. You stated
upfront exactly what it was doing. And let Linus decide if it was right or
not. Maybe it was a misjudgment on your part. But chalk it up as a learning
experience. I personally find you a good maintainer.

-- Steve

Re: [GIT PULL] printk for 5.20

[Linus Torvalds]

On Wed, Aug 3, 2022 at 8:43 AM Petr Mladek <pmladek@suse.com> wrote:
>
> On Tue 2022-08-02 20:19:34, Linus Torvalds wrote:
> > And printing messages to a console is not some "oh, we'll just stop
> > doing that because you asked for PREEMPT_RT".
>
> My thinking was that PREEMPT_RT was used only by some rather small
> community that was very well aware of the upstream status. I kind of
> though that this was their choice.

Oh, I agree that it probably is a pretty small community.

And I also think that people who are really into RT are basically
always going to have extra patches anyway - I think the bulk of the
core stuff has made it upstream, but not *all* has made it.

And the "real RT" people tend to also have long lead times - it's not
just about "we need guaranteed latency", it also tends to be about
"our hardware is special and stays around for years" too - and likely
wouldn't ever really use upstream kernels directly anyway.

In fact, I don't think anybody can currently even enable PREEMPT_RT in
an upstream kernel anyway without extra patches. Much of the RT
infrastructure has been merged, but some of the grottier parts are
literally just "to make it easier to maintain the real external
patch".

So I agree with you that in reality it probably wouldn't really affect
very many people, if any.

I suspect the most immediate effect would literally be people who want
to experiment with it, "just because".

Not the serious RT users who probably have special hardware anyway and
are likely to also have special debug interfaces (exactly _because_
they have special latency concerns).

So that's why I'd suspect that the actual effect would be on people
who just want to tinker with it, and download the necessary RT patches
and set up some data acquisition station for their own use or
whatever.

But thinking some more about it, even the "serious RT" people almost
certainly don't really want some kind of static "disable it all". Not
even if it was a separate Kconfig question like I suggested.

You'd most likely want it to be dynamic, because things like "log to
console" is different at bootup when the system hasn't started yet -
you can't really have realtime response when your hardware hasn't even
initialized yet - and when things are actually running.

So I think even then you really just want a "turn off console logging"
dynamic flag, not a Kconfig option.

Which I think we already have, in the form of log levels. No?

          Linus

Re: [GIT PULL] printk for 5.20

[Sebastian Andrzej Siewior]

On 2022-08-03 09:08:07 [-0700], Linus Torvalds wrote:
> But thinking some more about it, even the "serious RT" people almost
> certainly don't really want some kind of static "disable it all". Not
> even if it was a separate Kconfig question like I suggested.
> 
> You'd most likely want it to be dynamic, because things like "log to
> console" is different at bootup when the system hasn't started yet -
> you can't really have realtime response when your hardware hasn't even
> initialized yet - and when things are actually running.
> 
> So I think even then you really just want a "turn off console logging"
> dynamic flag, not a Kconfig option.

Having such a flag would mean we have to set it starting with
SYSTEM_SCHEDULING because otherwise we risk a deadlock once the console
driver tries to acquire its lock. The printk code invokes the console
drivers with disabled interrupts and the console driver use spinlock_t
locking which become sleeping locks on PREEMPT_RT and must not be
acquired with disabled interrupts.
The long term solution is to have a printing thread for the console
driver and the atomic console support to print directly in an emergency
case.

> Which I think we already have, in the form of log levels. No?
> 
>           Linus

Sebastian

Re: [GIT PULL] printk for 5.20

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Wed, Aug 3, 2022 at 8:43 AM Petr Mladek <pmladek@suse.com> wrote:
> >
> > On Tue 2022-08-02 20:19:34, Linus Torvalds wrote:
> > > And printing messages to a console is not some "oh, we'll just stop
> > > doing that because you asked for PREEMPT_RT".
> >
> > My thinking was that PREEMPT_RT was used only by some rather small
> > community that was very well aware of the upstream status. I kind of
> > though that this was their choice.
> 
> Oh, I agree that it probably is a pretty small community.
> 
> And I also think that people who are really into RT are basically
> always going to have extra patches anyway - I think the bulk of the
> core stuff has made it upstream, but not *all* has made it.
> 
> And the "real RT" people tend to also have long lead times - it's not
> just about "we need guaranteed latency", it also tends to be about
> "our hardware is special and stays around for years" too - and likely
> wouldn't ever really use upstream kernels directly anyway.
> 
> In fact, I don't think anybody can currently even enable PREEMPT_RT in
> an upstream kernel anyway without extra patches. Much of the RT
> infrastructure has been merged, but some of the grottier parts are
> literally just "to make it easier to maintain the real external
> patch".
> 
> So I agree with you that in reality it probably wouldn't really affect
> very many people, if any.
> 
> I suspect the most immediate effect would literally be people who want
> to experiment with it, "just because".
> 
> Not the serious RT users who probably have special hardware anyway and
> are likely to also have special debug interfaces (exactly _because_
> they have special latency concerns).

As a side note, Red Hat is productizing -rt, and in general lots of systems 
with non-broken hardware will work mostly fine under -rt. For the really 
hairy hard-realtime usecases a lot of verification is done - often as part 
of the project.

With all the 'edge computing' usecases arising & the automotive industry 
getting much more software-intense, I think it's a safe policy to make -rt 
less special & adhere to the same quality and upstream maintenance 
standards as regular Linux distributions.

In a few years PREEMPT_RT won't be all that special anymore, and working 
consoles are very much part of a usable product.

> So that's why I'd suspect that the actual effect would be on people who 
> just want to tinker with it, and download the necessary RT patches and 
> set up some data acquisition station for their own use or whatever.
> 
> But thinking some more about it, even the "serious RT" people almost 
> certainly don't really want some kind of static "disable it all". Not 
> even if it was a separate Kconfig question like I suggested.
> 
> You'd most likely want it to be dynamic, because things like "log to 
> console" is different at bootup when the system hasn't started yet - you 
> can't really have realtime response when your hardware hasn't even 
> initialized yet - and when things are actually running.
> 
> So I think even then you really just want a "turn off console logging" 
> dynamic flag, not a Kconfig option.
> 
> Which I think we already have, in the form of log levels. No?

Yeah:

  CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
  CONFIG_CONSOLE_LOGLEVEL_QUIET=4

What we could do is to set the default console loglevel really low by on 
PREEMPT_RT - say to 1. Serious crashes would still show up - but random 
console chatter wouldn't.

Thanks,

	Ingo

Re: [GIT PULL] printk for 5.20

[Thomas Gleixner]

Linus,

On Tue, Aug 02 2022 at 20:19, Linus Torvalds wrote:
> On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote:
>>
>> - Completely disable printing on consoles with CONFIG_RT.
> And guys, I want to make it really clear how disappointed I am with
> the printk tree lately. There seems to be some kind of hardline
> religious fervor having taken over to make these kinds of "this is how
> it has to be done, screw any sanity or common sense".
...
> Put another way: not only am I not pulling this, I'm concerned that I
> will not be pulling printk patches in the future either because of
> where these pull requests seem to be trending.

I really have to stand up for the printk maintainers here.

Especially Petr has done an extraordinary job in the past few years.
There have been hickups, but with such a semantically ill defined
mechanism like printk() that's not a surprise at all.

Let me add some historical background here.

In Sept. 2019, i.e. almost 3 years ago, we all - including you - sat
together at Plumbers in Lisbon and agreed that printk() in it's back
then form is a nightmare not only for PREEMPT_RT. We also agreed back
then that seperating out the console writes into individual printk
threads makes sense because it also gets rid of the fully serializing
nature of printk() which exists for historical but not for technical
reasons. Also the magic oops_in_progress heuristics have been declared
to be just duct tape, as the still existing (after 25+ years of Linux)
situation with graphics demonstrates on a daily basis. We all agreed
that a dedicated atomic_write() which also allows to prioritize your
favourite pet pieve of making laptops more debugable by utilizing
persistant storage simpler and what's more important more reliable.

It took whopping 2.5 years to get to the point to reach this seperation
as it required to rewrite the buffers and other infrastructure. This was
very responsible and cautionosly guided by Petr and the other members of
the printk() maintainer team. Along with that went quite some
improvements like realistic timestamping and other details which matter
for dmesg power users.

The printk threads were added in the 5.19 merge window and unfortunately
reverted between 5.19-rc3 and 5.19-rc4 for the very wrong reason:

   Some embedded boards failed to boot.

     The root cause is missing locking in the init functions of the
     related UART drivers.

     This is not a problem of threaded printk(), Those are existing bugs
     in these drivers which can be triggered on a 5.18 kernel. They are
     hard to trigger and nobody cared so far because they were neither
     bisectable nor reliably reproducible. The threaded printk() change
     made them reproducible and the bisect pointed at the messenger and
     not at the root cause.

Nevertheless it was decided to throw away valuable work for no real good
reason. That revert costs me constantly ~2 seconds of boot time on one my
main development machines. Not much, but ~10% and I know that quite some
folks in the fastboot camp give a leg for 10%.

The right decision would have been to offer this by boot or config
parameter and not to throw the baby out with the bathwater.

I know it's my fault because I was AFK that week...

Coming back to the commit in question which made you (rightfully) upset.

I agree that it should never have happened, but OTOH it's a very clear
message of developer frustration to you:

On one hand you can't get tired of "praising" the RT people about their
responsible approach to solve fundamental shortcomigs in the kernel, but
at the same time you're pulling whatever new fancy "technology" which
comes around the corner and then makes my and your inbox full of
security issues. That's obviously not a problem, right?

Neither is it a problem that all of these "feature" developers can
rightfully ignore PREEMPT_RT and offload all the resulting problems to
the RT people, right?

I might be wrong, but you seem to have a fundamental misunderstanding of
the scope and importance of PREEEMPT_RT:

>  Oh, I agree that it probably is a pretty small community.

Pull the rug under that "pretty small community" and the industrial
world is comming to a grinding halt. That grinding halt will affect your
power grid, your water supply and quite some other essential things
which are listed under Civil Infrastructure. Plus the new fangled edge
computing, modern cars ....

There is a reason that at least some major industry players have
invested into the RT project for the last couple of years. All major
distro players have commercially supported RT offerings for years and
they are surely not doing this just for fun.

There are freeriders as always. Space*, car*, distro* ,,,,

> And I also think that people who are really into RT are basically
> always going to have extra patches anyway - I think the bulk of the
> core stuff has made it upstream, but not *all* has made it.

At the point of this writing the only outstanding issue is printk, which
prevents PREEMPT_RT from being enabled in your tree on x86[64] and
ARM[64].

This still does not justify the commit in question, but clearly points
out that there is an attitude problem.

Which attitude problem?

   The attitude problem that PREEMPT_RT is just a niche, but the larger
   kernel community is happy to consume the benefits of that allegded
   niche without giving anything back.

Think about that.

Thanks,

        Thomas

Re: [GIT PULL] printk for 5.20

[Daniel Vetter]

On Tue, Aug 09, 2022 at 03:18:20AM +0200, Thomas Gleixner wrote:
> On Tue, Aug 02 2022 at 20:19, Linus Torvalds wrote:
> > On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote:
> >>
> >> - Completely disable printing on consoles with CONFIG_RT.
> > And guys, I want to make it really clear how disappointed I am with
> > the printk tree lately. There seems to be some kind of hardline
> > religious fervor having taken over to make these kinds of "this is how
> > it has to be done, screw any sanity or common sense".
> ...
> > Put another way: not only am I not pulling this, I'm concerned that I
> > will not be pulling printk patches in the future either because of
> > where these pull requests seem to be trending.
> 
> I really have to stand up for the printk maintainers here.
> 
> Especially Petr has done an extraordinary job in the past few years.
> There have been hickups, but with such a semantically ill defined
> mechanism like printk() that's not a surprise at all.
> 
> Let me add some historical background here.
> 
> In Sept. 2019, i.e. almost 3 years ago, we all - including you - sat
> together at Plumbers in Lisbon and agreed that printk() in it's back
> then form is a nightmare not only for PREEMPT_RT. We also agreed back
> then that seperating out the console writes into individual printk
> threads makes sense because it also gets rid of the fully serializing
> nature of printk() which exists for historical but not for technical
> reasons. Also the magic oops_in_progress heuristics have been declared
> to be just duct tape, as the still existing (after 25+ years of Linux)
> situation with graphics demonstrates on a daily basis. We all agreed
> that a dedicated atomic_write() which also allows to prioritize your
> favourite pet pieve of making laptops more debugable by utilizing
> persistant storage simpler and what's more important more reliable.

Cutting the more detail stuff below because I'm justing chiming in on the
overall direction.

I want this. Where "want" means by default and across the board, and
"this" the entire printk/console_lock redesign we discussed in Lisbon with
threaded printing and properly separate out emergency output and
everything else. Because ffs I'm so fed up having to deal with
console_lock in graphcis drivers.

This has nothing to do with rt or being niche or whatever but just that
Petr, John and others are tackling some really nasty corners that everyone
just covered with industrial quantities of duct-tape in the past years
(decades?) and finally there's real movement (but a bit slower than I
hoped for really). I visted linutronix folks a few weeks ago and we talked
a bit what we could all do when this mess is sorted and I want it all.

*graphics maintainer out*
-Daniel

> It took whopping 2.5 years to get to the point to reach this seperation
> as it required to rewrite the buffers and other infrastructure. This was
> very responsible and cautionosly guided by Petr and the other members of
> the printk() maintainer team. Along with that went quite some
> improvements like realistic timestamping and other details which matter
> for dmesg power users.
> 
> The printk threads were added in the 5.19 merge window and unfortunately
> reverted between 5.19-rc3 and 5.19-rc4 for the very wrong reason:
> 
>    Some embedded boards failed to boot.
> 
>      The root cause is missing locking in the init functions of the
>      related UART drivers.
> 
>      This is not a problem of threaded printk(), Those are existing bugs
>      in these drivers which can be triggered on a 5.18 kernel. They are
>      hard to trigger and nobody cared so far because they were neither
>      bisectable nor reliably reproducible. The threaded printk() change
>      made them reproducible and the bisect pointed at the messenger and
>      not at the root cause.
> 
> Nevertheless it was decided to throw away valuable work for no real good
> reason. That revert costs me constantly ~2 seconds of boot time on one my
> main development machines. Not much, but ~10% and I know that quite some
> folks in the fastboot camp give a leg for 10%.
> 
> The right decision would have been to offer this by boot or config
> parameter and not to throw the baby out with the bathwater.
> 
> I know it's my fault because I was AFK that week...
> 
> Coming back to the commit in question which made you (rightfully) upset.
> 
> I agree that it should never have happened, but OTOH it's a very clear
> message of developer frustration to you:
> 
> On one hand you can't get tired of "praising" the RT people about their
> responsible approach to solve fundamental shortcomigs in the kernel, but
> at the same time you're pulling whatever new fancy "technology" which
> comes around the corner and then makes my and your inbox full of
> security issues. That's obviously not a problem, right?
> 
> Neither is it a problem that all of these "feature" developers can
> rightfully ignore PREEMPT_RT and offload all the resulting problems to
> the RT people, right?
> 
> I might be wrong, but you seem to have a fundamental misunderstanding of
> the scope and importance of PREEEMPT_RT:
> 
> >  Oh, I agree that it probably is a pretty small community.
> 
> Pull the rug under that "pretty small community" and the industrial
> world is comming to a grinding halt. That grinding halt will affect your
> power grid, your water supply and quite some other essential things
> which are listed under Civil Infrastructure. Plus the new fangled edge
> computing, modern cars ....
> 
> There is a reason that at least some major industry players have
> invested into the RT project for the last couple of years. All major
> distro players have commercially supported RT offerings for years and
> they are surely not doing this just for fun.
> 
> There are freeriders as always. Space*, car*, distro* ,,,,
> 
> > And I also think that people who are really into RT are basically
> > always going to have extra patches anyway - I think the bulk of the
> > core stuff has made it upstream, but not *all* has made it.
> 
> At the point of this writing the only outstanding issue is printk, which
> prevents PREEMPT_RT from being enabled in your tree on x86[64] and
> ARM[64].
> 
> This still does not justify the commit in question, but clearly points
> out that there is an attitude problem.
> 
> Which attitude problem?
> 
>    The attitude problem that PREEMPT_RT is just a niche, but the larger
>    kernel community is happy to consume the benefits of that allegded
>    niche without giving anything back.
> 
> Think about that.
> 
> Thanks,
> 
>         Thomas

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [GIT PULL] printk for 5.20

[Daniel Vetter]

On Tue, Aug 09, 2022 at 03:18:20AM +0200, Thomas Gleixner wrote:
> On Tue, Aug 02 2022 at 20:19, Linus Torvalds wrote:
> > On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote:
> >>
> >> - Completely disable printing on consoles with CONFIG_RT.
> > And guys, I want to make it really clear how disappointed I am with
> > the printk tree lately. There seems to be some kind of hardline
> > religious fervor having taken over to make these kinds of "this is how
> > it has to be done, screw any sanity or common sense".
> ...
> > Put another way: not only am I not pulling this, I'm concerned that I
> > will not be pulling printk patches in the future either because of
> > where these pull requests seem to be trending.
> 
> I really have to stand up for the printk maintainers here.
> 
> Especially Petr has done an extraordinary job in the past few years.
> There have been hickups, but with such a semantically ill defined
> mechanism like printk() that's not a surprise at all.
> 
> Let me add some historical background here.
> 
> In Sept. 2019, i.e. almost 3 years ago, we all - including you - sat
> together at Plumbers in Lisbon and agreed that printk() in it's back
> then form is a nightmare not only for PREEMPT_RT. We also agreed back
> then that seperating out the console writes into individual printk
> threads makes sense because it also gets rid of the fully serializing
> nature of printk() which exists for historical but not for technical
> reasons. Also the magic oops_in_progress heuristics have been declared
> to be just duct tape, as the still existing (after 25+ years of Linux)
> situation with graphics demonstrates on a daily basis. We all agreed
> that a dedicated atomic_write() which also allows to prioritize your
> favourite pet pieve of making laptops more debugable by utilizing
> persistant storage simpler and what's more important more reliable.

Cutting the more detail stuff below because I'm justing chiming in on the
overall direction.

I want this. Where "want" means by default and across the board, and
"this" the entire printk/console_lock redesign we discussed in Lisbon with
threaded printing and properly separate out emergency output and
everything else. Because ffs I'm so fed up having to deal with
console_lock in graphcis drivers.

This has nothing to do with rt or being niche or whatever but just that
Petr, John and others are tackling some really nasty corners that everyone
just covered with industrial quantities of duct-tape in the past years
(decades?) and finally there's real movement (but a bit slower than I
hoped for really). I visted linutronix folks a few weeks ago and we talked
a bit what we could all do when this mess is sorted and I want it all.

*graphics maintainer out*
-Daniel

> It took whopping 2.5 years to get to the point to reach this seperation
> as it required to rewrite the buffers and other infrastructure. This was
> very responsible and cautionosly guided by Petr and the other members of
> the printk() maintainer team. Along with that went quite some
> improvements like realistic timestamping and other details which matter
> for dmesg power users.
> 
> The printk threads were added in the 5.19 merge window and unfortunately
> reverted between 5.19-rc3 and 5.19-rc4 for the very wrong reason:
> 
>    Some embedded boards failed to boot.
> 
>      The root cause is missing locking in the init functions of the
>      related UART drivers.
> 
>      This is not a problem of threaded printk(), Those are existing bugs
>      in these drivers which can be triggered on a 5.18 kernel. They are
>      hard to trigger and nobody cared so far because they were neither
>      bisectable nor reliably reproducible. The threaded printk() change
>      made them reproducible and the bisect pointed at the messenger and
>      not at the root cause.
> 
> Nevertheless it was decided to throw away valuable work for no real good
> reason. That revert costs me constantly ~2 seconds of boot time on one my
> main development machines. Not much, but ~10% and I know that quite some
> folks in the fastboot camp give a leg for 10%.
> 
> The right decision would have been to offer this by boot or config
> parameter and not to throw the baby out with the bathwater.
> 
> I know it's my fault because I was AFK that week...
> 
> Coming back to the commit in question which made you (rightfully) upset.
> 
> I agree that it should never have happened, but OTOH it's a very clear
> message of developer frustration to you:
> 
> On one hand you can't get tired of "praising" the RT people about their
> responsible approach to solve fundamental shortcomigs in the kernel, but
> at the same time you're pulling whatever new fancy "technology" which
> comes around the corner and then makes my and your inbox full of
> security issues. That's obviously not a problem, right?
> 
> Neither is it a problem that all of these "feature" developers can
> rightfully ignore PREEMPT_RT and offload all the resulting problems to
> the RT people, right?
> 
> I might be wrong, but you seem to have a fundamental misunderstanding of
> the scope and importance of PREEEMPT_RT:
> 
> >  Oh, I agree that it probably is a pretty small community.
> 
> Pull the rug under that "pretty small community" and the industrial
> world is comming to a grinding halt. That grinding halt will affect your
> power grid, your water supply and quite some other essential things
> which are listed under Civil Infrastructure. Plus the new fangled edge
> computing, modern cars ....
> 
> There is a reason that at least some major industry players have
> invested into the RT project for the last couple of years. All major
> distro players have commercially supported RT offerings for years and
> they are surely not doing this just for fun.
> 
> There are freeriders as always. Space*, car*, distro* ,,,,
> 
> > And I also think that people who are really into RT are basically
> > always going to have extra patches anyway - I think the bulk of the
> > core stuff has made it upstream, but not *all* has made it.
> 
> At the point of this writing the only outstanding issue is printk, which
> prevents PREEMPT_RT from being enabled in your tree on x86[64] and
> ARM[64].
> 
> This still does not justify the commit in question, but clearly points
> out that there is an attitude problem.
> 
> Which attitude problem?
> 
>    The attitude problem that PREEMPT_RT is just a niche, but the larger
>    kernel community is happy to consume the benefits of that allegded
>    niche without giving anything back.
> 
> Think about that.
> 
> Thanks,
> 
>         Thomas

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch