* [GIT PULL] printk for 5.20 @ 2022-08-01 15:07 Petr Mladek 2022-08-01 15:45 ` David Laight 2022-08-03 3:19 ` Linus Torvalds 0 siblings, 2 replies; 13+ messages in thread From: Petr Mladek @ 2022-08-01 15:07 UTC (permalink / raw) To: Linus Torvalds Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel Hi Linus, please pull the latest printk changes from git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git tags/printk-for-5.20 ===================================== - Completely disable printing on consoles with CONFIG_RT. They break RT guarantees. They are too slow to be called with raw spinlocks. They can't be called with normal spinlocks in an atomic context because RT spinlocks might sleep. It is going to get fixed by the kthreads and atomic consoles. In the meantime, the debugging of CONFIG_RT kernel will be possible only by the user space kernel log readers. - Allow to read kernel log in gdb even on 32 bits systems. - More granular check of the buffer usage in printf selftest. - Clang warning fix. ---------------------------------------------------------------- Antonio Borneo (1): scripts/gdb: fix 'lx-dmesg' on 32 bits arch Justin Stitt (1): lib/test_printf.c: fix clang -Wformat warnings Petr Mladek (2): Merge branch 'for-5.20' into for-linus Merge branch 'rework/kthreads' into for-linus Rasmus Villemoes (1): lib/test_printf.c: split write-beyond-buffer check in two Sebastian Andrzej Siewior (1): printk: Skip console drivers on PREEMPT_RT. kernel/printk/printk.c | 10 ++++++++++ lib/test_printf.c | 21 +++++++++++++++++---- scripts/gdb/linux/dmesg.py | 9 +++------ scripts/gdb/linux/utils.py | 14 ++++++++++++-- 4 files changed, 42 insertions(+), 12 deletions(-) ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [GIT PULL] printk for 5.20 2022-08-01 15:07 [GIT PULL] printk for 5.20 Petr Mladek @ 2022-08-01 15:45 ` David Laight 2022-08-02 7:37 ` Petr Mladek 2022-08-03 3:19 ` Linus Torvalds 1 sibling, 1 reply; 13+ messages in thread From: David Laight @ 2022-08-01 15:45 UTC (permalink / raw) To: 'Petr Mladek', Linus Torvalds Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel > - Completely disable printing on consoles with CONFIG_RT. They break > RT guarantees. They are too slow to be called with raw spinlocks. > They can't be called with normal spinlocks in an atomic context because > RT spinlocks might sleep. It is going to get fixed by the kthreads and > atomic consoles. In the meantime, the debugging of CONFIG_RT kernel > will be possible only by the user space kernel log readers. SYSV used to be like that if you were running syslogd. Used to make debugging some issues completely impossible. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-01 15:45 ` David Laight @ 2022-08-02 7:37 ` Petr Mladek 0 siblings, 0 replies; 13+ messages in thread From: Petr Mladek @ 2022-08-02 7:37 UTC (permalink / raw) To: David Laight Cc: Linus Torvalds, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On Mon 2022-08-01 15:45:13, David Laight wrote: > > - Completely disable printing on consoles with CONFIG_RT. They break > > RT guarantees. They are too slow to be called with raw spinlocks. > > They can't be called with normal spinlocks in an atomic context because > > RT spinlocks might sleep. It is going to get fixed by the kthreads and > > atomic consoles. In the meantime, the debugging of CONFIG_RT kernel > > will be possible only by the user space kernel log readers. > > SYSV used to be like that if you were running syslogd. > Used to make debugging some issues completely impossible. Sure. My understanding is that this is how RT guys want it in 4.19. There is no quick magic solution. Either printk() would break RT guarantees or messages will not be seen on consoles. The blank consoles were chosen as the least bad solution. Of course, we are still working on the printk kthreads and atomic consoles that would allow debugging using consoles on RT again. Best Regards, Petr ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-01 15:07 [GIT PULL] printk for 5.20 Petr Mladek 2022-08-01 15:45 ` David Laight @ 2022-08-03 3:19 ` Linus Torvalds 2022-08-03 14:35 ` Sebastian Andrzej Siewior ` (2 more replies) 1 sibling, 3 replies; 13+ messages in thread From: Linus Torvalds @ 2022-08-03 3:19 UTC (permalink / raw) To: Petr Mladek Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote: > > - Completely disable printing on consoles with CONFIG_RT. I don't think this is acceptable. We don't suddenly change behavior just because some random developer has decided "this is the RightThing(tm) to do". Users matter. For all we know, there may be random users who are playing around with PREEMPT_RT. They don't *have* to, but they want to. Just saying "you get no console because you wanted to try it out" is simply not acceptable. It's also NOT SANE. Seriously, even if you have strict RT requirements, you may also have strict debugging requirements, and if something goes wrong, you want to KNOW ABOUT IT. At that point, your RT rules may well fly out the window, because you have more serious problems. End result: no way will I accept this kind of completely arbitrary and frankly not very intelligent patch. If people want to disable console printing, that's THEIR CHOICE. It could be a new config variable where you ASK people about what they want. Not this kind of idiotic tying together of things. And guys, I want to make it really clear how disappointed I am with the printk tree lately. There seems to be some kind of hardline religious fervor having taken over to make these kinds of "this is how it has to be done, screw any sanity or common sense". There is exactly one thing you should hold sacred: don't break people's setups. All the rest is just engineering, and a HUGE part of "engineering" is to realize that everything is a trade-off. Linux kernel development is a pragmatic thing where existing users and existing code matters, and you don't get to just throw it all away because you have some odd personal hangup. And printing messages to a console is not some "oh, we'll just stop doing that because you asked for PREEMPT_RT". Put another way: not only am I not pulling this, I'm concerned that I will not be pulling printk patches in the future either because of where these pull requests seem to be trending. Linus ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 3:19 ` Linus Torvalds @ 2022-08-03 14:35 ` Sebastian Andrzej Siewior 2022-08-03 14:46 ` David Laight 2022-08-03 15:43 ` Petr Mladek 2022-08-09 1:18 ` Thomas Gleixner 2 siblings, 1 reply; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2022-08-03 14:35 UTC (permalink / raw) To: Linus Torvalds Cc: Petr Mladek, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On 2022-08-02 20:19:34 [-0700], Linus Torvalds wrote: > Seriously, even if you have strict RT requirements, you may also have > strict debugging requirements, and if something goes wrong, you want > to KNOW ABOUT IT. At that point, your RT rules may well fly out the > window, because you have more serious problems. > > End result: no way will I accept this kind of completely arbitrary and > frankly not very intelligent patch. > > If people want to disable console printing, that's THEIR CHOICE. It > could be a new config variable where you ASK people about what they > want. Not this kind of idiotic tying together of things. I don't argue about the loss in debugging with disabled consoles and I spare you the time explaining how it got there. Petr, do you revert it yourself or do you want me to send a revert? > Linus Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [GIT PULL] printk for 5.20 2022-08-03 14:35 ` Sebastian Andrzej Siewior @ 2022-08-03 14:46 ` David Laight 0 siblings, 0 replies; 13+ messages in thread From: David Laight @ 2022-08-03 14:46 UTC (permalink / raw) To: 'Sebastian Andrzej Siewior', Linus Torvalds Cc: Petr Mladek, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel > On 2022-08-02 20:19:34 [-0700], Linus Torvalds wrote: > > Seriously, even if you have strict RT requirements, you may also have > > strict debugging requirements, and if something goes wrong, you want > > to KNOW ABOUT IT. At that point, your RT rules may well fly out the > > window, because you have more serious problems. > > > > End result: no way will I accept this kind of completely arbitrary and > > frankly not very intelligent patch. > > > > If people want to disable console printing, that's THEIR CHOICE. It > > could be a new config variable where you ASK people about what they > > want. Not this kind of idiotic tying together of things. There is already the sysctl kernel.printk That can be used to limit the effects of console logging. I don't actually remember the 'threaded console' stuff being enabled/disabled in much the same way. I end up setting '6 4 1 7' to avoid serial port delays. But (in some sense) it might be useful to have the supressed 'info' messages output by a thread. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 3:19 ` Linus Torvalds 2022-08-03 14:35 ` Sebastian Andrzej Siewior @ 2022-08-03 15:43 ` Petr Mladek 2022-08-03 16:06 ` Steven Rostedt 2022-08-03 16:08 ` Linus Torvalds 2022-08-09 1:18 ` Thomas Gleixner 2 siblings, 2 replies; 13+ messages in thread From: Petr Mladek @ 2022-08-03 15:43 UTC (permalink / raw) To: Linus Torvalds Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On Tue 2022-08-02 20:19:34, Linus Torvalds wrote: > On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote: > > > > - Completely disable printing on consoles with CONFIG_RT. > > I don't think this is acceptable. > > We don't suddenly change behavior just because some random developer > has decided "this is the RightThing(tm) to do". > > Users matter. I fully agree. > For all we know, there may be random users who are playing around with > PREEMPT_RT. They don't *have* to, but they want to. > > Just saying "you get no console because you wanted to try it out" is > simply not acceptable. This is where I probably made a mistake. I know that PREEMPT_RT is not production ready in upstream. And I am not sure what people playing with it expect. My first reaction was that the patch was a joke. I tried to formulate the concerns somehow, see https://lore.kernel.org/all/Yt6MzEEFfpyTBIIj@alley/ Hmm, I ignored my intuition and let people familiar with PREEMPT_RT decide. I knew that John was working on the proper solution so it was not supposed to be final one. > Seriously, even if you have strict RT requirements, you may also have > strict debugging requirements, and if something goes wrong, you want > to KNOW ABOUT IT. At that point, your RT rules may well fly out the > window, because you have more serious problems. > > End result: no way will I accept this kind of completely arbitrary and > frankly not very intelligent patch. > > If people want to disable console printing, that's THEIR CHOICE. It > could be a new config variable where you ASK people about what they > want. Not this kind of idiotic tying together of things. > > And guys, I want to make it really clear how disappointed I am with > the printk tree lately. There seems to be some kind of hardline > religious fervor having taken over to make these kinds of "this is how > it has to be done, screw any sanity or common sense". > > There is exactly one thing you should hold sacred: don't break > people's setups. All the rest is just engineering, and a HUGE part of > "engineering" is to realize that everything is a trade-off. > > Linux kernel development is a pragmatic thing where existing users and > existing code matters, and you don't get to just throw it all away > because you have some odd personal hangup. > > And printing messages to a console is not some "oh, we'll just stop > doing that because you asked for PREEMPT_RT". My thinking was that PREEMPT_RT was used only by some rather small community that was very well aware of the upstream status. I kind of though that this was their choice. I think that I underestimated political and human influences. > Put another way: not only am I not pulling this, I'm concerned that I > will not be pulling printk patches in the future either because of > where these pull requests seem to be trending. I admit that _I did a big mistake_ by this "disable consoles on RT" patch. It broke many principles and it was a real hack. On the positive side. My intuition told me that it was very controversial. This is why I clearly described the effect. And it was the very first sentence in the commit message. I think that I made it _very visible_. The previous merge window was different. We tried to get into mainline a feature that many people wanted for years (since 2012). We though that it was ready but it wasn't and we took it back in time. Otherwise, I think that I am quite demanding maintainer. I focus on that the change must make sense, must not break existing behavior, any user interface must be sane, the code must be readable and maintainable. I do mistakes. But I have learned big lessons last and this merge window. I am going to believe more into my intuition and be more strict. I am going to take a break and think twice before sending any further pull request. Best Regards, Petr ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 15:43 ` Petr Mladek @ 2022-08-03 16:06 ` Steven Rostedt 2022-08-03 16:08 ` Linus Torvalds 1 sibling, 0 replies; 13+ messages in thread From: Steven Rostedt @ 2022-08-03 16:06 UTC (permalink / raw) To: Petr Mladek Cc: Linus Torvalds, Sergey Senozhatsky, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On Wed, 3 Aug 2022 17:43:12 +0200 Petr Mladek <pmladek@suse.com> wrote: > On the positive side. My intuition told me that it was very controversial. > This is why I clearly described the effect. And it was the very first > sentence in the commit message. I think that I made it _very visible_. Right. I do not see you doing anything to lose Linus's trust. You stated upfront exactly what it was doing. And let Linus decide if it was right or not. Maybe it was a misjudgment on your part. But chalk it up as a learning experience. I personally find you a good maintainer. -- Steve ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 15:43 ` Petr Mladek 2022-08-03 16:06 ` Steven Rostedt @ 2022-08-03 16:08 ` Linus Torvalds 2022-08-03 16:34 ` Sebastian Andrzej Siewior 2022-08-03 17:08 ` Ingo Molnar 1 sibling, 2 replies; 13+ messages in thread From: Linus Torvalds @ 2022-08-03 16:08 UTC (permalink / raw) To: Petr Mladek Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On Wed, Aug 3, 2022 at 8:43 AM Petr Mladek <pmladek@suse.com> wrote: > > On Tue 2022-08-02 20:19:34, Linus Torvalds wrote: > > And printing messages to a console is not some "oh, we'll just stop > > doing that because you asked for PREEMPT_RT". > > My thinking was that PREEMPT_RT was used only by some rather small > community that was very well aware of the upstream status. I kind of > though that this was their choice. Oh, I agree that it probably is a pretty small community. And I also think that people who are really into RT are basically always going to have extra patches anyway - I think the bulk of the core stuff has made it upstream, but not *all* has made it. And the "real RT" people tend to also have long lead times - it's not just about "we need guaranteed latency", it also tends to be about "our hardware is special and stays around for years" too - and likely wouldn't ever really use upstream kernels directly anyway. In fact, I don't think anybody can currently even enable PREEMPT_RT in an upstream kernel anyway without extra patches. Much of the RT infrastructure has been merged, but some of the grottier parts are literally just "to make it easier to maintain the real external patch". So I agree with you that in reality it probably wouldn't really affect very many people, if any. I suspect the most immediate effect would literally be people who want to experiment with it, "just because". Not the serious RT users who probably have special hardware anyway and are likely to also have special debug interfaces (exactly _because_ they have special latency concerns). So that's why I'd suspect that the actual effect would be on people who just want to tinker with it, and download the necessary RT patches and set up some data acquisition station for their own use or whatever. But thinking some more about it, even the "serious RT" people almost certainly don't really want some kind of static "disable it all". Not even if it was a separate Kconfig question like I suggested. You'd most likely want it to be dynamic, because things like "log to console" is different at bootup when the system hasn't started yet - you can't really have realtime response when your hardware hasn't even initialized yet - and when things are actually running. So I think even then you really just want a "turn off console logging" dynamic flag, not a Kconfig option. Which I think we already have, in the form of log levels. No? Linus ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 16:08 ` Linus Torvalds @ 2022-08-03 16:34 ` Sebastian Andrzej Siewior 2022-08-03 17:08 ` Ingo Molnar 1 sibling, 0 replies; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2022-08-03 16:34 UTC (permalink / raw) To: Linus Torvalds Cc: Petr Mladek, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel On 2022-08-03 09:08:07 [-0700], Linus Torvalds wrote: > But thinking some more about it, even the "serious RT" people almost > certainly don't really want some kind of static "disable it all". Not > even if it was a separate Kconfig question like I suggested. > > You'd most likely want it to be dynamic, because things like "log to > console" is different at bootup when the system hasn't started yet - > you can't really have realtime response when your hardware hasn't even > initialized yet - and when things are actually running. > > So I think even then you really just want a "turn off console logging" > dynamic flag, not a Kconfig option. Having such a flag would mean we have to set it starting with SYSTEM_SCHEDULING because otherwise we risk a deadlock once the console driver tries to acquire its lock. The printk code invokes the console drivers with disabled interrupts and the console driver use spinlock_t locking which become sleeping locks on PREEMPT_RT and must not be acquired with disabled interrupts. The long term solution is to have a printing thread for the console driver and the atomic console support to print directly in an emergency case. > Which I think we already have, in the form of log levels. No? > > Linus Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 16:08 ` Linus Torvalds 2022-08-03 16:34 ` Sebastian Andrzej Siewior @ 2022-08-03 17:08 ` Ingo Molnar 1 sibling, 0 replies; 13+ messages in thread From: Ingo Molnar @ 2022-08-03 17:08 UTC (permalink / raw) To: Linus Torvalds Cc: Petr Mladek, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Thomas Gleixner, Jan Kara, Peter Zijlstra, linux-kernel * Linus Torvalds <torvalds@linux-foundation.org> wrote: > On Wed, Aug 3, 2022 at 8:43 AM Petr Mladek <pmladek@suse.com> wrote: > > > > On Tue 2022-08-02 20:19:34, Linus Torvalds wrote: > > > And printing messages to a console is not some "oh, we'll just stop > > > doing that because you asked for PREEMPT_RT". > > > > My thinking was that PREEMPT_RT was used only by some rather small > > community that was very well aware of the upstream status. I kind of > > though that this was their choice. > > Oh, I agree that it probably is a pretty small community. > > And I also think that people who are really into RT are basically > always going to have extra patches anyway - I think the bulk of the > core stuff has made it upstream, but not *all* has made it. > > And the "real RT" people tend to also have long lead times - it's not > just about "we need guaranteed latency", it also tends to be about > "our hardware is special and stays around for years" too - and likely > wouldn't ever really use upstream kernels directly anyway. > > In fact, I don't think anybody can currently even enable PREEMPT_RT in > an upstream kernel anyway without extra patches. Much of the RT > infrastructure has been merged, but some of the grottier parts are > literally just "to make it easier to maintain the real external > patch". > > So I agree with you that in reality it probably wouldn't really affect > very many people, if any. > > I suspect the most immediate effect would literally be people who want > to experiment with it, "just because". > > Not the serious RT users who probably have special hardware anyway and > are likely to also have special debug interfaces (exactly _because_ > they have special latency concerns). As a side note, Red Hat is productizing -rt, and in general lots of systems with non-broken hardware will work mostly fine under -rt. For the really hairy hard-realtime usecases a lot of verification is done - often as part of the project. With all the 'edge computing' usecases arising & the automotive industry getting much more software-intense, I think it's a safe policy to make -rt less special & adhere to the same quality and upstream maintenance standards as regular Linux distributions. In a few years PREEMPT_RT won't be all that special anymore, and working consoles are very much part of a usable product. > So that's why I'd suspect that the actual effect would be on people who > just want to tinker with it, and download the necessary RT patches and > set up some data acquisition station for their own use or whatever. > > But thinking some more about it, even the "serious RT" people almost > certainly don't really want some kind of static "disable it all". Not > even if it was a separate Kconfig question like I suggested. > > You'd most likely want it to be dynamic, because things like "log to > console" is different at bootup when the system hasn't started yet - you > can't really have realtime response when your hardware hasn't even > initialized yet - and when things are actually running. > > So I think even then you really just want a "turn off console logging" > dynamic flag, not a Kconfig option. > > Which I think we already have, in the form of log levels. No? Yeah: CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7 CONFIG_CONSOLE_LOGLEVEL_QUIET=4 What we could do is to set the default console loglevel really low by on PREEMPT_RT - say to 1. Serious crashes would still show up - but random console chatter wouldn't. Thanks, Ingo ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-03 3:19 ` Linus Torvalds 2022-08-03 14:35 ` Sebastian Andrzej Siewior 2022-08-03 15:43 ` Petr Mladek @ 2022-08-09 1:18 ` Thomas Gleixner 2022-08-11 13:38 ` Daniel Vetter 2 siblings, 1 reply; 13+ messages in thread From: Thomas Gleixner @ 2022-08-09 1:18 UTC (permalink / raw) To: Linus Torvalds, Petr Mladek Cc: Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Jan Kara, Peter Zijlstra, linux-kernel, Clark Williams, Mel Gorman Linus, On Tue, Aug 02 2022 at 20:19, Linus Torvalds wrote: > On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote: >> >> - Completely disable printing on consoles with CONFIG_RT. > And guys, I want to make it really clear how disappointed I am with > the printk tree lately. There seems to be some kind of hardline > religious fervor having taken over to make these kinds of "this is how > it has to be done, screw any sanity or common sense". ... > Put another way: not only am I not pulling this, I'm concerned that I > will not be pulling printk patches in the future either because of > where these pull requests seem to be trending. I really have to stand up for the printk maintainers here. Especially Petr has done an extraordinary job in the past few years. There have been hickups, but with such a semantically ill defined mechanism like printk() that's not a surprise at all. Let me add some historical background here. In Sept. 2019, i.e. almost 3 years ago, we all - including you - sat together at Plumbers in Lisbon and agreed that printk() in it's back then form is a nightmare not only for PREEMPT_RT. We also agreed back then that seperating out the console writes into individual printk threads makes sense because it also gets rid of the fully serializing nature of printk() which exists for historical but not for technical reasons. Also the magic oops_in_progress heuristics have been declared to be just duct tape, as the still existing (after 25+ years of Linux) situation with graphics demonstrates on a daily basis. We all agreed that a dedicated atomic_write() which also allows to prioritize your favourite pet pieve of making laptops more debugable by utilizing persistant storage simpler and what's more important more reliable. It took whopping 2.5 years to get to the point to reach this seperation as it required to rewrite the buffers and other infrastructure. This was very responsible and cautionosly guided by Petr and the other members of the printk() maintainer team. Along with that went quite some improvements like realistic timestamping and other details which matter for dmesg power users. The printk threads were added in the 5.19 merge window and unfortunately reverted between 5.19-rc3 and 5.19-rc4 for the very wrong reason: Some embedded boards failed to boot. The root cause is missing locking in the init functions of the related UART drivers. This is not a problem of threaded printk(), Those are existing bugs in these drivers which can be triggered on a 5.18 kernel. They are hard to trigger and nobody cared so far because they were neither bisectable nor reliably reproducible. The threaded printk() change made them reproducible and the bisect pointed at the messenger and not at the root cause. Nevertheless it was decided to throw away valuable work for no real good reason. That revert costs me constantly ~2 seconds of boot time on one my main development machines. Not much, but ~10% and I know that quite some folks in the fastboot camp give a leg for 10%. The right decision would have been to offer this by boot or config parameter and not to throw the baby out with the bathwater. I know it's my fault because I was AFK that week... Coming back to the commit in question which made you (rightfully) upset. I agree that it should never have happened, but OTOH it's a very clear message of developer frustration to you: On one hand you can't get tired of "praising" the RT people about their responsible approach to solve fundamental shortcomigs in the kernel, but at the same time you're pulling whatever new fancy "technology" which comes around the corner and then makes my and your inbox full of security issues. That's obviously not a problem, right? Neither is it a problem that all of these "feature" developers can rightfully ignore PREEMPT_RT and offload all the resulting problems to the RT people, right? I might be wrong, but you seem to have a fundamental misunderstanding of the scope and importance of PREEEMPT_RT: > Oh, I agree that it probably is a pretty small community. Pull the rug under that "pretty small community" and the industrial world is comming to a grinding halt. That grinding halt will affect your power grid, your water supply and quite some other essential things which are listed under Civil Infrastructure. Plus the new fangled edge computing, modern cars .... There is a reason that at least some major industry players have invested into the RT project for the last couple of years. All major distro players have commercially supported RT offerings for years and they are surely not doing this just for fun. There are freeriders as always. Space*, car*, distro* ,,,, > And I also think that people who are really into RT are basically > always going to have extra patches anyway - I think the bulk of the > core stuff has made it upstream, but not *all* has made it. At the point of this writing the only outstanding issue is printk, which prevents PREEMPT_RT from being enabled in your tree on x86[64] and ARM[64]. This still does not justify the commit in question, but clearly points out that there is an attitude problem. Which attitude problem? The attitude problem that PREEMPT_RT is just a niche, but the larger kernel community is happy to consume the benefits of that allegded niche without giving anything back. Think about that. Thanks, Thomas ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [GIT PULL] printk for 5.20 2022-08-09 1:18 ` Thomas Gleixner @ 2022-08-11 13:38 ` Daniel Vetter 0 siblings, 0 replies; 13+ messages in thread From: Daniel Vetter @ 2022-08-11 13:38 UTC (permalink / raw) To: Thomas Gleixner Cc: Linus Torvalds, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, John Ogness, Andy Shevchenko, Rasmus Villemoes, Sebastian Andrzej Siewior, Jan Kara, Peter Zijlstra, linux-kernel, Clark Williams, Mel Gorman, DRI Development On Tue, Aug 09, 2022 at 03:18:20AM +0200, Thomas Gleixner wrote: > On Tue, Aug 02 2022 at 20:19, Linus Torvalds wrote: > > On Mon, Aug 1, 2022 at 8:08 AM Petr Mladek <pmladek@suse.com> wrote: > >> > >> - Completely disable printing on consoles with CONFIG_RT. > > And guys, I want to make it really clear how disappointed I am with > > the printk tree lately. There seems to be some kind of hardline > > religious fervor having taken over to make these kinds of "this is how > > it has to be done, screw any sanity or common sense". > ... > > Put another way: not only am I not pulling this, I'm concerned that I > > will not be pulling printk patches in the future either because of > > where these pull requests seem to be trending. > > I really have to stand up for the printk maintainers here. > > Especially Petr has done an extraordinary job in the past few years. > There have been hickups, but with such a semantically ill defined > mechanism like printk() that's not a surprise at all. > > Let me add some historical background here. > > In Sept. 2019, i.e. almost 3 years ago, we all - including you - sat > together at Plumbers in Lisbon and agreed that printk() in it's back > then form is a nightmare not only for PREEMPT_RT. We also agreed back > then that seperating out the console writes into individual printk > threads makes sense because it also gets rid of the fully serializing > nature of printk() which exists for historical but not for technical > reasons. Also the magic oops_in_progress heuristics have been declared > to be just duct tape, as the still existing (after 25+ years of Linux) > situation with graphics demonstrates on a daily basis. We all agreed > that a dedicated atomic_write() which also allows to prioritize your > favourite pet pieve of making laptops more debugable by utilizing > persistant storage simpler and what's more important more reliable. Cutting the more detail stuff below because I'm justing chiming in on the overall direction. I want this. Where "want" means by default and across the board, and "this" the entire printk/console_lock redesign we discussed in Lisbon with threaded printing and properly separate out emergency output and everything else. Because ffs I'm so fed up having to deal with console_lock in graphcis drivers. This has nothing to do with rt or being niche or whatever but just that Petr, John and others are tackling some really nasty corners that everyone just covered with industrial quantities of duct-tape in the past years (decades?) and finally there's real movement (but a bit slower than I hoped for really). I visted linutronix folks a few weeks ago and we talked a bit what we could all do when this mess is sorted and I want it all. *graphics maintainer out* -Daniel > It took whopping 2.5 years to get to the point to reach this seperation > as it required to rewrite the buffers and other infrastructure. This was > very responsible and cautionosly guided by Petr and the other members of > the printk() maintainer team. Along with that went quite some > improvements like realistic timestamping and other details which matter > for dmesg power users. > > The printk threads were added in the 5.19 merge window and unfortunately > reverted between 5.19-rc3 and 5.19-rc4 for the very wrong reason: > > Some embedded boards failed to boot. > > The root cause is missing locking in the init functions of the > related UART drivers. > > This is not a problem of threaded printk(), Those are existing bugs > in these drivers which can be triggered on a 5.18 kernel. They are > hard to trigger and nobody cared so far because they were neither > bisectable nor reliably reproducible. The threaded printk() change > made them reproducible and the bisect pointed at the messenger and > not at the root cause. > > Nevertheless it was decided to throw away valuable work for no real good > reason. That revert costs me constantly ~2 seconds of boot time on one my > main development machines. Not much, but ~10% and I know that quite some > folks in the fastboot camp give a leg for 10%. > > The right decision would have been to offer this by boot or config > parameter and not to throw the baby out with the bathwater. > > I know it's my fault because I was AFK that week... > > Coming back to the commit in question which made you (rightfully) upset. > > I agree that it should never have happened, but OTOH it's a very clear > message of developer frustration to you: > > On one hand you can't get tired of "praising" the RT people about their > responsible approach to solve fundamental shortcomigs in the kernel, but > at the same time you're pulling whatever new fancy "technology" which > comes around the corner and then makes my and your inbox full of > security issues. That's obviously not a problem, right? > > Neither is it a problem that all of these "feature" developers can > rightfully ignore PREEMPT_RT and offload all the resulting problems to > the RT people, right? > > I might be wrong, but you seem to have a fundamental misunderstanding of > the scope and importance of PREEEMPT_RT: > > > Oh, I agree that it probably is a pretty small community. > > Pull the rug under that "pretty small community" and the industrial > world is comming to a grinding halt. That grinding halt will affect your > power grid, your water supply and quite some other essential things > which are listed under Civil Infrastructure. Plus the new fangled edge > computing, modern cars .... > > There is a reason that at least some major industry players have > invested into the RT project for the last couple of years. All major > distro players have commercially supported RT offerings for years and > they are surely not doing this just for fun. > > There are freeriders as always. Space*, car*, distro* ,,,, > > > And I also think that people who are really into RT are basically > > always going to have extra patches anyway - I think the bulk of the > > core stuff has made it upstream, but not *all* has made it. > > At the point of this writing the only outstanding issue is printk, which > prevents PREEMPT_RT from being enabled in your tree on x86[64] and > ARM[64]. > > This still does not justify the commit in question, but clearly points > out that there is an attitude problem. > > Which attitude problem? > > The attitude problem that PREEMPT_RT is just a niche, but the larger > kernel community is happy to consume the benefits of that allegded > niche without giving anything back. > > Think about that. > > Thanks, > > Thomas -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2022-08-11 13:38 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-08-01 15:07 [GIT PULL] printk for 5.20 Petr Mladek 2022-08-01 15:45 ` David Laight 2022-08-02 7:37 ` Petr Mladek 2022-08-03 3:19 ` Linus Torvalds 2022-08-03 14:35 ` Sebastian Andrzej Siewior 2022-08-03 14:46 ` David Laight 2022-08-03 15:43 ` Petr Mladek 2022-08-03 16:06 ` Steven Rostedt 2022-08-03 16:08 ` Linus Torvalds 2022-08-03 16:34 ` Sebastian Andrzej Siewior 2022-08-03 17:08 ` Ingo Molnar 2022-08-09 1:18 ` Thomas Gleixner 2022-08-11 13:38 ` Daniel Vetter
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).