From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> To: Peter Zijlstra <peterz@infradead.org> Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>, linux-kernel@vger.kernel.org, Petr Mladek <pmladek@suse.com>, Steven Rostedt <rostedt@goodmis.org>, Daniel Wang <wonderfly@google.com>, Andrew Morton <akpm@linux-foundation.org>, Linus Torvalds <torvalds@linux-foundation.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Alan Cox <gnomes@lxorguk.ukuu.org.uk>, Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>, linux-serial@vger.kernel.org, Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Subject: Re: [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Date: Wed, 17 Oct 2018 13:32:51 +0900 [thread overview] Message-ID: <20181017043251.GC1068@jagdpanzerIV> (raw) In-Reply-To: <20181016125415.GA3121@hirez.programming.kicks-ass.net> On (10/16/18 14:54), Peter Zijlstra wrote: > On Tue, Oct 16, 2018 at 09:27:34PM +0900, Sergey Senozhatsky wrote: > > per-CPU printk_safe _semi-magic_ makes some things simple to handle. > > We can't just remove per-CPU buffers and add a wake_up_process() at > > the bottom of vprintk_emit(). Because this will deadlock: > > > > printk() > > wake_up_process() > > try_to_wake_up() > > raw_spin_lock_irqsave() > > <NMI> > > printk() > > wake_up_process() > > try_to_wake_up() > > raw_spin_lock_irqsave() > > > > So we still need some amount of per-CPU printk() semi-magic anyway. > > All we need is 4 max-line-length buffers per-CPU. Nothing more. OK, similar to what Steven did with cpu_buffer->current_context. > The above trainwreck is the direct result of forcing synchronous > printk'ing (which I'm otherwise a big fan of, but regular console > drivers stink and are unfixable). Yep. > > And printk-kthread offloding will not eliminate the need of > > printk_deferred(). > > Why not? printk() will reduce to a lockless buffer insert. IOW _all_ > printk is deferred. Aha! Interesting. I didn't realize you were talking about "all printk()-s are deferred". OK, jump to the last part of this mail. > All you need are 4 max-line-length buffers per CPU and a global/shared > lockless buffer. > > printk will determine the current context: > > task, softirq, hardirq or NMI > > and pick the corresponding per-cpu line buffer and do the vsnprintf() > thing. Then we have the actual line length and content. With the length > we reserve the bytes from the global buffer, we memcpy into the buffer > and commit. > > Done. > > The printk thread will observe it lags behind the buffer head and will > start printk-ing crud from task context. [you can skip this part] This probably will be a bit more hairy. logbuf is written to by many sources and is read from by many sides, including user-space [both read() and write()]. So we will need more flags/magic around memcpy(). A simple, "grab the logbuf entry, set the proper offset to point to the next available logbuf record and then do memcpy()" won't suffice. We need a flag for "memcpy() complete, we can read this entry". Otherwise: CPU0 CPU1 CPU2 CPU3 printk printk printk_kthread logbuf_entry A logbuf_entry B syslog(read all) call_console_drivers memcpy memcpy read unfinished print unfinished A and B A and B [..] > > We do, however, have loads of problems with all those dependencies which > > come from serial drivers and friends: timekeeping, scheduler (scheduler > > is brilliant and cool, but we do have some deadlocks in printk because of > > it ;), tty, net, MM, wq, etc. So I generally like the idea of "detached > > serial consoles" (that's what I call it). IOW, elimination of the direct > > printk -> serial console path. > > Right; we need to get rid of that in the generic case. Only allow > lockless consoles (earlycon) to be used synchonously. With maybe a > special case for the BUG/PANIC case to push things out ASAP. [..] > > So, unless I'm missing something, things are not entirely that simple: > > - throw away printk_safe semi-magic > > - add a lockless logbuf > > - add wake_up_process() to vprintk_emit(). > > No, no wakups. irq_work to wake the printk-thread, at most. All right. OK. So we are on the same page here: printk has internal locks - logbuf spin_lock; and external locks - all the scheduler locks, console_sem, net, tty, wq, you name it. printk() is not aware of those external locks; the only way to fix it is to remove them from printk(). And that's why "turn printk() into printk_deferred() and fix printk() deadlocks in general case" was my final proposal at the 2016 KS, NM, USA [1] (grep for printk_deferred). I mentioned this idea several times since then, and even sent a patch, doing this "printk is now printk_deferred unless we are in panic" thing. As far as I remember, back then the idea/patch were rejected [2], and one of reviewers even hinted that I was crazy :-) I have absolutely no issues with that, but, considering past experiences, I'd really like to: - Have more opinions on this. People please speak out. - Have clear "let's do it" from Cc-ed people. If we are really doing this, then let's split it and have incremental changes. Namely, what I suggest is: - keep internal printk lock - logbuf lock for now; we know how to handle it. I promise. - keep printk_safe for now, we need it to deal with logbuf lock - keep printk_safe completely internal to printk - add printk_kthread - do printk()->irq_work()->wake_up_process(printk_kthread) change and remove external locks dependency - use direct printk() for panic() case - do something about early_printk That's big enough already. From there, once we land this thing, we can start building new logbuf, stealing code from Steven, improving per-CPU buffers and so on. Are we doing this? [1] https://lwn.net/Articles/705938/ [2] https://lore.kernel.org/lkml/20170202090722.GW6515@twins.programming.kicks-ass.net/ -ss
WARNING: multiple messages have this Message-ID (diff)
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> To: Peter Zijlstra <peterz@infradead.org> Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>, linux-kernel@vger.kernel.org, Petr Mladek <pmladek@suse.com>, Steven Rostedt <rostedt@goodmis.org>, Daniel Wang <wonderfly@google.com>, Andrew Morton <akpm@linux-foundation.org>, Linus Torvalds <torvalds@linux-foundation.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Alan Cox <gnomes@lxorguk.ukuu.org.uk>, Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>, linux-serial@vger.kernel.org, Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Subject: Re: [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Date: Wed, 17 Oct 2018 13:32:51 +0900 [thread overview] Message-ID: <20181017043251.GC1068@jagdpanzerIV> (raw) In-Reply-To: <20181016125415.GA3121@hirez.programming.kicks-ass.net> On (10/16/18 14:54), Peter Zijlstra wrote: > On Tue, Oct 16, 2018 at 09:27:34PM +0900, Sergey Senozhatsky wrote: > > per-CPU printk_safe _semi-magic_ makes some things simple to handle. > > We can't just remove per-CPU buffers and add a wake_up_process() at > > the bottom of vprintk_emit(). Because this will deadlock: > > > > printk() > > wake_up_process() > > try_to_wake_up() > > raw_spin_lock_irqsave() > > <NMI> > > printk() > > wake_up_process() > > try_to_wake_up() > > raw_spin_lock_irqsave() > > > > So we still need some amount of per-CPU printk() semi-magic anyway. > > All we need is 4 max-line-length buffers per-CPU. Nothing more. OK, similar to what Steven did with cpu_buffer->current_context. > The above trainwreck is the direct result of forcing synchronous > printk'ing (which I'm otherwise a big fan of, but regular console > drivers stink and are unfixable). Yep. > > And printk-kthread offloding will not eliminate the need of > > printk_deferred(). > > Why not? printk() will reduce to a lockless buffer insert. IOW _all_ > printk is deferred. Aha! Interesting. I didn't realize you were talking about "all printk()-s are deferred". OK, jump to the last part of this mail. > All you need are 4 max-line-length buffers per CPU and a global/shared > lockless buffer. > > printk will determine the current context: > > task, softirq, hardirq or NMI > > and pick the corresponding per-cpu line buffer and do the vsnprintf() > thing. Then we have the actual line length and content. With the length > we reserve the bytes from the global buffer, we memcpy into the buffer > and commit. > > Done. > > The printk thread will observe it lags behind the buffer head and will > start printk-ing crud from task context. [you can skip this part] This probably will be a bit more hairy. logbuf is written to by many sources and is read from by many sides, including user-space [both read() and write()]. So we will need more flags/magic around memcpy(). A simple, "grab the logbuf entry, set the proper offset to point to the next available logbuf record and then do memcpy()" won't suffice. We need a flag for "memcpy() complete, we can read this entry". Otherwise: CPU0 CPU1 CPU2 CPU3 printk printk printk_kthread logbuf_entry A logbuf_entry B syslog(read all) call_console_drivers memcpy memcpy read unfinished print unfinished A and B A and B [..] > > We do, however, have loads of problems with all those dependencies which > > come from serial drivers and friends: timekeeping, scheduler (scheduler > > is brilliant and cool, but we do have some deadlocks in printk because of > > it ;), tty, net, MM, wq, etc. So I generally like the idea of "detached > > serial consoles" (that's what I call it). IOW, elimination of the direct > > printk -> serial console path. > > Right; we need to get rid of that in the generic case. Only allow > lockless consoles (earlycon) to be used synchonously. With maybe a > special case for the BUG/PANIC case to push things out ASAP. [..] > > So, unless I'm missing something, things are not entirely that simple: > > - throw away printk_safe semi-magic > > - add a lockless logbuf > > - add wake_up_process() to vprintk_emit(). > > No, no wakups. irq_work to wake the printk-thread, at most. All right. OK. So we are on the same page here: printk has internal locks - logbuf spin_lock; and external locks - all the scheduler locks, console_sem, net, tty, wq, you name it. printk() is not aware of those external locks; the only way to fix it is to remove them from printk(). And that's why "turn printk() into printk_deferred() and fix printk() deadlocks in general case" was my final proposal at the 2016 KS, NM, USA [1] (grep for printk_deferred). I mentioned this idea several times since then, and even sent a patch, doing this "printk is now printk_deferred unless we are in panic" thing. As far as I remember, back then the idea/patch were rejected [2], and one of reviewers even hinted that I was crazy :-) I have absolutely no issues with that, but, considering past experiences, I'd really like to: - Have more opinions on this. People please speak out. - Have clear "let's do it" from Cc-ed people. If we are really doing this, then let's split it and have incremental changes. Namely, what I suggest is: - keep internal printk lock - logbuf lock for now; we know how to handle it. I promise. - keep printk_safe for now, we need it to deal with logbuf lock - keep printk_safe completely internal to printk - add printk_kthread - do printk()->irq_work()->wake_up_process(printk_kthread) change and remove external locks dependency - use direct printk() for panic() case - do something about early_printk That's big enough already. >From there, once we land this thing, we can start building new logbuf, stealing code from Steven, improving per-CPU buffers and so on. Are we doing this? [1] https://lwn.net/Articles/705938/ [2] https://lore.kernel.org/lkml/20170202090722.GW6515@twins.programming.kicks-ass.net/ -ss
next prev parent reply other threads:[~2018-10-17 4:32 UTC|newest] Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-10-16 5:04 [RFC][PATCHv2 0/4] less deadlock prone serial consoles Sergey Senozhatsky 2018-10-16 5:04 ` [RFC][PATCHv2 1/4] panic: avoid deadlocks in re-entrant console drivers Sergey Senozhatsky 2018-10-16 5:04 ` Sergey Senozhatsky 2018-10-17 4:48 ` Sergey Senozhatsky 2018-10-23 11:07 ` Petr Mladek 2018-10-23 11:54 ` Sergey Senozhatsky 2018-10-23 12:04 ` Sergey Senozhatsky 2018-10-23 12:12 ` Sergey Senozhatsky 2018-10-25 9:06 ` Petr Mladek 2018-10-25 9:31 ` Sergey Senozhatsky 2018-10-25 8:29 ` Petr Mladek 2018-10-25 9:05 ` Sergey Senozhatsky 2018-10-25 10:10 ` [PATCHv3] " Sergey Senozhatsky 2018-10-25 10:10 ` Sergey Senozhatsky 2018-10-25 10:51 ` kbuild test robot 2018-10-25 10:51 ` kbuild test robot 2018-10-25 11:56 ` Sergey Senozhatsky 2018-10-25 11:56 ` Sergey Senozhatsky 2018-10-31 12:27 ` Petr Mladek 2018-11-01 1:48 ` Sergey Senozhatsky 2018-11-01 8:08 ` Petr Mladek 2018-11-22 13:12 ` Petr Mladek 2018-12-12 0:53 ` Daniel Wang 2018-12-12 5:23 ` Sergey Senozhatsky 2018-12-12 5:59 ` Daniel Wang 2018-12-12 6:06 ` Sergey Senozhatsky 2018-12-12 6:09 ` Daniel Wang 2018-10-16 5:04 ` [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Sergey Senozhatsky 2018-10-16 7:27 ` Peter Zijlstra 2018-10-16 11:40 ` Petr Mladek 2018-10-16 12:17 ` Peter Zijlstra 2018-10-17 10:50 ` Petr Mladek 2018-10-17 14:00 ` Peter Zijlstra 2018-10-22 14:30 ` Petr Mladek 2018-10-16 12:27 ` Sergey Senozhatsky 2018-10-16 12:38 ` Peter Zijlstra 2018-10-16 12:54 ` Peter Zijlstra 2018-10-16 14:21 ` Peter Zijlstra 2018-10-17 4:32 ` Sergey Senozhatsky [this message] 2018-10-17 4:32 ` Sergey Senozhatsky 2018-10-17 7:57 ` Peter Zijlstra 2018-10-17 13:36 ` Sergey Senozhatsky 2018-10-23 6:25 ` Sergey Senozhatsky 2018-10-16 5:04 ` [RFC][PATCHv2 3/4] serial: introduce uart_port locking helpers Sergey Senozhatsky 2018-12-08 3:12 ` Sergey Senozhatsky 2018-12-12 11:08 ` Greg Kroah-Hartman 2018-10-16 5:04 ` [RFC][PATCHv2 4/4] tty: 8250: switch to " Sergey Senozhatsky 2018-10-16 7:23 ` [RFC][PATCHv2 0/4] less deadlock prone serial consoles Peter Zijlstra 2018-10-16 8:12 ` Sergey Senozhatsky
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181017043251.GC1068@jagdpanzerIV \ --to=sergey.senozhatsky.work@gmail.com \ --cc=akpm@linux-foundation.org \ --cc=gnomes@lxorguk.ukuu.org.uk \ --cc=gregkh@linuxfoundation.org \ --cc=jslaby@suse.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-serial@vger.kernel.org \ --cc=peterz@infradead.org \ --cc=pfeiner@google.com \ --cc=pmladek@suse.com \ --cc=rostedt@goodmis.org \ --cc=sergey.senozhatsky@gmail.com \ --cc=torvalds@linux-foundation.org \ --cc=wonderfly@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.