Linux-rt-users archive on lore.kernel.org
 help / color / Atom feed
* [5.4-rt] kdb: push 'bt' command output to console immediately.
@ 2020-05-20 15:16 Joe Korty
  2020-05-20 15:33 ` John Ogness
  0 siblings, 1 reply; 6+ messages in thread
From: Joe Korty @ 2020-05-20 15:16 UTC (permalink / raw)
  To: John Ogness; +Cc: Sebastian Andrzej Siewior, linux-rt-users

[5.4-rt] kdb: push 'bt' command output to console immediately.

The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
'bt' command now prints a single line then returns to the
kdb prompt.  There is no stack trace being shown.

The missing stack trace is in fact being generated, but
is instead being spooled into an internal buffer.

On kdb return-to-kernel (ie, the 'go' command), the spooled
output is displayed.

This behavour occurs because kdb calls a non-kdb function,
stack_dump(), to decode and dump the stack.  That function
uses printk's.  In an NMI handler (of which kgdb/kdb is an
example), printk spools all output generated, and prints
that spooled output once the handler exits.

To fix this, mainline has long had a tap in printk that
forces it to divert to kdb_printf whenever kdb invokes
some out-of-kdb function.  That tap, in 5.4-rt, is for some
reason no longer functional.  This might have something to
do with the printk subsystem partial rewrite now present
in the rt patch.

Here is my test sequence.  This example assumes one is
using a serial console not a KVM.

   echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
   echo g > /proc/sysrq-trigger
   bt
   go

Here are the kernels I ran the above test on:

   4.19.120-rt52 -- Pass
   5.0.21-rt16   -- Pass (*)
   5.2.21-rt15   -- Fail
   5.4.40-rt24   -- Fail

* - Passed on 2nd boot.  First boot locked the system up
    on the 'echo g'.  No oops.

I have attached a small patch that Seems To Work.  It
taps earlier into printk than the official tap does.

Signed-off-by: Joe Korty <joe.korty@concurrent-rt.com>

Index: b/kernel/printk/printk.c
===================================================================
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2054,7 +2054,10 @@ asmlinkage __visible int printk(const ch
 	int r;
 
 	va_start(args, fmt);
-	r = vprintk_func(fmt, args);
+	if (kdb_trap_printk)
+		r = vkdb_printf(KDB_MSGSRC_INTERNAL, fmt, args);
+	else
+		r = vprintk_func(fmt, args);
 	va_end(args);
 
 	return r;

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [5.4-rt] kdb: push 'bt' command output to console immediately.
  2020-05-20 15:16 [5.4-rt] kdb: push 'bt' command output to console immediately Joe Korty
@ 2020-05-20 15:33 ` John Ogness
  2020-05-20 15:40   ` Joe Korty
  2020-05-26 16:44   ` Sebastian Andrzej Siewior
  0 siblings, 2 replies; 6+ messages in thread
From: John Ogness @ 2020-05-20 15:33 UTC (permalink / raw)
  To: Joe Korty; +Cc: Sebastian Andrzej Siewior, linux-rt-users

On 2020-05-20, Joe Korty <joe.korty@concurrent-rt.com> wrote:
> [5.4-rt] kdb: push 'bt' command output to console immediately.
>
> The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
> 'bt' command now prints a single line then returns to the
> kdb prompt.  There is no stack trace being shown.
...
>
> I have attached a small patch that Seems To Work.  It
> taps earlier into printk than the official tap does.

On LKML a similar patch was recently posted[0]. It would probably be
better to follow that (patching vprintk_func and using
KDB_MSGSRC_PRINTK).

John Ogness

[0] https://lkml.kernel.org/r/20200520102233.GC3464@linux-b0ei

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [5.4-rt] kdb: push 'bt' command output to console immediately.
  2020-05-20 15:33 ` John Ogness
@ 2020-05-20 15:40   ` Joe Korty
  2020-05-26 16:44   ` Sebastian Andrzej Siewior
  1 sibling, 0 replies; 6+ messages in thread
From: Joe Korty @ 2020-05-20 15:40 UTC (permalink / raw)
  To: John Ogness; +Cc: Sebastian Andrzej Siewior, linux-rt-users

On Wed, May 20, 2020 at 05:33:25PM +0200, John Ogness wrote:
> On 2020-05-20, Joe Korty <joe.korty@concurrent-rt.com> wrote:
> > [5.4-rt] kdb: push 'bt' command output to console immediately.
> >
> > The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
> > 'bt' command now prints a single line then returns to the
> > kdb prompt.  There is no stack trace being shown.
> ...
> >
> > I have attached a small patch that Seems To Work.  It
> > taps earlier into printk than the official tap does.
> 
> On LKML a similar patch was recently posted[0]. It would probably be
> better to follow that (patching vprintk_func and using
> KDB_MSGSRC_PRINTK).
> 
> John Ogness
> 
> [0] https://lkml.kernel.org/r/20200520102233.GC3464@linux-b0ei


Thanks.  It is weird that both this and my bug report were posted
on the same day !

Joe

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [5.4-rt] kdb: push 'bt' command output to console immediately.
  2020-05-20 15:33 ` John Ogness
  2020-05-20 15:40   ` Joe Korty
@ 2020-05-26 16:44   ` Sebastian Andrzej Siewior
  2020-05-26 16:55     ` Joe Korty
  1 sibling, 1 reply; 6+ messages in thread
From: Sebastian Andrzej Siewior @ 2020-05-26 16:44 UTC (permalink / raw)
  To: John Ogness; +Cc: Joe Korty, linux-rt-users

On 2020-05-20 17:33:25 [+0200], John Ogness wrote:
> On 2020-05-20, Joe Korty <joe.korty@concurrent-rt.com> wrote:
> > [5.4-rt] kdb: push 'bt' command output to console immediately.
> >
> > The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
> > 'bt' command now prints a single line then returns to the
> > kdb prompt.  There is no stack trace being shown.
> ...
> >
> > I have attached a small patch that Seems To Work.  It
> > taps earlier into printk than the official tap does.
> 
> On LKML a similar patch was recently posted[0]. It would probably be
> better to follow that (patching vprintk_func and using
> KDB_MSGSRC_PRINTK).

Should I do here anything?

> John Ogness

Sebastian

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [5.4-rt] kdb: push 'bt' command output to console immediately.
  2020-05-26 16:44   ` Sebastian Andrzej Siewior
@ 2020-05-26 16:55     ` Joe Korty
  2020-06-26 13:05       ` Joe Korty
  0 siblings, 1 reply; 6+ messages in thread
From: Joe Korty @ 2020-05-26 16:55 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior; +Cc: John Ogness, linux-rt-users

On Tue, May 26, 2020 at 06:44:49PM +0200, Sebastian Andrzej Siewior wrote:
> On 2020-05-20 17:33:25 [+0200], John Ogness wrote:
> > On 2020-05-20, Joe Korty <joe.korty@concurrent-rt.com> wrote:
> > > [5.4-rt] kdb: push 'bt' command output to console immediately.
> > >
> > > The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
> > > 'bt' command now prints a single line then returns to the
> > > kdb prompt.  There is no stack trace being shown.
> > ...
> > >
> > > I have attached a small patch that Seems To Work.  It
> > > taps earlier into printk than the official tap does.
> > 
> > On LKML a similar patch was recently posted[0]. It would probably be
> > better to follow that (patching vprintk_func and using
> > KDB_MSGSRC_PRINTK).
> 
> Should I do here anything?

Hi John,
Probably not.

Since the bug is in mainline, not rt, ideally rt should
just wait for the fix you so graciously found for me to
enter mainline and propagate down to the various stable
trees.

That should be OK since the bug is more of an annoyance
rather than something critical, so I feel there is no rush
to get an official fix in.

For myself, I have applied the fix to my tree, and I will
delete it when and whenever it shows up in the mainline
branches I care about.

Joe

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [5.4-rt] kdb: push 'bt' command output to console immediately.
  2020-05-26 16:55     ` Joe Korty
@ 2020-06-26 13:05       ` Joe Korty
  0 siblings, 0 replies; 6+ messages in thread
From: Joe Korty @ 2020-06-26 13:05 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior; +Cc: John Ogness, linux-rt-users

On Tue, May 26, 2020 at 12:55:02PM -0400, Joe Korty wrote:
> On Tue, May 26, 2020 at 06:44:49PM +0200, Sebastian Andrzej Siewior wrote:
> > On 2020-05-20 17:33:25 [+0200], John Ogness wrote:
> > > On 2020-05-20, Joe Korty <joe.korty@concurrent-rt.com> wrote:
> > > > [5.4-rt] kdb: push 'bt' command output to console immediately.
> > > >
> > > > The rt patch for 5.4 and 5.2 broke kdb slightly.  The kdb
> > > > 'bt' command now prints a single line then returns to the
> > > > kdb prompt.  There is no stack trace being shown.
> > > ...
> > > >
> > > > I have attached a small patch that Seems To Work.  It
> > > > taps earlier into printk than the official tap does.
> > > 
> > > On LKML a similar patch was recently posted[0]. It would probably be
> > > better to follow that (patching vprintk_func and using
> > > KDB_MSGSRC_PRINTK).
> > 
> > Should I do here anything?
> 
> Hi John,
> Probably not.
> 
> Since the bug is in mainline, not rt, ideally rt should
> just wait for the fix you so graciously found for me to
> enter mainline and propagate down to the various stable
> trees.
...



Hi Sebastian,
Oops, my mistake .. the bug is in rt, not mainline.  The
status of the long-term rt's w.r.t. this patch is:

       5.6-rt    -- already has fix
       5.4-rt    -- needs fix
       4.19-rt   -- needs fix
       4.14-rt   -- needs fix
       4.9-rt    -- needs fix
       4.4-rt    -- needs fix

For your convenience, I've attached the needed patch.

Regards,
Joe

> From: Matt Fleming <matt@codeblueprint.co.uk>
> To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: linux-rt@vger.kernel.org, linux-kernel@vger.kernel.org,
>	Daniel Wagner <wagi@monom.org>,
>	Matt Fleming <matt@codeblueprint.co.uk>
>Subject: [PATCH RT] signal: Prevent double-free of user struct
>Date: Tue,  7 Apr 2020 10:54:13 +0100

The way user struct reference counting works changed significantly with,

  fda31c50292a ("signal: avoid double atomic counter increments for user accounting")

Now user structs are only freed once the last pending signal is
dequeued. Make sigqueue_free_current() follow this new convention to
avoid freeing the user struct multiple times and triggering this
warning:

 refcount_t: underflow; use-after-free.
 WARNING: CPU: 0 PID: 6794 at lib/refcount.c:288 refcount_dec_not_one+0x45/0x50
 Call Trace:
  refcount_dec_and_lock_irqsave+0x16/0x60
  free_uid+0x31/0xa0
  ? schedule_hrtimeout_range_clock+0x104/0x110
  __dequeue_signal+0x17c/0x190
  dequeue_signal+0x5a/0x1b0
  do_sigtimedwait+0x208/0x250
  __x64_sys_rt_sigtimedwait+0x6f/0xd0
  do_syscall_64+0x72/0x200
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
Reported-by: Daniel Wagner <wagi@monom.org>

Index: b/kernel/signal.c
===================================================================
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -494,8 +494,8 @@ static void sigqueue_free_current(struct
 
 	up = q->user;
 	if (rt_prio(current->normal_prio) && !put_task_cache(current, q)) {
-		atomic_dec(&up->sigpending);
-		free_uid(up);
+		if (atomic_dec_and_test(&up->sigpending))
+			free_uid(up);
 	} else
 		  __sigqueue_free(q);
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-20 15:16 [5.4-rt] kdb: push 'bt' command output to console immediately Joe Korty
2020-05-20 15:33 ` John Ogness
2020-05-20 15:40   ` Joe Korty
2020-05-26 16:44   ` Sebastian Andrzej Siewior
2020-05-26 16:55     ` Joe Korty
2020-06-26 13:05       ` Joe Korty

Linux-rt-users archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rt-users/0 linux-rt-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rt-users linux-rt-users/ https://lore.kernel.org/linux-rt-users \
		linux-rt-users@vger.kernel.org
	public-inbox-index linux-rt-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rt-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git