linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Thomas Gleixner <tglx@linutronix.de>
To: Brian Starkey <brian.starkey@arm.com>
Cc: Eric Dumazet <edumazet@google.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Alexander Potapenko <glider@google.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: Re: Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let ksoftirqd do its job"
Date: Fri, 18 Nov 2016 01:40:43 +0100 (CET)	[thread overview]
Message-ID: <alpine.DEB.2.20.1611180125400.3640@nanos> (raw)
In-Reply-To: <20161117164200.GA24653@e106950-lin.cambridge.arm.com>

Brian,

On Thu, 17 Nov 2016, Brian Starkey wrote:
> No joy with this patch :-(
> 
> I had to add an ioaddr argument because apparently that macro depends
> on local context (yuck...), but it doesn't help my issue.
> 
> FWIW I don't see any timeouts, either with or without the patch.
> (I don't know for sure, but I would guess that the model of the
> network card doesn't model whatever stall that loop is checking for.
> It probably just completes all MMU operations immediately)

Is there a chance that you enable trace points at the kernel command line?

  trace_event=sched_wakeup,sched_switch,irq_handler_entry,irq_handler_exit,softirq_raise,softirq_entry,softirq_exit

should be enough for a start. All we need aside of that is a trigger to
stop the trace so we can actually see the events around the time where
things go stale.

I assume that the whole issue is visible throughout the slow progress of
init towards a working system, so for a start it would be sufficient to add
something like this into the startup sequence at some point:

 mount -t debugfs debugfs /sys/kernel/debug
 echo 0 >/sys/kernel/debug/tracing/tracing_on

The only interesting challange is to get the trace data out of the
system. The trace is accessible via:

 cat /sys/kernel/tracing/trace 

So if your ssh works at some point, that might be an option or you just try
to store it over NFS (which will be slow, but better than nothing). Maybe
you have a better idea :)

Thanks,

	tglx

  reply	other threads:[~2016-11-18  0:43 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-11-16 13:55 Regression: Failed boots bisected to 4cd13c21b207 "softirq: Let ksoftirqd do its job" Brian Starkey
2016-11-16 15:52 ` Eric Dumazet
2016-11-16 18:01   ` Brian Starkey
2016-11-16 18:49     ` Eric Dumazet
2016-11-16 21:01       ` Brian Starkey
2016-11-17 15:29         ` Eric Dumazet
2016-11-17 16:42           ` Brian Starkey
2016-11-18  0:40             ` Thomas Gleixner [this message]
2016-11-18 20:23               ` Brian Starkey
2016-11-22 10:33               ` Brian Starkey
2016-11-22 14:29                 ` Eric Dumazet
2016-11-22 15:27                   ` Brian Starkey
2016-11-22 16:09                     ` Eric Dumazet
2016-11-23 18:21                       ` Brian Starkey
2016-11-23 20:03                         ` Eric Dumazet
2016-11-25 13:14                           ` Brian Starkey
2017-02-06 18:46                             ` Will Deacon
2017-02-06 18:49                               ` Russell King - ARM Linux
2017-02-08  9:46                                 ` Will Deacon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.20.1611180125400.3640@nanos \
    --to=tglx@linutronix.de \
    --cc=akpm@linux-foundation.org \
    --cc=bigeasy@linutronix.de \
    --cc=brian.starkey@arm.com \
    --cc=edumazet@google.com \
    --cc=glider@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).