All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Paul E. McKenney" <paulmck@kernel.org>
To: peterz@infradead.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Vladimir Oltean <olteanv@gmail.com>,
	Kurt Kanzenbach <kurt.kanzenbach@linutronix.de>,
	Alison Wang <alison.wang@nxp.com>,
	catalin.marinas@arm.com, will@kernel.org, mw@semihalf.com,
	leoyang.li@nxp.com, vladimir.oltean@nxp.com,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org,
	Anna-Maria Gleixner <anna-maria@linutronix.de>
Subject: Re: [RFC PATCH] arm64: defconfig: Disable fine-grained task level IRQ time accounting
Date: Thu, 6 Aug 2020 06:27:10 -0700	[thread overview]
Message-ID: <20200806132710.GL4295@paulmck-ThinkPad-P72> (raw)
In-Reply-To: <20200806114545.GA2674@hirez.programming.kicks-ass.net>

On Thu, Aug 06, 2020 at 01:45:45PM +0200, peterz@infradead.org wrote:
> On Thu, Aug 06, 2020 at 11:41:06AM +0200, Thomas Gleixner wrote:
> > peterz@infradead.org writes:
> > > On Wed, Aug 05, 2020 at 02:56:49PM +0100, Valentin Schneider wrote:
> > >
> > >> I've been tempted to say the test case is a bit bogus, but am not familiar
> > >> enough with the RT throttling details to stand that ground. That said, from
> > >> both looking at the execution and the stress-ng source code, it seems to
> > >> unconditionally spawn 32 FIFO-50 tasks (there's even an option to make
> > >> these FIFO-99!!!), which is quite a crowd on monoCPU systems.
> > >
> > > Oh, so it's a case of: we do stupid without tuning and the system falls
> > > over. I can live with that.
> > 
> > It's not a question of whether you can live with that behaviour for a
> > particular silly test case.
> > 
> > The same happens with a single RT runaway task with enough interrupt
> > load on a UP machine. Just validated that. 
> 
> Of course.
> 
> > And that has nothing to do
> > with a silly test case. Sporadic runaways due to a bug in a once per
> > week code path simply can happen and having the safety net working
> > depending on a config option selected or not is just wrong.
> 
> The safety thing is concerned with RT tasks. It doesn't pretend to help
> with runnaway IRQs, never has, never will.

Getting into the time machine back to the 1990s...

DYNIX/ptx had a discretionary mechanism to deal with excessive interrupts.
There was a function that long-running interrupt handlers were supposed
to call periodically that would return false if the system felt that
the CPU had done enough interrupts for the time being.  In that case,
the interrupt handler was supposed to schedule itself for a later time,
but leave the interrupt unacknowledged in order to prevent retriggering
in the meantime.

Of course, this mechanism would be rather less helpful in Linux.

For one, Linux has way more device drivers and way more oddball devices.
In contrast, the few devices that DYNIX/ptx supported were carefully
selected, and the selection criteria included being able to put up
with this sort of thing.  Also, the fact that there was but a handful
of device drivers meant that changes like this could be more easily
propagated through all drivers.

Also, Linux supports way more workloads.  In contrast, DYNIX/ptx could
pick a small percentage of each CPU that would be permitted to be used
by hardware interrupt handlers.  As in there are probably Linux workloads
that run >90% of some poor CPU within hardware interrupt handlers.

But reminiscing anyway on the off-chance that this inspires someone
to come up with an idea that would work well in the Linux environment.

							Thanx, Paul

> The further extreme is an interrupt storm, those have always taken a
> machine down.
> 
> Accounting unrelated IRQ time to RT tasks is equally wrong, the task
> execution is unrelated to the IRQs. The config option at least offers
> insight into where time goes -- and it's a config option because doing
> time accounting on interrupts adds overhead :/
> 
> This really is a no-win all round.
> 
> The only 'sensible' option here is threaded IRQs, where the IRQ line
> gets disabled until the handler thread has ran, that also helps with IRQ
> storms.

WARNING: multiple messages have this Message-ID (diff)
From: "Paul E. McKenney" <paulmck@kernel.org>
To: peterz@infradead.org
Cc: mw@semihalf.com, Anna-Maria Gleixner <anna-maria@linutronix.de>,
	catalin.marinas@arm.com, Alison Wang <alison.wang@nxp.com>,
	linux-kernel@vger.kernel.org, leoyang.li@nxp.com,
	vladimir.oltean@nxp.com,
	Kurt Kanzenbach <kurt.kanzenbach@linutronix.de>,
	Thomas Gleixner <tglx@linutronix.de>,
	Vladimir Oltean <olteanv@gmail.com>,
	will@kernel.org, Valentin Schneider <valentin.schneider@arm.com>,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [RFC PATCH] arm64: defconfig: Disable fine-grained task level IRQ time accounting
Date: Thu, 6 Aug 2020 06:27:10 -0700	[thread overview]
Message-ID: <20200806132710.GL4295@paulmck-ThinkPad-P72> (raw)
In-Reply-To: <20200806114545.GA2674@hirez.programming.kicks-ass.net>

On Thu, Aug 06, 2020 at 01:45:45PM +0200, peterz@infradead.org wrote:
> On Thu, Aug 06, 2020 at 11:41:06AM +0200, Thomas Gleixner wrote:
> > peterz@infradead.org writes:
> > > On Wed, Aug 05, 2020 at 02:56:49PM +0100, Valentin Schneider wrote:
> > >
> > >> I've been tempted to say the test case is a bit bogus, but am not familiar
> > >> enough with the RT throttling details to stand that ground. That said, from
> > >> both looking at the execution and the stress-ng source code, it seems to
> > >> unconditionally spawn 32 FIFO-50 tasks (there's even an option to make
> > >> these FIFO-99!!!), which is quite a crowd on monoCPU systems.
> > >
> > > Oh, so it's a case of: we do stupid without tuning and the system falls
> > > over. I can live with that.
> > 
> > It's not a question of whether you can live with that behaviour for a
> > particular silly test case.
> > 
> > The same happens with a single RT runaway task with enough interrupt
> > load on a UP machine. Just validated that. 
> 
> Of course.
> 
> > And that has nothing to do
> > with a silly test case. Sporadic runaways due to a bug in a once per
> > week code path simply can happen and having the safety net working
> > depending on a config option selected or not is just wrong.
> 
> The safety thing is concerned with RT tasks. It doesn't pretend to help
> with runnaway IRQs, never has, never will.

Getting into the time machine back to the 1990s...

DYNIX/ptx had a discretionary mechanism to deal with excessive interrupts.
There was a function that long-running interrupt handlers were supposed
to call periodically that would return false if the system felt that
the CPU had done enough interrupts for the time being.  In that case,
the interrupt handler was supposed to schedule itself for a later time,
but leave the interrupt unacknowledged in order to prevent retriggering
in the meantime.

Of course, this mechanism would be rather less helpful in Linux.

For one, Linux has way more device drivers and way more oddball devices.
In contrast, the few devices that DYNIX/ptx supported were carefully
selected, and the selection criteria included being able to put up
with this sort of thing.  Also, the fact that there was but a handful
of device drivers meant that changes like this could be more easily
propagated through all drivers.

Also, Linux supports way more workloads.  In contrast, DYNIX/ptx could
pick a small percentage of each CPU that would be permitted to be used
by hardware interrupt handlers.  As in there are probably Linux workloads
that run >90% of some poor CPU within hardware interrupt handlers.

But reminiscing anyway on the off-chance that this inspires someone
to come up with an idea that would work well in the Linux environment.

							Thanx, Paul

> The further extreme is an interrupt storm, those have always taken a
> machine down.
> 
> Accounting unrelated IRQ time to RT tasks is equally wrong, the task
> execution is unrelated to the IRQs. The config option at least offers
> insight into where time goes -- and it's a config option because doing
> time accounting on interrupts adds overhead :/
> 
> This really is a no-win all round.
> 
> The only 'sensible' option here is threaded IRQs, where the IRQ line
> gets disabled until the handler thread has ran, that also helps with IRQ
> storms.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  reply	other threads:[~2020-08-06 17:32 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-29  3:39 [RFC PATCH] arm64: defconfig: Disable fine-grained task level IRQ time accounting Alison Wang
2020-07-29  3:39 ` Alison Wang
2020-07-29  8:40 ` Kurt Kanzenbach
2020-07-29  8:40   ` Kurt Kanzenbach
2020-07-29  8:50   ` [EXT] " Alison Wang
2020-07-29  8:50     ` Alison Wang
2020-07-29  9:49   ` Vladimir Oltean
2020-07-29  9:49     ` Vladimir Oltean
2020-07-30  7:23     ` Kurt Kanzenbach
2020-07-30  7:23       ` Kurt Kanzenbach
2020-07-30  8:22       ` Vladimir Oltean
2020-07-30  8:22         ` Vladimir Oltean
2020-08-03  8:04         ` Kurt Kanzenbach
2020-08-03  8:04           ` Kurt Kanzenbach
2020-08-03  8:16           ` Vladimir Oltean
2020-08-03  8:16             ` Vladimir Oltean
2020-08-03  9:51             ` Robin Murphy
2020-08-03  9:51               ` Robin Murphy
2020-08-03 11:38               ` Vladimir Oltean
2020-08-03 11:38                 ` Vladimir Oltean
2020-08-03 11:48                 ` Valentin Schneider
2020-08-03 11:48                   ` Valentin Schneider
2020-08-03 13:24                   ` Marc Zyngier
2020-08-03 13:24                     ` Marc Zyngier
2020-08-03 10:02             ` Thomas Gleixner
2020-08-03 10:02               ` Thomas Gleixner
2020-08-03 10:49           ` Thomas Gleixner
2020-08-03 10:49             ` Thomas Gleixner
2020-08-03 11:41             ` Vladimir Oltean
2020-08-03 11:41               ` Vladimir Oltean
2020-08-03 15:13               ` Thomas Gleixner
2020-08-03 15:13                 ` Thomas Gleixner
2020-08-03 15:47                 ` Valentin Schneider
2020-08-03 15:47                   ` Valentin Schneider
2020-08-03 16:14                   ` Vladimir Oltean
2020-08-03 16:14                     ` Vladimir Oltean
2020-08-03 19:22                   ` Thomas Gleixner
2020-08-03 19:22                     ` Thomas Gleixner
2020-08-03 23:59                     ` Valentin Schneider
2020-08-03 23:59                       ` Valentin Schneider
2020-08-05  8:50                       ` Dietmar Eggemann
2020-08-05  8:50                         ` Dietmar Eggemann
2020-08-05 13:40                     ` peterz
2020-08-05 13:40                       ` peterz
2020-08-05 13:56                       ` Valentin Schneider
2020-08-05 13:56                         ` Valentin Schneider
2020-08-05 15:31                         ` peterz
2020-08-05 15:31                           ` peterz
2020-08-06  9:41                           ` Thomas Gleixner
2020-08-06  9:41                             ` Thomas Gleixner
2020-08-06 11:45                             ` peterz
2020-08-06 11:45                               ` peterz
2020-08-06 13:27                               ` Paul E. McKenney [this message]
2020-08-06 13:27                                 ` Paul E. McKenney
2020-08-06 19:03                                 ` Thomas Gleixner
2020-08-06 19:03                                   ` Thomas Gleixner
2020-08-06 20:39                                   ` Paul E. McKenney
2020-08-06 20:39                                     ` Paul E. McKenney
2020-08-06 18:58                               ` Thomas Gleixner
2020-08-06 18:58                                 ` Thomas Gleixner
2020-08-06  9:34                       ` Thomas Gleixner
2020-08-06  9:34                         ` Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200806132710.GL4295@paulmck-ThinkPad-P72 \
    --to=paulmck@kernel.org \
    --cc=alison.wang@nxp.com \
    --cc=anna-maria@linutronix.de \
    --cc=catalin.marinas@arm.com \
    --cc=kurt.kanzenbach@linutronix.de \
    --cc=leoyang.li@nxp.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mw@semihalf.com \
    --cc=olteanv@gmail.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --cc=valentin.schneider@arm.com \
    --cc=vladimir.oltean@nxp.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.