From: John Garry <john.garry@huawei.com>
To: Thomas Gleixner <tglx@linutronix.de>, Marc Zyngier <maz@kernel.org>
Cc: <linux-kernel@vger.kernel.org>,
Jason Cooper <jason@lakedaemon.net>,
"Ming Lei" <ming.lei@redhat.com>,
"chenxiang (M)" <chenxiang66@hisilicon.com>
Subject: Re: [PATCH] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs
Date: Tue, 21 Jan 2020 10:46:41 +0000 [thread overview]
Message-ID: <88d64d51-4344-e908-b55b-0583b0137ddf@huawei.com> (raw)
In-Reply-To: <87h80q2aoc.fsf@nanos.tec.linutronix.de>
On 20/01/2020 19:24, Thomas Gleixner wrote:
> Marc,
>
> Marc Zyngier <maz@kernel.org> writes:
>> We're stuck between a rock and a hard place here:
>>
>> (1) We place all interrupts on the least loaded CPU that matches
>> the affinity -> results in performance issues on some funky
>> HW (like D05's SAS controller).
I think that the software driver was more of the issue in that case,
which I'm fixing in the driver by spreading the interrupts properly.
But I am not sure which other platforms rely on this behavior.
>>
>> (2) We place managed interrupts on the least loaded CPU that matches
>> the affinity -> we have artificial load on NUMA boundaries, and
>> reduced spread of overlapping managed interrupts.
>>
>> (3) We don't account for non-managed LPIs, and we run the risk of
>> unpredictable performance because we don't really know where
>> the *other* interrupts are.
>>
>> My personal preference would be to go for (1), as in my original post.
That seems reasonable, but I like how x86 accounts only for managed
interrupt count per-cpu when choosing the target cpu (for a managed
interrupt).
>> I find (3) the least appealing, because we don't track things anymore.
>> (2) feels like "the least of all evils", as it is a decent performance
>> gain, seems to give predictable performance, and doesn't regress lesser
>> systems...
>>
>> I'm definitely open to suggestions here.
>
> The way x86 does it and that's mostly ok except for some really broken
> setups is:
>
> 1) Non-managed interrupts:
>
> If the interrupt is bound to a node, then we try to find a target
>
> I) in the intersection of affinity mask and node mask.
>
> II) in the nodemask itself
>
> Yes we ignore affinity mask there because that's pretty much
> the same as if the given affinity does not contain an online
> CPU.
>
> If all of that fails then we try the nodeless mode
>
> If the interrupt is not bound to a node, then we try to find a target
>
> I) in the intersection of affinity mask and online mask.
>
> II) in the onlinemask itself
>
> Each step searches for the CPU in the searched mask which has the
> least number of total interrupts assigned.
>
> 2) Managed interrupts
>
> For managed interrupts we just search in the intersection of assigned
> mask and online CPUs for the CPU with the least number of managed
> interrupts.
As above, this is something which I prefer we do.
>
> If no CPU is online then the interrupt is shutdown anyway, so no
> fallback required.
>
> Don't know whether that's something you can map to ARM64, but I assume
> the principle of trying to enforce NUMA locality plus balancing the
> number of interrupts makes sense in general.
I guess that we could use irq matrix code directly if we wanted to go
this way. That's why it is in a common location...
Cheers,
John
next prev parent reply other threads:[~2020-01-21 10:46 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-19 19:05 [PATCH] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs Marc Zyngier
2020-01-20 16:13 ` John Garry
2020-01-20 17:42 ` Marc Zyngier
2020-01-20 18:21 ` John Garry
2020-01-20 18:45 ` Marc Zyngier
2020-01-20 19:24 ` Thomas Gleixner
2020-01-21 10:46 ` John Garry [this message]
2020-03-10 11:33 ` John Garry
2020-03-11 13:18 ` Marc Zyngier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=88d64d51-4344-e908-b55b-0583b0137ddf@huawei.com \
--to=john.garry@huawei.com \
--cc=chenxiang66@hisilicon.com \
--cc=jason@lakedaemon.net \
--cc=linux-kernel@vger.kernel.org \
--cc=maz@kernel.org \
--cc=ming.lei@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).