From: Marc Zyngier <firstname.lastname@example.org> To: John Garry <email@example.com> Cc: Ming Lei <firstname.lastname@example.org>, <email@example.com>, "chenxiang (M)" <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org> Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt Date: Mon, 16 Dec 2019 11:40:24 +0000 Message-ID: <email@example.com> (raw) In-Reply-To: <firstname.lastname@example.org> On 2019-12-16 10:47, John Garry wrote: > On 14/12/2019 13:56, Marc Zyngier wrote: >> On Fri, 13 Dec 2019 15:43:07 +0000 >> John Garry <email@example.com> wrote: >> [...] >> >>> john@ubuntu:~$ ./dump-io-irq-affinity >>> kernel version: >>> Linux ubuntu 5.5.0-rc1-00003-g7adc5d7ec1ca-dirty #1440 SMP PREEMPT >>> Fri Dec 13 14:53:19 GMT 2019 aarch64 aarch64 aarch64 GNU/Linux >>> PCI name is 04:00.0: nvme0n1 >>> irq 56, cpu list 75, effective list 5 >>> irq 60, cpu list 24-28, effective list 10 >> The NUMA selection code definitely gets in the way. And to be >> honest, >> this NUMA thing is only there for the benefit of a terminally broken >> implementation (Cavium ThunderX), which we should have never >> supported >> the first place. >> Let's rework this and simply use the managed affinity whenever >> available instead. It may well be that it will break TX1, but I care >> about it just as much as Cavium/Marvell does... > > I'm just wondering if non-managed interrupts should be included in > the load balancing calculation? Couldn't irqbalance (if active) start > moving non-managed interrupts around anyway? But they are, aren't they? See what we do in irq_set_affinity: + atomic_inc(per_cpu_ptr(&cpu_lpi_count, cpu)); + atomic_dec(per_cpu_ptr(&cpu_lpi_count, + its_dev->event_map.col_map[id])); We don't try to "rebalance" anything based on that though, not that I think we should. > >> Please give this new patch a shot on your system (my D05 doesn't >> have >> any managed devices): > > We could consider supporting platform msi managed interrupts, but I > doubt the value. It shouldn't be hard to do, and most of the existing code could be moved to the generic level. As for the value, I'm not convinced either. For example D05 uses the MBIGEN as an intermediate interrupt controller, so MSIs are from the PoV of MBIGEN, and not the SAS device attached to it. Not the best design... >> >> https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=irq/its-balance-mappings&id=1e987d83b8d880d56c9a2d8a86289631da94e55a >> > > I quickly tested that in my NVMe env, and I see a performance boost > of 1055K -> 1206K IOPS. Results at bottom. OK, that's encouraging. > Here's the irq mapping dump: [...] Looks good. > I'm still getting the CPU lockup (even on CPUs which have a single > NVMe completion interrupt assigned), which taints these results. That > lockup needs to be fixed. Is this interrupt screaming to the point where it prevents the completion thread from making forward progress? What if you don't use threaded interrupts? > We'll check on our SAS env also. I did already hack something up > similar to your change and again we saw a boost there. OK. Please keep me posted. If the result is overall positive, I'll push this into -next for some soaking. Thanks, M. -- Jazz is not dead. It just smells funny...
next prev parent reply index Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry 2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry 2019-12-06 15:22 ` Marc Zyngier 2019-12-06 16:16 ` John Garry 2019-12-07 8:03 ` Ming Lei 2019-12-09 14:30 ` John Garry 2019-12-09 15:09 ` Hannes Reinecke 2019-12-09 15:17 ` Marc Zyngier 2019-12-09 15:25 ` Hannes Reinecke 2019-12-09 15:36 ` Marc Zyngier 2019-12-09 15:49 ` Qais Yousef 2019-12-09 15:55 ` Marc Zyngier 2019-12-10 1:43 ` Ming Lei 2019-12-10 9:45 ` John Garry 2019-12-10 10:06 ` Ming Lei 2019-12-10 10:28 ` Marc Zyngier 2019-12-10 10:59 ` John Garry 2019-12-10 11:36 ` Marc Zyngier 2019-12-10 12:05 ` John Garry 2019-12-10 18:32 ` Marc Zyngier 2019-12-11 9:41 ` John Garry 2019-12-13 10:07 ` John Garry 2019-12-13 10:31 ` Marc Zyngier 2019-12-13 12:08 ` John Garry 2019-12-14 10:59 ` Marc Zyngier 2019-12-11 17:09 ` John Garry 2019-12-12 22:38 ` Ming Lei 2019-12-13 11:12 ` John Garry 2019-12-13 13:18 ` Ming Lei 2019-12-13 15:43 ` John Garry 2019-12-13 17:12 ` Ming Lei 2019-12-13 17:50 ` John Garry 2019-12-14 13:56 ` Marc Zyngier 2019-12-16 10:47 ` John Garry 2019-12-16 11:40 ` Marc Zyngier [this message] 2019-12-16 14:17 ` John Garry 2019-12-16 18:00 ` Marc Zyngier 2019-12-16 18:50 ` John Garry 2019-12-20 11:30 ` John Garry 2019-12-20 14:43 ` Marc Zyngier 2019-12-20 15:38 ` John Garry 2019-12-20 16:16 ` Marc Zyngier 2019-12-20 23:31 ` Ming Lei 2019-12-23 9:07 ` Marc Zyngier 2019-12-23 10:26 ` John Garry 2019-12-23 10:47 ` Marc Zyngier 2019-12-23 11:35 ` John Garry 2019-12-24 1:59 ` Ming Lei 2019-12-24 11:20 ` Marc Zyngier 2019-12-25 0:48 ` Ming Lei 2020-01-02 10:35 ` John Garry 2020-01-03 0:46 ` Ming Lei 2020-01-03 10:41 ` John Garry 2020-01-03 11:29 ` Ming Lei 2020-01-03 11:50 ` John Garry 2020-01-04 12:03 ` Ming Lei 2020-05-30 7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
LKML Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \ firstname.lastname@example.org public-inbox-index lkml Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel AGPL code for this site: git clone https://public-inbox.org/public-inbox.git