From: Marc Zyngier <firstname.lastname@example.org> To: John Garry <email@example.com> Cc: Ming Lei <firstname.lastname@example.org>, <email@example.com>, "chenxiang (M)" <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org> Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt Date: Fri, 13 Dec 2019 10:31:10 +0000 Message-ID: <email@example.com> (raw) In-Reply-To: <firstname.lastname@example.org> Hi John, On 2019-12-13 10:07, John Garry wrote: > On 11/12/2019 09:41, John Garry wrote: >> On 10/12/2019 18:32, Marc Zyngier wrote: >>>>>> The ITS code will make the lowest online CPU in the affinity >>>>>> mask >>>>>> the >>>>>> target CPU for the interrupt, which may result in some CPUs >>>>>> handling >>>>>> so many interrupts. >>>>> If what you want is for the*default* affinity to be spread >>>>> around, >>>>> that should be achieved pretty easily. Let me have a think about >>>>> how >>>>> to do that. >>>> Cool, I anticipate that it should help my case. >>>> >>>> I can also seek out some NVMe cards to see how it would help a >>>> more >>>> "generic" scenario. >>> Can you give the following a go? It probably has all kind of warts >>> on >>> top of the quality debug information, but I managed to get my D05 >>> and >>> a couple of guests to boot with it. It will probably eat your data, >>> so use caution!;-) >>> >> Hi Marc, >> Ok, we'll give it a spin. >> Thanks, >> John > > Hi Marc, > > JFYI, we're still testing this and the patch itself seems to work as > intended. > > Here's the kernel log if you just want to see how the interrupts are > getting assigned: > https://pastebin.com/hh3r810g It is a bit hard to make sense of this dump, specially on such a wide machine (I want one!) without really knowing the topology of the system. > For me, I did get a performance boost for NVMe testing, but my > colleague Xiang Chen saw a drop for our storage test of interest - > that's the HiSi SAS controller. We're trying to make sense of it now. One of the difference is that with this patch, the initial affinity is picked inside the NUMA node that matches the ITS. In your case, that's either node 0 or 2. But it is unclear whether which CPUs these map to. Given that I see interrupts mapped to CPUs 0-23 on one side, and 48-71 on the other, it looks like half of your machine gets starved, and that may be because no ITS targets the NUMA nodes they are part of. It would be interesting to see what happens if you manually set the affinity of the interrupts outside of the NUMA node. Thanks, M. -- Jazz is not dead. It just smells funny...
next prev parent reply index Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry 2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry 2019-12-06 15:22 ` Marc Zyngier 2019-12-06 16:16 ` John Garry 2019-12-07 8:03 ` Ming Lei 2019-12-09 14:30 ` John Garry 2019-12-09 15:09 ` Hannes Reinecke 2019-12-09 15:17 ` Marc Zyngier 2019-12-09 15:25 ` Hannes Reinecke 2019-12-09 15:36 ` Marc Zyngier 2019-12-09 15:49 ` Qais Yousef 2019-12-09 15:55 ` Marc Zyngier 2019-12-10 1:43 ` Ming Lei 2019-12-10 9:45 ` John Garry 2019-12-10 10:06 ` Ming Lei 2019-12-10 10:28 ` Marc Zyngier 2019-12-10 10:59 ` John Garry 2019-12-10 11:36 ` Marc Zyngier 2019-12-10 12:05 ` John Garry 2019-12-10 18:32 ` Marc Zyngier 2019-12-11 9:41 ` John Garry 2019-12-13 10:07 ` John Garry 2019-12-13 10:31 ` Marc Zyngier [this message] 2019-12-13 12:08 ` John Garry 2019-12-14 10:59 ` Marc Zyngier 2019-12-11 17:09 ` John Garry 2019-12-12 22:38 ` Ming Lei 2019-12-13 11:12 ` John Garry 2019-12-13 13:18 ` Ming Lei 2019-12-13 15:43 ` John Garry 2019-12-13 17:12 ` Ming Lei 2019-12-13 17:50 ` John Garry 2019-12-14 13:56 ` Marc Zyngier 2019-12-16 10:47 ` John Garry 2019-12-16 11:40 ` Marc Zyngier 2019-12-16 14:17 ` John Garry 2019-12-16 18:00 ` Marc Zyngier 2019-12-16 18:50 ` John Garry 2019-12-20 11:30 ` John Garry 2019-12-20 14:43 ` Marc Zyngier 2019-12-20 15:38 ` John Garry 2019-12-20 16:16 ` Marc Zyngier 2019-12-20 23:31 ` Ming Lei 2019-12-23 9:07 ` Marc Zyngier 2019-12-23 10:26 ` John Garry 2019-12-23 10:47 ` Marc Zyngier 2019-12-23 11:35 ` John Garry 2019-12-24 1:59 ` Ming Lei 2019-12-24 11:20 ` Marc Zyngier 2019-12-25 0:48 ` Ming Lei 2020-01-02 10:35 ` John Garry 2020-01-03 0:46 ` Ming Lei 2020-01-03 10:41 ` John Garry 2020-01-03 11:29 ` Ming Lei 2020-01-03 11:50 ` John Garry 2020-01-04 12:03 ` Ming Lei 2020-05-30 7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
LKML Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \ email@example.com public-inbox-index lkml Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel AGPL code for this site: git clone https://public-inbox.org/public-inbox.git