From: Marc Zyngier <email@example.com> To: John Garry <firstname.lastname@example.org> Cc: Ming Lei <email@example.com>, <firstname.lastname@example.org>, "chenxiang (M)" <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com> Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt Date: Fri, 20 Dec 2019 14:43:36 +0000 Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <email@example.com> Hi John, On 2019-12-20 11:30, John Garry wrote: >>> So you enqueue requests from CPU0 only? It seems a bit odd... >> No, but maybe I wasn't clear enough. I'll give an overview: >> For D06 SAS controller - which is a multi-queue PCI device - we use >> managed interrupts. The HW has 16 submission/completion queues, so for >> 96 cores, we have an even spread of 6 CPUs assigned per queue; and >> this per-queue CPU mask is the interrupt affinity mask. So CPU0-5 >> would submit any IO on queue0, CPU6-11 on queue2, and so on. PCI NVMe >> is essentially the same. >> These are the environments which we're trying to promote >> performance. >> Then for D05 SAS controller - which is multi-queue platform device >> (mbigen) - we don't use managed interrupts. We still submit IO from >> any CPU, but we choose the queue to submit IO on a round-robin basis >> to promote some isolation, i.e. reduce inter-queue lock contention, so >> the queue chosen has nothing to do with the CPU. >> And with your change we may submit on cpu4 but service the interrupt >> on cpu30, as an example. While previously we would always service on >> cpu0. The old way still isn't ideal, I'll admit. >> For this env, we would just like to maintain the same performance. >> And it's here that we see the performance drop. >> > > Hi Marc, > > We've got some more results and it looks promising. > > So with your patch we get a performance boost of 3180.1K -> 3294.9K > IOPS in the D06 SAS env. Then when we change the driver to use > threaded interrupt handler (mainline currently uses tasklet), we get > a > boost again up to 3415K IOPS. > > Now this is essentially the same figure we had with using threaded > handler + the gen irq change in spreading the handler CPU affinity. > We > did also test your patch + gen irq change and got a performance drop, > to 3347K IOPS. > > So tentatively I'd say your patch may be all we need. OK. > FYI, here is how the effective affinity is looking for both SAS > controllers with your patch: > > 74:02.0 > irq 81, cpu list 24-29, effective list 24 cq > irq 82, cpu list 30-35, effective list 30 cq Cool. [...] > As for your patch itself, I'm still concerned of possible regressions > if we don't apply this effective interrupt affinity spread policy to > only managed interrupts. I'll try and revise that as I post the patch, probably at some point between now and Christmas. I still think we should find a way to address this for the D05 SAS driver though, maybe by managing the affinity yourself in the driver. But this requires experimentation. > JFYI, about NVMe CPU lockup issue, there are 2 works on going here: > > https://firstname.lastname@example.org/T/#t > > https://email@example.com/T/#t I've also managed to trigger some of them now that I have access to a decent box with nvme storage. Out of curiosity, have you tried with the SMMU disabled? I'm wondering whether we hit some livelock condition on unmapping buffers... > Cheers, > John > > Ps. Thanks to Xiang Chen for all the work here in getting these > results. Yup, much appreciated! Thanks, M. -- Jazz is not dead. It just smells funny...
next prev parent reply index Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry 2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry 2019-12-06 15:22 ` Marc Zyngier 2019-12-06 16:16 ` John Garry 2019-12-07 8:03 ` Ming Lei 2019-12-09 14:30 ` John Garry 2019-12-09 15:09 ` Hannes Reinecke 2019-12-09 15:17 ` Marc Zyngier 2019-12-09 15:25 ` Hannes Reinecke 2019-12-09 15:36 ` Marc Zyngier 2019-12-09 15:49 ` Qais Yousef 2019-12-09 15:55 ` Marc Zyngier 2019-12-10 1:43 ` Ming Lei 2019-12-10 9:45 ` John Garry 2019-12-10 10:06 ` Ming Lei 2019-12-10 10:28 ` Marc Zyngier 2019-12-10 10:59 ` John Garry 2019-12-10 11:36 ` Marc Zyngier 2019-12-10 12:05 ` John Garry 2019-12-10 18:32 ` Marc Zyngier 2019-12-11 9:41 ` John Garry 2019-12-13 10:07 ` John Garry 2019-12-13 10:31 ` Marc Zyngier 2019-12-13 12:08 ` John Garry 2019-12-14 10:59 ` Marc Zyngier 2019-12-11 17:09 ` John Garry 2019-12-12 22:38 ` Ming Lei 2019-12-13 11:12 ` John Garry 2019-12-13 13:18 ` Ming Lei 2019-12-13 15:43 ` John Garry 2019-12-13 17:12 ` Ming Lei 2019-12-13 17:50 ` John Garry 2019-12-14 13:56 ` Marc Zyngier 2019-12-16 10:47 ` John Garry 2019-12-16 11:40 ` Marc Zyngier 2019-12-16 14:17 ` John Garry 2019-12-16 18:00 ` Marc Zyngier 2019-12-16 18:50 ` John Garry 2019-12-20 11:30 ` John Garry 2019-12-20 14:43 ` Marc Zyngier [this message] 2019-12-20 15:38 ` John Garry 2019-12-20 16:16 ` Marc Zyngier 2019-12-20 23:31 ` Ming Lei 2019-12-23 9:07 ` Marc Zyngier 2019-12-23 10:26 ` John Garry 2019-12-23 10:47 ` Marc Zyngier 2019-12-23 11:35 ` John Garry 2019-12-24 1:59 ` Ming Lei 2019-12-24 11:20 ` Marc Zyngier 2019-12-25 0:48 ` Ming Lei 2020-01-02 10:35 ` John Garry 2020-01-03 0:46 ` Ming Lei 2020-01-03 10:41 ` John Garry 2020-01-03 11:29 ` Ming Lei 2020-01-03 11:50 ` John Garry 2020-01-04 12:03 ` Ming Lei 2020-05-30 7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
LKML Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \ firstname.lastname@example.org public-inbox-index lkml Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel AGPL code for this site: git clone https://public-inbox.org/public-inbox.git