LKML Archive on lore.kernel.org
 help / color / Atom feed
From: John Garry <john.garry@huawei.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Ming Lei <ming.lei@redhat.com>, <tglx@linutronix.de>,
	"chenxiang (M)" <chenxiang66@hisilicon.com>,
	<bigeasy@linutronix.de>, <linux-kernel@vger.kernel.org>,
	<hare@suse.com>, <hch@lst.de>, <axboe@kernel.dk>,
	<bvanassche@acm.org>, <peterz@infradead.org>, <mingo@redhat.com>
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt
Date: Fri, 20 Dec 2019 11:30:44 +0000
Message-ID: <687cbcc4-89d9-63ea-a246-ce2abaae501a@huawei.com> (raw)
In-Reply-To: <ac5b5a25-df2e-18e9-6b0f-60af8c7cec3b@huawei.com>

>> So you enqueue requests from CPU0 only? It seems a bit odd...
> 
> No, but maybe I wasn't clear enough. I'll give an overview:
> 
> For D06 SAS controller - which is a multi-queue PCI device - we use 
> managed interrupts. The HW has 16 submission/completion queues, so for 
> 96 cores, we have an even spread of 6 CPUs assigned per queue; and this 
> per-queue CPU mask is the interrupt affinity mask. So CPU0-5 would 
> submit any IO on queue0, CPU6-11 on queue2, and so on. PCI NVMe is 
> essentially the same.
> 
> These are the environments which we're trying to promote performance.
> 
> Then for D05 SAS controller - which is multi-queue platform device 
> (mbigen) - we don't use managed interrupts. We still submit IO from any 
> CPU, but we choose the queue to submit IO on a round-robin basis to 
> promote some isolation, i.e. reduce inter-queue lock contention, so the 
> queue chosen has nothing to do with the CPU.
> 
> And with your change we may submit on cpu4 but service the interrupt on 
> cpu30, as an example. While previously we would always service on cpu0. 
> The old way still isn't ideal, I'll admit.
> 
> For this env, we would just like to maintain the same performance. And 
> it's here that we see the performance drop.
> 

Hi Marc,

We've got some more results and it looks promising.

So with your patch we get a performance boost of 3180.1K -> 3294.9K IOPS 
in the D06 SAS env. Then when we change the driver to use threaded 
interrupt handler (mainline currently uses tasklet), we get a boost 
again up to 3415K IOPS.

Now this is essentially the same figure we had with using threaded 
handler + the gen irq change in spreading the handler CPU affinity. We 
did also test your patch + gen irq change and got a performance drop, to 
3347K IOPS.

So tentatively I'd say your patch may be all we need.

FYI, here is how the effective affinity is looking for both SAS 
controllers with your patch:

74:02.0
irq 81, cpu list 24-29, effective list 24 cq
irq 82, cpu list 30-35, effective list 30 cq
irq 83, cpu list 36-41, effective list 36 cq
irq 84, cpu list 42-47, effective list 42 cq
irq 85, cpu list 48-53, effective list 48 cq
irq 86, cpu list 54-59, effective list 56 cq
irq 87, cpu list 60-65, effective list 60 cq
irq 88, cpu list 66-71, effective list 66 cq
irq 89, cpu list 72-77, effective list 72 cq
irq 90, cpu list 78-83, effective list 78 cq
irq 91, cpu list 84-89, effective list 84 cq
irq 92, cpu list 90-95, effective list 90 cq
irq 93, cpu list 0-5, effective list 0 cq
irq 94, cpu list 6-11, effective list 6 cq
irq 95, cpu list 12-17, effective list 12 cq
irq 96, cpu list 18-23, effective list 18 cq

74:04.0
irq 113, cpu list 24-29, effective list 25 cq
irq 114, cpu list 30-35, effective list 31 cq
irq 115, cpu list 36-41, effective list 37 cq
irq 116, cpu list 42-47, effective list 43 cq
irq 117, cpu list 48-53, effective list 49 cq
irq 118, cpu list 54-59, effective list 57 cq
irq 119, cpu list 60-65, effective list 61 cq
irq 120, cpu list 66-71, effective list 67 cq
irq 121, cpu list 72-77, effective list 73 cq
irq 122, cpu list 78-83, effective list 79 cq
irq 123, cpu list 84-89, effective list 85 cq
irq 124, cpu list 90-95, effective list 91 cq
irq 125, cpu list 0-5, effective list 1 cq
irq 126, cpu list 6-11, effective list 7 cq
irq 127, cpu list 12-17, effective list 17 cq
irq 128, cpu list 18-23, effective list 19 cq

As for your patch itself, I'm still concerned of possible regressions if 
we don't apply this effective interrupt affinity spread policy to only 
managed interrupts.

JFYI, about NVMe CPU lockup issue, there are 2 works on going here:
https://lore.kernel.org/linux-nvme/20191209175622.1964-1-kbusch@kernel.org/T/#t
https://lore.kernel.org/linux-block/20191218071942.22336-1-ming.lei@redhat.com/T/#t

Cheers,
John

Ps. Thanks to Xiang Chen for all the work here in getting these results.

>>
>>>>>> Please give this new patch a shot on your system (my D05 doesn't have
>>>>>> any managed devices):
>>>>>
>>>>> We could consider supporting platform msi managed interrupts, but I 


  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry
2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry
2019-12-06 15:22   ` Marc Zyngier
2019-12-06 16:16     ` John Garry
2019-12-07  8:03   ` Ming Lei
2019-12-09 14:30     ` John Garry
2019-12-09 15:09       ` Hannes Reinecke
2019-12-09 15:17         ` Marc Zyngier
2019-12-09 15:25           ` Hannes Reinecke
2019-12-09 15:36             ` Marc Zyngier
2019-12-09 15:49           ` Qais Yousef
2019-12-09 15:55             ` Marc Zyngier
2019-12-10  1:43       ` Ming Lei
2019-12-10  9:45         ` John Garry
2019-12-10 10:06           ` Ming Lei
2019-12-10 10:28           ` Marc Zyngier
2019-12-10 10:59             ` John Garry
2019-12-10 11:36               ` Marc Zyngier
2019-12-10 12:05                 ` John Garry
2019-12-10 18:32                   ` Marc Zyngier
2019-12-11  9:41                     ` John Garry
2019-12-13 10:07                       ` John Garry
2019-12-13 10:31                         ` Marc Zyngier
2019-12-13 12:08                           ` John Garry
2019-12-14 10:59                             ` Marc Zyngier
2019-12-11 17:09         ` John Garry
2019-12-12 22:38           ` Ming Lei
2019-12-13 11:12             ` John Garry
2019-12-13 13:18               ` Ming Lei
2019-12-13 15:43                 ` John Garry
2019-12-13 17:12                   ` Ming Lei
2019-12-13 17:50                     ` John Garry
2019-12-14 13:56                   ` Marc Zyngier
2019-12-16 10:47                     ` John Garry
2019-12-16 11:40                       ` Marc Zyngier
2019-12-16 14:17                         ` John Garry
2019-12-16 18:00                           ` Marc Zyngier
2019-12-16 18:50                             ` John Garry
2019-12-20 11:30                               ` John Garry [this message]
2019-12-20 14:43                                 ` Marc Zyngier
2019-12-20 15:38                                   ` John Garry
2019-12-20 16:16                                     ` Marc Zyngier
2019-12-20 23:31                                     ` Ming Lei
2019-12-23  9:07                                       ` Marc Zyngier
2019-12-23 10:26                                         ` John Garry
2019-12-23 10:47                                           ` Marc Zyngier
2019-12-23 11:35                                             ` John Garry
2019-12-24  1:59                                             ` Ming Lei
2019-12-24 11:20                                               ` Marc Zyngier
2019-12-25  0:48                                                 ` Ming Lei
2020-01-02 10:35                                                   ` John Garry
2020-01-03  0:46                                                     ` Ming Lei
2020-01-03 10:41                                                       ` John Garry
2020-01-03 11:29                                                         ` Ming Lei
2020-01-03 11:50                                                           ` John Garry
2020-01-04 12:03                                                             ` Ming Lei
2020-05-30  7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=687cbcc4-89d9-63ea-a246-ce2abaae501a@huawei.com \
    --to=john.garry@huawei.com \
    --cc=axboe@kernel.dk \
    --cc=bigeasy@linutronix.de \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git