LKML Archive on lore.kernel.org
 help / color / Atom feed
From: John Garry <john.garry@huawei.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Ming Lei <ming.lei@redhat.com>, <tglx@linutronix.de>,
	<chenxiang66@hisilicon.com>, <bigeasy@linutronix.de>,
	<linux-kernel@vger.kernel.org>, <hare@suse.com>, <hch@lst.de>,
	<axboe@kernel.dk>, <bvanassche@acm.org>, <peterz@infradead.org>,
	<mingo@redhat.com>
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt
Date: Tue, 10 Dec 2019 12:05:20 +0000
Message-ID: <06d1e2ff-9ec7-2262-25a0-4503cb204b0b@huawei.com> (raw)
In-Reply-To: <6e513d25d8b0c6b95d37a64df0c27b78@www.loen.fr>

On 10/12/2019 11:36, Marc Zyngier wrote:
> On 2019-12-10 10:59, John Garry wrote:
>>>>
>>>> There is no lockup, just a potential performance boost in this change.
>>>>
>>>> My colleague Xiang Chen can provide specifics of the test, as he is
>>>> the one running it.
>>>>
>>>> But one key bit of info - which I did not think most relevant before
>>>> - that is we have 2x SAS controllers running the throughput test on
>>>> the same host.
>>>>
>>>> As such, the completion queue interrupts would be spread identically
>>>> over the CPUs for each controller. I notice that ARM GICv3 ITS
>>>> interrupt controller (which we use) does not use the generic irq
>>>> matrix allocator, which I think would really help with this.
>>>>
>>>> Hi Marc,
>>>>
>>>> Is there any reason for which we couldn't utilise of the generic irq
>>>> matrix allocator for GICv3?
>>>
>>
>> Hi Marc,
>>
>>> For a start, the ITS code predates the matrix allocator by about three
>>> years. Also, my understanding of this allocator is that it allows
>>> x86 to cope with a very small number of possible interrupt vectors
>>> per CPU. The ITS doesn't have such issue, as:
>>> 1) the namespace is global, and not per CPU
>>> 2) the namespace is *huge*
>>> Now, what property of the matrix allocator is the ITS code missing?
>>> I'd be more than happy to improve it.
>>
>> I think specifically the property that the matrix allocator will try
>> to find a CPU for irq affinity which "has the lowest number of managed
>> IRQs allocated" - I'm quoting the comment on 
>> matrix_find_best_cpu_managed().
> 
> But that decision is due to allocation constraints. You can have at most
> 256 interrupts per CPU, so the allocator tries to balance it.
> 
> On the contrary, the ITS does care about how many interrupt target any
> given CPU. The whole 2^24 interrupt namespace can be thrown at a single
> CPU.
> 
>> The ITS code will make the lowest online CPU in the affinity mask the
>> target CPU for the interrupt, which may result in some CPUs handling
>> so many interrupts.
> 
> If what you want is for the *default* affinity to be spread around,
> that should be achieved pretty easily. Let me have a think about how
> to do that.

Cool, I anticipate that it should help my case.

I can also seek out some NVMe cards to see how it would help a more 
"generic" scenario.

Cheers,
John

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry
2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry
2019-12-06 15:22   ` Marc Zyngier
2019-12-06 16:16     ` John Garry
2019-12-07  8:03   ` Ming Lei
2019-12-09 14:30     ` John Garry
2019-12-09 15:09       ` Hannes Reinecke
2019-12-09 15:17         ` Marc Zyngier
2019-12-09 15:25           ` Hannes Reinecke
2019-12-09 15:36             ` Marc Zyngier
2019-12-09 15:49           ` Qais Yousef
2019-12-09 15:55             ` Marc Zyngier
2019-12-10  1:43       ` Ming Lei
2019-12-10  9:45         ` John Garry
2019-12-10 10:06           ` Ming Lei
2019-12-10 10:28           ` Marc Zyngier
2019-12-10 10:59             ` John Garry
2019-12-10 11:36               ` Marc Zyngier
2019-12-10 12:05                 ` John Garry [this message]
2019-12-10 18:32                   ` Marc Zyngier
2019-12-11  9:41                     ` John Garry
2019-12-13 10:07                       ` John Garry
2019-12-13 10:31                         ` Marc Zyngier
2019-12-13 12:08                           ` John Garry
2019-12-14 10:59                             ` Marc Zyngier
2019-12-11 17:09         ` John Garry
2019-12-12 22:38           ` Ming Lei
2019-12-13 11:12             ` John Garry
2019-12-13 13:18               ` Ming Lei
2019-12-13 15:43                 ` John Garry
2019-12-13 17:12                   ` Ming Lei
2019-12-13 17:50                     ` John Garry
2019-12-14 13:56                   ` Marc Zyngier
2019-12-16 10:47                     ` John Garry
2019-12-16 11:40                       ` Marc Zyngier
2019-12-16 14:17                         ` John Garry
2019-12-16 18:00                           ` Marc Zyngier
2019-12-16 18:50                             ` John Garry
2019-12-20 11:30                               ` John Garry
2019-12-20 14:43                                 ` Marc Zyngier
2019-12-20 15:38                                   ` John Garry
2019-12-20 16:16                                     ` Marc Zyngier
2019-12-20 23:31                                     ` Ming Lei
2019-12-23  9:07                                       ` Marc Zyngier
2019-12-23 10:26                                         ` John Garry
2019-12-23 10:47                                           ` Marc Zyngier
2019-12-23 11:35                                             ` John Garry
2019-12-24  1:59                                             ` Ming Lei
2019-12-24 11:20                                               ` Marc Zyngier
2019-12-25  0:48                                                 ` Ming Lei
2020-01-02 10:35                                                   ` John Garry
2020-01-03  0:46                                                     ` Ming Lei
2020-01-03 10:41                                                       ` John Garry
2020-01-03 11:29                                                         ` Ming Lei
2020-01-03 11:50                                                           ` John Garry
2020-01-04 12:03                                                             ` Ming Lei
2020-05-30  7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=06d1e2ff-9ec7-2262-25a0-4503cb204b0b@huawei.com \
    --to=john.garry@huawei.com \
    --cc=axboe@kernel.dk \
    --cc=bigeasy@linutronix.de \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git