LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Hannes Reinecke <hare@suse.de>
To: John Garry <john.garry@huawei.com>, Ming Lei <ming.lei@redhat.com>
Cc: tglx@linutronix.de, chenxiang66@hisilicon.com,
	bigeasy@linutronix.de, linux-kernel@vger.kernel.org,
	maz@kernel.org, hare@suse.com, hch@lst.de, axboe@kernel.dk,
	bvanassche@acm.org, peterz@infradead.org, mingo@redhat.com
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt
Date: Mon, 9 Dec 2019 16:09:43 +0100
Message-ID: <305198e5-f76f-ded4-946b-9cfade18f08c@suse.de> (raw)
In-Reply-To: <78a10958-fdc9-0576-0c39-6079b9749d39@huawei.com>

On 12/9/19 3:30 PM, John Garry wrote:
> On 07/12/2019 08:03, Ming Lei wrote:
>> On Fri, Dec 06, 2019 at 10:35:04PM +0800, John Garry wrote:
>>> Currently the cpu allowed mask for the threaded part of a threaded irq
>>> handler will be set to the effective affinity of the hard irq.
>>>
>>> Typically the effective affinity of the hard irq will be for a single 
>>> cpu. As such,
>>> the threaded handler would always run on the same cpu as the hard irq.
>>>
>>> We have seen scenarios in high data-rate throughput testing that the cpu
>>> handling the interrupt can be totally saturated handling both the hard
>>> interrupt and threaded handler parts, limiting throughput.
>>
> 
> Hi Ming,
> 
>> Frankly speaking, I never observed that single CPU is saturated by one 
>> storage
>> completion queue's interrupt load. Because CPU is still much quicker than
>> current storage device.
>>
>> If there are more drives, one CPU won't handle more than one 
>> queue(drive)'s
>> interrupt if (nr_drive * nr_hw_queues) < nr_cpu_cores.
> 
> Are things this simple? I mean, can you guarantee that fio processes are 
> evenly distributed as such?
> 
I would assume that it does, seeing that that was the primary goal of 
fio ...

>>
>> So could you describe your case in a bit detail? Then we can confirm
>> if this change is really needed.
> 
> The issue is that the CPU is saturated in servicing the hard and 
> threaded part of the interrupt together - here's the sort of thing which 
> we saw previously:
> Before:
> CPU    %usr    %sys    %irq    %soft    %idle
> all    2.9    13.1    1.2    4.6    78.2
> 0    0.0    29.3    10.1    58.6    2.0
> 1    18.2    39.4    0.0    1.0    41.4
> 2    0.0    2.0    0.0    0.0    98.0
> 
> CPU0 has no effectively no idle.
> 
> Then, by allowing the threaded part to roam:
> After:
> CPU    %usr    %sys    %irq    %soft    %idle
> all    3.5    18.4    2.7    6.8    68.6
> 0    0.0    20.6    29.9    29.9    19.6
> 1    0.0    39.8    0.0    50.0    10.2
> 
> Note: I think that I may be able to reduce the irq hard part load in the 
> endpoint driver, but not that much such that we see still this issue.
> 
Well ... to get a _real_ comparison you would need to specify the number 
of irqs handled (and the resulting IOPS) alongside the cpu load.
It might well be that by spreading out the interrupts to other CPUs 
we're increasing the latency, thus trivially reducing the load ...

My idea here is slightly different: can't we leverage SMT?
Most modern CPUs do SMT (I guess even ARM does it nowadays)
(Yes, I know about spectre and things. We're talking performance here :-)

So for 2-way SMT one could move the submisson queue on one side, and the 
completion queue handling (ie the irq handling) on the other side.
Due to SMT we shouldn't suffer from cache misses (keep fingers crossed),
and might even get better performance.

John, would such a scenario work on your boxes?
IE can we tweak the interrupt and queue assignment?

Initially I would love to test things out, just to see what'll be 
happening; might be that it doesn't bring any benefit at all, but it'd 
be interesting to test out anyway.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry
2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry
2019-12-06 15:22   ` Marc Zyngier
2019-12-06 16:16     ` John Garry
2019-12-07  8:03   ` Ming Lei
2019-12-09 14:30     ` John Garry
2019-12-09 15:09       ` Hannes Reinecke [this message]
2019-12-09 15:17         ` Marc Zyngier
2019-12-09 15:25           ` Hannes Reinecke
2019-12-09 15:36             ` Marc Zyngier
2019-12-09 15:49           ` Qais Yousef
2019-12-09 15:55             ` Marc Zyngier
2019-12-10  1:43       ` Ming Lei
2019-12-10  9:45         ` John Garry
2019-12-10 10:06           ` Ming Lei
2019-12-10 10:28           ` Marc Zyngier
2019-12-10 10:59             ` John Garry
2019-12-10 11:36               ` Marc Zyngier
2019-12-10 12:05                 ` John Garry
2019-12-10 18:32                   ` Marc Zyngier
2019-12-11  9:41                     ` John Garry
2019-12-13 10:07                       ` John Garry
2019-12-13 10:31                         ` Marc Zyngier
2019-12-13 12:08                           ` John Garry
2019-12-14 10:59                             ` Marc Zyngier
2019-12-11 17:09         ` John Garry
2019-12-12 22:38           ` Ming Lei
2019-12-13 11:12             ` John Garry
2019-12-13 13:18               ` Ming Lei
2019-12-13 15:43                 ` John Garry
2019-12-13 17:12                   ` Ming Lei
2019-12-13 17:50                     ` John Garry
2019-12-14 13:56                   ` Marc Zyngier
2019-12-16 10:47                     ` John Garry
2019-12-16 11:40                       ` Marc Zyngier
2019-12-16 14:17                         ` John Garry
2019-12-16 18:00                           ` Marc Zyngier
2019-12-16 18:50                             ` John Garry
2019-12-20 11:30                               ` John Garry
2019-12-20 14:43                                 ` Marc Zyngier
2019-12-20 15:38                                   ` John Garry
2019-12-20 16:16                                     ` Marc Zyngier
2019-12-20 23:31                                     ` Ming Lei
2019-12-23  9:07                                       ` Marc Zyngier
2019-12-23 10:26                                         ` John Garry
2019-12-23 10:47                                           ` Marc Zyngier
2019-12-23 11:35                                             ` John Garry
2019-12-24  1:59                                             ` Ming Lei
2019-12-24 11:20                                               ` Marc Zyngier
2019-12-25  0:48                                                 ` Ming Lei
2020-01-02 10:35                                                   ` John Garry
2020-01-03  0:46                                                     ` Ming Lei
2020-01-03 10:41                                                       ` John Garry
2020-01-03 11:29                                                         ` Ming Lei
2020-01-03 11:50                                                           ` John Garry
2020-01-04 12:03                                                             ` Ming Lei
2020-05-30  7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=305198e5-f76f-ded4-946b-9cfade18f08c@suse.de \
    --to=hare@suse.de \
    --cc=axboe@kernel.dk \
    --cc=bigeasy@linutronix.de \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=john.garry@huawei.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git