LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Marc Zyngier <maz@kernel.org>
To: John Garry <john.garry@huawei.com>
Cc: Ming Lei <ming.lei@redhat.com>, <tglx@linutronix.de>,
	"chenxiang (M)" <chenxiang66@hisilicon.com>,
	<bigeasy@linutronix.de>, <linux-kernel@vger.kernel.org>,
	<hare@suse.com>, <hch@lst.de>, <axboe@kernel.dk>,
	<bvanassche@acm.org>, <peterz@infradead.org>, <mingo@redhat.com>
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt
Date: Sat, 14 Dec 2019 10:59:11 +0000
Message-ID: <86eex7i35s.wl-maz@kernel.org> (raw)
In-Reply-To: <174bfdbe-0c44-298b-35e9-8466e77528df@huawei.com>

On Fri, 13 Dec 2019 12:08:54 +0000,
John Garry <john.garry@huawei.com> wrote:
> 
> Hi Marc,
> 
> >> JFYI, we're still testing this and the patch itself seems to work as
> >> intended.
> >> 
> >> Here's the kernel log if you just want to see how the interrupts are
> >> getting assigned:
> >> https://pastebin.com/hh3r810g
> > 
> > It is a bit hard to make sense of this dump, specially on such a wide
> > machine (I want one!) 
> 
> So do I :) That's the newer "D06CS" board.
> 
> without really knowing the topology of the system.
> 
> So it's 2x socket, each socket has 2x CPU dies, and each die has 6
> clusters of 4 CPUs, which gives 96 in total.
> 
> > 
> >> For me, I did get a performance boost for NVMe testing, but my
> >> colleague Xiang Chen saw a drop for our storage test of interest  -
> >> that's the HiSi SAS controller. We're trying to make sense of it now.
> > 
> > One of the difference is that with this patch, the initial affinity
> > is picked inside the NUMA node that matches the ITS. 
> 
> Is that even for managed interrupts? We're testing the storage
> controller which uses managed interrupts. I should have made that
> clearer.

The ITS driver doesn't care about the fact that an interrupt affinity
is 'managed' or not. And I don't think a low-level driver should, as
it will just follow whatever interrupt affinity it is requested to
use. If a managed interrupt has some requirements, then these
requirements better be explicit in terms of CPU affinity.

> In your case,
> > that's either node 0 or 2. But it is unclear whether which CPUs these
> > map to.
> > 
> > Given that I see interrupts mapped to CPUs 0-23 on one side, and 48-71
> > on the other, it looks like half of your machine gets starved, 
> 
> Seems that way.
> 
> So this is a mystery to me:
> 
> [   23.584192] picked CPU62 IRQ147
> 
> 147:          0          0          0          0          0          0
> 0          0          0          0          0          0  0          0
> 0          0          0          0          0       0          0
> 0          0          0          0 0          0          0          0
> 0          0          0     0          0          0          0
> 0          0          0          0          0          0          0
> 0          0    0          0          0          0          0
> 0          0         0          0          0          0          0
> 0   0          0          0          0          0          0
> 0        0          0          0          0          0          0  0
> 0          0          0          0          0          0       0
> 0          0          0          0          0 0          0          0
> 0          0          0          0     0          0          0
> 0          0   ITS-MSI 94404626 Edge      hisi_sas_v3_hw cq
> 
> 
> and
> 
> [   25.896728] picked CPU62 IRQ183
> 
> 183:          0          0          0          0          0          0
> 0          0          0          0          0          0  0          0
> 0          0          0          0          0       0          0
> 0          0          0          0 0          0          0          0
> 0          0          0     0          0          0          0
> 0          0          0          0          0          0          0
> 0          0    0          0          0          0          0
> 0          0         0          0          0          0          0
> 0   0          0          0          0          0          0
> 0        0          0          0          0          0          0  0
> 0          0          0          0          0          0       0
> 0          0          0          0          0 0          0          0
> 0          0          0          0     0          0          0
> 0          0   ITS-MSI 94437398 Edge      hisi_sas_v3_hw cq
> 
> 
> But mpstat reports for CPU62:
> 
> 12:44:58 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft
> %steal  %guest  %gnice   %idle
> 12:45:00 AM   62    6.54    0.00   42.99    0.00    6.54   12.15
> 0.00    0.00    6.54   25.23
> 
> I don't know what interrupts they are...

Clearly, they aren't your SAS interrupts. But the debug print do not
mean that these are the only interrupts that are targeting
CPU62. Looking at the 62nd column of /proc/interrupts should tell you
what fires (and my bet is on something like the timer).

> It's the "hisi_sas_v3_hw cq" interrupts which we're interested in.

Clearly, they aren't firing.

> and that
> > may be because no ITS targets the NUMA nodes they are part of.
> 
> So both storage controllers (which we're interested in for this test)
> are on socket #0, node #0.
> 
>  It would
> > be interesting to see what happens if you manually set the affinity
> > of the interrupts outside of the NUMA node.
> > 
> 
> Again, managed, so I don't think it's possible.

OK, we need to get back to what the actual requirements of a 'managed'
interrupt are, because there is clearly something that hasn't made it
into the core code...

	M.

-- 
Jazz is not dead, it just smells funny.

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry
2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry
2019-12-06 15:22   ` Marc Zyngier
2019-12-06 16:16     ` John Garry
2019-12-07  8:03   ` Ming Lei
2019-12-09 14:30     ` John Garry
2019-12-09 15:09       ` Hannes Reinecke
2019-12-09 15:17         ` Marc Zyngier
2019-12-09 15:25           ` Hannes Reinecke
2019-12-09 15:36             ` Marc Zyngier
2019-12-09 15:49           ` Qais Yousef
2019-12-09 15:55             ` Marc Zyngier
2019-12-10  1:43       ` Ming Lei
2019-12-10  9:45         ` John Garry
2019-12-10 10:06           ` Ming Lei
2019-12-10 10:28           ` Marc Zyngier
2019-12-10 10:59             ` John Garry
2019-12-10 11:36               ` Marc Zyngier
2019-12-10 12:05                 ` John Garry
2019-12-10 18:32                   ` Marc Zyngier
2019-12-11  9:41                     ` John Garry
2019-12-13 10:07                       ` John Garry
2019-12-13 10:31                         ` Marc Zyngier
2019-12-13 12:08                           ` John Garry
2019-12-14 10:59                             ` Marc Zyngier [this message]
2019-12-11 17:09         ` John Garry
2019-12-12 22:38           ` Ming Lei
2019-12-13 11:12             ` John Garry
2019-12-13 13:18               ` Ming Lei
2019-12-13 15:43                 ` John Garry
2019-12-13 17:12                   ` Ming Lei
2019-12-13 17:50                     ` John Garry
2019-12-14 13:56                   ` Marc Zyngier
2019-12-16 10:47                     ` John Garry
2019-12-16 11:40                       ` Marc Zyngier
2019-12-16 14:17                         ` John Garry
2019-12-16 18:00                           ` Marc Zyngier
2019-12-16 18:50                             ` John Garry
2019-12-20 11:30                               ` John Garry
2019-12-20 14:43                                 ` Marc Zyngier
2019-12-20 15:38                                   ` John Garry
2019-12-20 16:16                                     ` Marc Zyngier
2019-12-20 23:31                                     ` Ming Lei
2019-12-23  9:07                                       ` Marc Zyngier
2019-12-23 10:26                                         ` John Garry
2019-12-23 10:47                                           ` Marc Zyngier
2019-12-23 11:35                                             ` John Garry
2019-12-24  1:59                                             ` Ming Lei
2019-12-24 11:20                                               ` Marc Zyngier
2019-12-25  0:48                                                 ` Ming Lei
2020-01-02 10:35                                                   ` John Garry
2020-01-03  0:46                                                     ` Ming Lei
2020-01-03 10:41                                                       ` John Garry
2020-01-03 11:29                                                         ` Ming Lei
2020-01-03 11:50                                                           ` John Garry
2020-01-04 12:03                                                             ` Ming Lei
2020-05-30  7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=86eex7i35s.wl-maz@kernel.org \
    --to=maz@kernel.org \
    --cc=axboe@kernel.dk \
    --cc=bigeasy@linutronix.de \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=john.garry@huawei.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git