LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Ming Lei <ming.lei@redhat.com>
To: Marc Zyngier <maz@kernel.org>
Cc: John Garry <john.garry@huawei.com>,
	tglx@linutronix.de, "chenxiang (M)" <chenxiang66@hisilicon.com>,
	bigeasy@linutronix.de, linux-kernel@vger.kernel.org,
	hare@suse.com, hch@lst.de, axboe@kernel.dk, bvanassche@acm.org,
	peterz@infradead.org, mingo@redhat.com,
	Zhang Yi <yi.zhang@redhat.com>
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt
Date: Tue, 24 Dec 2019 09:59:26 +0800
Message-ID: <20191224015926.GC13083@ming.t460p> (raw)
In-Reply-To: <e815b5451ea86e99d42045f7067f455a@www.loen.fr>

On Mon, Dec 23, 2019 at 10:47:07AM +0000, Marc Zyngier wrote:
> On 2019-12-23 10:26, John Garry wrote:
> > > > > > I've also managed to trigger some of them now that I have
> > > > > access to
> > > > > > a decent box with nvme storage.
> > > > > 
> > > > > I only have 2x NVMe SSDs when this occurs - I should not be
> > > > > hitting this...
> > > > > 
> > > > > Out of curiosity, have you tried
> > > > > > with the SMMU disabled? I'm wondering whether we hit some
> > > > > livelock
> > > > > > condition on unmapping buffers...
> > > > > 
> > > > > No, but I can give it a try. Doing that should lower the CPU
> > > > > usage, though,
> > > > > so maybe masks the issue - probably not.
> > > > 
> > > > Lots of CPU lockup can is performance issue if there isn't
> > > > obvious bug.
> > > > 
> > > > I am wondering if you may explain it a bit why enabling SMMU may
> > > > save
> > > > CPU a it?
> > > The other way around. mapping/unmapping IOVAs doesn't comes for
> > > free.
> > > I'm trying to find out whether the NVMe map/unmap patterns trigger
> > > something unexpected in the SMMU driver, but that's a very long
> > > shot.
> > 
> > So I tested v5.5-rc3 with and without the SMMU enabled, and without
> > the SMMU enabled I don't get the lockup.
> 
> OK, so my hunch wasn't completely off... At least we have something
> to look into.
> 
> [...]
> 
> > Obviously this is not conclusive, especially with such limited
> > testing - 5 minute runs each. The CPU load goes up when disabling the
> > SMMU, but that could be attributed to extra throughput (1183K ->
> > 1539K) loading.
> > 
> > I do notice that since we complete the NVMe request in irq context,
> > we also do the DMA unmap, i.e. talk to the SMMU, in the same context,
> > which is less than ideal.
> 
> It depends on how much overhead invalidating the TLB adds to the
> equation, but we should be able to do some tracing and find out.
> 
> > I need to finish for the Christmas break today, so can't check this
> > much further ATM.
> 
> No worries. May I suggest creating a new thread in the new year, maybe
> involving Robin and Will as well?

Zhang Yi has observed the CPU lockup issue once when running heavy IO on
single nvme drive, and please CC him if you have new patch to try.

Then looks the DMA unmap cost is too big on aarch64 if SMMU is involved.


Thanks,
Ming


  parent reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-06 14:35 [PATCH RFC 0/1] Threaded handler uses irq affinity for when the interrupt is managed John Garry
2019-12-06 14:35 ` [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt John Garry
2019-12-06 15:22   ` Marc Zyngier
2019-12-06 16:16     ` John Garry
2019-12-07  8:03   ` Ming Lei
2019-12-09 14:30     ` John Garry
2019-12-09 15:09       ` Hannes Reinecke
2019-12-09 15:17         ` Marc Zyngier
2019-12-09 15:25           ` Hannes Reinecke
2019-12-09 15:36             ` Marc Zyngier
2019-12-09 15:49           ` Qais Yousef
2019-12-09 15:55             ` Marc Zyngier
2019-12-10  1:43       ` Ming Lei
2019-12-10  9:45         ` John Garry
2019-12-10 10:06           ` Ming Lei
2019-12-10 10:28           ` Marc Zyngier
2019-12-10 10:59             ` John Garry
2019-12-10 11:36               ` Marc Zyngier
2019-12-10 12:05                 ` John Garry
2019-12-10 18:32                   ` Marc Zyngier
2019-12-11  9:41                     ` John Garry
2019-12-13 10:07                       ` John Garry
2019-12-13 10:31                         ` Marc Zyngier
2019-12-13 12:08                           ` John Garry
2019-12-14 10:59                             ` Marc Zyngier
2019-12-11 17:09         ` John Garry
2019-12-12 22:38           ` Ming Lei
2019-12-13 11:12             ` John Garry
2019-12-13 13:18               ` Ming Lei
2019-12-13 15:43                 ` John Garry
2019-12-13 17:12                   ` Ming Lei
2019-12-13 17:50                     ` John Garry
2019-12-14 13:56                   ` Marc Zyngier
2019-12-16 10:47                     ` John Garry
2019-12-16 11:40                       ` Marc Zyngier
2019-12-16 14:17                         ` John Garry
2019-12-16 18:00                           ` Marc Zyngier
2019-12-16 18:50                             ` John Garry
2019-12-20 11:30                               ` John Garry
2019-12-20 14:43                                 ` Marc Zyngier
2019-12-20 15:38                                   ` John Garry
2019-12-20 16:16                                     ` Marc Zyngier
2019-12-20 23:31                                     ` Ming Lei
2019-12-23  9:07                                       ` Marc Zyngier
2019-12-23 10:26                                         ` John Garry
2019-12-23 10:47                                           ` Marc Zyngier
2019-12-23 11:35                                             ` John Garry
2019-12-24  1:59                                             ` Ming Lei [this message]
2019-12-24 11:20                                               ` Marc Zyngier
2019-12-25  0:48                                                 ` Ming Lei
2020-01-02 10:35                                                   ` John Garry
2020-01-03  0:46                                                     ` Ming Lei
2020-01-03 10:41                                                       ` John Garry
2020-01-03 11:29                                                         ` Ming Lei
2020-01-03 11:50                                                           ` John Garry
2020-01-04 12:03                                                             ` Ming Lei
2020-05-30  7:46 ` [tip: irq/core] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs tip-bot2 for Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191224015926.GC13083@ming.t460p \
    --to=ming.lei@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=bigeasy@linutronix.de \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=john.garry@huawei.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --cc=yi.zhang@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git