From: "Derrick, Jonathan" <jonathan.derrick@intel.com>
To: "kbusch@kernel.org" <kbusch@kernel.org>,
"ming.lei@redhat.com" <ming.lei@redhat.com>
Cc: "tglx@linutronix.de" <tglx@linutronix.de>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"hch@lst.de" <hch@lst.de>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
"axboe@kernel.dk" <axboe@kernel.dk>
Subject: Re: [PATCH V5 2/2] genirq/affinity: Spread vectors on node according to nr_cpu ratio
Date: Fri, 16 Aug 2019 16:57:32 +0000 [thread overview]
Message-ID: <3bdaa2bbd7b7c7e4cd15bee6f8df074df1fc1077.camel@intel.com> (raw)
In-Reply-To: <20190816155353.GA6883@localhost.localdomain>
[-- Attachment #1: Type: text/plain, Size: 2462 bytes --]
On Fri, 2019-08-16 at 09:53 -0600, Keith Busch wrote:
> On Thu, Aug 15, 2019 at 07:28:49PM -0700, Ming Lei wrote:
> > Now __irq_build_affinity_masks() spreads vectors evenly per node, and
> > all vectors may not be spread in case that each numa node has different
> > CPU number, then the warning in irq_build_affinity_masks() can
> > be triggered.
> >
> > Improve current spreading algorithm by assigning vectors according to
> > the ratio of node's nr_cpu to nr_remaining_cpus, meantime running the
> > assignment from smaller nodes to bigger nodes to guarantee that every
> > active node gets allocated at least one vector, then we can avoid
> > cross-node spread in normal situation.
> >
> > Meantime the reported warning can be fixed.
> >
> > Another big goodness is that the spread approach becomes more fair if
> > node has different CPU number.
> >
> > For example, on the following machine:
> > [root@ktest-01 ~]# lscpu
> > ...
> > CPU(s): 16
> > On-line CPU(s) list: 0-15
> > Thread(s) per core: 1
> > Core(s) per socket: 8
> > Socket(s): 2
> > NUMA node(s): 2
> > ...
> > NUMA node0 CPU(s): 0,1,3,5-9,11,13-15
> > NUMA node1 CPU(s): 2,4,10,12
> >
> > When driver requests to allocate 8 vectors, the following spread can
> > be got:
> > irq 31, cpu list 2,4
> > irq 32, cpu list 10,12
> > irq 33, cpu list 0-1
> > irq 34, cpu list 3,5
> > irq 35, cpu list 6-7
> > irq 36, cpu list 8-9
> > irq 37, cpu list 11,13
> > irq 38, cpu list 14-15
> >
> > Without this patch, kernel warning is triggered on above situation, and
> > allocation result was supposed to be 4 vectors for each node.
> >
> > Cc: Christoph Hellwig <hch@lst.de>
> > Cc: Keith Busch <kbusch@kernel.org>
> > Cc: linux-nvme@lists.infradead.org,
> > Cc: Jon Derrick <jonathan.derrick@intel.com>
> > Cc: Jens Axboe <axboe@kernel.dk>
> > Reported-by: Jon Derrick <jonathan.derrick@intel.com>
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
>
> I had every intention to thoroughly test this on imbalanced node
> configurations, but that's not going to happen anytime soon. It looks
> correct to me, so I'll append my review here.
>
I can only test this with 2 nodes but I have varied nr_cpus as well as
using different devices with fewer and more vectors than CPUs. Spread
looks good.
Thank you
Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
[snip]
[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3278 bytes --]
next prev parent reply other threads:[~2019-08-16 16:57 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-16 2:28 [PATCH V5 0/2] genriq/affinity: Make vectors allocation fair Ming Lei
2019-08-16 2:28 ` [PATCH V5 1/2] genirq/affinity: Improve __irq_build_affinity_masks() Ming Lei
2019-08-27 14:36 ` [tip: irq/core] " tip-bot2 for Ming Lei
2019-08-16 2:28 ` [PATCH V5 2/2] genirq/affinity: Spread vectors on node according to nr_cpu ratio Ming Lei
2019-08-16 15:53 ` Keith Busch
2019-08-16 16:28 ` Ming Lei
2019-08-16 16:57 ` Derrick, Jonathan [this message]
2019-08-27 14:36 ` [tip: irq/core] " tip-bot2 for Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3bdaa2bbd7b7c7e4cd15bee6f8df074df1fc1077.camel@intel.com \
--to=jonathan.derrick@intel.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=ming.lei@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).