From: Ming Lei <ming.lei@redhat.com>
To: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
Christoph Hellwig <hch@infradead.org>,
Mike Snitzer <snitzer@redhat.com>,
linux-scsi@vger.kernel.org, Hannes Reinecke <hare@suse.de>,
Arun Easi <arun.easi@cavium.com>, Omar Sandoval <osandov@fb.com>,
"Martin K . Petersen" <martin.petersen@oracle.com>,
James Bottomley <james.bottomley@hansenpartnership.com>,
Christoph Hellwig <hch@lst.de>,
Don Brace <don.brace@microsemi.com>,
Peter Rivera <peter.rivera@broadcom.com>,
Laurence Oberman <loberman@redhat.com>
Subject: Re: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance via .host_tagset
Date: Thu, 8 Mar 2018 19:06:26 +0800 [thread overview]
Message-ID: <20180308110625.GC31062@ming.t460p> (raw)
In-Reply-To: <45ba4cc7328786fd5b536362b686b5a3@mail.gmail.com>
On Thu, Mar 08, 2018 at 03:34:31PM +0530, Kashyap Desai wrote:
> > -----Original Message-----
> > From: Ming Lei [mailto:ming.lei@redhat.com]
> > Sent: Thursday, March 8, 2018 6:46 AM
> > To: Kashyap Desai
> > Cc: Jens Axboe; linux-block@vger.kernel.org; Christoph Hellwig; Mike
> Snitzer;
> > linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar Sandoval;
> > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace;
> Peter
> > Rivera; Laurence Oberman
> > Subject: Re: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance
> via
> > .host_tagset
> >
> > On Wed, Mar 07, 2018 at 10:58:34PM +0530, Kashyap Desai wrote:
> > > > >
> > > > > Also one observation using V3 series patch. I am seeing below
> > > > > Affinity mapping whereas I have only 72 logical CPUs. It means we
> > > > > are really not going to use all reply queues.
> > > > > e.a If I bind fio jobs on CPU 18-20, I am seeing only one reply
> > > > > queue is used and that may lead to performance drop as well.
> > > >
> > > > If the mapping is in such shape, I guess it should be quite
> > > > difficult to
> > > figure out
> > > > one perfect way to solve this situation because one reply queue has
> > > > to
> > > handle
> > > > IOs submitted from 4~5 CPUs at average.
> > >
> > > 4.15.0-rc1 kernel has below mapping - I am not sure which commit id in
> "
> > > linux_4.16-rc-host-tags-v3.2" is changing the mapping of IRQ to CPU.
> > > It
> >
> > I guess the mapping you posted is read from /proc/irq/126/smp_affinity.
> >
> > If yes, no any patch in linux_4.16-rc-host-tags-v3.2 should change IRQ
> affinity
> > code, which is done in irq_create_affinity_masks(), as you saw, no any
> patch
> > in linux_4.16-rc-host-tags-v3.2 touches that code.
> >
> > Could you simply apply the patches in linux_4.16-rc-host-tags-v3.2
> against
> > 4.15-rc1 kernel and see any difference?
> >
> > > will be really good if we can fall back to below mapping once again.
> > > Current repo linux_4.16-rc-host-tags-v3.2 is giving lots of random
> > > mapping of CPU - MSIx. And that will be problematic in performance
> run.
> > >
> > > As I posted earlier, latest repo will only allow us to use *18* reply
> >
> > Looks not see this report before, could you share us how you conclude
> that?
> > The only patch changing reply queue is the following one:
> >
> > https://marc.info/?l=linux-block&m=151972611911593&w=2
> >
> > But not see any issue in this patch yet, can you recover to 72 reply
> queues
> > after reverting the patch in above link?
> Ming -
>
> While testing, my system went bad. I debug further and understood that
> affinity mapping was changed due to below commit -
> 84676c1f21e8ff54befe985f4f14dc1edc10046b
>
> [PATCH] genirq/affinity: assign vectors to all possible CPUs
>
> Because of above change, we end up using very less reply queue. Many reply
> queues on my setup was mapped to offline/not-available CPUs. This may be
> primary contributing to odd performance impact and it may not be truly due
> to V3/V4 patch series.
Seems a good news, :-)
>
> I am planning to check your V3 and V4 series after removing above commit
> ID (for performance impact.).
You can run your test on a server in which all CPUs are kept as online
for avoiding this issue.
Or you can apply the following patchset for avoiding this issue:
https://marc.info/?l=linux-block&m=152050646332092&w=2
>
> It is good if we spread possible CPUs (instead of online cpus) to all irq
> vectors considering - We should have at least *one* online CPU mapped to
> the vector.
Right, that is exactly what the above patchset does.
Thanks,
Ming
next prev parent reply other threads:[~2018-03-08 11:06 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-27 10:07 [PATCH V3 0/8] blk-mq & scsi: fix reply queue selection and improve host wide tagset Ming Lei
2018-02-27 10:07 ` [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue Ming Lei
2018-03-01 16:18 ` Don Brace
2018-03-01 19:01 ` Laurence Oberman
2018-03-01 21:19 ` Laurence Oberman
2018-03-02 2:16 ` Ming Lei
2018-03-02 14:09 ` Laurence Oberman
2018-03-02 15:03 ` Don Brace
2018-03-02 21:53 ` Laurence Oberman
2018-03-05 2:07 ` Ming Lei
2018-03-06 17:55 ` Martin K. Petersen
2018-03-06 19:24 ` Martin K. Petersen
2018-03-07 0:00 ` Ming Lei
2018-03-07 3:14 ` Martin K. Petersen
2018-03-07 14:11 ` Laurence Oberman
2018-03-08 13:42 ` Ming Lei
2018-03-08 20:56 ` Laurence Oberman
2018-03-05 7:23 ` Kashyap Desai
2018-03-05 14:35 ` Don Brace
2018-03-05 15:19 ` Mike Snitzer
2018-03-02 0:47 ` Ming Lei
2018-03-08 7:50 ` Christoph Hellwig
2018-03-08 8:15 ` Ming Lei
2018-03-08 8:41 ` Hannes Reinecke
2018-03-08 9:19 ` Ming Lei
2018-03-08 15:31 ` Bart Van Assche
2018-02-27 10:07 ` [PATCH V3 2/8] scsi: megaraid_sas: " Ming Lei
2018-02-27 10:07 ` [PATCH V3 3/8] blk-mq: introduce 'start_tag' field to 'struct blk_mq_tags' Ming Lei
2018-03-08 7:51 ` Christoph Hellwig
2018-02-27 10:07 ` [PATCH V3 4/8] blk-mq: introduce BLK_MQ_F_HOST_TAGS Ming Lei
2018-03-08 7:52 ` Christoph Hellwig
2018-03-08 9:35 ` Ming Lei
2018-02-27 10:07 ` [PATCH V3 5/8] scsi: Add template flag 'host_tagset' Ming Lei
2018-02-27 10:07 ` [PATCH V3 6/8] block: null_blk: introduce module parameter of 'g_host_tags' Ming Lei
2018-02-27 10:07 ` [PATCH V3 7/8] scsi: hpsa: improve scsi_mq performance via .host_tagset Ming Lei
2018-03-08 7:54 ` Christoph Hellwig
2018-03-08 10:59 ` Ming Lei
2018-02-27 10:07 ` [PATCH V3 8/8] scsi: megaraid: " Ming Lei
2018-02-28 14:58 ` Kashyap Desai
2018-02-28 15:21 ` Ming Lei
2018-02-28 16:22 ` Laurence Oberman
2018-03-01 5:24 ` Kashyap Desai
2018-03-01 7:58 ` Ming Lei
2018-03-07 5:27 ` Ming Lei
2018-03-07 15:01 ` Kashyap Desai
2018-03-07 16:05 ` Ming Lei
2018-03-07 17:28 ` Kashyap Desai
2018-03-08 1:15 ` Ming Lei
2018-03-08 10:04 ` Kashyap Desai
2018-03-08 11:06 ` Ming Lei [this message]
2018-03-08 11:23 ` Ming Lei
2018-03-09 6:56 ` Kashyap Desai
2018-03-09 8:13 ` Ming Lei
2018-03-01 21:46 ` [PATCH V3 0/8] blk-mq & scsi: fix reply queue selection and improve host wide tagset Laurence Oberman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180308110625.GC31062@ming.t460p \
--to=ming.lei@redhat.com \
--cc=arun.easi@cavium.com \
--cc=axboe@kernel.dk \
--cc=don.brace@microsemi.com \
--cc=hare@suse.de \
--cc=hch@infradead.org \
--cc=hch@lst.de \
--cc=james.bottomley@hansenpartnership.com \
--cc=kashyap.desai@broadcom.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=loberman@redhat.com \
--cc=martin.petersen@oracle.com \
--cc=osandov@fb.com \
--cc=peter.rivera@broadcom.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).