From: Kashyap Desai <kashyap.desai@broadcom.com>
To: Laurence Oberman <loberman@redhat.com>, Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
Christoph Hellwig <hch@infradead.org>,
Mike Snitzer <snitzer@redhat.com>,
linux-scsi@vger.kernel.org, Hannes Reinecke <hare@suse.de>,
Arun Easi <arun.easi@cavium.com>, Omar Sandoval <osandov@fb.com>,
"Martin K . Petersen" <martin.petersen@oracle.com>,
James Bottomley <james.bottomley@hansenpartnership.com>,
Christoph Hellwig <hch@lst.de>,
Don Brace <don.brace@microsemi.com>,
Peter Rivera <peter.rivera@broadcom.com>
Subject: RE: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance via .host_tagset
Date: Thu, 1 Mar 2018 10:54:17 +0530 [thread overview]
Message-ID: <9ccb507aaa8fcec81fcf83d0ee2ba4c2@mail.gmail.com> (raw)
In-Reply-To: <1519834930.7327.1.camel@redhat.com>
> -----Original Message-----
> From: Laurence Oberman [mailto:loberman@redhat.com]
> Sent: Wednesday, February 28, 2018 9:52 PM
> To: Ming Lei; Kashyap Desai
> Cc: Jens Axboe; linux-block@vger.kernel.org; Christoph Hellwig; Mike
> Snitzer;
> linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar Sandoval;
> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter
> Rivera
> Subject: Re: [PATCH V3 8/8] scsi: megaraid: improve scsi_mq performance
> via
> .host_tagset
>
> On Wed, 2018-02-28 at 23:21 +0800, Ming Lei wrote:
> > On Wed, Feb 28, 2018 at 08:28:48PM +0530, Kashyap Desai wrote:
> > > Ming -
> > >
> > > Quick testing on my setup - Performance slightly degraded (4-5%
> > > drop)for megaraid_sas driver with this patch. (From 1610K IOPS it
> > > goes to
> > > 1544K)
> > > I confirm that after applying this patch, we have #queue = #numa
> > > node.
> > >
> > > ls -l
> > > /sys/devices/pci0000:80/0000:80:02.0/0000:83:00.0/host10/target10:2
> > > :23/10:
> > > 2:23:0/block/sdy/mq
> > > total 0
> > > drwxr-xr-x. 18 root root 0 Feb 28 09:53 0 drwxr-xr-x. 18 root root 0
> > > Feb 28 09:53 1
> >
> > OK, thanks for your test.
> >
> > As I mentioned to you, this patch should have improved performance on
> > megaraid_sas, but the current slight degrade might be caused by
> > scsi_host_queue_ready() in scsi_queue_rq(), I guess.
> >
> > With .host_tagset enabled and use per-numa-node hw queue, request can
> > be queued to lld more frequently/quick than single queue, then the
> > cost of
> > atomic_inc_return(&host->host_busy) may be increased much meantime,
> > think about millions of such operations, and finally slight IOPS drop
> > is observed when the hw queue depth becomes half of .can_queue.
> >
> > >
> > >
> > > I would suggest to skip megaraid_sas driver changes using
> > > shared_tagset until and unless there is obvious gain. If overall
> > > interface of using shared_tagset is commit in kernel tree, we will
> > > investigate (megaraid_sas
> > > driver) in future about real benefit of using it.
> >
> > I'd suggest to not merge it until it is proved that performance can be
> > improved in real device.
Noted.
> >
> > I will try to work to remove the expensive atomic_inc_return(&host-
> > >host_busy)
> > from scsi_queue_rq(), since it isn't needed for SCSI_MQ, once it is
> > done, will ask you to test again.
Ming - Do you mean removing host_busy stats from scsi_queue_rq() will still
provide correct value in host_busy whenever IO reach to LLD ?
> >
> >
> > Thanks,
> > Ming
>
> I will test this here as well
> I just put the Megaraid card in to my system here
>
> Kashyap, do you have ssd's on the back-end and are you you using jbods or
> virtual devices. Let me have your config.
> I only have 6G sas shelves though.
Laurence -
I am using 12 SSD drives in JBOD mode OR single drive R0 mode. Single SSD
is capable of ~138K IOPS (4K RR).
With all 12 SSDs performance scale linearly and goes upto ~1610K IOPS.
I think if you have 6G SAS fully loaded, you may need more number of drives
to reach 1600K IOPs (sequential load with nomerges=2 on HDD is required to
avoid IO merge at block layer.)
SSD model I am using is - HGST - " HUSMH8020BSS200"
Here is lscpu output of my setup -
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 1726.217
BogoMIPS: 4199.37
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
>
> Regards
> Laurence
next prev parent reply other threads:[~2018-03-01 5:24 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-27 10:07 [PATCH V3 0/8] blk-mq & scsi: fix reply queue selection and improve host wide tagset Ming Lei
2018-02-27 10:07 ` [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue Ming Lei
2018-03-01 16:18 ` Don Brace
2018-03-01 19:01 ` Laurence Oberman
2018-03-01 21:19 ` Laurence Oberman
2018-03-02 2:16 ` Ming Lei
2018-03-02 14:09 ` Laurence Oberman
2018-03-02 15:03 ` Don Brace
2018-03-02 21:53 ` Laurence Oberman
2018-03-05 2:07 ` Ming Lei
2018-03-06 17:55 ` Martin K. Petersen
2018-03-06 19:24 ` Martin K. Petersen
2018-03-07 0:00 ` Ming Lei
2018-03-07 3:14 ` Martin K. Petersen
2018-03-07 14:11 ` Laurence Oberman
2018-03-08 13:42 ` Ming Lei
2018-03-08 20:56 ` Laurence Oberman
2018-03-05 7:23 ` Kashyap Desai
2018-03-05 14:35 ` Don Brace
2018-03-05 15:19 ` Mike Snitzer
2018-03-02 0:47 ` Ming Lei
2018-03-08 7:50 ` Christoph Hellwig
2018-03-08 8:15 ` Ming Lei
2018-03-08 8:41 ` Hannes Reinecke
2018-03-08 9:19 ` Ming Lei
2018-03-08 15:31 ` Bart Van Assche
2018-02-27 10:07 ` [PATCH V3 2/8] scsi: megaraid_sas: " Ming Lei
2018-02-27 10:07 ` [PATCH V3 3/8] blk-mq: introduce 'start_tag' field to 'struct blk_mq_tags' Ming Lei
2018-03-08 7:51 ` Christoph Hellwig
2018-02-27 10:07 ` [PATCH V3 4/8] blk-mq: introduce BLK_MQ_F_HOST_TAGS Ming Lei
2018-03-08 7:52 ` Christoph Hellwig
2018-03-08 9:35 ` Ming Lei
2018-02-27 10:07 ` [PATCH V3 5/8] scsi: Add template flag 'host_tagset' Ming Lei
2018-02-27 10:07 ` [PATCH V3 6/8] block: null_blk: introduce module parameter of 'g_host_tags' Ming Lei
2018-02-27 10:07 ` [PATCH V3 7/8] scsi: hpsa: improve scsi_mq performance via .host_tagset Ming Lei
2018-03-08 7:54 ` Christoph Hellwig
2018-03-08 10:59 ` Ming Lei
2018-02-27 10:07 ` [PATCH V3 8/8] scsi: megaraid: " Ming Lei
2018-02-28 14:58 ` Kashyap Desai
2018-02-28 15:21 ` Ming Lei
2018-02-28 16:22 ` Laurence Oberman
2018-03-01 5:24 ` Kashyap Desai [this message]
2018-03-01 7:58 ` Ming Lei
2018-03-07 5:27 ` Ming Lei
2018-03-07 15:01 ` Kashyap Desai
2018-03-07 16:05 ` Ming Lei
2018-03-07 17:28 ` Kashyap Desai
2018-03-08 1:15 ` Ming Lei
2018-03-08 10:04 ` Kashyap Desai
2018-03-08 11:06 ` Ming Lei
2018-03-08 11:23 ` Ming Lei
2018-03-09 6:56 ` Kashyap Desai
2018-03-09 8:13 ` Ming Lei
2018-03-01 21:46 ` [PATCH V3 0/8] blk-mq & scsi: fix reply queue selection and improve host wide tagset Laurence Oberman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9ccb507aaa8fcec81fcf83d0ee2ba4c2@mail.gmail.com \
--to=kashyap.desai@broadcom.com \
--cc=arun.easi@cavium.com \
--cc=axboe@kernel.dk \
--cc=don.brace@microsemi.com \
--cc=hare@suse.de \
--cc=hch@infradead.org \
--cc=hch@lst.de \
--cc=james.bottomley@hansenpartnership.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=loberman@redhat.com \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=peter.rivera@broadcom.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).