From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hannes Reinecke Subject: Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion Date: Wed, 07 Jan 2015 17:57:07 +0100 Message-ID: <54AD6563.4040603@suse.de> References: <54AD5DDD.2090808@dev.mellanox.co.il> Reply-To: open-iscsi-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <54AD5DDD.2090808-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , To: Sagi Grimberg , lsf-pc-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Cc: linux-scsi , target-devel , open-iscsi-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org, Mike Christie List-Id: linux-scsi@vger.kernel.org On 01/07/2015 05:25 PM, Sagi Grimberg wrote: > Hi everyone, >=20 > Now that scsi-mq is fully included, we need an iSCSI initiator that > would use it to achieve scalable performance. The need is even greater > for iSCSI offload devices and transports that support multiple HW > queues. As iSER maintainer I'd like to discuss the way we would choose > to implement that in iSCSI. >=20 > My measurements show that iSER initiator can scale up to ~2.1M IOPs > with multiple sessions but only ~630K IOPs with a single session where > the most significant bottleneck the (single) core processing > completions. >=20 > In the existing single connection per session model, given that command > ordering must be preserved session-wide, we end up in a serial command > execution over a single connection which is basically a single queue > model. The best fit seems to be plugging iSCSI MCS as a multi-queued > scsi LLDD. In this model, a hardware context will have a 1x1 mapping > with an iSCSI connection (TCP socket or a HW queue). >=20 > iSCSI MCS and it's role in the presence of dm-multipath layer was > discussed several times in the past decade(s). The basic need for MCS is > implementing a multi-queue data path, so perhaps we may want to avoid > doing any type link aggregation or load balancing to not overlap > dm-multipath. For example we can implement ERL=3D0 (which is basically th= e > scsi-mq ERL) and/or restrict a session to a single portal. >=20 > As I see it, the todo's are: > 1. Getting MCS to work (kernel + user-space) with ERL=3D0 and a > round-robin connection selection (per scsi command execution). > 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and > using blk-mq based queue (conn) selection. > 3. Rework iSCSI core locking scheme to avoid session-wide locking > as much as possible. > 4. Use blk-mq pre-allocation and tagging facilities. >=20 > I've recently started looking into this. I would like the community to > agree (or debate) on this scheme and also talk about implementation > with anyone who is also interested in this. >=20 Yes, that's a really good topic. I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good. Mike? Any comments? Cheers, Hannes --=20 Dr. Hannes Reinecke zSeries & Storage hare-l3A5Bk7waGM@public.gmane.org +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N=C3=BCrnberg GF: J. Hawn, J. Guild, F. Imend=C3=B6rffer, HRB 16746 (AG N=C3=BCrnberg) --=20 You received this message because you are subscribed to the Google Groups "= open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to open-iscsi+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to open-iscsi-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.