All of lore.kernel.org
 help / color / mirror / Atom feed
From: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
To: Sagi Grimberg
	<sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>,
	Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org,
	sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org
Subject: Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names
Date: Sun, 26 Jul 2015 12:32:54 -0500	[thread overview]
Message-ID: <55B519C6.2030205@opengridcomputing.com> (raw)
In-Reply-To: <55B4BDE3.8040801-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>

On 7/26/2015 6:00 AM, Sagi Grimberg wrote:
> On 7/26/2015 1:43 PM, Christoph Hellwig wrote:
>> On Sun, Jul 26, 2015 at 01:08:16PM +0300, Sagi Grimberg wrote:
>>> I've given this some thought and I think we should avoid splitting
>>> logic from PI and iWARP. The reason (other than code duplication) is
>>> that currently the iser target support only up to 1MB IOs. I have some
>>> code (not done yet) to support larger IOs by using multiple
>>> registrations  per IO (with or without PI).
>>
>> Just curious: How is this going to work with iSER only having a single
>> rkey/offset/len field?
>>
>
> Good question,
>
> On the wire iser sends a single rkey, but the target is allowed to
> transfer the data however it wants to.
>
> Say that the local target HCA supports only 32 pages (128K bytes for 4K
> pages) registration and the initiator sent:
> rkey=0x1234
> address=0xffffaaaa
> length=512K
>
> The target would allocate a 512K buffer and:
> register offset 0-128K to lkey=0x1
> register offset 128K-256K to lkey=0x2
> register offset 256K-384K to lkey=0x3
> register offset 384K-512K to lkey=0x4
>
> then constructs sg_list as:
> sg_list[0] = {addr=buf, length=128K, lkey=0x1}
> sg_list[1] = {addr=buf+128K, length=128K, lkey=0x2}
> sg_list[2] = {addr=buf+256K, length=128K, lkey=0x3}
> sg_list[3] = {addr=buf+384K, length=128K, lkey=0x4}
>
> Then set rdma_read wr with:
> rdma_r_wr.sg_list=&sg_list
> rdma_r_wr.rdma.addr=0xffffaaaa
> rdma_r_wr.rdma.rkey=0x1234
>
> post_send(rdma_r_wr);
>
> Ideally, the post contains a chain of all 4 registrations and the
> rdma_read (and an opportunistic good scsi response).

Just to be clear: This example is for IB only, correct?  IW would 
require rkeys with REMOTE_WRITE and 4 read wrs.  And you're ignoring 
invalidation wrs (or read-with-inv) in the example...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

WARNING: multiple messages have this Message-ID (diff)
From: Steve Wise <swise@opengridcomputing.com>
To: Sagi Grimberg <sagig@dev.mellanox.co.il>,
	Christoph Hellwig <hch@infradead.org>
Cc: dledford@redhat.com, infinipath@intel.com, sagig@mellanox.com,
	ogerlitz@mellanox.com, roid@mellanox.com,
	linux-rdma@vger.kernel.org, eli@mellanox.com,
	target-devel@vger.kernel.org, linux-nfs@vger.kernel.org,
	bfields@fieldses.org
Subject: Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names
Date: Sun, 26 Jul 2015 12:32:54 -0500	[thread overview]
Message-ID: <55B519C6.2030205@opengridcomputing.com> (raw)
In-Reply-To: <55B4BDE3.8040801@dev.mellanox.co.il>

On 7/26/2015 6:00 AM, Sagi Grimberg wrote:
> On 7/26/2015 1:43 PM, Christoph Hellwig wrote:
>> On Sun, Jul 26, 2015 at 01:08:16PM +0300, Sagi Grimberg wrote:
>>> I've given this some thought and I think we should avoid splitting
>>> logic from PI and iWARP. The reason (other than code duplication) is
>>> that currently the iser target support only up to 1MB IOs. I have some
>>> code (not done yet) to support larger IOs by using multiple
>>> registrations  per IO (with or without PI).
>>
>> Just curious: How is this going to work with iSER only having a single
>> rkey/offset/len field?
>>
>
> Good question,
>
> On the wire iser sends a single rkey, but the target is allowed to
> transfer the data however it wants to.
>
> Say that the local target HCA supports only 32 pages (128K bytes for 4K
> pages) registration and the initiator sent:
> rkey=0x1234
> address=0xffffaaaa
> length=512K
>
> The target would allocate a 512K buffer and:
> register offset 0-128K to lkey=0x1
> register offset 128K-256K to lkey=0x2
> register offset 256K-384K to lkey=0x3
> register offset 384K-512K to lkey=0x4
>
> then constructs sg_list as:
> sg_list[0] = {addr=buf, length=128K, lkey=0x1}
> sg_list[1] = {addr=buf+128K, length=128K, lkey=0x2}
> sg_list[2] = {addr=buf+256K, length=128K, lkey=0x3}
> sg_list[3] = {addr=buf+384K, length=128K, lkey=0x4}
>
> Then set rdma_read wr with:
> rdma_r_wr.sg_list=&sg_list
> rdma_r_wr.rdma.addr=0xffffaaaa
> rdma_r_wr.rdma.rkey=0x1234
>
> post_send(rdma_r_wr);
>
> Ideally, the post contains a chain of all 4 registrations and the
> rdma_read (and an opportunistic good scsi response).

Just to be clear: This example is for IB only, correct?  IW would 
require rkeys with REMOTE_WRITE and 4 read wrs.  And you're ignoring 
invalidation wrs (or read-with-inv) in the example...


  parent reply	other threads:[~2015-07-26 17:32 UTC|newest]

Thread overview: 76+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-07-24 16:18 [PATCH V6 0/9] iSER support for iWARP Steve Wise
2015-07-24 16:18 ` Steve Wise
     [not found] ` <20150724161331.25617.8475.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:18   ` [PATCH V6 1/9] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161820.25617.63886.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:41       ` Jason Gunthorpe
2015-07-24 16:41         ` Jason Gunthorpe
2015-07-24 18:40         ` Steve Wise
2015-07-24 18:40           ` Steve Wise
2015-07-24 19:14           ` Jason Gunthorpe
2015-07-26  9:58             ` Sagi Grimberg
2015-07-26  9:57           ` Sagi Grimberg
2015-07-26  9:57             ` Sagi Grimberg
2015-07-24 16:18   ` [PATCH V6 2/9] mlx4, mlx5, mthca: Expose max_sge_rd correctly Steve Wise
2015-07-24 16:18     ` Steve Wise
2015-07-24 16:18   ` [PATCH V6 3/9] ipath,qib: " Steve Wise
2015-07-24 16:18     ` Steve Wise
2015-07-24 16:18   ` [PATCH V6 4/9] svcrdma: Use max_sge_rd for destination read depths Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161837.25617.48584.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 18:51       ` Steve Wise
2015-07-24 18:51         ` Steve Wise
2015-07-26  9:58         ` Sagi Grimberg
     [not found]           ` <55B4AF63.3040108-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 10:42             ` Christoph Hellwig
2015-07-26 10:42               ` Christoph Hellwig
2015-07-24 16:18   ` [PATCH V6 6/9] isert: Rename IO functions to more descriptive names Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161848.25617.26092.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-26 10:08       ` Sagi Grimberg
2015-07-26 10:08         ` Sagi Grimberg
     [not found]         ` <55B4B190.7070305-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 10:43           ` Christoph Hellwig
2015-07-26 10:43             ` Christoph Hellwig
     [not found]             ` <20150726104328.GB18944-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-26 11:00               ` Sagi Grimberg
2015-07-26 11:00                 ` Sagi Grimberg
2015-07-26 15:53                 ` Christoph Hellwig
2015-07-26 16:44                   ` Sagi Grimberg
     [not found]                 ` <55B4BDE3.8040801-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 17:32                   ` Steve Wise [this message]
2015-07-26 17:32                     ` Steve Wise
2015-07-26 17:40                     ` Sagi Grimberg
     [not found]                       ` <55B51B7A.8030008-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 20:15                         ` Steve Wise
2015-07-26 20:15                           ` Steve Wise
2015-07-26 20:17           ` Steve Wise
2015-07-26 20:17             ` Steve Wise
     [not found]             ` <55B5404F.5040404-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
2015-07-27 21:45               ` Steve Wise
2015-07-27 21:45                 ` Steve Wise
2015-08-03 19:32               ` Steve Wise
2015-08-03 19:32                 ` Steve Wise
2015-08-04 17:26                 ` Sagi Grimberg
     [not found]                   ` <55C0F5B7.4040100-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-08-04 17:44                     ` Steve Wise
2015-08-04 17:44                       ` Steve Wise
2015-08-05 21:23                   ` Steve Wise
2015-08-05 21:23                     ` Steve Wise
2015-08-06 15:37                     ` Sagi Grimberg
2015-08-06 15:37                       ` Sagi Grimberg
     [not found]                       ` <55C37F43.6080106-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-08-12 22:15                         ` Steve Wise
2015-08-12 22:15                           ` Steve Wise
2015-08-13 13:09                           ` Sagi Grimberg
2015-07-24 16:18 ` [PATCH V6 5/9] RDMA/isert: Limit read depth based on the device max_sge_rd capability Steve Wise
2015-07-24 16:18 ` [PATCH V6 7/9] isert: Use the device's max fastreg page list depth Steve Wise
2015-07-24 16:18 ` [PATCH V6 8/9] isert: Use local_dma_lkey whenever possible Steve Wise
     [not found]   ` <20150724161859.25617.17286.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:49     ` Jason Gunthorpe
2015-07-24 16:49       ` Jason Gunthorpe
     [not found]       ` <20150724164909.GB25480-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-24 18:41         ` Steve Wise
2015-07-24 18:41           ` Steve Wise
2015-07-24 16:19 ` [PATCH V6 9/9] isert: Support iWARP transports using FRMRs Steve Wise
2015-07-24 16:57   ` Jason Gunthorpe
2015-07-24 18:48     ` Steve Wise
2015-07-24 18:48       ` Steve Wise
2015-07-24 19:24       ` Jason Gunthorpe
2015-07-24 19:24         ` Jason Gunthorpe
2015-07-24 19:57         ` Steve Wise
2015-07-24 19:57           ` Steve Wise
2015-07-24 22:11           ` Steve Wise
2015-07-24 22:11             ` Steve Wise
2015-07-24 22:38             ` Jason Gunthorpe
2015-07-24 22:38               ` Jason Gunthorpe
     [not found]         ` <20150724192411.GC26225-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-26 10:23           ` Sagi Grimberg
2015-07-26 10:23             ` Sagi Grimberg
2015-07-27 17:07             ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=55B519C6.2030205@opengridcomputing.com \
    --to=swise-7bpotxp6k4+p2yhjcf5u+vpxobypeauw@public.gmane.org \
    --cc=bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org \
    --cc=dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
    --cc=eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \
    --cc=infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
    --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org \
    --cc=sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.