All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
To: Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Cc: Steve Wise
	<swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>,
	dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org,
	sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org,
	target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org
Subject: Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names
Date: Sun, 26 Jul 2015 14:00:51 +0300	[thread overview]
Message-ID: <55B4BDE3.8040801@dev.mellanox.co.il> (raw)
In-Reply-To: <20150726104328.GB18944-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>

On 7/26/2015 1:43 PM, Christoph Hellwig wrote:
> On Sun, Jul 26, 2015 at 01:08:16PM +0300, Sagi Grimberg wrote:
>> I've given this some thought and I think we should avoid splitting
>> logic from PI and iWARP. The reason (other than code duplication) is
>> that currently the iser target support only up to 1MB IOs. I have some
>> code (not done yet) to support larger IOs by using multiple
>> registrations  per IO (with or without PI).
>
> Just curious: How is this going to work with iSER only having a single
> rkey/offset/len field?
>

Good question,

On the wire iser sends a single rkey, but the target is allowed to
transfer the data however it wants to.

Say that the local target HCA supports only 32 pages (128K bytes for 4K
pages) registration and the initiator sent:
rkey=0x1234
address=0xffffaaaa
length=512K

The target would allocate a 512K buffer and:
register offset 0-128K to lkey=0x1
register offset 128K-256K to lkey=0x2
register offset 256K-384K to lkey=0x3
register offset 384K-512K to lkey=0x4

then constructs sg_list as:
sg_list[0] = {addr=buf, length=128K, lkey=0x1}
sg_list[1] = {addr=buf+128K, length=128K, lkey=0x2}
sg_list[2] = {addr=buf+256K, length=128K, lkey=0x3}
sg_list[3] = {addr=buf+384K, length=128K, lkey=0x4}

Then set rdma_read wr with:
rdma_r_wr.sg_list=&sg_list
rdma_r_wr.rdma.addr=0xffffaaaa
rdma_r_wr.rdma.rkey=0x1234

post_send(rdma_r_wr);

Ideally, the post contains a chain of all 4 registrations and the
rdma_read (and an opportunistic good scsi response).
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

WARNING: multiple messages have this Message-ID (diff)
From: Sagi Grimberg <sagig@dev.mellanox.co.il>
To: Christoph Hellwig <hch@infradead.org>
Cc: Steve Wise <swise@opengridcomputing.com>,
	dledford@redhat.com, infinipath@intel.com, sagig@mellanox.com,
	ogerlitz@mellanox.com, roid@mellanox.com,
	linux-rdma@vger.kernel.org, eli@mellanox.com,
	target-devel@vger.kernel.org, linux-nfs@vger.kernel.org,
	bfields@fieldses.org
Subject: Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names
Date: Sun, 26 Jul 2015 14:00:51 +0300	[thread overview]
Message-ID: <55B4BDE3.8040801@dev.mellanox.co.il> (raw)
In-Reply-To: <20150726104328.GB18944@infradead.org>

On 7/26/2015 1:43 PM, Christoph Hellwig wrote:
> On Sun, Jul 26, 2015 at 01:08:16PM +0300, Sagi Grimberg wrote:
>> I've given this some thought and I think we should avoid splitting
>> logic from PI and iWARP. The reason (other than code duplication) is
>> that currently the iser target support only up to 1MB IOs. I have some
>> code (not done yet) to support larger IOs by using multiple
>> registrations  per IO (with or without PI).
>
> Just curious: How is this going to work with iSER only having a single
> rkey/offset/len field?
>

Good question,

On the wire iser sends a single rkey, but the target is allowed to
transfer the data however it wants to.

Say that the local target HCA supports only 32 pages (128K bytes for 4K
pages) registration and the initiator sent:
rkey=0x1234
address=0xffffaaaa
length=512K

The target would allocate a 512K buffer and:
register offset 0-128K to lkey=0x1
register offset 128K-256K to lkey=0x2
register offset 256K-384K to lkey=0x3
register offset 384K-512K to lkey=0x4

then constructs sg_list as:
sg_list[0] = {addr=buf, length=128K, lkey=0x1}
sg_list[1] = {addr=buf+128K, length=128K, lkey=0x2}
sg_list[2] = {addr=buf+256K, length=128K, lkey=0x3}
sg_list[3] = {addr=buf+384K, length=128K, lkey=0x4}

Then set rdma_read wr with:
rdma_r_wr.sg_list=&sg_list
rdma_r_wr.rdma.addr=0xffffaaaa
rdma_r_wr.rdma.rkey=0x1234

post_send(rdma_r_wr);

Ideally, the post contains a chain of all 4 registrations and the
rdma_read (and an opportunistic good scsi response).

  parent reply	other threads:[~2015-07-26 11:00 UTC|newest]

Thread overview: 76+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-07-24 16:18 [PATCH V6 0/9] iSER support for iWARP Steve Wise
2015-07-24 16:18 ` Steve Wise
     [not found] ` <20150724161331.25617.8475.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:18   ` [PATCH V6 1/9] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161820.25617.63886.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:41       ` Jason Gunthorpe
2015-07-24 16:41         ` Jason Gunthorpe
2015-07-24 18:40         ` Steve Wise
2015-07-24 18:40           ` Steve Wise
2015-07-24 19:14           ` Jason Gunthorpe
2015-07-26  9:58             ` Sagi Grimberg
2015-07-26  9:57           ` Sagi Grimberg
2015-07-26  9:57             ` Sagi Grimberg
2015-07-24 16:18   ` [PATCH V6 2/9] mlx4, mlx5, mthca: Expose max_sge_rd correctly Steve Wise
2015-07-24 16:18     ` Steve Wise
2015-07-24 16:18   ` [PATCH V6 3/9] ipath,qib: " Steve Wise
2015-07-24 16:18     ` Steve Wise
2015-07-24 16:18   ` [PATCH V6 4/9] svcrdma: Use max_sge_rd for destination read depths Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161837.25617.48584.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 18:51       ` Steve Wise
2015-07-24 18:51         ` Steve Wise
2015-07-26  9:58         ` Sagi Grimberg
     [not found]           ` <55B4AF63.3040108-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 10:42             ` Christoph Hellwig
2015-07-26 10:42               ` Christoph Hellwig
2015-07-24 16:18   ` [PATCH V6 6/9] isert: Rename IO functions to more descriptive names Steve Wise
2015-07-24 16:18     ` Steve Wise
     [not found]     ` <20150724161848.25617.26092.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-26 10:08       ` Sagi Grimberg
2015-07-26 10:08         ` Sagi Grimberg
     [not found]         ` <55B4B190.7070305-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 10:43           ` Christoph Hellwig
2015-07-26 10:43             ` Christoph Hellwig
     [not found]             ` <20150726104328.GB18944-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-26 11:00               ` Sagi Grimberg [this message]
2015-07-26 11:00                 ` Sagi Grimberg
2015-07-26 15:53                 ` Christoph Hellwig
2015-07-26 16:44                   ` Sagi Grimberg
     [not found]                 ` <55B4BDE3.8040801-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 17:32                   ` Steve Wise
2015-07-26 17:32                     ` Steve Wise
2015-07-26 17:40                     ` Sagi Grimberg
     [not found]                       ` <55B51B7A.8030008-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-26 20:15                         ` Steve Wise
2015-07-26 20:15                           ` Steve Wise
2015-07-26 20:17           ` Steve Wise
2015-07-26 20:17             ` Steve Wise
     [not found]             ` <55B5404F.5040404-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
2015-07-27 21:45               ` Steve Wise
2015-07-27 21:45                 ` Steve Wise
2015-08-03 19:32               ` Steve Wise
2015-08-03 19:32                 ` Steve Wise
2015-08-04 17:26                 ` Sagi Grimberg
     [not found]                   ` <55C0F5B7.4040100-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-08-04 17:44                     ` Steve Wise
2015-08-04 17:44                       ` Steve Wise
2015-08-05 21:23                   ` Steve Wise
2015-08-05 21:23                     ` Steve Wise
2015-08-06 15:37                     ` Sagi Grimberg
2015-08-06 15:37                       ` Sagi Grimberg
     [not found]                       ` <55C37F43.6080106-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-08-12 22:15                         ` Steve Wise
2015-08-12 22:15                           ` Steve Wise
2015-08-13 13:09                           ` Sagi Grimberg
2015-07-24 16:18 ` [PATCH V6 5/9] RDMA/isert: Limit read depth based on the device max_sge_rd capability Steve Wise
2015-07-24 16:18 ` [PATCH V6 7/9] isert: Use the device's max fastreg page list depth Steve Wise
2015-07-24 16:18 ` [PATCH V6 8/9] isert: Use local_dma_lkey whenever possible Steve Wise
     [not found]   ` <20150724161859.25617.17286.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-24 16:49     ` Jason Gunthorpe
2015-07-24 16:49       ` Jason Gunthorpe
     [not found]       ` <20150724164909.GB25480-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-24 18:41         ` Steve Wise
2015-07-24 18:41           ` Steve Wise
2015-07-24 16:19 ` [PATCH V6 9/9] isert: Support iWARP transports using FRMRs Steve Wise
2015-07-24 16:57   ` Jason Gunthorpe
2015-07-24 18:48     ` Steve Wise
2015-07-24 18:48       ` Steve Wise
2015-07-24 19:24       ` Jason Gunthorpe
2015-07-24 19:24         ` Jason Gunthorpe
2015-07-24 19:57         ` Steve Wise
2015-07-24 19:57           ` Steve Wise
2015-07-24 22:11           ` Steve Wise
2015-07-24 22:11             ` Steve Wise
2015-07-24 22:38             ` Jason Gunthorpe
2015-07-24 22:38               ` Jason Gunthorpe
     [not found]         ` <20150724192411.GC26225-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-26 10:23           ` Sagi Grimberg
2015-07-26 10:23             ` Sagi Grimberg
2015-07-27 17:07             ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=55B4BDE3.8040801@dev.mellanox.co.il \
    --to=sagig-ldsdmyg8hgv8yrgs2mwiifqbs+8scbdb@public.gmane.org \
    --cc=bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org \
    --cc=dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
    --cc=eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \
    --cc=infinipath-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
    --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \
    --cc=swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org \
    --cc=target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.