* [PATCH for-next 01/10] IB/iser: Fix module init not cleaning up on error flow
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
[not found] ` <1447691861-3796-2-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-17 8:56 ` Christoph Hellwig
2015-11-16 16:37 ` [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr Sagi Grimberg
` (7 subsequent siblings)
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Roi Dayan
From: Roi Dayan <roid@mellanox.com>
destroy workqueue on transport register error
release kmem cache on workqueue alloc error
Signed-off-by: Roi Dayan <roid@mellanox.com>
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/iser/iscsi_iser.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c
index 8017f3a049fb..f1ddad34b001 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.c
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.c
@@ -1063,7 +1063,8 @@ static int __init iser_init(void)
release_wq = alloc_workqueue("release workqueue", 0, 0);
if (!release_wq) {
iser_err("failed to allocate release workqueue\n");
- return -ENOMEM;
+ err = -ENOMEM;
+ goto err_alloc_wq;
}
iscsi_iser_scsi_transport = iscsi_register_transport(
@@ -1071,12 +1072,14 @@ static int __init iser_init(void)
if (!iscsi_iser_scsi_transport) {
iser_err("iscsi_register_transport failed\n");
err = -EINVAL;
- goto register_transport_failure;
+ goto err_reg;
}
return 0;
-register_transport_failure:
+err_reg:
+ destroy_workqueue(release_wq);
+err_alloc_wq:
kmem_cache_destroy(ig.desc_cache);
return err;
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
2015-11-16 16:37 ` [PATCH for-next 01/10] IB/iser: Fix module init not cleaning up on error flow Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
[not found] ` <1447691861-3796-3-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-17 8:57 ` Christoph Hellwig
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (6 subsequent siblings)
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel; +Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise
This is a pre-step before adding remote invalidate
support. Also, every ULP should prefer fastreg over
fmr.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/iser/iser_memory.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c
index ea765fb9664d..1103aa6e8d00 100644
--- a/drivers/infiniband/ulp/iser/iser_memory.c
+++ b/drivers/infiniband/ulp/iser/iser_memory.c
@@ -71,15 +71,14 @@ int iser_assign_reg_ops(struct iser_device *device)
{
struct ib_device_attr *dev_attr = &device->dev_attr;
- /* Assign function handles - based on FMR support */
+ if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
+ iser_info("FastReg supported, using FastReg for registration\n");
+ device->reg_ops = &fastreg_ops;
+ } else
if (device->ib_device->alloc_fmr && device->ib_device->dealloc_fmr &&
device->ib_device->map_phys_fmr && device->ib_device->unmap_fmr) {
iser_info("FMR supported, using FMR for registration\n");
device->reg_ops = &fmr_ops;
- } else
- if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
- iser_info("FastReg supported, using FastReg for registration\n");
- device->reg_ops = &fastreg_ops;
} else {
iser_err("IB device does not support FMRs nor FastRegs, can't register memory\n");
return -1;
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
[parent not found: <1447691861-3796-3-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
[not found] ` <1447691861-3796-3-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-17 7:41 ` Or Gerlitz
2015-11-17 9:35 ` Sagi Grimberg
0 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 7:41 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Steve Wise
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> This is a pre-step before adding remote invalidate support.
Wouldn't that introduce performance regression on ConnectX3 devices?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-17 7:41 ` Or Gerlitz
@ 2015-11-17 9:35 ` Sagi Grimberg
2015-11-17 10:09 ` Or Gerlitz
0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-17 9:35 UTC (permalink / raw)
To: Or Gerlitz, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise
> On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
>> This is a pre-step before adding remote invalidate support.
>
> Wouldn't that introduce performance regression on ConnectX3 devices?
With remote invalidate it shouldn't make much of a difference. Plus,
I'd really like us to start phasing out from FMRs...
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-17 9:35 ` Sagi Grimberg
@ 2015-11-17 10:09 ` Or Gerlitz
2015-11-17 10:46 ` Sagi Grimberg
0 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 10:09 UTC (permalink / raw)
To: Sagi Grimberg, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise
On 11/17/2015 11:35 AM, Sagi Grimberg wrote:
>> On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
>>> This is a pre-step before adding remote invalidate support.
>>
>> Wouldn't that introduce performance regression on ConnectX3 devices?
>
> With remote invalidate it shouldn't make much of a difference.
Why? the invalidate is just one part of the story, we are doing a
mapping on IO submission
and CX3 has strong ordering on FRWRs, right?
> Plus, I'd really like us to start phasing out from FMRs...
We should make sure not to introduce performance regression for HW which
has
such a big existing install base and is well selling in 2015/16
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-17 10:09 ` Or Gerlitz
@ 2015-11-17 10:46 ` Sagi Grimberg
0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-17 10:46 UTC (permalink / raw)
To: Or Gerlitz, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise
> Why? the invalidate is just one part of the story, we are doing a
> mapping on IO submission
> and CX3 has strong ordering on FRWRs, right?
Yes, this is correct.
We'll test on CX3 to see if this introduces a regression.
> We should make sure not to introduce performance regression for HW which
> has such a big existing install base and is well selling in 2015/16
You are right.
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-16 16:37 ` [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr Sagi Grimberg
[not found] ` <1447691861-3796-3-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-17 8:57 ` Christoph Hellwig
2015-11-17 9:35 ` Sagi Grimberg
1 sibling, 1 reply; 74+ messages in thread
From: Christoph Hellwig @ 2015-11-17 8:57 UTC (permalink / raw)
To: Sagi Grimberg
Cc: linux-rdma, target-devel, Nicholas A. Bellinger, Or Gerlitz, Steve Wise
> + if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
> + iser_info("FastReg supported, using FastReg for registration\n");
> + device->reg_ops = &fastreg_ops;
> + } else
> if (device->ib_device->alloc_fmr && device->ib_device->dealloc_fmr &&
Should an 'else if'. If the line gets to long it might be time to add a
rdma_cap_fmr() helper to ib_verbs.h
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 02/10] IB/iser: Default to fastreg instead of fmr
2015-11-17 8:57 ` Christoph Hellwig
@ 2015-11-17 9:35 ` Sagi Grimberg
0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-17 9:35 UTC (permalink / raw)
To: Christoph Hellwig, Sagi Grimberg
Cc: linux-rdma, target-devel, Nicholas A. Bellinger, Or Gerlitz, Steve Wise
On 17/11/2015 10:57, Christoph Hellwig wrote:
>> + if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
>> + iser_info("FastReg supported, using FastReg for registration\n");
>> + device->reg_ops = &fastreg_ops;
>> + } else
>> if (device->ib_device->alloc_fmr && device->ib_device->dealloc_fmr &&
>
> Should an 'else if'. If the line gets to long it might be time to add a
> rdma_cap_fmr() helper to ib_verbs.h
OK, I will.
^ permalink raw reply [flat|nested] 74+ messages in thread
[parent not found: <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* [PATCH for-next 03/10] IB/iser: Don't register memory for all immediatedata writes
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-16 16:37 ` Sagi Grimberg
2015-11-17 7:47 ` Or Gerlitz
2015-11-16 16:37 ` [PATCH for-next 07/10] iser-target: Declare correct flags when accepting a connection Sagi Grimberg
` (2 subsequent siblings)
3 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Jenny Derzhavetz
From: Jenny Derzhavetz <jennyf-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
When all the task data is sent as immeidatedata, we are
allowed to use the local_dma_lkey as it is not sent to
the wire.
Signed-off-by: Jenny Derzhavetz <jennyf-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/ulp/iser/iscsi_iser.h | 3 ++-
drivers/infiniband/ulp/iser/iser_initiator.c | 5 +++--
drivers/infiniband/ulp/iser/iser_memory.c | 9 +++++----
3 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h b/drivers/infiniband/ulp/iser/iscsi_iser.h
index 8a5998e6a407..fb7fa7aa113c 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.h
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.h
@@ -651,7 +651,8 @@ void iser_finalize_rdma_unaligned_sg(struct iscsi_iser_task *iser_task,
enum iser_data_dir cmd_dir);
int iser_reg_rdma_mem(struct iscsi_iser_task *task,
- enum iser_data_dir dir);
+ enum iser_data_dir dir,
+ bool all_imm);
void iser_unreg_rdma_mem(struct iscsi_iser_task *task,
enum iser_data_dir dir);
diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c
index ffd00c420729..07bf26427ee7 100644
--- a/drivers/infiniband/ulp/iser/iser_initiator.c
+++ b/drivers/infiniband/ulp/iser/iser_initiator.c
@@ -72,7 +72,7 @@ static int iser_prepare_read_cmd(struct iscsi_task *task)
return err;
}
- err = iser_reg_rdma_mem(iser_task, ISER_DIR_IN);
+ err = iser_reg_rdma_mem(iser_task, ISER_DIR_IN, false);
if (err) {
iser_err("Failed to set up Data-IN RDMA\n");
return err;
@@ -126,7 +126,8 @@ iser_prepare_write_cmd(struct iscsi_task *task,
return err;
}
- err = iser_reg_rdma_mem(iser_task, ISER_DIR_OUT);
+ err = iser_reg_rdma_mem(iser_task, ISER_DIR_OUT,
+ buf_out->data_len == imm_sz);
if (err != 0) {
iser_err("Failed to register write cmd RDMA mem\n");
return err;
diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c
index 1103aa6e8d00..e500d0df7b85 100644
--- a/drivers/infiniband/ulp/iser/iser_memory.c
+++ b/drivers/infiniband/ulp/iser/iser_memory.c
@@ -250,7 +250,7 @@ iser_reg_dma(struct iser_device *device, struct iser_data_buf *mem,
struct scatterlist *sg = mem->sg;
reg->sge.lkey = device->pd->local_dma_lkey;
- reg->rkey = device->mr->rkey;
+ reg->rkey = device->mr ? device->mr->rkey : 0;
reg->sge.addr = ib_sg_dma_address(device->ib_device, &sg[0]);
reg->sge.length = ib_sg_dma_len(device->ib_device, &sg[0]);
@@ -553,7 +553,8 @@ iser_reg_data_sg(struct iscsi_iser_task *task,
}
int iser_reg_rdma_mem(struct iscsi_iser_task *task,
- enum iser_data_dir dir)
+ enum iser_data_dir dir,
+ bool all_imm)
{
struct ib_conn *ib_conn = &task->iser_conn->ib_conn;
struct iser_device *device = ib_conn->device;
@@ -564,8 +565,8 @@ int iser_reg_rdma_mem(struct iscsi_iser_task *task,
bool use_dma_key;
int err;
- use_dma_key = (mem->dma_nents == 1 && !iser_always_reg &&
- scsi_get_prot_op(task->sc) == SCSI_PROT_NORMAL);
+ use_dma_key = mem->dma_nents == 1 && (all_imm || !iser_always_reg) &&
+ scsi_get_prot_op(task->sc) == SCSI_PROT_NORMAL;
if (!use_dma_key) {
desc = device->reg_ops->reg_desc_get(ib_conn);
--
1.8.4.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 03/10] IB/iser: Don't register memory for all immediatedata writes
2015-11-16 16:37 ` [PATCH for-next 03/10] IB/iser: Don't register memory for all immediatedata writes Sagi Grimberg
@ 2015-11-17 7:47 ` Or Gerlitz
[not found] ` <564ADB7D.20806-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 7:47 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> --- a/drivers/infiniband/ulp/iser/iser_memory.c
> +++ b/drivers/infiniband/ulp/iser/iser_memory.c
> @@ -250,7 +250,7 @@ iser_reg_dma(struct iser_device *device, struct iser_data_buf *mem,
> struct scatterlist *sg = mem->sg;
>
> reg->sge.lkey = device->pd->local_dma_lkey;
> - reg->rkey = device->mr->rkey;
> + reg->rkey = device->mr ? device->mr->rkey : 0;
> reg->sge.addr = ib_sg_dma_address(device->ib_device, &sg[0]);
> reg->sge.length = ib_sg_dma_len(device->ib_device, &sg[0]);
>
what's the role of this hunk? why it belongs here? you are testing
device->mr but this is
something global and has nothing to do specially with specific IOs for
which this patch
aims to act
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH for-next 07/10] iser-target: Declare correct flags when accepting a connection
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-16 16:37 ` [PATCH for-next 03/10] IB/iser: Don't register memory for all immediatedata writes Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
[not found] ` <1447691861-3796-8-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-16 16:37 ` [PATCH for-next 09/10] IB/iser: Increment the rkey when registering and not when invalidating Sagi Grimberg
2015-11-17 8:10 ` [PATCH for-next 00/10] iSER support for remote invalidate Or Gerlitz
3 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Jenny Derzhavetz
From: Jenny Derzhavetz <jennyf-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
iser target does not support zero based virtual addresses and
send with invalidate, so it should declare that it doesn't.
Signed-off-by: Jenny Derzhavetz <jennyf-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/ulp/isert/ib_isert.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index bdda42638629..667238f6253f 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -3099,12 +3099,18 @@ isert_rdma_accept(struct isert_conn *isert_conn)
struct rdma_cm_id *cm_id = isert_conn->cm_id;
struct rdma_conn_param cp;
int ret;
+ struct iser_cm_hdr rsp_hdr;
memset(&cp, 0, sizeof(struct rdma_conn_param));
cp.initiator_depth = isert_conn->initiator_depth;
cp.retry_count = 7;
cp.rnr_retry_count = 7;
+ memset(&rsp_hdr, 0, sizeof(rsp_hdr));
+ rsp_hdr.flags = (ISERT_ZBVA_NOT_USED | ISERT_SEND_W_INV_NOT_USED);
+ cp.private_data = (void *)&rsp_hdr;
+ cp.private_data_len = sizeof(rsp_hdr);
+
ret = rdma_accept(cm_id, &cp);
if (ret) {
isert_err("rdma_accept() failed with: %d\n", ret);
--
1.8.4.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH for-next 09/10] IB/iser: Increment the rkey when registering and not when invalidating
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-16 16:37 ` [PATCH for-next 03/10] IB/iser: Don't register memory for all immediatedata writes Sagi Grimberg
2015-11-16 16:37 ` [PATCH for-next 07/10] iser-target: Declare correct flags when accepting a connection Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
2015-11-17 8:13 ` Or Gerlitz
2015-11-17 8:10 ` [PATCH for-next 00/10] iSER support for remote invalidate Or Gerlitz
3 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise
With remote invalidate we won't local invalidate
but we still want to increment the rkey.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Jenny Derzhavetz <jennyf-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/ulp/iser/iser_memory.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c
index fb676ae25c42..52f64073037b 100644
--- a/drivers/infiniband/ulp/iser/iser_memory.c
+++ b/drivers/infiniband/ulp/iser/iser_memory.c
@@ -412,19 +412,14 @@ iser_set_prot_checks(struct scsi_cmnd *sc, u8 *mask)
*mask |= ISER_CHECK_GUARD;
}
-static void
+static inline void
iser_inv_rkey(struct ib_send_wr *inv_wr, struct ib_mr *mr)
{
- u32 rkey;
-
inv_wr->opcode = IB_WR_LOCAL_INV;
inv_wr->wr_id = ISER_FASTREG_LI_WRID;
inv_wr->ex.invalidate_rkey = mr->rkey;
inv_wr->send_flags = 0;
inv_wr->num_sge = 0;
-
- rkey = ib_inc_rkey(mr->rkey);
- ib_update_fast_reg_key(mr, rkey);
}
static int
@@ -437,6 +432,7 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
struct iser_tx_desc *tx_desc = &iser_task->desc;
struct ib_sig_attrs *sig_attrs = &tx_desc->sig_attrs;
struct ib_sig_handover_wr *wr;
+ struct ib_mr *mr = pi_ctx->sig_mr;
int ret;
memset(sig_attrs, 0, sizeof(*sig_attrs));
@@ -447,7 +443,9 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
iser_set_prot_checks(iser_task->sc, &sig_attrs->check_mask);
if (pi_ctx->sig_mr_valid)
- iser_inv_rkey(iser_tx_next_wr(tx_desc), pi_ctx->sig_mr);
+ iser_inv_rkey(iser_tx_next_wr(tx_desc), mr);
+
+ ib_update_fast_reg_key(mr, ib_inc_rkey(mr->rkey));
wr = sig_handover_wr(iser_tx_next_wr(tx_desc));
wr->wr.opcode = IB_WR_REG_SIG_MR;
@@ -456,7 +454,7 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
wr->wr.num_sge = 1;
wr->wr.send_flags = 0;
wr->sig_attrs = sig_attrs;
- wr->sig_mr = pi_ctx->sig_mr;
+ wr->sig_mr = mr;
if (scsi_prot_sg_count(iser_task->sc))
wr->prot = &prot_reg->sge;
else
@@ -466,8 +464,8 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
IB_ACCESS_REMOTE_WRITE;
pi_ctx->sig_mr_valid = 1;
- sig_reg->sge.lkey = pi_ctx->sig_mr->lkey;
- sig_reg->rkey = pi_ctx->sig_mr->rkey;
+ sig_reg->sge.lkey = mr->lkey;
+ sig_reg->rkey = mr->rkey;
sig_reg->sge.addr = 0;
sig_reg->sge.length = scsi_transfer_length(iser_task->sc);
@@ -491,6 +489,8 @@ static int iser_fast_reg_mr(struct iscsi_iser_task *iser_task,
if (rsc->mr_valid)
iser_inv_rkey(iser_tx_next_wr(tx_desc), mr);
+ ib_update_fast_reg_key(mr, ib_inc_rkey(mr->rkey));
+
n = ib_map_mr_sg(mr, mem->sg, mem->size, SIZE_4K);
if (unlikely(n != mem->size)) {
iser_err("failed to map sg (%d/%d)\n",
--
1.8.4.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 09/10] IB/iser: Increment the rkey when registering and not when invalidating
2015-11-16 16:37 ` [PATCH for-next 09/10] IB/iser: Increment the rkey when registering and not when invalidating Sagi Grimberg
@ 2015-11-17 8:13 ` Or Gerlitz
0 siblings, 0 replies; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 8:13 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma, target-devel; +Cc: Nicholas A. Bellinger, Steve Wise
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> With remote invalidate we won't local invalidate
> but we still want to increment the rkey.
To ease with review, as this is low level pre-patch, please push it to
downstream area
of the series, e.g before the patch that introduces include/scsi/iser.h
-- also use comma
to make the sentence more readable, maybe something like: "With remote
invalidate we won't
do local invalidate, but we still want to increment the rkey."
Or.
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 00/10] iSER support for remote invalidate
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (2 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 09/10] IB/iser: Increment the rkey when registering and not when invalidating Sagi Grimberg
@ 2015-11-17 8:10 ` Or Gerlitz
[not found] ` <564AE0E8.7030705-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
3 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 8:10 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Steve Wise
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> This patchset adds remote invalidation support to iser initiator and
> target. The support negotiation for this feature is based on IBTA
> annex 12 "Support for iSCSI Extensions for RDMA" carried in rdma_cm
> private data.
On which public git tree/branch this series applies?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH for-next 04/10] IB/iser: set intuitive values for mr_valid
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (2 preceding siblings ...)
[not found] ` <1447691861-3796-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-16 16:37 ` Sagi Grimberg
2015-11-17 7:43 ` Or Gerlitz
2015-11-16 16:37 ` [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions Sagi Grimberg
` (4 subsequent siblings)
8 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Jenny Derzhavetz
From: Jenny Derzhavetz <jennyf@mellanox.com>
This parameter is described as "is mr valid indicator".
In other words, it indicates whether memory registration
is valid or not. So intuitive values would be:
mr_valid=True, when memory registration is valid and
mr_valid=False otherwise.
Signed-off-by: Jenny Derzhavetz <jennyf@mellanox.com>
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/iser/iser_memory.c | 8 ++++----
drivers/infiniband/ulp/iser/iser_verbs.c | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband/ulp/iser/iser_memory.c
index e500d0df7b85..fb676ae25c42 100644
--- a/drivers/infiniband/ulp/iser/iser_memory.c
+++ b/drivers/infiniband/ulp/iser/iser_memory.c
@@ -446,7 +446,7 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
iser_set_prot_checks(iser_task->sc, &sig_attrs->check_mask);
- if (!pi_ctx->sig_mr_valid)
+ if (pi_ctx->sig_mr_valid)
iser_inv_rkey(iser_tx_next_wr(tx_desc), pi_ctx->sig_mr);
wr = sig_handover_wr(iser_tx_next_wr(tx_desc));
@@ -464,7 +464,7 @@ iser_reg_sig_mr(struct iscsi_iser_task *iser_task,
wr->access_flags = IB_ACCESS_LOCAL_WRITE |
IB_ACCESS_REMOTE_READ |
IB_ACCESS_REMOTE_WRITE;
- pi_ctx->sig_mr_valid = 0;
+ pi_ctx->sig_mr_valid = 1;
sig_reg->sge.lkey = pi_ctx->sig_mr->lkey;
sig_reg->rkey = pi_ctx->sig_mr->rkey;
@@ -488,7 +488,7 @@ static int iser_fast_reg_mr(struct iscsi_iser_task *iser_task,
struct ib_reg_wr *wr;
int n;
- if (!rsc->mr_valid)
+ if (rsc->mr_valid)
iser_inv_rkey(iser_tx_next_wr(tx_desc), mr);
n = ib_map_mr_sg(mr, mem->sg, mem->size, SIZE_4K);
@@ -509,7 +509,7 @@ static int iser_fast_reg_mr(struct iscsi_iser_task *iser_task,
IB_ACCESS_REMOTE_WRITE |
IB_ACCESS_REMOTE_READ;
- rsc->mr_valid = 0;
+ rsc->mr_valid = 1;
reg->sge.lkey = mr->lkey;
reg->rkey = mr->rkey;
diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c
index a66b9dea96d8..09121e774d14 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -299,7 +299,7 @@ iser_alloc_reg_res(struct ib_device *ib_device,
iser_err("Failed to allocate ib_fast_reg_mr err=%d\n", ret);
return ret;
}
- res->mr_valid = 1;
+ res->mr_valid = 0;
return 0;
}
@@ -336,7 +336,7 @@ iser_alloc_pi_ctx(struct ib_device *ib_device,
ret = PTR_ERR(pi_ctx->sig_mr);
goto sig_mr_failure;
}
- pi_ctx->sig_mr_valid = 1;
+ pi_ctx->sig_mr_valid = 0;
desc->pi_ctx->sig_protected = 0;
return 0;
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 04/10] IB/iser: set intuitive values for mr_valid
2015-11-16 16:37 ` [PATCH for-next 04/10] IB/iser: set intuitive values for mr_valid Sagi Grimberg
@ 2015-11-17 7:43 ` Or Gerlitz
[not found] ` <564ADAB5.3080208-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 7:43 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> This parameter is described as "is mr valid indicator".
> In other words, it indicates whether memory registration
> is valid or not. So intuitive values would be:
> mr_valid=True, when memory registration is valid and
> mr_valid=False otherwise.
and what was there before? the other way around?
on what cases the indicator is false (with the new semantics)?
Also, we have a convention of starting commit titles with capital
letters, please follow on that "Set iser [...]"
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (3 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 04/10] IB/iser: set intuitive values for mr_valid Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
2015-11-17 8:59 ` Christoph Hellwig
[not found] ` <1447691861-3796-6-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-16 16:37 ` [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h Sagi Grimberg
` (3 subsequent siblings)
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, linux-scsi
The iser RDMA_CM negotiation protocol is shared by
the initiator and the target, so have a shared header
for the defines and structure. Move relevant items from
the initiator and target headers.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
Signed-off-by: Jenny Derzhavetz <jennyf@mellanox.com>
Cc: linux-scsi <linux-scsi@vger.kernel.org>
---
drivers/infiniband/ulp/iser/iscsi_iser.h | 41 ++--------
drivers/infiniband/ulp/iser/iser_initiator.c | 6 +-
drivers/infiniband/ulp/iser/iser_verbs.c | 7 +-
drivers/infiniband/ulp/isert/ib_isert.c | 22 +++---
drivers/infiniband/ulp/isert/ib_isert.h | 6 +-
include/scsi/iser.h | 109 +++++++++++++++++++++++++++
6 files changed, 135 insertions(+), 56 deletions(-)
create mode 100644 include/scsi/iser.h
diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h b/drivers/infiniband/ulp/iser/iscsi_iser.h
index fb7fa7aa113c..096d5234bbea 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.h
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.h
@@ -48,6 +48,7 @@
#include <scsi/scsi_transport_iscsi.h>
#include <scsi/scsi_cmnd.h>
#include <scsi/scsi_device.h>
+#include <scsi/iser.h>
#include <linux/interrupt.h>
#include <linux/wait.h>
@@ -154,43 +155,11 @@
#define ISER_WC_BATCH_COUNT 16
#define ISER_SIGNAL_CMD_COUNT 32
-#define ISER_VER 0x10
-#define ISER_WSV 0x08
-#define ISER_RSV 0x04
-
#define ISER_FASTREG_LI_WRID 0xffffffffffffffffULL
#define ISER_BEACON_WRID 0xfffffffffffffffeULL
-/**
- * struct iser_hdr - iSER header
- *
- * @flags: flags support (zbva, remote_inv)
- * @rsvd: reserved
- * @write_stag: write rkey
- * @write_va: write virtual address
- * @reaf_stag: read rkey
- * @read_va: read virtual address
- */
-struct iser_hdr {
- u8 flags;
- u8 rsvd[3];
- __be32 write_stag;
- __be64 write_va;
- __be32 read_stag;
- __be64 read_va;
-} __attribute__((packed));
-
-
-#define ISER_ZBVA_NOT_SUPPORTED 0x80
-#define ISER_SEND_W_INV_NOT_SUPPORTED 0x40
-
-struct iser_cm_hdr {
- u8 flags;
- u8 rsvd[3];
-} __packed;
-
-/* Constant PDU lengths calculations */
-#define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct iscsi_hdr))
+/*Constant PDU lengths calculations */
+#define ISER_HEADERS_LEN (sizeof(struct iser_ctrl) + sizeof(struct iscsi_hdr))
#define ISER_RECV_DATA_SEG_LEN 128
#define ISER_RX_PAYLOAD_SIZE (ISER_HEADERS_LEN + ISER_RECV_DATA_SEG_LEN)
@@ -287,7 +256,7 @@ enum iser_desc_type {
* @sig_attrs: Signature attributes
*/
struct iser_tx_desc {
- struct iser_hdr iser_header;
+ struct iser_ctrl iser_header;
struct iscsi_hdr iscsi_header;
enum iser_desc_type type;
u64 dma_addr;
@@ -318,7 +287,7 @@ struct iser_tx_desc {
* @pad: for sense data TODO: Modify to maximum sense length supported
*/
struct iser_rx_desc {
- struct iser_hdr iser_header;
+ struct iser_ctrl iser_header;
struct iscsi_hdr iscsi_header;
char data[ISER_RECV_DATA_SEG_LEN];
u64 dma_addr;
diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c
index 07bf26427ee7..6a968e350c14 100644
--- a/drivers/infiniband/ulp/iser/iser_initiator.c
+++ b/drivers/infiniband/ulp/iser/iser_initiator.c
@@ -51,7 +51,7 @@ static int iser_prepare_read_cmd(struct iscsi_task *task)
struct iscsi_iser_task *iser_task = task->dd_data;
struct iser_mem_reg *mem_reg;
int err;
- struct iser_hdr *hdr = &iser_task->desc.iser_header;
+ struct iser_ctrl *hdr = &iser_task->desc.iser_header;
struct iser_data_buf *buf_in = &iser_task->data[ISER_DIR_IN];
err = iser_dma_map_task_data(iser_task,
@@ -104,7 +104,7 @@ iser_prepare_write_cmd(struct iscsi_task *task,
struct iscsi_iser_task *iser_task = task->dd_data;
struct iser_mem_reg *mem_reg;
int err;
- struct iser_hdr *hdr = &iser_task->desc.iser_header;
+ struct iser_ctrl *hdr = &iser_task->desc.iser_header;
struct iser_data_buf *buf_out = &iser_task->data[ISER_DIR_OUT];
struct ib_sge *tx_dsg = &iser_task->desc.tx_sg[1];
@@ -167,7 +167,7 @@ static void iser_create_send_desc(struct iser_conn *iser_conn,
ib_dma_sync_single_for_cpu(device->ib_device,
tx_desc->dma_addr, ISER_HEADERS_LEN, DMA_TO_DEVICE);
- memset(&tx_desc->iser_header, 0, sizeof(struct iser_hdr));
+ memset(&tx_desc->iser_header, 0, sizeof(struct iser_ctrl));
tx_desc->iser_header.flags = ISER_VER;
tx_desc->num_sge = 1;
}
diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c
index 09121e774d14..74161a566852 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -847,10 +847,9 @@ static void iser_route_handler(struct rdma_cm_id *cma_id)
conn_param.rnr_retry_count = 6;
memset(&req_hdr, 0, sizeof(req_hdr));
- req_hdr.flags = (ISER_ZBVA_NOT_SUPPORTED |
- ISER_SEND_W_INV_NOT_SUPPORTED);
- conn_param.private_data = (void *)&req_hdr;
- conn_param.private_data_len = sizeof(struct iser_cm_hdr);
+ req_hdr.flags = (ISER_ZBVA_NOT_SUP | ISER_SEND_W_INV_NOT_SUP);
+ conn_param.private_data = (void *)&req_hdr;
+ conn_param.private_data_len = sizeof(struct iser_cm_hdr);
ret = rdma_connect(cma_id, &conn_param);
if (ret) {
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 7dbae56e576a..97ac9ffaf1eb 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1054,8 +1054,8 @@ isert_create_send_desc(struct isert_conn *isert_conn,
ib_dma_sync_single_for_cpu(ib_dev, tx_desc->dma_addr,
ISER_HEADERS_LEN, DMA_TO_DEVICE);
- memset(&tx_desc->iser_header, 0, sizeof(struct iser_hdr));
- tx_desc->iser_header.flags = ISER_VER;
+ memset(&tx_desc->iser_header, 0, sizeof(struct iser_ctrl));
+ tx_desc->iser_header.flags = ISCSI_CTRL;
tx_desc->num_sge = 1;
tx_desc->isert_cmd = isert_cmd;
@@ -1547,22 +1547,22 @@ isert_rx_opcode(struct isert_conn *isert_conn, struct iser_rx_desc *rx_desc,
static void
isert_rx_do_work(struct iser_rx_desc *rx_desc, struct isert_conn *isert_conn)
{
- struct iser_hdr *iser_hdr = &rx_desc->iser_header;
+ struct iser_ctrl *iser_ctrl = &rx_desc->iser_header;
uint64_t read_va = 0, write_va = 0;
uint32_t read_stag = 0, write_stag = 0;
int rc;
- switch (iser_hdr->flags & 0xF0) {
+ switch (iser_ctrl->flags & 0xF0) {
case ISCSI_CTRL:
- if (iser_hdr->flags & ISER_RSV) {
- read_stag = be32_to_cpu(iser_hdr->read_stag);
- read_va = be64_to_cpu(iser_hdr->read_va);
+ if (iser_ctrl->flags & ISER_RSV) {
+ read_stag = be32_to_cpu(iser_ctrl->read_stag);
+ read_va = be64_to_cpu(iser_ctrl->read_va);
isert_dbg("ISER_RSV: read_stag: 0x%x read_va: 0x%llx\n",
read_stag, (unsigned long long)read_va);
}
- if (iser_hdr->flags & ISER_WSV) {
- write_stag = be32_to_cpu(iser_hdr->write_stag);
- write_va = be64_to_cpu(iser_hdr->write_va);
+ if (iser_ctrl->flags & ISER_WSV) {
+ write_stag = be32_to_cpu(iser_ctrl->write_stag);
+ write_va = be64_to_cpu(iser_ctrl->write_va);
isert_dbg("ISER_WSV: write_stag: 0x%x write_va: 0x%llx\n",
write_stag, (unsigned long long)write_va);
}
@@ -1573,7 +1573,7 @@ isert_rx_do_work(struct iser_rx_desc *rx_desc, struct isert_conn *isert_conn)
isert_err("iSER Hello message\n");
break;
default:
- isert_warn("Unknown iSER hdr flags: 0x%02x\n", iser_hdr->flags);
+ isert_warn("Unknown iSER hdr flags: 0x%02x\n", iser_ctrl->flags);
break;
}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 3d7fbc47c343..03ddc3e0087f 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -3,6 +3,8 @@
#include <linux/in6.h>
#include <rdma/ib_verbs.h>
#include <rdma/rdma_cm.h>
+#include <scsi/iser.h>
+
#define DRV_NAME "isert"
#define PFX DRV_NAME ": "
@@ -56,7 +58,7 @@ enum iser_conn_state {
};
struct iser_rx_desc {
- struct iser_hdr iser_header;
+ struct iser_ctrl iser_header;
struct iscsi_hdr iscsi_header;
char data[ISER_RECV_DATA_SEG_LEN];
u64 dma_addr;
@@ -65,7 +67,7 @@ struct iser_rx_desc {
} __packed;
struct iser_tx_desc {
- struct iser_hdr iser_header;
+ struct iser_ctrl iser_header;
struct iscsi_hdr iscsi_header;
enum isert_desc_type type;
u64 dma_addr;
diff --git a/include/scsi/iser.h b/include/scsi/iser.h
new file mode 100644
index 000000000000..16098de23f4b
--- /dev/null
+++ b/include/scsi/iser.h
@@ -0,0 +1,109 @@
+/*
+ * Copyright (c) 2015 Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#ifndef ISCSI_ISER_H
+#define ISCSI_ISER_H
+
+#define ISER_ZBVA_NOT_SUP 0x80
+#define ISER_SEND_W_INV_NOT_SUP 0x40
+#define ISERT_ZBVA_NOT_USED 0x80
+#define ISERT_SEND_W_INV_NOT_USED 0x40
+
+#define ISCSI_CTRL 0x10
+#define ISER_HELLO 0x20
+#define ISER_HELLORPLY 0x30
+
+#define ISER_VER 0x10
+#define ISER_WSV 0x08
+#define ISER_RSV 0x04
+
+/**
+ * struct iser_cm_hdr - iSER CM header (from iSER Annex A12)
+ *
+ * @flags: flags support (zbva, send_w_inv)
+ * @rsvd: reserved
+ */
+struct iser_cm_hdr {
+ u8 flags;
+ u8 rsvd[3];
+} __packed;
+
+/**
+ * struct iser_ctrl - iSER header of iSCSI control PDU
+ *
+ * @flags: opcode and read/write valid bits
+ * @rsvd: reserved
+ * @write_stag: write rkey
+ * @write_va: write virtual address
+ * @reaf_stag: read rkey
+ * @read_va: read virtual address
+ */
+struct iser_ctrl {
+ u8 flags;
+ u8 rsvd[3];
+ __be32 write_stag;
+ __be64 write_va;
+ __be32 read_stag;
+ __be64 read_va;
+} __packed;
+
+/**
+ * struct iser_hello - iSER Hello header
+ *
+ * @opcode: opcode (must be set to ISER_HELLO)
+ * @max_min_ver: maximum and minimum iser versions
+ * @iser_ird: iSER IRD
+ * @rsvd: reserved
+ */
+struct iser_hello {
+ u8 opcode;
+ u8 max_min_ver;
+ u16 iser_ird;
+ u8 rsvd[20];
+} __packed;
+
+/**
+ * struct iser_hello_rep - iSER Hello reply header
+ *
+ * @opcode_rej: opcode (must be set to ISER_HELLORPLY)
+ * lower bit is reject bit
+ * @max_cur_ver: maximum and current iser versions
+ * @iser_ord: iSER ORD
+ * @rsvd: reserved
+ */
+struct iser_hello_rep {
+ u8 opcode_rej;
+ u8 max_cur_ver;
+ u16 iser_ord;
+ u8 rsvd[20];
+} __packed;
+
+#endif /* ISCSI_ISER_H */
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions
2015-11-16 16:37 ` [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions Sagi Grimberg
@ 2015-11-17 8:59 ` Christoph Hellwig
[not found] ` <20151117085933.GC19578-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
[not found] ` <1447691861-3796-6-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
1 sibling, 1 reply; 74+ messages in thread
From: Christoph Hellwig @ 2015-11-17 8:59 UTC (permalink / raw)
To: Sagi Grimberg
Cc: linux-rdma, target-devel, Nicholas A. Bellinger, Or Gerlitz,
Steve Wise, linux-scsi
On Mon, Nov 16, 2015 at 06:37:36PM +0200, Sagi Grimberg wrote:
> The iser RDMA_CM negotiation protocol is shared by
> the initiator and the target, so have a shared header
> for the defines and structure. Move relevant items from
> the initiator and target headers.
Nice!
But should a header that just documents protocol constants really
have a copyright header?
^ permalink raw reply [flat|nested] 74+ messages in thread
[parent not found: <1447691861-3796-6-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions
[not found] ` <1447691861-3796-6-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-19 7:20 ` Or Gerlitz
[not found] ` <564D7835.4010407-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-19 7:20 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Steve Wise, linux-scsi
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> +/**
> + * struct iser_hello - iSER Hello header
> + *
> + * @opcode: opcode (must be set to ISER_HELLO)
> + * @max_min_ver: maximum and minimum iser versions
> + * @iser_ird: iSER IRD
> + * @rsvd: reserved
> + */
> +struct iser_hello {
> + u8 opcode;
> + u8 max_min_ver;
> + u16 iser_ird;
> + u8 rsvd[20];
> +} __packed;
> +
> +/**
> + * struct iser_hello_rep - iSER Hello reply header
> + *
> + * @opcode_rej: opcode (must be set to ISER_HELLORPLY)
> + * lower bit is reject bit
> + * @max_cur_ver: maximum and current iser versions
> + * @iser_ord: iSER ORD
> + * @rsvd: reserved
> + */
> +struct iser_hello_rep {
> + u8 opcode_rej;
> + u8 max_cur_ver;
> + u16 iser_ord;
> + u8 rsvd[20];
> +} __packed;
> +
I don't see the point to include these two defs, we don't use them and
Steve even got iser to work
over iwarp without them, so why care? we should only leave
>
> +#define ISER_HELLO 0x20
> +#define ISER_HELLORPLY 0x30
to allow warnings on them if we get such packets
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (4 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 05/10] iser: Have initiator and target to share protocol structures and definitions Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
2015-11-17 7:57 ` Or Gerlitz
2015-11-17 9:00 ` Christoph Hellwig
2015-11-16 16:37 ` [PATCH for-next 08/10] iser-target: Support the remote invalidation exception Sagi Grimberg
` (2 subsequent siblings)
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel; +Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise
We don't need iser_proto.h anymore, remove it and
move (non-protocol) declarations to ib_isert.h
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
Signed-off-by: Jenny Derzhavetz <jennyf@mellanox.com>
---
drivers/infiniband/ulp/isert/ib_isert.c | 1 -
drivers/infiniband/ulp/isert/ib_isert.h | 31 ++++++++++++++++++++
drivers/infiniband/ulp/isert/isert_proto.h | 47 ------------------------------
3 files changed, 31 insertions(+), 48 deletions(-)
delete mode 100644 drivers/infiniband/ulp/isert/isert_proto.h
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 97ac9ffaf1eb..bdda42638629 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -29,7 +29,6 @@
#include <target/iscsi/iscsi_transport.h>
#include <linux/semaphore.h>
-#include "isert_proto.h"
#include "ib_isert.h"
#define ISERT_MAX_CONN 8
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 03ddc3e0087f..00edd5b68ca8 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -33,6 +33,37 @@
#define isert_err(fmt, arg...) \
pr_err(PFX "%s: " fmt, __func__ , ## arg)
+/*Constant PDU lengths calculations */
+#define ISER_HEADERS_LEN (sizeof(struct iser_ctrl) + sizeof(struct iscsi_hdr))
+#define ISER_RECV_DATA_SEG_LEN 8192
+#define ISER_RX_PAYLOAD_SIZE (ISER_HEADERS_LEN + ISER_RECV_DATA_SEG_LEN)
+#define ISER_RX_LOGIN_SIZE (ISER_HEADERS_LEN + ISCSI_DEF_MAX_RECV_SEG_LEN)
+
+/* QP settings */
+/* Maximal bounds on received asynchronous PDUs */
+#define ISERT_MAX_TX_MISC_PDUS 4 /* NOOP_IN(2) , ASYNC_EVENT(2) */
+
+#define ISERT_MAX_RX_MISC_PDUS 6 /*
+ * NOOP_OUT(2), TEXT(1),
+ * SCSI_TMFUNC(2), LOGOUT(1)
+ */
+
+#define ISCSI_DEF_XMIT_CMDS_MAX 128 /* from libiscsi.h, must be power of 2 */
+
+#define ISERT_QP_MAX_RECV_DTOS (ISCSI_DEF_XMIT_CMDS_MAX)
+
+#define ISERT_MIN_POSTED_RX (ISCSI_DEF_XMIT_CMDS_MAX >> 2)
+
+#define ISERT_INFLIGHT_DATAOUTS 8
+
+#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX * \
+ (1 + ISERT_INFLIGHT_DATAOUTS) + \
+ ISERT_MAX_TX_MISC_PDUS + \
+ ISERT_MAX_RX_MISC_PDUS)
+
+#define ISER_RX_PAD_SIZE (ISER_RECV_DATA_SEG_LEN + 4096 - \
+ (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct ib_sge)))
+
#define ISCSI_ISER_SG_TABLESIZE 256
#define ISER_FASTREG_LI_WRID 0xffffffffffffffffULL
#define ISER_BEACON_WRID 0xfffffffffffffffeULL
diff --git a/drivers/infiniband/ulp/isert/isert_proto.h b/drivers/infiniband/ulp/isert/isert_proto.h
deleted file mode 100644
index 4dccd313b777..000000000000
--- a/drivers/infiniband/ulp/isert/isert_proto.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/* From iscsi_iser.h */
-
-struct iser_hdr {
- u8 flags;
- u8 rsvd[3];
- __be32 write_stag; /* write rkey */
- __be64 write_va;
- __be32 read_stag; /* read rkey */
- __be64 read_va;
-} __packed;
-
-/*Constant PDU lengths calculations */
-#define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct iscsi_hdr))
-
-#define ISER_RECV_DATA_SEG_LEN 8192
-#define ISER_RX_PAYLOAD_SIZE (ISER_HEADERS_LEN + ISER_RECV_DATA_SEG_LEN)
-#define ISER_RX_LOGIN_SIZE (ISER_HEADERS_LEN + ISCSI_DEF_MAX_RECV_SEG_LEN)
-
-/* QP settings */
-/* Maximal bounds on received asynchronous PDUs */
-#define ISERT_MAX_TX_MISC_PDUS 4 /* NOOP_IN(2) , ASYNC_EVENT(2) */
-
-#define ISERT_MAX_RX_MISC_PDUS 6 /* NOOP_OUT(2), TEXT(1), *
- * SCSI_TMFUNC(2), LOGOUT(1) */
-
-#define ISCSI_DEF_XMIT_CMDS_MAX 128 /* from libiscsi.h, must be power of 2 */
-
-#define ISERT_QP_MAX_RECV_DTOS (ISCSI_DEF_XMIT_CMDS_MAX)
-
-#define ISERT_MIN_POSTED_RX (ISCSI_DEF_XMIT_CMDS_MAX >> 2)
-
-#define ISERT_INFLIGHT_DATAOUTS 8
-
-#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX * \
- (1 + ISERT_INFLIGHT_DATAOUTS) + \
- ISERT_MAX_TX_MISC_PDUS + \
- ISERT_MAX_RX_MISC_PDUS)
-
-#define ISER_RX_PAD_SIZE (ISER_RECV_DATA_SEG_LEN + 4096 - \
- (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct ib_sge)))
-
-#define ISER_VER 0x10
-#define ISER_WSV 0x08
-#define ISER_RSV 0x04
-#define ISCSI_CTRL 0x10
-#define ISER_HELLO 0x20
-#define ISER_HELLORPLY 0x30
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h
2015-11-16 16:37 ` [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h Sagi Grimberg
@ 2015-11-17 7:57 ` Or Gerlitz
2015-11-17 9:45 ` Sagi Grimberg
2015-11-17 9:00 ` Christoph Hellwig
1 sibling, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 7:57 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma, target-devel; +Cc: Nicholas A. Bellinger, Steve Wise
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> -/* From iscsi_iser.h */
> -
> -struct iser_hdr {
> - u8 flags;
> - u8 rsvd[3];
> - __be32 write_stag; /* write rkey */
> - __be64 write_va;
> - __be32 read_stag; /* read rkey */
> - __be64 read_va;
> -} __packed;
> -
> -/*Constant PDU lengths calculations */
> -#define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct iscsi_hdr))
move this part of the cleanup to the preceding patch, so it can be
reviewed and bisected into
regardless of this or any downstream patch of the series.
Or.
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h
2015-11-17 7:57 ` Or Gerlitz
@ 2015-11-17 9:45 ` Sagi Grimberg
0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-17 9:45 UTC (permalink / raw)
To: Or Gerlitz, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise
On 17/11/2015 09:57, Or Gerlitz wrote:
> On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
>> -/* From iscsi_iser.h */
>> -
>> -struct iser_hdr {
>> - u8 flags;
>> - u8 rsvd[3];
>> - __be32 write_stag; /* write rkey */
>> - __be64 write_va;
>> - __be32 read_stag; /* read rkey */
>> - __be64 read_va;
>> -} __packed;
>> -
>> -/*Constant PDU lengths calculations */
>> -#define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct
>> iscsi_hdr))
>
> move this part of the cleanup to the preceding patch, so it can be
> reviewed and bisected into
> regardless of this or any downstream patch of the series.
You're right... will fix..
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h
2015-11-16 16:37 ` [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h Sagi Grimberg
2015-11-17 7:57 ` Or Gerlitz
@ 2015-11-17 9:00 ` Christoph Hellwig
[not found] ` <20151117090046.GD19578-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
1 sibling, 1 reply; 74+ messages in thread
From: Christoph Hellwig @ 2015-11-17 9:00 UTC (permalink / raw)
To: Sagi Grimberg
Cc: linux-rdma, target-devel, Nicholas A. Bellinger, Or Gerlitz, Steve Wise
Sorry for the nitpicking, but:
> +/*Constant PDU lengths calculations */
This really needs a space before the 'C'.
> +#define ISER_HEADERS_LEN (sizeof(struct iser_ctrl) + sizeof(struct iscsi_hdr))
Also fixing lines over 8- characters would be nice.
Otherwise looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH for-next 08/10] iser-target: Support the remote invalidation exception
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (5 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 06/10] iser-target: Remove unused file iser_proto.h Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
[not found] ` <1447691861-3796-9-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-19 7:25 ` Or Gerlitz
2015-11-16 16:37 ` [PATCH for-next 10/10] IB/iser: " Sagi Grimberg
2015-11-17 9:35 ` [PATCH for-next 00/10] iSER support for remote invalidate Christoph Hellwig
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Jenny Derzhavetz
From: Jenny Derzhavetz <jennyf@mellanox.com>
We'll use remote invalidate, according to negotiation result
during connection establishment. If the initiator declared that
it supports the remote invalidate exception then the target will
use IB_WR_SEND_WITH_INV with the correct rkey for the response.
Signed-off-by: Jenny Derzhavetz <jennyf@mellanox.com>
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/isert/ib_isert.c | 39 +++++++++++++++++++++++++++------
drivers/infiniband/ulp/isert/ib_isert.h | 2 ++
2 files changed, 34 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 667238f6253f..035fd12ec7ae 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -679,6 +679,25 @@ out_login_buf:
return ret;
}
+static void
+isert_set_nego_params(struct isert_conn *isert_conn,
+ struct rdma_conn_param *param)
+{
+ struct isert_device *device = isert_conn->device;
+
+ /* Set max inflight RDMA READ requests */
+ isert_conn->initiator_depth = min_t(u8, param->initiator_depth,
+ device->dev_attr.max_qp_init_rd_atom);
+ isert_dbg("Using initiator_depth: %u\n", isert_conn->initiator_depth);
+
+ if (param->private_data) {
+ u8 flags = *(u8 *)param->private_data;
+
+ isert_conn->snd_w_inv = !(flags & ISER_SEND_W_INV_NOT_SUP);
+ isert_info("Using remote invalidation\n");
+ }
+}
+
static int
isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
{
@@ -717,11 +736,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
}
isert_conn->device = device;
- /* Set max inflight RDMA READ requests */
- isert_conn->initiator_depth = min_t(u8,
- event->param.conn.initiator_depth,
- device->dev_attr.max_qp_init_rd_atom);
- isert_dbg("Using initiator_depth: %u\n", isert_conn->initiator_depth);
+ isert_set_nego_params(isert_conn, &event->param.conn);
ret = isert_conn_setup_qp(isert_conn, cma_id);
if (ret)
@@ -1100,7 +1115,14 @@ isert_init_send_wr(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
isert_cmd->rdma_wr.iser_ib_op = ISER_IB_SEND;
send_wr->wr_id = (uintptr_t)&isert_cmd->tx_desc;
- send_wr->opcode = IB_WR_SEND;
+
+ if (isert_conn->snd_w_inv && isert_cmd->inv_rkey) {
+ send_wr->opcode = IB_WR_SEND_WITH_INV;
+ send_wr->ex.invalidate_rkey = isert_cmd->inv_rkey;
+ } else {
+ send_wr->opcode = IB_WR_SEND;
+ }
+
send_wr->sg_list = &tx_desc->tx_sg[0];
send_wr->num_sge = isert_cmd->tx_desc.num_sge;
send_wr->send_flags = IB_SEND_SIGNALED;
@@ -1489,6 +1511,7 @@ isert_rx_opcode(struct isert_conn *isert_conn, struct iser_rx_desc *rx_desc,
isert_cmd->read_va = read_va;
isert_cmd->write_stag = write_stag;
isert_cmd->write_va = write_va;
+ isert_cmd->inv_rkey = read_stag ? read_stag : write_stag;
ret = isert_handle_scsi_cmd(isert_conn, isert_cmd, cmd,
rx_desc, (unsigned char *)hdr);
@@ -3107,7 +3130,9 @@ isert_rdma_accept(struct isert_conn *isert_conn)
cp.rnr_retry_count = 7;
memset(&rsp_hdr, 0, sizeof(rsp_hdr));
- rsp_hdr.flags = (ISERT_ZBVA_NOT_USED | ISERT_SEND_W_INV_NOT_USED);
+ rsp_hdr.flags = ISERT_ZBVA_NOT_USED;
+ if (!isert_conn->snd_w_inv)
+ rsp_hdr.flags = rsp_hdr.flags | ISERT_SEND_W_INV_NOT_USED;
cp.private_data = (void *)&rsp_hdr;
cp.private_data_len = sizeof(rsp_hdr);
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 00edd5b68ca8..b23086d3d08b 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -162,6 +162,7 @@ struct isert_cmd {
uint32_t write_stag;
uint64_t read_va;
uint64_t write_va;
+ uint32_t inv_rkey;
u64 pdu_buf_dma;
u32 pdu_buf_len;
struct isert_conn *conn;
@@ -209,6 +210,7 @@ struct isert_conn {
struct work_struct release_work;
struct ib_recv_wr beacon;
bool logout_posted;
+ bool snd_w_inv;
};
#define ISERT_MAX_CQ 64
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (6 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 08/10] iser-target: Support the remote invalidation exception Sagi Grimberg
@ 2015-11-16 16:37 ` Sagi Grimberg
[not found] ` <1447691861-3796-11-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-17 8:08 ` Or Gerlitz
2015-11-17 9:35 ` [PATCH for-next 00/10] iSER support for remote invalidate Christoph Hellwig
8 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-16 16:37 UTC (permalink / raw)
To: linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Or Gerlitz, Steve Wise, Jenny Derzhavetz
From: Jenny Derzhavetz <jennyf@mellanox.com>
Declare that we support remote invalidation and be able to detect
the invalidated rkey so we won't invalidate it locally. The spec
mandates that we must not rely on the taget remote invalidate our
rkey so we must check it upon a receive (scsi response) completion.
Signed-off-by: Jenny Derzhavetz <jennyf@mellanox.com>
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/iser/iscsi_iser.h | 3 +-
drivers/infiniband/ulp/iser/iser_initiator.c | 55 +++++++++++++++++++++++++++-
drivers/infiniband/ulp/iser/iser_verbs.c | 19 +++++++---
3 files changed, 70 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h b/drivers/infiniband/ulp/iser/iscsi_iser.h
index 096d5234bbea..2e0a24ba18ed 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.h
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.h
@@ -519,6 +519,7 @@ struct iser_conn {
u32 num_rx_descs;
unsigned short scsi_sg_tablesize;
unsigned int scsi_max_sectors;
+ bool snd_w_inv;
};
/**
@@ -603,7 +604,7 @@ int iser_conn_terminate(struct iser_conn *iser_conn);
void iser_release_work(struct work_struct *work);
void iser_rcv_completion(struct iser_rx_desc *desc,
- unsigned long dto_xfer_len,
+ struct ib_wc *wc,
struct ib_conn *ib_conn);
void iser_snd_completion(struct iser_tx_desc *desc,
diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c
index 6a968e350c14..df6a4b70ffc9 100644
--- a/drivers/infiniband/ulp/iser/iser_initiator.c
+++ b/drivers/infiniband/ulp/iser/iser_initiator.c
@@ -563,11 +563,57 @@ send_control_error:
return err;
}
+static int
+iser_check_remote_inv(struct iser_conn *iser_conn,
+ struct ib_wc *wc,
+ struct iscsi_hdr *hdr)
+{
+ if (wc->wc_flags & IB_WC_WITH_INVALIDATE) {
+ struct iscsi_task *task;
+ struct iscsi_iser_task *iser_task;
+ u32 rkey = wc->ex.invalidate_rkey;
+
+ iser_dbg("conn %p: remote invalidation\n", iser_conn);
+ if (unlikely(!iser_conn->snd_w_inv)) {
+ iser_err("conn %p: unexepected remote invalidation,"
+ "terminating connection\n", iser_conn);
+ return -EPROTO;
+ }
+
+ task = iscsi_itt_to_ctask(iser_conn->iscsi_conn, hdr->itt);
+ if (likely(task)) {
+ struct iser_mem_reg *reg;
+ struct iser_fr_desc *desc;
+
+ iser_task = task->dd_data;
+ if (iser_task->dir[ISER_DIR_IN])
+ reg = &iser_task->rdma_reg[ISER_DIR_IN];
+ else
+ reg = &iser_task->rdma_reg[ISER_DIR_OUT];
+ desc = reg->mem_h;
+
+ if (likely(rkey == desc->rsc.mr->rkey)) {
+ desc->rsc.mr_valid = 0;
+ } else if (likely(rkey == desc->pi_ctx->sig_mr->rkey)) {
+ desc->pi_ctx->sig_mr_valid = 0;
+ } else {
+ iser_err("invalidation of unknown rkey=0x%x\n", rkey);
+ return -EINVAL;
+ }
+ } else {
+ iser_err("failed to get task for itt=%d\n", hdr->itt);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
/**
* iser_rcv_dto_completion - recv DTO completion
*/
void iser_rcv_completion(struct iser_rx_desc *rx_desc,
- unsigned long rx_xfer_len,
+ struct ib_wc *wc,
struct ib_conn *ib_conn)
{
struct iser_conn *iser_conn = container_of(ib_conn, struct iser_conn,
@@ -575,6 +621,7 @@ void iser_rcv_completion(struct iser_rx_desc *rx_desc,
struct iscsi_hdr *hdr;
u64 rx_dma;
int rx_buflen, outstanding, count, err;
+ unsigned long rx_xfer_len = wc->byte_len;
/* differentiate between login to all other PDUs */
if ((char *)rx_desc == iser_conn->login_resp_buf) {
@@ -593,6 +640,12 @@ void iser_rcv_completion(struct iser_rx_desc *rx_desc,
iser_dbg("op 0x%x itt 0x%x dlen %d\n", hdr->opcode,
hdr->itt, (int)(rx_xfer_len - ISER_HEADERS_LEN));
+ if (iser_check_remote_inv(iser_conn, wc, hdr)) {
+ iscsi_conn_failure(iser_conn->iscsi_conn,
+ ISCSI_ERR_CONN_FAILED);
+ return;
+ }
+
iscsi_iser_recv(iser_conn->iscsi_conn, hdr, rx_desc->data,
rx_xfer_len - ISER_HEADERS_LEN);
diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c
index 74161a566852..5dd7cbd8e2e2 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -847,7 +847,7 @@ static void iser_route_handler(struct rdma_cm_id *cma_id)
conn_param.rnr_retry_count = 6;
memset(&req_hdr, 0, sizeof(req_hdr));
- req_hdr.flags = (ISER_ZBVA_NOT_SUP | ISER_SEND_W_INV_NOT_SUP);
+ req_hdr.flags = ISER_ZBVA_NOT_SUP;
conn_param.private_data = (void *)&req_hdr;
conn_param.private_data_len = sizeof(struct iser_cm_hdr);
@@ -862,7 +862,8 @@ failure:
iser_connect_error(cma_id);
}
-static void iser_connected_handler(struct rdma_cm_id *cma_id)
+static void iser_connected_handler(struct rdma_cm_id *cma_id,
+ const void *private_data)
{
struct iser_conn *iser_conn;
struct ib_qp_attr attr;
@@ -876,6 +877,15 @@ static void iser_connected_handler(struct rdma_cm_id *cma_id)
(void)ib_query_qp(cma_id->qp, &attr, ~0, &init_attr);
iser_info("remote qpn:%x my qpn:%x\n", attr.dest_qp_num, cma_id->qp->qp_num);
+ if (private_data) {
+ u8 flags = *(u8 *)private_data;
+
+ iser_conn->snd_w_inv = !(flags & ISER_SEND_W_INV_NOT_SUP);
+ }
+
+ iser_info("conn %p: negotiated %s invalidation\n",
+ iser_conn, iser_conn->snd_w_inv ? "remote" : "local");
+
iser_conn->state = ISER_CONN_UP;
complete(&iser_conn->up_completion);
}
@@ -927,7 +937,7 @@ static int iser_cma_handler(struct rdma_cm_id *cma_id, struct rdma_cm_event *eve
iser_route_handler(cma_id);
break;
case RDMA_CM_EVENT_ESTABLISHED:
- iser_connected_handler(cma_id);
+ iser_connected_handler(cma_id, event->param.conn.private_data);
break;
case RDMA_CM_EVENT_ADDR_ERROR:
case RDMA_CM_EVENT_ROUTE_ERROR:
@@ -1205,8 +1215,7 @@ static void iser_handle_wc(struct ib_wc *wc)
if (likely(wc->status == IB_WC_SUCCESS)) {
if (wc->opcode == IB_WC_RECV) {
rx_desc = (struct iser_rx_desc *)(uintptr_t)wc->wr_id;
- iser_rcv_completion(rx_desc, wc->byte_len,
- ib_conn);
+ iser_rcv_completion(rx_desc, wc, ib_conn);
} else
if (wc->opcode == IB_WC_SEND) {
tx_desc = (struct iser_tx_desc *)(uintptr_t)wc->wr_id;
--
1.8.4.3
^ permalink raw reply related [flat|nested] 74+ messages in thread
[parent not found: <1447691861-3796-11-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
[not found] ` <1447691861-3796-11-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-17 8:03 ` Or Gerlitz
[not found] ` <564ADF68.1050408-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-17 8:05 ` Or Gerlitz
1 sibling, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 8:03 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> void iser_rcv_completion(struct iser_rx_desc *rx_desc,
> - unsigned long rx_xfer_len,
> + struct ib_wc *wc,
> struct ib_conn *ib_conn)
> {
> struct iser_conn *iser_conn = container_of(ib_conn, struct iser_conn,
> @@ -575,6 +621,7 @@ void iser_rcv_completion(struct iser_rx_desc *rx_desc,
> struct iscsi_hdr *hdr;
> u64 rx_dma;
> int rx_buflen, outstanding, count, err;
> + unsigned long rx_xfer_len = wc->byte_len;
unrelated small enhancement which has nothing to do with this patch,
remove it from here
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
[not found] ` <1447691861-3796-11-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-11-17 8:03 ` Or Gerlitz
@ 2015-11-17 8:05 ` Or Gerlitz
2015-11-17 9:53 ` Sagi Grimberg
1 sibling, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 8:05 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
target-devel-u79uwXL29TY76Z2rM5mHXA
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> + if (iser_task->dir[ISER_DIR_IN])
> + reg = &iser_task->rdma_reg[ISER_DIR_IN];
> + else
> + reg = &iser_task->rdma_reg[ISER_DIR_OUT];
and what happens with bidirectional commands?!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
2015-11-17 8:05 ` Or Gerlitz
@ 2015-11-17 9:53 ` Sagi Grimberg
2015-11-17 10:04 ` Or Gerlitz
0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2015-11-17 9:53 UTC (permalink / raw)
To: Or Gerlitz, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 17/11/2015 10:05, Or Gerlitz wrote:
> On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
>> + if (iser_task->dir[ISER_DIR_IN])
>> + reg = &iser_task->rdma_reg[ISER_DIR_IN];
>> + else
>> + reg = &iser_task->rdma_reg[ISER_DIR_OUT];
>
> and what happens with bidirectional commands?!
It won't work, iSER lacks support for bidirectional commands for as
long as I'm involved with it and until we either decide that we care
enough to implement it both in the target and initiator sides or some
array with iser bidi support shows up I don't know how much its worth
our attention.
Thoughts?
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
2015-11-17 9:53 ` Sagi Grimberg
@ 2015-11-17 10:04 ` Or Gerlitz
2015-11-17 10:15 ` Christoph Hellwig
[not found] ` <564AFB93.9010602-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 2 replies; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 10:04 UTC (permalink / raw)
To: Sagi Grimberg, Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/17/2015 11:53 AM, Sagi Grimberg wrote:
> [...] iSER lacks support for bidirectional commands for as long as I'm
> involved with it
Why? we don't support and when did we broke it after the initial 2.6.18
upstreaming of the driver
Also, do we refuse to queuecommand a bidi? where?
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
2015-11-17 10:04 ` Or Gerlitz
@ 2015-11-17 10:15 ` Christoph Hellwig
[not found] ` <564AFB93.9010602-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
1 sibling, 0 replies; 74+ messages in thread
From: Christoph Hellwig @ 2015-11-17 10:15 UTC (permalink / raw)
To: Or Gerlitz
Cc: Sagi Grimberg, Sagi Grimberg, linux-rdma, target-devel,
Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On Tue, Nov 17, 2015 at 12:04:03PM +0200, Or Gerlitz wrote:
> Also, do we refuse to queuecommand a bidi? where?
Or, can you please do the basic research first? Thanks! (Hint: check
how few drivers support bidi commands, and how it's enabled)
^ permalink raw reply [flat|nested] 74+ messages in thread
[parent not found: <564AFB93.9010602-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH for-next 10/10] IB/iser: Support the remote invalidation exception
2015-11-16 16:37 ` [PATCH for-next 10/10] IB/iser: " Sagi Grimberg
[not found] ` <1447691861-3796-11-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-11-17 8:08 ` Or Gerlitz
[not found] ` <564AE06C.9020504-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
1 sibling, 1 reply; 74+ messages in thread
From: Or Gerlitz @ 2015-11-17 8:08 UTC (permalink / raw)
To: Sagi Grimberg, linux-rdma, target-devel
Cc: Nicholas A. Bellinger, Steve Wise, Jenny Derzhavetz
On 11/16/2015 6:37 PM, Sagi Grimberg wrote:
> --- a/drivers/infiniband/ulp/iser/iser_verbs.c
> +++ b/drivers/infiniband/ulp/iser/iser_verbs.c
> @@ -847,7 +847,7 @@ static void iser_route_handler(struct rdma_cm_id *cma_id)
> conn_param.rnr_retry_count = 6;
>
> memset(&req_hdr, 0, sizeof(req_hdr));
> - req_hdr.flags = (ISER_ZBVA_NOT_SUP | ISER_SEND_W_INV_NOT_SUP);
> + req_hdr.flags = ISER_ZBVA_NOT_SUP;
isn't there a property of the **local** device we need to check before
advertizing that
to the target? to be on the safe side, I would do that only over
devices that support
IB_DEVICE_MEM_MGT_EXTENSIONS, as non-local invalidations are part of the
BMME ext
of IBTA, right?
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH for-next 00/10] iSER support for remote invalidate
2015-11-16 16:37 [PATCH for-next 00/10] iSER support for remote invalidate Sagi Grimberg
` (7 preceding siblings ...)
2015-11-16 16:37 ` [PATCH for-next 10/10] IB/iser: " Sagi Grimberg
@ 2015-11-17 9:35 ` Christoph Hellwig
8 siblings, 0 replies; 74+ messages in thread
From: Christoph Hellwig @ 2015-11-17 9:35 UTC (permalink / raw)
To: Sagi Grimberg
Cc: linux-rdma, target-devel, Nicholas A. Bellinger, Or Gerlitz, Steve Wise
On Mon, Nov 16, 2015 at 06:37:31PM +0200, Sagi Grimberg wrote:
> This patchset adds remote invalidation support to iser initiator and
> target. The support negotiation for this feature is based on IBTA
> annex 12 "Support for iSCSI Extensions for RDMA" carried in rdma_cm
> private data.
Oh, okay - this answers my question where the private data comes from,
might be worth documenting it in the header what standard each bit
comes from.
^ permalink raw reply [flat|nested] 74+ messages in thread