From: Krishnamraju Eraparaju <krishna2@chelsio.com>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: sagi@grimberg.me, linux-rdma@vger.kernel.org, jgg@nvidia.com,
dledford@redhat.com, oren@nvidia.com, maxg@mellanox.com
Subject: Re: [PATCH 1/1] IB/isert: add module param to set sg_tablesize for IO cmd
Date: Wed, 14 Oct 2020 05:18:56 +0530 [thread overview]
Message-ID: <20201013234726.GA8423@chelsio.com> (raw)
In-Reply-To: <1d4f4c66-5c87-c2c2-c45b-6ab6a30481ea@nvidia.com>
On Tuesday, October 10/13/20, 2020 at 20:43:03 +0300, Max Gurtovoy wrote:
> Krishna,
>
> did this patch fix the issue you reported ?
With this patch applied, I don't see the reported issue anymore.
Thanks for the patch!
>
>
> On 10/11/2020 12:06 PM, Max Gurtovoy wrote:
> >From: Max Gurtovoy <maxg@mellanox.com>
> >
> >Currently, iser target support max IO size of 16MiB by default. For some
> >adapters, allocating this amount of resources might reduce the total
> >number of possible connections that can be created. For those adapters,
> >it's preferred to reduce the max IO size to be able to create more
> >connections. Since there is no handshake procedure for max IO size in
> >iser protocol, set the default max IO size to 1MiB and add a module
> >parameter for enabling the option to control it for suitable adapters.
> >
> >Reported-by: Krishnamraju Eraparaju <krishna2@chelsio.com>
> >Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
> >---
> > drivers/infiniband/ulp/isert/ib_isert.c | 27 ++++++++++++++++++++++++-
> > drivers/infiniband/ulp/isert/ib_isert.h | 6 ++++++
> > 2 files changed, 32 insertions(+), 1 deletion(-)
> >
> >diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
> >index 695f701dc43d..5a47f1bbca96 100644
> >--- a/drivers/infiniband/ulp/isert/ib_isert.c
> >+++ b/drivers/infiniband/ulp/isert/ib_isert.c
> >@@ -28,6 +28,18 @@ static int isert_debug_level;
> > module_param_named(debug_level, isert_debug_level, int, 0644);
> > MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0 (default:0)");
> >+static int isert_sg_tablesize_set(const char *val,
> >+ const struct kernel_param *kp);
> >+static const struct kernel_param_ops sg_tablesize_ops = {
> >+ .set = isert_sg_tablesize_set,
> >+ .get = param_get_int,
> >+};
> >+
> >+static int isert_sg_tablesize = ISCSI_ISER_SG_TABLESIZE;
> >+module_param_cb(sg_tablesize, &sg_tablesize_ops, &isert_sg_tablesize, 0644);
> >+MODULE_PARM_DESC(sg_tablesize,
> >+ "Number of gather/scatter entries in a single scsi command, should >= 128 (default: 256, max: 4096)");
> >+
> > static DEFINE_MUTEX(device_list_mutex);
> > static LIST_HEAD(device_list);
> > static struct workqueue_struct *isert_comp_wq;
> >@@ -47,6 +59,19 @@ static void isert_send_done(struct ib_cq *cq, struct ib_wc *wc);
> > static void isert_login_recv_done(struct ib_cq *cq, struct ib_wc *wc);
> > static void isert_login_send_done(struct ib_cq *cq, struct ib_wc *wc);
> >+static int isert_sg_tablesize_set(const char *val, const struct kernel_param *kp)
> >+{
> >+ int n = 0, ret;
> >+
> >+ ret = kstrtoint(val, 10, &n);
> >+ if (ret != 0 || n < ISCSI_ISER_MIN_SG_TABLESIZE ||
> >+ n > ISCSI_ISER_MAX_SG_TABLESIZE)
> >+ return -EINVAL;
> >+
> >+ return param_set_int(val, kp);
> >+}
> >+
> >+
> > static inline bool
> > isert_prot_cmd(struct isert_conn *conn, struct se_cmd *cmd)
> > {
> >@@ -101,7 +126,7 @@ isert_create_qp(struct isert_conn *isert_conn,
> > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1;
> > attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1;
> > factor = rdma_rw_mr_factor(device->ib_device, cma_id->port_num,
> >- ISCSI_ISER_MAX_SG_TABLESIZE);
> >+ isert_sg_tablesize);
> > attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX * factor;
> > attr.cap.max_send_sge = device->ib_device->attrs.max_send_sge;
> > attr.cap.max_recv_sge = 1;
> >diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
> >index 7fee4a65e181..90ef215bf755 100644
> >--- a/drivers/infiniband/ulp/isert/ib_isert.h
> >+++ b/drivers/infiniband/ulp/isert/ib_isert.h
> >@@ -65,6 +65,12 @@
> > */
> > #define ISER_RX_SIZE (ISCSI_DEF_MAX_RECV_SEG_LEN + 1024)
> >+/* Default I/O size is 1MB */
> >+#define ISCSI_ISER_SG_TABLESIZE 256
> >+
> >+/* Minimum I/O size is 512KB */
> >+#define ISCSI_ISER_MIN_SG_TABLESIZE 128
> >+
> > /* Maximum support is 16MB I/O size */
> > #define ISCSI_ISER_MAX_SG_TABLESIZE 4096
next prev parent reply other threads:[~2020-10-13 23:50 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-11 9:06 [PATCH 1/1] IB/isert: add module param to set sg_tablesize for IO cmd Max Gurtovoy
2020-10-13 17:43 ` Max Gurtovoy
2020-10-13 23:48 ` Krishnamraju Eraparaju [this message]
2020-10-13 22:21 ` Sagi Grimberg
2020-10-14 0:11 ` Max Gurtovoy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201013234726.GA8423@chelsio.com \
--to=krishna2@chelsio.com \
--cc=dledford@redhat.com \
--cc=jgg@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=maxg@mellanox.com \
--cc=mgurtovoy@nvidia.com \
--cc=oren@nvidia.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).