* iSERT SQ overflow with single target and multi luns @ 2020-07-07 12:36 Krishnamraju Eraparaju 2020-07-07 13:23 ` Max Gurtovoy 0 siblings, 1 reply; 4+ messages in thread From: Krishnamraju Eraparaju @ 2020-07-07 12:36 UTC (permalink / raw) To: Sagi Grimberg, Max Gurtovoy Cc: linux-rdma, Nirranjan Kirubaharan, Potnuri Bharat Teja, krishna2 Looks like the commit 07173c3e(block: enable multipage bvecs) has uncovered iSER SQ sizing issue. Here is how I hit the issue: Created two luns under single target, then run the below script on each lun(parallelly). while [ 1 ] do iozone -i 0 -i 1 -I -+d -s 100000 -r 16384 -w done Then failures like below are logged in dmesg output, due to iw_cxgb4 SQ getting full at iSER target. "isert: isert_rdma_rw_ctx_post: Cmd: 00000000cb75342a failed to post RDMA res" This issue won't occur if luns are created on seperate targets. Also, the issue won't occur if I revert the multipage bvecs(07173c3e) changes at initator. Currently SQ is being sized this way: attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + (ISCSI_DEF_XMIT_CMDS_MAX:128 * factor:3) = 523. I tried increaseing the SQ size and observed that the issue is not occuring when attr.cap.max_send_wr is 562. Looks like the avg length of RDMA READ/WRITE operations has increased after "multipage bvecs" changes. Queueing many large sized RDMA READ/WRITE WRs may cause backpressure and increases the chances of SQ getting full at provider driver. Notice the length(0x7f000 & 0x2000) of each RDMA READ operation below, for Before and After case. Before "multipage bvecs" RDMA READ: [ +0.001903] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x78] [ +0.000007] iser: iser_fast_reg_mr: lkey=0x8a41 rkey=0x8a41 addr=0x446166000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:120 READ tags RKEY:0X8A41 VA:0X446166000 [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6f] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x13b51 rkey=0x13b51 addr=0x443b25000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:111 READ tags RKEY:0X13B51 VA:0X443B25000 [ +0.000022] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xe] [ +0.000001] iser: iser_fast_reg_mr: lkey=0xa371 rkey=0xa371 addr=0x4461a4000 length=0x2000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:14 READ tags RKEY:0XA371 VA:0X4461A4000 [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x79] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x12f4f rkey=0x12f4f addr=0x4461a9000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:121 READ tags RKEY:0X12F4F VA:0X4461A9000 [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] [ +0.000003] iser: iser_fast_reg_mr: lkey=0xe040 rkey=0xe040 addr=0x447e67000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags RKEY:0XE040 VA:0X447E67000 [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7b] [ +0.000001] iser: iser_fast_reg_mr: lkey=0xb149 rkey=0xb149 addr=0x3d0366000 length=0x2000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:123 READ tags RKEY:0XB149 VA:0X3D0366000 [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xb] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x1014c rkey=0x1014c addr=0x3d0368000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:11 READ tags RKEY:0X1014C VA:0X3D0368000 [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x62] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x7c3b rkey=0x7c3b addr=0x3d03e7000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:98 READ tags RKEY:0X7C3B VA:0X3D03E7000 [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x11] [ +0.000001] iser: iser_fast_reg_mr: lkey=0x11752 rkey=0x11752 addr=0x3d6de6000 length=0x2000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:17 READ tags RKEY:0X11752 VA:0X3D6DE6000 [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x77] After "multipage bvecs" RDMA READ: [ +0.002455] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] [ +0.000006] iser: iser_fast_reg_mr: lkey=0x7991 rkey=0x7991 addr=0x3d2819000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags RKEY:0X7991 VA:0X3D2819000 [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7e] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8c9b rkey=0x8c9b addr=0x3d2898000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:126 READ tags RKEY:0X8C9B VA:0X3D2898000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7f] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x856d rkey=0x856d addr=0x3d2917000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:127 READ tags RKEY:0X856D VA:0X3D2917000 [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x1] [ +0.000002] iser: iser_fast_reg_mr: lkey=0x9b55 rkey=0x9b55 addr=0x3d2999000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:1 READ tags RKEY:0X9B55 VA:0X3D2999000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x2] [ +0.000002] iser: iser_fast_reg_mr: lkey=0x86cf rkey=0x86cf addr=0x3d2018000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:2 READ tags RKEY:0X86CF VA:0X3D2018000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x3] [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8062 rkey=0x8062 addr=0x3d2097000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:3 READ tags RKEY:0X8062 VA:0X3D2097000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x4] [ +0.000002] iser: iser_fast_reg_mr: lkey=0xc34b rkey=0xc34b addr=0x3d2116000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:4 READ tags RKEY:0XC34B VA:0X3D2116000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x5] [ +0.000002] iser: iser_fast_reg_mr: lkey=0x8b6d rkey=0x8b6d addr=0x3d2195000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:5 READ tags RKEY:0X8B6D VA:0X3D2195000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6] [ +0.000002] iser: iser_fast_reg_mr: lkey=0xce56 rkey=0xce56 addr=0x3d0e14000 length=0x7f000 [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:6 READ tags RKEY:0XCE56 VA:0X3D0E14000 [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7] [ +0.000002] iser: iser_fast_reg_mr: lkey=0xba45 rkey=0xba45 addr=0x3d0e93000 length=0x7f000 [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:7 READ tags RKEY:0XBA45 VA:0X3D0E93000 [ +0.000002] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x8] Hence, I feel iSER target SQ is undersized and needs to be sized properly to hold max possible entries. I might be wrong. Please take a look. Thanks, Krishna. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: iSERT SQ overflow with single target and multi luns 2020-07-07 12:36 iSERT SQ overflow with single target and multi luns Krishnamraju Eraparaju @ 2020-07-07 13:23 ` Max Gurtovoy 2020-07-07 17:16 ` Krishnamraju Eraparaju 0 siblings, 1 reply; 4+ messages in thread From: Max Gurtovoy @ 2020-07-07 13:23 UTC (permalink / raw) To: Krishnamraju Eraparaju, Sagi Grimberg Cc: linux-rdma, Nirranjan Kirubaharan, Potnuri Bharat Teja Hi Krishna, thanks for debugging this. please try the following untested patch: diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index b7df38e..49f5f05 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -119,7 +119,7 @@ { struct isert_device *device = isert_conn->device; struct ib_qp_init_attr attr; - int ret; + int ret, factor; memset(&attr, 0, sizeof(struct ib_qp_init_attr)); attr.event_handler = isert_qp_event_callback; @@ -128,7 +128,9 @@ attr.recv_cq = comp->cq; attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1; attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1; - attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX; + factor = rdma_rw_mr_factor(device->ib_device, cma_id->port_num, + ISCSI_ISER_MAX_SG_TABLESIZE); + attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX * factor; attr.cap.max_send_sge = device->ib_device->attrs.max_send_sge; attr.cap.max_recv_sge = 1; attr.sq_sig_type = IB_SIGNAL_REQ_WR; diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index 3b296ba..c9ccf1d 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -63,7 +63,8 @@ (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct ib_sge) + \ sizeof(struct ib_cqe) + sizeof(bool))) -#define ISCSI_ISER_SG_TABLESIZE 256 +/* Maximum support is 16MB I/O size */ +#define ISCSI_ISER_MAX_SG_TABLESIZE 4096 enum isert_desc_type { ISCSI_TX_CONTROL, On 7/7/2020 3:36 PM, Krishnamraju Eraparaju wrote: > Looks like the commit 07173c3e(block: enable multipage bvecs) has > uncovered iSER SQ sizing issue. > > Here is how I hit the issue: > Created two luns under single target, then run the below script on each > lun(parallelly). > > while [ 1 ] > do > iozone -i 0 -i 1 -I -+d -s 100000 -r 16384 -w > done > > > Then failures like below are logged in dmesg output, due to iw_cxgb4 SQ > getting full at iSER target. > "isert: isert_rdma_rw_ctx_post: Cmd: 00000000cb75342a failed to post > RDMA res" > > > This issue won't occur if luns are created on seperate targets. > Also, the issue won't occur if I revert the multipage bvecs(07173c3e) > changes at initator. > > > Currently SQ is being sized this way: > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + > (ISCSI_DEF_XMIT_CMDS_MAX:128 * factor:3) = 523. > I tried increaseing the SQ size and observed that the issue is not > occuring when attr.cap.max_send_wr is 562. > > > Looks like the avg length of RDMA READ/WRITE operations has increased > after "multipage bvecs" changes. > Queueing many large sized RDMA READ/WRITE WRs may cause backpressure and > increases the chances of SQ getting full at provider driver. > Notice the length(0x7f000 & 0x2000) of each RDMA READ operation below, > for Before and After case. > > Before "multipage bvecs" RDMA READ: > [ +0.001903] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x78] > [ +0.000007] iser: iser_fast_reg_mr: lkey=0x8a41 rkey=0x8a41 > addr=0x446166000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:120 READ tags > RKEY:0X8A41 VA:0X446166000 > [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6f] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x13b51 rkey=0x13b51 > addr=0x443b25000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:111 READ tags > RKEY:0X13B51 VA:0X443B25000 > [ +0.000022] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xe] > [ +0.000001] iser: iser_fast_reg_mr: lkey=0xa371 rkey=0xa371 > addr=0x4461a4000 length=0x2000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:14 READ tags > RKEY:0XA371 VA:0X4461A4000 > [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x79] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x12f4f rkey=0x12f4f > addr=0x4461a9000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:121 READ tags > RKEY:0X12F4F VA:0X4461A9000 > [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0xe040 rkey=0xe040 > addr=0x447e67000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags > RKEY:0XE040 VA:0X447E67000 > [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7b] > [ +0.000001] iser: iser_fast_reg_mr: lkey=0xb149 rkey=0xb149 > addr=0x3d0366000 length=0x2000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:123 READ tags > RKEY:0XB149 VA:0X3D0366000 > [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xb] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x1014c rkey=0x1014c > addr=0x3d0368000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:11 READ tags > RKEY:0X1014C VA:0X3D0368000 > [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x62] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x7c3b rkey=0x7c3b > addr=0x3d03e7000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:98 READ tags > RKEY:0X7C3B VA:0X3D03E7000 > [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x11] > [ +0.000001] iser: iser_fast_reg_mr: lkey=0x11752 rkey=0x11752 > addr=0x3d6de6000 length=0x2000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:17 READ tags > RKEY:0X11752 VA:0X3D6DE6000 > [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x77] > > > After "multipage bvecs" RDMA READ: > [ +0.002455] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] > [ +0.000006] iser: iser_fast_reg_mr: lkey=0x7991 rkey=0x7991 > addr=0x3d2819000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags > RKEY:0X7991 VA:0X3D2819000 > [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7e] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8c9b rkey=0x8c9b > addr=0x3d2898000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:126 READ tags > RKEY:0X8C9B VA:0X3D2898000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7f] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x856d rkey=0x856d > addr=0x3d2917000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:127 READ tags > RKEY:0X856D VA:0X3D2917000 > [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x1] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0x9b55 rkey=0x9b55 > addr=0x3d2999000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:1 READ tags > RKEY:0X9B55 VA:0X3D2999000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x2] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0x86cf rkey=0x86cf > addr=0x3d2018000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:2 READ tags > RKEY:0X86CF VA:0X3D2018000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x3] > [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8062 rkey=0x8062 > addr=0x3d2097000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:3 READ tags > RKEY:0X8062 VA:0X3D2097000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x4] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0xc34b rkey=0xc34b > addr=0x3d2116000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:4 READ tags > RKEY:0XC34B VA:0X3D2116000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x5] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0x8b6d rkey=0x8b6d > addr=0x3d2195000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:5 READ tags > RKEY:0X8B6D VA:0X3D2195000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0xce56 rkey=0xce56 > addr=0x3d0e14000 length=0x7f000 > [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:6 READ tags > RKEY:0XCE56 VA:0X3D0E14000 > [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7] > [ +0.000002] iser: iser_fast_reg_mr: lkey=0xba45 rkey=0xba45 > addr=0x3d0e93000 length=0x7f000 > [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:7 READ tags > RKEY:0XBA45 VA:0X3D0E93000 > [ +0.000002] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x8] > > Hence, I feel iSER target SQ is undersized and needs to be sized > properly to hold max possible entries. I might be wrong. > > Please take a look. > > Thanks, > Krishna. ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: iSERT SQ overflow with single target and multi luns 2020-07-07 13:23 ` Max Gurtovoy @ 2020-07-07 17:16 ` Krishnamraju Eraparaju 2020-07-07 18:53 ` Max Gurtovoy 0 siblings, 1 reply; 4+ messages in thread From: Krishnamraju Eraparaju @ 2020-07-07 17:16 UTC (permalink / raw) To: Max Gurtovoy Cc: Sagi Grimberg, linux-rdma, Nirranjan Kirubaharan, Potnuri Bharat Teja, krishna2 Hi Max, Thanks for the quick response! I just tested your patch, it's working without any issue. Just for reference, here is what I observed with Chelsio iWARP adapter, after applying your patch: attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + (rdma_ctxs:4096 * factor:3) = 12427 rdma_ctxs:4096 comes from ==> rdma_rw_mr_factor(ISCSI_ISER_MAX_SG_TABLESIZE:4096)=32 * ISCSI_DEF_XMIT_CMDS_MAX:128 where mr_pages is 128 Finally, 12427 size got capped to 8190 due to "dev->attrs.max_qp_wr" in rdma_rw_init_qp(). So final value of attr.cap.max_send_wr is 8190 for Chelsio adapters. Thanks, Krishna. On Tuesday, July 07/07/20, 2020 at 16:23:30 +0300, Max Gurtovoy wrote: > Hi Krishna, > > thanks for debugging this. > > please try the following untested patch: > > > diff --git a/drivers/infiniband/ulp/isert/ib_isert.c > b/drivers/infiniband/ulp/isert/ib_isert.c > index b7df38e..49f5f05 100644 > --- a/drivers/infiniband/ulp/isert/ib_isert.c > +++ b/drivers/infiniband/ulp/isert/ib_isert.c > @@ -119,7 +119,7 @@ > { > struct isert_device *device = isert_conn->device; > struct ib_qp_init_attr attr; > - int ret; > + int ret, factor; > > memset(&attr, 0, sizeof(struct ib_qp_init_attr)); > attr.event_handler = isert_qp_event_callback; > @@ -128,7 +128,9 @@ > attr.recv_cq = comp->cq; > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1; > attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1; > - attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX; > + factor = rdma_rw_mr_factor(device->ib_device, cma_id->port_num, > + ISCSI_ISER_MAX_SG_TABLESIZE); > + attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX * factor; > attr.cap.max_send_sge = device->ib_device->attrs.max_send_sge; > attr.cap.max_recv_sge = 1; > attr.sq_sig_type = IB_SIGNAL_REQ_WR; > diff --git a/drivers/infiniband/ulp/isert/ib_isert.h > b/drivers/infiniband/ulp/isert/ib_isert.h > index 3b296ba..c9ccf1d 100644 > --- a/drivers/infiniband/ulp/isert/ib_isert.h > +++ b/drivers/infiniband/ulp/isert/ib_isert.h > @@ -63,7 +63,8 @@ > (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct > ib_sge) + \ > sizeof(struct ib_cqe) + sizeof(bool))) > > -#define ISCSI_ISER_SG_TABLESIZE 256 > +/* Maximum support is 16MB I/O size */ > +#define ISCSI_ISER_MAX_SG_TABLESIZE 4096 > > enum isert_desc_type { > ISCSI_TX_CONTROL, > > > On 7/7/2020 3:36 PM, Krishnamraju Eraparaju wrote: > >Looks like the commit 07173c3e(block: enable multipage bvecs) has > >uncovered iSER SQ sizing issue. > > > >Here is how I hit the issue: > >Created two luns under single target, then run the below script on each > >lun(parallelly). > > > > while [ 1 ] > > do > > iozone -i 0 -i 1 -I -+d -s 100000 -r 16384 -w > > done > > > > > >Then failures like below are logged in dmesg output, due to iw_cxgb4 SQ > >getting full at iSER target. > > "isert: isert_rdma_rw_ctx_post: Cmd: 00000000cb75342a failed to post > >RDMA res" > > > > > >This issue won't occur if luns are created on seperate targets. > >Also, the issue won't occur if I revert the multipage bvecs(07173c3e) > >changes at initator. > > > > > >Currently SQ is being sized this way: > >attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + > >(ISCSI_DEF_XMIT_CMDS_MAX:128 * factor:3) = 523. > >I tried increaseing the SQ size and observed that the issue is not > >occuring when attr.cap.max_send_wr is 562. > > > > > >Looks like the avg length of RDMA READ/WRITE operations has increased > >after "multipage bvecs" changes. > >Queueing many large sized RDMA READ/WRITE WRs may cause backpressure and > >increases the chances of SQ getting full at provider driver. > >Notice the length(0x7f000 & 0x2000) of each RDMA READ operation below, > >for Before and After case. > > > >Before "multipage bvecs" RDMA READ: > >[ +0.001903] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x78] > >[ +0.000007] iser: iser_fast_reg_mr: lkey=0x8a41 rkey=0x8a41 > >addr=0x446166000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:120 READ tags > >RKEY:0X8A41 VA:0X446166000 > >[ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6f] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x13b51 rkey=0x13b51 > >addr=0x443b25000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:111 READ tags > >RKEY:0X13B51 VA:0X443B25000 > >[ +0.000022] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xe] > >[ +0.000001] iser: iser_fast_reg_mr: lkey=0xa371 rkey=0xa371 > >addr=0x4461a4000 length=0x2000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:14 READ tags > >RKEY:0XA371 VA:0X4461A4000 > >[ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x79] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x12f4f rkey=0x12f4f > >addr=0x4461a9000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:121 READ tags > >RKEY:0X12F4F VA:0X4461A9000 > >[ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0xe040 rkey=0xe040 > >addr=0x447e67000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags > >RKEY:0XE040 VA:0X447E67000 > >[ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7b] > >[ +0.000001] iser: iser_fast_reg_mr: lkey=0xb149 rkey=0xb149 > >addr=0x3d0366000 length=0x2000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:123 READ tags > >RKEY:0XB149 VA:0X3D0366000 > >[ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xb] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x1014c rkey=0x1014c > >addr=0x3d0368000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:11 READ tags > >RKEY:0X1014C VA:0X3D0368000 > >[ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x62] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x7c3b rkey=0x7c3b > >addr=0x3d03e7000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:98 READ tags > >RKEY:0X7C3B VA:0X3D03E7000 > >[ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x11] > >[ +0.000001] iser: iser_fast_reg_mr: lkey=0x11752 rkey=0x11752 > >addr=0x3d6de6000 length=0x2000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:17 READ tags > >RKEY:0X11752 VA:0X3D6DE6000 > >[ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x77] > > > > > >After "multipage bvecs" RDMA READ: > >[ +0.002455] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] > >[ +0.000006] iser: iser_fast_reg_mr: lkey=0x7991 rkey=0x7991 > >addr=0x3d2819000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags > >RKEY:0X7991 VA:0X3D2819000 > >[ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7e] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x8c9b rkey=0x8c9b > >addr=0x3d2898000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:126 READ tags > >RKEY:0X8C9B VA:0X3D2898000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7f] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x856d rkey=0x856d > >addr=0x3d2917000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:127 READ tags > >RKEY:0X856D VA:0X3D2917000 > >[ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x1] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0x9b55 rkey=0x9b55 > >addr=0x3d2999000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:1 READ tags > >RKEY:0X9B55 VA:0X3D2999000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x2] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0x86cf rkey=0x86cf > >addr=0x3d2018000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:2 READ tags > >RKEY:0X86CF VA:0X3D2018000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x3] > >[ +0.000003] iser: iser_fast_reg_mr: lkey=0x8062 rkey=0x8062 > >addr=0x3d2097000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:3 READ tags > >RKEY:0X8062 VA:0X3D2097000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x4] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0xc34b rkey=0xc34b > >addr=0x3d2116000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:4 READ tags > >RKEY:0XC34B VA:0X3D2116000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x5] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0x8b6d rkey=0x8b6d > >addr=0x3d2195000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:5 READ tags > >RKEY:0X8B6D VA:0X3D2195000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0xce56 rkey=0xce56 > >addr=0x3d0e14000 length=0x7f000 > >[ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:6 READ tags > >RKEY:0XCE56 VA:0X3D0E14000 > >[ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7] > >[ +0.000002] iser: iser_fast_reg_mr: lkey=0xba45 rkey=0xba45 > >addr=0x3d0e93000 length=0x7f000 > >[ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:7 READ tags > >RKEY:0XBA45 VA:0X3D0E93000 > >[ +0.000002] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x8] > > > >Hence, I feel iSER target SQ is undersized and needs to be sized > >properly to hold max possible entries. I might be wrong. > > > >Please take a look. > > > >Thanks, > >Krishna. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: iSERT SQ overflow with single target and multi luns 2020-07-07 17:16 ` Krishnamraju Eraparaju @ 2020-07-07 18:53 ` Max Gurtovoy 0 siblings, 0 replies; 4+ messages in thread From: Max Gurtovoy @ 2020-07-07 18:53 UTC (permalink / raw) To: Krishnamraju Eraparaju Cc: Sagi Grimberg, linux-rdma, Nirranjan Kirubaharan, Potnuri Bharat Teja great news. Sagi, can you review the below before I send the formal patch ? BTW, we solved a similar issue with NVMf/RDMA target recently.. On 7/7/2020 8:16 PM, Krishnamraju Eraparaju wrote: > Hi Max, > > Thanks for the quick response! > > I just tested your patch, it's working without any issue. > > > Just for reference, here is what I observed with Chelsio iWARP adapter, > after applying your patch: > > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + (rdma_ctxs:4096 * > factor:3) = 12427 > > > rdma_ctxs:4096 comes from ==> > rdma_rw_mr_factor(ISCSI_ISER_MAX_SG_TABLESIZE:4096)=32 * > ISCSI_DEF_XMIT_CMDS_MAX:128 where mr_pages is 128 > > > Finally, 12427 size got capped to 8190 due to "dev->attrs.max_qp_wr" in > rdma_rw_init_qp(). > > So final value of attr.cap.max_send_wr is 8190 for Chelsio adapters. > > > > Thanks, > Krishna. > On Tuesday, July 07/07/20, 2020 at 16:23:30 +0300, Max Gurtovoy wrote: >> Hi Krishna, >> >> thanks for debugging this. >> >> please try the following untested patch: >> >> >> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c >> b/drivers/infiniband/ulp/isert/ib_isert.c >> index b7df38e..49f5f05 100644 >> --- a/drivers/infiniband/ulp/isert/ib_isert.c >> +++ b/drivers/infiniband/ulp/isert/ib_isert.c >> @@ -119,7 +119,7 @@ >> { >> struct isert_device *device = isert_conn->device; >> struct ib_qp_init_attr attr; >> - int ret; >> + int ret, factor; >> >> memset(&attr, 0, sizeof(struct ib_qp_init_attr)); >> attr.event_handler = isert_qp_event_callback; >> @@ -128,7 +128,9 @@ >> attr.recv_cq = comp->cq; >> attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1; >> attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1; >> - attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX; >> + factor = rdma_rw_mr_factor(device->ib_device, cma_id->port_num, >> + ISCSI_ISER_MAX_SG_TABLESIZE); >> + attr.cap.max_rdma_ctxs = ISCSI_DEF_XMIT_CMDS_MAX * factor; >> attr.cap.max_send_sge = device->ib_device->attrs.max_send_sge; >> attr.cap.max_recv_sge = 1; >> attr.sq_sig_type = IB_SIGNAL_REQ_WR; >> diff --git a/drivers/infiniband/ulp/isert/ib_isert.h >> b/drivers/infiniband/ulp/isert/ib_isert.h >> index 3b296ba..c9ccf1d 100644 >> --- a/drivers/infiniband/ulp/isert/ib_isert.h >> +++ b/drivers/infiniband/ulp/isert/ib_isert.h >> @@ -63,7 +63,8 @@ >> (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct >> ib_sge) + \ >> sizeof(struct ib_cqe) + sizeof(bool))) >> >> -#define ISCSI_ISER_SG_TABLESIZE 256 >> +/* Maximum support is 16MB I/O size */ >> +#define ISCSI_ISER_MAX_SG_TABLESIZE 4096 >> >> enum isert_desc_type { >> ISCSI_TX_CONTROL, >> >> >> On 7/7/2020 3:36 PM, Krishnamraju Eraparaju wrote: >>> Looks like the commit 07173c3e(block: enable multipage bvecs) has >>> uncovered iSER SQ sizing issue. >>> >>> Here is how I hit the issue: >>> Created two luns under single target, then run the below script on each >>> lun(parallelly). >>> >>> while [ 1 ] >>> do >>> iozone -i 0 -i 1 -I -+d -s 100000 -r 16384 -w >>> done >>> >>> >>> Then failures like below are logged in dmesg output, due to iw_cxgb4 SQ >>> getting full at iSER target. >>> "isert: isert_rdma_rw_ctx_post: Cmd: 00000000cb75342a failed to post >>> RDMA res" >>> >>> >>> This issue won't occur if luns are created on seperate targets. >>> Also, the issue won't occur if I revert the multipage bvecs(07173c3e) >>> changes at initator. >>> >>> >>> Currently SQ is being sized this way: >>> attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS:138 +1 + >>> (ISCSI_DEF_XMIT_CMDS_MAX:128 * factor:3) = 523. >>> I tried increaseing the SQ size and observed that the issue is not >>> occuring when attr.cap.max_send_wr is 562. >>> >>> >>> Looks like the avg length of RDMA READ/WRITE operations has increased >>> after "multipage bvecs" changes. >>> Queueing many large sized RDMA READ/WRITE WRs may cause backpressure and >>> increases the chances of SQ getting full at provider driver. >>> Notice the length(0x7f000 & 0x2000) of each RDMA READ operation below, >>> for Before and After case. >>> >>> Before "multipage bvecs" RDMA READ: >>> [ +0.001903] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x78] >>> [ +0.000007] iser: iser_fast_reg_mr: lkey=0x8a41 rkey=0x8a41 >>> addr=0x446166000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:120 READ tags >>> RKEY:0X8A41 VA:0X446166000 >>> [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6f] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x13b51 rkey=0x13b51 >>> addr=0x443b25000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:111 READ tags >>> RKEY:0X13B51 VA:0X443B25000 >>> [ +0.000022] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xe] >>> [ +0.000001] iser: iser_fast_reg_mr: lkey=0xa371 rkey=0xa371 >>> addr=0x4461a4000 length=0x2000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:14 READ tags >>> RKEY:0XA371 VA:0X4461A4000 >>> [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x79] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x12f4f rkey=0x12f4f >>> addr=0x4461a9000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:121 READ tags >>> RKEY:0X12F4F VA:0X4461A9000 >>> [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0xe040 rkey=0xe040 >>> addr=0x447e67000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags >>> RKEY:0XE040 VA:0X447E67000 >>> [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7b] >>> [ +0.000001] iser: iser_fast_reg_mr: lkey=0xb149 rkey=0xb149 >>> addr=0x3d0366000 length=0x2000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:123 READ tags >>> RKEY:0XB149 VA:0X3D0366000 >>> [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0xb] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x1014c rkey=0x1014c >>> addr=0x3d0368000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:11 READ tags >>> RKEY:0X1014C VA:0X3D0368000 >>> [ +0.000007] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x62] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x7c3b rkey=0x7c3b >>> addr=0x3d03e7000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:98 READ tags >>> RKEY:0X7C3B VA:0X3D03E7000 >>> [ +0.000021] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x11] >>> [ +0.000001] iser: iser_fast_reg_mr: lkey=0x11752 rkey=0x11752 >>> addr=0x3d6de6000 length=0x2000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:17 READ tags >>> RKEY:0X11752 VA:0X3D6DE6000 >>> [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x77] >>> >>> >>> After "multipage bvecs" RDMA READ: >>> [ +0.002455] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7d] >>> [ +0.000006] iser: iser_fast_reg_mr: lkey=0x7991 rkey=0x7991 >>> addr=0x3d2819000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:125 READ tags >>> RKEY:0X7991 VA:0X3D2819000 >>> [ +0.000005] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7e] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8c9b rkey=0x8c9b >>> addr=0x3d2898000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:126 READ tags >>> RKEY:0X8C9B VA:0X3D2898000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7f] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x856d rkey=0x856d >>> addr=0x3d2917000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:127 READ tags >>> RKEY:0X856D VA:0X3D2917000 >>> [ +0.000004] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x1] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0x9b55 rkey=0x9b55 >>> addr=0x3d2999000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:1 READ tags >>> RKEY:0X9B55 VA:0X3D2999000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x2] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0x86cf rkey=0x86cf >>> addr=0x3d2018000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:2 READ tags >>> RKEY:0X86CF VA:0X3D2018000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x3] >>> [ +0.000003] iser: iser_fast_reg_mr: lkey=0x8062 rkey=0x8062 >>> addr=0x3d2097000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:3 READ tags >>> RKEY:0X8062 VA:0X3D2097000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x4] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0xc34b rkey=0xc34b >>> addr=0x3d2116000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:4 READ tags >>> RKEY:0XC34B VA:0X3D2116000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x5] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0x8b6d rkey=0x8b6d >>> addr=0x3d2195000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:5 READ tags >>> RKEY:0X8B6D VA:0X3D2195000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x6] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0xce56 rkey=0xce56 >>> addr=0x3d0e14000 length=0x7f000 >>> [ +0.000000] iser: iser_prepare_read_cmd: Cmd itt:6 READ tags >>> RKEY:0XCE56 VA:0X3D0E14000 >>> [ +0.000003] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x7] >>> [ +0.000002] iser: iser_fast_reg_mr: lkey=0xba45 rkey=0xba45 >>> addr=0x3d0e93000 length=0x7f000 >>> [ +0.000001] iser: iser_prepare_read_cmd: Cmd itt:7 READ tags >>> RKEY:0XBA45 VA:0X3D0E93000 >>> [ +0.000002] iser: iscsi_iser_task_xmit: ctask xmit [cid 0 itt 0x8] >>> >>> Hence, I feel iSER target SQ is undersized and needs to be sized >>> properly to hold max possible entries. I might be wrong. >>> >>> Please take a look. >>> >>> Thanks, >>> Krishna. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-07-07 18:53 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-07 12:36 iSERT SQ overflow with single target and multi luns Krishnamraju Eraparaju 2020-07-07 13:23 ` Max Gurtovoy 2020-07-07 17:16 ` Krishnamraju Eraparaju 2020-07-07 18:53 ` Max Gurtovoy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).