From: Potnuri Bharat Teja <bharat-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org> To: nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org, sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org Cc: target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, bharat-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org Subject: [PATCH RFC] iser-target: avoid reinitializing rdma contexts for isert commands Date: Tue, 28 Nov 2017 23:58:07 +0530 [thread overview] Message-ID: <1511893687-29298-1-git-send-email-bharat@chelsio.com> (raw) isert commands that failed during isert_rdma_rw_ctx_post() are queued to Queue-Full(QF) queue and are scheduled to be reposted during queue-full queue processing. During this reposting, the rdma contexts are initialised again in isert_rdma_rw_ctx_post(), which is leaking significant memory. unreferenced object 0xffff8830201d9640 (size 64): comm "kworker/0:2", pid 195, jiffies 4295374851 (age 4528.436s) hex dump (first 32 bytes): 00 60 8b cb 2e 00 00 00 00 10 00 00 00 00 00 00 .`.............. 00 90 e3 cb 2e 00 00 00 00 10 00 00 00 00 00 00 ................ backtrace: [<ffffffff8170711e>] kmemleak_alloc+0x4e/0xb0 [<ffffffff811f8ba5>] __kmalloc+0x125/0x2b0 [<ffffffffa046b24f>] rdma_rw_ctx_init+0x15f/0x6f0 [ib_core] [<ffffffffa07ab644>] isert_rdma_rw_ctx_post+0xc4/0x3c0 [ib_isert] [<ffffffffa07ad972>] isert_put_datain+0x112/0x1c0 [ib_isert] [<ffffffffa07dddce>] lio_queue_data_in+0x2e/0x30 [iscsi_target_mod] [<ffffffffa076c322>] target_qf_do_work+0x2b2/0x4b0 [target_core_mod] [<ffffffff81080c3b>] process_one_work+0x1db/0x5d0 [<ffffffff8108107d>] worker_thread+0x4d/0x3e0 [<ffffffff81088667>] kthread+0x117/0x150 [<ffffffff81713fa7>] ret_from_fork+0x27/0x40 [<ffffffffffffffff>] 0xffffffffffffffff Here is patch to use the older rdma contexts while reposting the isert commands intead of reinitialising them. Signed-off-by: Potnuri Bharat Teja <bharat-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org> --- drivers/infiniband/ulp/isert/ib_isert.c | 7 +++++++ drivers/infiniband/ulp/isert/ib_isert.h | 1 + 2 files changed, 8 insertions(+) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 720dfb3a1ac2..fd55163801a3 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2123,6 +2123,9 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, u32 rkey, offset; int ret; + if (cmd->ctx_init_done) + goto rdma_ctx_post; + if (dir == DMA_FROM_DEVICE) { addr = cmd->write_va; rkey = cmd->write_stag; @@ -2150,11 +2153,15 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, se_cmd->t_data_sg, se_cmd->t_data_nents, offset, addr, rkey, dir); } + if (ret < 0) { isert_err("Cmd: %p failed to prepare RDMA res\n", cmd); return ret; } + cmd->ctx_init_done = true; + +rdma_ctx_post: ret = rdma_rw_ctx_post(&cmd->rw, conn->qp, port_num, cqe, chain_wr); if (ret < 0) isert_err("Cmd: %p failed to post RDMA res\n", cmd); diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index d6fd248320ae..3b296bac4f60 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -126,6 +126,7 @@ struct isert_cmd { struct rdma_rw_ctx rw; struct work_struct comp_work; struct scatterlist sg; + bool ctx_init_done; }; static inline struct isert_cmd *tx_desc_to_cmd(struct iser_tx_desc *desc) -- 2.5.3 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Potnuri Bharat Teja <bharat@chelsio.com> To: nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org, sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org Cc: target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, bharat-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org Subject: [PATCH RFC] iser-target: avoid reinitializing rdma contexts for isert commands Date: Tue, 28 Nov 2017 18:40:07 +0000 [thread overview] Message-ID: <1511893687-29298-1-git-send-email-bharat@chelsio.com> (raw) isert commands that failed during isert_rdma_rw_ctx_post() are queued to Queue-Full(QF) queue and are scheduled to be reposted during queue-full queue processing. During this reposting, the rdma contexts are initialised again in isert_rdma_rw_ctx_post(), which is leaking significant memory. unreferenced object 0xffff8830201d9640 (size 64): comm "kworker/0:2", pid 195, jiffies 4295374851 (age 4528.436s) hex dump (first 32 bytes): 00 60 8b cb 2e 00 00 00 00 10 00 00 00 00 00 00 .`.............. 00 90 e3 cb 2e 00 00 00 00 10 00 00 00 00 00 00 ................ backtrace: [<ffffffff8170711e>] kmemleak_alloc+0x4e/0xb0 [<ffffffff811f8ba5>] __kmalloc+0x125/0x2b0 [<ffffffffa046b24f>] rdma_rw_ctx_init+0x15f/0x6f0 [ib_core] [<ffffffffa07ab644>] isert_rdma_rw_ctx_post+0xc4/0x3c0 [ib_isert] [<ffffffffa07ad972>] isert_put_datain+0x112/0x1c0 [ib_isert] [<ffffffffa07dddce>] lio_queue_data_in+0x2e/0x30 [iscsi_target_mod] [<ffffffffa076c322>] target_qf_do_work+0x2b2/0x4b0 [target_core_mod] [<ffffffff81080c3b>] process_one_work+0x1db/0x5d0 [<ffffffff8108107d>] worker_thread+0x4d/0x3e0 [<ffffffff81088667>] kthread+0x117/0x150 [<ffffffff81713fa7>] ret_from_fork+0x27/0x40 [<ffffffffffffffff>] 0xffffffffffffffff Here is patch to use the older rdma contexts while reposting the isert commands intead of reinitialising them. Signed-off-by: Potnuri Bharat Teja <bharat@chelsio.com> --- drivers/infiniband/ulp/isert/ib_isert.c | 7 +++++++ drivers/infiniband/ulp/isert/ib_isert.h | 1 + 2 files changed, 8 insertions(+) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 720dfb3a1ac2..fd55163801a3 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2123,6 +2123,9 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, u32 rkey, offset; int ret; + if (cmd->ctx_init_done) + goto rdma_ctx_post; + if (dir = DMA_FROM_DEVICE) { addr = cmd->write_va; rkey = cmd->write_stag; @@ -2150,11 +2153,15 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, se_cmd->t_data_sg, se_cmd->t_data_nents, offset, addr, rkey, dir); } + if (ret < 0) { isert_err("Cmd: %p failed to prepare RDMA res\n", cmd); return ret; } + cmd->ctx_init_done = true; + +rdma_ctx_post: ret = rdma_rw_ctx_post(&cmd->rw, conn->qp, port_num, cqe, chain_wr); if (ret < 0) isert_err("Cmd: %p failed to post RDMA res\n", cmd); diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index d6fd248320ae..3b296bac4f60 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -126,6 +126,7 @@ struct isert_cmd { struct rdma_rw_ctx rw; struct work_struct comp_work; struct scatterlist sg; + bool ctx_init_done; }; static inline struct isert_cmd *tx_desc_to_cmd(struct iser_tx_desc *desc) -- 2.5.3
next reply other threads:[~2017-11-28 18:28 UTC|newest] Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-11-28 18:28 Potnuri Bharat Teja [this message] 2017-11-28 18:40 ` [PATCH RFC] iser-target: avoid reinitializing rdma contexts for isert commands Potnuri Bharat Teja 2017-11-29 22:50 ` Sagi Grimberg 2017-11-29 22:50 ` Sagi Grimberg 2017-11-30 10:34 ` Potnuri Bharat Teja 2017-11-30 10:46 ` Potnuri Bharat Teja 2017-12-04 18:31 ` Sagi Grimberg 2017-12-04 18:31 ` Sagi Grimberg 2017-12-13 17:51 ` [RFC] " Jason Gunthorpe 2017-12-13 17:51 ` Jason Gunthorpe 2017-12-13 18:48 ` Potnuri Bharat Teja 2017-12-13 18:49 ` Potnuri Bharat Teja 2017-12-18 23:14 ` Jason Gunthorpe 2017-12-18 23:14 ` Jason Gunthorpe [not found] ` <20171218231439.GA31621-uk2M96/98Pc@public.gmane.org> 2018-01-13 6:20 ` Nicholas A. Bellinger 2018-01-13 6:20 ` Nicholas A. Bellinger [not found] ` <1515824443.24576.108.camel-XoQW25Eq2zs8TOCF0fvnoXxStJ4P+DSV@public.gmane.org> 2018-01-13 18:04 ` Jason Gunthorpe 2018-01-13 18:04 ` Jason Gunthorpe 2018-01-14 9:26 ` Sagi Grimberg 2018-01-14 9:26 ` Sagi Grimberg [not found] ` <143bd378-193e-0bbd-1ee3-25299af61aa8-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2018-01-14 18:23 ` Jason Gunthorpe 2018-01-14 18:23 ` Jason Gunthorpe [not found] ` <20180114182344.GB9088-uk2M96/98Pc@public.gmane.org> 2018-01-15 4:39 ` Nicholas A. Bellinger 2018-01-15 4:39 ` Nicholas A. Bellinger
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1511893687-29298-1-git-send-email-bharat@chelsio.com \ --to=bharat-ut6up61k2wzbdgjk7y7tuq@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org \ --cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \ --cc=target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.