From: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: [PATCH v3 4/6] svcrdma: Add infrastructure to send backwards direction RPC/RDMA calls Date: Mon, 07 Dec 2015 15:42:56 -0500 [thread overview] Message-ID: <20151207204256.12988.76473.stgit@klimt.1015granger.net> (raw) In-Reply-To: <20151207203851.12988.97804.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org> To support the NFSv4.1 backchannel on RDMA connections, add a mechanism for sending a backwards-direction RPC/RDMA call on a connection established by a client. Signed-off-by: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> --- include/linux/sunrpc/svc_rdma.h | 2 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index f71c625..bf9b17b 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -215,6 +215,8 @@ extern int rdma_read_chunk_frmr(struct svcxprt_rdma *, struct svc_rqst *, extern int svc_rdma_sendto(struct svc_rqst *); extern struct rpcrdma_read_chunk * svc_rdma_get_read_chunk(struct rpcrdma_msg *); +extern int svc_rdma_bc_post_send(struct svcxprt_rdma *, + struct svc_rdma_op_ctxt *, struct xdr_buf *); /* svc_rdma_transport.c */ extern int svc_rdma_send(struct svcxprt_rdma *, struct ib_send_wr *); diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index bad5eaa..846df63 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -648,3 +648,64 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) svc_rdma_put_context(ctxt, 0); return ret; } + +/* Send a backwards direction RPC call. + * + * Caller holds the connection's mutex and has already marshaled the + * RPC/RDMA request. Before sending the request, this API also posts + * an extra receive buffer to catch the bc reply for this request. + */ +int svc_rdma_bc_post_send(struct svcxprt_rdma *rdma, + struct svc_rdma_op_ctxt *ctxt, struct xdr_buf *sndbuf) +{ + struct svc_rdma_req_map *vec; + struct ib_send_wr send_wr; + int ret; + + vec = svc_rdma_get_req_map(); + ret = map_xdr(rdma, sndbuf, vec); + if (ret) + goto out; + + /* Post a recv buffer to handle reply for this request */ + ret = svc_rdma_post_recv(rdma); + if (ret) { + pr_err("svcrdma: Failed to post bc receive buffer, err=%d. " + "Closing transport %p.\n", ret, rdma); + set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags); + ret = -ENOTCONN; + goto out; + } + + ctxt->wr_op = IB_WR_SEND; + ctxt->direction = DMA_TO_DEVICE; + ctxt->sge[0].lkey = rdma->sc_dma_lkey; + ctxt->sge[0].length = sndbuf->len; + ctxt->sge[0].addr = + ib_dma_map_page(rdma->sc_cm_id->device, ctxt->pages[0], 0, + sndbuf->len, DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdma->sc_cm_id->device, ctxt->sge[0].addr)) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } + atomic_inc(&rdma->sc_dma_used); + + memset(&send_wr, 0, sizeof send_wr); + send_wr.wr_id = (unsigned long)ctxt; + send_wr.sg_list = ctxt->sge; + send_wr.num_sge = 1; + send_wr.opcode = IB_WR_SEND; + send_wr.send_flags = IB_SEND_SIGNALED; + + ret = svc_rdma_send(rdma, &send_wr); + if (ret) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } +out: + svc_rdma_put_req_map(vec); + dprintk("svcrdma: %s returns %d\n", __func__, ret); + return ret; +} -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Chuck Lever <chuck.lever@oracle.com> To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH v3 4/6] svcrdma: Add infrastructure to send backwards direction RPC/RDMA calls Date: Mon, 07 Dec 2015 15:42:56 -0500 [thread overview] Message-ID: <20151207204256.12988.76473.stgit@klimt.1015granger.net> (raw) In-Reply-To: <20151207203851.12988.97804.stgit@klimt.1015granger.net> To support the NFSv4.1 backchannel on RDMA connections, add a mechanism for sending a backwards-direction RPC/RDMA call on a connection established by a client. Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- include/linux/sunrpc/svc_rdma.h | 2 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index f71c625..bf9b17b 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -215,6 +215,8 @@ extern int rdma_read_chunk_frmr(struct svcxprt_rdma *, struct svc_rqst *, extern int svc_rdma_sendto(struct svc_rqst *); extern struct rpcrdma_read_chunk * svc_rdma_get_read_chunk(struct rpcrdma_msg *); +extern int svc_rdma_bc_post_send(struct svcxprt_rdma *, + struct svc_rdma_op_ctxt *, struct xdr_buf *); /* svc_rdma_transport.c */ extern int svc_rdma_send(struct svcxprt_rdma *, struct ib_send_wr *); diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index bad5eaa..846df63 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -648,3 +648,64 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) svc_rdma_put_context(ctxt, 0); return ret; } + +/* Send a backwards direction RPC call. + * + * Caller holds the connection's mutex and has already marshaled the + * RPC/RDMA request. Before sending the request, this API also posts + * an extra receive buffer to catch the bc reply for this request. + */ +int svc_rdma_bc_post_send(struct svcxprt_rdma *rdma, + struct svc_rdma_op_ctxt *ctxt, struct xdr_buf *sndbuf) +{ + struct svc_rdma_req_map *vec; + struct ib_send_wr send_wr; + int ret; + + vec = svc_rdma_get_req_map(); + ret = map_xdr(rdma, sndbuf, vec); + if (ret) + goto out; + + /* Post a recv buffer to handle reply for this request */ + ret = svc_rdma_post_recv(rdma); + if (ret) { + pr_err("svcrdma: Failed to post bc receive buffer, err=%d. " + "Closing transport %p.\n", ret, rdma); + set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags); + ret = -ENOTCONN; + goto out; + } + + ctxt->wr_op = IB_WR_SEND; + ctxt->direction = DMA_TO_DEVICE; + ctxt->sge[0].lkey = rdma->sc_dma_lkey; + ctxt->sge[0].length = sndbuf->len; + ctxt->sge[0].addr = + ib_dma_map_page(rdma->sc_cm_id->device, ctxt->pages[0], 0, + sndbuf->len, DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdma->sc_cm_id->device, ctxt->sge[0].addr)) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } + atomic_inc(&rdma->sc_dma_used); + + memset(&send_wr, 0, sizeof send_wr); + send_wr.wr_id = (unsigned long)ctxt; + send_wr.sg_list = ctxt->sge; + send_wr.num_sge = 1; + send_wr.opcode = IB_WR_SEND; + send_wr.send_flags = IB_SEND_SIGNALED; + + ret = svc_rdma_send(rdma, &send_wr); + if (ret) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } +out: + svc_rdma_put_req_map(vec); + dprintk("svcrdma: %s returns %d\n", __func__, ret); + return ret; +}
next prev parent reply other threads:[~2015-12-07 20:42 UTC|newest] Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-12-07 20:42 [PATCH v3 0/6] NFS/RDMA server patches for 4.5 Chuck Lever 2015-12-07 20:42 ` Chuck Lever [not found] ` <20151207203851.12988.97804.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2015-12-07 20:42 ` [PATCH v3 1/6] svcrdma: Do not send XDR roundup bytes for a write chunk Chuck Lever 2015-12-07 20:42 ` Chuck Lever [not found] ` <20151207204231.12988.59287.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2015-12-13 3:14 ` Tom Talpey 2015-12-13 3:14 ` Tom Talpey [not found] ` <566CE288.6000808-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-12-13 19:44 ` Chuck Lever 2015-12-13 19:44 ` Chuck Lever 2015-12-07 20:42 ` [PATCH v3 2/6] svcrdma: Improve allocation of struct svc_rdma_op_ctxt Chuck Lever 2015-12-07 20:42 ` Chuck Lever 2015-12-07 20:42 ` [PATCH v3 3/6] svcrdma: Define maximum number of backchannel requests Chuck Lever 2015-12-07 20:42 ` Chuck Lever 2015-12-07 20:42 ` Chuck Lever [this message] 2015-12-07 20:42 ` [PATCH v3 4/6] svcrdma: Add infrastructure to send backwards direction RPC/RDMA calls Chuck Lever 2015-12-07 20:43 ` [PATCH v3 5/6] svcrdma: Add infrastructure to receive backwards direction RPC/RDMA replies Chuck Lever 2015-12-07 20:43 ` Chuck Lever [not found] ` <20151207204305.12988.2749.stgit-Hs+gFlyCn65vLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2015-12-13 3:24 ` Tom Talpey 2015-12-13 3:24 ` Tom Talpey [not found] ` <566CE4E7.9050307-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-12-13 20:27 ` Chuck Lever 2015-12-13 20:27 ` Chuck Lever 2015-12-07 20:43 ` [PATCH v3 6/6] xprtrdma: Add class for RDMA backwards direction transport Chuck Lever 2015-12-07 20:43 ` Chuck Lever
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20151207204256.12988.76473.stgit@klimt.1015granger.net \ --to=chuck.lever-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.