From: Chuck Lever <chuck.lever@oracle.com>
To: Anna Schumaker <schumaker.anna@gmail.com>
Cc: linux-rdma <linux-rdma@vger.kernel.org>,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR)
Date: Tue, 20 Nov 2018 13:07:34 -0500 [thread overview]
Message-ID: <08D50748-8C07-4EF1-B2BB-3DA30A3598BF@oracle.com> (raw)
In-Reply-To: <f1d83adeef9acf690c523df43c2f542b97a0bc33.camel@gmail.com>
> On Nov 20, 2018, at 1:02 PM, Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> Hi Chuck,
>
> On Mon, 2018-11-19 at 10:46 -0500, Chuck Lever wrote:
>> Place the associated RPC transaction's XID in the upper 32 bits of
>> each RDMA segment's rdma_offset field. These bits are currently
>> always zero.
>>
>> There are two reasons to do this:
>>
>> - The R_key only has 8 bits that are different from registration to
>> registration. The XID adds more uniqueness to each RDMA segment to
>> reduce the likelihood of a software bug on the server reading from
>> or writing into memory it's not supposed to.
>>
>> - On-the-wire RDMA Read and Write operations do not otherwise carry
>> any identifier that matches them up to an RPC. The XID in the
>> upper 32 bits will act as an eye-catcher in network captures.
>>
>> Suggested-by: Tom Talpey <ttalpey@microsoft.com>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> net/sunrpc/xprtrdma/frwr_ops.c | 3 ++-
>> net/sunrpc/xprtrdma/rpc_rdma.c | 6 +++---
>> net/sunrpc/xprtrdma/xprt_rdma.h | 2 +-
>> 3 files changed, 6 insertions(+), 5 deletions(-)
>>
>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
>> index 49b314d..3b260d2 100644
>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>> @@ -344,7 +344,7 @@
>> */
>> static struct rpcrdma_mr_seg *
>> frwr_op_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg,
>> - int nsegs, bool writing, struct rpcrdma_mr **out)
>> + int nsegs, bool writing, u32 xid, struct rpcrdma_mr **out)
>> {
>> struct rpcrdma_ia *ia = &r_xprt->rx_ia;
>> bool holes_ok = ia->ri_mrtype == IB_MR_TYPE_SG_GAPS;
>> @@ -398,6 +398,7 @@
>> if (unlikely(n != mr->mr_nents))
>> goto out_mapmr_err;
>>
>> + ibmr->iova |= ((u64)cpu_to_be32(xid)) << 32;
>
> My mount command hangs once we make this change (I got bored and killed it after
> about 5 minutes). This is with NFS over soft-RoCE in a kvm virtual machine, and
> I see the behavior with all NFS versions.
>
> I hope this helps!
OK. Can you capture ftrace output to show us where the hang occurs?
I use this on the NFS client:
# trace-cmd record -e sunrpc -e rpcrdma
^C
# trace-cmd report | less (or > /tmp/trace.log)
Thanks!
> Anna
>
>> key = (u8)(ibmr->rkey & 0x000000FF);
>> ib_update_fast_reg_key(ibmr, ++key);
>>
>> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
>> index 9f53e02..89a2db2 100644
>> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
>> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
>> @@ -357,7 +357,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt
>> *r_xprt,
>>
>> do {
>> seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs,
>> - false, &mr);
>> + false, rqst->rq_xid, &mr);
>> if (IS_ERR(seg))
>> return PTR_ERR(seg);
>> rpcrdma_mr_push(mr, &req->rl_registered);
>> @@ -415,7 +415,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt
>> *r_xprt,
>> nchunks = 0;
>> do {
>> seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs,
>> - true, &mr);
>> + true, rqst->rq_xid, &mr);
>> if (IS_ERR(seg))
>> return PTR_ERR(seg);
>> rpcrdma_mr_push(mr, &req->rl_registered);
>> @@ -473,7 +473,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt
>> *r_xprt,
>> nchunks = 0;
>> do {
>> seg = r_xprt->rx_ia.ri_ops->ro_map(r_xprt, seg, nsegs,
>> - true, &mr);
>> + true, rqst->rq_xid, &mr);
>> if (IS_ERR(seg))
>> return PTR_ERR(seg);
>> rpcrdma_mr_push(mr, &req->rl_registered);
>> diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h
>> index a13ccb6..2ae1ee2 100644
>> --- a/net/sunrpc/xprtrdma/xprt_rdma.h
>> +++ b/net/sunrpc/xprtrdma/xprt_rdma.h
>> @@ -472,7 +472,7 @@ struct rpcrdma_memreg_ops {
>> struct rpcrdma_mr_seg *
>> (*ro_map)(struct rpcrdma_xprt *,
>> struct rpcrdma_mr_seg *, int, bool,
>> - struct rpcrdma_mr **);
>> + u32, struct rpcrdma_mr **);
>> int (*ro_send)(struct rpcrdma_ia *ia,
>> struct rpcrdma_req *req);
>> void (*ro_reminv)(struct rpcrdma_rep *rep,
--
Chuck Lever
next prev parent reply other threads:[~2018-11-20 18:07 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-19 15:45 [PATCH v1 0/4] NFS/RDMA client for v4.21 (part 1) Chuck Lever
2018-11-19 15:45 ` [PATCH v1 1/4] xprtrdma: Remove support for FMR memory registration Chuck Lever
2018-11-19 16:16 ` Bart Van Assche
2018-11-19 19:09 ` Leon Romanovsky
2018-11-19 20:52 ` Bart Van Assche
2018-11-20 5:37 ` Leon Romanovsky
2018-11-19 22:41 ` Jason Gunthorpe
2018-11-19 22:56 ` Chuck Lever
2018-11-19 23:10 ` Jason Gunthorpe
2018-11-20 15:22 ` Dennis Dalessandro
2018-11-19 15:45 ` [PATCH v1 2/4] xprtrdma: mrs_create off-by-one Chuck Lever
2018-11-19 15:46 ` [PATCH v1 3/4] xprtrdma: Reduce max_frwr_depth Chuck Lever
2018-11-19 15:46 ` [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR) Chuck Lever
2018-11-19 17:47 ` Olga Kornievskaia
2018-11-19 17:58 ` Chuck Lever
2018-11-19 18:08 ` Olga Kornievskaia
2018-11-19 18:18 ` Chuck Lever
2018-11-19 18:47 ` Olga Kornievskaia
2018-11-19 18:58 ` Chuck Lever
2018-11-19 21:22 ` Olga Kornievskaia
2018-11-19 21:32 ` Chuck Lever
2018-11-19 21:42 ` Mora, Jorge
2018-11-19 22:46 ` Jason Gunthorpe
2018-11-20 2:45 ` Tom Talpey
2018-11-20 3:09 ` Jason Gunthorpe
2018-11-20 3:25 ` Tom Talpey
2018-11-20 3:32 ` Jason Gunthorpe
2018-11-20 3:38 ` Tom Talpey
2018-11-20 18:02 ` Anna Schumaker
2018-11-20 18:07 ` Chuck Lever [this message]
[not found] ` <94ff7ec712e086bfdd9c217a5f97c293a07151b9.camel@gmail.com>
2018-11-20 21:31 ` Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=08D50748-8C07-4EF1-B2BB-3DA30A3598BF@oracle.com \
--to=chuck.lever@oracle.com \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=schumaker.anna@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).