From: Chuck Lever <chuck.lever@oracle.com>
To: Anna Schumaker <schumaker.anna@gmail.com>
Cc: linux-rdma@vger.kernel.org,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v4 07/30] xprtrdma: Fix ri_max_segs and the result of ro_maxpages
Date: Tue, 18 Dec 2018 14:39:24 -0500 [thread overview]
Message-ID: <66C22632-967A-40C9-B30C-F407920E4DBA@oracle.com> (raw)
In-Reply-To: <ca236b957f74998d0104d9865b2d65943850d5b4.camel@gmail.com>
> On Dec 18, 2018, at 2:35 PM, Anna Schumaker <schumaker.anna@gmail.com> wrote:
>
> Hi Chuck,
>
> On Mon, 2018-12-17 at 11:39 -0500, Chuck Lever wrote:
>> With certain combinations of krb5i/p, MR size, and r/wsize, I/O can
>> fail with EMSGSIZE. This is because the calculated value of
>> ri_max_segs (the max number of MRs per RPC) exceeded
>> RPCRDMA_MAX_HDR_SEGS, which caused Read or Write list encoding to
>> walk off the end of the transport header.
>>
>> Once that was addressed, the ro_maxpages result has to be corrected
>> to account for the number of MRs needed for Reply chunks, which is
>> 2 MRs smaller than a normal Read or Write chunk.
>>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> net/sunrpc/xprtrdma/fmr_ops.c | 7 +++++--
>> net/sunrpc/xprtrdma/frwr_ops.c | 7 +++++--
>> net/sunrpc/xprtrdma/transport.c | 6 ++++--
>> 3 files changed, 14 insertions(+), 6 deletions(-)
>>
>> diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c
>> index 7f5632c..78a0224 100644
>> --- a/net/sunrpc/xprtrdma/fmr_ops.c
>> +++ b/net/sunrpc/xprtrdma/fmr_ops.c
>> @@ -176,7 +176,10 @@ enum {
>>
>> ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS /
>> RPCRDMA_MAX_FMR_SGES);
>> - ia->ri_max_segs += 2; /* segments for head and tail buffers */
>> + /* Reply chunks require segments for head and tail buffers */
>> + ia->ri_max_segs += 2;
>> + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS)
>> + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS;
>> return 0;
>> }
>>
>> @@ -186,7 +189,7 @@ enum {
>> fmr_op_maxpages(struct rpcrdma_xprt *r_xprt)
>> {
>> return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS,
>> - RPCRDMA_MAX_HDR_SEGS * RPCRDMA_MAX_FMR_SGES);
>> + (ia->ri_max_segs - 2) * RPCRDMA_MAX_FMR_SGES);
>
> ia isn't defined in this function. Should that be r_xprt->rx_ia.ri_max_segs
> instead?
Yeah. I compile-tested the tip of the branch, and so missed
this.
I'm going to have to send you a v5, so I'll fix this up in
the next post of this series.
> Thanks,
> Anna
>
>> }
>>
>> /* Use the ib_map_phys_fmr() verb to register a memory region
>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
>> index 27222c0..f587e44 100644
>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>> @@ -244,7 +244,10 @@
>>
>> ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS /
>> ia->ri_max_frwr_depth);
>> - ia->ri_max_segs += 2; /* segments for head and tail buffers */
>> + /* Reply chunks require segments for head and tail buffers */
>> + ia->ri_max_segs += 2;
>> + if (ia->ri_max_segs > RPCRDMA_MAX_HDR_SEGS)
>> + ia->ri_max_segs = RPCRDMA_MAX_HDR_SEGS;
>> return 0;
>> }
>>
>> @@ -257,7 +260,7 @@
>> struct rpcrdma_ia *ia = &r_xprt->rx_ia;
>>
>> return min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS,
>> - RPCRDMA_MAX_HDR_SEGS * ia->ri_max_frwr_depth);
>> + (ia->ri_max_segs - 2) * ia->ri_max_frwr_depth);
>> }
>>
>> static void
>> diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
>> index a16296b..fbb14bf 100644
>> --- a/net/sunrpc/xprtrdma/transport.c
>> +++ b/net/sunrpc/xprtrdma/transport.c
>> @@ -704,8 +704,10 @@
>> * %-ENOTCONN if the caller should reconnect and call again
>> * %-EAGAIN if the caller should call again
>> * %-ENOBUFS if the caller should call again after a delay
>> - * %-EIO if a permanent error occurred and the request was not
>> - * sent. Do not try to send this message again.
>> + * %-EMSGSIZE if encoding ran out of buffer space. The request
>> + * was not sent. Do not try to send this message again.
>> + * %-EIO if an I/O error occurred. The request was not sent.
>> + * Do not try to send this message again.
>> */
>> static int
>> xprt_rdma_send_request(struct rpc_rqst *rqst)
--
Chuck Lever
next prev parent reply other threads:[~2018-12-18 19:39 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-17 16:39 [PATCH v4 00/30] NFS/RDMA client for next Chuck Lever
2018-12-17 16:39 ` [PATCH v4 01/30] xprtrdma: Yet another double DMA-unmap Chuck Lever
2018-12-17 16:39 ` [PATCH v4 02/30] xprtrdma: Ensure MRs are DMA-unmapped when posting LOCAL_INV fails Chuck Lever
2018-12-17 16:39 ` [PATCH v4 03/30] xprtrdma: Refactor Receive accounting Chuck Lever
2018-12-17 16:39 ` [PATCH v4 04/30] xprtrdma: Replace rpcrdma_receive_wq with a per-xprt workqueue Chuck Lever
2018-12-17 16:39 ` [PATCH v4 05/30] xprtrdma: No qp_event disconnect Chuck Lever
2018-12-17 16:39 ` [PATCH v4 06/30] xprtrdma: Don't wake pending tasks until disconnect is done Chuck Lever
2018-12-17 17:28 ` Trond Myklebust
2018-12-17 18:37 ` Chuck Lever
2018-12-17 18:55 ` Trond Myklebust
2018-12-17 19:00 ` Chuck Lever
2018-12-17 19:09 ` Trond Myklebust
2018-12-17 19:19 ` Chuck Lever
2018-12-17 19:26 ` Trond Myklebust
2018-12-17 16:39 ` [PATCH v4 07/30] xprtrdma: Fix ri_max_segs and the result of ro_maxpages Chuck Lever
2018-12-18 19:35 ` Anna Schumaker
2018-12-18 19:39 ` Chuck Lever [this message]
2018-12-17 16:40 ` [PATCH v4 08/30] xprtrdma: Reduce max_frwr_depth Chuck Lever
2018-12-17 16:40 ` [PATCH v4 09/30] xprtrdma: Remove support for FMR memory registration Chuck Lever
2018-12-17 16:40 ` [PATCH v4 10/30] xprtrdma: Remove rpcrdma_memreg_ops Chuck Lever
2018-12-17 16:40 ` [PATCH v4 11/30] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR) Chuck Lever
2018-12-17 16:40 ` [PATCH v4 12/30] NFS: Make "port=" mount option optional for RDMA mounts Chuck Lever
2018-12-17 16:40 ` [PATCH v4 13/30] xprtrdma: Recognize XDRBUF_SPARSE_PAGES Chuck Lever
2018-12-17 16:40 ` [PATCH v4 14/30] xprtrdma: Remove request_module from backchannel Chuck Lever
2018-12-17 16:40 ` [PATCH v4 15/30] xprtrdma: Expose transport header errors Chuck Lever
2018-12-17 16:40 ` [PATCH v4 16/30] xprtrdma: Simplify locking that protects the rl_allreqs list Chuck Lever
2018-12-17 16:40 ` [PATCH v4 17/30] xprtrdma: Cull dprintk() call sites Chuck Lever
2018-12-17 16:40 ` [PATCH v4 18/30] xprtrdma: Remove unused fields from rpcrdma_ia Chuck Lever
2018-12-17 16:41 ` [PATCH v4 19/30] xprtrdma: Clean up of xprtrdma chunk trace points Chuck Lever
2018-12-17 16:41 ` [PATCH v4 20/30] xprtrdma: Relocate the xprtrdma_mr_map " Chuck Lever
2018-12-17 16:41 ` [PATCH v4 21/30] xprtrdma: Add trace points for calls to transport switch methods Chuck Lever
2018-12-17 16:41 ` [PATCH v4 22/30] xprtrdma: Trace mapping, alloc, and dereg failures Chuck Lever
2018-12-17 16:41 ` [PATCH v4 23/30] NFS: Fix NFSv4 symbolic trace point output Chuck Lever
2018-12-17 16:41 ` [PATCH v4 24/30] SUNRPC: Simplify defining common RPC trace events Chuck Lever
2018-12-17 16:41 ` [PATCH v4 25/30] SUNRPC: Fix some kernel doc complaints Chuck Lever
2018-12-17 16:41 ` [PATCH v4 26/30] xprtrdma: Update comments in frwr_op_send Chuck Lever
2018-12-17 16:41 ` [PATCH v4 27/30] xprtrdma: Replace outdated comment for rpcrdma_ep_post Chuck Lever
2018-12-17 16:41 ` [PATCH v4 28/30] xprtrdma: Add documenting comment for rpcrdma_buffer_destroy Chuck Lever
2018-12-17 16:41 ` [PATCH v4 29/30] xprtrdma: Clarify comments in rpcrdma_ia_remove Chuck Lever
2018-12-17 16:42 ` [PATCH v4 30/30] xprtrdma: Don't leak freed MRs Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=66C22632-967A-40C9-B30C-F407920E4DBA@oracle.com \
--to=chuck.lever@oracle.com \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=schumaker.anna@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).