From: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: anna.schumaker-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: [PATCH v3 04/12] SUNRPC: Proper metric accounting when RPC is not transmitted Date: Tue, 29 Nov 2016 10:52:32 -0500 [thread overview] Message-ID: <20161129155232.23061.9794.stgit@manet.1015granger.net> (raw) In-Reply-To: <20161129154845.23061.29385.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> I noticed recently that during an xfstests on a krb5i mount, the retransmit count for certain operations had gone negative, and the backlog value became unreasonably large. I recall that Andy has pointed this out to me in the past. When call_refresh fails to find a valid credential for an RPC, the RPC exits immediately without sending anything on the wire. This leaves rq_ntrans, rq_xtime, and rq_rtt set to zero. The solution for om_queue is to not add the to RPC's running backlog queue total whenever rq_xtime is zero. For om_ntrans, it's a bit more difficult. A zero rq_ntrans causes om_ops to become larger than om_ntrans. The design of the RPC metrics API assumes that ntrans will always be equal to or larger than the ops count. The result is that when an RPC fails to find credentials, the RPC operation's reported retransmit count, which is computed in user space as the difference between ops and ntrans, goes negative. Ideally the kernel API should report a separate retransmit and "exited before initial transmission" metric, so that user space can sort out the difference properly. To avoid kernel API changes and changes to the way rq_ntrans is used when performing transport locking, account for untransmitted RPCs so that om_ntrans keeps up with om_ops: always add one or more to om_ntrans. Signed-off-by: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> --- net/sunrpc/stats.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/stats.c b/net/sunrpc/stats.c index 2ecb994..caeb01a 100644 --- a/net/sunrpc/stats.c +++ b/net/sunrpc/stats.c @@ -157,15 +157,17 @@ void rpc_count_iostats_metrics(const struct rpc_task *task, spin_lock(&op_metrics->om_lock); op_metrics->om_ops++; - op_metrics->om_ntrans += req->rq_ntrans; + /* kernel API: om_ops must never become larger than om_ntrans */ + op_metrics->om_ntrans += max(req->rq_ntrans, 1); op_metrics->om_timeouts += task->tk_timeouts; op_metrics->om_bytes_sent += req->rq_xmit_bytes_sent; op_metrics->om_bytes_recv += req->rq_reply_bytes_recvd; - delta = ktime_sub(req->rq_xtime, task->tk_start); - op_metrics->om_queue = ktime_add(op_metrics->om_queue, delta); - + if (ktime_to_ns(req->rq_xtime)) { + delta = ktime_sub(req->rq_xtime, task->tk_start); + op_metrics->om_queue = ktime_add(op_metrics->om_queue, delta); + } op_metrics->om_rtt = ktime_add(op_metrics->om_rtt, req->rq_rtt); delta = ktime_sub(now, task->tk_start); -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Chuck Lever <chuck.lever@oracle.com> To: anna.schumaker@netapp.com Cc: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH v3 04/12] SUNRPC: Proper metric accounting when RPC is not transmitted Date: Tue, 29 Nov 2016 10:52:32 -0500 [thread overview] Message-ID: <20161129155232.23061.9794.stgit@manet.1015granger.net> (raw) In-Reply-To: <20161129154845.23061.29385.stgit@manet.1015granger.net> I noticed recently that during an xfstests on a krb5i mount, the retransmit count for certain operations had gone negative, and the backlog value became unreasonably large. I recall that Andy has pointed this out to me in the past. When call_refresh fails to find a valid credential for an RPC, the RPC exits immediately without sending anything on the wire. This leaves rq_ntrans, rq_xtime, and rq_rtt set to zero. The solution for om_queue is to not add the to RPC's running backlog queue total whenever rq_xtime is zero. For om_ntrans, it's a bit more difficult. A zero rq_ntrans causes om_ops to become larger than om_ntrans. The design of the RPC metrics API assumes that ntrans will always be equal to or larger than the ops count. The result is that when an RPC fails to find credentials, the RPC operation's reported retransmit count, which is computed in user space as the difference between ops and ntrans, goes negative. Ideally the kernel API should report a separate retransmit and "exited before initial transmission" metric, so that user space can sort out the difference properly. To avoid kernel API changes and changes to the way rq_ntrans is used when performing transport locking, account for untransmitted RPCs so that om_ntrans keeps up with om_ops: always add one or more to om_ntrans. Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- net/sunrpc/stats.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/stats.c b/net/sunrpc/stats.c index 2ecb994..caeb01a 100644 --- a/net/sunrpc/stats.c +++ b/net/sunrpc/stats.c @@ -157,15 +157,17 @@ void rpc_count_iostats_metrics(const struct rpc_task *task, spin_lock(&op_metrics->om_lock); op_metrics->om_ops++; - op_metrics->om_ntrans += req->rq_ntrans; + /* kernel API: om_ops must never become larger than om_ntrans */ + op_metrics->om_ntrans += max(req->rq_ntrans, 1); op_metrics->om_timeouts += task->tk_timeouts; op_metrics->om_bytes_sent += req->rq_xmit_bytes_sent; op_metrics->om_bytes_recv += req->rq_reply_bytes_recvd; - delta = ktime_sub(req->rq_xtime, task->tk_start); - op_metrics->om_queue = ktime_add(op_metrics->om_queue, delta); - + if (ktime_to_ns(req->rq_xtime)) { + delta = ktime_sub(req->rq_xtime, task->tk_start); + op_metrics->om_queue = ktime_add(op_metrics->om_queue, delta); + } op_metrics->om_rtt = ktime_add(op_metrics->om_rtt, req->rq_rtt); delta = ktime_sub(now, task->tk_start);
next prev parent reply other threads:[~2016-11-29 15:52 UTC|newest] Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-11-29 15:51 [PATCH v3 00/12] client-side NFS/RDMA patches proposed for v4.10 Chuck Lever 2016-11-29 15:51 ` Chuck Lever [not found] ` <20161129154845.23061.29385.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2016-11-29 15:52 ` [PATCH v3 01/12] xprtrdma: Cap size of callback buffer resources Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:52 ` [PATCH v3 02/12] xprtrdma: Make FRWR send queue entry accounting more accurate Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:52 ` [PATCH v3 03/12] xprtrdma: Support for SG_GAP devices Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:52 ` Chuck Lever [this message] 2016-11-29 15:52 ` [PATCH v3 04/12] SUNRPC: Proper metric accounting when RPC is not transmitted Chuck Lever 2016-11-29 15:52 ` [PATCH v3 05/12] xprtrdma: Address coverity complaint about wait_for_completion() Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:52 ` [PATCH v3 06/12] xprtrdma: Avoid calls to ro_unmap_safe() Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:52 ` [PATCH v3 07/12] xprtrdma: Refactor FRMR invalidation Chuck Lever 2016-11-29 15:52 ` Chuck Lever 2016-11-29 15:53 ` [PATCH v3 08/12] xprtrdma: Update documenting comment Chuck Lever 2016-11-29 15:53 ` Chuck Lever 2016-11-29 15:53 ` [PATCH v3 09/12] xprtrdma: Squelch "max send, max recv" messages at connect time Chuck Lever 2016-11-29 15:53 ` Chuck Lever 2016-11-29 15:53 ` [PATCH v3 10/12] xprtrdma: Shorten QP access error message Chuck Lever 2016-11-29 15:53 ` Chuck Lever 2016-11-29 15:53 ` [PATCH v3 11/12] xprtrdma: Update dprintk in rpcrdma_count_chunks Chuck Lever 2016-11-29 15:53 ` Chuck Lever 2016-11-29 15:53 ` [PATCH v3 12/12] xprtrdma: Relocate connection helper functions Chuck Lever 2016-11-29 15:53 ` Chuck Lever
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20161129155232.23061.9794.stgit@manet.1015granger.net \ --to=chuck.lever-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=anna.schumaker-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.