[PATCH v2 00/34] Convert RPC client transmission to a queued model

[PATCH v2 00/34] Convert RPC client transmission to a queued model

[Trond Myklebust]

For historical reasons, the RPC client is heavily serialised during the
process of transmitting a request by the XPRT_LOCK. A request is
required to take that lock before it can start XDR encoding, and it is
required to hold it until it is done transmitting. In essence the lock
protects the following functions:

- Stream based transport connect/reconnect
- RPCSEC_GSS encoding of the RPC message
- Transmission of a single RPC message

The following patch set assumes that we do not need to do much to
improve performance of the connect/reconnect case, as that is supposed
to be a rare occurrence.

The set looks at dealing with RPCSEC_GSS issues by removing serialisation
while encoding, and simply assuming that if we detect after grabbing the
XPRT_LOCK that we're about to transmit a message with a sequence number
that has fallen outside the window allowed by RFC2203, then we can
abort the transmission of that message, and schedule it for re-encoding.
Since window sizes are typically expected to lie above 100 messages or
so, we expect these cases where we miss the window to be rare, in
general.

Finally, we look at trying to avoid the requirement that every request
must go through the process of being woken up to grab the XPRT_LOCK in
order to transmit itself by allowing a request that currently holds the
XPRT_LOCK to grab other requests from an ordered queue, and to transmit
them too. The bulk of the changes in this patchset are dedicated to
providing this functionality.

In addition, the XPRT_LOCK queue provides some extra functionality:
- Throttling of the TCP slot allocation (as Chuck pointed out)
- Fair queuing, to ensure batch jobs don't crowd out interactive ones

The patchset does add functionality to ensure that the resulting
transmission queue is fair, and also fixes up the RPC wait queues to
ensure that they don't compromise fairness.
For now, this patchset discards the TCP slot throttling. We may still
want to throttle in the case where the connection is lost, but if we
do so, we should ensure we do not serialise all requests when in the
connected state.

---
v2: - Address feedback by Chuck.
      - Handle UDP/RDMA credits correctly
      - Remove throttling of TCP slot allocations
      - Minor nits
    - Clean up the write_space handling
    - Fair queueing


Trond Myklebust (34):
  SUNRPC: Clean up initialisation of the struct rpc_rqst
  SUNRPC: If there is no reply expected, bail early from call_decode
  SUNRPC: The transmitted message must lie in the RPCSEC window of
    validity
  SUNRPC: Simplify identification of when the message send/receive is
    complete
  SUNRPC: Avoid holding locks across the XDR encoding of the RPC message
  SUNRPC: Rename TCP receive-specific state variables
  SUNRPC: Move reset of TCP state variables into the reconnect code
  SUNRPC: Add socket transmit queue offset tracking
  SUNRPC: Simplify dealing with aborted partially transmitted messages
  SUNRPC: Refactor the transport request pinning
  SUNRPC: Add a helper to wake up a sleeping rpc_task and set its status
  SUNRPC: Don't wake queued RPC calls multiple times in xprt_transmit
  SUNRPC: Rename xprt->recv_lock to xprt->queue_lock
  SUNRPC: Refactor xprt_transmit() to remove the reply queue code
  SUNRPC: Refactor xprt_transmit() to remove wait for reply code
  SUNRPC: Minor cleanup for call_transmit()
  SUNRPC: Distinguish between the slot allocation list and receive queue
  NFS: Add a transmission queue for RPC requests
  SUNRPC: Refactor RPC call encoding
  SUNRPC: Treat the task and request as separate in the
    xprt_ops->send_request()
  SUNRPC: Don't reset the request 'bytes_sent' counter when releasing
    XPRT_LOCK
  SUNRPC: Simplify xprt_prepare_transmit()
  SUNRPC: Move RPC retransmission stat counter to xprt_transmit()
  SUNRPC: Fix up the back channel transmit
  SUNRPC: Support for congestion control when queuing is enabled
  SUNRPC: Improve latency for interactive tasks
  SUNRPC: Allow calls to xprt_transmit() to drain the entire transmit
    queue
  SUNRPC: Queue the request for transmission immediately after encoding
  SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue
  SUNRPC: Allow soft RPC calls to time out when waiting for the
    XPRT_LOCK
  SUNRPC: Turn off throttling of RPC slots for TCP sockets
  SUNRPC: Clean up transport write space handling
  SUNRPC: Cleanup: remove the unused 'task' argument from the
    request_send()
  SUNRPC: Queue fairness for all.

 include/linux/sunrpc/auth.h                |   2 +
 include/linux/sunrpc/auth_gss.h            |   1 +
 include/linux/sunrpc/sched.h               |   9 +-
 include/linux/sunrpc/svc_xprt.h            |   1 -
 include/linux/sunrpc/xprt.h                |  31 +-
 include/linux/sunrpc/xprtsock.h            |  23 +-
 include/trace/events/sunrpc.h              |  10 +-
 net/sunrpc/auth.c                          |  10 +
 net/sunrpc/auth_gss/auth_gss.c             |  41 ++
 net/sunrpc/backchannel_rqst.c              |   4 +-
 net/sunrpc/clnt.c                          | 152 ++---
 net/sunrpc/sched.c                         | 189 +++---
 net/sunrpc/svc_xprt.c                      |   2 -
 net/sunrpc/svcsock.c                       |   6 +-
 net/sunrpc/xprt.c                          | 679 +++++++++++++--------
 net/sunrpc/xprtrdma/backchannel.c          |   7 +-
 net/sunrpc/xprtrdma/rpc_rdma.c             |  12 +-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  14 +-
 net/sunrpc/xprtrdma/transport.c            |  10 +-
 net/sunrpc/xprtsock.c                      | 359 ++++++-----
 20 files changed, 919 insertions(+), 643 deletions(-)

-- 
2.17.1

[PATCH v2 01/34] SUNRPC: Clean up initialisation of the struct rpc_rqst

[Trond Myklebust]

Move the initialisation back into xprt.c.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h |  1 -
 net/sunrpc/clnt.c           |  1 -
 net/sunrpc/xprt.c           | 91 +++++++++++++++++++++----------------
 3 files changed, 51 insertions(+), 42 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 336fd1a19cca..3d80524e92d6 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -325,7 +325,6 @@ struct xprt_class {
 struct rpc_xprt		*xprt_create_transport(struct xprt_create *args);
 void			xprt_connect(struct rpc_task *task);
 void			xprt_reserve(struct rpc_task *task);
-void			xprt_request_init(struct rpc_task *task);
 void			xprt_retry_reserve(struct rpc_task *task);
 int			xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task);
 int			xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 8ea2f5fadd96..bc9d020bf71f 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1558,7 +1558,6 @@ call_reserveresult(struct rpc_task *task)
 	task->tk_status = 0;
 	if (status >= 0) {
 		if (task->tk_rqstp) {
-			xprt_request_init(task);
 			task->tk_action = call_refresh;
 			return;
 		}
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index a8db2e3f8904..6aa09edc9567 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1250,6 +1250,55 @@ void xprt_free(struct rpc_xprt *xprt)
 }
 EXPORT_SYMBOL_GPL(xprt_free);
 
+static __be32
+xprt_alloc_xid(struct rpc_xprt *xprt)
+{
+	__be32 xid;
+
+	spin_lock(&xprt->reserve_lock);
+	xid = (__force __be32)xprt->xid++;
+	spin_unlock(&xprt->reserve_lock);
+	return xid;
+}
+
+static void
+xprt_init_xid(struct rpc_xprt *xprt)
+{
+	xprt->xid = prandom_u32();
+}
+
+static void
+xprt_request_init(struct rpc_task *task)
+{
+	struct rpc_xprt *xprt = task->tk_xprt;
+	struct rpc_rqst	*req = task->tk_rqstp;
+
+	INIT_LIST_HEAD(&req->rq_list);
+	req->rq_timeout = task->tk_client->cl_timeout->to_initval;
+	req->rq_task	= task;
+	req->rq_xprt    = xprt;
+	req->rq_buffer  = NULL;
+	req->rq_xid	= xprt_alloc_xid(xprt);
+	req->rq_connect_cookie = xprt->connect_cookie - 1;
+	req->rq_bytes_sent = 0;
+	req->rq_snd_buf.len = 0;
+	req->rq_snd_buf.buflen = 0;
+	req->rq_rcv_buf.len = 0;
+	req->rq_rcv_buf.buflen = 0;
+	req->rq_release_snd_buf = NULL;
+	xprt_reset_majortimeo(req);
+	dprintk("RPC: %5u reserved req %p xid %08x\n", task->tk_pid,
+			req, ntohl(req->rq_xid));
+}
+
+static void
+xprt_do_reserve(struct rpc_xprt *xprt, struct rpc_task *task)
+{
+	xprt->ops->alloc_slot(xprt, task);
+	if (task->tk_rqstp != NULL)
+		xprt_request_init(task);
+}
+
 /**
  * xprt_reserve - allocate an RPC request slot
  * @task: RPC task requesting a slot allocation
@@ -1269,7 +1318,7 @@ void xprt_reserve(struct rpc_task *task)
 	task->tk_timeout = 0;
 	task->tk_status = -EAGAIN;
 	if (!xprt_throttle_congested(xprt, task))
-		xprt->ops->alloc_slot(xprt, task);
+		xprt_do_reserve(xprt, task);
 }
 
 /**
@@ -1291,45 +1340,7 @@ void xprt_retry_reserve(struct rpc_task *task)
 
 	task->tk_timeout = 0;
 	task->tk_status = -EAGAIN;
-	xprt->ops->alloc_slot(xprt, task);
-}
-
-static inline __be32 xprt_alloc_xid(struct rpc_xprt *xprt)
-{
-	__be32 xid;
-
-	spin_lock(&xprt->reserve_lock);
-	xid = (__force __be32)xprt->xid++;
-	spin_unlock(&xprt->reserve_lock);
-	return xid;
-}
-
-static inline void xprt_init_xid(struct rpc_xprt *xprt)
-{
-	xprt->xid = prandom_u32();
-}
-
-void xprt_request_init(struct rpc_task *task)
-{
-	struct rpc_xprt *xprt = task->tk_xprt;
-	struct rpc_rqst	*req = task->tk_rqstp;
-
-	INIT_LIST_HEAD(&req->rq_list);
-	req->rq_timeout = task->tk_client->cl_timeout->to_initval;
-	req->rq_task	= task;
-	req->rq_xprt    = xprt;
-	req->rq_buffer  = NULL;
-	req->rq_xid	= xprt_alloc_xid(xprt);
-	req->rq_connect_cookie = xprt->connect_cookie - 1;
-	req->rq_bytes_sent = 0;
-	req->rq_snd_buf.len = 0;
-	req->rq_snd_buf.buflen = 0;
-	req->rq_rcv_buf.len = 0;
-	req->rq_rcv_buf.buflen = 0;
-	req->rq_release_snd_buf = NULL;
-	xprt_reset_majortimeo(req);
-	dprintk("RPC: %5u reserved req %p xid %08x\n", task->tk_pid,
-			req, ntohl(req->rq_xid));
+	xprt_do_reserve(xprt, task);
 }
 
 /**
-- 
2.17.1

[PATCH v2 02/34] SUNRPC: If there is no reply expected, bail early from call_decode

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index bc9d020bf71f..4f1ec8013332 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2260,6 +2260,11 @@ call_decode(struct rpc_task *task)
 
 	dprint_status(task);
 
+	if (!decode) {
+		task->tk_action = rpc_exit_task;
+		return;
+	}
+
 	if (task->tk_flags & RPC_CALL_MAJORSEEN) {
 		if (clnt->cl_chatty) {
 			printk(KERN_NOTICE "%s: server %s OK\n",
@@ -2297,13 +2302,11 @@ call_decode(struct rpc_task *task)
 			goto out_retry;
 		return;
 	}
-
 	task->tk_action = rpc_exit_task;
 
-	if (decode) {
-		task->tk_status = rpcauth_unwrap_resp(task, decode, req, p,
-						      task->tk_msg.rpc_resp);
-	}
+	task->tk_status = rpcauth_unwrap_resp(task, decode, req, p,
+					      task->tk_msg.rpc_resp);
+
 	dprintk("RPC: %5u call_decode result %d\n", task->tk_pid,
 			task->tk_status);
 	return;
-- 
2.17.1

[PATCH v2 03/34] SUNRPC: The transmitted message must lie in the RPCSEC window of validity

[Trond Myklebust]

If a message has been encoded using RPCSEC_GSS, the server is
maintaining a window of sequence numbers that it considers valid.
The client should normally be tracking that window, and needs to
verify that the sequence number used by the message being transmitted
still lies inside the window of validity.

So far, we've been able to assume this condition would be realised
automatically, since the client has been encoding the message only
after taking the socket lock. Once we change that condition, we
will need the explicit check.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/auth.h     |  2 ++
 include/linux/sunrpc/auth_gss.h |  1 +
 net/sunrpc/auth.c               | 10 ++++++++
 net/sunrpc/auth_gss/auth_gss.c  | 41 +++++++++++++++++++++++++++++++++
 net/sunrpc/clnt.c               |  3 +++
 net/sunrpc/xprt.c               |  7 ++++++
 6 files changed, 64 insertions(+)

diff --git a/include/linux/sunrpc/auth.h b/include/linux/sunrpc/auth.h
index 58a6765c1c5e..2c97a3933ef9 100644
--- a/include/linux/sunrpc/auth.h
+++ b/include/linux/sunrpc/auth.h
@@ -157,6 +157,7 @@ struct rpc_credops {
 	int			(*crkey_timeout)(struct rpc_cred *);
 	bool			(*crkey_to_expire)(struct rpc_cred *);
 	char *			(*crstringify_acceptor)(struct rpc_cred *);
+	bool			(*crneed_reencode)(struct rpc_task *);
 };
 
 extern const struct rpc_authops	authunix_ops;
@@ -192,6 +193,7 @@ __be32 *		rpcauth_marshcred(struct rpc_task *, __be32 *);
 __be32 *		rpcauth_checkverf(struct rpc_task *, __be32 *);
 int			rpcauth_wrap_req(struct rpc_task *task, kxdreproc_t encode, void *rqstp, __be32 *data, void *obj);
 int			rpcauth_unwrap_resp(struct rpc_task *task, kxdrdproc_t decode, void *rqstp, __be32 *data, void *obj);
+bool			rpcauth_xmit_need_reencode(struct rpc_task *task);
 int			rpcauth_refreshcred(struct rpc_task *);
 void			rpcauth_invalcred(struct rpc_task *);
 int			rpcauth_uptodatecred(struct rpc_task *);
diff --git a/include/linux/sunrpc/auth_gss.h b/include/linux/sunrpc/auth_gss.h
index 0c9eac351aab..30427b729070 100644
--- a/include/linux/sunrpc/auth_gss.h
+++ b/include/linux/sunrpc/auth_gss.h
@@ -70,6 +70,7 @@ struct gss_cl_ctx {
 	refcount_t		count;
 	enum rpc_gss_proc	gc_proc;
 	u32			gc_seq;
+	u32			gc_seq_xmit;
 	spinlock_t		gc_seq_lock;
 	struct gss_ctx		*gc_gss_ctx;
 	struct xdr_netobj	gc_wire_ctx;
diff --git a/net/sunrpc/auth.c b/net/sunrpc/auth.c
index 305ecea92170..59df5cdba0ac 100644
--- a/net/sunrpc/auth.c
+++ b/net/sunrpc/auth.c
@@ -817,6 +817,16 @@ rpcauth_unwrap_resp(struct rpc_task *task, kxdrdproc_t decode, void *rqstp,
 	return rpcauth_unwrap_req_decode(decode, rqstp, data, obj);
 }
 
+bool
+rpcauth_xmit_need_reencode(struct rpc_task *task)
+{
+	struct rpc_cred *cred = task->tk_rqstp->rq_cred;
+
+	if (!cred || !cred->cr_ops->crneed_reencode)
+		return false;
+	return cred->cr_ops->crneed_reencode(task);
+}
+
 int
 rpcauth_refreshcred(struct rpc_task *task)
 {
diff --git a/net/sunrpc/auth_gss/auth_gss.c b/net/sunrpc/auth_gss/auth_gss.c
index 21c0aa0a0d1d..c898a7c75e84 100644
--- a/net/sunrpc/auth_gss/auth_gss.c
+++ b/net/sunrpc/auth_gss/auth_gss.c
@@ -1984,6 +1984,46 @@ gss_unwrap_req_decode(kxdrdproc_t decode, struct rpc_rqst *rqstp,
 	return decode(rqstp, &xdr, obj);
 }
 
+static bool
+gss_seq_is_newer(u32 new, u32 old)
+{
+	return (s32)(new - old) > 0;
+}
+
+static bool
+gss_xmit_need_reencode(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_cred *cred = req->rq_cred;
+	struct gss_cl_ctx *ctx = gss_cred_get_ctx(cred);
+	u32 win, seq_xmit;
+	bool ret = true;
+
+	if (!ctx)
+		return true;
+
+	if (gss_seq_is_newer(req->rq_seqno, READ_ONCE(ctx->gc_seq)))
+		goto out;
+
+	seq_xmit = READ_ONCE(ctx->gc_seq_xmit);
+	while (gss_seq_is_newer(req->rq_seqno, seq_xmit)) {
+		u32 tmp = seq_xmit;
+
+		seq_xmit = cmpxchg(&ctx->gc_seq_xmit, tmp, req->rq_seqno);
+		if (seq_xmit == tmp) {
+			ret = false;
+			goto out;
+		}
+	}
+
+	win = ctx->gc_win;
+	if (win > 0)
+		ret = !gss_seq_is_newer(req->rq_seqno, seq_xmit - win);
+out:
+	gss_put_ctx(ctx);
+	return ret;
+}
+
 static int
 gss_unwrap_resp(struct rpc_task *task,
 		kxdrdproc_t decode, void *rqstp, __be32 *p, void *obj)
@@ -2052,6 +2092,7 @@ static const struct rpc_credops gss_credops = {
 	.crunwrap_resp		= gss_unwrap_resp,
 	.crkey_timeout		= gss_key_timeout,
 	.crstringify_acceptor	= gss_stringify_acceptor,
+	.crneed_reencode	= gss_xmit_need_reencode,
 };
 
 static const struct rpc_credops gss_nullops = {
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 4f1ec8013332..d41b5ac1d4e8 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2184,6 +2184,9 @@ call_status(struct rpc_task *task)
 		/* shutdown or soft timeout */
 		rpc_exit(task, status);
 		break;
+	case -EBADMSG:
+		task->tk_action = call_transmit;
+		break;
 	default:
 		if (clnt->cl_chatty)
 			printk("%s: RPC call returned error %d\n",
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 6aa09edc9567..3973e10ea2bd 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1014,6 +1014,13 @@ void xprt_transmit(struct rpc_task *task)
 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, req->rq_slen);
 
 	if (!req->rq_reply_bytes_recvd) {
+
+		/* Verify that our message lies in the RPCSEC_GSS window */
+		if (!req->rq_bytes_sent && rpcauth_xmit_need_reencode(task)) {
+			task->tk_status = -EBADMSG;
+			return;
+		}
+
 		if (list_empty(&req->rq_list) && rpc_reply_expected(task)) {
 			/*
 			 * Add to the list only if we're expecting a reply
-- 
2.17.1

[PATCH v2 04/34] SUNRPC: Simplify identification of when the message send/receive is complete

[Trond Myklebust]

Add states to indicate that the message send and receive are not yet
complete.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/sched.h |  6 ++++--
 net/sunrpc/clnt.c            | 19 +++++++------------
 net/sunrpc/xprt.c            | 17 ++++++++++++++---
 3 files changed, 25 insertions(+), 17 deletions(-)

diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
index 592653becd91..9e655df70131 100644
--- a/include/linux/sunrpc/sched.h
+++ b/include/linux/sunrpc/sched.h
@@ -140,8 +140,10 @@ struct rpc_task_setup {
 #define RPC_TASK_RUNNING	0
 #define RPC_TASK_QUEUED		1
 #define RPC_TASK_ACTIVE		2
-#define RPC_TASK_MSG_RECV	3
-#define RPC_TASK_MSG_RECV_WAIT	4
+#define RPC_TASK_NEED_XMIT	3
+#define RPC_TASK_NEED_RECV	4
+#define RPC_TASK_MSG_RECV	5
+#define RPC_TASK_MSG_RECV_WAIT	6
 
 #define RPC_IS_RUNNING(t)	test_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
 #define rpc_set_running(t)	set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index d41b5ac1d4e8..e5ac35e803ad 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1156,6 +1156,7 @@ struct rpc_task *rpc_run_bc_task(struct rpc_rqst *req)
 	 */
 	xbufp->len = xbufp->head[0].iov_len + xbufp->page_len +
 			xbufp->tail[0].iov_len;
+	set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 
 	task->tk_action = call_bc_transmit;
 	atomic_inc(&task->tk_count);
@@ -1720,17 +1721,10 @@ call_allocate(struct rpc_task *task)
 	rpc_exit(task, -ERESTARTSYS);
 }
 
-static inline int
+static int
 rpc_task_need_encode(struct rpc_task *task)
 {
-	return task->tk_rqstp->rq_snd_buf.len == 0;
-}
-
-static inline void
-rpc_task_force_reencode(struct rpc_task *task)
-{
-	task->tk_rqstp->rq_snd_buf.len = 0;
-	task->tk_rqstp->rq_bytes_sent = 0;
+	return test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate) == 0;
 }
 
 /*
@@ -1765,6 +1759,8 @@ rpc_xdr_encode(struct rpc_task *task)
 
 	task->tk_status = rpcauth_wrap_req(task, encode, req, p,
 			task->tk_msg.rpc_argp);
+	if (task->tk_status == 0)
+		set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 }
 
 /*
@@ -1999,7 +1995,6 @@ call_transmit_status(struct rpc_task *task)
 	 */
 	if (task->tk_status == 0) {
 		xprt_end_transmit(task);
-		rpc_task_force_reencode(task);
 		return;
 	}
 
@@ -2010,7 +2005,6 @@ call_transmit_status(struct rpc_task *task)
 	default:
 		dprint_status(task);
 		xprt_end_transmit(task);
-		rpc_task_force_reencode(task);
 		break;
 		/*
 		 * Special cases: if we've been waiting on the
@@ -2038,7 +2032,7 @@ call_transmit_status(struct rpc_task *task)
 	case -EADDRINUSE:
 	case -ENOTCONN:
 	case -EPIPE:
-		rpc_task_force_reencode(task);
+		break;
 	}
 }
 
@@ -2185,6 +2179,7 @@ call_status(struct rpc_task *task)
 		rpc_exit(task, status);
 		break;
 	case -EBADMSG:
+		clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 		task->tk_action = call_transmit;
 		break;
 	default:
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 3973e10ea2bd..45d580cd93ac 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -936,10 +936,18 @@ void xprt_complete_rqst(struct rpc_task *task, int copied)
 	/* req->rq_reply_bytes_recvd */
 	smp_wmb();
 	req->rq_reply_bytes_recvd = copied;
+	clear_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
 	rpc_wake_up_queued_task(&xprt->pending, task);
 }
 EXPORT_SYMBOL_GPL(xprt_complete_rqst);
 
+static bool
+xprt_request_data_received(struct rpc_task *task)
+{
+	return !test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate) &&
+		task->tk_rqstp->rq_reply_bytes_recvd != 0;
+}
+
 static void xprt_timer(struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
@@ -1031,12 +1039,13 @@ void xprt_transmit(struct rpc_task *task)
 			/* Add request to the receive list */
 			spin_lock(&xprt->recv_lock);
 			list_add_tail(&req->rq_list, &xprt->recv);
+			set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
 			spin_unlock(&xprt->recv_lock);
 			xprt_reset_majortimeo(req);
 			/* Turn off autodisconnect */
 			del_singleshot_timer_sync(&xprt->timer);
 		}
-	} else if (!req->rq_bytes_sent)
+	} else if (xprt_request_data_received(task) && !req->rq_bytes_sent)
 		return;
 
 	connect_cookie = xprt->connect_cookie;
@@ -1046,9 +1055,11 @@ void xprt_transmit(struct rpc_task *task)
 		task->tk_status = status;
 		return;
 	}
+
 	xprt_inject_disconnect(xprt);
 
 	dprintk("RPC: %5u xmit complete\n", task->tk_pid);
+	clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 	task->tk_flags |= RPC_TASK_SENT;
 	spin_lock_bh(&xprt->transport_lock);
 
@@ -1062,14 +1073,14 @@ void xprt_transmit(struct rpc_task *task)
 	spin_unlock_bh(&xprt->transport_lock);
 
 	req->rq_connect_cookie = connect_cookie;
-	if (rpc_reply_expected(task) && !READ_ONCE(req->rq_reply_bytes_recvd)) {
+	if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 		/*
 		 * Sleep on the pending queue if we're expecting a reply.
 		 * The spinlock ensures atomicity between the test of
 		 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
 		 */
 		spin_lock(&xprt->recv_lock);
-		if (!req->rq_reply_bytes_recvd) {
+		if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 			rpc_sleep_on(&xprt->pending, task, xprt_timer);
 			/*
 			 * Send an extra queue wakeup call if the
-- 
2.17.1

[PATCH v2 05/34] SUNRPC: Avoid holding locks across the XDR encoding of the RPC message

[Trond Myklebust]

Currently, we grab the socket bit lock before we allow the message
to be XDR encoded. That significantly slows down the transmission
rate, since we serialise on a potentially blocking operation.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index e5ac35e803ad..a858366cd15d 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1949,9 +1949,6 @@ call_transmit(struct rpc_task *task)
 	task->tk_action = call_status;
 	if (task->tk_status < 0)
 		return;
-	if (!xprt_prepare_transmit(task))
-		return;
-	task->tk_action = call_transmit_status;
 	/* Encode here so that rpcsec_gss can use correct sequence number. */
 	if (rpc_task_need_encode(task)) {
 		rpc_xdr_encode(task);
@@ -1965,6 +1962,9 @@ call_transmit(struct rpc_task *task)
 			return;
 		}
 	}
+	if (!xprt_prepare_transmit(task))
+		return;
+	task->tk_action = call_transmit_status;
 	xprt_transmit(task);
 	if (task->tk_status < 0)
 		return;
-- 
2.17.1

[PATCH v2 06/34] SUNRPC: Rename TCP receive-specific state variables

[Trond Myklebust]

Since we will want to introduce similar TCP state variables for the
transmission of requests, let's rename the existing ones to label
that they are for the receive side.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprtsock.h |  16 +--
 include/trace/events/sunrpc.h   |  10 +-
 net/sunrpc/xprtsock.c           | 178 ++++++++++++++++----------------
 3 files changed, 103 insertions(+), 101 deletions(-)

diff --git a/include/linux/sunrpc/xprtsock.h b/include/linux/sunrpc/xprtsock.h
index ae0f99b9b965..90d5ca8e65f4 100644
--- a/include/linux/sunrpc/xprtsock.h
+++ b/include/linux/sunrpc/xprtsock.h
@@ -30,15 +30,17 @@ struct sock_xprt {
 	/*
 	 * State of TCP reply receive
 	 */
-	__be32			tcp_fraghdr,
-				tcp_xid,
-				tcp_calldir;
+	struct {
+		__be32		fraghdr,
+				xid,
+				calldir;
 
-	u32			tcp_offset,
-				tcp_reclen;
+		u32		offset,
+				len;
 
-	unsigned long		tcp_copied,
-				tcp_flags;
+		unsigned long	copied,
+				flags;
+	} recv;
 
 	/*
 	 * Connection of transports
diff --git a/include/trace/events/sunrpc.h b/include/trace/events/sunrpc.h
index bbb08a3ef5cc..0aa347194e0f 100644
--- a/include/trace/events/sunrpc.h
+++ b/include/trace/events/sunrpc.h
@@ -525,11 +525,11 @@ TRACE_EVENT(xs_tcp_data_recv,
 	TP_fast_assign(
 		__assign_str(addr, xs->xprt.address_strings[RPC_DISPLAY_ADDR]);
 		__assign_str(port, xs->xprt.address_strings[RPC_DISPLAY_PORT]);
-		__entry->xid = be32_to_cpu(xs->tcp_xid);
-		__entry->flags = xs->tcp_flags;
-		__entry->copied = xs->tcp_copied;
-		__entry->reclen = xs->tcp_reclen;
-		__entry->offset = xs->tcp_offset;
+		__entry->xid = be32_to_cpu(xs->recv.xid);
+		__entry->flags = xs->recv.flags;
+		__entry->copied = xs->recv.copied;
+		__entry->reclen = xs->recv.len;
+		__entry->offset = xs->recv.offset;
 	),
 
 	TP_printk("peer=[%s]:%s xid=0x%08x flags=%s copied=%lu reclen=%u offset=%lu",
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 6b7539c0466e..cd7d093721ae 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1169,42 +1169,42 @@ static inline void xs_tcp_read_fraghdr(struct rpc_xprt *xprt, struct xdr_skb_rea
 	size_t len, used;
 	char *p;
 
-	p = ((char *) &transport->tcp_fraghdr) + transport->tcp_offset;
-	len = sizeof(transport->tcp_fraghdr) - transport->tcp_offset;
+	p = ((char *) &transport->recv.fraghdr) + transport->recv.offset;
+	len = sizeof(transport->recv.fraghdr) - transport->recv.offset;
 	used = xdr_skb_read_bits(desc, p, len);
-	transport->tcp_offset += used;
+	transport->recv.offset += used;
 	if (used != len)
 		return;
 
-	transport->tcp_reclen = ntohl(transport->tcp_fraghdr);
-	if (transport->tcp_reclen & RPC_LAST_STREAM_FRAGMENT)
-		transport->tcp_flags |= TCP_RCV_LAST_FRAG;
+	transport->recv.len = ntohl(transport->recv.fraghdr);
+	if (transport->recv.len & RPC_LAST_STREAM_FRAGMENT)
+		transport->recv.flags |= TCP_RCV_LAST_FRAG;
 	else
-		transport->tcp_flags &= ~TCP_RCV_LAST_FRAG;
-	transport->tcp_reclen &= RPC_FRAGMENT_SIZE_MASK;
+		transport->recv.flags &= ~TCP_RCV_LAST_FRAG;
+	transport->recv.len &= RPC_FRAGMENT_SIZE_MASK;
 
-	transport->tcp_flags &= ~TCP_RCV_COPY_FRAGHDR;
-	transport->tcp_offset = 0;
+	transport->recv.flags &= ~TCP_RCV_COPY_FRAGHDR;
+	transport->recv.offset = 0;
 
 	/* Sanity check of the record length */
-	if (unlikely(transport->tcp_reclen < 8)) {
+	if (unlikely(transport->recv.len < 8)) {
 		dprintk("RPC:       invalid TCP record fragment length\n");
 		xs_tcp_force_close(xprt);
 		return;
 	}
 	dprintk("RPC:       reading TCP record fragment of length %d\n",
-			transport->tcp_reclen);
+			transport->recv.len);
 }
 
 static void xs_tcp_check_fraghdr(struct sock_xprt *transport)
 {
-	if (transport->tcp_offset == transport->tcp_reclen) {
-		transport->tcp_flags |= TCP_RCV_COPY_FRAGHDR;
-		transport->tcp_offset = 0;
-		if (transport->tcp_flags & TCP_RCV_LAST_FRAG) {
-			transport->tcp_flags &= ~TCP_RCV_COPY_DATA;
-			transport->tcp_flags |= TCP_RCV_COPY_XID;
-			transport->tcp_copied = 0;
+	if (transport->recv.offset == transport->recv.len) {
+		transport->recv.flags |= TCP_RCV_COPY_FRAGHDR;
+		transport->recv.offset = 0;
+		if (transport->recv.flags & TCP_RCV_LAST_FRAG) {
+			transport->recv.flags &= ~TCP_RCV_COPY_DATA;
+			transport->recv.flags |= TCP_RCV_COPY_XID;
+			transport->recv.copied = 0;
 		}
 	}
 }
@@ -1214,20 +1214,20 @@ static inline void xs_tcp_read_xid(struct sock_xprt *transport, struct xdr_skb_r
 	size_t len, used;
 	char *p;
 
-	len = sizeof(transport->tcp_xid) - transport->tcp_offset;
+	len = sizeof(transport->recv.xid) - transport->recv.offset;
 	dprintk("RPC:       reading XID (%zu bytes)\n", len);
-	p = ((char *) &transport->tcp_xid) + transport->tcp_offset;
+	p = ((char *) &transport->recv.xid) + transport->recv.offset;
 	used = xdr_skb_read_bits(desc, p, len);
-	transport->tcp_offset += used;
+	transport->recv.offset += used;
 	if (used != len)
 		return;
-	transport->tcp_flags &= ~TCP_RCV_COPY_XID;
-	transport->tcp_flags |= TCP_RCV_READ_CALLDIR;
-	transport->tcp_copied = 4;
+	transport->recv.flags &= ~TCP_RCV_COPY_XID;
+	transport->recv.flags |= TCP_RCV_READ_CALLDIR;
+	transport->recv.copied = 4;
 	dprintk("RPC:       reading %s XID %08x\n",
-			(transport->tcp_flags & TCP_RPC_REPLY) ? "reply for"
+			(transport->recv.flags & TCP_RPC_REPLY) ? "reply for"
 							      : "request with",
-			ntohl(transport->tcp_xid));
+			ntohl(transport->recv.xid));
 	xs_tcp_check_fraghdr(transport);
 }
 
@@ -1239,34 +1239,34 @@ static inline void xs_tcp_read_calldir(struct sock_xprt *transport,
 	char *p;
 
 	/*
-	 * We want transport->tcp_offset to be 8 at the end of this routine
+	 * We want transport->recv.offset to be 8 at the end of this routine
 	 * (4 bytes for the xid and 4 bytes for the call/reply flag).
 	 * When this function is called for the first time,
-	 * transport->tcp_offset is 4 (after having already read the xid).
+	 * transport->recv.offset is 4 (after having already read the xid).
 	 */
-	offset = transport->tcp_offset - sizeof(transport->tcp_xid);
-	len = sizeof(transport->tcp_calldir) - offset;
+	offset = transport->recv.offset - sizeof(transport->recv.xid);
+	len = sizeof(transport->recv.calldir) - offset;
 	dprintk("RPC:       reading CALL/REPLY flag (%zu bytes)\n", len);
-	p = ((char *) &transport->tcp_calldir) + offset;
+	p = ((char *) &transport->recv.calldir) + offset;
 	used = xdr_skb_read_bits(desc, p, len);
-	transport->tcp_offset += used;
+	transport->recv.offset += used;
 	if (used != len)
 		return;
-	transport->tcp_flags &= ~TCP_RCV_READ_CALLDIR;
+	transport->recv.flags &= ~TCP_RCV_READ_CALLDIR;
 	/*
 	 * We don't yet have the XDR buffer, so we will write the calldir
 	 * out after we get the buffer from the 'struct rpc_rqst'
 	 */
-	switch (ntohl(transport->tcp_calldir)) {
+	switch (ntohl(transport->recv.calldir)) {
 	case RPC_REPLY:
-		transport->tcp_flags |= TCP_RCV_COPY_CALLDIR;
-		transport->tcp_flags |= TCP_RCV_COPY_DATA;
-		transport->tcp_flags |= TCP_RPC_REPLY;
+		transport->recv.flags |= TCP_RCV_COPY_CALLDIR;
+		transport->recv.flags |= TCP_RCV_COPY_DATA;
+		transport->recv.flags |= TCP_RPC_REPLY;
 		break;
 	case RPC_CALL:
-		transport->tcp_flags |= TCP_RCV_COPY_CALLDIR;
-		transport->tcp_flags |= TCP_RCV_COPY_DATA;
-		transport->tcp_flags &= ~TCP_RPC_REPLY;
+		transport->recv.flags |= TCP_RCV_COPY_CALLDIR;
+		transport->recv.flags |= TCP_RCV_COPY_DATA;
+		transport->recv.flags &= ~TCP_RPC_REPLY;
 		break;
 	default:
 		dprintk("RPC:       invalid request message type\n");
@@ -1287,21 +1287,21 @@ static inline void xs_tcp_read_common(struct rpc_xprt *xprt,
 
 	rcvbuf = &req->rq_private_buf;
 
-	if (transport->tcp_flags & TCP_RCV_COPY_CALLDIR) {
+	if (transport->recv.flags & TCP_RCV_COPY_CALLDIR) {
 		/*
 		 * Save the RPC direction in the XDR buffer
 		 */
-		memcpy(rcvbuf->head[0].iov_base + transport->tcp_copied,
-			&transport->tcp_calldir,
-			sizeof(transport->tcp_calldir));
-		transport->tcp_copied += sizeof(transport->tcp_calldir);
-		transport->tcp_flags &= ~TCP_RCV_COPY_CALLDIR;
+		memcpy(rcvbuf->head[0].iov_base + transport->recv.copied,
+			&transport->recv.calldir,
+			sizeof(transport->recv.calldir));
+		transport->recv.copied += sizeof(transport->recv.calldir);
+		transport->recv.flags &= ~TCP_RCV_COPY_CALLDIR;
 	}
 
 	len = desc->count;
-	if (len > transport->tcp_reclen - transport->tcp_offset)
-		desc->count = transport->tcp_reclen - transport->tcp_offset;
-	r = xdr_partial_copy_from_skb(rcvbuf, transport->tcp_copied,
+	if (len > transport->recv.len - transport->recv.offset)
+		desc->count = transport->recv.len - transport->recv.offset;
+	r = xdr_partial_copy_from_skb(rcvbuf, transport->recv.copied,
 					  desc, xdr_skb_read_bits);
 
 	if (desc->count) {
@@ -1314,31 +1314,31 @@ static inline void xs_tcp_read_common(struct rpc_xprt *xprt,
 		 * Any remaining data from this record will
 		 * be discarded.
 		 */
-		transport->tcp_flags &= ~TCP_RCV_COPY_DATA;
+		transport->recv.flags &= ~TCP_RCV_COPY_DATA;
 		dprintk("RPC:       XID %08x truncated request\n",
-				ntohl(transport->tcp_xid));
-		dprintk("RPC:       xprt = %p, tcp_copied = %lu, "
-				"tcp_offset = %u, tcp_reclen = %u\n",
-				xprt, transport->tcp_copied,
-				transport->tcp_offset, transport->tcp_reclen);
+				ntohl(transport->recv.xid));
+		dprintk("RPC:       xprt = %p, recv.copied = %lu, "
+				"recv.offset = %u, recv.len = %u\n",
+				xprt, transport->recv.copied,
+				transport->recv.offset, transport->recv.len);
 		return;
 	}
 
-	transport->tcp_copied += r;
-	transport->tcp_offset += r;
+	transport->recv.copied += r;
+	transport->recv.offset += r;
 	desc->count = len - r;
 
 	dprintk("RPC:       XID %08x read %zd bytes\n",
-			ntohl(transport->tcp_xid), r);
-	dprintk("RPC:       xprt = %p, tcp_copied = %lu, tcp_offset = %u, "
-			"tcp_reclen = %u\n", xprt, transport->tcp_copied,
-			transport->tcp_offset, transport->tcp_reclen);
-
-	if (transport->tcp_copied == req->rq_private_buf.buflen)
-		transport->tcp_flags &= ~TCP_RCV_COPY_DATA;
-	else if (transport->tcp_offset == transport->tcp_reclen) {
-		if (transport->tcp_flags & TCP_RCV_LAST_FRAG)
-			transport->tcp_flags &= ~TCP_RCV_COPY_DATA;
+			ntohl(transport->recv.xid), r);
+	dprintk("RPC:       xprt = %p, recv.copied = %lu, recv.offset = %u, "
+			"recv.len = %u\n", xprt, transport->recv.copied,
+			transport->recv.offset, transport->recv.len);
+
+	if (transport->recv.copied == req->rq_private_buf.buflen)
+		transport->recv.flags &= ~TCP_RCV_COPY_DATA;
+	else if (transport->recv.offset == transport->recv.len) {
+		if (transport->recv.flags & TCP_RCV_LAST_FRAG)
+			transport->recv.flags &= ~TCP_RCV_COPY_DATA;
 	}
 }
 
@@ -1353,14 +1353,14 @@ static inline int xs_tcp_read_reply(struct rpc_xprt *xprt,
 				container_of(xprt, struct sock_xprt, xprt);
 	struct rpc_rqst *req;
 
-	dprintk("RPC:       read reply XID %08x\n", ntohl(transport->tcp_xid));
+	dprintk("RPC:       read reply XID %08x\n", ntohl(transport->recv.xid));
 
 	/* Find and lock the request corresponding to this xid */
 	spin_lock(&xprt->recv_lock);
-	req = xprt_lookup_rqst(xprt, transport->tcp_xid);
+	req = xprt_lookup_rqst(xprt, transport->recv.xid);
 	if (!req) {
 		dprintk("RPC:       XID %08x request not found!\n",
-				ntohl(transport->tcp_xid));
+				ntohl(transport->recv.xid));
 		spin_unlock(&xprt->recv_lock);
 		return -1;
 	}
@@ -1370,8 +1370,8 @@ static inline int xs_tcp_read_reply(struct rpc_xprt *xprt,
 	xs_tcp_read_common(xprt, desc, req);
 
 	spin_lock(&xprt->recv_lock);
-	if (!(transport->tcp_flags & TCP_RCV_COPY_DATA))
-		xprt_complete_rqst(req->rq_task, transport->tcp_copied);
+	if (!(transport->recv.flags & TCP_RCV_COPY_DATA))
+		xprt_complete_rqst(req->rq_task, transport->recv.copied);
 	xprt_unpin_rqst(req);
 	spin_unlock(&xprt->recv_lock);
 	return 0;
@@ -1393,7 +1393,7 @@ static int xs_tcp_read_callback(struct rpc_xprt *xprt,
 	struct rpc_rqst *req;
 
 	/* Look up the request corresponding to the given XID */
-	req = xprt_lookup_bc_request(xprt, transport->tcp_xid);
+	req = xprt_lookup_bc_request(xprt, transport->recv.xid);
 	if (req == NULL) {
 		printk(KERN_WARNING "Callback slot table overflowed\n");
 		xprt_force_disconnect(xprt);
@@ -1403,8 +1403,8 @@ static int xs_tcp_read_callback(struct rpc_xprt *xprt,
 	dprintk("RPC:       read callback  XID %08x\n", ntohl(req->rq_xid));
 	xs_tcp_read_common(xprt, desc, req);
 
-	if (!(transport->tcp_flags & TCP_RCV_COPY_DATA))
-		xprt_complete_bc_request(req, transport->tcp_copied);
+	if (!(transport->recv.flags & TCP_RCV_COPY_DATA))
+		xprt_complete_bc_request(req, transport->recv.copied);
 
 	return 0;
 }
@@ -1415,7 +1415,7 @@ static inline int _xs_tcp_read_data(struct rpc_xprt *xprt,
 	struct sock_xprt *transport =
 				container_of(xprt, struct sock_xprt, xprt);
 
-	return (transport->tcp_flags & TCP_RPC_REPLY) ?
+	return (transport->recv.flags & TCP_RPC_REPLY) ?
 		xs_tcp_read_reply(xprt, desc) :
 		xs_tcp_read_callback(xprt, desc);
 }
@@ -1458,9 +1458,9 @@ static void xs_tcp_read_data(struct rpc_xprt *xprt,
 	else {
 		/*
 		 * The transport_lock protects the request handling.
-		 * There's no need to hold it to update the tcp_flags.
+		 * There's no need to hold it to update the recv.flags.
 		 */
-		transport->tcp_flags &= ~TCP_RCV_COPY_DATA;
+		transport->recv.flags &= ~TCP_RCV_COPY_DATA;
 	}
 }
 
@@ -1468,12 +1468,12 @@ static inline void xs_tcp_read_discard(struct sock_xprt *transport, struct xdr_s
 {
 	size_t len;
 
-	len = transport->tcp_reclen - transport->tcp_offset;
+	len = transport->recv.len - transport->recv.offset;
 	if (len > desc->count)
 		len = desc->count;
 	desc->count -= len;
 	desc->offset += len;
-	transport->tcp_offset += len;
+	transport->recv.offset += len;
 	dprintk("RPC:       discarded %zu bytes\n", len);
 	xs_tcp_check_fraghdr(transport);
 }
@@ -1494,22 +1494,22 @@ static int xs_tcp_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb, uns
 		trace_xs_tcp_data_recv(transport);
 		/* Read in a new fragment marker if necessary */
 		/* Can we ever really expect to get completely empty fragments? */
-		if (transport->tcp_flags & TCP_RCV_COPY_FRAGHDR) {
+		if (transport->recv.flags & TCP_RCV_COPY_FRAGHDR) {
 			xs_tcp_read_fraghdr(xprt, &desc);
 			continue;
 		}
 		/* Read in the xid if necessary */
-		if (transport->tcp_flags & TCP_RCV_COPY_XID) {
+		if (transport->recv.flags & TCP_RCV_COPY_XID) {
 			xs_tcp_read_xid(transport, &desc);
 			continue;
 		}
 		/* Read in the call/reply flag */
-		if (transport->tcp_flags & TCP_RCV_READ_CALLDIR) {
+		if (transport->recv.flags & TCP_RCV_READ_CALLDIR) {
 			xs_tcp_read_calldir(transport, &desc);
 			continue;
 		}
 		/* Read in the request data */
-		if (transport->tcp_flags & TCP_RCV_COPY_DATA) {
+		if (transport->recv.flags & TCP_RCV_COPY_DATA) {
 			xs_tcp_read_data(xprt, &desc);
 			continue;
 		}
@@ -1602,10 +1602,10 @@ static void xs_tcp_state_change(struct sock *sk)
 		if (!xprt_test_and_set_connected(xprt)) {
 
 			/* Reset TCP record info */
-			transport->tcp_offset = 0;
-			transport->tcp_reclen = 0;
-			transport->tcp_copied = 0;
-			transport->tcp_flags =
+			transport->recv.offset = 0;
+			transport->recv.len = 0;
+			transport->recv.copied = 0;
+			transport->recv.flags =
 				TCP_RCV_COPY_FRAGHDR | TCP_RCV_COPY_XID;
 			xprt->connect_cookie++;
 			clear_bit(XPRT_SOCK_CONNECTING, &transport->sock_state);
-- 
2.17.1

[PATCH v2 07/34] SUNRPC: Move reset of TCP state variables into the reconnect code

[Trond Myklebust]

Rather than resetting state variables in socket state_change() callback,
do it in the sunrpc TCP connect function itself.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprtsock.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index cd7d093721ae..ec1e3f93e707 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1600,13 +1600,6 @@ static void xs_tcp_state_change(struct sock *sk)
 	case TCP_ESTABLISHED:
 		spin_lock(&xprt->transport_lock);
 		if (!xprt_test_and_set_connected(xprt)) {
-
-			/* Reset TCP record info */
-			transport->recv.offset = 0;
-			transport->recv.len = 0;
-			transport->recv.copied = 0;
-			transport->recv.flags =
-				TCP_RCV_COPY_FRAGHDR | TCP_RCV_COPY_XID;
 			xprt->connect_cookie++;
 			clear_bit(XPRT_SOCK_CONNECTING, &transport->sock_state);
 			xprt_clear_connecting(xprt);
@@ -2386,6 +2379,12 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 
 	xs_set_memalloc(xprt);
 
+	/* Reset TCP record info */
+	transport->recv.offset = 0;
+	transport->recv.len = 0;
+	transport->recv.copied = 0;
+	transport->recv.flags = TCP_RCV_COPY_FRAGHDR | TCP_RCV_COPY_XID;
+
 	/* Tell the socket layer to start connecting... */
 	xprt->stat.connect_count++;
 	xprt->stat.connect_start = jiffies;
-- 
2.17.1

[PATCH v2 08/34] SUNRPC: Add socket transmit queue offset tracking

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprtsock.h |  7 ++++++
 net/sunrpc/xprtsock.c           | 40 ++++++++++++++++++---------------
 2 files changed, 29 insertions(+), 18 deletions(-)

diff --git a/include/linux/sunrpc/xprtsock.h b/include/linux/sunrpc/xprtsock.h
index 90d5ca8e65f4..005cfb6e7238 100644
--- a/include/linux/sunrpc/xprtsock.h
+++ b/include/linux/sunrpc/xprtsock.h
@@ -42,6 +42,13 @@ struct sock_xprt {
 				flags;
 	} recv;
 
+	/*
+	 * State of TCP transmit queue
+	 */
+	struct {
+		u32		offset;
+	} xmit;
+
 	/*
 	 * Connection of transports
 	 */
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index ec1e3f93e707..629cc45e1e6c 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -461,7 +461,7 @@ static int xs_nospace(struct rpc_task *task)
 	int ret = -EAGAIN;
 
 	dprintk("RPC: %5u xmit incomplete (%u left of %u)\n",
-			task->tk_pid, req->rq_slen - req->rq_bytes_sent,
+			task->tk_pid, req->rq_slen - transport->xmit.offset,
 			req->rq_slen);
 
 	/* Protect against races with write_space */
@@ -528,19 +528,22 @@ static int xs_local_send_request(struct rpc_task *task)
 			req->rq_svec->iov_base, req->rq_svec->iov_len);
 
 	req->rq_xtime = ktime_get();
-	status = xs_sendpages(transport->sock, NULL, 0, xdr, req->rq_bytes_sent,
+	status = xs_sendpages(transport->sock, NULL, 0, xdr,
+			      transport->xmit.offset,
 			      true, &sent);
 	dprintk("RPC:       %s(%u) = %d\n",
-			__func__, xdr->len - req->rq_bytes_sent, status);
+			__func__, xdr->len - transport->xmit.offset, status);
 
 	if (status == -EAGAIN && sock_writeable(transport->inet))
 		status = -ENOBUFS;
 
 	if (likely(sent > 0) || status == 0) {
-		req->rq_bytes_sent += sent;
-		req->rq_xmit_bytes_sent += sent;
+		transport->xmit.offset += sent;
+		req->rq_bytes_sent = transport->xmit.offset;
 		if (likely(req->rq_bytes_sent >= req->rq_slen)) {
+			req->rq_xmit_bytes_sent += transport->xmit.offset;
 			req->rq_bytes_sent = 0;
+			transport->xmit.offset = 0;
 			return 0;
 		}
 		status = -EAGAIN;
@@ -592,10 +595,10 @@ static int xs_udp_send_request(struct rpc_task *task)
 		return -ENOTCONN;
 	req->rq_xtime = ktime_get();
 	status = xs_sendpages(transport->sock, xs_addr(xprt), xprt->addrlen,
-			      xdr, req->rq_bytes_sent, true, &sent);
+			      xdr, 0, true, &sent);
 
 	dprintk("RPC:       xs_udp_send_request(%u) = %d\n",
-			xdr->len - req->rq_bytes_sent, status);
+			xdr->len, status);
 
 	/* firewall is blocking us, don't return -EAGAIN or we end up looping */
 	if (status == -EPERM)
@@ -684,17 +687,20 @@ static int xs_tcp_send_request(struct rpc_task *task)
 	while (1) {
 		sent = 0;
 		status = xs_sendpages(transport->sock, NULL, 0, xdr,
-				      req->rq_bytes_sent, zerocopy, &sent);
+				      transport->xmit.offset,
+				      zerocopy, &sent);
 
 		dprintk("RPC:       xs_tcp_send_request(%u) = %d\n",
-				xdr->len - req->rq_bytes_sent, status);
+				xdr->len - transport->xmit.offset, status);
 
 		/* If we've sent the entire packet, immediately
 		 * reset the count of bytes sent. */
-		req->rq_bytes_sent += sent;
-		req->rq_xmit_bytes_sent += sent;
+		transport->xmit.offset += sent;
+		req->rq_bytes_sent = transport->xmit.offset;
 		if (likely(req->rq_bytes_sent >= req->rq_slen)) {
+			req->rq_xmit_bytes_sent += transport->xmit.offset;
 			req->rq_bytes_sent = 0;
+			transport->xmit.offset = 0;
 			return 0;
 		}
 
@@ -760,18 +766,13 @@ static int xs_tcp_send_request(struct rpc_task *task)
  */
 static void xs_tcp_release_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 {
-	struct rpc_rqst *req;
+	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
 
 	if (task != xprt->snd_task)
 		return;
 	if (task == NULL)
 		goto out_release;
-	req = task->tk_rqstp;
-	if (req == NULL)
-		goto out_release;
-	if (req->rq_bytes_sent == 0)
-		goto out_release;
-	if (req->rq_bytes_sent == req->rq_snd_buf.len)
+	if (transport->xmit.offset == 0 || !xprt_connected(xprt))
 		goto out_release;
 	set_bit(XPRT_CLOSE_WAIT, &xprt->state);
 out_release:
@@ -2021,6 +2022,8 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 		write_unlock_bh(&sk->sk_callback_lock);
 	}
 
+	transport->xmit.offset = 0;
+
 	/* Tell the socket layer to start connecting... */
 	xprt->stat.connect_count++;
 	xprt->stat.connect_start = jiffies;
@@ -2384,6 +2387,7 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 	transport->recv.len = 0;
 	transport->recv.copied = 0;
 	transport->recv.flags = TCP_RCV_COPY_FRAGHDR | TCP_RCV_COPY_XID;
+	transport->xmit.offset = 0;
 
 	/* Tell the socket layer to start connecting... */
 	xprt->stat.connect_count++;
-- 
2.17.1

[PATCH v2 09/34] SUNRPC: Simplify dealing with aborted partially transmitted messages

[Trond Myklebust]

If the previous message was only partially transmitted, we need to close
the socket in order to avoid corruption of the message stream. To do so,
we currently hijack the unlocking of the socket in order to schedule
the close.
Now that we track the message offset in the socket state, we can move
that kind of checking out of the socket lock code, which is needed to
allow messages to remain queued after dropping the socket lock.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprtsock.c | 51 +++++++++++++++++++++----------------------
 1 file changed, 25 insertions(+), 26 deletions(-)

diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 629cc45e1e6c..3fbccebd0b10 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -491,6 +491,16 @@ static int xs_nospace(struct rpc_task *task)
 	return ret;
 }
 
+/*
+ * Determine if the previous message in the stream was aborted before it
+ * could complete transmission.
+ */
+static bool
+xs_send_request_was_aborted(struct sock_xprt *transport, struct rpc_rqst *req)
+{
+	return transport->xmit.offset != 0 && req->rq_bytes_sent == 0;
+}
+
 /*
  * Construct a stream transport record marker in @buf.
  */
@@ -522,6 +532,12 @@ static int xs_local_send_request(struct rpc_task *task)
 	int status;
 	int sent = 0;
 
+	/* Close the stream if the previous transmission was incomplete */
+	if (xs_send_request_was_aborted(transport, req)) {
+		xs_close(xprt);
+		return -ENOTCONN;
+	}
+
 	xs_encode_stream_record_marker(&req->rq_snd_buf);
 
 	xs_pktdump("packet data:",
@@ -665,6 +681,13 @@ static int xs_tcp_send_request(struct rpc_task *task)
 	int status;
 	int sent;
 
+	/* Close the stream if the previous transmission was incomplete */
+	if (xs_send_request_was_aborted(transport, req)) {
+		if (transport->sock != NULL)
+			kernel_sock_shutdown(transport->sock, SHUT_RDWR);
+		return -ENOTCONN;
+	}
+
 	xs_encode_stream_record_marker(&req->rq_snd_buf);
 
 	xs_pktdump("packet data:",
@@ -755,30 +778,6 @@ static int xs_tcp_send_request(struct rpc_task *task)
 	return status;
 }
 
-/**
- * xs_tcp_release_xprt - clean up after a tcp transmission
- * @xprt: transport
- * @task: rpc task
- *
- * This cleans up if an error causes us to abort the transmission of a request.
- * In this case, the socket may need to be reset in order to avoid confusing
- * the server.
- */
-static void xs_tcp_release_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
-{
-	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
-
-	if (task != xprt->snd_task)
-		return;
-	if (task == NULL)
-		goto out_release;
-	if (transport->xmit.offset == 0 || !xprt_connected(xprt))
-		goto out_release;
-	set_bit(XPRT_CLOSE_WAIT, &xprt->state);
-out_release:
-	xprt_release_xprt(xprt, task);
-}
-
 static void xs_save_old_callbacks(struct sock_xprt *transport, struct sock *sk)
 {
 	transport->old_data_ready = sk->sk_data_ready;
@@ -2764,7 +2763,7 @@ static void bc_destroy(struct rpc_xprt *xprt)
 
 static const struct rpc_xprt_ops xs_local_ops = {
 	.reserve_xprt		= xprt_reserve_xprt,
-	.release_xprt		= xs_tcp_release_xprt,
+	.release_xprt		= xprt_release_xprt,
 	.alloc_slot		= xprt_alloc_slot,
 	.free_slot		= xprt_free_slot,
 	.rpcbind		= xs_local_rpcbind,
@@ -2806,7 +2805,7 @@ static const struct rpc_xprt_ops xs_udp_ops = {
 
 static const struct rpc_xprt_ops xs_tcp_ops = {
 	.reserve_xprt		= xprt_reserve_xprt,
-	.release_xprt		= xs_tcp_release_xprt,
+	.release_xprt		= xprt_release_xprt,
 	.alloc_slot		= xprt_lock_and_alloc_slot,
 	.free_slot		= xprt_free_slot,
 	.rpcbind		= rpcb_getport_async,
-- 
2.17.1

[PATCH v2 10/34] SUNRPC: Refactor the transport request pinning

[Trond Myklebust]

We are going to need to pin for both send and receive.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/sched.h |  2 --
 include/linux/sunrpc/xprt.h  |  1 +
 net/sunrpc/xprt.c            | 37 +++++++++++++++++-------------------
 3 files changed, 18 insertions(+), 22 deletions(-)

diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
index 9e655df70131..a4a42b3a1f03 100644
--- a/include/linux/sunrpc/sched.h
+++ b/include/linux/sunrpc/sched.h
@@ -142,8 +142,6 @@ struct rpc_task_setup {
 #define RPC_TASK_ACTIVE		2
 #define RPC_TASK_NEED_XMIT	3
 #define RPC_TASK_NEED_RECV	4
-#define RPC_TASK_MSG_RECV	5
-#define RPC_TASK_MSG_RECV_WAIT	6
 
 #define RPC_IS_RUNNING(t)	test_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
 #define rpc_set_running(t)	set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 3d80524e92d6..bd743c51a865 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -103,6 +103,7 @@ struct rpc_rqst {
 						/* A cookie used to track the
 						   state of the transport
 						   connection */
+	atomic_t		rq_pin;
 	
 	/*
 	 * Partial send handling
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 45d580cd93ac..bf8fc1a5dbd1 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -847,16 +847,22 @@ struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid)
 }
 EXPORT_SYMBOL_GPL(xprt_lookup_rqst);
 
+static bool
+xprt_is_pinned_rqst(struct rpc_rqst *req)
+{
+	return atomic_read(&req->rq_pin) != 0;
+}
+
 /**
  * xprt_pin_rqst - Pin a request on the transport receive list
  * @req: Request to pin
  *
  * Caller must ensure this is atomic with the call to xprt_lookup_rqst()
- * so should be holding the xprt transport lock.
+ * so should be holding the xprt receive lock.
  */
 void xprt_pin_rqst(struct rpc_rqst *req)
 {
-	set_bit(RPC_TASK_MSG_RECV, &req->rq_task->tk_runstate);
+	atomic_inc(&req->rq_pin);
 }
 EXPORT_SYMBOL_GPL(xprt_pin_rqst);
 
@@ -864,31 +870,18 @@ EXPORT_SYMBOL_GPL(xprt_pin_rqst);
  * xprt_unpin_rqst - Unpin a request on the transport receive list
  * @req: Request to pin
  *
- * Caller should be holding the xprt transport lock.
+ * Caller should be holding the xprt receive lock.
  */
 void xprt_unpin_rqst(struct rpc_rqst *req)
 {
-	struct rpc_task *task = req->rq_task;
-
-	clear_bit(RPC_TASK_MSG_RECV, &task->tk_runstate);
-	if (test_bit(RPC_TASK_MSG_RECV_WAIT, &task->tk_runstate))
-		wake_up_bit(&task->tk_runstate, RPC_TASK_MSG_RECV);
+	if (atomic_dec_and_test(&req->rq_pin))
+		wake_up_var(&req->rq_pin);
 }
 EXPORT_SYMBOL_GPL(xprt_unpin_rqst);
 
 static void xprt_wait_on_pinned_rqst(struct rpc_rqst *req)
-__must_hold(&req->rq_xprt->recv_lock)
 {
-	struct rpc_task *task = req->rq_task;
-
-	if (task && test_bit(RPC_TASK_MSG_RECV, &task->tk_runstate)) {
-		spin_unlock(&req->rq_xprt->recv_lock);
-		set_bit(RPC_TASK_MSG_RECV_WAIT, &task->tk_runstate);
-		wait_on_bit(&task->tk_runstate, RPC_TASK_MSG_RECV,
-				TASK_UNINTERRUPTIBLE);
-		clear_bit(RPC_TASK_MSG_RECV_WAIT, &task->tk_runstate);
-		spin_lock(&req->rq_xprt->recv_lock);
-	}
+	wait_var_event(&req->rq_pin, !xprt_is_pinned_rqst(req));
 }
 
 /**
@@ -1388,7 +1381,11 @@ void xprt_release(struct rpc_task *task)
 	spin_lock(&xprt->recv_lock);
 	if (!list_empty(&req->rq_list)) {
 		list_del_init(&req->rq_list);
-		xprt_wait_on_pinned_rqst(req);
+		if (atomic_read(&req->rq_pin)) {
+			spin_unlock(&xprt->recv_lock);
+			xprt_wait_on_pinned_rqst(req);
+			spin_lock(&xprt->recv_lock);
+		}
 	}
 	spin_unlock(&xprt->recv_lock);
 	spin_lock_bh(&xprt->transport_lock);
-- 
2.17.1

[PATCH v2 11/34] SUNRPC: Add a helper to wake up a sleeping rpc_task and set its status

[Trond Myklebust]

Add a helper that will wake up a task that is sleeping on a specific
queue, and will set the value of task->tk_status. This is mainly
intended for use by the transport layer to notify the task of an
error condition.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/sched.h |  3 ++
 net/sunrpc/sched.c           | 63 ++++++++++++++++++++++++++++++------
 2 files changed, 56 insertions(+), 10 deletions(-)

diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
index a4a42b3a1f03..c5bc779feb00 100644
--- a/include/linux/sunrpc/sched.h
+++ b/include/linux/sunrpc/sched.h
@@ -234,6 +234,9 @@ void rpc_wake_up_queued_task_on_wq(struct workqueue_struct *wq,
 		struct rpc_task *task);
 void		rpc_wake_up_queued_task(struct rpc_wait_queue *,
 					struct rpc_task *);
+void		rpc_wake_up_queued_task_set_status(struct rpc_wait_queue *,
+						   struct rpc_task *,
+						   int);
 void		rpc_wake_up(struct rpc_wait_queue *);
 struct rpc_task *rpc_wake_up_next(struct rpc_wait_queue *);
 struct rpc_task *rpc_wake_up_first_on_wq(struct workqueue_struct *wq,
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 3fe5d60ab0e2..104c056daf83 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -440,14 +440,28 @@ static void __rpc_do_wake_up_task_on_wq(struct workqueue_struct *wq,
 /*
  * Wake up a queued task while the queue lock is being held
  */
-static void rpc_wake_up_task_on_wq_queue_locked(struct workqueue_struct *wq,
-		struct rpc_wait_queue *queue, struct rpc_task *task)
+static struct rpc_task *
+rpc_wake_up_task_on_wq_queue_action_locked(struct workqueue_struct *wq,
+		struct rpc_wait_queue *queue, struct rpc_task *task,
+		bool (*action)(struct rpc_task *, void *), void *data)
 {
 	if (RPC_IS_QUEUED(task)) {
 		smp_rmb();
-		if (task->tk_waitqueue == queue)
-			__rpc_do_wake_up_task_on_wq(wq, queue, task);
+		if (task->tk_waitqueue == queue) {
+			if (action == NULL || action(task, data)) {
+				__rpc_do_wake_up_task_on_wq(wq, queue, task);
+				return task;
+			}
+		}
 	}
+	return NULL;
+}
+
+static void
+rpc_wake_up_task_on_wq_queue_locked(struct workqueue_struct *wq,
+		struct rpc_wait_queue *queue, struct rpc_task *task)
+{
+	rpc_wake_up_task_on_wq_queue_action_locked(wq, queue, task, NULL, NULL);
 }
 
 /*
@@ -481,6 +495,38 @@ void rpc_wake_up_queued_task(struct rpc_wait_queue *queue, struct rpc_task *task
 }
 EXPORT_SYMBOL_GPL(rpc_wake_up_queued_task);
 
+static bool rpc_task_action_set_status(struct rpc_task *task, void *status)
+{
+	task->tk_status = *(int *)status;
+	return true;
+}
+
+static void
+rpc_wake_up_task_queue_set_status_locked(struct rpc_wait_queue *queue,
+		struct rpc_task *task, int status)
+{
+	rpc_wake_up_task_on_wq_queue_action_locked(rpciod_workqueue, queue,
+			task, rpc_task_action_set_status, &status);
+}
+
+/**
+ * rpc_wake_up_queued_task_set_status - wake up a task and set task->tk_status
+ * @queue: pointer to rpc_wait_queue
+ * @task: pointer to rpc_task
+ * @status: integer error value
+ *
+ * If @task is queued on @queue, then it is woken up, and @task->tk_status is
+ * set to the value of @status.
+ */
+void
+rpc_wake_up_queued_task_set_status(struct rpc_wait_queue *queue,
+		struct rpc_task *task, int status)
+{
+	spin_lock_bh(&queue->lock);
+	rpc_wake_up_task_queue_set_status_locked(queue, task, status);
+	spin_unlock_bh(&queue->lock);
+}
+
 /*
  * Wake up the next task on a priority queue.
  */
@@ -553,12 +599,9 @@ struct rpc_task *rpc_wake_up_first_on_wq(struct workqueue_struct *wq,
 			queue, rpc_qname(queue));
 	spin_lock_bh(&queue->lock);
 	task = __rpc_find_next_queued(queue);
-	if (task != NULL) {
-		if (func(task, data))
-			rpc_wake_up_task_on_wq_queue_locked(wq, queue, task);
-		else
-			task = NULL;
-	}
+	if (task != NULL)
+		task = rpc_wake_up_task_on_wq_queue_action_locked(wq, queue,
+				task, func, data);
 	spin_unlock_bh(&queue->lock);
 
 	return task;
-- 
2.17.1

[PATCH v2 12/34] SUNRPC: Don't wake queued RPC calls multiple times in xprt_transmit

[Trond Myklebust]

Rather than waking up the entire queue of RPC messages a second time,
just wake up the task that was put to sleep.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprt.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index bf8fc1a5dbd1..0441e6c8153d 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1075,13 +1075,10 @@ void xprt_transmit(struct rpc_task *task)
 		spin_lock(&xprt->recv_lock);
 		if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 			rpc_sleep_on(&xprt->pending, task, xprt_timer);
-			/*
-			 * Send an extra queue wakeup call if the
-			 * connection was dropped in case the call to
-			 * rpc_sleep_on() raced.
-			 */
+			/* Wake up immediately if the connection was dropped */
 			if (!xprt_connected(xprt))
-				xprt_wake_pending_tasks(xprt, -ENOTCONN);
+				rpc_wake_up_queued_task_set_status(&xprt->pending,
+						task, -ENOTCONN);
 		}
 		spin_unlock(&xprt->recv_lock);
 	}
-- 
2.17.1

[PATCH v2 13/34] SUNRPC: Rename xprt->recv_lock to xprt->queue_lock

[Trond Myklebust]

We will use the same lock to protect both the transmit and receive queues.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h                |  2 +-
 net/sunrpc/svcsock.c                       |  6 ++---
 net/sunrpc/xprt.c                          | 24 ++++++++---------
 net/sunrpc/xprtrdma/rpc_rdma.c             | 10 ++++----
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  4 +--
 net/sunrpc/xprtsock.c                      | 30 +++++++++++-----------
 6 files changed, 38 insertions(+), 38 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index bd743c51a865..c25d0a5fda69 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -235,7 +235,7 @@ struct rpc_xprt {
 	 */
 	spinlock_t		transport_lock;	/* lock transport info */
 	spinlock_t		reserve_lock;	/* lock slot table */
-	spinlock_t		recv_lock;	/* lock receive list */
+	spinlock_t		queue_lock;	/* send/receive queue lock */
 	u32			xid;		/* Next XID value to use */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
 	struct svc_xprt		*bc_xprt;	/* NFSv4.1 backchannel */
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 5445145e639c..db8bb6b3a2b0 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1004,7 +1004,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 
 	if (!bc_xprt)
 		return -EAGAIN;
-	spin_lock(&bc_xprt->recv_lock);
+	spin_lock(&bc_xprt->queue_lock);
 	req = xprt_lookup_rqst(bc_xprt, xid);
 	if (!req)
 		goto unlock_notfound;
@@ -1022,7 +1022,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 	memcpy(dst->iov_base, src->iov_base, src->iov_len);
 	xprt_complete_rqst(req->rq_task, rqstp->rq_arg.len);
 	rqstp->rq_arg.len = 0;
-	spin_unlock(&bc_xprt->recv_lock);
+	spin_unlock(&bc_xprt->queue_lock);
 	return 0;
 unlock_notfound:
 	printk(KERN_NOTICE
@@ -1031,7 +1031,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 		__func__, ntohl(calldir),
 		bc_xprt, ntohl(xid));
 unlock_eagain:
-	spin_unlock(&bc_xprt->recv_lock);
+	spin_unlock(&bc_xprt->queue_lock);
 	return -EAGAIN;
 }
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 0441e6c8153d..eda305de9f77 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -826,7 +826,7 @@ static void xprt_connect_status(struct rpc_task *task)
  * @xprt: transport on which the original request was transmitted
  * @xid: RPC XID of incoming reply
  *
- * Caller holds xprt->recv_lock.
+ * Caller holds xprt->queue_lock.
  */
 struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid)
 {
@@ -888,7 +888,7 @@ static void xprt_wait_on_pinned_rqst(struct rpc_rqst *req)
  * xprt_update_rtt - Update RPC RTT statistics
  * @task: RPC request that recently completed
  *
- * Caller holds xprt->recv_lock.
+ * Caller holds xprt->queue_lock.
  */
 void xprt_update_rtt(struct rpc_task *task)
 {
@@ -910,7 +910,7 @@ EXPORT_SYMBOL_GPL(xprt_update_rtt);
  * @task: RPC request that recently completed
  * @copied: actual number of bytes received from the transport
  *
- * Caller holds xprt->recv_lock.
+ * Caller holds xprt->queue_lock.
  */
 void xprt_complete_rqst(struct rpc_task *task, int copied)
 {
@@ -1030,10 +1030,10 @@ void xprt_transmit(struct rpc_task *task)
 			memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
 					sizeof(req->rq_private_buf));
 			/* Add request to the receive list */
-			spin_lock(&xprt->recv_lock);
+			spin_lock(&xprt->queue_lock);
 			list_add_tail(&req->rq_list, &xprt->recv);
 			set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
-			spin_unlock(&xprt->recv_lock);
+			spin_unlock(&xprt->queue_lock);
 			xprt_reset_majortimeo(req);
 			/* Turn off autodisconnect */
 			del_singleshot_timer_sync(&xprt->timer);
@@ -1072,7 +1072,7 @@ void xprt_transmit(struct rpc_task *task)
 		 * The spinlock ensures atomicity between the test of
 		 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
 		 */
-		spin_lock(&xprt->recv_lock);
+		spin_lock(&xprt->queue_lock);
 		if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 			rpc_sleep_on(&xprt->pending, task, xprt_timer);
 			/* Wake up immediately if the connection was dropped */
@@ -1080,7 +1080,7 @@ void xprt_transmit(struct rpc_task *task)
 				rpc_wake_up_queued_task_set_status(&xprt->pending,
 						task, -ENOTCONN);
 		}
-		spin_unlock(&xprt->recv_lock);
+		spin_unlock(&xprt->queue_lock);
 	}
 }
 
@@ -1375,16 +1375,16 @@ void xprt_release(struct rpc_task *task)
 		task->tk_ops->rpc_count_stats(task, task->tk_calldata);
 	else if (task->tk_client)
 		rpc_count_iostats(task, task->tk_client->cl_metrics);
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	if (!list_empty(&req->rq_list)) {
 		list_del_init(&req->rq_list);
 		if (atomic_read(&req->rq_pin)) {
-			spin_unlock(&xprt->recv_lock);
+			spin_unlock(&xprt->queue_lock);
 			xprt_wait_on_pinned_rqst(req);
-			spin_lock(&xprt->recv_lock);
+			spin_lock(&xprt->queue_lock);
 		}
 	}
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	spin_lock_bh(&xprt->transport_lock);
 	xprt->ops->release_xprt(xprt, task);
 	if (xprt->ops->release_request)
@@ -1414,7 +1414,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 
 	spin_lock_init(&xprt->transport_lock);
 	spin_lock_init(&xprt->reserve_lock);
-	spin_lock_init(&xprt->recv_lock);
+	spin_lock_init(&xprt->queue_lock);
 
 	INIT_LIST_HEAD(&xprt->free);
 	INIT_LIST_HEAD(&xprt->recv);
diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index c8ae983c6cc0..0020dc401215 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -1238,7 +1238,7 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep)
 		goto out_badheader;
 
 out:
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	cwnd = xprt->cwnd;
 	xprt->cwnd = r_xprt->rx_buf.rb_credits << RPC_CWNDSHIFT;
 	if (xprt->cwnd > cwnd)
@@ -1246,7 +1246,7 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep)
 
 	xprt_complete_rqst(rqst->rq_task, status);
 	xprt_unpin_rqst(rqst);
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	return;
 
 /* If the incoming reply terminated a pending RPC, the next
@@ -1345,7 +1345,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 	/* Match incoming rpcrdma_rep to an rpcrdma_req to
 	 * get context for handling any incoming chunks.
 	 */
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	rqst = xprt_lookup_rqst(xprt, rep->rr_xid);
 	if (!rqst)
 		goto out_norqst;
@@ -1357,7 +1357,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 		credits = buf->rb_max_requests;
 	buf->rb_credits = credits;
 
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 
 	req = rpcr_to_rdmar(rqst);
 	req->rl_reply = rep;
@@ -1378,7 +1378,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
  * is corrupt.
  */
 out_norqst:
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	trace_xprtrdma_reply_rqst(rep);
 	goto repost;
 
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index a68180090554..09b12b7568fe 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -56,7 +56,7 @@ int svc_rdma_handle_bc_reply(struct rpc_xprt *xprt, __be32 *rdma_resp,
 	if (src->iov_len < 24)
 		goto out_shortreply;
 
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	req = xprt_lookup_rqst(xprt, xid);
 	if (!req)
 		goto out_notfound;
@@ -86,7 +86,7 @@ int svc_rdma_handle_bc_reply(struct rpc_xprt *xprt, __be32 *rdma_resp,
 	rcvbuf->len = 0;
 
 out_unlock:
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 out:
 	return ret;
 
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 3fbccebd0b10..8d6404259ff9 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -966,12 +966,12 @@ static void xs_local_data_read_skb(struct rpc_xprt *xprt,
 		return;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	rovr = xprt_lookup_rqst(xprt, *xp);
 	if (!rovr)
 		goto out_unlock;
 	xprt_pin_rqst(rovr);
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	task = rovr->rq_task;
 
 	copied = rovr->rq_private_buf.buflen;
@@ -980,16 +980,16 @@ static void xs_local_data_read_skb(struct rpc_xprt *xprt,
 
 	if (xs_local_copy_to_xdr(&rovr->rq_private_buf, skb)) {
 		dprintk("RPC:       sk_buff copy failed\n");
-		spin_lock(&xprt->recv_lock);
+		spin_lock(&xprt->queue_lock);
 		goto out_unpin;
 	}
 
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	xprt_complete_rqst(task, copied);
 out_unpin:
 	xprt_unpin_rqst(rovr);
  out_unlock:
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 }
 
 static void xs_local_data_receive(struct sock_xprt *transport)
@@ -1058,13 +1058,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 		return;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	rovr = xprt_lookup_rqst(xprt, *xp);
 	if (!rovr)
 		goto out_unlock;
 	xprt_pin_rqst(rovr);
 	xprt_update_rtt(rovr->rq_task);
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	task = rovr->rq_task;
 
 	if ((copied = rovr->rq_private_buf.buflen) > repsize)
@@ -1072,7 +1072,7 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 
 	/* Suck it into the iovec, verify checksum if not done by hw. */
 	if (csum_partial_copy_to_xdr(&rovr->rq_private_buf, skb)) {
-		spin_lock(&xprt->recv_lock);
+		spin_lock(&xprt->queue_lock);
 		__UDPX_INC_STATS(sk, UDP_MIB_INERRORS);
 		goto out_unpin;
 	}
@@ -1081,13 +1081,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 	spin_lock_bh(&xprt->transport_lock);
 	xprt_adjust_cwnd(xprt, task, copied);
 	spin_unlock_bh(&xprt->transport_lock);
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	xprt_complete_rqst(task, copied);
 	__UDPX_INC_STATS(sk, UDP_MIB_INDATAGRAMS);
 out_unpin:
 	xprt_unpin_rqst(rovr);
  out_unlock:
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 }
 
 static void xs_udp_data_receive(struct sock_xprt *transport)
@@ -1356,24 +1356,24 @@ static inline int xs_tcp_read_reply(struct rpc_xprt *xprt,
 	dprintk("RPC:       read reply XID %08x\n", ntohl(transport->recv.xid));
 
 	/* Find and lock the request corresponding to this xid */
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	req = xprt_lookup_rqst(xprt, transport->recv.xid);
 	if (!req) {
 		dprintk("RPC:       XID %08x request not found!\n",
 				ntohl(transport->recv.xid));
-		spin_unlock(&xprt->recv_lock);
+		spin_unlock(&xprt->queue_lock);
 		return -1;
 	}
 	xprt_pin_rqst(req);
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 
 	xs_tcp_read_common(xprt, desc, req);
 
-	spin_lock(&xprt->recv_lock);
+	spin_lock(&xprt->queue_lock);
 	if (!(transport->recv.flags & TCP_RCV_COPY_DATA))
 		xprt_complete_rqst(req->rq_task, transport->recv.copied);
 	xprt_unpin_rqst(req);
-	spin_unlock(&xprt->recv_lock);
+	spin_unlock(&xprt->queue_lock);
 	return 0;
 }
 
-- 
2.17.1

[PATCH v2 14/34] SUNRPC: Refactor xprt_transmit() to remove the reply queue code

[Trond Myklebust]

Separate out the action of adding a request to the reply queue so that the
backchannel code can simply skip calling it altogether.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h |   1 +
 net/sunrpc/clnt.c           |   5 ++
 net/sunrpc/xprt.c           | 100 ++++++++++++++++++++++--------------
 3 files changed, 68 insertions(+), 38 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index c25d0a5fda69..0250294c904a 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -334,6 +334,7 @@ void			xprt_free_slot(struct rpc_xprt *xprt,
 				       struct rpc_rqst *req);
 void			xprt_lock_and_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task);
 bool			xprt_prepare_transmit(struct rpc_task *task);
+void			xprt_request_enqueue_receive(struct rpc_task *task);
 void			xprt_transmit(struct rpc_task *task);
 void			xprt_end_transmit(struct rpc_task *task);
 int			xprt_adjust_timeout(struct rpc_rqst *req);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index a858366cd15d..414966273a3f 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1962,6 +1962,11 @@ call_transmit(struct rpc_task *task)
 			return;
 		}
 	}
+
+	/* Add task to reply queue before transmission to avoid races */
+	if (rpc_reply_expected(task))
+		xprt_request_enqueue_receive(task);
+
 	if (!xprt_prepare_transmit(task))
 		return;
 	task->tk_action = call_transmit_status;
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index eda305de9f77..6a5f2dbdc00f 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -884,6 +884,57 @@ static void xprt_wait_on_pinned_rqst(struct rpc_rqst *req)
 	wait_var_event(&req->rq_pin, !xprt_is_pinned_rqst(req));
 }
 
+static bool
+xprt_request_data_received(struct rpc_task *task)
+{
+	return !test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate) &&
+		READ_ONCE(task->tk_rqstp->rq_reply_bytes_recvd) != 0;
+}
+
+/**
+ * xprt_request_enqueue_receive - Add an request to the receive queue
+ * @task: RPC task
+ *
+ */
+void
+xprt_request_enqueue_receive(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *xprt = req->rq_xprt;
+
+	spin_lock(&xprt->queue_lock);
+	if (xprt_request_data_received(task) || !list_empty(&req->rq_list)) {
+		spin_unlock(&xprt->queue_lock);
+		return;
+	}
+
+	/* Update the softirq receive buffer */
+	memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
+			sizeof(req->rq_private_buf));
+
+	/* Add request to the receive list */
+	list_add_tail(&req->rq_list, &xprt->recv);
+	set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
+	spin_unlock(&xprt->queue_lock);
+
+	xprt_reset_majortimeo(req);
+	/* Turn off autodisconnect */
+	del_singleshot_timer_sync(&xprt->timer);
+}
+
+/**
+ * xprt_request_dequeue_receive_locked - Remove a request from the receive queue
+ * @task: RPC task
+ *
+ * Caller must hold xprt->queue_lock.
+ */
+static void
+xprt_request_dequeue_receive_locked(struct rpc_task *task)
+{
+	clear_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
+	list_del_init(&task->tk_rqstp->rq_list);
+}
+
 /**
  * xprt_update_rtt - Update RPC RTT statistics
  * @task: RPC request that recently completed
@@ -923,24 +974,16 @@ void xprt_complete_rqst(struct rpc_task *task, int copied)
 
 	xprt->stat.recvs++;
 
-	list_del_init(&req->rq_list);
 	req->rq_private_buf.len = copied;
 	/* Ensure all writes are done before we update */
 	/* req->rq_reply_bytes_recvd */
 	smp_wmb();
 	req->rq_reply_bytes_recvd = copied;
-	clear_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
+	xprt_request_dequeue_receive_locked(task);
 	rpc_wake_up_queued_task(&xprt->pending, task);
 }
 EXPORT_SYMBOL_GPL(xprt_complete_rqst);
 
-static bool
-xprt_request_data_received(struct rpc_task *task)
-{
-	return !test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate) &&
-		task->tk_rqstp->rq_reply_bytes_recvd != 0;
-}
-
 static void xprt_timer(struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
@@ -1014,32 +1057,15 @@ void xprt_transmit(struct rpc_task *task)
 
 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, req->rq_slen);
 
-	if (!req->rq_reply_bytes_recvd) {
-
+	if (!req->rq_bytes_sent) {
+		if (xprt_request_data_received(task))
+			return;
 		/* Verify that our message lies in the RPCSEC_GSS window */
-		if (!req->rq_bytes_sent && rpcauth_xmit_need_reencode(task)) {
+		if (rpcauth_xmit_need_reencode(task)) {
 			task->tk_status = -EBADMSG;
 			return;
 		}
-
-		if (list_empty(&req->rq_list) && rpc_reply_expected(task)) {
-			/*
-			 * Add to the list only if we're expecting a reply
-			 */
-			/* Update the softirq receive buffer */
-			memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
-					sizeof(req->rq_private_buf));
-			/* Add request to the receive list */
-			spin_lock(&xprt->queue_lock);
-			list_add_tail(&req->rq_list, &xprt->recv);
-			set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
-			spin_unlock(&xprt->queue_lock);
-			xprt_reset_majortimeo(req);
-			/* Turn off autodisconnect */
-			del_singleshot_timer_sync(&xprt->timer);
-		}
-	} else if (xprt_request_data_received(task) && !req->rq_bytes_sent)
-		return;
+	}
 
 	connect_cookie = xprt->connect_cookie;
 	status = xprt->ops->send_request(task);
@@ -1376,13 +1402,11 @@ void xprt_release(struct rpc_task *task)
 	else if (task->tk_client)
 		rpc_count_iostats(task, task->tk_client->cl_metrics);
 	spin_lock(&xprt->queue_lock);
-	if (!list_empty(&req->rq_list)) {
-		list_del_init(&req->rq_list);
-		if (atomic_read(&req->rq_pin)) {
-			spin_unlock(&xprt->queue_lock);
-			xprt_wait_on_pinned_rqst(req);
-			spin_lock(&xprt->queue_lock);
-		}
+	xprt_request_dequeue_receive_locked(task);
+	while (xprt_is_pinned_rqst(req)) {
+		spin_unlock(&xprt->queue_lock);
+		xprt_wait_on_pinned_rqst(req);
+		spin_lock(&xprt->queue_lock);
 	}
 	spin_unlock(&xprt->queue_lock);
 	spin_lock_bh(&xprt->transport_lock);
-- 
2.17.1

[PATCH v2 15/34] SUNRPC: Refactor xprt_transmit() to remove wait for reply code

[Trond Myklebust]

Allow the caller in clnt.c to call into the code to wait for a reply
after calling xprt_transmit(). Again, the reason is that the backchannel
code does not need this functionality.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h |  1 +
 net/sunrpc/clnt.c           | 10 +-----
 net/sunrpc/xprt.c           | 72 ++++++++++++++++++++++++++-----------
 3 files changed, 53 insertions(+), 30 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 0250294c904a..4fa2af087cff 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -335,6 +335,7 @@ void			xprt_free_slot(struct rpc_xprt *xprt,
 void			xprt_lock_and_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task);
 bool			xprt_prepare_transmit(struct rpc_task *task);
 void			xprt_request_enqueue_receive(struct rpc_task *task);
+void			xprt_request_wait_receive(struct rpc_task *task);
 void			xprt_transmit(struct rpc_task *task);
 void			xprt_end_transmit(struct rpc_task *task);
 int			xprt_adjust_timeout(struct rpc_rqst *req);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 414966273a3f..775d6e80b6e8 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1975,15 +1975,6 @@ call_transmit(struct rpc_task *task)
 		return;
 	if (is_retrans)
 		task->tk_client->cl_stats->rpcretrans++;
-	/*
-	 * On success, ensure that we call xprt_end_transmit() before sleeping
-	 * in order to allow access to the socket to other RPC requests.
-	 */
-	call_transmit_status(task);
-	if (rpc_reply_expected(task))
-		return;
-	task->tk_action = rpc_exit_task;
-	rpc_wake_up_queued_task(&task->tk_rqstp->rq_xprt->pending, task);
 }
 
 /*
@@ -2000,6 +1991,7 @@ call_transmit_status(struct rpc_task *task)
 	 */
 	if (task->tk_status == 0) {
 		xprt_end_transmit(task);
+		xprt_request_wait_receive(task);
 		return;
 	}
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 6a5f2dbdc00f..1fba837e5390 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -654,6 +654,22 @@ void xprt_force_disconnect(struct rpc_xprt *xprt)
 }
 EXPORT_SYMBOL_GPL(xprt_force_disconnect);
 
+static unsigned int
+xprt_connect_cookie(struct rpc_xprt *xprt)
+{
+	return READ_ONCE(xprt->connect_cookie);
+}
+
+static bool
+xprt_request_retransmit_after_disconnect(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *xprt = req->rq_xprt;
+
+	return req->rq_connect_cookie != xprt_connect_cookie(xprt) ||
+		!xprt_connected(xprt);
+}
+
 /**
  * xprt_conditional_disconnect - force a transport to disconnect
  * @xprt: transport to disconnect
@@ -1000,6 +1016,39 @@ static void xprt_timer(struct rpc_task *task)
 		task->tk_status = 0;
 }
 
+/**
+ * xprt_request_wait_receive - wait for the reply to an RPC request
+ * @task: RPC task about to send a request
+ *
+ */
+void xprt_request_wait_receive(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *xprt = req->rq_xprt;
+
+	if (!test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate))
+		return;
+	/*
+	 * Sleep on the pending queue if we're expecting a reply.
+	 * The spinlock ensures atomicity between the test of
+	 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
+	 */
+	spin_lock(&xprt->queue_lock);
+	if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
+		xprt->ops->set_retrans_timeout(task);
+		rpc_sleep_on(&xprt->pending, task, xprt_timer);
+		/*
+		 * Send an extra queue wakeup call if the
+		 * connection was dropped in case the call to
+		 * rpc_sleep_on() raced.
+		 */
+		if (xprt_request_retransmit_after_disconnect(task))
+			rpc_wake_up_queued_task_set_status(&xprt->pending,
+					task, -ENOTCONN);
+	}
+	spin_unlock(&xprt->queue_lock);
+}
+
 /**
  * xprt_prepare_transmit - reserve the transport before sending a request
  * @task: RPC task about to send a request
@@ -1019,9 +1068,8 @@ bool xprt_prepare_transmit(struct rpc_task *task)
 			task->tk_status = req->rq_reply_bytes_recvd;
 			goto out_unlock;
 		}
-		if ((task->tk_flags & RPC_TASK_NO_RETRANS_TIMEOUT)
-		    && xprt_connected(xprt)
-		    && req->rq_connect_cookie == xprt->connect_cookie) {
+		if ((task->tk_flags & RPC_TASK_NO_RETRANS_TIMEOUT) &&
+		    !xprt_request_retransmit_after_disconnect(task)) {
 			xprt->ops->set_retrans_timeout(task);
 			rpc_sleep_on(&xprt->pending, task, xprt_timer);
 			goto out_unlock;
@@ -1082,8 +1130,6 @@ void xprt_transmit(struct rpc_task *task)
 	task->tk_flags |= RPC_TASK_SENT;
 	spin_lock_bh(&xprt->transport_lock);
 
-	xprt->ops->set_retrans_timeout(task);
-
 	xprt->stat.sends++;
 	xprt->stat.req_u += xprt->stat.sends - xprt->stat.recvs;
 	xprt->stat.bklog_u += xprt->backlog.qlen;
@@ -1092,22 +1138,6 @@ void xprt_transmit(struct rpc_task *task)
 	spin_unlock_bh(&xprt->transport_lock);
 
 	req->rq_connect_cookie = connect_cookie;
-	if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
-		/*
-		 * Sleep on the pending queue if we're expecting a reply.
-		 * The spinlock ensures atomicity between the test of
-		 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
-		 */
-		spin_lock(&xprt->queue_lock);
-		if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
-			rpc_sleep_on(&xprt->pending, task, xprt_timer);
-			/* Wake up immediately if the connection was dropped */
-			if (!xprt_connected(xprt))
-				rpc_wake_up_queued_task_set_status(&xprt->pending,
-						task, -ENOTCONN);
-		}
-		spin_unlock(&xprt->queue_lock);
-	}
 }
 
 static void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task)
-- 
2.17.1

[PATCH v2 16/34] SUNRPC: Minor cleanup for call_transmit()

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 31 ++++++++++++++-----------------
 1 file changed, 14 insertions(+), 17 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 775d6e80b6e8..5fbd9875544e 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1946,9 +1946,7 @@ call_transmit(struct rpc_task *task)
 
 	dprint_status(task);
 
-	task->tk_action = call_status;
-	if (task->tk_status < 0)
-		return;
+	task->tk_action = call_transmit_status;
 	/* Encode here so that rpcsec_gss can use correct sequence number. */
 	if (rpc_task_need_encode(task)) {
 		rpc_xdr_encode(task);
@@ -1969,7 +1967,6 @@ call_transmit(struct rpc_task *task)
 
 	if (!xprt_prepare_transmit(task))
 		return;
-	task->tk_action = call_transmit_status;
 	xprt_transmit(task);
 	if (task->tk_status < 0)
 		return;
@@ -1996,19 +1993,28 @@ call_transmit_status(struct rpc_task *task)
 	}
 
 	switch (task->tk_status) {
-	case -EAGAIN:
-	case -ENOBUFS:
-		break;
 	default:
 		dprint_status(task);
 		xprt_end_transmit(task);
 		break;
+	case -EBADMSG:
+		clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
+		task->tk_action = call_transmit;
+		xprt_end_transmit(task);
+		break;
 		/*
 		 * Special cases: if we've been waiting on the
 		 * socket's write_space() callback, or if the
 		 * socket just returned a connection error,
 		 * then hold onto the transport lock.
 		 */
+	case -ENOBUFS:
+		rpc_delay(task, HZ>>2);
+		/* fall through */
+	case -EAGAIN:
+		task->tk_action = call_transmit;
+		task->tk_status = 0;
+		break;
 	case -ECONNREFUSED:
 	case -EHOSTDOWN:
 	case -ENETDOWN:
@@ -2163,22 +2169,13 @@ call_status(struct rpc_task *task)
 		/* fall through */
 	case -EPIPE:
 	case -ENOTCONN:
-		task->tk_action = call_bind;
-		break;
-	case -ENOBUFS:
-		rpc_delay(task, HZ>>2);
-		/* fall through */
 	case -EAGAIN:
-		task->tk_action = call_transmit;
+		task->tk_action = call_bind;
 		break;
 	case -EIO:
 		/* shutdown or soft timeout */
 		rpc_exit(task, status);
 		break;
-	case -EBADMSG:
-		clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
-		task->tk_action = call_transmit;
-		break;
 	default:
 		if (clnt->cl_chatty)
 			printk("%s: RPC call returned error %d\n",
-- 
2.17.1

[PATCH v2 17/34] SUNRPC: Distinguish between the slot allocation list and receive queue

[Trond Myklebust]

When storing a struct rpc_rqst on the slot allocation list, we currently
use the same field 'rq_list' as we use to store the request on the
receive queue. Since the structure is never on both lists at the same
time, this is OK.
However, for clarity, let's make that a union with different names for
the different lists so that we can more easily distinguish between
the two states.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h       |  9 +++++++--
 net/sunrpc/backchannel_rqst.c     |  2 +-
 net/sunrpc/xprt.c                 | 16 ++++++++--------
 net/sunrpc/xprtrdma/backchannel.c |  2 +-
 4 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 4fa2af087cff..9cec2d0811f2 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -82,7 +82,11 @@ struct rpc_rqst {
 	struct page		**rq_enc_pages;	/* scratch pages for use by
 						   gss privacy code */
 	void (*rq_release_snd_buf)(struct rpc_rqst *); /* release rq_enc_pages */
-	struct list_head	rq_list;
+
+	union {
+		struct list_head	rq_list;	/* Slot allocation list */
+		struct list_head	rq_recv;	/* Receive queue */
+	};
 
 	void			*rq_buffer;	/* Call XDR encode buffer */
 	size_t			rq_callsize;
@@ -249,7 +253,8 @@ struct rpc_xprt {
 	struct list_head	bc_pa_list;	/* List of preallocated
 						 * backchannel rpc_rqst's */
 #endif /* CONFIG_SUNRPC_BACKCHANNEL */
-	struct list_head	recv;
+
+	struct list_head	recv_queue;	/* Receive queue */
 
 	struct {
 		unsigned long		bind_count,	/* total number of binds */
diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c
index 3c15a99b9700..92e9ad30ec2f 100644
--- a/net/sunrpc/backchannel_rqst.c
+++ b/net/sunrpc/backchannel_rqst.c
@@ -91,7 +91,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt *xprt, gfp_t gfp_flags)
 		return NULL;
 
 	req->rq_xprt = xprt;
-	INIT_LIST_HEAD(&req->rq_list);
+	INIT_LIST_HEAD(&req->rq_recv);
 	INIT_LIST_HEAD(&req->rq_bc_list);
 
 	/* Preallocate one XDR receive buffer */
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 1fba837e5390..7f53e97a624f 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -708,7 +708,7 @@ static void
 xprt_schedule_autodisconnect(struct rpc_xprt *xprt)
 	__must_hold(&xprt->transport_lock)
 {
-	if (list_empty(&xprt->recv) && xprt_has_timer(xprt))
+	if (list_empty(&xprt->recv_queue) && xprt_has_timer(xprt))
 		mod_timer(&xprt->timer, xprt->last_used + xprt->idle_timeout);
 }
 
@@ -718,7 +718,7 @@ xprt_init_autodisconnect(struct timer_list *t)
 	struct rpc_xprt *xprt = from_timer(xprt, t, timer);
 
 	spin_lock(&xprt->transport_lock);
-	if (!list_empty(&xprt->recv))
+	if (!list_empty(&xprt->recv_queue))
 		goto out_abort;
 	/* Reset xprt->last_used to avoid connect/autodisconnect cycling */
 	xprt->last_used = jiffies;
@@ -848,7 +848,7 @@ struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid)
 {
 	struct rpc_rqst *entry;
 
-	list_for_each_entry(entry, &xprt->recv, rq_list)
+	list_for_each_entry(entry, &xprt->recv_queue, rq_recv)
 		if (entry->rq_xid == xid) {
 			trace_xprt_lookup_rqst(xprt, xid, 0);
 			entry->rq_rtt = ktime_sub(ktime_get(), entry->rq_xtime);
@@ -919,7 +919,7 @@ xprt_request_enqueue_receive(struct rpc_task *task)
 	struct rpc_xprt *xprt = req->rq_xprt;
 
 	spin_lock(&xprt->queue_lock);
-	if (xprt_request_data_received(task) || !list_empty(&req->rq_list)) {
+	if (xprt_request_data_received(task) || !list_empty(&req->rq_recv)) {
 		spin_unlock(&xprt->queue_lock);
 		return;
 	}
@@ -929,7 +929,7 @@ xprt_request_enqueue_receive(struct rpc_task *task)
 			sizeof(req->rq_private_buf));
 
 	/* Add request to the receive list */
-	list_add_tail(&req->rq_list, &xprt->recv);
+	list_add_tail(&req->rq_recv, &xprt->recv_queue);
 	set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
 	spin_unlock(&xprt->queue_lock);
 
@@ -948,7 +948,7 @@ static void
 xprt_request_dequeue_receive_locked(struct rpc_task *task)
 {
 	clear_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
-	list_del_init(&task->tk_rqstp->rq_list);
+	list_del_init(&task->tk_rqstp->rq_recv);
 }
 
 /**
@@ -1337,7 +1337,7 @@ xprt_request_init(struct rpc_task *task)
 	struct rpc_xprt *xprt = task->tk_xprt;
 	struct rpc_rqst	*req = task->tk_rqstp;
 
-	INIT_LIST_HEAD(&req->rq_list);
+	INIT_LIST_HEAD(&req->rq_recv);
 	req->rq_timeout = task->tk_client->cl_timeout->to_initval;
 	req->rq_task	= task;
 	req->rq_xprt    = xprt;
@@ -1471,7 +1471,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 	spin_lock_init(&xprt->queue_lock);
 
 	INIT_LIST_HEAD(&xprt->free);
-	INIT_LIST_HEAD(&xprt->recv);
+	INIT_LIST_HEAD(&xprt->recv_queue);
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 	spin_lock_init(&xprt->bc_pa_lock);
 	INIT_LIST_HEAD(&xprt->bc_pa_list);
diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c
index 90adeff4c06b..40c7c7306a99 100644
--- a/net/sunrpc/xprtrdma/backchannel.c
+++ b/net/sunrpc/xprtrdma/backchannel.c
@@ -51,7 +51,7 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt,
 		rqst = &req->rl_slot;
 
 		rqst->rq_xprt = xprt;
-		INIT_LIST_HEAD(&rqst->rq_list);
+		INIT_LIST_HEAD(&rqst->rq_recv);
 		INIT_LIST_HEAD(&rqst->rq_bc_list);
 		__set_bit(RPC_BC_PA_IN_USE, &rqst->rq_bc_pa_state);
 		spin_lock_bh(&xprt->bc_pa_lock);
-- 
2.17.1

[PATCH v2 18/34] NFS: Add a transmission queue for RPC requests

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h       |  6 +++
 net/sunrpc/backchannel_rqst.c     |  1 +
 net/sunrpc/clnt.c                 |  6 +--
 net/sunrpc/xprt.c                 | 74 +++++++++++++++++++++++++++----
 net/sunrpc/xprtrdma/backchannel.c |  1 +
 5 files changed, 77 insertions(+), 11 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 9cec2d0811f2..81a6c2c8dfc7 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -88,6 +88,8 @@ struct rpc_rqst {
 		struct list_head	rq_recv;	/* Receive queue */
 	};
 
+	struct list_head	rq_xmit;	/* Send queue */
+
 	void			*rq_buffer;	/* Call XDR encode buffer */
 	size_t			rq_callsize;
 	void			*rq_rbuffer;	/* Reply XDR decode buffer */
@@ -242,6 +244,9 @@ struct rpc_xprt {
 	spinlock_t		queue_lock;	/* send/receive queue lock */
 	u32			xid;		/* Next XID value to use */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
+
+	struct list_head	xmit_queue;	/* Send queue */
+
 	struct svc_xprt		*bc_xprt;	/* NFSv4.1 backchannel */
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 	struct svc_serv		*bc_serv;       /* The RPC service which will */
@@ -339,6 +344,7 @@ void			xprt_free_slot(struct rpc_xprt *xprt,
 				       struct rpc_rqst *req);
 void			xprt_lock_and_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task);
 bool			xprt_prepare_transmit(struct rpc_task *task);
+void			xprt_request_enqueue_transmit(struct rpc_task *task);
 void			xprt_request_enqueue_receive(struct rpc_task *task);
 void			xprt_request_wait_receive(struct rpc_task *task);
 void			xprt_transmit(struct rpc_task *task);
diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c
index 92e9ad30ec2f..39b394b7dae3 100644
--- a/net/sunrpc/backchannel_rqst.c
+++ b/net/sunrpc/backchannel_rqst.c
@@ -92,6 +92,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt *xprt, gfp_t gfp_flags)
 
 	req->rq_xprt = xprt;
 	INIT_LIST_HEAD(&req->rq_recv);
+	INIT_LIST_HEAD(&req->rq_xmit);
 	INIT_LIST_HEAD(&req->rq_bc_list);
 
 	/* Preallocate one XDR receive buffer */
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 5fbd9875544e..a817f70d6192 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1759,8 +1759,6 @@ rpc_xdr_encode(struct rpc_task *task)
 
 	task->tk_status = rpcauth_wrap_req(task, encode, req, p,
 			task->tk_msg.rpc_argp);
-	if (task->tk_status == 0)
-		set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 }
 
 /*
@@ -1959,11 +1957,13 @@ call_transmit(struct rpc_task *task)
 				rpc_exit(task, task->tk_status);
 			return;
 		}
+		set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 	}
 
 	/* Add task to reply queue before transmission to avoid races */
 	if (rpc_reply_expected(task))
 		xprt_request_enqueue_receive(task);
+	xprt_request_enqueue_transmit(task);
 
 	if (!xprt_prepare_transmit(task))
 		return;
@@ -1998,7 +1998,6 @@ call_transmit_status(struct rpc_task *task)
 		xprt_end_transmit(task);
 		break;
 	case -EBADMSG:
-		clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 		task->tk_action = call_transmit;
 		xprt_end_transmit(task);
 		break;
@@ -2049,6 +2048,7 @@ call_bc_transmit(struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
 
+	xprt_request_enqueue_transmit(task);
 	if (!xprt_prepare_transmit(task))
 		goto out_retry;
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 7f53e97a624f..8e8c345eedf7 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1049,6 +1049,64 @@ void xprt_request_wait_receive(struct rpc_task *task)
 	spin_unlock(&xprt->queue_lock);
 }
 
+static bool
+xprt_request_need_transmit(struct rpc_task *task)
+{
+	return !(task->tk_flags & RPC_TASK_NO_RETRANS_TIMEOUT) ||
+		xprt_request_retransmit_after_disconnect(task);
+}
+
+/**
+ * xprt_request_enqueue_transmit - queue a task for transmission
+ * @task: pointer to rpc_task
+ *
+ * Add a task to the transmission queue.
+ */
+void
+xprt_request_enqueue_transmit(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *xprt = req->rq_xprt;
+
+	spin_lock(&xprt->queue_lock);
+	if (list_empty(&req->rq_xmit) && xprt_request_need_transmit(task) &&
+	    test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+		list_add_tail(&req->rq_xmit, &xprt->xmit_queue);
+	spin_unlock(&xprt->queue_lock);
+}
+
+/**
+ * xprt_request_dequeue_transmit_locked - remove a task from the transmission queue
+ * @task: pointer to rpc_task
+ *
+ * Remove a task from the transmission queue
+ * Caller must hold xprt->queue_lock
+ */
+static void
+xprt_request_dequeue_transmit_locked(struct rpc_task *task)
+{
+	xprt_task_clear_bytes_sent(task);
+	clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
+	list_del_init(&task->tk_rqstp->rq_xmit);
+}
+
+/**
+ * xprt_request_dequeue_transmit - remove a task from the transmission queue
+ * @task: pointer to rpc_task
+ *
+ * Remove a task from the transmission queue
+ */
+static void
+xprt_request_dequeue_transmit(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *xprt = req->rq_xprt;
+
+	spin_lock(&xprt->queue_lock);
+	xprt_request_dequeue_transmit_locked(task);
+	spin_unlock(&xprt->queue_lock);
+}
+
 /**
  * xprt_prepare_transmit - reserve the transport before sending a request
  * @task: RPC task about to send a request
@@ -1068,12 +1126,8 @@ bool xprt_prepare_transmit(struct rpc_task *task)
 			task->tk_status = req->rq_reply_bytes_recvd;
 			goto out_unlock;
 		}
-		if ((task->tk_flags & RPC_TASK_NO_RETRANS_TIMEOUT) &&
-		    !xprt_request_retransmit_after_disconnect(task)) {
-			xprt->ops->set_retrans_timeout(task);
-			rpc_sleep_on(&xprt->pending, task, xprt_timer);
+		if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
 			goto out_unlock;
-		}
 	}
 	if (!xprt->ops->reserve_xprt(xprt, task)) {
 		task->tk_status = -EAGAIN;
@@ -1107,11 +1161,11 @@ void xprt_transmit(struct rpc_task *task)
 
 	if (!req->rq_bytes_sent) {
 		if (xprt_request_data_received(task))
-			return;
+			goto out_dequeue;
 		/* Verify that our message lies in the RPCSEC_GSS window */
 		if (rpcauth_xmit_need_reencode(task)) {
 			task->tk_status = -EBADMSG;
-			return;
+			goto out_dequeue;
 		}
 	}
 
@@ -1126,7 +1180,6 @@ void xprt_transmit(struct rpc_task *task)
 	xprt_inject_disconnect(xprt);
 
 	dprintk("RPC: %5u xmit complete\n", task->tk_pid);
-	clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 	task->tk_flags |= RPC_TASK_SENT;
 	spin_lock_bh(&xprt->transport_lock);
 
@@ -1138,6 +1191,8 @@ void xprt_transmit(struct rpc_task *task)
 	spin_unlock_bh(&xprt->transport_lock);
 
 	req->rq_connect_cookie = connect_cookie;
+out_dequeue:
+	xprt_request_dequeue_transmit(task);
 }
 
 static void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task)
@@ -1338,6 +1393,7 @@ xprt_request_init(struct rpc_task *task)
 	struct rpc_rqst	*req = task->tk_rqstp;
 
 	INIT_LIST_HEAD(&req->rq_recv);
+	INIT_LIST_HEAD(&req->rq_xmit);
 	req->rq_timeout = task->tk_client->cl_timeout->to_initval;
 	req->rq_task	= task;
 	req->rq_xprt    = xprt;
@@ -1433,6 +1489,7 @@ void xprt_release(struct rpc_task *task)
 		rpc_count_iostats(task, task->tk_client->cl_metrics);
 	spin_lock(&xprt->queue_lock);
 	xprt_request_dequeue_receive_locked(task);
+	xprt_request_dequeue_transmit_locked(task);
 	while (xprt_is_pinned_rqst(req)) {
 		spin_unlock(&xprt->queue_lock);
 		xprt_wait_on_pinned_rqst(req);
@@ -1472,6 +1529,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 
 	INIT_LIST_HEAD(&xprt->free);
 	INIT_LIST_HEAD(&xprt->recv_queue);
+	INIT_LIST_HEAD(&xprt->xmit_queue);
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 	spin_lock_init(&xprt->bc_pa_lock);
 	INIT_LIST_HEAD(&xprt->bc_pa_list);
diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c
index 40c7c7306a99..fc01fdabbbce 100644
--- a/net/sunrpc/xprtrdma/backchannel.c
+++ b/net/sunrpc/xprtrdma/backchannel.c
@@ -52,6 +52,7 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt,
 
 		rqst->rq_xprt = xprt;
 		INIT_LIST_HEAD(&rqst->rq_recv);
+		INIT_LIST_HEAD(&rqst->rq_xmit);
 		INIT_LIST_HEAD(&rqst->rq_bc_list);
 		__set_bit(RPC_BC_PA_IN_USE, &rqst->rq_bc_pa_state);
 		spin_lock_bh(&xprt->bc_pa_lock);
-- 
2.17.1

[PATCH v2 19/34] SUNRPC: Refactor RPC call encoding

[Trond Myklebust]

Move the call encoding so that it occurs before the transport connection
etc.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 67 +++++++++++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 28 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index a817f70d6192..497a30762a6d 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -61,6 +61,7 @@ static void	call_start(struct rpc_task *task);
 static void	call_reserve(struct rpc_task *task);
 static void	call_reserveresult(struct rpc_task *task);
 static void	call_allocate(struct rpc_task *task);
+static void	call_encode(struct rpc_task *task);
 static void	call_decode(struct rpc_task *task);
 static void	call_bind(struct rpc_task *task);
 static void	call_bind_status(struct rpc_task *task);
@@ -1680,7 +1681,7 @@ call_allocate(struct rpc_task *task)
 	dprint_status(task);
 
 	task->tk_status = 0;
-	task->tk_action = call_bind;
+	task->tk_action = call_encode;
 
 	if (req->rq_buffer)
 		return;
@@ -1727,9 +1728,6 @@ rpc_task_need_encode(struct rpc_task *task)
 	return test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate) == 0;
 }
 
-/*
- * 3.	Encode arguments of an RPC call
- */
 static void
 rpc_xdr_encode(struct rpc_task *task)
 {
@@ -1745,6 +1743,7 @@ rpc_xdr_encode(struct rpc_task *task)
 	xdr_buf_init(&req->rq_rcv_buf,
 		     req->rq_rbuffer,
 		     req->rq_rcvsize);
+	req->rq_bytes_sent = 0;
 
 	p = rpc_encode_header(task);
 	if (p == NULL) {
@@ -1761,6 +1760,34 @@ rpc_xdr_encode(struct rpc_task *task)
 			task->tk_msg.rpc_argp);
 }
 
+/*
+ * 3.	Encode arguments of an RPC call
+ */
+static void
+call_encode(struct rpc_task *task)
+{
+	if (!rpc_task_need_encode(task))
+		goto out;
+	/* Encode here so that rpcsec_gss can use correct sequence number. */
+	rpc_xdr_encode(task);
+	/* Did the encode result in an error condition? */
+	if (task->tk_status != 0) {
+		/* Was the error nonfatal? */
+		if (task->tk_status == -EAGAIN)
+			rpc_delay(task, HZ >> 4);
+		else
+			rpc_exit(task, task->tk_status);
+		return;
+	}
+
+	/* Add task to reply queue before transmission to avoid races */
+	if (rpc_reply_expected(task))
+		xprt_request_enqueue_receive(task);
+	set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
+out:
+	task->tk_action = call_bind;
+}
+
 /*
  * 4.	Get the server port number if not yet set
  */
@@ -1945,25 +1972,9 @@ call_transmit(struct rpc_task *task)
 	dprint_status(task);
 
 	task->tk_action = call_transmit_status;
-	/* Encode here so that rpcsec_gss can use correct sequence number. */
-	if (rpc_task_need_encode(task)) {
-		rpc_xdr_encode(task);
-		/* Did the encode result in an error condition? */
-		if (task->tk_status != 0) {
-			/* Was the error nonfatal? */
-			if (task->tk_status == -EAGAIN)
-				rpc_delay(task, HZ >> 4);
-			else
-				rpc_exit(task, task->tk_status);
-			return;
-		}
-		set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
-	}
-
-	/* Add task to reply queue before transmission to avoid races */
-	if (rpc_reply_expected(task))
-		xprt_request_enqueue_receive(task);
 	xprt_request_enqueue_transmit(task);
+	if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+		return;
 
 	if (!xprt_prepare_transmit(task))
 		return;
@@ -1998,8 +2009,8 @@ call_transmit_status(struct rpc_task *task)
 		xprt_end_transmit(task);
 		break;
 	case -EBADMSG:
-		task->tk_action = call_transmit;
 		xprt_end_transmit(task);
+		task->tk_action = call_encode;
 		break;
 		/*
 		 * Special cases: if we've been waiting on the
@@ -2170,7 +2181,7 @@ call_status(struct rpc_task *task)
 	case -EPIPE:
 	case -ENOTCONN:
 	case -EAGAIN:
-		task->tk_action = call_bind;
+		task->tk_action = call_encode;
 		break;
 	case -EIO:
 		/* shutdown or soft timeout */
@@ -2235,7 +2246,7 @@ call_timeout(struct rpc_task *task)
 	rpcauth_invalcred(task);
 
 retry:
-	task->tk_action = call_bind;
+	task->tk_action = call_encode;
 	task->tk_status = 0;
 }
 
@@ -2279,7 +2290,7 @@ call_decode(struct rpc_task *task)
 
 	if (req->rq_rcv_buf.len < 12) {
 		if (!RPC_IS_SOFT(task)) {
-			task->tk_action = call_bind;
+			task->tk_action = call_encode;
 			goto out_retry;
 		}
 		dprintk("RPC:       %s: too small RPC reply size (%d bytes)\n",
@@ -2410,7 +2421,7 @@ rpc_verify_header(struct rpc_task *task)
 			task->tk_garb_retry--;
 			dprintk("RPC: %5u %s: retry garbled creds\n",
 					task->tk_pid, __func__);
-			task->tk_action = call_bind;
+			task->tk_action = call_encode;
 			goto out_retry;
 		case RPC_AUTH_TOOWEAK:
 			printk(KERN_NOTICE "RPC: server %s requires stronger "
@@ -2479,7 +2490,7 @@ rpc_verify_header(struct rpc_task *task)
 		task->tk_garb_retry--;
 		dprintk("RPC: %5u %s: retrying\n",
 				task->tk_pid, __func__);
-		task->tk_action = call_bind;
+		task->tk_action = call_encode;
 out_retry:
 		return ERR_PTR(-EAGAIN);
 	}
-- 
2.17.1

[PATCH v2 20/34] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request()

[Trond Myklebust]

When we shift to using the transmit queue, then the task that holds the
write lock will not necessarily be the same as the one being transmitted.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h                |  2 +-
 net/sunrpc/xprt.c                          |  2 +-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  3 +--
 net/sunrpc/xprtrdma/transport.c            |  5 ++--
 net/sunrpc/xprtsock.c                      | 27 +++++++++++-----------
 5 files changed, 18 insertions(+), 21 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 81a6c2c8dfc7..6d91acfe0644 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -140,7 +140,7 @@ struct rpc_xprt_ops {
 	void		(*connect)(struct rpc_xprt *xprt, struct rpc_task *task);
 	int		(*buf_alloc)(struct rpc_task *task);
 	void		(*buf_free)(struct rpc_task *task);
-	int		(*send_request)(struct rpc_task *task);
+	int		(*send_request)(struct rpc_rqst *req, struct rpc_task *task);
 	void		(*set_retrans_timeout)(struct rpc_task *task);
 	void		(*timer)(struct rpc_xprt *xprt, struct rpc_task *task);
 	void		(*release_request)(struct rpc_task *task);
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 8e8c345eedf7..7c80f93562e5 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1170,7 +1170,7 @@ void xprt_transmit(struct rpc_task *task)
 	}
 
 	connect_cookie = xprt->connect_cookie;
-	status = xprt->ops->send_request(task);
+	status = xprt->ops->send_request(req, task);
 	trace_xprt_transmit(xprt, req->rq_xid, status);
 	if (status != 0) {
 		task->tk_status = status;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index 09b12b7568fe..d1618c70edb4 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -215,9 +215,8 @@ rpcrdma_bc_send_request(struct svcxprt_rdma *rdma, struct rpc_rqst *rqst)
  * connection.
  */
 static int
-xprt_rdma_bc_send_request(struct rpc_task *task)
+xprt_rdma_bc_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
 {
-	struct rpc_rqst *rqst = task->tk_rqstp;
 	struct svc_xprt *sxprt = rqst->rq_xprt->bc_xprt;
 	struct svcxprt_rdma *rdma;
 	int ret;
diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
index 143ce2579ba9..fa684bf4d090 100644
--- a/net/sunrpc/xprtrdma/transport.c
+++ b/net/sunrpc/xprtrdma/transport.c
@@ -706,9 +706,8 @@ xprt_rdma_free(struct rpc_task *task)
  *		sent. Do not try to send this message again.
  */
 static int
-xprt_rdma_send_request(struct rpc_task *task)
+xprt_rdma_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
 {
-	struct rpc_rqst *rqst = task->tk_rqstp;
 	struct rpc_xprt *xprt = rqst->rq_xprt;
 	struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
 	struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt);
@@ -741,7 +740,7 @@ xprt_rdma_send_request(struct rpc_task *task)
 	/* An RPC with no reply will throw off credit accounting,
 	 * so drop the connection to reset the credit grant.
 	 */
-	if (!rpc_reply_expected(task))
+	if (!rpc_reply_expected(rqst->rq_task))
 		goto drop_connection;
 	return 0;
 
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 8d6404259ff9..b8143eded4af 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -449,12 +449,12 @@ static void xs_nospace_callback(struct rpc_task *task)
 
 /**
  * xs_nospace - place task on wait queue if transmit was incomplete
+ * @req: pointer to RPC request
  * @task: task to put to sleep
  *
  */
-static int xs_nospace(struct rpc_task *task)
+static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
 	struct sock *sk = transport->inet;
@@ -513,6 +513,7 @@ static inline void xs_encode_stream_record_marker(struct xdr_buf *buf)
 
 /**
  * xs_local_send_request - write an RPC request to an AF_LOCAL socket
+ * @req: pointer to RPC request
  * @task: RPC task that manages the state of an RPC request
  *
  * Return values:
@@ -522,9 +523,8 @@ static inline void xs_encode_stream_record_marker(struct xdr_buf *buf)
  * ENOTCONN:	Caller needs to invoke connect logic then call again
  *    other:	Some other error occured, the request was not sent
  */
-static int xs_local_send_request(struct rpc_task *task)
+static int xs_local_send_request(struct rpc_rqst *req, struct rpc_task *task)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport =
 				container_of(xprt, struct sock_xprt, xprt);
@@ -569,7 +569,7 @@ static int xs_local_send_request(struct rpc_task *task)
 	case -ENOBUFS:
 		break;
 	case -EAGAIN:
-		status = xs_nospace(task);
+		status = xs_nospace(req, task);
 		break;
 	default:
 		dprintk("RPC:       sendmsg returned unrecognized error %d\n",
@@ -585,6 +585,7 @@ static int xs_local_send_request(struct rpc_task *task)
 
 /**
  * xs_udp_send_request - write an RPC request to a UDP socket
+ * @req: pointer to RPC request
  * @task: address of RPC task that manages the state of an RPC request
  *
  * Return values:
@@ -594,9 +595,8 @@ static int xs_local_send_request(struct rpc_task *task)
  * ENOTCONN:	Caller needs to invoke connect logic then call again
  *    other:	Some other error occurred, the request was not sent
  */
-static int xs_udp_send_request(struct rpc_task *task)
+static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
 	struct xdr_buf *xdr = &req->rq_snd_buf;
@@ -638,7 +638,7 @@ static int xs_udp_send_request(struct rpc_task *task)
 		/* Should we call xs_close() here? */
 		break;
 	case -EAGAIN:
-		status = xs_nospace(task);
+		status = xs_nospace(req, task);
 		break;
 	case -ENETUNREACH:
 	case -ENOBUFS:
@@ -658,6 +658,7 @@ static int xs_udp_send_request(struct rpc_task *task)
 
 /**
  * xs_tcp_send_request - write an RPC request to a TCP socket
+ * @req: pointer to RPC request
  * @task: address of RPC task that manages the state of an RPC request
  *
  * Return values:
@@ -670,9 +671,8 @@ static int xs_udp_send_request(struct rpc_task *task)
  * XXX: In the case of soft timeouts, should we eventually give up
  *	if sendmsg is not able to make progress?
  */
-static int xs_tcp_send_request(struct rpc_task *task)
+static int xs_tcp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
 	struct xdr_buf *xdr = &req->rq_snd_buf;
@@ -697,7 +697,7 @@ static int xs_tcp_send_request(struct rpc_task *task)
 	 * completes while the socket holds a reference to the pages,
 	 * then we may end up resending corrupted data.
 	 */
-	if (task->tk_flags & RPC_TASK_SENT)
+	if (req->rq_task->tk_flags & RPC_TASK_SENT)
 		zerocopy = false;
 
 	if (test_bit(XPRT_SOCK_UPD_TIMEOUT, &transport->sock_state))
@@ -761,7 +761,7 @@ static int xs_tcp_send_request(struct rpc_task *task)
 		/* Should we call xs_close() here? */
 		break;
 	case -EAGAIN:
-		status = xs_nospace(task);
+		status = xs_nospace(req, task);
 		break;
 	case -ECONNRESET:
 	case -ECONNREFUSED:
@@ -2706,9 +2706,8 @@ static int bc_sendto(struct rpc_rqst *req)
 /*
  * The send routine. Borrows from svc_send
  */
-static int bc_send_request(struct rpc_task *task)
+static int bc_send_request(struct rpc_rqst *req, struct rpc_task *task)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
 	struct svc_xprt	*xprt;
 	int len;
 
-- 
2.17.1

[PATCH v2 21/34] SUNRPC: Don't reset the request 'bytes_sent' counter when releasing XPRT_LOCK

[Trond Myklebust]

If the request is still on the queue, this will be incorrect behaviour.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c |  4 ----
 net/sunrpc/xprt.c | 14 --------------
 2 files changed, 18 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 497a30762a6d..032b7042adb6 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2132,15 +2132,11 @@ static void
 call_status(struct rpc_task *task)
 {
 	struct rpc_clnt	*clnt = task->tk_client;
-	struct rpc_rqst	*req = task->tk_rqstp;
 	int		status;
 
 	if (!task->tk_msg.rpc_proc->p_proc)
 		trace_xprt_ping(task->tk_xprt, task->tk_status);
 
-	if (req->rq_reply_bytes_recvd > 0 && !req->rq_bytes_sent)
-		task->tk_status = req->rq_reply_bytes_recvd;
-
 	dprint_status(task);
 
 	status = task->tk_status;
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 7c80f93562e5..305aa9570873 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -332,15 +332,6 @@ static void __xprt_lock_write_next_cong(struct rpc_xprt *xprt)
 	xprt_clear_locked(xprt);
 }
 
-static void xprt_task_clear_bytes_sent(struct rpc_task *task)
-{
-	if (task != NULL) {
-		struct rpc_rqst *req = task->tk_rqstp;
-		if (req != NULL)
-			req->rq_bytes_sent = 0;
-	}
-}
-
 /**
  * xprt_release_xprt - allow other requests to use a transport
  * @xprt: transport with other tasks potentially waiting
@@ -351,7 +342,6 @@ static void xprt_task_clear_bytes_sent(struct rpc_task *task)
 void xprt_release_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 {
 	if (xprt->snd_task == task) {
-		xprt_task_clear_bytes_sent(task);
 		xprt_clear_locked(xprt);
 		__xprt_lock_write_next(xprt);
 	}
@@ -369,7 +359,6 @@ EXPORT_SYMBOL_GPL(xprt_release_xprt);
 void xprt_release_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 {
 	if (xprt->snd_task == task) {
-		xprt_task_clear_bytes_sent(task);
 		xprt_clear_locked(xprt);
 		__xprt_lock_write_next_cong(xprt);
 	}
@@ -742,7 +731,6 @@ bool xprt_lock_connect(struct rpc_xprt *xprt,
 		goto out;
 	if (xprt->snd_task != task)
 		goto out;
-	xprt_task_clear_bytes_sent(task);
 	xprt->snd_task = cookie;
 	ret = true;
 out:
@@ -788,7 +776,6 @@ void xprt_connect(struct rpc_task *task)
 		xprt->ops->close(xprt);
 
 	if (!xprt_connected(xprt)) {
-		task->tk_rqstp->rq_bytes_sent = 0;
 		task->tk_timeout = task->tk_rqstp->rq_timeout;
 		task->tk_rqstp->rq_connect_cookie = xprt->connect_cookie;
 		rpc_sleep_on(&xprt->pending, task, xprt_connect_status);
@@ -1085,7 +1072,6 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 static void
 xprt_request_dequeue_transmit_locked(struct rpc_task *task)
 {
-	xprt_task_clear_bytes_sent(task);
 	clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
 	list_del_init(&task->tk_rqstp->rq_xmit);
 }
-- 
2.17.1

[PATCH v2 22/34] SUNRPC: Simplify xprt_prepare_transmit()

[Trond Myklebust]

Remove the checks for whether or not we need to transmit, and whether
or not a reply has been received. Those are already handled in
call_transmit() itself.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprt.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 305aa9570873..f1301d391399 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1102,27 +1102,12 @@ bool xprt_prepare_transmit(struct rpc_task *task)
 {
 	struct rpc_rqst	*req = task->tk_rqstp;
 	struct rpc_xprt	*xprt = req->rq_xprt;
-	bool ret = false;
 
 	dprintk("RPC: %5u xprt_prepare_transmit\n", task->tk_pid);
 
-	spin_lock_bh(&xprt->transport_lock);
-	if (!req->rq_bytes_sent) {
-		if (req->rq_reply_bytes_recvd) {
-			task->tk_status = req->rq_reply_bytes_recvd;
-			goto out_unlock;
-		}
-		if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
-			goto out_unlock;
-	}
-	if (!xprt->ops->reserve_xprt(xprt, task)) {
-		task->tk_status = -EAGAIN;
-		goto out_unlock;
-	}
-	ret = true;
-out_unlock:
-	spin_unlock_bh(&xprt->transport_lock);
-	return ret;
+	if (!xprt_lock_write(xprt, task))
+		return false;
+	return true;
 }
 
 void xprt_end_transmit(struct rpc_task *task)
-- 
2.17.1

[PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c |  6 ------
 net/sunrpc/xprt.c | 13 ++++++-------
 2 files changed, 6 insertions(+), 13 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 032b7042adb6..21713bed812a 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1967,8 +1967,6 @@ call_connect_status(struct rpc_task *task)
 static void
 call_transmit(struct rpc_task *task)
 {
-	int is_retrans = RPC_WAS_SENT(task);
-
 	dprint_status(task);
 
 	task->tk_action = call_transmit_status;
@@ -1979,10 +1977,6 @@ call_transmit(struct rpc_task *task)
 	if (!xprt_prepare_transmit(task))
 		return;
 	xprt_transmit(task);
-	if (task->tk_status < 0)
-		return;
-	if (is_retrans)
-		task->tk_client->cl_stats->rpcretrans++;
 }
 
 /*
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index f1301d391399..e2f5b4668a66 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -191,8 +191,6 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 		goto out_sleep;
 	}
 	xprt->snd_task = task;
-	if (req != NULL)
-		req->rq_ntrans++;
 
 	return 1;
 
@@ -247,7 +245,6 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 	}
 	if (__xprt_get_cong(xprt, task)) {
 		xprt->snd_task = task;
-		req->rq_ntrans++;
 		return 1;
 	}
 	xprt_clear_locked(xprt);
@@ -281,12 +278,8 @@ static inline int xprt_lock_write(struct rpc_xprt *xprt, struct rpc_task *task)
 static bool __xprt_lock_write_func(struct rpc_task *task, void *data)
 {
 	struct rpc_xprt *xprt = data;
-	struct rpc_rqst *req;
 
-	req = task->tk_rqstp;
 	xprt->snd_task = task;
-	if (req)
-		req->rq_ntrans++;
 	return true;
 }
 
@@ -1126,6 +1119,7 @@ void xprt_transmit(struct rpc_task *task)
 	struct rpc_rqst	*req = task->tk_rqstp;
 	struct rpc_xprt	*xprt = req->rq_xprt;
 	unsigned int connect_cookie;
+	int is_retrans = RPC_WAS_SENT(task);
 	int status;
 
 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, req->rq_slen);
@@ -1140,6 +1134,8 @@ void xprt_transmit(struct rpc_task *task)
 		}
 	}
 
+	req->rq_ntrans++;
+
 	connect_cookie = xprt->connect_cookie;
 	status = xprt->ops->send_request(req, task);
 	trace_xprt_transmit(xprt, req->rq_xid, status);
@@ -1148,6 +1144,9 @@ void xprt_transmit(struct rpc_task *task)
 		return;
 	}
 
+	if (is_retrans)
+		task->tk_client->cl_stats->rpcretrans++;
+
 	xprt_inject_disconnect(xprt);
 
 	dprintk("RPC: %5u xmit complete\n", task->tk_pid);
-- 
2.17.1

[PATCH v2 24/34] SUNRPC: Fix up the back channel transmit

[Trond Myklebust]

Fix up the back channel code to recognise that it has already been
transmitted, so does not need to be called again.
Also ensure that we set req->rq_task.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 21713bed812a..00384bde593e 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1150,6 +1150,7 @@ struct rpc_task *rpc_run_bc_task(struct rpc_rqst *req)
 	 */
 	task = rpc_new_task(&task_setup_data);
 	task->tk_rqstp = req;
+	req->rq_task = task;
 
 	/*
 	 * Set up the xdr_buf length.
@@ -2054,6 +2055,9 @@ call_bc_transmit(struct rpc_task *task)
 	struct rpc_rqst *req = task->tk_rqstp;
 
 	xprt_request_enqueue_transmit(task);
+	if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+		goto out_wakeup;
+
 	if (!xprt_prepare_transmit(task))
 		goto out_retry;
 
@@ -2108,6 +2112,7 @@ call_bc_transmit(struct rpc_task *task)
 			"error: %d\n", task->tk_status);
 		break;
 	}
+out_wakeup:
 	rpc_wake_up_queued_task(&req->rq_xprt->pending, task);
 out_done:
 	task->tk_action = rpc_exit_task;
-- 
2.17.1

[PATCH v2 25/34] SUNRPC: Support for congestion control when queuing is enabled

[Trond Myklebust]

Both RDMA and UDP transports require the request to get a "congestion control"
credit before they can be transmitted. Right now, this is done when
the request locks the socket. We'd like it to happen when a request attempts
to be transmitted for the first time.
In order to support retransmission of requests that already hold such
credits, we also want to ensure that they get queued first, so that we
don't deadlock with requests that have yet to obtain a credit.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h       |  1 +
 net/sunrpc/clnt.c                 |  5 +++
 net/sunrpc/xprt.c                 | 70 ++++++++++++++++++-------------
 net/sunrpc/xprtrdma/backchannel.c |  3 ++
 net/sunrpc/xprtrdma/transport.c   |  3 ++
 net/sunrpc/xprtsock.c             |  4 ++
 6 files changed, 57 insertions(+), 29 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 6d91acfe0644..b23c757bebfc 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -395,6 +395,7 @@ void			xprt_complete_rqst(struct rpc_task *task, int copied);
 void			xprt_pin_rqst(struct rpc_rqst *req);
 void			xprt_unpin_rqst(struct rpc_rqst *req);
 void			xprt_release_rqst_cong(struct rpc_task *task);
+bool			xprt_request_get_cong(struct rpc_xprt *xprt, struct rpc_rqst *req);
 void			xprt_disconnect_done(struct rpc_xprt *xprt);
 void			xprt_force_disconnect(struct rpc_xprt *xprt);
 void			xprt_conditional_disconnect(struct rpc_xprt *xprt, unsigned int cookie);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 00384bde593e..52494baca7bc 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2003,6 +2003,11 @@ call_transmit_status(struct rpc_task *task)
 		dprint_status(task);
 		xprt_end_transmit(task);
 		break;
+	case -EBADSLT:
+		xprt_end_transmit(task);
+		task->tk_action = call_transmit;
+		task->tk_status = 0;
+		break;
 	case -EBADMSG:
 		xprt_end_transmit(task);
 		task->tk_action = call_encode;
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index e2f5b4668a66..a7a93d61567f 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -68,8 +68,6 @@
 static void	 xprt_init(struct rpc_xprt *xprt, struct net *net);
 static __be32	xprt_alloc_xid(struct rpc_xprt *xprt);
 static void	xprt_connect_status(struct rpc_task *task);
-static int      __xprt_get_cong(struct rpc_xprt *, struct rpc_task *);
-static void     __xprt_put_cong(struct rpc_xprt *, struct rpc_rqst *);
 static void	 xprt_destroy(struct rpc_xprt *xprt);
 
 static DEFINE_SPINLOCK(xprt_list_lock);
@@ -228,6 +226,7 @@ static void xprt_clear_locked(struct rpc_xprt *xprt)
  * Same as xprt_reserve_xprt, but Van Jacobson congestion control is
  * integrated into the decision of whether a request is allowed to be
  * woken up and given access to the transport.
+ * Note that the lock is only granted if we know there are free slots.
  */
 int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 {
@@ -243,14 +242,12 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 		xprt->snd_task = task;
 		return 1;
 	}
-	if (__xprt_get_cong(xprt, task)) {
+	if (!RPCXPRT_CONGESTED(xprt)) {
 		xprt->snd_task = task;
 		return 1;
 	}
 	xprt_clear_locked(xprt);
 out_sleep:
-	if (req)
-		__xprt_put_cong(xprt, req);
 	dprintk("RPC: %5u failed to lock transport %p\n", task->tk_pid, xprt);
 	task->tk_timeout = 0;
 	task->tk_status = -EAGAIN;
@@ -294,24 +291,6 @@ static void __xprt_lock_write_next(struct rpc_xprt *xprt)
 	xprt_clear_locked(xprt);
 }
 
-static bool __xprt_lock_write_cong_func(struct rpc_task *task, void *data)
-{
-	struct rpc_xprt *xprt = data;
-	struct rpc_rqst *req;
-
-	req = task->tk_rqstp;
-	if (req == NULL) {
-		xprt->snd_task = task;
-		return true;
-	}
-	if (__xprt_get_cong(xprt, task)) {
-		xprt->snd_task = task;
-		req->rq_ntrans++;
-		return true;
-	}
-	return false;
-}
-
 static void __xprt_lock_write_next_cong(struct rpc_xprt *xprt)
 {
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state))
@@ -319,7 +298,7 @@ static void __xprt_lock_write_next_cong(struct rpc_xprt *xprt)
 	if (RPCXPRT_CONGESTED(xprt))
 		goto out_unlock;
 	if (rpc_wake_up_first_on_wq(xprtiod_workqueue, &xprt->sending,
-				__xprt_lock_write_cong_func, xprt))
+				__xprt_lock_write_func, xprt))
 		return;
 out_unlock:
 	xprt_clear_locked(xprt);
@@ -370,14 +349,12 @@ static inline void xprt_release_write(struct rpc_xprt *xprt, struct rpc_task *ta
  * overflowed. Put the task to sleep if this is the case.
  */
 static int
-__xprt_get_cong(struct rpc_xprt *xprt, struct rpc_task *task)
+__xprt_get_cong(struct rpc_xprt *xprt, struct rpc_rqst *req)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
-
 	if (req->rq_cong)
 		return 1;
 	dprintk("RPC: %5u xprt_cwnd_limited cong = %lu cwnd = %lu\n",
-			task->tk_pid, xprt->cong, xprt->cwnd);
+			req->rq_task->tk_pid, xprt->cong, xprt->cwnd);
 	if (RPCXPRT_CONGESTED(xprt))
 		return 0;
 	req->rq_cong = 1;
@@ -399,6 +376,25 @@ __xprt_put_cong(struct rpc_xprt *xprt, struct rpc_rqst *req)
 	__xprt_lock_write_next_cong(xprt);
 }
 
+/**
+ * xprt_request_get_cong - Request congestion control credits
+ * @xprt: pointer to transport
+ * @req: pointer to RPC request
+ *
+ * Useful for transports that require congestion control.
+ */
+bool
+xprt_request_get_cong(struct rpc_xprt *xprt, struct rpc_rqst *req)
+{
+	bool ret = false;
+
+	spin_lock_bh(&xprt->transport_lock);
+	ret = __xprt_get_cong(xprt, req) != 0;
+	spin_unlock_bh(&xprt->transport_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(xprt_request_get_cong);
+
 /**
  * xprt_release_rqst_cong - housekeeping when request is complete
  * @task: RPC request that recently completed
@@ -1050,8 +1046,24 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 
 	spin_lock(&xprt->queue_lock);
 	if (list_empty(&req->rq_xmit) && xprt_request_need_transmit(task) &&
-	    test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+	    test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate)) {
+		/*
+		 * Requests that carry congestion control credits are added
+		 * to the head of the list to avoid starvation issues.
+		 */
+		if (req->rq_cong) {
+			struct rpc_rqst *pos;
+			list_for_each_entry(pos, &xprt->xmit_queue, rq_xmit) {
+				if (pos->rq_cong)
+					continue;
+				/* Note: req is added _before_ pos */
+				list_add_tail(&req->rq_xmit, &pos->rq_xmit);
+				goto out;
+			}
+		}
 		list_add_tail(&req->rq_xmit, &xprt->xmit_queue);
+	}
+out:
 	spin_unlock(&xprt->queue_lock);
 }
 
diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c
index fc01fdabbbce..14fc4596075e 100644
--- a/net/sunrpc/xprtrdma/backchannel.c
+++ b/net/sunrpc/xprtrdma/backchannel.c
@@ -202,6 +202,9 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst)
 	if (!xprt_connected(rqst->rq_xprt))
 		goto drop_connection;
 
+	if (!xprt_request_get_cong(rqst->rq_xprt, rqst))
+		return -EBADSLT;
+
 	rc = rpcrdma_bc_marshal_reply(rqst);
 	if (rc < 0)
 		goto failed_marshal;
diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
index fa684bf4d090..9ff322e53f37 100644
--- a/net/sunrpc/xprtrdma/transport.c
+++ b/net/sunrpc/xprtrdma/transport.c
@@ -721,6 +721,9 @@ xprt_rdma_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
 	if (!xprt_connected(xprt))
 		goto drop_connection;
 
+	if (!xprt_request_get_cong(xprt, rqst))
+		return -EBADSLT;
+
 	rc = rpcrdma_marshal_req(r_xprt, rqst);
 	if (rc < 0)
 		goto failed_marshal;
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index b8143eded4af..8831e84a058a 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -609,6 +609,10 @@ static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 
 	if (!xprt_bound(xprt))
 		return -ENOTCONN;
+
+	if (!xprt_request_get_cong(xprt, req))
+		return -EBADSLT;
+
 	req->rq_xtime = ktime_get();
 	status = xs_sendpages(transport->sock, xs_addr(xprt), xprt->addrlen,
 			      xdr, 0, true, &sent);
-- 
2.17.1

[PATCH v2 26/34] SUNRPC: Improve latency for interactive tasks

[Trond Myklebust]

One of the intentions with the priority queues was to ensure that no
single process can hog the transport. The field task->tk_owner therefore
identifies the RPC call's origin, and is intended to allow the RPC layer
to organise queues for fairness.
This commit therefore modifies the transmit queue to group requests
by task->tk_owner, and ensures that we round robin among those groups.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h       |  1 +
 net/sunrpc/backchannel_rqst.c     |  1 +
 net/sunrpc/xprt.c                 | 23 +++++++++++++++++++++--
 net/sunrpc/xprtrdma/backchannel.c |  1 +
 4 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index b23c757bebfc..5dd64bdb28cd 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -89,6 +89,7 @@ struct rpc_rqst {
 	};
 
 	struct list_head	rq_xmit;	/* Send queue */
+	struct list_head	rq_xmit2;	/* Send queue */
 
 	void			*rq_buffer;	/* Call XDR encode buffer */
 	size_t			rq_callsize;
diff --git a/net/sunrpc/backchannel_rqst.c b/net/sunrpc/backchannel_rqst.c
index 39b394b7dae3..e43dbab2e3f0 100644
--- a/net/sunrpc/backchannel_rqst.c
+++ b/net/sunrpc/backchannel_rqst.c
@@ -93,6 +93,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt *xprt, gfp_t gfp_flags)
 	req->rq_xprt = xprt;
 	INIT_LIST_HEAD(&req->rq_recv);
 	INIT_LIST_HEAD(&req->rq_xmit);
+	INIT_LIST_HEAD(&req->rq_xmit2);
 	INIT_LIST_HEAD(&req->rq_bc_list);
 
 	/* Preallocate one XDR receive buffer */
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index a7a93d61567f..8a4c5260eecd 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1043,6 +1043,7 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = req->rq_xprt;
+	struct rpc_rqst *pos;
 
 	spin_lock(&xprt->queue_lock);
 	if (list_empty(&req->rq_xmit) && xprt_request_need_transmit(task) &&
@@ -1052,7 +1053,6 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 		 * to the head of the list to avoid starvation issues.
 		 */
 		if (req->rq_cong) {
-			struct rpc_rqst *pos;
 			list_for_each_entry(pos, &xprt->xmit_queue, rq_xmit) {
 				if (pos->rq_cong)
 					continue;
@@ -1060,6 +1060,13 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 				list_add_tail(&req->rq_xmit, &pos->rq_xmit);
 				goto out;
 			}
+		} else {
+			list_for_each_entry(pos, &xprt->xmit_queue, rq_xmit) {
+				if (pos->rq_task->tk_owner != task->tk_owner)
+					continue;
+				list_add_tail(&req->rq_xmit2, &pos->rq_xmit2);
+				goto out;
+			}
 		}
 		list_add_tail(&req->rq_xmit, &xprt->xmit_queue);
 	}
@@ -1077,8 +1084,19 @@ xprt_request_enqueue_transmit(struct rpc_task *task)
 static void
 xprt_request_dequeue_transmit_locked(struct rpc_task *task)
 {
+	struct rpc_rqst *req = task->tk_rqstp;
+
 	clear_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
-	list_del_init(&task->tk_rqstp->rq_xmit);
+	if (!list_empty(&req->rq_xmit)) {
+		list_del_init(&req->rq_xmit);
+		if (!list_empty(&req->rq_xmit2)) {
+			struct rpc_rqst *next = list_first_entry(&req->rq_xmit2,
+					struct rpc_rqst, rq_xmit2);
+			list_del_init(&req->rq_xmit2);
+			list_add_tail(&next->rq_xmit, &next->rq_xprt->xmit_queue);
+		}
+	} else if (!list_empty(&req->rq_xmit2))
+		list_del_init(&req->rq_xmit2);
 }
 
 /**
@@ -1376,6 +1394,7 @@ xprt_request_init(struct rpc_task *task)
 
 	INIT_LIST_HEAD(&req->rq_recv);
 	INIT_LIST_HEAD(&req->rq_xmit);
+	INIT_LIST_HEAD(&req->rq_xmit2);
 	req->rq_timeout = task->tk_client->cl_timeout->to_initval;
 	req->rq_task	= task;
 	req->rq_xprt    = xprt;
diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c
index 14fc4596075e..92fffe765731 100644
--- a/net/sunrpc/xprtrdma/backchannel.c
+++ b/net/sunrpc/xprtrdma/backchannel.c
@@ -53,6 +53,7 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt,
 		rqst->rq_xprt = xprt;
 		INIT_LIST_HEAD(&rqst->rq_recv);
 		INIT_LIST_HEAD(&rqst->rq_xmit);
+		INIT_LIST_HEAD(&rqst->rq_xmit2);
 		INIT_LIST_HEAD(&rqst->rq_bc_list);
 		__set_bit(RPC_BC_PA_IN_USE, &rqst->rq_bc_pa_state);
 		spin_lock_bh(&xprt->bc_pa_lock);
-- 
2.17.1

[PATCH v2 27/34] SUNRPC: Allow calls to xprt_transmit() to drain the entire transmit queue

[Trond Myklebust]

Rather than forcing each and every RPC task to grab the socket write
lock in order to send itself, we allow whichever task is holding the
write lock to attempt to drain the entire transmit queue.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprt.c | 79 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 66 insertions(+), 13 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 8a4c5260eecd..ba9af25d14de 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1139,15 +1139,20 @@ void xprt_end_transmit(struct rpc_task *task)
 }
 
 /**
- * xprt_transmit - send an RPC request on a transport
- * @task: controlling RPC task
+ * xprt_request_transmit - send an RPC request on a transport
+ * @req: pointer to request to transmit
+ * @snd_task: RPC task that owns the transport lock
  *
- * We have to copy the iovec because sendmsg fiddles with its contents.
+ * This performs the transmission of a single request.
+ * Note that if the request is not the same as snd_task, then it
+ * does need to be pinned.
+ * Returns '0' on success.
  */
-void xprt_transmit(struct rpc_task *task)
+static int
+xprt_request_transmit(struct rpc_rqst *req, struct rpc_task *snd_task)
 {
-	struct rpc_rqst	*req = task->tk_rqstp;
-	struct rpc_xprt	*xprt = req->rq_xprt;
+	struct rpc_xprt *xprt = req->rq_xprt;
+	struct rpc_task *task = req->rq_task;
 	unsigned int connect_cookie;
 	int is_retrans = RPC_WAS_SENT(task);
 	int status;
@@ -1155,11 +1160,13 @@ void xprt_transmit(struct rpc_task *task)
 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, req->rq_slen);
 
 	if (!req->rq_bytes_sent) {
-		if (xprt_request_data_received(task))
+		if (xprt_request_data_received(task)) {
+			status = 0;
 			goto out_dequeue;
+		}
 		/* Verify that our message lies in the RPCSEC_GSS window */
 		if (rpcauth_xmit_need_reencode(task)) {
-			task->tk_status = -EBADMSG;
+			status = -EBADMSG;
 			goto out_dequeue;
 		}
 	}
@@ -1167,12 +1174,10 @@ void xprt_transmit(struct rpc_task *task)
 	req->rq_ntrans++;
 
 	connect_cookie = xprt->connect_cookie;
-	status = xprt->ops->send_request(req, task);
+	status = xprt->ops->send_request(req, snd_task);
 	trace_xprt_transmit(xprt, req->rq_xid, status);
-	if (status != 0) {
-		task->tk_status = status;
-		return;
-	}
+	if (status != 0)
+		return status;
 
 	if (is_retrans)
 		task->tk_client->cl_stats->rpcretrans++;
@@ -1193,6 +1198,54 @@ void xprt_transmit(struct rpc_task *task)
 	req->rq_connect_cookie = connect_cookie;
 out_dequeue:
 	xprt_request_dequeue_transmit(task);
+	rpc_wake_up_queued_task_set_status(&xprt->sending, task, status);
+	return status;
+}
+
+/**
+ * xprt_transmit - send an RPC request on a transport
+ * @task: controlling RPC task
+ *
+ * Attempts to drain the transmit queue. On exit, either the transport
+ * signalled an error that needs to be handled before transmission can
+ * resume, or @task finished transmitting, and detected that it already
+ * received a reply.
+ */
+void
+xprt_transmit(struct rpc_task *task)
+{
+	struct rpc_rqst *next, *req = task->tk_rqstp;
+	struct rpc_xprt	*xprt = req->rq_xprt;
+	LIST_HEAD(head);
+	int status;
+
+	task->tk_status = -EAGAIN;
+	spin_lock(&xprt->queue_lock);
+	/* Avoid livelock by moving the xmit_queue contents to a private list */
+	list_splice_init(&xprt->xmit_queue, &head);
+	while (!list_empty(&head)) {
+		next = list_first_entry(&head, struct rpc_rqst, rq_xmit);
+		xprt_pin_rqst(next);
+		spin_unlock(&xprt->queue_lock);
+		status = xprt_request_transmit(next, task);
+		if (status == -EBADMSG && next != req)
+			status = 0;
+		cond_resched();
+		spin_lock(&xprt->queue_lock);
+		xprt_unpin_rqst(next);
+		if (status == 0) {
+			if (!xprt_request_data_received(task) ||
+			    test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+				continue;
+		} else if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+			rpc_wake_up_queued_task(&xprt->pending, task);
+		else
+			task->tk_status = status;
+		/* On early exit, splice back the list contents */
+		list_splice(&head, &xprt->xmit_queue);
+		break;
+	}
+	spin_unlock(&xprt->queue_lock);
 }
 
 static void xprt_add_backlog(struct rpc_xprt *xprt, struct rpc_task *task)
-- 
2.17.1

[PATCH v2 28/34] SUNRPC: Queue the request for transmission immediately after encoding

[Trond Myklebust]

Move up the call to xprt_request_enqueue_transmit() to call_encode()
so that the queue order reflects the order in which slots were
allocated.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/clnt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 52494baca7bc..4f8803413499 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1785,6 +1785,7 @@ call_encode(struct rpc_task *task)
 	if (rpc_reply_expected(task))
 		xprt_request_enqueue_receive(task);
 	set_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate);
+	xprt_request_enqueue_transmit(task);
 out:
 	task->tk_action = call_bind;
 }
@@ -1971,7 +1972,6 @@ call_transmit(struct rpc_task *task)
 	dprint_status(task);
 
 	task->tk_action = call_transmit_status;
-	xprt_request_enqueue_transmit(task);
 	if (!test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
 		return;
 
-- 
2.17.1

[PATCH v2 29/34] SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue

[Trond Myklebust]

We no longer need priority semantics on the xprt->sending queue, because
the order in which tasks are sent is now dictated by their position in
the send queue.
Note that the backlog queue remains a priority queue, meaning that
slot resources are still managed in order of task priority.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprt.c | 29 ++++++-----------------------
 1 file changed, 6 insertions(+), 23 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index ba9af25d14de..ebe5235c484b 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -181,7 +181,6 @@ EXPORT_SYMBOL_GPL(xprt_load_transport);
 int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
-	int priority;
 
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state)) {
 		if (task == xprt->snd_task)
@@ -197,13 +196,7 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 			task->tk_pid, xprt);
 	task->tk_timeout = 0;
 	task->tk_status = -EAGAIN;
-	if (req == NULL)
-		priority = RPC_PRIORITY_LOW;
-	else if (!req->rq_ntrans)
-		priority = RPC_PRIORITY_NORMAL;
-	else
-		priority = RPC_PRIORITY_HIGH;
-	rpc_sleep_on_priority(&xprt->sending, task, NULL, priority);
+	rpc_sleep_on(&xprt->sending, task, NULL);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(xprt_reserve_xprt);
@@ -231,7 +224,6 @@ static void xprt_clear_locked(struct rpc_xprt *xprt)
 int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 {
 	struct rpc_rqst *req = task->tk_rqstp;
-	int priority;
 
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state)) {
 		if (task == xprt->snd_task)
@@ -251,13 +243,7 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 	dprintk("RPC: %5u failed to lock transport %p\n", task->tk_pid, xprt);
 	task->tk_timeout = 0;
 	task->tk_status = -EAGAIN;
-	if (req == NULL)
-		priority = RPC_PRIORITY_LOW;
-	else if (!req->rq_ntrans)
-		priority = RPC_PRIORITY_NORMAL;
-	else
-		priority = RPC_PRIORITY_HIGH;
-	rpc_sleep_on_priority(&xprt->sending, task, NULL, priority);
+	rpc_sleep_on(&xprt->sending, task, NULL);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(xprt_reserve_xprt_cong);
@@ -285,8 +271,7 @@ static void __xprt_lock_write_next(struct rpc_xprt *xprt)
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state))
 		return;
 
-	if (rpc_wake_up_first_on_wq(xprtiod_workqueue, &xprt->sending,
-				__xprt_lock_write_func, xprt))
+	if (rpc_wake_up_first(&xprt->sending, __xprt_lock_write_func, xprt))
 		return;
 	xprt_clear_locked(xprt);
 }
@@ -297,8 +282,7 @@ static void __xprt_lock_write_next_cong(struct rpc_xprt *xprt)
 		return;
 	if (RPCXPRT_CONGESTED(xprt))
 		goto out_unlock;
-	if (rpc_wake_up_first_on_wq(xprtiod_workqueue, &xprt->sending,
-				__xprt_lock_write_func, xprt))
+	if (rpc_wake_up_first(&xprt->sending, __xprt_lock_write_func, xprt))
 		return;
 out_unlock:
 	xprt_clear_locked(xprt);
@@ -495,8 +479,7 @@ void xprt_write_space(struct rpc_xprt *xprt)
 	if (xprt->snd_task) {
 		dprintk("RPC:       write space: waking waiting task on "
 				"xprt %p\n", xprt);
-		rpc_wake_up_queued_task_on_wq(xprtiod_workqueue,
-				&xprt->pending, xprt->snd_task);
+		rpc_wake_up_queued_task(&xprt->pending, xprt->snd_task);
 	}
 	spin_unlock_bh(&xprt->transport_lock);
 }
@@ -1596,7 +1579,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 
 	rpc_init_wait_queue(&xprt->binding, "xprt_binding");
 	rpc_init_wait_queue(&xprt->pending, "xprt_pending");
-	rpc_init_priority_wait_queue(&xprt->sending, "xprt_sending");
+	rpc_init_wait_queue(&xprt->sending, "xprt_sending");
 	rpc_init_priority_wait_queue(&xprt->backlog, "xprt_backlog");
 
 	xprt_init_xid(xprt);
-- 
2.17.1

[PATCH v2 30/34] SUNRPC: Allow soft RPC calls to time out when waiting for the XPRT_LOCK

[Trond Myklebust]

This no longer causes them to lose their place in the transmission queue.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/xprt.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index ebe5235c484b..bd818b97cb09 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -194,7 +194,7 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 out_sleep:
 	dprintk("RPC: %5u failed to lock transport %p\n",
 			task->tk_pid, xprt);
-	task->tk_timeout = 0;
+	task->tk_timeout = RPC_IS_SOFT(task) ? req->rq_timeout : 0;
 	task->tk_status = -EAGAIN;
 	rpc_sleep_on(&xprt->sending, task, NULL);
 	return 0;
@@ -241,7 +241,7 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 	xprt_clear_locked(xprt);
 out_sleep:
 	dprintk("RPC: %5u failed to lock transport %p\n", task->tk_pid, xprt);
-	task->tk_timeout = 0;
+	task->tk_timeout = RPC_IS_SOFT(task) ? req->rq_timeout : 0;
 	task->tk_status = -EAGAIN;
 	rpc_sleep_on(&xprt->sending, task, NULL);
 	return 0;
-- 
2.17.1

[PATCH v2 31/34] SUNRPC: Turn off throttling of RPC slots for TCP sockets

[Trond Myklebust]

The theory was that we would need to grab the socket lock anyway, so we
might as well use it to gate the allocation of RPC slots for a TCP
socket.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h |  1 -
 net/sunrpc/xprt.c           | 14 --------------
 net/sunrpc/xprtsock.c       |  2 +-
 3 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 5dd64bdb28cd..d623bebab4f9 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -343,7 +343,6 @@ int			xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task);
 void			xprt_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task);
 void			xprt_free_slot(struct rpc_xprt *xprt,
 				       struct rpc_rqst *req);
-void			xprt_lock_and_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task);
 bool			xprt_prepare_transmit(struct rpc_task *task);
 void			xprt_request_enqueue_transmit(struct rpc_task *task);
 void			xprt_request_enqueue_receive(struct rpc_task *task);
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index bd818b97cb09..4068d0ea3a21 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1325,20 +1325,6 @@ void xprt_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task)
 }
 EXPORT_SYMBOL_GPL(xprt_alloc_slot);
 
-void xprt_lock_and_alloc_slot(struct rpc_xprt *xprt, struct rpc_task *task)
-{
-	/* Note: grabbing the xprt_lock_write() ensures that we throttle
-	 * new slot allocation if the transport is congested (i.e. when
-	 * reconnecting a stream transport or when out of socket write
-	 * buffer space).
-	 */
-	if (xprt_lock_write(xprt, task)) {
-		xprt_alloc_slot(xprt, task);
-		xprt_release_write(xprt, task);
-	}
-}
-EXPORT_SYMBOL_GPL(xprt_lock_and_alloc_slot);
-
 void xprt_free_slot(struct rpc_xprt *xprt, struct rpc_rqst *req)
 {
 	spin_lock(&xprt->reserve_lock);
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 8831e84a058a..f54e8110f4c6 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -2809,7 +2809,7 @@ static const struct rpc_xprt_ops xs_udp_ops = {
 static const struct rpc_xprt_ops xs_tcp_ops = {
 	.reserve_xprt		= xprt_reserve_xprt,
 	.release_xprt		= xprt_release_xprt,
-	.alloc_slot		= xprt_lock_and_alloc_slot,
+	.alloc_slot		= xprt_alloc_slot,
 	.free_slot		= xprt_free_slot,
 	.rpcbind		= rpcb_getport_async,
 	.set_port		= xs_set_port,
-- 
2.17.1

[PATCH v2 32/34] SUNRPC: Clean up transport write space handling

[Trond Myklebust]

Treat socket write space handling in the same way we now treat transport
congestion: by denying the XPRT_LOCK until the transport signals that it
has free buffer space.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/svc_xprt.h            |  1 -
 include/linux/sunrpc/xprt.h                |  5 +-
 net/sunrpc/clnt.c                          | 11 +---
 net/sunrpc/svc_xprt.c                      |  2 -
 net/sunrpc/xprt.c                          | 69 ++++++++++++++--------
 net/sunrpc/xprtrdma/rpc_rdma.c             |  2 +-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  7 +--
 net/sunrpc/xprtsock.c                      | 33 ++++-------
 8 files changed, 62 insertions(+), 68 deletions(-)

diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h
index c3d72066d4b1..6b7a86c4d6e6 100644
--- a/include/linux/sunrpc/svc_xprt.h
+++ b/include/linux/sunrpc/svc_xprt.h
@@ -84,7 +84,6 @@ struct svc_xprt {
 	struct sockaddr_storage	xpt_remote;	/* remote peer's address */
 	size_t			xpt_remotelen;	/* length of address */
 	char			xpt_remotebuf[INET6_ADDRSTRLEN + 10];
-	struct rpc_wait_queue	xpt_bc_pending;	/* backchannel wait queue */
 	struct list_head	xpt_users;	/* callbacks on free */
 
 	struct net		*xpt_net;
diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index d623bebab4f9..ba4a9d83102a 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -386,8 +386,8 @@ int			xprt_load_transport(const char *);
 void			xprt_set_retrans_timeout_def(struct rpc_task *task);
 void			xprt_set_retrans_timeout_rtt(struct rpc_task *task);
 void			xprt_wake_pending_tasks(struct rpc_xprt *xprt, int status);
-void			xprt_wait_for_buffer_space(struct rpc_task *task, rpc_action action);
-void			xprt_write_space(struct rpc_xprt *xprt);
+void			xprt_wait_for_buffer_space(struct rpc_xprt *xprt);
+bool			xprt_write_space(struct rpc_xprt *xprt);
 void			xprt_adjust_cwnd(struct rpc_xprt *xprt, struct rpc_task *task, int result);
 struct rpc_rqst *	xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid);
 void			xprt_update_rtt(struct rpc_task *task);
@@ -414,6 +414,7 @@ void			xprt_unlock_connect(struct rpc_xprt *, void *);
 #define XPRT_BINDING		(5)
 #define XPRT_CLOSING		(6)
 #define XPRT_CONGESTED		(9)
+#define XPRT_WRITE_SPACE	(10)
 
 static inline void xprt_set_connected(struct rpc_xprt *xprt)
 {
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 4f8803413499..fb19c8a85e68 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1978,6 +1978,7 @@ call_transmit(struct rpc_task *task)
 	if (!xprt_prepare_transmit(task))
 		return;
 	xprt_transmit(task);
+	xprt_end_transmit(task);
 }
 
 /*
@@ -1993,7 +1994,6 @@ call_transmit_status(struct rpc_task *task)
 	 * test first.
 	 */
 	if (task->tk_status == 0) {
-		xprt_end_transmit(task);
 		xprt_request_wait_receive(task);
 		return;
 	}
@@ -2001,15 +2001,12 @@ call_transmit_status(struct rpc_task *task)
 	switch (task->tk_status) {
 	default:
 		dprint_status(task);
-		xprt_end_transmit(task);
 		break;
 	case -EBADSLT:
-		xprt_end_transmit(task);
 		task->tk_action = call_transmit;
 		task->tk_status = 0;
 		break;
 	case -EBADMSG:
-		xprt_end_transmit(task);
 		task->tk_action = call_encode;
 		break;
 		/*
@@ -2032,7 +2029,6 @@ call_transmit_status(struct rpc_task *task)
 	case -ENETUNREACH:
 	case -EPERM:
 		if (RPC_IS_SOFTCONN(task)) {
-			xprt_end_transmit(task);
 			if (!task->tk_msg.rpc_proc->p_proc)
 				trace_xprt_ping(task->tk_xprt,
 						task->tk_status);
@@ -2076,9 +2072,6 @@ call_bc_transmit(struct rpc_task *task)
 
 	xprt_transmit(task);
 
-	if (task->tk_status == -EAGAIN)
-		goto out_nospace;
-
 	xprt_end_transmit(task);
 	dprint_status(task);
 	switch (task->tk_status) {
@@ -2094,6 +2087,8 @@ call_bc_transmit(struct rpc_task *task)
 	case -ENOTCONN:
 	case -EPIPE:
 		break;
+	case -EAGAIN:
+		goto out_nospace;
 	case -ETIMEDOUT:
 		/*
 		 * Problem reaching the server.  Disconnect and let the
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index 5185efb9027b..87533fbb96cf 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -171,7 +171,6 @@ void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 	mutex_init(&xprt->xpt_mutex);
 	spin_lock_init(&xprt->xpt_lock);
 	set_bit(XPT_BUSY, &xprt->xpt_flags);
-	rpc_init_wait_queue(&xprt->xpt_bc_pending, "xpt_bc_pending");
 	xprt->xpt_net = get_net(net);
 	strcpy(xprt->xpt_remotebuf, "uninitialized");
 }
@@ -895,7 +894,6 @@ int svc_send(struct svc_rqst *rqstp)
 	else
 		len = xprt->xpt_ops->xpo_sendto(rqstp);
 	mutex_unlock(&xprt->xpt_mutex);
-	rpc_wake_up(&xprt->xpt_bc_pending);
 	trace_svc_send(rqstp, len);
 	svc_xprt_release(rqstp);
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 4068d0ea3a21..baa454b0d855 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -169,6 +169,17 @@ int xprt_load_transport(const char *transport_name)
 }
 EXPORT_SYMBOL_GPL(xprt_load_transport);
 
+static void xprt_clear_locked(struct rpc_xprt *xprt)
+{
+	xprt->snd_task = NULL;
+	if (!test_bit(XPRT_CLOSE_WAIT, &xprt->state)) {
+		smp_mb__before_atomic();
+		clear_bit(XPRT_LOCKED, &xprt->state);
+		smp_mb__after_atomic();
+	} else
+		queue_work(xprtiod_workqueue, &xprt->task_cleanup);
+}
+
 /**
  * xprt_reserve_xprt - serialize write access to transports
  * @task: task that is requesting access to the transport
@@ -187,10 +198,14 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 			return 1;
 		goto out_sleep;
 	}
+	if (test_bit(XPRT_WRITE_SPACE, &xprt->state))
+		goto out_unlock;
 	xprt->snd_task = task;
 
 	return 1;
 
+out_unlock:
+	xprt_clear_locked(xprt);
 out_sleep:
 	dprintk("RPC: %5u failed to lock transport %p\n",
 			task->tk_pid, xprt);
@@ -201,17 +216,6 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task)
 }
 EXPORT_SYMBOL_GPL(xprt_reserve_xprt);
 
-static void xprt_clear_locked(struct rpc_xprt *xprt)
-{
-	xprt->snd_task = NULL;
-	if (!test_bit(XPRT_CLOSE_WAIT, &xprt->state)) {
-		smp_mb__before_atomic();
-		clear_bit(XPRT_LOCKED, &xprt->state);
-		smp_mb__after_atomic();
-	} else
-		queue_work(xprtiod_workqueue, &xprt->task_cleanup);
-}
-
 /*
  * xprt_reserve_xprt_cong - serialize write access to transports
  * @task: task that is requesting access to the transport
@@ -234,10 +238,13 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, struct rpc_task *task)
 		xprt->snd_task = task;
 		return 1;
 	}
+	if (test_bit(XPRT_WRITE_SPACE, &xprt->state))
+		goto out_unlock;
 	if (!RPCXPRT_CONGESTED(xprt)) {
 		xprt->snd_task = task;
 		return 1;
 	}
+out_unlock:
 	xprt_clear_locked(xprt);
 out_sleep:
 	dprintk("RPC: %5u failed to lock transport %p\n", task->tk_pid, xprt);
@@ -270,9 +277,11 @@ static void __xprt_lock_write_next(struct rpc_xprt *xprt)
 {
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state))
 		return;
-
+	if (test_bit(XPRT_WRITE_SPACE, &xprt->state))
+		goto out_unlock;
 	if (rpc_wake_up_first(&xprt->sending, __xprt_lock_write_func, xprt))
 		return;
+out_unlock:
 	xprt_clear_locked(xprt);
 }
 
@@ -280,6 +289,8 @@ static void __xprt_lock_write_next_cong(struct rpc_xprt *xprt)
 {
 	if (test_and_set_bit(XPRT_LOCKED, &xprt->state))
 		return;
+	if (test_bit(XPRT_WRITE_SPACE, &xprt->state))
+		goto out_unlock;
 	if (RPCXPRT_CONGESTED(xprt))
 		goto out_unlock;
 	if (rpc_wake_up_first(&xprt->sending, __xprt_lock_write_func, xprt))
@@ -450,38 +461,43 @@ EXPORT_SYMBOL_GPL(xprt_wake_pending_tasks);
 
 /**
  * xprt_wait_for_buffer_space - wait for transport output buffer to clear
- * @task: task to be put to sleep
- * @action: function pointer to be executed after wait
+ * @xprt: transport
  *
  * Note that we only set the timer for the case of RPC_IS_SOFT(), since
  * we don't in general want to force a socket disconnection due to
  * an incomplete RPC call transmission.
  */
-void xprt_wait_for_buffer_space(struct rpc_task *task, rpc_action action)
+void xprt_wait_for_buffer_space(struct rpc_xprt *xprt)
 {
-	struct rpc_rqst *req = task->tk_rqstp;
-	struct rpc_xprt *xprt = req->rq_xprt;
-
-	task->tk_timeout = RPC_IS_SOFT(task) ? req->rq_timeout : 0;
-	rpc_sleep_on(&xprt->pending, task, action);
+	set_bit(XPRT_WRITE_SPACE, &xprt->state);
 }
 EXPORT_SYMBOL_GPL(xprt_wait_for_buffer_space);
 
+static bool
+xprt_clear_write_space_locked(struct rpc_xprt *xprt)
+{
+	if (test_and_clear_bit(XPRT_WRITE_SPACE, &xprt->state)) {
+		__xprt_lock_write_next(xprt);
+		dprintk("RPC:       write space: waking waiting task on "
+				"xprt %p\n", xprt);
+		return true;
+	}
+	return false;
+}
+
 /**
  * xprt_write_space - wake the task waiting for transport output buffer space
  * @xprt: transport with waiting tasks
  *
  * Can be called in a soft IRQ context, so xprt_write_space never sleeps.
  */
-void xprt_write_space(struct rpc_xprt *xprt)
+bool xprt_write_space(struct rpc_xprt *xprt)
 {
+	bool ret;
 	spin_lock_bh(&xprt->transport_lock);
-	if (xprt->snd_task) {
-		dprintk("RPC:       write space: waking waiting task on "
-				"xprt %p\n", xprt);
-		rpc_wake_up_queued_task(&xprt->pending, xprt->snd_task);
-	}
+	ret = xprt_clear_write_space_locked(xprt);
 	spin_unlock_bh(&xprt->transport_lock);
+	return ret;
 }
 EXPORT_SYMBOL_GPL(xprt_write_space);
 
@@ -592,6 +608,7 @@ void xprt_disconnect_done(struct rpc_xprt *xprt)
 	dprintk("RPC:       disconnected transport %p\n", xprt);
 	spin_lock_bh(&xprt->transport_lock);
 	xprt_clear_connected(xprt);
+	xprt_clear_write_space_locked(xprt);
 	xprt_wake_pending_tasks(xprt, -EAGAIN);
 	spin_unlock_bh(&xprt->transport_lock);
 }
diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index 0020dc401215..53fa95d60015 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -866,7 +866,7 @@ rpcrdma_marshal_req(struct rpcrdma_xprt *r_xprt, struct rpc_rqst *rqst)
 out_err:
 	switch (ret) {
 	case -EAGAIN:
-		xprt_wait_for_buffer_space(rqst->rq_task, NULL);
+		xprt_wait_for_buffer_space(rqst->rq_xprt);
 		break;
 	case -ENOBUFS:
 		break;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index d1618c70edb4..35a8c3aab302 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -224,12 +224,7 @@ xprt_rdma_bc_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
 	dprintk("svcrdma: sending bc call with xid: %08x\n",
 		be32_to_cpu(rqst->rq_xid));
 
-	if (!mutex_trylock(&sxprt->xpt_mutex)) {
-		rpc_sleep_on(&sxprt->xpt_bc_pending, task, NULL);
-		if (!mutex_trylock(&sxprt->xpt_mutex))
-			return -EAGAIN;
-		rpc_wake_up_queued_task(&sxprt->xpt_bc_pending, task);
-	}
+	mutex_lock(&sxprt->xpt_mutex);
 
 	ret = -ENOTCONN;
 	rdma = container_of(sxprt, struct svcxprt_rdma, sc_xprt);
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index f54e8110f4c6..ef8d0e81cbda 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -440,20 +440,12 @@ static int xs_sendpages(struct socket *sock, struct sockaddr *addr, int addrlen,
 	return err;
 }
 
-static void xs_nospace_callback(struct rpc_task *task)
-{
-	struct sock_xprt *transport = container_of(task->tk_rqstp->rq_xprt, struct sock_xprt, xprt);
-
-	transport->inet->sk_write_pending--;
-}
-
 /**
- * xs_nospace - place task on wait queue if transmit was incomplete
+ * xs_nospace - handle transmit was incomplete
  * @req: pointer to RPC request
- * @task: task to put to sleep
  *
  */
-static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
+static int xs_nospace(struct rpc_rqst *req)
 {
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
@@ -461,7 +453,8 @@ static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
 	int ret = -EAGAIN;
 
 	dprintk("RPC: %5u xmit incomplete (%u left of %u)\n",
-			task->tk_pid, req->rq_slen - transport->xmit.offset,
+			req->rq_task->tk_pid,
+			req->rq_slen - transport->xmit.offset,
 			req->rq_slen);
 
 	/* Protect against races with write_space */
@@ -471,7 +464,7 @@ static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
 	if (xprt_connected(xprt)) {
 		/* wait for more buffer space */
 		sk->sk_write_pending++;
-		xprt_wait_for_buffer_space(task, xs_nospace_callback);
+		xprt_wait_for_buffer_space(xprt);
 	} else
 		ret = -ENOTCONN;
 
@@ -569,7 +562,7 @@ static int xs_local_send_request(struct rpc_rqst *req, struct rpc_task *task)
 	case -ENOBUFS:
 		break;
 	case -EAGAIN:
-		status = xs_nospace(req, task);
+		status = xs_nospace(req);
 		break;
 	default:
 		dprintk("RPC:       sendmsg returned unrecognized error %d\n",
@@ -642,7 +635,7 @@ static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 		/* Should we call xs_close() here? */
 		break;
 	case -EAGAIN:
-		status = xs_nospace(req, task);
+		status = xs_nospace(req);
 		break;
 	case -ENETUNREACH:
 	case -ENOBUFS:
@@ -765,7 +758,7 @@ static int xs_tcp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 		/* Should we call xs_close() here? */
 		break;
 	case -EAGAIN:
-		status = xs_nospace(req, task);
+		status = xs_nospace(req);
 		break;
 	case -ECONNRESET:
 	case -ECONNREFUSED:
@@ -1672,7 +1665,8 @@ static void xs_write_space(struct sock *sk)
 	if (!wq || test_and_clear_bit(SOCKWQ_ASYNC_NOSPACE, &wq->flags) == 0)
 		goto out;
 
-	xprt_write_space(xprt);
+	if (xprt_write_space(xprt))
+		sk->sk_write_pending--;
 out:
 	rcu_read_unlock();
 }
@@ -2725,12 +2719,7 @@ static int bc_send_request(struct rpc_rqst *req, struct rpc_task *task)
 	 * Grab the mutex to serialize data as the connection is shared
 	 * with the fore channel
 	 */
-	if (!mutex_trylock(&xprt->xpt_mutex)) {
-		rpc_sleep_on(&xprt->xpt_bc_pending, task, NULL);
-		if (!mutex_trylock(&xprt->xpt_mutex))
-			return -EAGAIN;
-		rpc_wake_up_queued_task(&xprt->xpt_bc_pending, task);
-	}
+	mutex_lock(&xprt->xpt_mutex);
 	if (test_bit(XPT_DEAD, &xprt->xpt_flags))
 		len = -ENOTCONN;
 	else
-- 
2.17.1

[PATCH v2 33/34] SUNRPC: Cleanup: remove the unused 'task' argument from the request_send()

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/xprt.h                |  2 +-
 net/sunrpc/xprt.c                          |  2 +-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  2 +-
 net/sunrpc/xprtrdma/transport.c            |  4 ++--
 net/sunrpc/xprtsock.c                      | 11 ++++-------
 5 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index ba4a9d83102a..0120e53b4ce9 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -141,7 +141,7 @@ struct rpc_xprt_ops {
 	void		(*connect)(struct rpc_xprt *xprt, struct rpc_task *task);
 	int		(*buf_alloc)(struct rpc_task *task);
 	void		(*buf_free)(struct rpc_task *task);
-	int		(*send_request)(struct rpc_rqst *req, struct rpc_task *task);
+	int		(*send_request)(struct rpc_rqst *req);
 	void		(*set_retrans_timeout)(struct rpc_task *task);
 	void		(*timer)(struct rpc_xprt *xprt, struct rpc_task *task);
 	void		(*release_request)(struct rpc_task *task);
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index baa454b0d855..147921f60b5a 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1174,7 +1174,7 @@ xprt_request_transmit(struct rpc_rqst *req, struct rpc_task *snd_task)
 	req->rq_ntrans++;
 
 	connect_cookie = xprt->connect_cookie;
-	status = xprt->ops->send_request(req, snd_task);
+	status = xprt->ops->send_request(req);
 	trace_xprt_transmit(xprt, req->rq_xid, status);
 	if (status != 0)
 		return status;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index 35a8c3aab302..992312504cfd 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -215,7 +215,7 @@ rpcrdma_bc_send_request(struct svcxprt_rdma *rdma, struct rpc_rqst *rqst)
  * connection.
  */
 static int
-xprt_rdma_bc_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
+xprt_rdma_bc_send_request(struct rpc_rqst *rqst)
 {
 	struct svc_xprt *sxprt = rqst->rq_xprt->bc_xprt;
 	struct svcxprt_rdma *rdma;
diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
index 9ff322e53f37..a5a6a4a353f2 100644
--- a/net/sunrpc/xprtrdma/transport.c
+++ b/net/sunrpc/xprtrdma/transport.c
@@ -693,7 +693,7 @@ xprt_rdma_free(struct rpc_task *task)
 
 /**
  * xprt_rdma_send_request - marshal and send an RPC request
- * @task: RPC task with an RPC message in rq_snd_buf
+ * @rqst: RPC message in rq_snd_buf
  *
  * Caller holds the transport's write lock.
  *
@@ -706,7 +706,7 @@ xprt_rdma_free(struct rpc_task *task)
  *		sent. Do not try to send this message again.
  */
 static int
-xprt_rdma_send_request(struct rpc_rqst *rqst, struct rpc_task *task)
+xprt_rdma_send_request(struct rpc_rqst *rqst)
 {
 	struct rpc_xprt *xprt = rqst->rq_xprt;
 	struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index ef8d0e81cbda..f16406228ead 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -507,7 +507,6 @@ static inline void xs_encode_stream_record_marker(struct xdr_buf *buf)
 /**
  * xs_local_send_request - write an RPC request to an AF_LOCAL socket
  * @req: pointer to RPC request
- * @task: RPC task that manages the state of an RPC request
  *
  * Return values:
  *        0:	The request has been sent
@@ -516,7 +515,7 @@ static inline void xs_encode_stream_record_marker(struct xdr_buf *buf)
  * ENOTCONN:	Caller needs to invoke connect logic then call again
  *    other:	Some other error occured, the request was not sent
  */
-static int xs_local_send_request(struct rpc_rqst *req, struct rpc_task *task)
+static int xs_local_send_request(struct rpc_rqst *req)
 {
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport =
@@ -579,7 +578,6 @@ static int xs_local_send_request(struct rpc_rqst *req, struct rpc_task *task)
 /**
  * xs_udp_send_request - write an RPC request to a UDP socket
  * @req: pointer to RPC request
- * @task: address of RPC task that manages the state of an RPC request
  *
  * Return values:
  *        0:	The request has been sent
@@ -588,7 +586,7 @@ static int xs_local_send_request(struct rpc_rqst *req, struct rpc_task *task)
  * ENOTCONN:	Caller needs to invoke connect logic then call again
  *    other:	Some other error occurred, the request was not sent
  */
-static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
+static int xs_udp_send_request(struct rpc_rqst *req)
 {
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
@@ -656,7 +654,6 @@ static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
 /**
  * xs_tcp_send_request - write an RPC request to a TCP socket
  * @req: pointer to RPC request
- * @task: address of RPC task that manages the state of an RPC request
  *
  * Return values:
  *        0:	The request has been sent
@@ -668,7 +665,7 @@ static int xs_udp_send_request(struct rpc_rqst *req, struct rpc_task *task)
  * XXX: In the case of soft timeouts, should we eventually give up
  *	if sendmsg is not able to make progress?
  */
-static int xs_tcp_send_request(struct rpc_rqst *req, struct rpc_task *task)
+static int xs_tcp_send_request(struct rpc_rqst *req)
 {
 	struct rpc_xprt *xprt = req->rq_xprt;
 	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
@@ -2704,7 +2701,7 @@ static int bc_sendto(struct rpc_rqst *req)
 /*
  * The send routine. Borrows from svc_send
  */
-static int bc_send_request(struct rpc_rqst *req, struct rpc_task *task)
+static int bc_send_request(struct rpc_rqst *req)
 {
 	struct svc_xprt	*xprt;
 	int len;
-- 
2.17.1

[PATCH v2 34/34] SUNRPC: Queue fairness for all.

[Trond Myklebust]

Fix up the priority queue to not batch by owner, but by queue, so that
we allow '1 << priority' elements to be dequeued before switching to
the next priority queue.
The owner field is still used to wake up requests in round robin order
by owner to avoid single processes hogging the RPC layer by loading the
queues. We extend this property to non-priority queues as well.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 include/linux/sunrpc/sched.h |   2 -
 net/sunrpc/sched.c           | 126 ++++++++++++++++-------------------
 2 files changed, 57 insertions(+), 71 deletions(-)

diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
index c5bc779feb00..869e2ee787fa 100644
--- a/include/linux/sunrpc/sched.h
+++ b/include/linux/sunrpc/sched.h
@@ -188,7 +188,6 @@ struct rpc_timer {
 struct rpc_wait_queue {
 	spinlock_t		lock;
 	struct list_head	tasks[RPC_NR_PRIORITY];	/* task queue for each priority level */
-	pid_t			owner;			/* process id of last task serviced */
 	unsigned char		maxpriority;		/* maximum priority (0 if queue is not a priority queue) */
 	unsigned char		priority;		/* current priority */
 	unsigned char		nr;			/* # tasks remaining for cookie */
@@ -204,7 +203,6 @@ struct rpc_wait_queue {
  * from a single cookie.  The aim is to improve
  * performance of NFS operations such as read/write.
  */
-#define RPC_BATCH_COUNT			16
 #define RPC_IS_PRIORITY(q)		((q)->maxpriority > 0)
 
 /*
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 104c056daf83..1120857eb1df 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -99,64 +99,77 @@ __rpc_add_timer(struct rpc_wait_queue *queue, struct rpc_task *task)
 	list_add(&task->u.tk_wait.timer_list, &queue->timer_list.list);
 }
 
-static void rpc_rotate_queue_owner(struct rpc_wait_queue *queue)
-{
-	struct list_head *q = &queue->tasks[queue->priority];
-	struct rpc_task *task;
-
-	if (!list_empty(q)) {
-		task = list_first_entry(q, struct rpc_task, u.tk_wait.list);
-		if (task->tk_owner == queue->owner)
-			list_move_tail(&task->u.tk_wait.list, q);
-	}
-}
-
 static void rpc_set_waitqueue_priority(struct rpc_wait_queue *queue, int priority)
 {
 	if (queue->priority != priority) {
-		/* Fairness: rotate the list when changing priority */
-		rpc_rotate_queue_owner(queue);
 		queue->priority = priority;
+		queue->nr = 1U << priority;
 	}
 }
 
-static void rpc_set_waitqueue_owner(struct rpc_wait_queue *queue, pid_t pid)
-{
-	queue->owner = pid;
-	queue->nr = RPC_BATCH_COUNT;
-}
-
 static void rpc_reset_waitqueue_priority(struct rpc_wait_queue *queue)
 {
 	rpc_set_waitqueue_priority(queue, queue->maxpriority);
-	rpc_set_waitqueue_owner(queue, 0);
 }
 
 /*
- * Add new request to a priority queue.
+ * Add a request to a queue list
  */
-static void __rpc_add_wait_queue_priority(struct rpc_wait_queue *queue,
-		struct rpc_task *task,
-		unsigned char queue_priority)
+static void
+__rpc_list_enqueue_task(struct list_head *q, struct rpc_task *task)
 {
-	struct list_head *q;
 	struct rpc_task *t;
-
-	INIT_LIST_HEAD(&task->u.tk_wait.links);
-	if (unlikely(queue_priority > queue->maxpriority))
-		queue_priority = queue->maxpriority;
-	if (queue_priority > queue->priority)
-		rpc_set_waitqueue_priority(queue, queue_priority);
-	q = &queue->tasks[queue_priority];
 	list_for_each_entry(t, q, u.tk_wait.list) {
 		if (t->tk_owner == task->tk_owner) {
-			list_add_tail(&task->u.tk_wait.list, &t->u.tk_wait.links);
+			list_add_tail(&task->u.tk_wait.links,
+					&t->u.tk_wait.links);
+			/* Cache the queue head in task->u.tk_wait.list */
+			task->u.tk_wait.list.next = q;
+			task->u.tk_wait.list.prev = NULL;
 			return;
 		}
 	}
+	INIT_LIST_HEAD(&task->u.tk_wait.links);
 	list_add_tail(&task->u.tk_wait.list, q);
 }
 
+/*
+ * Remove request from a queue list
+ */
+static void
+__rpc_list_dequeue_task(struct rpc_task *task)
+{
+	struct list_head *q;
+	struct rpc_task *t;
+
+	if (task->u.tk_wait.list.prev == NULL) {
+		list_del(&task->u.tk_wait.links);
+		return;
+	}
+	if (!list_empty(&task->u.tk_wait.links)) {
+		t = list_first_entry(&task->u.tk_wait.links,
+				struct rpc_task,
+				u.tk_wait.links);
+		/* Assume __rpc_list_enqueue_task() cached the queue head */
+		q = t->u.tk_wait.list.next;
+		list_add_tail(&t->u.tk_wait.list, q);
+		list_del(&task->u.tk_wait.links);
+	}
+	list_del(&task->u.tk_wait.list);
+}
+
+/*
+ * Add new request to a priority queue.
+ */
+static void __rpc_add_wait_queue_priority(struct rpc_wait_queue *queue,
+		struct rpc_task *task,
+		unsigned char queue_priority)
+{
+	if (unlikely(queue_priority > queue->maxpriority))
+		queue_priority = queue->maxpriority;
+	__rpc_list_enqueue_task(&queue->tasks[queue_priority], task);
+}
+
 /*
  * Add new request to wait queue.
  *
@@ -175,10 +188,11 @@ static void __rpc_add_wait_queue(struct rpc_wait_queue *queue,
 
 	if (RPC_IS_PRIORITY(queue))
 		__rpc_add_wait_queue_priority(queue, task, queue_priority);
-	else if (RPC_IS_SWAPPER(task))
+	else if (RPC_IS_SWAPPER(task)) {
 		list_add(&task->u.tk_wait.list, &queue->tasks[0]);
-	else
-		list_add_tail(&task->u.tk_wait.list, &queue->tasks[0]);
+		INIT_LIST_HEAD(&task->u.tk_wait.links);
+	} else
+		__rpc_list_enqueue_task(&queue->tasks[0], task);
 	task->tk_waitqueue = queue;
 	queue->qlen++;
 	/* barrier matches the read in rpc_wake_up_task_queue_locked() */
@@ -189,20 +203,6 @@ static void __rpc_add_wait_queue(struct rpc_wait_queue *queue,
 			task->tk_pid, queue, rpc_qname(queue));
 }
 
-/*
- * Remove request from a priority queue.
- */
-static void __rpc_remove_wait_queue_priority(struct rpc_task *task)
-{
-	struct rpc_task *t;
-
-	if (!list_empty(&task->u.tk_wait.links)) {
-		t = list_entry(task->u.tk_wait.links.next, struct rpc_task, u.tk_wait.list);
-		list_move(&t->u.tk_wait.list, &task->u.tk_wait.list);
-		list_splice_init(&task->u.tk_wait.links, &t->u.tk_wait.links);
-	}
-}
-
 /*
  * Remove request from queue.
  * Note: must be called with spin lock held.
@@ -210,9 +210,7 @@ static void __rpc_remove_wait_queue_priority(struct rpc_task *task)
 static void __rpc_remove_wait_queue(struct rpc_wait_queue *queue, struct rpc_task *task)
 {
 	__rpc_disable_timer(queue, task);
-	if (RPC_IS_PRIORITY(queue))
-		__rpc_remove_wait_queue_priority(task);
-	list_del(&task->u.tk_wait.list);
+	__rpc_list_dequeue_task(task);
 	queue->qlen--;
 	dprintk("RPC: %5u removed from queue %p \"%s\"\n",
 			task->tk_pid, queue, rpc_qname(queue));
@@ -536,20 +534,12 @@ static struct rpc_task *__rpc_find_next_queued_priority(struct rpc_wait_queue *q
 	struct rpc_task *task;
 
 	/*
-	 * Service a batch of tasks from a single owner.
+	 * Service a batch of tasks from a single queue.
 	 */
 	q = &queue->tasks[queue->priority];
-	if (!list_empty(q)) {
-		task = list_entry(q->next, struct rpc_task, u.tk_wait.list);
-		if (queue->owner == task->tk_owner) {
-			if (--queue->nr)
-				goto out;
-			list_move_tail(&task->u.tk_wait.list, q);
-		}
-		/*
-		 * Check if we need to switch queues.
-		 */
-		goto new_owner;
+	if (!list_empty(q) && --queue->nr) {
+		task = list_first_entry(q, struct rpc_task, u.tk_wait.list);
+		goto out;
 	}
 
 	/*
@@ -561,7 +551,7 @@ static struct rpc_task *__rpc_find_next_queued_priority(struct rpc_wait_queue *q
 		else
 			q = q - 1;
 		if (!list_empty(q)) {
-			task = list_entry(q->next, struct rpc_task, u.tk_wait.list);
+			task = list_first_entry(q, struct rpc_task, u.tk_wait.list);
 			goto new_queue;
 		}
 	} while (q != &queue->tasks[queue->priority]);
@@ -571,8 +561,6 @@ static struct rpc_task *__rpc_find_next_queued_priority(struct rpc_wait_queue *q
 
 new_queue:
 	rpc_set_waitqueue_priority(queue, (unsigned int)(q - &queue->tasks[0]));
-new_owner:
-	rpc_set_waitqueue_owner(queue, task->tk_owner);
 out:
 	return task;
 }
-- 
2.17.1

Re: [PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Chuck Lever]

> On Sep 4, 2018, at 5:05 PM, Trond Myklebust <trondmy@gmail.com> wrote:
>=20
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
> net/sunrpc/clnt.c |  6 ------
> net/sunrpc/xprt.c | 13 ++++++-------
> 2 files changed, 6 insertions(+), 13 deletions(-)
>=20
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index 032b7042adb6..21713bed812a 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -1967,8 +1967,6 @@ call_connect_status(struct rpc_task *task)
> static void
> call_transmit(struct rpc_task *task)
> {
> -	int is_retrans =3D RPC_WAS_SENT(task);
> -
> 	dprint_status(task);
>=20
> 	task->tk_action =3D call_transmit_status;
> @@ -1979,10 +1977,6 @@ call_transmit(struct rpc_task *task)
> 	if (!xprt_prepare_transmit(task))
> 		return;
> 	xprt_transmit(task);
> -	if (task->tk_status < 0)
> -		return;
> -	if (is_retrans)
> -		task->tk_client->cl_stats->rpcretrans++;
> }
>=20
> /*
> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> index f1301d391399..e2f5b4668a66 100644
> --- a/net/sunrpc/xprt.c
> +++ b/net/sunrpc/xprt.c
> @@ -191,8 +191,6 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt, =
struct rpc_task *task)
> 		goto out_sleep;
> 	}
> 	xprt->snd_task =3D task;
> -	if (req !=3D NULL)
> -		req->rq_ntrans++;
>=20
> 	return 1;
>=20
> @@ -247,7 +245,6 @@ int xprt_reserve_xprt_cong(struct rpc_xprt *xprt, =
struct rpc_task *task)
> 	}
> 	if (__xprt_get_cong(xprt, task)) {
> 		xprt->snd_task =3D task;
> -		req->rq_ntrans++;
> 		return 1;
> 	}
> 	xprt_clear_locked(xprt);
> @@ -281,12 +278,8 @@ static inline int xprt_lock_write(struct rpc_xprt =
*xprt, struct rpc_task *task)
> static bool __xprt_lock_write_func(struct rpc_task *task, void *data)
> {
> 	struct rpc_xprt *xprt =3D data;
> -	struct rpc_rqst *req;
>=20
> -	req =3D task->tk_rqstp;
> 	xprt->snd_task =3D task;
> -	if (req)
> -		req->rq_ntrans++;
> 	return true;
> }
>=20
> @@ -1126,6 +1119,7 @@ void xprt_transmit(struct rpc_task *task)
> 	struct rpc_rqst	*req =3D task->tk_rqstp;
> 	struct rpc_xprt	*xprt =3D req->rq_xprt;
> 	unsigned int connect_cookie;
> +	int is_retrans =3D RPC_WAS_SENT(task);
> 	int status;
>=20
> 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, =
req->rq_slen);
> @@ -1140,6 +1134,8 @@ void xprt_transmit(struct rpc_task *task)
> 		}
> 	}
>=20
> +	req->rq_ntrans++;
> +

rq_ntrans is used in two places:

1. rpc_update_rtt
2. rpc_count_iostats_metrics

Both of these appear to assume that rq_ntrans is counting
full successful transmissions (ie, calls that appear on the
wire), not invocations of ->send_request().

Can this counter be moved down to where rpcretrans is updated?


> 	connect_cookie =3D xprt->connect_cookie;
> 	status =3D xprt->ops->send_request(req, task);
> 	trace_xprt_transmit(xprt, req->rq_xid, status);
> @@ -1148,6 +1144,9 @@ void xprt_transmit(struct rpc_task *task)
> 		return;
> 	}
>=20
> +	if (is_retrans)
> +		task->tk_client->cl_stats->rpcretrans++;
> +
> 	xprt_inject_disconnect(xprt);
>=20
> 	dprintk("RPC: %5u xmit complete\n", task->tk_pid);
> --=20
> 2.17.1
>=20

--
Chuck Lever

Re: [PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Trond Myklebust]

T24gV2VkLCAyMDE4LTA5LTA1IGF0IDEwOjMwIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
PiBPbiBTZXAgNCwgMjAxOCwgYXQgNTowNSBQTSwgVHJvbmQgTXlrbGVidXN0IDx0cm9uZG15QGdt
YWlsLmNvbT4NCj4gPiB3cm90ZToNCj4gPiANCj4gPiBTaWduZWQtb2ZmLWJ5OiBUcm9uZCBNeWts
ZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBoYW1tZXJzcGFjZS5jb20+DQo+ID4gLS0tDQo+ID4gbmV0
L3N1bnJwYy9jbG50LmMgfCAgNiAtLS0tLS0NCj4gPiBuZXQvc3VucnBjL3hwcnQuYyB8IDEzICsr
KysrKy0tLS0tLS0NCj4gPiAyIGZpbGVzIGNoYW5nZWQsIDYgaW5zZXJ0aW9ucygrKSwgMTMgZGVs
ZXRpb25zKC0pDQo+ID4gDQo+ID4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMvY2xudC5jIGIvbmV0
L3N1bnJwYy9jbG50LmMNCj4gPiBpbmRleCAwMzJiNzA0MmFkYjYuLjIxNzEzYmVkODEyYSAxMDA2
NDQNCj4gPiAtLS0gYS9uZXQvc3VucnBjL2NsbnQuYw0KPiA+ICsrKyBiL25ldC9zdW5ycGMvY2xu
dC5jDQo+ID4gQEAgLTE5NjcsOCArMTk2Nyw2IEBAIGNhbGxfY29ubmVjdF9zdGF0dXMoc3RydWN0
IHJwY190YXNrICp0YXNrKQ0KPiA+IHN0YXRpYyB2b2lkDQo+ID4gY2FsbF90cmFuc21pdChzdHJ1
Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ID4gew0KPiA+IC0JaW50IGlzX3JldHJhbnMgPSBSUENfV0FT
X1NFTlQodGFzayk7DQo+ID4gLQ0KPiA+IAlkcHJpbnRfc3RhdHVzKHRhc2spOw0KPiA+IA0KPiA+
IAl0YXNrLT50a19hY3Rpb24gPSBjYWxsX3RyYW5zbWl0X3N0YXR1czsNCj4gPiBAQCAtMTk3OSwx
MCArMTk3Nyw2IEBAIGNhbGxfdHJhbnNtaXQoc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+IAlp
ZiAoIXhwcnRfcHJlcGFyZV90cmFuc21pdCh0YXNrKSkNCj4gPiAJCXJldHVybjsNCj4gPiAJeHBy
dF90cmFuc21pdCh0YXNrKTsNCj4gPiAtCWlmICh0YXNrLT50a19zdGF0dXMgPCAwKQ0KPiA+IC0J
CXJldHVybjsNCj4gPiAtCWlmIChpc19yZXRyYW5zKQ0KPiA+IC0JCXRhc2stPnRrX2NsaWVudC0+
Y2xfc3RhdHMtPnJwY3JldHJhbnMrKzsNCj4gPiB9DQo+ID4gDQo+ID4gLyoNCj4gPiBkaWZmIC0t
Z2l0IGEvbmV0L3N1bnJwYy94cHJ0LmMgYi9uZXQvc3VucnBjL3hwcnQuYw0KPiA+IGluZGV4IGYx
MzAxZDM5MTM5OS4uZTJmNWI0NjY4YTY2IDEwMDY0NA0KPiA+IC0tLSBhL25ldC9zdW5ycGMveHBy
dC5jDQo+ID4gKysrIGIvbmV0L3N1bnJwYy94cHJ0LmMNCj4gPiBAQCAtMTkxLDggKzE5MSw2IEBA
IGludCB4cHJ0X3Jlc2VydmVfeHBydChzdHJ1Y3QgcnBjX3hwcnQgKnhwcnQsDQo+ID4gc3RydWN0
IHJwY190YXNrICp0YXNrKQ0KPiA+IAkJZ290byBvdXRfc2xlZXA7DQo+ID4gCX0NCj4gPiAJeHBy
dC0+c25kX3Rhc2sgPSB0YXNrOw0KPiA+IC0JaWYgKHJlcSAhPSBOVUxMKQ0KPiA+IC0JCXJlcS0+
cnFfbnRyYW5zKys7DQo+ID4gDQo+ID4gCXJldHVybiAxOw0KPiA+IA0KPiA+IEBAIC0yNDcsNyAr
MjQ1LDYgQEAgaW50IHhwcnRfcmVzZXJ2ZV94cHJ0X2Nvbmcoc3RydWN0IHJwY194cHJ0DQo+ID4g
KnhwcnQsIHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4gPiAJfQ0KPiA+IAlpZiAoX194cHJ0X2dl
dF9jb25nKHhwcnQsIHRhc2spKSB7DQo+ID4gCQl4cHJ0LT5zbmRfdGFzayA9IHRhc2s7DQo+ID4g
LQkJcmVxLT5ycV9udHJhbnMrKzsNCj4gPiAJCXJldHVybiAxOw0KPiA+IAl9DQo+ID4gCXhwcnRf
Y2xlYXJfbG9ja2VkKHhwcnQpOw0KPiA+IEBAIC0yODEsMTIgKzI3OCw4IEBAIHN0YXRpYyBpbmxp
bmUgaW50IHhwcnRfbG9ja193cml0ZShzdHJ1Y3QNCj4gPiBycGNfeHBydCAqeHBydCwgc3RydWN0
IHJwY190YXNrICp0YXNrKQ0KPiA+IHN0YXRpYyBib29sIF9feHBydF9sb2NrX3dyaXRlX2Z1bmMo
c3RydWN0IHJwY190YXNrICp0YXNrLCB2b2lkDQo+ID4gKmRhdGEpDQo+ID4gew0KPiA+IAlzdHJ1
Y3QgcnBjX3hwcnQgKnhwcnQgPSBkYXRhOw0KPiA+IC0Jc3RydWN0IHJwY19ycXN0ICpyZXE7DQo+
ID4gDQo+ID4gLQlyZXEgPSB0YXNrLT50a19ycXN0cDsNCj4gPiAJeHBydC0+c25kX3Rhc2sgPSB0
YXNrOw0KPiA+IC0JaWYgKHJlcSkNCj4gPiAtCQlyZXEtPnJxX250cmFucysrOw0KPiA+IAlyZXR1
cm4gdHJ1ZTsNCj4gPiB9DQo+ID4gDQo+ID4gQEAgLTExMjYsNiArMTExOSw3IEBAIHZvaWQgeHBy
dF90cmFuc21pdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ID4gCXN0cnVjdCBycGNfcnFzdAkq
cmVxID0gdGFzay0+dGtfcnFzdHA7DQo+ID4gCXN0cnVjdCBycGNfeHBydAkqeHBydCA9IHJlcS0+
cnFfeHBydDsNCj4gPiAJdW5zaWduZWQgaW50IGNvbm5lY3RfY29va2llOw0KPiA+ICsJaW50IGlz
X3JldHJhbnMgPSBSUENfV0FTX1NFTlQodGFzayk7DQo+ID4gCWludCBzdGF0dXM7DQo+ID4gDQo+
ID4gCWRwcmludGsoIlJQQzogJTV1IHhwcnRfdHJhbnNtaXQoJXUpXG4iLCB0YXNrLT50a19waWQs
IHJlcS0NCj4gPiA+cnFfc2xlbik7DQo+ID4gQEAgLTExNDAsNiArMTEzNCw4IEBAIHZvaWQgeHBy
dF90cmFuc21pdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ID4gCQl9DQo+ID4gCX0NCj4gPiAN
Cj4gPiArCXJlcS0+cnFfbnRyYW5zKys7DQo+ID4gKw0KPiANCj4gcnFfbnRyYW5zIGlzIHVzZWQg
aW4gdHdvIHBsYWNlczoNCj4gDQo+IDEuIHJwY191cGRhdGVfcnR0DQo+IDIuIHJwY19jb3VudF9p
b3N0YXRzX21ldHJpY3MNCj4gDQo+IEJvdGggb2YgdGhlc2UgYXBwZWFyIHRvIGFzc3VtZSB0aGF0
IHJxX250cmFucyBpcyBjb3VudGluZw0KPiBmdWxsIHN1Y2Nlc3NmdWwgdHJhbnNtaXNzaW9ucyAo
aWUsIGNhbGxzIHRoYXQgYXBwZWFyIG9uIHRoZQ0KPiB3aXJlKSwgbm90IGludm9jYXRpb25zIG9m
IC0+c2VuZF9yZXF1ZXN0KCkuDQo+IA0KPiBDYW4gdGhpcyBjb3VudGVyIGJlIG1vdmVkIGRvd24g
dG8gd2hlcmUgcnBjcmV0cmFucyBpcyB1cGRhdGVkPw0KPiANCg0KVGhlIHJlYXNvbiB3aHkgSSBk
b24ndCB3YW50IHRvIHVwZGF0ZSByZXEtPnJxX250cmFucyBhZnRlciB0cmFuc21pc3Npb24NCmlz
IGJlY2F1c2UgdGhhdCB3b3VsZCByYWNlIHdpdGggdGhlIHJlcGx5IGZyb20gdGhlIHNlcnZlciAo
d2hpY2ggaXMgbm90DQpibG9ja2VkIGJ5IHRoZSBYUFJUX0xPQ0spLiBJbiB0aGF0IHJhY2Ugc2Nl
bmFyaW8sIHJwY191cGRhdGVfcnR0KCkNCmNvdWxkIHVwZGF0ZSB0aGUgUlRUIHZhbHVlIGJlZm9y
ZSB3ZSd2ZSB1cGRhdGVkIHRoZSByZXEtPnJxX250cmFucywNCm1lYW5pbmcgdGhhdCB3ZSBtaWdo
dCBiZSBtZWFzdXJpbmcgdGhlIHJlc3BvbnNlIHRpbWUgYWdhaW5zdCBhIHJlcGx5IHRvDQp0aGUg
b3JpZ2luYWwgdHJhbnNtaXNzaW9uIG9yIHRvIHRoZSByZXRyYW5zbWlzc2lvbiB0aGF0IHdhcyBq
dXN0IHNlbnQuDQpUaGF0IHdvdWxkIGVuZCB1cCBicmVha2luZyBWYW4gSmFjb2JzZW4gY29uZ2Vz
dGlvbiBjb250cm9sLCBzaW5jZSB3ZQ0KcmlzayBzaWduaWZpY2FudGx5IHVuZGVyZXN0aW1hdGlu
ZyB0aGUgUlRUIHZhbHVlLg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVu
dCBtYWludGFpbmVyLCBIYW1tZXJzcGFjZQ0KdHJvbmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNv
bQ0KDQoNCg==

Re: [PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Chuck Lever]

> On Sep 5, 2018, at 11:28 AM, Trond Myklebust <trondmy@hammerspace.com> =
wrote:
>=20
> On Wed, 2018-09-05 at 10:30 -0400, Chuck Lever wrote:
>>> On Sep 4, 2018, at 5:05 PM, Trond Myklebust <trondmy@gmail.com>
>>> wrote:
>>>=20
>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
>>> ---
>>> net/sunrpc/clnt.c |  6 ------
>>> net/sunrpc/xprt.c | 13 ++++++-------
>>> 2 files changed, 6 insertions(+), 13 deletions(-)
>>>=20
>>> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
>>> index 032b7042adb6..21713bed812a 100644
>>> --- a/net/sunrpc/clnt.c
>>> +++ b/net/sunrpc/clnt.c
>>> @@ -1967,8 +1967,6 @@ call_connect_status(struct rpc_task *task)
>>> static void
>>> call_transmit(struct rpc_task *task)
>>> {
>>> -	int is_retrans =3D RPC_WAS_SENT(task);
>>> -
>>> 	dprint_status(task);
>>>=20
>>> 	task->tk_action =3D call_transmit_status;
>>> @@ -1979,10 +1977,6 @@ call_transmit(struct rpc_task *task)
>>> 	if (!xprt_prepare_transmit(task))
>>> 		return;
>>> 	xprt_transmit(task);
>>> -	if (task->tk_status < 0)
>>> -		return;
>>> -	if (is_retrans)
>>> -		task->tk_client->cl_stats->rpcretrans++;
>>> }
>>>=20
>>> /*
>>> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
>>> index f1301d391399..e2f5b4668a66 100644
>>> --- a/net/sunrpc/xprt.c
>>> +++ b/net/sunrpc/xprt.c
>>> @@ -191,8 +191,6 @@ int xprt_reserve_xprt(struct rpc_xprt *xprt,
>>> struct rpc_task *task)
>>> 		goto out_sleep;
>>> 	}
>>> 	xprt->snd_task =3D task;
>>> -	if (req !=3D NULL)
>>> -		req->rq_ntrans++;
>>>=20
>>> 	return 1;
>>>=20
>>> @@ -247,7 +245,6 @@ int xprt_reserve_xprt_cong(struct rpc_xprt
>>> *xprt, struct rpc_task *task)
>>> 	}
>>> 	if (__xprt_get_cong(xprt, task)) {
>>> 		xprt->snd_task =3D task;
>>> -		req->rq_ntrans++;
>>> 		return 1;
>>> 	}
>>> 	xprt_clear_locked(xprt);
>>> @@ -281,12 +278,8 @@ static inline int xprt_lock_write(struct
>>> rpc_xprt *xprt, struct rpc_task *task)
>>> static bool __xprt_lock_write_func(struct rpc_task *task, void
>>> *data)
>>> {
>>> 	struct rpc_xprt *xprt =3D data;
>>> -	struct rpc_rqst *req;
>>>=20
>>> -	req =3D task->tk_rqstp;
>>> 	xprt->snd_task =3D task;
>>> -	if (req)
>>> -		req->rq_ntrans++;
>>> 	return true;
>>> }
>>>=20
>>> @@ -1126,6 +1119,7 @@ void xprt_transmit(struct rpc_task *task)
>>> 	struct rpc_rqst	*req =3D task->tk_rqstp;
>>> 	struct rpc_xprt	*xprt =3D req->rq_xprt;
>>> 	unsigned int connect_cookie;
>>> +	int is_retrans =3D RPC_WAS_SENT(task);
>>> 	int status;
>>>=20
>>> 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid, req-
>>>> rq_slen);
>>> @@ -1140,6 +1134,8 @@ void xprt_transmit(struct rpc_task *task)
>>> 		}
>>> 	}
>>>=20
>>> +	req->rq_ntrans++;
>>> +
>>=20
>> rq_ntrans is used in two places:
>>=20
>> 1. rpc_update_rtt
>> 2. rpc_count_iostats_metrics
>>=20
>> Both of these appear to assume that rq_ntrans is counting
>> full successful transmissions (ie, calls that appear on the
>> wire), not invocations of ->send_request().
>>=20
>> Can this counter be moved down to where rpcretrans is updated?
>>=20
>=20
> The reason why I don't want to update req->rq_ntrans after =
transmission
> is because that would race with the reply from the server (which is =
not
> blocked by the XPRT_LOCK). In that race scenario, rpc_update_rtt()
> could update the RTT value before we've updated the req->rq_ntrans,
> meaning that we might be measuring the response time against a reply =
to
> the original transmission or to the retransmission that was just sent.
> That would end up breaking Van Jacobsen congestion control, since we
> risk significantly underestimating the RTT value.

But for transports that don't use xprt_update_rtt, error
returns from ->send_request() (like -EAGAIN or -ENOBUFS)
means that rq_ntrans is counted twice, and it shows up as
false retransmissions.

What if rq_ntrans was bumped in the transport code instead?


--
Chuck Lever

Re: [PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Trond Myklebust]

T24gV2VkLCAyMDE4LTA5LTA1IGF0IDExOjMxIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
PiBPbiBTZXAgNSwgMjAxOCwgYXQgMTE6MjggQU0sIFRyb25kIE15a2xlYnVzdCA8DQo+ID4gdHJv
bmRteUBoYW1tZXJzcGFjZS5jb20+IHdyb3RlOg0KPiA+IA0KPiA+IE9uIFdlZCwgMjAxOC0wOS0w
NSBhdCAxMDozMCAtMDQwMCwgQ2h1Y2sgTGV2ZXIgd3JvdGU6DQo+ID4gPiA+IE9uIFNlcCA0LCAy
MDE4LCBhdCA1OjA1IFBNLCBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kbXlAZ21haWwuY29tPg0KPiA+
ID4gPiB3cm90ZToNCj4gPiA+ID4gDQo+ID4gPiA+IFNpZ25lZC1vZmYtYnk6IFRyb25kIE15a2xl
YnVzdCA8dHJvbmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KPiA+ID4gPiA+DQo+ID4gPiA+
IC0tLQ0KPiA+ID4gPiBuZXQvc3VucnBjL2NsbnQuYyB8ICA2IC0tLS0tLQ0KPiA+ID4gPiBuZXQv
c3VucnBjL3hwcnQuYyB8IDEzICsrKysrKy0tLS0tLS0NCj4gPiA+ID4gMiBmaWxlcyBjaGFuZ2Vk
LCA2IGluc2VydGlvbnMoKyksIDEzIGRlbGV0aW9ucygtKQ0KPiA+ID4gPiANCj4gPiA+ID4gZGlm
ZiAtLWdpdCBhL25ldC9zdW5ycGMvY2xudC5jIGIvbmV0L3N1bnJwYy9jbG50LmMNCj4gPiA+ID4g
aW5kZXggMDMyYjcwNDJhZGI2Li4yMTcxM2JlZDgxMmEgMTAwNjQ0DQo+ID4gPiA+IC0tLSBhL25l
dC9zdW5ycGMvY2xudC5jDQo+ID4gPiA+ICsrKyBiL25ldC9zdW5ycGMvY2xudC5jDQo+ID4gPiA+
IEBAIC0xOTY3LDggKzE5NjcsNiBAQCBjYWxsX2Nvbm5lY3Rfc3RhdHVzKHN0cnVjdCBycGNfdGFz
aw0KPiA+ID4gPiAqdGFzaykNCj4gPiA+ID4gc3RhdGljIHZvaWQNCj4gPiA+ID4gY2FsbF90cmFu
c21pdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ID4gPiA+IHsNCj4gPiA+ID4gLQlpbnQgaXNf
cmV0cmFucyA9IFJQQ19XQVNfU0VOVCh0YXNrKTsNCj4gPiA+ID4gLQ0KPiA+ID4gPiAJZHByaW50
X3N0YXR1cyh0YXNrKTsNCj4gPiA+ID4gDQo+ID4gPiA+IAl0YXNrLT50a19hY3Rpb24gPSBjYWxs
X3RyYW5zbWl0X3N0YXR1czsNCj4gPiA+ID4gQEAgLTE5NzksMTAgKzE5NzcsNiBAQCBjYWxsX3Ry
YW5zbWl0KHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4gPiA+ID4gCWlmICgheHBydF9wcmVwYXJl
X3RyYW5zbWl0KHRhc2spKQ0KPiA+ID4gPiAJCXJldHVybjsNCj4gPiA+ID4gCXhwcnRfdHJhbnNt
aXQodGFzayk7DQo+ID4gPiA+IC0JaWYgKHRhc2stPnRrX3N0YXR1cyA8IDApDQo+ID4gPiA+IC0J
CXJldHVybjsNCj4gPiA+ID4gLQlpZiAoaXNfcmV0cmFucykNCj4gPiA+ID4gLQkJdGFzay0+dGtf
Y2xpZW50LT5jbF9zdGF0cy0+cnBjcmV0cmFucysrOw0KPiA+ID4gPiB9DQo+ID4gPiA+IA0KPiA+
ID4gPiAvKg0KPiA+ID4gPiBkaWZmIC0tZ2l0IGEvbmV0L3N1bnJwYy94cHJ0LmMgYi9uZXQvc3Vu
cnBjL3hwcnQuYw0KPiA+ID4gPiBpbmRleCBmMTMwMWQzOTEzOTkuLmUyZjViNDY2OGE2NiAxMDA2
NDQNCj4gPiA+ID4gLS0tIGEvbmV0L3N1bnJwYy94cHJ0LmMNCj4gPiA+ID4gKysrIGIvbmV0L3N1
bnJwYy94cHJ0LmMNCj4gPiA+ID4gQEAgLTE5MSw4ICsxOTEsNiBAQCBpbnQgeHBydF9yZXNlcnZl
X3hwcnQoc3RydWN0IHJwY194cHJ0DQo+ID4gPiA+ICp4cHJ0LA0KPiA+ID4gPiBzdHJ1Y3QgcnBj
X3Rhc2sgKnRhc2spDQo+ID4gPiA+IAkJZ290byBvdXRfc2xlZXA7DQo+ID4gPiA+IAl9DQo+ID4g
PiA+IAl4cHJ0LT5zbmRfdGFzayA9IHRhc2s7DQo+ID4gPiA+IC0JaWYgKHJlcSAhPSBOVUxMKQ0K
PiA+ID4gPiAtCQlyZXEtPnJxX250cmFucysrOw0KPiA+ID4gPiANCj4gPiA+ID4gCXJldHVybiAx
Ow0KPiA+ID4gPiANCj4gPiA+ID4gQEAgLTI0Nyw3ICsyNDUsNiBAQCBpbnQgeHBydF9yZXNlcnZl
X3hwcnRfY29uZyhzdHJ1Y3QgcnBjX3hwcnQNCj4gPiA+ID4gKnhwcnQsIHN0cnVjdCBycGNfdGFz
ayAqdGFzaykNCj4gPiA+ID4gCX0NCj4gPiA+ID4gCWlmIChfX3hwcnRfZ2V0X2NvbmcoeHBydCwg
dGFzaykpIHsNCj4gPiA+ID4gCQl4cHJ0LT5zbmRfdGFzayA9IHRhc2s7DQo+ID4gPiA+IC0JCXJl
cS0+cnFfbnRyYW5zKys7DQo+ID4gPiA+IAkJcmV0dXJuIDE7DQo+ID4gPiA+IAl9DQo+ID4gPiA+
IAl4cHJ0X2NsZWFyX2xvY2tlZCh4cHJ0KTsNCj4gPiA+ID4gQEAgLTI4MSwxMiArMjc4LDggQEAg
c3RhdGljIGlubGluZSBpbnQgeHBydF9sb2NrX3dyaXRlKHN0cnVjdA0KPiA+ID4gPiBycGNfeHBy
dCAqeHBydCwgc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ID4gPiBzdGF0aWMgYm9vbCBfX3hw
cnRfbG9ja193cml0ZV9mdW5jKHN0cnVjdCBycGNfdGFzayAqdGFzaywgdm9pZA0KPiA+ID4gPiAq
ZGF0YSkNCj4gPiA+ID4gew0KPiA+ID4gPiAJc3RydWN0IHJwY194cHJ0ICp4cHJ0ID0gZGF0YTsN
Cj4gPiA+ID4gLQlzdHJ1Y3QgcnBjX3Jxc3QgKnJlcTsNCj4gPiA+ID4gDQo+ID4gPiA+IC0JcmVx
ID0gdGFzay0+dGtfcnFzdHA7DQo+ID4gPiA+IAl4cHJ0LT5zbmRfdGFzayA9IHRhc2s7DQo+ID4g
PiA+IC0JaWYgKHJlcSkNCj4gPiA+ID4gLQkJcmVxLT5ycV9udHJhbnMrKzsNCj4gPiA+ID4gCXJl
dHVybiB0cnVlOw0KPiA+ID4gPiB9DQo+ID4gPiA+IA0KPiA+ID4gPiBAQCAtMTEyNiw2ICsxMTE5
LDcgQEAgdm9pZCB4cHJ0X3RyYW5zbWl0KHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4gPiA+ID4g
CXN0cnVjdCBycGNfcnFzdAkqcmVxID0gdGFzay0+dGtfcnFzdHA7DQo+ID4gPiA+IAlzdHJ1Y3Qg
cnBjX3hwcnQJKnhwcnQgPSByZXEtPnJxX3hwcnQ7DQo+ID4gPiA+IAl1bnNpZ25lZCBpbnQgY29u
bmVjdF9jb29raWU7DQo+ID4gPiA+ICsJaW50IGlzX3JldHJhbnMgPSBSUENfV0FTX1NFTlQodGFz
ayk7DQo+ID4gPiA+IAlpbnQgc3RhdHVzOw0KPiA+ID4gPiANCj4gPiA+ID4gCWRwcmludGsoIlJQ
QzogJTV1IHhwcnRfdHJhbnNtaXQoJXUpXG4iLCB0YXNrLT50a19waWQsDQo+ID4gPiA+IHJlcS0N
Cj4gPiA+ID4gPiBycV9zbGVuKTsNCj4gPiA+ID4gDQo+ID4gPiA+IEBAIC0xMTQwLDYgKzExMzQs
OCBAQCB2b2lkIHhwcnRfdHJhbnNtaXQoc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ID4gPiAJ
CX0NCj4gPiA+ID4gCX0NCj4gPiA+ID4gDQo+ID4gPiA+ICsJcmVxLT5ycV9udHJhbnMrKzsNCj4g
PiA+ID4gKw0KPiA+ID4gDQo+ID4gPiBycV9udHJhbnMgaXMgdXNlZCBpbiB0d28gcGxhY2VzOg0K
PiA+ID4gDQo+ID4gPiAxLiBycGNfdXBkYXRlX3J0dA0KPiA+ID4gMi4gcnBjX2NvdW50X2lvc3Rh
dHNfbWV0cmljcw0KPiA+ID4gDQo+ID4gPiBCb3RoIG9mIHRoZXNlIGFwcGVhciB0byBhc3N1bWUg
dGhhdCBycV9udHJhbnMgaXMgY291bnRpbmcNCj4gPiA+IGZ1bGwgc3VjY2Vzc2Z1bCB0cmFuc21p
c3Npb25zIChpZSwgY2FsbHMgdGhhdCBhcHBlYXIgb24gdGhlDQo+ID4gPiB3aXJlKSwgbm90IGlu
dm9jYXRpb25zIG9mIC0+c2VuZF9yZXF1ZXN0KCkuDQo+ID4gPiANCj4gPiA+IENhbiB0aGlzIGNv
dW50ZXIgYmUgbW92ZWQgZG93biB0byB3aGVyZSBycGNyZXRyYW5zIGlzIHVwZGF0ZWQ/DQo+ID4g
PiANCj4gPiANCj4gPiBUaGUgcmVhc29uIHdoeSBJIGRvbid0IHdhbnQgdG8gdXBkYXRlIHJlcS0+
cnFfbnRyYW5zIGFmdGVyDQo+ID4gdHJhbnNtaXNzaW9uDQo+ID4gaXMgYmVjYXVzZSB0aGF0IHdv
dWxkIHJhY2Ugd2l0aCB0aGUgcmVwbHkgZnJvbSB0aGUgc2VydmVyICh3aGljaCBpcw0KPiA+IG5v
dA0KPiA+IGJsb2NrZWQgYnkgdGhlIFhQUlRfTE9DSykuIEluIHRoYXQgcmFjZSBzY2VuYXJpbywg
cnBjX3VwZGF0ZV9ydHQoKQ0KPiA+IGNvdWxkIHVwZGF0ZSB0aGUgUlRUIHZhbHVlIGJlZm9yZSB3
ZSd2ZSB1cGRhdGVkIHRoZSByZXEtPnJxX250cmFucywNCj4gPiBtZWFuaW5nIHRoYXQgd2UgbWln
aHQgYmUgbWVhc3VyaW5nIHRoZSByZXNwb25zZSB0aW1lIGFnYWluc3QgYQ0KPiA+IHJlcGx5IHRv
DQo+ID4gdGhlIG9yaWdpbmFsIHRyYW5zbWlzc2lvbiBvciB0byB0aGUgcmV0cmFuc21pc3Npb24g
dGhhdCB3YXMganVzdA0KPiA+IHNlbnQuDQo+ID4gVGhhdCB3b3VsZCBlbmQgdXAgYnJlYWtpbmcg
VmFuIEphY29ic2VuIGNvbmdlc3Rpb24gY29udHJvbCwgc2luY2UNCj4gPiB3ZQ0KPiA+IHJpc2sg
c2lnbmlmaWNhbnRseSB1bmRlcmVzdGltYXRpbmcgdGhlIFJUVCB2YWx1ZS4NCj4gDQo+IEJ1dCBm
b3IgdHJhbnNwb3J0cyB0aGF0IGRvbid0IHVzZSB4cHJ0X3VwZGF0ZV9ydHQsIGVycm9yDQo+IHJl
dHVybnMgZnJvbSAtPnNlbmRfcmVxdWVzdCgpIChsaWtlIC1FQUdBSU4gb3IgLUVOT0JVRlMpDQo+
IG1lYW5zIHRoYXQgcnFfbnRyYW5zIGlzIGNvdW50ZWQgdHdpY2UsIGFuZCBpdCBzaG93cyB1cCBh
cw0KPiBmYWxzZSByZXRyYW5zbWlzc2lvbnMuDQoNClRoYXQncyB0aGUgd2F5IGl0IGhhcyBhbHdh
eXMgd29ya2VkLiBUaGUgVmFuIEphY29ic2VuIGNvZGUgd2FzDQppbXBsZW1lbnRlZCBsb25nIGJl
Zm9yZSB0aGUgcGVyLXJlcXVlc3Qgc3RhdHMgd2VyZSBpbnRyb2R1Y2VkLCBhbmQgc28NCnRoZSBi
ZWhhdmlvdXIgb2YgcmVxLT5ycV9udHJhbnMgaGFzIGFsd2F5cyBiZWVuIHRvIHVwZGF0ZSB0aGUg
dmFsdWUNCmJlZm9yZSB3ZSB0cmFuc21pdC4NCg0KPiBXaGF0IGlmIHJxX250cmFucyB3YXMgYnVt
cGVkIGluIHRoZSB0cmFuc3BvcnQgY29kZSBpbnN0ZWFkPw0KDQpJZiB3ZSBtdXN0IGNoYW5nZSB0
aGUgYmVoYXZpb3VyLCB0aGVuIHdoeSBub3QganVzdCBoYXZlIHRoZSBnZW5lcmljDQpsYXllciAn
dW5idW1wJyB0aGUgY291bnRlciBvbiBhbiBpbmNvbXBsZXRlIHRyYW5zbWlzc2lvbj8NCg0KLS0g
DQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lciwgSGFtbWVyc3Bh
Y2UNCnRyb25kLm15a2xlYnVzdEBoYW1tZXJzcGFjZS5jb20NCg0KDQo=

Re: [PATCH v2 23/34] SUNRPC: Move RPC retransmission stat counter to xprt_transmit()

[Chuck Lever]

> On Sep 5, 2018, at 12:07 PM, Trond Myklebust <trondmy@hammerspace.com> =
wrote:
>=20
> On Wed, 2018-09-05 at 11:31 -0400, Chuck Lever wrote:
>>> On Sep 5, 2018, at 11:28 AM, Trond Myklebust <
>>> trondmy@hammerspace.com> wrote:
>>>=20
>>> On Wed, 2018-09-05 at 10:30 -0400, Chuck Lever wrote:
>>>>> On Sep 4, 2018, at 5:05 PM, Trond Myklebust <trondmy@gmail.com>
>>>>> wrote:
>>>>>=20
>>>>> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com
>>>>>>=20
>>>>> ---
>>>>> net/sunrpc/clnt.c |  6 ------
>>>>> net/sunrpc/xprt.c | 13 ++++++-------
>>>>> 2 files changed, 6 insertions(+), 13 deletions(-)
>>>>>=20
>>>>> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
>>>>> index 032b7042adb6..21713bed812a 100644
>>>>> --- a/net/sunrpc/clnt.c
>>>>> +++ b/net/sunrpc/clnt.c
>>>>> @@ -1967,8 +1967,6 @@ call_connect_status(struct rpc_task
>>>>> *task)
>>>>> static void
>>>>> call_transmit(struct rpc_task *task)
>>>>> {
>>>>> -	int is_retrans =3D RPC_WAS_SENT(task);
>>>>> -
>>>>> 	dprint_status(task);
>>>>>=20
>>>>> 	task->tk_action =3D call_transmit_status;
>>>>> @@ -1979,10 +1977,6 @@ call_transmit(struct rpc_task *task)
>>>>> 	if (!xprt_prepare_transmit(task))
>>>>> 		return;
>>>>> 	xprt_transmit(task);
>>>>> -	if (task->tk_status < 0)
>>>>> -		return;
>>>>> -	if (is_retrans)
>>>>> -		task->tk_client->cl_stats->rpcretrans++;
>>>>> }
>>>>>=20
>>>>> /*
>>>>> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
>>>>> index f1301d391399..e2f5b4668a66 100644
>>>>> --- a/net/sunrpc/xprt.c
>>>>> +++ b/net/sunrpc/xprt.c
>>>>> @@ -191,8 +191,6 @@ int xprt_reserve_xprt(struct rpc_xprt
>>>>> *xprt,
>>>>> struct rpc_task *task)
>>>>> 		goto out_sleep;
>>>>> 	}
>>>>> 	xprt->snd_task =3D task;
>>>>> -	if (req !=3D NULL)
>>>>> -		req->rq_ntrans++;
>>>>>=20
>>>>> 	return 1;
>>>>>=20
>>>>> @@ -247,7 +245,6 @@ int xprt_reserve_xprt_cong(struct rpc_xprt
>>>>> *xprt, struct rpc_task *task)
>>>>> 	}
>>>>> 	if (__xprt_get_cong(xprt, task)) {
>>>>> 		xprt->snd_task =3D task;
>>>>> -		req->rq_ntrans++;
>>>>> 		return 1;
>>>>> 	}
>>>>> 	xprt_clear_locked(xprt);
>>>>> @@ -281,12 +278,8 @@ static inline int xprt_lock_write(struct
>>>>> rpc_xprt *xprt, struct rpc_task *task)
>>>>> static bool __xprt_lock_write_func(struct rpc_task *task, void
>>>>> *data)
>>>>> {
>>>>> 	struct rpc_xprt *xprt =3D data;
>>>>> -	struct rpc_rqst *req;
>>>>>=20
>>>>> -	req =3D task->tk_rqstp;
>>>>> 	xprt->snd_task =3D task;
>>>>> -	if (req)
>>>>> -		req->rq_ntrans++;
>>>>> 	return true;
>>>>> }
>>>>>=20
>>>>> @@ -1126,6 +1119,7 @@ void xprt_transmit(struct rpc_task *task)
>>>>> 	struct rpc_rqst	*req =3D task->tk_rqstp;
>>>>> 	struct rpc_xprt	*xprt =3D req->rq_xprt;
>>>>> 	unsigned int connect_cookie;
>>>>> +	int is_retrans =3D RPC_WAS_SENT(task);
>>>>> 	int status;
>>>>>=20
>>>>> 	dprintk("RPC: %5u xprt_transmit(%u)\n", task->tk_pid,
>>>>> req-
>>>>>> rq_slen);
>>>>>=20
>>>>> @@ -1140,6 +1134,8 @@ void xprt_transmit(struct rpc_task *task)
>>>>> 		}
>>>>> 	}
>>>>>=20
>>>>> +	req->rq_ntrans++;
>>>>> +
>>>>=20
>>>> rq_ntrans is used in two places:
>>>>=20
>>>> 1. rpc_update_rtt
>>>> 2. rpc_count_iostats_metrics
>>>>=20
>>>> Both of these appear to assume that rq_ntrans is counting
>>>> full successful transmissions (ie, calls that appear on the
>>>> wire), not invocations of ->send_request().
>>>>=20
>>>> Can this counter be moved down to where rpcretrans is updated?
>>>>=20
>>>=20
>>> The reason why I don't want to update req->rq_ntrans after
>>> transmission
>>> is because that would race with the reply from the server (which is
>>> not
>>> blocked by the XPRT_LOCK). In that race scenario, rpc_update_rtt()
>>> could update the RTT value before we've updated the req->rq_ntrans,
>>> meaning that we might be measuring the response time against a
>>> reply to
>>> the original transmission or to the retransmission that was just
>>> sent.
>>> That would end up breaking Van Jacobsen congestion control, since
>>> we
>>> risk significantly underestimating the RTT value.
>>=20
>> But for transports that don't use xprt_update_rtt, error
>> returns from ->send_request() (like -EAGAIN or -ENOBUFS)
>> means that rq_ntrans is counted twice, and it shows up as
>> false retransmissions.
>=20
> That's the way it has always worked.

Not quite: the current xprt_reserve_xprt_cong has a latch -- rq_cong.
req->rq_ntrans is bumped only once when it gets a congestion credit.

The transport send lock is held until enough ->send_request calls
have been done to transmit the whole Call message, but rq_ntrans
is bumped only once per full RPC Call message.

Note that this code is a little dodgy for xprt_reserve_xprt: when
driving a connect, the RPC task takes and releases the transport
send lock twice, and both times it bumps rq_ntrans, which is a bug.
This patch appears to fix that.


> The Van Jacobsen code was
> implemented long before the per-request stats were introduced, and so
> the behaviour of req->rq_ntrans has always been to update the value
> before we transmit.

>> What if rq_ntrans was bumped in the transport code instead?
>=20
> If we must change the behaviour, then why not just have the generic
> layer 'unbump' the counter on an incomplete transmission?

OK. There's no way for a reply to arrive if the call hasn't been
fully transmitted, so a race is avoided in that case.

--
Chuck Lever

Re: [PATCH v2 29/34] SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue

[Schumaker, Anna]

SGkgVHJvbmQsDQoNCk9uIFR1ZSwgMjAxOC0wOS0wNCBhdCAxNzowNSAtMDQwMCwgVHJvbmQgTXlr
bGVidXN0IHdyb3RlOg0KPiBXZSBubyBsb25nZXIgbmVlZCBwcmlvcml0eSBzZW1hbnRpY3Mgb24g
dGhlIHhwcnQtPnNlbmRpbmcgcXVldWUsIGJlY2F1c2UNCj4gdGhlIG9yZGVyIGluIHdoaWNoIHRh
c2tzIGFyZSBzZW50IGlzIG5vdyBkaWN0YXRlZCBieSB0aGVpciBwb3NpdGlvbiBpbg0KPiB0aGUg
c2VuZCBxdWV1ZS4NCj4gTm90ZSB0aGF0IHRoZSBiYWNrbG9nIHF1ZXVlIHJlbWFpbnMgYSBwcmlv
cml0eSBxdWV1ZSwgbWVhbmluZyB0aGF0DQo+IHNsb3QgcmVzb3VyY2VzIGFyZSBzdGlsbCBtYW5h
Z2VkIGluIG9yZGVyIG9mIHRhc2sgcHJpb3JpdHkuDQo+IA0KPiBTaWduZWQtb2ZmLWJ5OiBUcm9u
ZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBoYW1tZXJzcGFjZS5jb20+DQo+IC0tLQ0KPiAg
bmV0L3N1bnJwYy94cHJ0LmMgfCAyOSArKysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAg
MSBmaWxlIGNoYW5nZWQsIDYgaW5zZXJ0aW9ucygrKSwgMjMgZGVsZXRpb25zKC0pDQo+IA0KPiBk
aWZmIC0tZ2l0IGEvbmV0L3N1bnJwYy94cHJ0LmMgYi9uZXQvc3VucnBjL3hwcnQuYw0KPiBpbmRl
eCBiYTlhZjI1ZDE0ZGUuLmViZTUyMzVjNDg0YiAxMDA2NDQNCj4gLS0tIGEvbmV0L3N1bnJwYy94
cHJ0LmMNCj4gKysrIGIvbmV0L3N1bnJwYy94cHJ0LmMNCj4gQEAgLTE4MSw3ICsxODEsNiBAQCBF
WFBPUlRfU1lNQk9MX0dQTCh4cHJ0X2xvYWRfdHJhbnNwb3J0KTsNCj4gIGludCB4cHJ0X3Jlc2Vy
dmVfeHBydChzdHJ1Y3QgcnBjX3hwcnQgKnhwcnQsIHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4g
IHsNCj4gIAlzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSA9IHRhc2stPnRrX3Jxc3RwOw0KDQpBcyBvZiB0
aGlzIHBhdGNoLCByZXEgaXMgbm8gbG9uZ2VyIHVzZWQgZWl0aGVyLiAgQ2FuIGl0IGJlIHJlbW92
ZWQ/DQoNClRoYW5rcywNCkFubmENCg0KPiAtCWludCBwcmlvcml0eTsNCj4gIA0KPiAgCWlmICh0
ZXN0X2FuZF9zZXRfYml0KFhQUlRfTE9DS0VELCAmeHBydC0+c3RhdGUpKSB7DQo+ICAJCWlmICh0
YXNrID09IHhwcnQtPnNuZF90YXNrKQ0KPiBAQCAtMTk3LDEzICsxOTYsNyBAQCBpbnQgeHBydF9y
ZXNlcnZlX3hwcnQoc3RydWN0IHJwY194cHJ0ICp4cHJ0LCBzdHJ1Y3QNCj4gcnBjX3Rhc2sgKnRh
c2spDQo+ICAJCQl0YXNrLT50a19waWQsIHhwcnQpOw0KPiAgCXRhc2stPnRrX3RpbWVvdXQgPSAw
Ow0KPiAgCXRhc2stPnRrX3N0YXR1cyA9IC1FQUdBSU47DQo+IC0JaWYgKHJlcSA9PSBOVUxMKQ0K
PiAtCQlwcmlvcml0eSA9IFJQQ19QUklPUklUWV9MT1c7DQo+IC0JZWxzZSBpZiAoIXJlcS0+cnFf
bnRyYW5zKQ0KPiAtCQlwcmlvcml0eSA9IFJQQ19QUklPUklUWV9OT1JNQUw7DQo+IC0JZWxzZQ0K
PiAtCQlwcmlvcml0eSA9IFJQQ19QUklPUklUWV9ISUdIOw0KPiAtCXJwY19zbGVlcF9vbl9wcmlv
cml0eSgmeHBydC0+c2VuZGluZywgdGFzaywgTlVMTCwgcHJpb3JpdHkpOw0KPiArCXJwY19zbGVl
cF9vbigmeHBydC0+c2VuZGluZywgdGFzaywgTlVMTCk7DQo+ICAJcmV0dXJuIDA7DQo+ICB9DQo+
ICBFWFBPUlRfU1lNQk9MX0dQTCh4cHJ0X3Jlc2VydmVfeHBydCk7DQo+IEBAIC0yMzEsNyArMjI0
LDYgQEAgc3RhdGljIHZvaWQgeHBydF9jbGVhcl9sb2NrZWQoc3RydWN0IHJwY194cHJ0ICp4cHJ0
KQ0KPiAgaW50IHhwcnRfcmVzZXJ2ZV94cHJ0X2Nvbmcoc3RydWN0IHJwY194cHJ0ICp4cHJ0LCBz
dHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICB7DQo+ICAJc3RydWN0IHJwY19ycXN0ICpyZXEgPSB0
YXNrLT50a19ycXN0cDsNCj4gLQlpbnQgcHJpb3JpdHk7DQo+ICANCj4gIAlpZiAodGVzdF9hbmRf
c2V0X2JpdChYUFJUX0xPQ0tFRCwgJnhwcnQtPnN0YXRlKSkgew0KPiAgCQlpZiAodGFzayA9PSB4
cHJ0LT5zbmRfdGFzaykNCj4gQEAgLTI1MSwxMyArMjQzLDcgQEAgaW50IHhwcnRfcmVzZXJ2ZV94
cHJ0X2Nvbmcoc3RydWN0IHJwY194cHJ0ICp4cHJ0LCBzdHJ1Y3QNCj4gcnBjX3Rhc2sgKnRhc2sp
DQo+ICAJZHByaW50aygiUlBDOiAlNXUgZmFpbGVkIHRvIGxvY2sgdHJhbnNwb3J0ICVwXG4iLCB0
YXNrLT50a19waWQsIHhwcnQpOw0KPiAgCXRhc2stPnRrX3RpbWVvdXQgPSAwOw0KPiAgCXRhc2st
PnRrX3N0YXR1cyA9IC1FQUdBSU47DQo+IC0JaWYgKHJlcSA9PSBOVUxMKQ0KPiAtCQlwcmlvcml0
eSA9IFJQQ19QUklPUklUWV9MT1c7DQo+IC0JZWxzZSBpZiAoIXJlcS0+cnFfbnRyYW5zKQ0KPiAt
CQlwcmlvcml0eSA9IFJQQ19QUklPUklUWV9OT1JNQUw7DQo+IC0JZWxzZQ0KPiAtCQlwcmlvcml0
eSA9IFJQQ19QUklPUklUWV9ISUdIOw0KPiAtCXJwY19zbGVlcF9vbl9wcmlvcml0eSgmeHBydC0+
c2VuZGluZywgdGFzaywgTlVMTCwgcHJpb3JpdHkpOw0KPiArCXJwY19zbGVlcF9vbigmeHBydC0+
c2VuZGluZywgdGFzaywgTlVMTCk7DQo+ICAJcmV0dXJuIDA7DQo+ICB9DQo+ICBFWFBPUlRfU1lN
Qk9MX0dQTCh4cHJ0X3Jlc2VydmVfeHBydF9jb25nKTsNCj4gQEAgLTI4NSw4ICsyNzEsNyBAQCBz
dGF0aWMgdm9pZCBfX3hwcnRfbG9ja193cml0ZV9uZXh0KHN0cnVjdCBycGNfeHBydCAqeHBydCkN
Cj4gIAlpZiAodGVzdF9hbmRfc2V0X2JpdChYUFJUX0xPQ0tFRCwgJnhwcnQtPnN0YXRlKSkNCj4g
IAkJcmV0dXJuOw0KPiAgDQo+IC0JaWYgKHJwY193YWtlX3VwX2ZpcnN0X29uX3dxKHhwcnRpb2Rf
d29ya3F1ZXVlLCAmeHBydC0+c2VuZGluZywNCj4gLQkJCQlfX3hwcnRfbG9ja193cml0ZV9mdW5j
LCB4cHJ0KSkNCj4gKwlpZiAocnBjX3dha2VfdXBfZmlyc3QoJnhwcnQtPnNlbmRpbmcsIF9feHBy
dF9sb2NrX3dyaXRlX2Z1bmMsIHhwcnQpKQ0KPiAgCQlyZXR1cm47DQo+ICAJeHBydF9jbGVhcl9s
b2NrZWQoeHBydCk7DQo+ICB9DQo+IEBAIC0yOTcsOCArMjgyLDcgQEAgc3RhdGljIHZvaWQgX194
cHJ0X2xvY2tfd3JpdGVfbmV4dF9jb25nKHN0cnVjdCBycGNfeHBydA0KPiAqeHBydCkNCj4gIAkJ
cmV0dXJuOw0KPiAgCWlmIChSUENYUFJUX0NPTkdFU1RFRCh4cHJ0KSkNCj4gIAkJZ290byBvdXRf
dW5sb2NrOw0KPiAtCWlmIChycGNfd2FrZV91cF9maXJzdF9vbl93cSh4cHJ0aW9kX3dvcmtxdWV1
ZSwgJnhwcnQtPnNlbmRpbmcsDQo+IC0JCQkJX194cHJ0X2xvY2tfd3JpdGVfZnVuYywgeHBydCkp
DQo+ICsJaWYgKHJwY193YWtlX3VwX2ZpcnN0KCZ4cHJ0LT5zZW5kaW5nLCBfX3hwcnRfbG9ja193
cml0ZV9mdW5jLCB4cHJ0KSkNCj4gIAkJcmV0dXJuOw0KPiAgb3V0X3VubG9jazoNCj4gIAl4cHJ0
X2NsZWFyX2xvY2tlZCh4cHJ0KTsNCj4gQEAgLTQ5NSw4ICs0NzksNyBAQCB2b2lkIHhwcnRfd3Jp
dGVfc3BhY2Uoc3RydWN0IHJwY194cHJ0ICp4cHJ0KQ0KPiAgCWlmICh4cHJ0LT5zbmRfdGFzaykg
ew0KPiAgCQlkcHJpbnRrKCJSUEM6ICAgICAgIHdyaXRlIHNwYWNlOiB3YWtpbmcgd2FpdGluZyB0
YXNrIG9uICINCj4gIAkJCQkieHBydCAlcFxuIiwgeHBydCk7DQo+IC0JCXJwY193YWtlX3VwX3F1
ZXVlZF90YXNrX29uX3dxKHhwcnRpb2Rfd29ya3F1ZXVlLA0KPiAtCQkJCSZ4cHJ0LT5wZW5kaW5n
LCB4cHJ0LT5zbmRfdGFzayk7DQo+ICsJCXJwY193YWtlX3VwX3F1ZXVlZF90YXNrKCZ4cHJ0LT5w
ZW5kaW5nLCB4cHJ0LT5zbmRfdGFzayk7DQo+ICAJfQ0KPiAgCXNwaW5fdW5sb2NrX2JoKCZ4cHJ0
LT50cmFuc3BvcnRfbG9jayk7DQo+ICB9DQo+IEBAIC0xNTk2LDcgKzE1NzksNyBAQCBzdGF0aWMg
dm9pZCB4cHJ0X2luaXQoc3RydWN0IHJwY194cHJ0ICp4cHJ0LCBzdHJ1Y3QgbmV0DQo+ICpuZXQp
DQo+ICANCj4gIAlycGNfaW5pdF93YWl0X3F1ZXVlKCZ4cHJ0LT5iaW5kaW5nLCAieHBydF9iaW5k
aW5nIik7DQo+ICAJcnBjX2luaXRfd2FpdF9xdWV1ZSgmeHBydC0+cGVuZGluZywgInhwcnRfcGVu
ZGluZyIpOw0KPiAtCXJwY19pbml0X3ByaW9yaXR5X3dhaXRfcXVldWUoJnhwcnQtPnNlbmRpbmcs
ICJ4cHJ0X3NlbmRpbmciKTsNCj4gKwlycGNfaW5pdF93YWl0X3F1ZXVlKCZ4cHJ0LT5zZW5kaW5n
LCAieHBydF9zZW5kaW5nIik7DQo+ICAJcnBjX2luaXRfcHJpb3JpdHlfd2FpdF9xdWV1ZSgmeHBy
dC0+YmFja2xvZywgInhwcnRfYmFja2xvZyIpOw0KPiAgDQo+ICAJeHBydF9pbml0X3hpZCh4cHJ0
KTsNCg==

Re: [PATCH v2 29/34] SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue

[Schumaker, Anna]

T24gVGh1LCAyMDE4LTA5LTA2IGF0IDEwOjE3IC0wNDAwLCBBbm5hIFNjaHVtYWtlciB3cm90ZToN
Cj4gSGkgVHJvbmQsDQo+IA0KPiBPbiBUdWUsIDIwMTgtMDktMDQgYXQgMTc6MDUgLTA0MDAsIFRy
b25kIE15a2xlYnVzdCB3cm90ZToNCj4gPiBXZSBubyBsb25nZXIgbmVlZCBwcmlvcml0eSBzZW1h
bnRpY3Mgb24gdGhlIHhwcnQtPnNlbmRpbmcgcXVldWUsIGJlY2F1c2UNCj4gPiB0aGUgb3JkZXIg
aW4gd2hpY2ggdGFza3MgYXJlIHNlbnQgaXMgbm93IGRpY3RhdGVkIGJ5IHRoZWlyIHBvc2l0aW9u
IGluDQo+ID4gdGhlIHNlbmQgcXVldWUuDQo+ID4gTm90ZSB0aGF0IHRoZSBiYWNrbG9nIHF1ZXVl
IHJlbWFpbnMgYSBwcmlvcml0eSBxdWV1ZSwgbWVhbmluZyB0aGF0DQo+ID4gc2xvdCByZXNvdXJj
ZXMgYXJlIHN0aWxsIG1hbmFnZWQgaW4gb3JkZXIgb2YgdGFzayBwcmlvcml0eS4NCj4gPiANCj4g
PiBTaWduZWQtb2ZmLWJ5OiBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBoYW1tZXJz
cGFjZS5jb20+DQo+ID4gLS0tDQo+ID4gIG5ldC9zdW5ycGMveHBydC5jIHwgMjkgKysrKysrLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiAgMSBmaWxlIGNoYW5nZWQsIDYgaW5zZXJ0aW9ucygr
KSwgMjMgZGVsZXRpb25zKC0pDQo+ID4gDQo+ID4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMveHBy
dC5jIGIvbmV0L3N1bnJwYy94cHJ0LmMNCj4gPiBpbmRleCBiYTlhZjI1ZDE0ZGUuLmViZTUyMzVj
NDg0YiAxMDA2NDQNCj4gPiAtLS0gYS9uZXQvc3VucnBjL3hwcnQuYw0KPiA+ICsrKyBiL25ldC9z
dW5ycGMveHBydC5jDQo+ID4gQEAgLTE4MSw3ICsxODEsNiBAQCBFWFBPUlRfU1lNQk9MX0dQTCh4
cHJ0X2xvYWRfdHJhbnNwb3J0KTsNCj4gPiAgaW50IHhwcnRfcmVzZXJ2ZV94cHJ0KHN0cnVjdCBy
cGNfeHBydCAqeHBydCwgc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ICB7DQo+ID4gIAlzdHJ1
Y3QgcnBjX3Jxc3QgKnJlcSA9IHRhc2stPnRrX3Jxc3RwOw0KPiANCj4gQXMgb2YgdGhpcyBwYXRj
aCwgcmVxIGlzIG5vIGxvbmdlciB1c2VkIGVpdGhlci4gIENhbiBpdCBiZSByZW1vdmVkPw0KDQpO
ZXZlciBtaW5kLCBJIGxvb2tlZCBhdCB0aGUgbmV4dCBwYXRjaCBhbmQgc2F3IHRoYXQgaXQncyB1
c2VkIGFnYWluLg0KDQo+IA0KPiBUaGFua3MsDQo+IEFubmENCj4gDQo+ID4gLQlpbnQgcHJpb3Jp
dHk7DQo+ID4gIA0KPiA+ICAJaWYgKHRlc3RfYW5kX3NldF9iaXQoWFBSVF9MT0NLRUQsICZ4cHJ0
LT5zdGF0ZSkpIHsNCj4gPiAgCQlpZiAodGFzayA9PSB4cHJ0LT5zbmRfdGFzaykNCj4gPiBAQCAt
MTk3LDEzICsxOTYsNyBAQCBpbnQgeHBydF9yZXNlcnZlX3hwcnQoc3RydWN0IHJwY194cHJ0ICp4
cHJ0LCBzdHJ1Y3QNCj4gPiBycGNfdGFzayAqdGFzaykNCj4gPiAgCQkJdGFzay0+dGtfcGlkLCB4
cHJ0KTsNCj4gPiAgCXRhc2stPnRrX3RpbWVvdXQgPSAwOw0KPiA+ICAJdGFzay0+dGtfc3RhdHVz
ID0gLUVBR0FJTjsNCj4gPiAtCWlmIChyZXEgPT0gTlVMTCkNCj4gPiAtCQlwcmlvcml0eSA9IFJQ
Q19QUklPUklUWV9MT1c7DQo+ID4gLQllbHNlIGlmICghcmVxLT5ycV9udHJhbnMpDQo+ID4gLQkJ
cHJpb3JpdHkgPSBSUENfUFJJT1JJVFlfTk9STUFMOw0KPiA+IC0JZWxzZQ0KPiA+IC0JCXByaW9y
aXR5ID0gUlBDX1BSSU9SSVRZX0hJR0g7DQo+ID4gLQlycGNfc2xlZXBfb25fcHJpb3JpdHkoJnhw
cnQtPnNlbmRpbmcsIHRhc2ssIE5VTEwsIHByaW9yaXR5KTsNCj4gPiArCXJwY19zbGVlcF9vbigm
eHBydC0+c2VuZGluZywgdGFzaywgTlVMTCk7DQo+ID4gIAlyZXR1cm4gMDsNCj4gPiAgfQ0KPiA+
ICBFWFBPUlRfU1lNQk9MX0dQTCh4cHJ0X3Jlc2VydmVfeHBydCk7DQo+ID4gQEAgLTIzMSw3ICsy
MjQsNiBAQCBzdGF0aWMgdm9pZCB4cHJ0X2NsZWFyX2xvY2tlZChzdHJ1Y3QgcnBjX3hwcnQgKnhw
cnQpDQo+ID4gIGludCB4cHJ0X3Jlc2VydmVfeHBydF9jb25nKHN0cnVjdCBycGNfeHBydCAqeHBy
dCwgc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ICB7DQo+ID4gIAlzdHJ1Y3QgcnBjX3Jxc3Qg
KnJlcSA9IHRhc2stPnRrX3Jxc3RwOw0KPiA+IC0JaW50IHByaW9yaXR5Ow0KPiA+ICANCj4gPiAg
CWlmICh0ZXN0X2FuZF9zZXRfYml0KFhQUlRfTE9DS0VELCAmeHBydC0+c3RhdGUpKSB7DQo+ID4g
IAkJaWYgKHRhc2sgPT0geHBydC0+c25kX3Rhc2spDQo+ID4gQEAgLTI1MSwxMyArMjQzLDcgQEAg
aW50IHhwcnRfcmVzZXJ2ZV94cHJ0X2Nvbmcoc3RydWN0IHJwY194cHJ0ICp4cHJ0LA0KPiA+IHN0
cnVjdA0KPiA+IHJwY190YXNrICp0YXNrKQ0KPiA+ICAJZHByaW50aygiUlBDOiAlNXUgZmFpbGVk
IHRvIGxvY2sgdHJhbnNwb3J0ICVwXG4iLCB0YXNrLT50a19waWQsIHhwcnQpOw0KPiA+ICAJdGFz
ay0+dGtfdGltZW91dCA9IDA7DQo+ID4gIAl0YXNrLT50a19zdGF0dXMgPSAtRUFHQUlOOw0KPiA+
IC0JaWYgKHJlcSA9PSBOVUxMKQ0KPiA+IC0JCXByaW9yaXR5ID0gUlBDX1BSSU9SSVRZX0xPVzsN
Cj4gPiAtCWVsc2UgaWYgKCFyZXEtPnJxX250cmFucykNCj4gPiAtCQlwcmlvcml0eSA9IFJQQ19Q
UklPUklUWV9OT1JNQUw7DQo+ID4gLQllbHNlDQo+ID4gLQkJcHJpb3JpdHkgPSBSUENfUFJJT1JJ
VFlfSElHSDsNCj4gPiAtCXJwY19zbGVlcF9vbl9wcmlvcml0eSgmeHBydC0+c2VuZGluZywgdGFz
aywgTlVMTCwgcHJpb3JpdHkpOw0KPiA+ICsJcnBjX3NsZWVwX29uKCZ4cHJ0LT5zZW5kaW5nLCB0
YXNrLCBOVUxMKTsNCj4gPiAgCXJldHVybiAwOw0KPiA+ICB9DQo+ID4gIEVYUE9SVF9TWU1CT0xf
R1BMKHhwcnRfcmVzZXJ2ZV94cHJ0X2NvbmcpOw0KPiA+IEBAIC0yODUsOCArMjcxLDcgQEAgc3Rh
dGljIHZvaWQgX194cHJ0X2xvY2tfd3JpdGVfbmV4dChzdHJ1Y3QgcnBjX3hwcnQNCj4gPiAqeHBy
dCkNCj4gPiAgCWlmICh0ZXN0X2FuZF9zZXRfYml0KFhQUlRfTE9DS0VELCAmeHBydC0+c3RhdGUp
KQ0KPiA+ICAJCXJldHVybjsNCj4gPiAgDQo+ID4gLQlpZiAocnBjX3dha2VfdXBfZmlyc3Rfb25f
d3EoeHBydGlvZF93b3JrcXVldWUsICZ4cHJ0LT5zZW5kaW5nLA0KPiA+IC0JCQkJX194cHJ0X2xv
Y2tfd3JpdGVfZnVuYywgeHBydCkpDQo+ID4gKwlpZiAocnBjX3dha2VfdXBfZmlyc3QoJnhwcnQt
PnNlbmRpbmcsIF9feHBydF9sb2NrX3dyaXRlX2Z1bmMsIHhwcnQpKQ0KPiA+ICAJCXJldHVybjsN
Cj4gPiAgCXhwcnRfY2xlYXJfbG9ja2VkKHhwcnQpOw0KPiA+ICB9DQo+ID4gQEAgLTI5Nyw4ICsy
ODIsNyBAQCBzdGF0aWMgdm9pZCBfX3hwcnRfbG9ja193cml0ZV9uZXh0X2Nvbmcoc3RydWN0IHJw
Y194cHJ0DQo+ID4gKnhwcnQpDQo+ID4gIAkJcmV0dXJuOw0KPiA+ICAJaWYgKFJQQ1hQUlRfQ09O
R0VTVEVEKHhwcnQpKQ0KPiA+ICAJCWdvdG8gb3V0X3VubG9jazsNCj4gPiAtCWlmIChycGNfd2Fr
ZV91cF9maXJzdF9vbl93cSh4cHJ0aW9kX3dvcmtxdWV1ZSwgJnhwcnQtPnNlbmRpbmcsDQo+ID4g
LQkJCQlfX3hwcnRfbG9ja193cml0ZV9mdW5jLCB4cHJ0KSkNCj4gPiArCWlmIChycGNfd2FrZV91
cF9maXJzdCgmeHBydC0+c2VuZGluZywgX194cHJ0X2xvY2tfd3JpdGVfZnVuYywgeHBydCkpDQo+
ID4gIAkJcmV0dXJuOw0KPiA+ICBvdXRfdW5sb2NrOg0KPiA+ICAJeHBydF9jbGVhcl9sb2NrZWQo
eHBydCk7DQo+ID4gQEAgLTQ5NSw4ICs0NzksNyBAQCB2b2lkIHhwcnRfd3JpdGVfc3BhY2Uoc3Ry
dWN0IHJwY194cHJ0ICp4cHJ0KQ0KPiA+ICAJaWYgKHhwcnQtPnNuZF90YXNrKSB7DQo+ID4gIAkJ
ZHByaW50aygiUlBDOiAgICAgICB3cml0ZSBzcGFjZTogd2FraW5nIHdhaXRpbmcgdGFzayBvbiAi
DQo+ID4gIAkJCQkieHBydCAlcFxuIiwgeHBydCk7DQo+ID4gLQkJcnBjX3dha2VfdXBfcXVldWVk
X3Rhc2tfb25fd3EoeHBydGlvZF93b3JrcXVldWUsDQo+ID4gLQkJCQkmeHBydC0+cGVuZGluZywg
eHBydC0+c25kX3Rhc2spOw0KPiA+ICsJCXJwY193YWtlX3VwX3F1ZXVlZF90YXNrKCZ4cHJ0LT5w
ZW5kaW5nLCB4cHJ0LT5zbmRfdGFzayk7DQo+ID4gIAl9DQo+ID4gIAlzcGluX3VubG9ja19iaCgm
eHBydC0+dHJhbnNwb3J0X2xvY2spOw0KPiA+ICB9DQo+ID4gQEAgLTE1OTYsNyArMTU3OSw3IEBA
IHN0YXRpYyB2b2lkIHhwcnRfaW5pdChzdHJ1Y3QgcnBjX3hwcnQgKnhwcnQsIHN0cnVjdA0KPiA+
IG5ldA0KPiA+ICpuZXQpDQo+ID4gIA0KPiA+ICAJcnBjX2luaXRfd2FpdF9xdWV1ZSgmeHBydC0+
YmluZGluZywgInhwcnRfYmluZGluZyIpOw0KPiA+ICAJcnBjX2luaXRfd2FpdF9xdWV1ZSgmeHBy
dC0+cGVuZGluZywgInhwcnRfcGVuZGluZyIpOw0KPiA+IC0JcnBjX2luaXRfcHJpb3JpdHlfd2Fp
dF9xdWV1ZSgmeHBydC0+c2VuZGluZywgInhwcnRfc2VuZGluZyIpOw0KPiA+ICsJcnBjX2luaXRf
d2FpdF9xdWV1ZSgmeHBydC0+c2VuZGluZywgInhwcnRfc2VuZGluZyIpOw0KPiA+ICAJcnBjX2lu
aXRfcHJpb3JpdHlfd2FpdF9xdWV1ZSgmeHBydC0+YmFja2xvZywgInhwcnRfYmFja2xvZyIpOw0K
PiA+ICANCj4gPiAgCXhwcnRfaW5pdF94aWQoeHBydCk7DQo=

Re: [PATCH v2 29/34] SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue

[Schumaker, Anna]

T24gVGh1LCAyMDE4LTA5LTA2IGF0IDEwOjE3IC0wNDAwLCBBbm5hIFNjaHVtYWtlciB3cm90ZToN
Cj4gSGkgVHJvbmQsDQo+IA0KPiBPbiBUdWUsIDIwMTgtMDktMDQgYXQgMTc6MDUgLTA0MDAsIFRy
b25kIE15a2xlYnVzdCB3cm90ZToNCj4gPiBXZSBubyBsb25nZXIgbmVlZCBwcmlvcml0eSBzZW1h
bnRpY3Mgb24gdGhlIHhwcnQtPnNlbmRpbmcgcXVldWUsIGJlY2F1c2UNCj4gPiB0aGUgb3JkZXIg
aW4gd2hpY2ggdGFza3MgYXJlIHNlbnQgaXMgbm93IGRpY3RhdGVkIGJ5IHRoZWlyIHBvc2l0aW9u
IGluDQo+ID4gdGhlIHNlbmQgcXVldWUuDQo+ID4gTm90ZSB0aGF0IHRoZSBiYWNrbG9nIHF1ZXVl
IHJlbWFpbnMgYSBwcmlvcml0eSBxdWV1ZSwgbWVhbmluZyB0aGF0DQo+ID4gc2xvdCByZXNvdXJj
ZXMgYXJlIHN0aWxsIG1hbmFnZWQgaW4gb3JkZXIgb2YgdGFzayBwcmlvcml0eS4NCj4gPiANCj4g
PiBTaWduZWQtb2ZmLWJ5OiBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBoYW1tZXJz
cGFjZS5jb20+DQo+ID4gLS0tDQo+ID4gIG5ldC9zdW5ycGMveHBydC5jIHwgMjkgKysrKysrLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiAgMSBmaWxlIGNoYW5nZWQsIDYgaW5zZXJ0aW9ucygr
KSwgMjMgZGVsZXRpb25zKC0pDQo+ID4gDQo+ID4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMveHBy
dC5jIGIvbmV0L3N1bnJwYy94cHJ0LmMNCj4gPiBpbmRleCBiYTlhZjI1ZDE0ZGUuLmViZTUyMzVj
NDg0YiAxMDA2NDQNCj4gPiAtLS0gYS9uZXQvc3VucnBjL3hwcnQuYw0KPiA+ICsrKyBiL25ldC9z
dW5ycGMveHBydC5jDQo+ID4gQEAgLTE4MSw3ICsxODEsNiBAQCBFWFBPUlRfU1lNQk9MX0dQTCh4
cHJ0X2xvYWRfdHJhbnNwb3J0KTsNCj4gPiAgaW50IHhwcnRfcmVzZXJ2ZV94cHJ0KHN0cnVjdCBy
cGNfeHBydCAqeHBydCwgc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ICB7DQo+ID4gIAlzdHJ1
Y3QgcnBjX3Jxc3QgKnJlcSA9IHRhc2stPnRrX3Jxc3RwOw0KPiANCj4gQXMgb2YgdGhpcyBwYXRj
aCwgcmVxIGlzIG5vIGxvbmdlciB1c2VkIGVpdGhlci4gIENhbiBpdCBiZSByZW1vdmVkPw0KDQpO
ZXZlciBtaW5kLCBJIGxvb2tlZCBhdCB0aGUgbmV4dCBwYXRjaCBhbmQgc2F3IHRoYXQgaXQncyB1
c2VkIGFnYWluLg0KDQo+IA0KPiBUaGFua3MsDQo+IEFubmENCj4gDQo+ID4gLQlpbnQgcHJpb3Jp
dHk7DQo+ID4gIA0KPiA+ICAJaWYgKHRlc3RfYW5kX3NldF9iaXQoWFBSVF9MT0NLRUQsICZ4cHJ0
LT5zdGF0ZSkpIHsNCj4gPiAgCQlpZiAodGFzayA9PSB4cHJ0LT5zbmRfdGFzaykNCj4gPiBAQCAt
MTk3LDEzICsxOTYsNyBAQCBpbnQgeHBydF9yZXNlcnZlX3hwcnQoc3RydWN0IHJwY194cHJ0ICp4
cHJ0LCBzdHJ1Y3QNCj4gPiBycGNfdGFzayAqdGFzaykNCj4gPiAgCQkJdGFzay0+dGtfcGlkLCB4
cHJ0KTsNCj4gPiAgCXRhc2stPnRrX3RpbWVvdXQgPSAwOw0KPiA+ICAJdGFzay0+dGtfc3RhdHVz
ID0gLUVBR0FJTjsNCj4gPiAtCWlmIChyZXEgPT0gTlVMTCkNCj4gPiAtCQlwcmlvcml0eSA9IFJQ
Q19QUklPUklUWV9MT1c7DQo+ID4gLQllbHNlIGlmICghcmVxLT5ycV9udHJhbnMpDQo+ID4gLQkJ
cHJpb3JpdHkgPSBSUENfUFJJT1JJVFlfTk9STUFMOw0KPiA+IC0JZWxzZQ0KPiA+IC0JCXByaW9y
aXR5ID0gUlBDX1BSSU9SSVRZX0hJR0g7DQo+ID4gLQlycGNfc2xlZXBfb25fcHJpb3JpdHkoJnhw
cnQtPnNlbmRpbmcsIHRhc2ssIE5VTEwsIHByaW9yaXR5KTsNCj4gPiArCXJwY19zbGVlcF9vbigm
eHBydC0+c2VuZGluZywgdGFzaywgTlVMTCk7DQo+ID4gIAlyZXR1cm4gMDsNCj4gPiAgfQ0KPiA+
ICBFWFBPUlRfU1lNQk9MX0dQTCh4cHJ0X3Jlc2VydmVfeHBydCk7DQo+ID4gQEAgLTIzMSw3ICsy
MjQsNiBAQCBzdGF0aWMgdm9pZCB4cHJ0X2NsZWFyX2xvY2tlZChzdHJ1Y3QgcnBjX3hwcnQgKnhw
cnQpDQo+ID4gIGludCB4cHJ0X3Jlc2VydmVfeHBydF9jb25nKHN0cnVjdCBycGNfeHBydCAqeHBy
dCwgc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiA+ICB7DQo+ID4gIAlzdHJ1Y3QgcnBjX3Jxc3Qg
KnJlcSA9IHRhc2stPnRrX3Jxc3RwOw0KPiA+IC0JaW50IHByaW9yaXR5Ow0KPiA+ICANCj4gPiAg
CWlmICh0ZXN0X2FuZF9zZXRfYml0KFhQUlRfTE9DS0VELCAmeHBydC0+c3RhdGUpKSB7DQo+ID4g
IAkJaWYgKHRhc2sgPT0geHBydC0+c25kX3Rhc2spDQo+ID4gQEAgLTI1MSwxMyArMjQzLDcgQEAg
aW50IHhwcnRfcmVzZXJ2ZV94cHJ0X2Nvbmcoc3RydWN0IHJwY194cHJ0ICp4cHJ0LA0KPiA+IHN0
cnVjdA0KPiA+IHJwY190YXNrICp0YXNrKQ0KPiA+ICAJZHByaW50aygiUlBDOiAlNXUgZmFpbGVk
IHRvIGxvY2sgdHJhbnNwb3J0ICVwXG4iLCB0YXNrLT50a19waWQsIHhwcnQpOw0KPiA+ICAJdGFz
ay0+dGtfdGltZW91dCA9IDA7DQo+ID4gIAl0YXNrLT50a19zdGF0dXMgPSAtRUFHQUlOOw0KPiA+
IC0JaWYgKHJlcSA9PSBOVUxMKQ0KPiA+IC0JCXByaW9yaXR5ID0gUlBDX1BSSU9SSVRZX0xPVzsN
Cj4gPiAtCWVsc2UgaWYgKCFyZXEtPnJxX250cmFucykNCj4gPiAtCQlwcmlvcml0eSA9IFJQQ19Q
UklPUklUWV9OT1JNQUw7DQo+ID4gLQllbHNlDQo+ID4gLQkJcHJpb3JpdHkgPSBSUENfUFJJT1JJ
VFlfSElHSDsNCj4gPiAtCXJwY19zbGVlcF9vbl9wcmlvcml0eSgmeHBydC0+c2VuZGluZywgdGFz
aywgTlVMTCwgcHJpb3JpdHkpOw0KPiA+ICsJcnBjX3NsZWVwX29uKCZ4cHJ0LT5zZW5kaW5nLCB0
YXNrLCBOVUxMKTsNCj4gPiAgCXJldHVybiAwOw0KPiA+ICB9DQo+ID4gIEVYUE9SVF9TWU1CT0xf
R1BMKHhwcnRfcmVzZXJ2ZV94cHJ0X2NvbmcpOw0KPiA+IEBAIC0yODUsOCArMjcxLDcgQEAgc3Rh
dGljIHZvaWQgX194cHJ0X2xvY2tfd3JpdGVfbmV4dChzdHJ1Y3QgcnBjX3hwcnQNCj4gPiAqeHBy
dCkNCj4gPiAgCWlmICh0ZXN0X2FuZF9zZXRfYml0KFhQUlRfTE9DS0VELCAmeHBydC0+c3RhdGUp
KQ0KPiA+ICAJCXJldHVybjsNCj4gPiAgDQo+ID4gLQlpZiAocnBjX3dha2VfdXBfZmlyc3Rfb25f
d3EoeHBydGlvZF93b3JrcXVldWUsICZ4cHJ0LT5zZW5kaW5nLA0KPiA+IC0JCQkJX194cHJ0X2xv
Y2tfd3JpdGVfZnVuYywgeHBydCkpDQo+ID4gKwlpZiAocnBjX3dha2VfdXBfZmlyc3QoJnhwcnQt
PnNlbmRpbmcsIF9feHBydF9sb2NrX3dyaXRlX2Z1bmMsIHhwcnQpKQ0KPiA+ICAJCXJldHVybjsN
Cj4gPiAgCXhwcnRfY2xlYXJfbG9ja2VkKHhwcnQpOw0KPiA+ICB9DQo+ID4gQEAgLTI5Nyw4ICsy
ODIsNyBAQCBzdGF0aWMgdm9pZCBfX3hwcnRfbG9ja193cml0ZV9uZXh0X2Nvbmcoc3RydWN0IHJw
Y194cHJ0DQo+ID4gKnhwcnQpDQo+ID4gIAkJcmV0dXJuOw0KPiA+ICAJaWYgKFJQQ1hQUlRfQ09O
R0VTVEVEKHhwcnQpKQ0KPiA+ICAJCWdvdG8gb3V0X3VubG9jazsNCj4gPiAtCWlmIChycGNfd2Fr
ZV91cF9maXJzdF9vbl93cSh4cHJ0aW9kX3dvcmtxdWV1ZSwgJnhwcnQtPnNlbmRpbmcsDQo+ID4g
LQkJCQlfX3hwcnRfbG9ja193cml0ZV9mdW5jLCB4cHJ0KSkNCj4gPiArCWlmIChycGNfd2FrZV91
cF9maXJzdCgmeHBydC0+c2VuZGluZywgX194cHJ0X2xvY2tfd3JpdGVfZnVuYywgeHBydCkpDQo+
ID4gIAkJcmV0dXJuOw0KPiA+ICBvdXRfdW5sb2NrOg0KPiA+ICAJeHBydF9jbGVhcl9sb2NrZWQo
eHBydCk7DQo+ID4gQEAgLTQ5NSw4ICs0NzksNyBAQCB2b2lkIHhwcnRfd3JpdGVfc3BhY2Uoc3Ry
dWN0IHJwY194cHJ0ICp4cHJ0KQ0KPiA+ICAJaWYgKHhwcnQtPnNuZF90YXNrKSB7DQo+ID4gIAkJ
ZHByaW50aygiUlBDOiAgICAgICB3cml0ZSBzcGFjZTogd2FraW5nIHdhaXRpbmcgdGFzayBvbiAi
DQo+ID4gIAkJCQkieHBydCAlcFxuIiwgeHBydCk7DQo+ID4gLQkJcnBjX3dha2VfdXBfcXVldWVk
X3Rhc2tfb25fd3EoeHBydGlvZF93b3JrcXVldWUsDQo+ID4gLQkJCQkmeHBydC0+cGVuZGluZywg
eHBydC0+c25kX3Rhc2spOw0KPiA+ICsJCXJwY193YWtlX3VwX3F1ZXVlZF90YXNrKCZ4cHJ0LT5w
ZW5kaW5nLCB4cHJ0LT5zbmRfdGFzayk7DQo+ID4gIAl9DQo+ID4gIAlzcGluX3VubG9ja19iaCgm
eHBydC0+dHJhbnNwb3J0X2xvY2spOw0KPiA+ICB9DQo+ID4gQEAgLTE1OTYsNyArMTU3OSw3IEBA
IHN0YXRpYyB2b2lkIHhwcnRfaW5pdChzdHJ1Y3QgcnBjX3hwcnQgKnhwcnQsIHN0cnVjdA0KPiA+
IG5ldA0KPiA+ICpuZXQpDQo+ID4gIA0KPiA+ICAJcnBjX2luaXRfd2FpdF9xdWV1ZSgmeHBydC0+
YmluZGluZywgInhwcnRfYmluZGluZyIpOw0KPiA+ICAJcnBjX2luaXRfd2FpdF9xdWV1ZSgmeHBy
dC0+cGVuZGluZywgInhwcnRfcGVuZGluZyIpOw0KPiA+IC0JcnBjX2luaXRfcHJpb3JpdHlfd2Fp
dF9xdWV1ZSgmeHBydC0+c2VuZGluZywgInhwcnRfc2VuZGluZyIpOw0KPiA+ICsJcnBjX2luaXRf
d2FpdF9xdWV1ZSgmeHBydC0+c2VuZGluZywgInhwcnRfc2VuZGluZyIpOw0KPiA+ICAJcnBjX2lu
aXRfcHJpb3JpdHlfd2FpdF9xdWV1ZSgmeHBydC0+YmFja2xvZywgInhwcnRfYmFja2xvZyIpOw0K
PiA+ICANCj4gPiAgCXhwcnRfaW5pdF94aWQoeHBydCk7DQo=

Re: [PATCH v2 20/34] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request()

[Schumaker, Anna]

SGkgVHJvbmQsDQoNCk9uIFR1ZSwgMjAxOC0wOS0wNCBhdCAxNzowNSAtMDQwMCwgVHJvbmQgTXlr
bGVidXN0IHdyb3RlOg0KPiBXaGVuIHdlIHNoaWZ0IHRvIHVzaW5nIHRoZSB0cmFuc21pdCBxdWV1
ZSwgdGhlbiB0aGUgdGFzayB0aGF0IGhvbGRzIHRoZQ0KPiB3cml0ZSBsb2NrIHdpbGwgbm90IG5l
Y2Vzc2FyaWx5IGJlIHRoZSBzYW1lIGFzIHRoZSBvbmUgYmVpbmcgdHJhbnNtaXR0ZWQuDQo+IA0K
PiBTaWduZWQtb2ZmLWJ5OiBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBoYW1tZXJz
cGFjZS5jb20+DQo+IC0tLQ0KPiAgaW5jbHVkZS9saW51eC9zdW5ycGMveHBydC5oICAgICAgICAg
ICAgICAgIHwgIDIgKy0NCj4gIG5ldC9zdW5ycGMveHBydC5jICAgICAgICAgICAgICAgICAgICAg
ICAgICB8ICAyICstDQo+ICBuZXQvc3VucnBjL3hwcnRyZG1hL3N2Y19yZG1hX2JhY2tjaGFubmVs
LmMgfCAgMyArLS0NCj4gIG5ldC9zdW5ycGMveHBydHJkbWEvdHJhbnNwb3J0LmMgICAgICAgICAg
ICB8ICA1ICsrLS0NCj4gIG5ldC9zdW5ycGMveHBydHNvY2suYyAgICAgICAgICAgICAgICAgICAg
ICB8IDI3ICsrKysrKysrKysrLS0tLS0tLS0tLS0NCj4gIDUgZmlsZXMgY2hhbmdlZCwgMTggaW5z
ZXJ0aW9ucygrKSwgMjEgZGVsZXRpb25zKC0pDQo+IA0KPiBkaWZmIC0tZ2l0IGEvaW5jbHVkZS9s
aW51eC9zdW5ycGMveHBydC5oIGIvaW5jbHVkZS9saW51eC9zdW5ycGMveHBydC5oDQo+IGluZGV4
IDgxYTZjMmM4ZGZjNy4uNmQ5MWFjZmUwNjQ0IDEwMDY0NA0KPiAtLS0gYS9pbmNsdWRlL2xpbnV4
L3N1bnJwYy94cHJ0LmgNCj4gKysrIGIvaW5jbHVkZS9saW51eC9zdW5ycGMveHBydC5oDQo+IEBA
IC0xNDAsNyArMTQwLDcgQEAgc3RydWN0IHJwY194cHJ0X29wcyB7DQo+ICAJdm9pZAkJKCpjb25u
ZWN0KShzdHJ1Y3QgcnBjX3hwcnQgKnhwcnQsIHN0cnVjdCBycGNfdGFzaw0KPiAqdGFzayk7DQo+
ICAJaW50CQkoKmJ1Zl9hbGxvYykoc3RydWN0IHJwY190YXNrICp0YXNrKTsNCj4gIAl2b2lkCQko
KmJ1Zl9mcmVlKShzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spOw0KPiAtCWludAkJKCpzZW5kX3JlcXVl
c3QpKHN0cnVjdCBycGNfdGFzayAqdGFzayk7DQo+ICsJaW50CQkoKnNlbmRfcmVxdWVzdCkoc3Ry
dWN0IHJwY19ycXN0ICpyZXEsIHN0cnVjdCBycGNfdGFzaw0KPiAqdGFzayk7DQo+ICAJdm9pZAkJ
KCpzZXRfcmV0cmFuc190aW1lb3V0KShzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spOw0KPiAgCXZvaWQJ
CSgqdGltZXIpKHN0cnVjdCBycGNfeHBydCAqeHBydCwgc3RydWN0IHJwY190YXNrDQo+ICp0YXNr
KTsNCj4gIAl2b2lkCQkoKnJlbGVhc2VfcmVxdWVzdCkoc3RydWN0IHJwY190YXNrICp0YXNrKTsN
Cj4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMveHBydC5jIGIvbmV0L3N1bnJwYy94cHJ0LmMNCj4g
aW5kZXggOGU4YzM0NWVlZGY3Li43YzgwZjkzNTYyZTUgMTAwNjQ0DQo+IC0tLSBhL25ldC9zdW5y
cGMveHBydC5jDQo+ICsrKyBiL25ldC9zdW5ycGMveHBydC5jDQo+IEBAIC0xMTcwLDcgKzExNzAs
NyBAQCB2b2lkIHhwcnRfdHJhbnNtaXQoc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiAgCX0NCj4g
IA0KPiAgCWNvbm5lY3RfY29va2llID0geHBydC0+Y29ubmVjdF9jb29raWU7DQo+IC0Jc3RhdHVz
ID0geHBydC0+b3BzLT5zZW5kX3JlcXVlc3QodGFzayk7DQo+ICsJc3RhdHVzID0geHBydC0+b3Bz
LT5zZW5kX3JlcXVlc3QocmVxLCB0YXNrKTsNCj4gIAl0cmFjZV94cHJ0X3RyYW5zbWl0KHhwcnQs
IHJlcS0+cnFfeGlkLCBzdGF0dXMpOw0KPiAgCWlmIChzdGF0dXMgIT0gMCkgew0KPiAgCQl0YXNr
LT50a19zdGF0dXMgPSBzdGF0dXM7DQo+IGRpZmYgLS1naXQgYS9uZXQvc3VucnBjL3hwcnRyZG1h
L3N2Y19yZG1hX2JhY2tjaGFubmVsLmMNCj4gYi9uZXQvc3VucnBjL3hwcnRyZG1hL3N2Y19yZG1h
X2JhY2tjaGFubmVsLmMNCj4gaW5kZXggMDliMTJiNzU2OGZlLi5kMTYxOGM3MGVkYjQgMTAwNjQ0
DQo+IC0tLSBhL25ldC9zdW5ycGMveHBydHJkbWEvc3ZjX3JkbWFfYmFja2NoYW5uZWwuYw0KPiAr
KysgYi9uZXQvc3VucnBjL3hwcnRyZG1hL3N2Y19yZG1hX2JhY2tjaGFubmVsLmMNCj4gQEAgLTIx
NSw5ICsyMTUsOCBAQCBycGNyZG1hX2JjX3NlbmRfcmVxdWVzdChzdHJ1Y3Qgc3ZjeHBydF9yZG1h
ICpyZG1hLCBzdHJ1Y3QNCj4gcnBjX3Jxc3QgKnJxc3QpDQo+ICAgKiBjb25uZWN0aW9uLg0KPiAg
ICovDQo+ICBzdGF0aWMgaW50DQo+IC14cHJ0X3JkbWFfYmNfc2VuZF9yZXF1ZXN0KHN0cnVjdCBy
cGNfdGFzayAqdGFzaykNCj4gK3hwcnRfcmRtYV9iY19zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY19y
cXN0ICpycXN0LCBzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICB7DQo+IC0Jc3RydWN0IHJwY19y
cXN0ICpycXN0ID0gdGFzay0+dGtfcnFzdHA7DQo+ICAJc3RydWN0IHN2Y194cHJ0ICpzeHBydCA9
IHJxc3QtPnJxX3hwcnQtPmJjX3hwcnQ7DQo+ICAJc3RydWN0IHN2Y3hwcnRfcmRtYSAqcmRtYTsN
Cj4gIAlpbnQgcmV0Ow0KPiBkaWZmIC0tZ2l0IGEvbmV0L3N1bnJwYy94cHJ0cmRtYS90cmFuc3Bv
cnQuYyBiL25ldC9zdW5ycGMveHBydHJkbWEvdHJhbnNwb3J0LmMNCj4gaW5kZXggMTQzY2UyNTc5
YmE5Li5mYTY4NGJmNGQwOTAgMTAwNjQ0DQo+IC0tLSBhL25ldC9zdW5ycGMveHBydHJkbWEvdHJh
bnNwb3J0LmMNCj4gKysrIGIvbmV0L3N1bnJwYy94cHJ0cmRtYS90cmFuc3BvcnQuYw0KPiBAQCAt
NzA2LDkgKzcwNiw4IEBAIHhwcnRfcmRtYV9mcmVlKHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4g
ICAqCQlzZW50LiBEbyBub3QgdHJ5IHRvIHNlbmQgdGhpcyBtZXNzYWdlIGFnYWluLg0KPiAgICov
DQo+ICBzdGF0aWMgaW50DQo+IC14cHJ0X3JkbWFfc2VuZF9yZXF1ZXN0KHN0cnVjdCBycGNfdGFz
ayAqdGFzaykNCj4gK3hwcnRfcmRtYV9zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY19ycXN0ICpycXN0
LCBzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICB7DQo+IC0Jc3RydWN0IHJwY19ycXN0ICpycXN0
ID0gdGFzay0+dGtfcnFzdHA7DQo+ICAJc3RydWN0IHJwY194cHJ0ICp4cHJ0ID0gcnFzdC0+cnFf
eHBydDsNCj4gIAlzdHJ1Y3QgcnBjcmRtYV9yZXEgKnJlcSA9IHJwY3JfdG9fcmRtYXIocnFzdCk7
DQo+ICAJc3RydWN0IHJwY3JkbWFfeHBydCAqcl94cHJ0ID0gcnBjeF90b19yZG1heCh4cHJ0KTsN
Cj4gQEAgLTc0MSw3ICs3NDAsNyBAQCB4cHJ0X3JkbWFfc2VuZF9yZXF1ZXN0KHN0cnVjdCBycGNf
dGFzayAqdGFzaykNCj4gIAkvKiBBbiBSUEMgd2l0aCBubyByZXBseSB3aWxsIHRocm93IG9mZiBj
cmVkaXQgYWNjb3VudGluZywNCj4gIAkgKiBzbyBkcm9wIHRoZSBjb25uZWN0aW9uIHRvIHJlc2V0
IHRoZSBjcmVkaXQgZ3JhbnQuDQo+ICAJICovDQo+IC0JaWYgKCFycGNfcmVwbHlfZXhwZWN0ZWQo
dGFzaykpDQo+ICsJaWYgKCFycGNfcmVwbHlfZXhwZWN0ZWQocnFzdC0+cnFfdGFzaykpDQo+ICAJ
CWdvdG8gZHJvcF9jb25uZWN0aW9uOw0KPiAgCXJldHVybiAwOw0KPiAgDQo+IGRpZmYgLS1naXQg
YS9uZXQvc3VucnBjL3hwcnRzb2NrLmMgYi9uZXQvc3VucnBjL3hwcnRzb2NrLmMNCj4gaW5kZXgg
OGQ2NDA0MjU5ZmY5Li5iODE0M2VkZWQ0YWYgMTAwNjQ0DQo+IC0tLSBhL25ldC9zdW5ycGMveHBy
dHNvY2suYw0KPiArKysgYi9uZXQvc3VucnBjL3hwcnRzb2NrLmMNCj4gQEAgLTQ0OSwxMiArNDQ5
LDEyIEBAIHN0YXRpYyB2b2lkIHhzX25vc3BhY2VfY2FsbGJhY2soc3RydWN0IHJwY190YXNrICp0
YXNrKQ0KPiAgDQo+ICAvKioNCj4gICAqIHhzX25vc3BhY2UgLSBwbGFjZSB0YXNrIG9uIHdhaXQg
cXVldWUgaWYgdHJhbnNtaXQgd2FzIGluY29tcGxldGUNCj4gKyAqIEByZXE6IHBvaW50ZXIgdG8g
UlBDIHJlcXVlc3QNCj4gICAqIEB0YXNrOiB0YXNrIHRvIHB1dCB0byBzbGVlcA0KPiAgICoNCj4g
ICAqLw0KPiAtc3RhdGljIGludCB4c19ub3NwYWNlKHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4g
K3N0YXRpYyBpbnQgeHNfbm9zcGFjZShzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSwgc3RydWN0IHJwY190
YXNrICp0YXNrKQ0KPiAgew0KPiAtCXN0cnVjdCBycGNfcnFzdCAqcmVxID0gdGFzay0+dGtfcnFz
dHA7DQo+ICAJc3RydWN0IHJwY194cHJ0ICp4cHJ0ID0gcmVxLT5ycV94cHJ0Ow0KPiAgCXN0cnVj
dCBzb2NrX3hwcnQgKnRyYW5zcG9ydCA9IGNvbnRhaW5lcl9vZih4cHJ0LCBzdHJ1Y3Qgc29ja194
cHJ0LA0KPiB4cHJ0KTsNCj4gIAlzdHJ1Y3Qgc29jayAqc2sgPSB0cmFuc3BvcnQtPmluZXQ7DQo+
IEBAIC01MTMsNiArNTEzLDcgQEAgc3RhdGljIGlubGluZSB2b2lkIHhzX2VuY29kZV9zdHJlYW1f
cmVjb3JkX21hcmtlcihzdHJ1Y3QNCj4geGRyX2J1ZiAqYnVmKQ0KPiAgDQo+ICAvKioNCj4gICAq
IHhzX2xvY2FsX3NlbmRfcmVxdWVzdCAtIHdyaXRlIGFuIFJQQyByZXF1ZXN0IHRvIGFuIEFGX0xP
Q0FMIHNvY2tldA0KPiArICogQHJlcTogcG9pbnRlciB0byBSUEMgcmVxdWVzdA0KPiAgICogQHRh
c2s6IFJQQyB0YXNrIHRoYXQgbWFuYWdlcyB0aGUgc3RhdGUgb2YgYW4gUlBDIHJlcXVlc3QNCj4g
ICAqDQo+ICAgKiBSZXR1cm4gdmFsdWVzOg0KPiBAQCAtNTIyLDkgKzUyMyw4IEBAIHN0YXRpYyBp
bmxpbmUgdm9pZCB4c19lbmNvZGVfc3RyZWFtX3JlY29yZF9tYXJrZXIoc3RydWN0DQo+IHhkcl9i
dWYgKmJ1ZikNCj4gICAqIEVOT1RDT05OOglDYWxsZXIgbmVlZHMgdG8gaW52b2tlIGNvbm5lY3Qg
bG9naWMgdGhlbiBjYWxsIGFnYWluDQo+ICAgKiAgICBvdGhlcjoJU29tZSBvdGhlciBlcnJvciBv
Y2N1cmVkLCB0aGUgcmVxdWVzdCB3YXMgbm90IHNlbnQNCj4gICAqLw0KPiAtc3RhdGljIGludCB4
c19sb2NhbF9zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiArc3RhdGljIGlu
dCB4c19sb2NhbF9zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY19ycXN0ICpyZXEsIHN0cnVjdCBycGNf
dGFzayAqdGFzaykNCj4gIHsNCj4gLQlzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSA9IHRhc2stPnRrX3Jx
c3RwOw0KPiAgCXN0cnVjdCBycGNfeHBydCAqeHBydCA9IHJlcS0+cnFfeHBydDsNCj4gIAlzdHJ1
Y3Qgc29ja194cHJ0ICp0cmFuc3BvcnQgPQ0KPiAgCQkJCWNvbnRhaW5lcl9vZih4cHJ0LCBzdHJ1
Y3Qgc29ja194cHJ0LCB4cHJ0KTsNCj4gQEAgLTU2OSw3ICs1NjksNyBAQCBzdGF0aWMgaW50IHhz
X2xvY2FsX3NlbmRfcmVxdWVzdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICAJY2FzZSAtRU5P
QlVGUzoNCj4gIAkJYnJlYWs7DQo+ICAJY2FzZSAtRUFHQUlOOg0KPiAtCQlzdGF0dXMgPSB4c19u
b3NwYWNlKHRhc2spOw0KPiArCQlzdGF0dXMgPSB4c19ub3NwYWNlKHJlcSwgdGFzayk7DQo+ICAJ
CWJyZWFrOw0KPiAgCWRlZmF1bHQ6DQo+ICAJCWRwcmludGsoIlJQQzogICAgICAgc2VuZG1zZyBy
ZXR1cm5lZCB1bnJlY29nbml6ZWQgZXJyb3IgJWRcbiIsDQo+IEBAIC01ODUsNiArNTg1LDcgQEAg
c3RhdGljIGludCB4c19sb2NhbF9zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY190YXNrICp0YXNrKQ0K
PiAgDQo+ICAvKioNCj4gICAqIHhzX3VkcF9zZW5kX3JlcXVlc3QgLSB3cml0ZSBhbiBSUEMgcmVx
dWVzdCB0byBhIFVEUCBzb2NrZXQNCj4gKyAqIEByZXE6IHBvaW50ZXIgdG8gUlBDIHJlcXVlc3QN
Cj4gICAqIEB0YXNrOiBhZGRyZXNzIG9mIFJQQyB0YXNrIHRoYXQgbWFuYWdlcyB0aGUgc3RhdGUg
b2YgYW4gUlBDIHJlcXVlc3QNCj4gICAqDQo+ICAgKiBSZXR1cm4gdmFsdWVzOg0KPiBAQCAtNTk0
LDkgKzU5NSw4IEBAIHN0YXRpYyBpbnQgeHNfbG9jYWxfc2VuZF9yZXF1ZXN0KHN0cnVjdCBycGNf
dGFzayAqdGFzaykNCj4gICAqIEVOT1RDT05OOglDYWxsZXIgbmVlZHMgdG8gaW52b2tlIGNvbm5l
Y3QgbG9naWMgdGhlbiBjYWxsIGFnYWluDQo+ICAgKiAgICBvdGhlcjoJU29tZSBvdGhlciBlcnJv
ciBvY2N1cnJlZCwgdGhlIHJlcXVlc3Qgd2FzIG5vdCBzZW50DQo+ICAgKi8NCj4gLXN0YXRpYyBp
bnQgeHNfdWRwX3NlbmRfcmVxdWVzdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICtzdGF0aWMg
aW50IHhzX3VkcF9zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY19ycXN0ICpyZXEsIHN0cnVjdCBycGNf
dGFzayAqdGFzaykNCj4gIHsNCj4gLQlzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSA9IHRhc2stPnRrX3Jx
c3RwOw0KPiAgCXN0cnVjdCBycGNfeHBydCAqeHBydCA9IHJlcS0+cnFfeHBydDsNCj4gIAlzdHJ1
Y3Qgc29ja194cHJ0ICp0cmFuc3BvcnQgPSBjb250YWluZXJfb2YoeHBydCwgc3RydWN0IHNvY2tf
eHBydCwNCj4geHBydCk7DQo+ICAJc3RydWN0IHhkcl9idWYgKnhkciA9ICZyZXEtPnJxX3NuZF9i
dWY7DQo+IEBAIC02MzgsNyArNjM4LDcgQEAgc3RhdGljIGludCB4c191ZHBfc2VuZF9yZXF1ZXN0
KHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4gIAkJLyogU2hvdWxkIHdlIGNhbGwgeHNfY2xvc2Uo
KSBoZXJlPyAqLw0KPiAgCQlicmVhazsNCj4gIAljYXNlIC1FQUdBSU46DQo+IC0JCXN0YXR1cyA9
IHhzX25vc3BhY2UodGFzayk7DQo+ICsJCXN0YXR1cyA9IHhzX25vc3BhY2UocmVxLCB0YXNrKTsN
Cj4gIAkJYnJlYWs7DQo+ICAJY2FzZSAtRU5FVFVOUkVBQ0g6DQo+ICAJY2FzZSAtRU5PQlVGUzoN
Cj4gQEAgLTY1OCw2ICs2NTgsNyBAQCBzdGF0aWMgaW50IHhzX3VkcF9zZW5kX3JlcXVlc3Qoc3Ry
dWN0IHJwY190YXNrICp0YXNrKQ0KPiAgDQo+ICAvKioNCj4gICAqIHhzX3RjcF9zZW5kX3JlcXVl
c3QgLSB3cml0ZSBhbiBSUEMgcmVxdWVzdCB0byBhIFRDUCBzb2NrZXQNCj4gKyAqIEByZXE6IHBv
aW50ZXIgdG8gUlBDIHJlcXVlc3QNCj4gICAqIEB0YXNrOiBhZGRyZXNzIG9mIFJQQyB0YXNrIHRo
YXQgbWFuYWdlcyB0aGUgc3RhdGUgb2YgYW4gUlBDIHJlcXVlc3QNCj4gICAqDQo+ICAgKiBSZXR1
cm4gdmFsdWVzOg0KPiBAQCAtNjcwLDkgKzY3MSw4IEBAIHN0YXRpYyBpbnQgeHNfdWRwX3NlbmRf
cmVxdWVzdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2spDQo+ICAgKiBYWFg6IEluIHRoZSBjYXNlIG9m
IHNvZnQgdGltZW91dHMsIHNob3VsZCB3ZSBldmVudHVhbGx5IGdpdmUgdXANCj4gICAqCWlmIHNl
bmRtc2cgaXMgbm90IGFibGUgdG8gbWFrZSBwcm9ncmVzcz8NCj4gICAqLw0KPiAtc3RhdGljIGlu
dCB4c190Y3Bfc2VuZF9yZXF1ZXN0KHN0cnVjdCBycGNfdGFzayAqdGFzaykNCj4gK3N0YXRpYyBp
bnQgeHNfdGNwX3NlbmRfcmVxdWVzdChzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSwgc3RydWN0IHJwY190
YXNrICp0YXNrKQ0KPiAgew0KPiAtCXN0cnVjdCBycGNfcnFzdCAqcmVxID0gdGFzay0+dGtfcnFz
dHA7DQo+ICAJc3RydWN0IHJwY194cHJ0ICp4cHJ0ID0gcmVxLT5ycV94cHJ0Ow0KPiAgCXN0cnVj
dCBzb2NrX3hwcnQgKnRyYW5zcG9ydCA9IGNvbnRhaW5lcl9vZih4cHJ0LCBzdHJ1Y3Qgc29ja194
cHJ0LA0KPiB4cHJ0KTsNCj4gIAlzdHJ1Y3QgeGRyX2J1ZiAqeGRyID0gJnJlcS0+cnFfc25kX2J1
ZjsNCj4gQEAgLTY5Nyw3ICs2OTcsNyBAQCBzdGF0aWMgaW50IHhzX3RjcF9zZW5kX3JlcXVlc3Qo
c3RydWN0IHJwY190YXNrICp0YXNrKQ0KPiAgCSAqIGNvbXBsZXRlcyB3aGlsZSB0aGUgc29ja2V0
IGhvbGRzIGEgcmVmZXJlbmNlIHRvIHRoZSBwYWdlcywNCj4gIAkgKiB0aGVuIHdlIG1heSBlbmQg
dXAgcmVzZW5kaW5nIGNvcnJ1cHRlZCBkYXRhLg0KPiAgCSAqLw0KPiAtCWlmICh0YXNrLT50a19m
bGFncyAmIFJQQ19UQVNLX1NFTlQpDQo+ICsJaWYgKHJlcS0+cnFfdGFzay0+dGtfZmxhZ3MgJiBS
UENfVEFTS19TRU5UKQ0KDQpDYW4gcmVxIG9yIHJxX3Rhc2sgYmUgbnVsbCBjb21pbmcgaW50byB0
aGlzIGZ1bmN0aW9uPyAgSSdtIHNlZWluZyB0aGUgZm9sbG93aW5nDQpudWxsIHBvaW50ZXIgZGVy
ZWZlcmVuY2Ugd2hpbGUgcnVubmluZyBjb25uZWN0YXRob24gdGVzdHMgb24gdjQuMSBhZnRlciB0
aGlzDQpwYXRjaDoNCg0KWyAgIDc3Ljc4MjA0NV0gQlVHOiB1bmFibGUgdG8gaGFuZGxlIGtlcm5l
bCBOVUxMIHBvaW50ZXIgZGVyZWZlcmVuY2UgYXQNCjAwMDAwMDAwMDAwMDAwZDQNClsgICA3Ny43
ODIxODRdIFBHRCAwIFA0RCAwIA0KWyAgIDc3Ljc4MjIzMl0gT29wczogMDAwMCBbIzFdIFBSRUVN
UFQgU01QIFBUSQ0KWyAgIDc3Ljc4MjI4Nl0gQ1BVOiAwIFBJRDogMTc5MCBDb21tOiBORlN2NCBj
YWxsYmFjayBOb3QgdGFpbnRlZCA0LjE5LjAtcmMyLQ0KQU5OQSsgIzEzNTANClsgICA3Ny43ODIz
ODJdIEhhcmR3YXJlIG5hbWU6IEJvY2hzIEJvY2hzLCBCSU9TIEJvY2hzIDAxLzAxLzIwMTENClsg
ICA3Ny43ODI0NjhdIFJJUDogMDAxMDp4c190Y3Bfc2VuZF9yZXF1ZXN0KzB4NWYvMHgyNTAgW3N1
bnJwY10NClsgICA3Ny43ODI1MzFdIENvZGU6IDc0IDBlIDhiIDk3IDZjIDAxIDAwIDAwIDg1IGQy
IDBmIDg0IDIxIDAxIDAwIDAwIDhiIDQzIDQwIDQ4DQo4YiA1MyAwOCA4MyBlOCAwNCAwZCAwMCAw
MCAwMCA4MCAwZiBjOCA4OSAwMiA0OCA4YiA4MyA4OCAwMCAwMCAwMCA8MGY+IGI3IGE4IGQ0DQow
MCAwMCAwMCA0OSA4YiA4NyA3OCAwNSAwMCAwMCA2NiBjMSBlZCAwYiA4MyBmNSAwMSA4Mw0KWyAg
IDc3Ljc4MjcxNl0gUlNQOiAwMDE4OmZmZmZjOTAwMDEwYTNkMTggRUZMQUdTOiAwMDAxMDI4Mg0K
WyAgIDc3Ljc4Mjc3NV0gUkFYOiAwMDAwMDAwMDAwMDAwMDAwIFJCWDogZmZmZjg4MDBiYTQ1MmUw
MCBSQ1g6IDAwMDAwMDAwMDAwMDAwMDANClsgICA3Ny43ODI4NDNdIFJEWDogZmZmZjg4MDBiODg4
MzAwMCBSU0k6IGZmZmY4ODAwYmE2YjcxMDAgUkRJOiBmZmZmODgwMGJhNDUyZTAwDQpbICAgNzcu
NzgyOTExXSBSQlA6IGZmZmY4ODAwYmE0NTJlMDAgUjA4OiAwMDAwMDAxODVlOTc3ZmM5IFIwOTog
ZmZmZmZmZmY4MjIyOWM5OA0KWyAgIDc3Ljc4Mjk4MF0gUjEwOiBmZmZmZmZmZjgyMjI5Yzk0IFIx
MTogMDAwMDAwMDAwMDAwMDAwMCBSMTI6IGZmZmY4ODAwYmE0NTJlMDgNClsgICA3Ny43ODMwNDhd
IFIxMzogZmZmZjg4MDBiYTQ1MmUwMCBSMTQ6IGZmZmZmZmZmYTAzMDM0YjAgUjE1OiBmZmZmODgw
MGI0YzExODAwDQpbICAgNzcuNzgzMTE4XSBGUzogIDAwMDAwMDAwMDAwMDAwMDAoMDAwMCkgR1M6
ZmZmZjg4MDBiY2EwMDAwMCgwMDAwKQ0Ka25sR1M6MDAwMDAwMDAwMDAwMDAwMA0KWyAgIDc3Ljc4
MzIxN10gQ1M6ICAwMDEwIERTOiAwMDAwIEVTOiAwMDAwIENSMDogMDAwMDAwMDA4MDA1MDAzMw0K
WyAgIDc3Ljc4MzI3OF0gQ1IyOiAwMDAwMDAwMDAwMDAwMGQ0IENSMzogMDAwMDAwMDAwMjAwYTAw
MSBDUjQ6IDAwMDAwMDAwMDAxNjA2ZjANClsgICA3Ny43ODMzNTFdIENhbGwgVHJhY2U6DQpbICAg
NzcuNzgzNDE4XSAgPyBycGNfd2FrZV91cF9zdGF0dXMrMHg3MC8weDcwIFtzdW5ycGNdDQpbICAg
NzcuNzgzNTQyXSAgPyBycGNfY2xudF90ZXN0X2FuZF9hZGRfeHBydCsweGIwLzB4YjAgW3N1bnJw
Y10NClsgICA3Ny43ODM2MzRdICB4cHJ0X3RyYW5zbWl0KzB4ODIvMHgyMzAgW3N1bnJwY10NClsg
ICA3Ny43ODM3MTFdICA/IHJwY193YWtlX3VwX3N0YXR1cysweDcwLzB4NzAgW3N1bnJwY10NClsg
ICA3Ny43ODM3ODZdICA/IHJwY19jbG50X3Rlc3RfYW5kX2FkZF94cHJ0KzB4YjAvMHhiMCBbc3Vu
cnBjXQ0KWyAgIDc3Ljc4NDc3Ml0gIGNhbGxfYmNfdHJhbnNtaXQrMHg1MC8weGYwIFtzdW5ycGNd
DQpbICAgNzcuNzg1NzM3XSAgX19ycGNfZXhlY3V0ZSsweDdhLzB4MzcwIFtzdW5ycGNdDQpbICAg
NzcuNzg2NjI2XSAgPyBrdm1fY2xvY2tfcmVhZCsweDE0LzB4MzANClsgICA3Ny43ODc1NDZdICA/
IGt0aW1lX2dldCsweDMxLzB4OTANClsgICA3Ny43ODgzOTNdICBycGNfcnVuX2JjX3Rhc2srMHg4
NS8weGMwIFtzdW5ycGNdDQpbICAgNzcuNzg5MjYwXSAgYmNfc3ZjX3Byb2Nlc3MrMHgyNzUvMHgy
ZjAgW3N1bnJwY10NClsgICA3Ny43OTAyMTZdICBuZnM0MV9jYWxsYmFja19zdmMrMHgxMjAvMHgx
OTAgW25mc3Y0XQ0KWyAgIDc3Ljc5MDYzMV0gID8gd2FpdF93b2tlbisweDgwLzB4ODANClsgICA3
Ny43OTA5OThdICA/IG5mc19tYXBfZ2lkX3RvX2dyb3VwKzB4MTEwLzB4MTEwIFtuZnN2NF0NClsg
ICA3Ny43OTEzNTZdICBrdGhyZWFkKzB4MTBkLzB4MTMwDQpbICAgNzcuNzkxNzA4XSAgPyBrdGhy
ZWFkX3BhcmsrMHg4MC8weDgwDQpbICAgNzcuNzkyMDQ3XSAgcmV0X2Zyb21fZm9yaysweDM1LzB4
NDANClsgICA3Ny43OTIyNzJdIE1vZHVsZXMgbGlua2VkIGluOiBuZnN2MyBycGNzZWNfZ3NzX2ty
YjUgbmZzdjQgbmZzIGZzY2FjaGUNCmNmZzgwMjExIHJwY3JkbWEgcmZraWxsIGNyY3QxMGRpZl9w
Y2xtdWwgY3JjMzJfcGNsbXVsIGNyYzMyY19pbnRlbA0KZ2hhc2hfY2xtdWxuaV9pbnRlbCBwY2Jj
IGpveWRldiBtb3VzZWRldiBhZXNuaV9pbnRlbCBldmRldiBhZXNfeDg2XzY0DQpjcnlwdG9fc2lt
ZCBpbnB1dF9sZWRzIGNyeXB0ZCBsZWRfY2xhc3MgbWFjX2hpZCBnbHVlX2hlbHBlciBwc21vdXNl
IHBjc3Brcg0KaW50ZWxfYWdwIG5mc2QgaW50ZWxfZ3R0IGkyY19waWl4NCBidXR0b24gYXV0aF9y
cGNnc3MgbmZzX2FjbCBsb2NrZCBncmFjZSBzdW5ycGMNCnNjaF9mcV9jb2RlbCBpcF90YWJsZXMg
eF90YWJsZXMgYXRhX2dlbmVyaWMgcGF0YV9hY3BpIHNlcmlvX3JhdyBhdGtiZCBsaWJwczINCmF0
YV9waWl4IGxpYmF0YSBpODA0MiBmbG9wcHkgc2VyaW8gdWhjaV9oY2Qgc2NzaV9tb2QgZWhjaV9w
Y2kgZWhjaV9oY2QgdXNiY29yZQ0KdXNiX2NvbW1vbiB4ZnMgdmlydGlvX2JhbGxvb24gdmlydGlv
X25ldCBuZXRfZmFpbG92ZXIgZmFpbG92ZXIgdmlydGlvX3BjaQ0KdmlydGlvX2JsayB2aXJ0aW9f
cmluZyB2aXJ0aW8NClsgICA3Ny43OTQzOTddIENSMjogMDAwMDAwMDAwMDAwMDBkNA0KWyAgIDc3
Ljc5NDY1MF0gLS0tWyBlbmQgdHJhY2UgOGQ4ODg4YjZiZGU4NGQzMyBdLS0tDQoNCg0KVGhhbmtz
LA0KQW5uYQ0KPiAgCQl6ZXJvY29weSA9IGZhbHNlOw0KPiAgDQo+ICAJaWYgKHRlc3RfYml0KFhQ
UlRfU09DS19VUERfVElNRU9VVCwgJnRyYW5zcG9ydC0+c29ja19zdGF0ZSkpDQo+IEBAIC03NjEs
NyArNzYxLDcgQEAgc3RhdGljIGludCB4c190Y3Bfc2VuZF9yZXF1ZXN0KHN0cnVjdCBycGNfdGFz
ayAqdGFzaykNCj4gIAkJLyogU2hvdWxkIHdlIGNhbGwgeHNfY2xvc2UoKSBoZXJlPyAqLw0KPiAg
CQlicmVhazsNCj4gIAljYXNlIC1FQUdBSU46DQo+IC0JCXN0YXR1cyA9IHhzX25vc3BhY2UodGFz
ayk7DQo+ICsJCXN0YXR1cyA9IHhzX25vc3BhY2UocmVxLCB0YXNrKTsNCj4gIAkJYnJlYWs7DQo+
ICAJY2FzZSAtRUNPTk5SRVNFVDoNCj4gIAljYXNlIC1FQ09OTlJFRlVTRUQ6DQo+IEBAIC0yNzA2
LDkgKzI3MDYsOCBAQCBzdGF0aWMgaW50IGJjX3NlbmR0byhzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSkN
Cj4gIC8qDQo+ICAgKiBUaGUgc2VuZCByb3V0aW5lLiBCb3Jyb3dzIGZyb20gc3ZjX3NlbmQNCj4g
ICAqLw0KPiAtc3RhdGljIGludCBiY19zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY190YXNrICp0YXNr
KQ0KPiArc3RhdGljIGludCBiY19zZW5kX3JlcXVlc3Qoc3RydWN0IHJwY19ycXN0ICpyZXEsIHN0
cnVjdCBycGNfdGFzayAqdGFzaykNCj4gIHsNCj4gLQlzdHJ1Y3QgcnBjX3Jxc3QgKnJlcSA9IHRh
c2stPnRrX3Jxc3RwOw0KPiAgCXN0cnVjdCBzdmNfeHBydAkqeHBydDsNCj4gIAlpbnQgbGVuOw0K
PiAgDQo=

Re: [PATCH v2 20/34] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request()

[Trond Myklebust]

On Thu, 2018-09-06 at 18:49 +0000, Schumaker, Anna wrote:
> Hi Trond,
> 
> On Tue, 2018-09-04 at 17:05 -0400, Trond Myklebust wrote:
> > When we shift to using the transmit queue, then the task that holds
> > the
> > write lock will not necessarily be the same as the one being
> > transmitted.
> > 
> > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > ---
> >  include/linux/sunrpc/xprt.h                |  2 +-
> >  net/sunrpc/xprt.c                          |  2 +-
> >  net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  3 +--
> >  net/sunrpc/xprtrdma/transport.c            |  5 ++--
> >  net/sunrpc/xprtsock.c                      | 27 +++++++++++-------
> > ----
> >  5 files changed, 18 insertions(+), 21 deletions(-)
> > 
> > diff --git a/include/linux/sunrpc/xprt.h
> > b/include/linux/sunrpc/xprt.h
> > index 81a6c2c8dfc7..6d91acfe0644 100644
> > --- a/include/linux/sunrpc/xprt.h
> > +++ b/include/linux/sunrpc/xprt.h
> > @@ -140,7 +140,7 @@ struct rpc_xprt_ops {
> >  	void		(*connect)(struct rpc_xprt *xprt, struct
> > rpc_task
> > *task);
> >  	int		(*buf_alloc)(struct rpc_task *task);
> >  	void		(*buf_free)(struct rpc_task *task);
> > -	int		(*send_request)(struct rpc_task *task);
> > +	int		(*send_request)(struct rpc_rqst *req, struct
> > rpc_task
> > *task);
> >  	void		(*set_retrans_timeout)(struct rpc_task
> > *task);
> >  	void		(*timer)(struct rpc_xprt *xprt, struct
> > rpc_task
> > *task);
> >  	void		(*release_request)(struct rpc_task *task);
> > diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> > index 8e8c345eedf7..7c80f93562e5 100644
> > --- a/net/sunrpc/xprt.c
> > +++ b/net/sunrpc/xprt.c
> > @@ -1170,7 +1170,7 @@ void xprt_transmit(struct rpc_task *task)
> >  	}
> >  
> >  	connect_cookie = xprt->connect_cookie;
> > -	status = xprt->ops->send_request(task);
> > +	status = xprt->ops->send_request(req, task);
> >  	trace_xprt_transmit(xprt, req->rq_xid, status);
> >  	if (status != 0) {
> >  		task->tk_status = status;
> > diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > index 09b12b7568fe..d1618c70edb4 100644
> > --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > @@ -215,9 +215,8 @@ rpcrdma_bc_send_request(struct svcxprt_rdma
> > *rdma, struct
> > rpc_rqst *rqst)
> >   * connection.
> >   */
> >  static int
> > -xprt_rdma_bc_send_request(struct rpc_task *task)
> > +xprt_rdma_bc_send_request(struct rpc_rqst *rqst, struct rpc_task
> > *task)
> >  {
> > -	struct rpc_rqst *rqst = task->tk_rqstp;
> >  	struct svc_xprt *sxprt = rqst->rq_xprt->bc_xprt;
> >  	struct svcxprt_rdma *rdma;
> >  	int ret;
> > diff --git a/net/sunrpc/xprtrdma/transport.c
> > b/net/sunrpc/xprtrdma/transport.c
> > index 143ce2579ba9..fa684bf4d090 100644
> > --- a/net/sunrpc/xprtrdma/transport.c
> > +++ b/net/sunrpc/xprtrdma/transport.c
> > @@ -706,9 +706,8 @@ xprt_rdma_free(struct rpc_task *task)
> >   *		sent. Do not try to send this message again.
> >   */
> >  static int
> > -xprt_rdma_send_request(struct rpc_task *task)
> > +xprt_rdma_send_request(struct rpc_rqst *rqst, struct rpc_task
> > *task)
> >  {
> > -	struct rpc_rqst *rqst = task->tk_rqstp;
> >  	struct rpc_xprt *xprt = rqst->rq_xprt;
> >  	struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
> >  	struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt);
> > @@ -741,7 +740,7 @@ xprt_rdma_send_request(struct rpc_task *task)
> >  	/* An RPC with no reply will throw off credit accounting,
> >  	 * so drop the connection to reset the credit grant.
> >  	 */
> > -	if (!rpc_reply_expected(task))
> > +	if (!rpc_reply_expected(rqst->rq_task))
> >  		goto drop_connection;
> >  	return 0;
> >  
> > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
> > index 8d6404259ff9..b8143eded4af 100644
> > --- a/net/sunrpc/xprtsock.c
> > +++ b/net/sunrpc/xprtsock.c
> > @@ -449,12 +449,12 @@ static void xs_nospace_callback(struct
> > rpc_task *task)
> >  
> >  /**
> >   * xs_nospace - place task on wait queue if transmit was
> > incomplete
> > + * @req: pointer to RPC request
> >   * @task: task to put to sleep
> >   *
> >   */
> > -static int xs_nospace(struct rpc_task *task)
> > +static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
> >  {
> > -	struct rpc_rqst *req = task->tk_rqstp;
> >  	struct rpc_xprt *xprt = req->rq_xprt;
> >  	struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt,
> > xprt);
> >  	struct sock *sk = transport->inet;
> > @@ -513,6 +513,7 @@ static inline void
> > xs_encode_stream_record_marker(struct
> > xdr_buf *buf)
> >  
> >  /**
> >   * xs_local_send_request - write an RPC request to an AF_LOCAL
> > socket
> > + * @req: pointer to RPC request
> >   * @task: RPC task that manages the state of an RPC request
> >   *
> >   * Return values:
> > @@ -522,9 +523,8 @@ static inline void
> > xs_encode_stream_record_marker(struct
> > xdr_buf *buf)
> >   * ENOTCONN:	Caller needs to invoke connect logic then call
> > again
> >   *    other:	Some other error occured, the request was not
> > sent
> >   */
> > -static int xs_local_send_request(struct rpc_task *task)
> > +static int xs_local_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> >  {
> > -	struct rpc_rqst *req = task->tk_rqstp;
> >  	struct rpc_xprt *xprt = req->rq_xprt;
> >  	struct sock_xprt *transport =
> >  				container_of(xprt, struct sock_xprt,
> > xprt);
> > @@ -569,7 +569,7 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> >  	case -ENOBUFS:
> >  		break;
> >  	case -EAGAIN:
> > -		status = xs_nospace(task);
> > +		status = xs_nospace(req, task);
> >  		break;
> >  	default:
> >  		dprintk("RPC:       sendmsg returned unrecognized error
> > %d\n",
> > @@ -585,6 +585,7 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> >  
> >  /**
> >   * xs_udp_send_request - write an RPC request to a UDP socket
> > + * @req: pointer to RPC request
> >   * @task: address of RPC task that manages the state of an RPC
> > request
> >   *
> >   * Return values:
> > @@ -594,9 +595,8 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> >   * ENOTCONN:	Caller needs to invoke connect logic then call
> > again
> >   *    other:	Some other error occurred, the request was not
> > sent
> >   */
> > -static int xs_udp_send_request(struct rpc_task *task)
> > +static int xs_udp_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> >  {
> > -	struct rpc_rqst *req = task->tk_rqstp;
> >  	struct rpc_xprt *xprt = req->rq_xprt;
> >  	struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt,
> > xprt);
> >  	struct xdr_buf *xdr = &req->rq_snd_buf;
> > @@ -638,7 +638,7 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> >  		/* Should we call xs_close() here? */
> >  		break;
> >  	case -EAGAIN:
> > -		status = xs_nospace(task);
> > +		status = xs_nospace(req, task);
> >  		break;
> >  	case -ENETUNREACH:
> >  	case -ENOBUFS:
> > @@ -658,6 +658,7 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> >  
> >  /**
> >   * xs_tcp_send_request - write an RPC request to a TCP socket
> > + * @req: pointer to RPC request
> >   * @task: address of RPC task that manages the state of an RPC
> > request
> >   *
> >   * Return values:
> > @@ -670,9 +671,8 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> >   * XXX: In the case of soft timeouts, should we eventually give up
> >   *	if sendmsg is not able to make progress?
> >   */
> > -static int xs_tcp_send_request(struct rpc_task *task)
> > +static int xs_tcp_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> >  {
> > -	struct rpc_rqst *req = task->tk_rqstp;
> >  	struct rpc_xprt *xprt = req->rq_xprt;
> >  	struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt,
> > xprt);
> >  	struct xdr_buf *xdr = &req->rq_snd_buf;
> > @@ -697,7 +697,7 @@ static int xs_tcp_send_request(struct rpc_task
> > *task)
> >  	 * completes while the socket holds a reference to the pages,
> >  	 * then we may end up resending corrupted data.
> >  	 */
> > -	if (task->tk_flags & RPC_TASK_SENT)
> > +	if (req->rq_task->tk_flags & RPC_TASK_SENT)
> 
> Can req or rq_task be null coming into this function?  I'm seeing the
> following
> null pointer dereference while running connectathon tests on v4.1
> after this
> patch:

The backchannels do need PATCH 24/34