From: Chuck Lever <chuck.lever@oracle.com>
To: "trondmy@kernel.org" <trondmy@kernel.org>
Cc: Bruce Fields <bfields@redhat.com>,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] SUNRPC: Use TCP_CORK to optimise send performance on the server
Date: Tue, 16 Feb 2021 16:27:10 +0000 [thread overview]
Message-ID: <635D6DE6-3E7B-43EB-93A4-075DE91897BB@oracle.com> (raw)
In-Reply-To: <20210213202532.23146-1-trondmy@kernel.org>
Hey Trond-
> On Feb 13, 2021, at 3:25 PM, trondmy@kernel.org wrote:
>
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
>
> Use a counter to keep track of how many requests are queued behind the
> xprt->xpt_mutex, and keep TCP_CORK set until the queue is empty.
>
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
Let's move this forward a little bit.
I'd like to see the previously discussed MSG_MORE simplification
integrated into this patch.
In addition to Daire's testing, I've done some testing:
- No behavior regressions noted
- No changes in large I/O throughput
- Slightly shorter RTTs on software build
And the current version of the patch is now in the for-rc branch
of https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/
to get broader testing exposure.
This work is a candidate for a second NFSD PR during the 5.12
merge window, along with the other patches currently in the
for-rc branch.
> ---
> include/linux/sunrpc/svcsock.h | 2 ++
> net/sunrpc/svcsock.c | 8 +++++++-
> 2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h
> index b7ac7fe68306..bcc555c7ae9c 100644
> --- a/include/linux/sunrpc/svcsock.h
> +++ b/include/linux/sunrpc/svcsock.h
> @@ -35,6 +35,8 @@ struct svc_sock {
> /* Total length of the data (not including fragment headers)
> * received so far in the fragments making up this rpc: */
> u32 sk_datalen;
> + /* Number of queued send requests */
> + atomic_t sk_sendqlen;
>
> struct page * sk_pages[RPCSVC_MAXPAGES]; /* received data */
> };
> diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
> index 5a809c64dc7b..231f510a4830 100644
> --- a/net/sunrpc/svcsock.c
> +++ b/net/sunrpc/svcsock.c
> @@ -1171,18 +1171,23 @@ static int svc_tcp_sendto(struct svc_rqst *rqstp)
>
> svc_tcp_release_rqst(rqstp);
>
> + atomic_inc(&svsk->sk_sendqlen);
> mutex_lock(&xprt->xpt_mutex);
> if (svc_xprt_is_dead(xprt))
> goto out_notconn;
> + tcp_sock_set_cork(svsk->sk_sk, true);
> err = svc_tcp_sendmsg(svsk->sk_sock, &msg, xdr, marker, &sent);
> xdr_free_bvec(xdr);
> trace_svcsock_tcp_send(xprt, err < 0 ? err : sent);
> if (err < 0 || sent != (xdr->len + sizeof(marker)))
> goto out_close;
> + if (atomic_dec_and_test(&svsk->sk_sendqlen))
> + tcp_sock_set_cork(svsk->sk_sk, false);
> mutex_unlock(&xprt->xpt_mutex);
> return sent;
>
> out_notconn:
> + atomic_dec(&svsk->sk_sendqlen);
> mutex_unlock(&xprt->xpt_mutex);
> return -ENOTCONN;
> out_close:
> @@ -1192,6 +1197,7 @@ static int svc_tcp_sendto(struct svc_rqst *rqstp)
> (err < 0) ? err : sent, xdr->len);
> set_bit(XPT_CLOSE, &xprt->xpt_flags);
> svc_xprt_enqueue(xprt);
> + atomic_dec(&svsk->sk_sendqlen);
> mutex_unlock(&xprt->xpt_mutex);
> return -EAGAIN;
> }
> @@ -1261,7 +1267,7 @@ static void svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
> svsk->sk_datalen = 0;
> memset(&svsk->sk_pages[0], 0, sizeof(svsk->sk_pages));
>
> - tcp_sk(sk)->nonagle |= TCP_NAGLE_OFF;
> + tcp_sock_set_nodelay(sk);
>
> set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags);
> switch (sk->sk_state) {
> --
> 2.29.2
>
--
Chuck Lever
next prev parent reply other threads:[~2021-02-16 16:28 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-13 20:25 [PATCH] SUNRPC: Use TCP_CORK to optimise send performance on the server trondmy
2021-02-13 21:53 ` Chuck Lever
2021-02-13 22:10 ` Trond Myklebust
2021-02-13 23:30 ` Chuck Lever
2021-02-14 14:13 ` Daire Byrne
2021-02-14 16:24 ` Trond Myklebust
2021-02-14 16:59 ` Chuck Lever
2021-02-15 14:08 ` Daire Byrne
2021-02-15 18:22 ` Chuck Lever
2021-02-14 16:21 ` Trond Myklebust
2021-02-14 17:27 ` Chuck Lever
2021-02-14 17:41 ` Trond Myklebust
2021-02-14 17:58 ` Chuck Lever
2021-02-14 18:18 ` Trond Myklebust
2021-02-14 19:43 ` Chuck Lever
2021-02-16 16:27 ` Chuck Lever [this message]
2021-02-16 17:10 ` Trond Myklebust
2021-02-16 17:12 ` Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=635D6DE6-3E7B-43EB-93A4-075DE91897BB@oracle.com \
--to=chuck.lever@oracle.com \
--cc=bfields@redhat.com \
--cc=linux-nfs@vger.kernel.org \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).