[PATCH RFC] NFSD: Reduce svc_rqst::rq_pages churn during READDIR operations

[PATCH RFC] NFSD: Reduce svc_rqst::rq_pages churn during READDIR operations

[Chuck Lever]

During NFSv2 and NFSv3 READDIR/PLUS operations, NFSD advances
rq_next_page to the full size of the client-requested buffer, then
releases all those pages at the end of the request. The next request
to use that nfsd thread has to refill the pages.

NFSD does this even when the dirlist in the reply is small. With
NFSv3 clients that send READDIR operations with large buffer sizes,
that can be 256 put_page/alloc_page pairs per READDIR request, even
though those pages often remain unused.

We can save some work by not releasing dirlist buffer pages that
were not used to form the READDIR Reply. I've left the NFSv2 code
alone since there are never more than three pages involved in an
NFSv2 READDIR operation.

Eventually we should nail down why these pages need to be released
at all in order to avoid allocating and releasing pages
unnecessarily.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfsd/nfs3proc.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c
index 8675851199f8..569bfff314f0 100644
--- a/fs/nfsd/nfs3proc.c
+++ b/fs/nfsd/nfs3proc.c
@@ -492,6 +492,7 @@ nfsd3_proc_readdir(struct svc_rqst *rqstp)
 		}
 		count += PAGE_SIZE;
 	}
+	rqstp->rq_next_page = p;
 	resp->count = count >> 2;
 	if (resp->offset) {
 		if (unlikely(resp->offset1)) {
@@ -557,6 +558,7 @@ nfsd3_proc_readdirplus(struct svc_rqst *rqstp)
 		}
 		count += PAGE_SIZE;
 	}
+	rqstp->rq_next_page = p;
 	resp->count = count >> 2;
 	if (resp->offset) {
 		if (unlikely(resp->offset1)) {

Re: [PATCH RFC] NFSD: Reduce svc_rqst::rq_pages churn during READDIR operations

[J. Bruce Fields]

Makes sense to me.--b.

On Fri, Jan 22, 2021 at 03:21:10PM -0500, Chuck Lever wrote:
> During NFSv2 and NFSv3 READDIR/PLUS operations, NFSD advances
> rq_next_page to the full size of the client-requested buffer, then
> releases all those pages at the end of the request. The next request
> to use that nfsd thread has to refill the pages.
> 
> NFSD does this even when the dirlist in the reply is small. With
> NFSv3 clients that send READDIR operations with large buffer sizes,
> that can be 256 put_page/alloc_page pairs per READDIR request, even
> though those pages often remain unused.
> 
> We can save some work by not releasing dirlist buffer pages that
> were not used to form the READDIR Reply. I've left the NFSv2 code
> alone since there are never more than three pages involved in an
> NFSv2 READDIR operation.
> 
> Eventually we should nail down why these pages need to be released
> at all in order to avoid allocating and releasing pages
> unnecessarily.
> 
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  fs/nfsd/nfs3proc.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c
> index 8675851199f8..569bfff314f0 100644
> --- a/fs/nfsd/nfs3proc.c
> +++ b/fs/nfsd/nfs3proc.c
> @@ -492,6 +492,7 @@ nfsd3_proc_readdir(struct svc_rqst *rqstp)
>  		}
>  		count += PAGE_SIZE;
>  	}
> +	rqstp->rq_next_page = p;
>  	resp->count = count >> 2;
>  	if (resp->offset) {
>  		if (unlikely(resp->offset1)) {
> @@ -557,6 +558,7 @@ nfsd3_proc_readdirplus(struct svc_rqst *rqstp)
>  		}
>  		count += PAGE_SIZE;
>  	}
> +	rqstp->rq_next_page = p;
>  	resp->count = count >> 2;
>  	if (resp->offset) {
>  		if (unlikely(resp->offset1)) {
>