From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38403C433E0 for ; Mon, 22 Feb 2021 15:23:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AE6CB64EEC for ; Mon, 22 Feb 2021 15:23:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE6CB64EEC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 430DA6B0078; Mon, 22 Feb 2021 10:23:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 406796B007B; Mon, 22 Feb 2021 10:23:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31BB66B007D; Mon, 22 Feb 2021 10:23:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 1B2EB6B0078 for ; Mon, 22 Feb 2021 10:23:31 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D39BC18019F0E for ; Mon, 22 Feb 2021 15:23:30 +0000 (UTC) X-FDA: 77846272980.17.14D6489 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf13.hostedemail.com (Postfix) with ESMTP id B4C66E00013D for ; Mon, 22 Feb 2021 15:23:27 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 287FD64E61; Mon, 22 Feb 2021 15:23:28 +0000 (UTC) Subject: [PATCH v2 4/4] SUNRPC: Cache pages that were replaced during a read splice From: Chuck Lever To: mgorman@techsingularity.net Cc: linux-nfs@vger.kernel.org, linux-mm@kvack.org, kuba@kernel.org Date: Mon, 22 Feb 2021 10:23:27 -0500 Message-ID: <161400740732.195066.3792261943053910900.stgit@klimt.1015granger.net> In-Reply-To: <161400722731.195066.9584156841718557193.stgit@klimt.1015granger.net> References: <161400722731.195066.9584156841718557193.stgit@klimt.1015granger.net> User-Agent: StGit/1.0-5-g755c MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B4C66E00013D X-Stat-Signature: snrw76oftc1opw696ccscwb4zjeqi8mf Received-SPF: none (kernel.org>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail.kernel.org; client-ip=198.145.29.99 X-HE-DKIM-Result: none/none X-HE-Tag: 1614007407-686978 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To avoid extra trips to the page allocator, don't free unused pages in nfsd_splice_actor(), but instead place them in a local cache. That cache is then used first when refilling rq_pages. On workloads that perform large NFS READs on splice-capable file systems, this saves a considerable amount of work. Suggested-by: NeilBrown Signed-off-by: Chuck Lever --- fs/nfsd/vfs.c | 4 ++-- include/linux/sunrpc/svc.h | 1 + include/linux/sunrpc/svc_xprt.h | 28 ++++++++++++++++++++++++++++ net/sunrpc/svc.c | 7 +++++++ net/sunrpc/svc_xprt.c | 12 ++++++++++++ 5 files changed, 50 insertions(+), 2 deletions(-) diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index d316e11923c5..25cf41eaf3c4 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -852,14 +852,14 @@ nfsd_splice_actor(struct pipe_inode_info *pipe, struct pipe_buffer *buf, if (rqstp->rq_res.page_len == 0) { get_page(page); - put_page(*rqstp->rq_next_page); + svc_rqst_put_unused_page(rqstp, *rqstp->rq_next_page); *(rqstp->rq_next_page++) = page; rqstp->rq_res.page_base = buf->offset; rqstp->rq_res.page_len = size; } else if (page != pp[-1]) { get_page(page); if (*rqstp->rq_next_page) - put_page(*rqstp->rq_next_page); + svc_rqst_put_unused_page(rqstp, *rqstp->rq_next_page); *(rqstp->rq_next_page++) = page; rqstp->rq_res.page_len += size; } else diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 31ee3b6047c3..340f4f3989c0 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -250,6 +250,7 @@ struct svc_rqst { struct xdr_stream rq_arg_stream; struct page *rq_scratch_page; struct xdr_buf rq_res; + struct list_head rq_unused_pages; struct page *rq_pages[RPCSVC_MAXPAGES + 1]; struct page * *rq_respages; /* points into rq_pages */ struct page * *rq_next_page; /* next reply page to use */ diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h index 571f605bc91e..49ef86499876 100644 --- a/include/linux/sunrpc/svc_xprt.h +++ b/include/linux/sunrpc/svc_xprt.h @@ -150,6 +150,34 @@ static inline void svc_xprt_get(struct svc_xprt *xprt) { kref_get(&xprt->xpt_ref); } + +/** + * svc_rqst_get_unused_page - Tap a page from the local cache + * @rqstp: svc_rqst with cached unused pages + * + * To save an allocator round trip, pages can be added to a + * local cache and re-used later by svc_alloc_arg(). + * + * Returns an unused page, or NULL if the cache is empty. + */ +static inline struct page *svc_rqst_get_unused_page(struct svc_rqst *rqstp) +{ + return list_first_entry_or_null(&rqstp->rq_unused_pages, + struct page, lru); +} + +/** + * svc_rqst_put_unused_page - Stash a page in the local cache + * @rqstp: svc_rqst with cached unused pages + * @page: page to cache + * + */ +static inline void svc_rqst_put_unused_page(struct svc_rqst *rqstp, + struct page *page) +{ + list_add(&page->lru, &rqstp->rq_unused_pages); +} + static inline void svc_xprt_set_local(struct svc_xprt *xprt, const struct sockaddr *sa, const size_t salen) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 61fb8a18552c..3920fa8f1146 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -570,6 +570,8 @@ svc_init_buffer(struct svc_rqst *rqstp, unsigned int size, int node) if (svc_is_backchannel(rqstp)) return 1; + INIT_LIST_HEAD(&rqstp->rq_unused_pages); + pages = size / PAGE_SIZE + 1; /* extra page as we hold both request and reply. * We assume one is at most one page */ @@ -593,8 +595,13 @@ svc_init_buffer(struct svc_rqst *rqstp, unsigned int size, int node) static void svc_release_buffer(struct svc_rqst *rqstp) { + struct page *page; unsigned int i; + while ((page = svc_rqst_get_unused_page(rqstp))) { + list_del(&page->lru); + put_page(page); + } for (i = 0; i < ARRAY_SIZE(rqstp->rq_pages); i++) if (rqstp->rq_pages[i]) put_page(rqstp->rq_pages[i]); diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 15aacfa5ca21..84210e546a66 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -678,6 +678,18 @@ static int svc_alloc_arg(struct svc_rqst *rqstp) for (needed = 0, i = 0; i < pages ; i++) if (!rqstp->rq_pages[i]) needed++; + if (needed) { + for (i = 0; i < pages; i++) { + if (!rqstp->rq_pages[i]) { + page = svc_rqst_get_unused_page(rqstp); + if (!page) + break; + list_del(&page->lru); + rqstp->rq_pages[i] = page; + needed--; + } + } + } if (needed) { LIST_HEAD(list);