From: Trond Myklebust <trondmy@hammerspace.com>
To: "chuck.lever@oracle.com" <chuck.lever@oracle.com>
Cc: "sashal@kernel.org" <sashal@kernel.org>,
"stable@vger.kernel.org" <stable@vger.kernel.org>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"daire@dneg.com" <daire@dneg.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: [PATCH AUTOSEL 5.10 03/45] SUNRPC: Handle TCP socket sends with kernel_sendpage() again
Date: Mon, 8 Feb 2021 20:12:31 +0000 [thread overview]
Message-ID: <6a137e45966fc297671d6f7218b9603d856c4604.camel@hammerspace.com> (raw)
In-Reply-To: <5CD4BF8B-402C-4735-BF04-52B8D62F5EED@oracle.com>
On Mon, 2021-02-08 at 19:48 +0000, Chuck Lever wrote:
>
>
> > On Feb 8, 2021, at 2:34 PM, Trond Myklebust <
> > trondmy@hammerspace.com> wrote:
> >
> > On Tue, 2021-01-19 at 20:25 -0500, Sasha Levin wrote:
> > > From: Chuck Lever <chuck.lever@oracle.com>
> > >
> > > [ Upstream commit 4a85a6a3320b4a622315d2e0ea91a1d2b013bce4 ]
> > >
> > > Daire Byrne reports a ~50% aggregrate throughput regression on
> > > his
> > > Linux NFS server after commit da1661b93bf4 ("SUNRPC: Teach server
> > > to
> > > use xprt_sock_sendmsg for socket sends"), which replaced
> > > kernel_send_page() calls in NFSD's socket send path with calls to
> > > sock_sendmsg() using iov_iter.
> > >
> > > Investigation showed that tcp_sendmsg() was not using zero-copy
> > > to
> > > send the xdr_buf's bvec pages, but instead was relying on memcpy.
> > > This means copying every byte of a large NFS READ payload.
> > >
> > > It looks like TLS sockets do indeed support a ->sendpage method,
> > > so it's really not necessary to use xprt_sock_sendmsg() to
> > > support
> > > TLS fully on the server. A mechanical reversion of da1661b93bf4
> > > is
> > > not possible at this point, but we can re-implement the server's
> > > TCP socket sendmsg path using kernel_sendpage().
> > >
> > > Reported-by: Daire Byrne <daire@dneg.com>
> > > BugLink: https://bugzilla.kernel.org/show_bug.cgi?id=209439
> > > Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> > > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > > ---
> > > net/sunrpc/svcsock.c | 86
> > > +++++++++++++++++++++++++++++++++++++++++++-
> > > 1 file changed, 85 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
> > > index c2752e2b9ce34..4404c491eb388 100644
> > > --- a/net/sunrpc/svcsock.c
> > > +++ b/net/sunrpc/svcsock.c
> > > @@ -1062,6 +1062,90 @@ static int svc_tcp_recvfrom(struct
> > > svc_rqst
> > > *rqstp)
> > > return 0; /* record not complete */
> > > }
> > >
> > > +static int svc_tcp_send_kvec(struct socket *sock, const struct
> > > kvec
> > > *vec,
> > > + int flags)
> > > +{
> > > + return kernel_sendpage(sock, virt_to_page(vec->iov_base),
> > > + offset_in_page(vec->iov_base),
> > > + vec->iov_len, flags);
>
> Thanks for your review!
>
> > I'm having trouble with this line. This looks like it is trying to
> > push
> > a slab page into kernel_sendpage().
>
> The head and tail kvec's in rq_res are not kmalloc'd, they are
> backed by pages in rqstp->rq_pages[].
>
>
> > What guarantees that the nfsd
> > thread won't call kfree() before the socket layer is done
> > transmitting
> > the page?
>
> If I understand correctly what Neil told us last week, the page
> reference count on those pages is set up so that one of
> svc_xprt_release() or the network layer does the final put_page(),
> in a safe fashion.
>
> Before da1661b93bf4 ("SUNRPC: Teach server to use xprt_sock_sendmsg
> for socket sends"), the original svc_send_common() code did this:
>
> - /* send head */
> - if (slen == xdr->head[0].iov_len)
> - flags = 0;
> - len = kernel_sendpage(sock, headpage, headoffset,
> - xdr->head[0].iov_len, flags);
> - if (len != xdr->head[0].iov_len)
> - goto out;
> - slen -= xdr->head[0].iov_len;
> - if (slen == 0)
> - goto out;
>
>
>
OK, so then only the argument kvec can be allocated on the slab (thanks
to svc_deferred_recv)? Is that correct?
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com
next prev parent reply other threads:[~2021-02-08 21:23 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-20 1:25 [PATCH AUTOSEL 5.10 01/45] ASoC: Intel: haswell: Add missing pm_ops Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 02/45] ASoC: rt711: mutex between calibration and power state changes Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 03/45] SUNRPC: Handle TCP socket sends with kernel_sendpage() again Sasha Levin
2021-02-08 19:34 ` Trond Myklebust
2021-02-08 19:48 ` Chuck Lever
2021-02-08 20:12 ` Trond Myklebust [this message]
2021-02-08 20:17 ` Chuck Lever
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 04/45] ASoC: AMD Renoir - add DMI entry for Lenovo ThinkPad E14 Gen 2 Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 05/45] HID: multitouch: Enable multi-input for Synaptics pointstick/touchpad device Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 06/45] HID: sony: select CONFIG_CRC32 Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 07/45] dm integrity: select CRYPTO_SKCIPHER Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 08/45] x86/hyperv: Fix kexec panic/hang issues Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 09/45] scsi: ufs: Relax the condition of UFSHCI_QUIRK_SKIP_MANUAL_WB_FLUSH_CTRL Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 10/45] scsi: ufs: Correct the LUN used in eh_device_reset_handler() callback Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 11/45] scsi: qedi: Correct max length of CHAP secret Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 12/45] scsi: scsi_debug: Fix memleak in scsi_debug_init() Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 13/45] scsi: sd: Suppress spurious errors when WRITE SAME is being disabled Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 14/45] riscv: Fix kernel time_init() Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 15/45] riscv: Fix sifive serial driver Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 16/45] riscv: Enable interrupts during syscalls with M-Mode Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 17/45] HID: logitech-dj: add the G602 receiver Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 18/45] HID: Ignore battery for Elan touchscreen on ASUS UX550 Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 19/45] clk: tegra30: Add hda clock default rates to clock driver Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 20/45] ALSA: hda/tegra: fix tegra-hda on tegra30 soc Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 21/45] r8152: Add Lenovo Powered USB-C Travel Hub Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 22/45] riscv: cacheinfo: Fix using smp_processor_id() in preemptible Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 23/45] arm64: make atomic helpers __always_inline Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 24/45] xen: Fix event channel callback via INTX/GSI Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 25/45] x86/xen: Add xen_no_vector_callback option to test PCI INTX delivery Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 26/45] x86/xen: Fix xen_hvm_smp_init() when vector callback not available Sasha Levin
2021-01-20 1:35 ` Boris Ostrovsky
2021-01-24 13:11 ` Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 27/45] net: stmmac: use __napi_schedule() for PREEMPT_RT Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 28/45] net: stmmac: Fixed mtu channged by cache aligned Sasha Levin
2021-01-20 6:08 ` Jakub Kicinski
2021-01-20 14:26 ` Sasha Levin
2021-01-21 14:39 ` [Linux-stm32] " Ahmad Fatoum
2021-01-21 16:02 ` Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 29/45] dts: phy: fix missing mdio device and probe failure of vsc8541-01 device Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 30/45] dts: phy: add GPIO number and active state used for phy reset Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 31/45] riscv: defconfig: enable gpio support for HiFive Unleashed Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 32/45] drm/amdgpu/psp: fix psp gfx ctrl cmds Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 33/45] drm/amd/display: disable dcn10 pipe split by default Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 34/45] HID: logitech-hidpp: Add product ID for MX Ergo in Bluetooth mode Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 35/45] drm/amd/display: Fix to be able to stop crc calculation Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 36/45] drm/nouveau/bios: fix issue shadowing expansion ROMs Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 37/45] drm/nouveau/privring: ack interrupts the same way as RM Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 38/45] drm/nouveau/i2c/gm200: increase width of aux semaphore owner fields Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 39/45] drm/nouveau/mmu: fix vram heap sizing Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 40/45] drm/nouveau/kms/nv50-: fix case where notifier buffer is at offset 0 Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 41/45] io_uring: flush timeouts that should already have expired Sasha Levin
2021-01-20 1:25 ` [PATCH AUTOSEL 5.10 42/45] libperf tests: If a test fails return non-zero Sasha Levin
2021-01-20 1:26 ` [PATCH AUTOSEL 5.10 43/45] libperf tests: Fail when failing to get a tracepoint id Sasha Levin
2021-01-20 1:26 ` [PATCH AUTOSEL 5.10 44/45] RISC-V: Set current memblock limit Sasha Levin
2021-01-20 1:26 ` [PATCH AUTOSEL 5.10 45/45] RISC-V: Fix maximum allowed phsyical memory for RV32 Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6a137e45966fc297671d6f7218b9603d856c4604.camel@hammerspace.com \
--to=trondmy@hammerspace.com \
--cc=chuck.lever@oracle.com \
--cc=daire@dneg.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=sashal@kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).