From: Jonathan Lemon <jonathan.lemon@gmail.com>
To: Eric Dumazet <edumazet@google.com>
Cc: netdev <netdev@vger.kernel.org>, kernel-team <kernel-team@fb.com>,
Christoph Hellwig <hch@lst.de>,
Robin Murphy <robin.murphy@arm.com>,
Andrew Morton <akpm@linux-foundation.org>,
David Miller <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Willem de Bruijn <willemb@google.com>,
Steffen Klassert <steffen.klassert@secunet.com>,
Saeed Mahameed <saeedm@mellanox.com>,
Maxim Mikityanskiy <maximmi@mellanox.com>,
bjorn.topel@intel.com, magnus.karlsson@intel.com,
borisp@mellanox.com, david@redhat.com
Subject: Re: [RFC PATCH v2 15/21] net/tcp: add MSG_NETDMA flag for sendmsg()
Date: Mon, 27 Jul 2020 19:11:30 -0700 [thread overview]
Message-ID: <20200728021130.bjrlcj7tzebfxsz3@bsd-mbp.dhcp.thefacebook.com> (raw)
In-Reply-To: <CANn89iKStB8=Exyopi1sufuYhA-rZvYVMOEm9LDgKTLBYiqSmA@mail.gmail.com>
On Mon, Jul 27, 2020 at 10:44:59AM -0700, Eric Dumazet wrote:
> On Mon, Jul 27, 2020 at 10:35 AM Jonathan Lemon
> <jonathan.lemon@gmail.com> wrote:
> >
> > On Mon, Jul 27, 2020 at 09:09:48AM -0700, Eric Dumazet wrote:
> > > On Mon, Jul 27, 2020 at 8:56 AM Jonathan Lemon <jonathan.lemon@gmail.com> wrote:
> > > >
> > > > On Mon, Jul 27, 2020 at 08:19:43AM -0700, Eric Dumazet wrote:
> > > > > On Mon, Jul 27, 2020 at 12:51 AM Jonathan Lemon
> > > > > <jonathan.lemon@gmail.com> wrote:
> > > > > >
> > > > > > This flag indicates that the attached data is a zero-copy send,
> > > > > > and the pages should be retrieved from the netgpu module. The
> > > > > > socket should should already have been attached to a netgpu queue.
> > > > > >
> > > > > > Signed-off-by: Jonathan Lemon <jonathan.lemon@gmail.com>
> > > > > > ---
> > > > > > include/linux/socket.h | 1 +
> > > > > > net/ipv4/tcp.c | 8 ++++++++
> > > > > > 2 files changed, 9 insertions(+)
> > > > > >
> > > > > > diff --git a/include/linux/socket.h b/include/linux/socket.h
> > > > > > index 04d2bc97f497..63816cc25dee 100644
> > > > > > --- a/include/linux/socket.h
> > > > > > +++ b/include/linux/socket.h
> > > > > > @@ -310,6 +310,7 @@ struct ucred {
> > > > > > */
> > > > > >
> > > > > > #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */
> > > > > > +#define MSG_NETDMA 0x8000000
> > > > > > #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */
> > > > > > #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file
> > > > > > descriptor received through
> > > > > > diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> > > > > > index 261c28ccc8f6..340ce319edc9 100644
> > > > > > --- a/net/ipv4/tcp.c
> > > > > > +++ b/net/ipv4/tcp.c
> > > > > > @@ -1214,6 +1214,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
> > > > > > uarg->zerocopy = 0;
> > > > > > }
> > > > > >
> > > > > > + if (flags & MSG_NETDMA && size && sock_flag(sk, SOCK_ZEROCOPY)) {
> > > > > > + zc = sk->sk_route_caps & NETIF_F_SG;
> > > > > > + if (!zc) {
> > > > > > + err = -EFAULT;
> > > > > > + goto out_err;
> > > > > > + }
> > > > > > + }
> > > > > >
> > > > >
> > > > > Sorry, no, we can not allow adding yet another branch into TCP fast
> > > > > path for yet another variant of zero copy.
> > > >
> > > > I'm not in disagreement with that statement, but the existing zerocopy
> > > > work makes some assumptions that aren't suitable. I take it that you'd
> > > > rather have things folded together so the old/new code works together?
> > >
> > > Exact. Forcing users to use MSG_NETDMA, yet reusing SOCK_ZEROCOPY is silly.
> > >
> > > SOCK_ZEROCOPY has been added to that user space and kernel would agree
> > > on MSG_ZEROCOPY being not a nop (as it was on old kernels)
> > >
> > > >
> > > > Allocating an extra structure for every skbuff isn't ideal in my book.
> > > >
> > >
> > > We do not allocate a structure for every skbuff. Please look again.
> >
> > I'm looking here:
> >
> > uarg = sock_zerocopy_realloc(sk, size, skb_zcopy(skb));
> >
> > Doesn't sock_zerocopy_realloc() allocate a new structure if the skb
> > doesn't have one already?
> >
> >
> > > > > Overall, I think your patch series desperately tries to add changes in
> > > > > TCP stack, while there is yet no proof
> > > > > that you have to use TCP transport between the peers.
> > > >
> > > > The goal is having a reliable transport without resorting to RDMA.
> > >
> > > And why should it be TCP ?
> > >
> > > Are you dealing with lost packets, retransmits, timers, and al ?
> >
> > Yes? If there was a true lossless medium, RDMA would have taken over by
> > now. Or are you suggesting that the transport protocol reliability
> > should be performed in userspace? (not all the world is QUIC yet)
> >
>
> The thing is : this patch series is a monster thing adding stuff that
> is going to impact 100% % of TCP flows,
> even if not used in this NETDMA context.
>
> So you need to convince us you are really desperate to get this in
> upstream linux.
>
> I have implemented TCP RX zero copy without adding a single line in
> standard TCP code.
That's a bit of an exaggeration, as I see skb_zcopy_*() calls scattered
around the normal TCP code path. I also haven't changed the normal TCP
path either, other than doing some of the same things as skb_zcopy_*().
(ignoring the ugly moron about padding out the TCP header, which I'll
put under a static_branch_unlikely).
The thing is, the existing zero copy code is zero-copy to /host/ memory,
which is not the same thing as zero-copy to other memory areas.
--
Jonathan
next prev parent reply other threads:[~2020-07-28 2:11 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20200727052846.4070247-1-jonathan.lemon@gmail.com>
[not found] ` <20200727052846.4070247-4-jonathan.lemon@gmail.com>
2020-07-27 5:58 ` [RFC PATCH v2 03/21] mm: Allow DMA mapping of pages which are not online Christoph Hellwig
2020-07-27 17:08 ` Jonathan Lemon
[not found] ` <20200727052846.4070247-22-jonathan.lemon@gmail.com>
2020-07-27 7:35 ` [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu Christoph Hellwig
2020-07-27 17:00 ` Jonathan Lemon
2020-07-27 18:24 ` Christoph Hellwig
2020-07-28 1:48 ` Jonathan Lemon
2020-07-28 6:47 ` Christoph Hellwig
2020-07-28 16:05 ` Jonathan Lemon
2020-07-28 16:10 ` Christoph Hellwig
2020-07-28 18:19 ` Jason Gunthorpe
2020-07-28 21:01 ` Jonathan Lemon
2020-07-28 21:14 ` Christoph Hellwig
2020-07-28 23:38 ` Jason Gunthorpe
[not found] ` <20200727052846.4070247-14-jonathan.lemon@gmail.com>
2020-07-27 15:19 ` [RFC PATCH v2 13/21] net/tcp: Pad TCP options out to a fixed size " Eric Dumazet
2020-07-27 17:20 ` Jonathan Lemon
[not found] ` <20200727052846.4070247-16-jonathan.lemon@gmail.com>
2020-07-27 15:19 ` [RFC PATCH v2 15/21] net/tcp: add MSG_NETDMA flag for sendmsg() Eric Dumazet
2020-07-27 15:55 ` Jonathan Lemon
2020-07-27 16:09 ` Eric Dumazet
2020-07-27 17:35 ` Jonathan Lemon
2020-07-27 17:44 ` Eric Dumazet
2020-07-28 2:11 ` Jonathan Lemon [this message]
2020-07-28 2:17 ` Eric Dumazet
2020-07-28 3:08 ` Jonathan Lemon
2020-07-28 6:50 ` Christoph Hellwig
[not found] ` <20200727052846.4070247-9-jonathan.lemon@gmail.com>
2020-07-27 15:24 ` [RFC PATCH v2 08/21] skbuff: add a zc_netgpu bitflag Eric Dumazet
2020-07-27 16:59 ` Jonathan Lemon
2020-07-27 17:08 ` Eric Dumazet
2020-07-27 17:16 ` Jonathan Lemon
2020-07-27 22:44 [RFC PATCH v2 00/21] netgpu: networking between NIC and GPU/CPU Jonathan Lemon
2020-07-27 22:44 ` [RFC PATCH v2 15/21] net/tcp: add MSG_NETDMA flag for sendmsg() Jonathan Lemon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200728021130.bjrlcj7tzebfxsz3@bsd-mbp.dhcp.thefacebook.com \
--to=jonathan.lemon@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=bjorn.topel@intel.com \
--cc=borisp@mellanox.com \
--cc=davem@davemloft.net \
--cc=david@redhat.com \
--cc=edumazet@google.com \
--cc=hch@lst.de \
--cc=kernel-team@fb.com \
--cc=kuba@kernel.org \
--cc=magnus.karlsson@intel.com \
--cc=maximmi@mellanox.com \
--cc=netdev@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=saeedm@mellanox.com \
--cc=steffen.klassert@secunet.com \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).