From: Eric Dumazet <edumazet@google.com>
To: Jonathan Lemon <jonathan.lemon@gmail.com>
Cc: netdev <netdev@vger.kernel.org>, kernel-team <kernel-team@fb.com>,
Christoph Hellwig <hch@lst.de>,
Robin Murphy <robin.murphy@arm.com>,
Andrew Morton <akpm@linux-foundation.org>,
David Miller <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Willem de Bruijn <willemb@google.com>,
Steffen Klassert <steffen.klassert@secunet.com>,
Saeed Mahameed <saeedm@mellanox.com>,
Maxim Mikityanskiy <maximmi@mellanox.com>,
bjorn.topel@intel.com, magnus.karlsson@intel.com,
borisp@mellanox.com, david@redhat.com
Subject: Re: [RFC PATCH v2 08/21] skbuff: add a zc_netgpu bitflag
Date: Mon, 27 Jul 2020 10:08:10 -0700 [thread overview]
Message-ID: <CANn89iJBLrmvtTmyBAwErejCEMdt6gLNNRX1XZ1oMorAWrbp4g@mail.gmail.com> (raw)
In-Reply-To: <20200727165904.twsfhafflw2ws4t4@bsd-mbp.dhcp.thefacebook.com>
On Mon, Jul 27, 2020 at 10:01 AM Jonathan Lemon
<jonathan.lemon@gmail.com> wrote:
>
> On Mon, Jul 27, 2020 at 08:24:55AM -0700, Eric Dumazet wrote:
> > On Mon, Jul 27, 2020 at 12:20 AM Jonathan Lemon
> > <jonathan.lemon@gmail.com> wrote:
> > >
> > > This could likely be moved elsewhere. The presence of the flag on
> > > the skb indicates that one of the fragments may contain zerocopy
> > > RX data, where the data is not accessible to the cpu.
> >
> > Why do we need yet another flag in skb exactly ?
> >
> > Please define what means "data not accessible to the cpu" ?
> >
> > This kind of change is a red flag for me.
>
> The architecture this is targeting is a ML cluster, where a 200Gbps NIC
> is attached to a PCIe switch which also has a GPU card attached. There
> are several of these, and the link(s) to the host cpu (which has another
> NIC attached) can't handle the incoming traffic.
>
> So what we're doing here is transferring the data directly from the NIC
> to the GPU via DMA. The host never sees the data, but can control it
> indirectly via the handles returned to userspace.
>
This seems to need a page/memory attribute or something.
skb should not have this knowledge, unless you are planning to make
sure that everything accessing skb data is going to test this new flag
and fail if it is set ?
> I'm not sure that a flag on the skb is the right location for this -
> perhaps moving it into skb_shared() instead would be better?
> --
> Jonathan
next prev parent reply other threads:[~2020-07-27 17:08 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20200727052846.4070247-1-jonathan.lemon@gmail.com>
[not found] ` <20200727052846.4070247-4-jonathan.lemon@gmail.com>
2020-07-27 5:58 ` [RFC PATCH v2 03/21] mm: Allow DMA mapping of pages which are not online Christoph Hellwig
2020-07-27 17:08 ` Jonathan Lemon
[not found] ` <20200727052846.4070247-22-jonathan.lemon@gmail.com>
2020-07-27 7:35 ` [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu Christoph Hellwig
2020-07-27 17:00 ` Jonathan Lemon
2020-07-27 18:24 ` Christoph Hellwig
2020-07-28 1:48 ` Jonathan Lemon
2020-07-28 6:47 ` Christoph Hellwig
2020-07-28 16:05 ` Jonathan Lemon
2020-07-28 16:10 ` Christoph Hellwig
2020-07-28 18:19 ` Jason Gunthorpe
2020-07-28 21:01 ` Jonathan Lemon
2020-07-28 21:14 ` Christoph Hellwig
2020-07-28 23:38 ` Jason Gunthorpe
[not found] ` <20200727052846.4070247-14-jonathan.lemon@gmail.com>
2020-07-27 15:19 ` [RFC PATCH v2 13/21] net/tcp: Pad TCP options out to a fixed size " Eric Dumazet
2020-07-27 17:20 ` Jonathan Lemon
[not found] ` <20200727052846.4070247-16-jonathan.lemon@gmail.com>
2020-07-27 15:19 ` [RFC PATCH v2 15/21] net/tcp: add MSG_NETDMA flag for sendmsg() Eric Dumazet
2020-07-27 15:55 ` Jonathan Lemon
2020-07-27 16:09 ` Eric Dumazet
2020-07-27 17:35 ` Jonathan Lemon
2020-07-27 17:44 ` Eric Dumazet
2020-07-28 2:11 ` Jonathan Lemon
2020-07-28 2:17 ` Eric Dumazet
2020-07-28 3:08 ` Jonathan Lemon
2020-07-28 6:50 ` Christoph Hellwig
[not found] ` <20200727052846.4070247-9-jonathan.lemon@gmail.com>
2020-07-27 15:24 ` [RFC PATCH v2 08/21] skbuff: add a zc_netgpu bitflag Eric Dumazet
2020-07-27 16:59 ` Jonathan Lemon
2020-07-27 17:08 ` Eric Dumazet [this message]
2020-07-27 17:16 ` Jonathan Lemon
2020-07-27 22:44 [RFC PATCH v2 00/21] netgpu: networking between NIC and GPU/CPU Jonathan Lemon
2020-07-27 22:44 ` [RFC PATCH v2 08/21] skbuff: add a zc_netgpu bitflag Jonathan Lemon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CANn89iJBLrmvtTmyBAwErejCEMdt6gLNNRX1XZ1oMorAWrbp4g@mail.gmail.com \
--to=edumazet@google.com \
--cc=akpm@linux-foundation.org \
--cc=bjorn.topel@intel.com \
--cc=borisp@mellanox.com \
--cc=davem@davemloft.net \
--cc=david@redhat.com \
--cc=hch@lst.de \
--cc=jonathan.lemon@gmail.com \
--cc=kernel-team@fb.com \
--cc=kuba@kernel.org \
--cc=magnus.karlsson@intel.com \
--cc=maximmi@mellanox.com \
--cc=netdev@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=saeedm@mellanox.com \
--cc=steffen.klassert@secunet.com \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).