From: Alexander Duyck <alexander.duyck@gmail.com>
To: Parav Pandit <parav@nvidia.com>
Cc: David Ahern <dsahern@gmail.com>, Jason Gunthorpe <jgg@nvidia.com>,
Saeed Mahameed <saeed@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Leon Romanovsky <leonro@nvidia.com>,
Netdev <netdev@vger.kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
David Ahern <dsahern@kernel.org>,
Jacob Keller <jacob.e.keller@intel.com>,
Sridhar Samudrala <sridhar.samudrala@intel.com>,
"Ertman, David M" <david.m.ertman@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
Kiran Patil <kiran.patil@intel.com>,
Greg KH <gregkh@linuxfoundation.org>
Subject: Re: [net-next v4 00/15] Add mlx5 subfunction support
Date: Fri, 18 Dec 2020 08:01:28 -0800 [thread overview]
Message-ID: <CAKgT0UdtEJ0Xe5icMOSj0dg-unEgTR8AwDrtdAWTKEH4D-0www@mail.gmail.com> (raw)
In-Reply-To: <BY5PR12MB43220950B3A93B9E548976C7DCC30@BY5PR12MB4322.namprd12.prod.outlook.com>
On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Alexander Duyck <alexander.duyck@gmail.com>
> > Sent: Friday, December 18, 2020 8:41 AM
> >
> > On Thu, Dec 17, 2020 at 5:30 PM David Ahern <dsahern@gmail.com> wrote:
> > >
> > > On 12/16/20 3:53 PM, Alexander Duyck wrote:
> > The problem is PCIe DMA wasn't designed to function as a network switch
> > fabric and when we start talking about a 400Gb NIC trying to handle over 256
> > subfunctions it will quickly reduce the receive/transmit throughput to gigabit
> > or less speeds when encountering hardware multicast/broadcast replication.
> > With 256 subfunctions a simple 60B ARP could consume more than 19KB of
> > PCIe bandwidth due to the packet having to be duplicated so many times. In
> > my mind it should be simpler to simply clone a single skb 256 times, forward
> > that to the switchdev ports, and have them perform a bypass (if available) to
> > deliver it to the subfunctions. That's why I was thinking it might be a good
> > time to look at addressing it.
> Linux tc framework is rich to address this and already used by openvswich for years now.
> Today arp broadcasts are not offloaded. They go through software path and replicated in the L2 domain.
> It is a solved problem for many years now.
When you say they are replicated in the L2 domain I assume you are
talking about the software switch connected to the switchdev ports. My
question is what are you doing with them after you have replicated
them? I'm assuming they are being sent to the other switchdev ports
which will require a DMA to transmit them, and another to receive them
on the VF/SF, or are you saying something else is going on here?
My argument is that this cuts into both the transmit and receive DMA
bandwidth of the NIC, and could easily be avoided in the case where SF
exists in the same kernel as the switchdev port by identifying the
multicast bit being set and simply bypassing the device.
next prev parent reply other threads:[~2020-12-18 16:02 UTC|newest]
Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-14 21:43 [net-next v4 00/15] Add mlx5 subfunction support Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 01/15] net/mlx5: Fix compilation warning for 32-bit platform Saeed Mahameed
2020-12-14 22:31 ` Alexander Duyck
2020-12-14 22:45 ` Saeed Mahameed
2020-12-15 4:59 ` Leon Romanovsky
2020-12-14 21:43 ` [net-next v4 02/15] devlink: Prepare code to fill multiple port function attributes Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 03/15] devlink: Introduce PCI SF port flavour and port attribute Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 04/15] devlink: Support add and delete devlink port Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 05/15] devlink: Support get and set state of port function Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 06/15] net/mlx5: Introduce vhca state event notifier Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 07/15] net/mlx5: SF, Add auxiliary device support Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 08/15] net/mlx5: SF, Add auxiliary device driver Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 09/15] net/mlx5: E-switch, Prepare eswitch to handle SF vport Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 10/15] net/mlx5: E-switch, Add eswitch helpers for " Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 11/15] net/mlx5: SF, Add port add delete functionality Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 12/15] net/mlx5: SF, Port function state change support Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 13/15] devlink: Add devlink port documentation Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 14/15] devlink: Extend devlink port documentation for subfunctions Saeed Mahameed
2020-12-14 21:43 ` [net-next v4 15/15] net/mlx5: Add devlink subfunction port documentation Saeed Mahameed
2020-12-15 1:53 ` [net-next v4 00/15] Add mlx5 subfunction support Alexander Duyck
2020-12-15 2:44 ` David Ahern
2020-12-15 16:16 ` Alexander Duyck
2020-12-15 16:59 ` Parav Pandit
2020-12-15 5:48 ` Parav Pandit
2020-12-15 18:47 ` Alexander Duyck
2020-12-15 20:05 ` Saeed Mahameed
2020-12-15 21:03 ` Jason Gunthorpe
2020-12-16 1:12 ` Edwin Peer
2020-12-16 2:39 ` Jason Gunthorpe
2020-12-16 3:12 ` Alexander Duyck
2020-12-15 20:59 ` David Ahern
2020-12-15 6:15 ` Saeed Mahameed
2020-12-15 19:12 ` Alexander Duyck
2020-12-15 20:35 ` Saeed Mahameed
2020-12-15 21:28 ` Jakub Kicinski
2020-12-16 6:50 ` Leon Romanovsky
2020-12-16 17:59 ` Saeed Mahameed
2020-12-15 21:41 ` Alexander Duyck
2020-12-16 0:19 ` Jason Gunthorpe
2020-12-16 2:19 ` Alexander Duyck
2020-12-16 3:03 ` Jason Gunthorpe
2020-12-16 4:13 ` Alexander Duyck
2020-12-16 4:45 ` Parav Pandit
2020-12-16 13:33 ` Jason Gunthorpe
2020-12-16 16:31 ` Alexander Duyck
2020-12-16 17:51 ` Jason Gunthorpe
2020-12-16 19:27 ` Alexander Duyck
2020-12-16 20:35 ` Jason Gunthorpe
2020-12-16 22:53 ` Alexander Duyck
2020-12-17 0:38 ` Jason Gunthorpe
2020-12-17 18:48 ` Alexander Duyck
2020-12-17 19:40 ` Jason Gunthorpe
2020-12-17 21:05 ` Alexander Duyck
2020-12-18 0:08 ` Jason Gunthorpe
2020-12-18 1:30 ` David Ahern
2020-12-18 3:11 ` Alexander Duyck
2020-12-18 3:55 ` David Ahern
2020-12-18 15:54 ` Alexander Duyck
2020-12-18 5:20 ` Parav Pandit
2020-12-18 5:36 ` Parav Pandit
2020-12-18 16:01 ` Alexander Duyck [this message]
2020-12-18 18:01 ` Parav Pandit
2020-12-18 19:22 ` Alexander Duyck
2020-12-18 20:18 ` Jason Gunthorpe
2020-12-19 0:03 ` Alexander Duyck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAKgT0UdtEJ0Xe5icMOSj0dg-unEgTR8AwDrtdAWTKEH4D-0www@mail.gmail.com \
--to=alexander.duyck@gmail.com \
--cc=dan.j.williams@intel.com \
--cc=davem@davemloft.net \
--cc=david.m.ertman@intel.com \
--cc=dsahern@gmail.com \
--cc=dsahern@kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=jacob.e.keller@intel.com \
--cc=jgg@nvidia.com \
--cc=kiran.patil@intel.com \
--cc=kuba@kernel.org \
--cc=leonro@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=parav@nvidia.com \
--cc=saeed@kernel.org \
--cc=sridhar.samudrala@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.