From: Parav Pandit <parav@mellanox.com>
To: Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
"davem@davemloft.net" <davem@davemloft.net>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
Saeed Mahameed <saeedm@mellanox.com>,
"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
"leon@kernel.org" <leon@kernel.org>,
"cohuck@redhat.com" <cohuck@redhat.com>,
Jiri Pirko <jiri@mellanox.com>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
Or Gerlitz <gerlitz.or@gmail.com>
Subject: RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
Date: Fri, 8 Nov 2019 01:49:09 +0000 [thread overview]
Message-ID: <AM0PR05MB48668E29F1811F496602079ED17B0@AM0PR05MB4866.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <20191107201627.68728686@cakuba>
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Thursday, November 7, 2019 7:16 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: alex.williamson@redhat.com; davem@davemloft.net;
> kvm@vger.kernel.org; netdev@vger.kernel.org; Saeed Mahameed
> <saeedm@mellanox.com>; kwankhede@nvidia.com; leon@kernel.org;
> cohuck@redhat.com; Jiri Pirko <jiri@mellanox.com>; linux-
> rdma@vger.kernel.org; Or Gerlitz <gerlitz.or@gmail.com>
> Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
>
> On Thu, 7 Nov 2019 20:52:29 +0000, Parav Pandit wrote:
> > > On Thu, 7 Nov 2019 10:04:48 -0600, Parav Pandit wrote:
> > > > Mellanox sub function capability allows users to create several
> > > > hundreds of networking and/or rdma devices without depending on
> > > > PCI SR-
> > > IOV support.
> > >
> > > You call the new port type "sub function" but the devlink port flavour is
> mdev.
> > >
> > Sub function is the internal driver structure. The abstract entity at user and
> stack level is mdev.
> > Hence the port flavour is mdev.
>
> FWIW I agree mdev as flavour seems like the right choice.
>
Ok.
> > > As I'm sure you remember you nacked my patches exposing NFP's PCI
> > > sub functions which are just regions of the BAR without any mdev
> > > capability. Am I in the clear to repost those now? Jiri?
> > >
> > For sure I didn't nack it. :-)
>
> Well, maybe the word "nack" wasn't exactly used :)
>
> > What I remember discussing offline/mailing list is
> > (a) exposing mdev/sub fuctions as devlink sub ports is not so good
> > abstraction
> > (b) user creating/deleting eswitch sub ports would be hard to fit in
> > the whole usage model
>
> Okay, so I can repost the "basic" sub functions?
>
I think so. Would you like post on top of this series as port flavour etc would come by default?
Also there is vfio/mdev dependency exist in this series...
> > > > Overview:
> > > > ---------
> > > > Mellanox ConnectX sub functions are exposed to user as a mediated
> > > > device (mdev) [2] as discussed in RFC [3] and further during
> > > > netdevconf0x13 at [4].
> > > >
> > > > mlx5 mediated device (mdev) enables users to create multiple
> > > > netdevices and/or RDMA devices from single PCI function.
> > > >
> > > > Each mdev maps to a mlx5 sub function.
> > > > mlx5 sub function is similar to PCI VF. However it doesn't have
> > > > its own PCI function and MSI-X vectors.
> > > >
> > > > mlx5 mdevs share common PCI resources such as PCI BAR region,
> > > > MSI-X interrupts.
> > > >
> > > > Each mdev has its own window in the PCI BAR region, which is
> > > > accessible only to that mdev and applications using it.
> > > >
> > > > Each mlx5 sub function has its own resource namespace for RDMA
> resources.
> > > >
> > > > mdevs are supported when eswitch mode of the devlink instance is
> > > > in switchdev mode described in devlink documentation [5].
> > >
> > > So presumably the mdevs don't spawn their own devlink instance
> > > today, but once mapped via VIRTIO to a VM they will create one?
> > >
> > mdev doesn't spawn the devlink instance today when mdev is created by
> user, like PCI.
> > When PCI bus driver enumerates and creates PCI device, there isn't a
> devlink instance for it.
> >
> > But, mdev's devlink instance is created when mlx5_core driver binds to the
> mdev device.
> > (again similar to PCI, when mlx5_core driver binds to PCI, its devlink
> instance is created ).
> >
> > I should have put the example in patch-15 which creates/deletes devlink
> instance of mdev.
> > I will revise the commit log of patch-15 to include that.
> > Good point.
>
> Thanks.
>
> > > It could be useful to specify.
> > >
> > Yes, its certainly useful. I missed to put the example in commit log of
> patch-15.
> >
> > > > Network side:
> > > > - By default the netdevice and the rdma device of mlx5 mdev cannot
> > > > send or receive any packets over the network or to any other mlx5
> mdev.
> > >
> > > Does this mean the frames don't fall back to the repr by default?
> > Probably I wasn't clear.
> > What I wanted to say is, that frames transmitted by mdev's netdevice and
> rdma devices don't go to network.
> > These frames goes to representor device.
> > User must configure representor to send/receive/steer traffic to mdev.
>
> 👍
next prev parent reply other threads:[~2019-11-08 1:49 UTC|newest]
Thread overview: 132+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-07 16:04 [PATCH net-next 00/19] Mellanox, mlx5 sub function support Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 01/19] net/mlx5: E-switch, Move devlink port close to eswitch port Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 02/19] net/mlx5: E-Switch, Add SF vport, vport-rep support Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 03/19] net/mlx5: Introduce SF table framework Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 04/19] net/mlx5: Introduce SF life cycle APIs to allocate/free Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 05/19] net/mlx5: E-Switch, Enable/disable SF's vport during SF life cycle Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 06/19] net/mlx5: Add support for mediated devices in switchdev mode Parav Pandit
2019-11-08 10:32 ` Jiri Pirko
2019-11-08 16:03 ` Parav Pandit
2019-11-08 16:22 ` Jiri Pirko
2019-11-08 16:29 ` Parav Pandit
2019-11-08 18:01 ` Jiri Pirko
2019-11-08 18:04 ` Jiri Pirko
2019-11-08 18:21 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 07/19] vfio/mdev: Introduce sha1 based mdev alias Parav Pandit
2019-11-08 11:04 ` Jiri Pirko
2019-11-08 15:59 ` Parav Pandit
2019-11-08 16:28 ` Jiri Pirko
2019-11-08 11:10 ` Cornelia Huck
2019-11-08 16:03 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 08/19] vfio/mdev: Make mdev alias unique among all mdevs Parav Pandit
2019-11-08 10:49 ` Jiri Pirko
2019-11-08 15:13 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 09/19] vfio/mdev: Expose mdev alias in sysfs tree Parav Pandit
2019-11-08 13:22 ` Jiri Pirko
2019-11-08 18:03 ` Alex Williamson
2019-11-08 18:16 ` Jiri Pirko
2019-11-07 16:08 ` [PATCH net-next 10/19] vfio/mdev: Introduce an API mdev_alias Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 11/19] vfio/mdev: Improvise mdev life cycle and parent removal scheme Parav Pandit
2019-11-08 13:01 ` Cornelia Huck
2019-11-08 16:12 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 12/19] devlink: Introduce mdev port flavour Parav Pandit
2019-11-07 20:38 ` Jakub Kicinski
2019-11-07 21:03 ` Parav Pandit
2019-11-08 1:17 ` Jakub Kicinski
2019-11-08 1:44 ` Parav Pandit
2019-11-08 2:20 ` Jakub Kicinski
2019-11-08 2:31 ` Parav Pandit
2019-11-08 9:46 ` Jiri Pirko
2019-11-08 15:45 ` Parav Pandit
2019-11-08 16:31 ` Jiri Pirko
2019-11-08 16:43 ` Parav Pandit
2019-11-08 18:11 ` Jiri Pirko
2019-11-08 18:23 ` Parav Pandit
2019-11-08 18:34 ` Jiri Pirko
2019-11-08 18:56 ` Parav Pandit
2019-11-08 9:30 ` Jiri Pirko
2019-11-08 15:41 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 13/19] net/mlx5: Register SF devlink port Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 14/19] net/mlx5: Share irqs between SFs and parent PCI device Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 15/19] net/mlx5: Add load/unload routines for SF driver binding Parav Pandit
2019-11-08 9:48 ` Jiri Pirko
2019-11-08 11:13 ` Jiri Pirko
2019-11-07 16:08 ` [PATCH net-next 16/19] net/mlx5: Implement dma ops and params for mediated device Parav Pandit
2019-11-07 20:42 ` Jakub Kicinski
2019-11-07 21:30 ` Parav Pandit
2019-11-08 1:16 ` Jakub Kicinski
2019-11-08 6:37 ` Christoph Hellwig
2019-11-08 15:29 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 17/19] net/mlx5: Add mdev driver to bind to mdev devices Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 18/19] Documentation: net: mlx5: Add mdev usage documentation Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 19/19] mtty: Optionally support mtty alias Parav Pandit
2019-11-08 6:26 ` Leon Romanovsky
2019-11-08 10:45 ` Jiri Pirko
2019-11-08 15:08 ` Parav Pandit
2019-11-08 15:15 ` Jiri Pirko
2019-11-08 13:46 ` Cornelia Huck
2019-11-08 15:10 ` Parav Pandit
2019-11-08 15:28 ` Cornelia Huck
2019-11-08 15:30 ` Parav Pandit
2019-11-08 17:54 ` Alex Williamson
2019-11-08 9:51 ` [PATCH net-next 01/19] net/mlx5: E-switch, Move devlink port close to eswitch port Jiri Pirko
2019-11-08 15:50 ` Parav Pandit
2019-11-07 17:03 ` [PATCH net-next 00/19] Mellanox, mlx5 sub function support Leon Romanovsky
2019-11-07 20:10 ` Parav Pandit
2019-11-08 6:20 ` Leon Romanovsky
2019-11-08 15:01 ` Parav Pandit
2019-11-07 20:32 ` Jakub Kicinski
2019-11-07 20:52 ` Parav Pandit
2019-11-08 1:16 ` Jakub Kicinski
2019-11-08 1:49 ` Parav Pandit [this message]
2019-11-08 2:12 ` Jakub Kicinski
2019-11-08 12:12 ` Jiri Pirko
2019-11-08 14:40 ` Jason Gunthorpe
2019-11-08 15:40 ` Parav Pandit
2019-11-08 19:12 ` Jakub Kicinski
2019-11-08 20:12 ` Jason Gunthorpe
2019-11-08 20:20 ` Parav Pandit
2019-11-08 20:32 ` Jason Gunthorpe
2019-11-08 20:52 ` gregkh
2019-11-08 20:34 ` Alex Williamson
2019-11-08 21:05 ` Jason Gunthorpe
2019-11-08 21:19 ` gregkh
2019-11-08 21:52 ` Alex Williamson
2019-11-08 22:48 ` Parav Pandit
2019-11-09 0:57 ` Jason Gunthorpe
2019-11-09 17:41 ` Jakub Kicinski
2019-11-10 19:04 ` Jason Gunthorpe
2019-11-10 19:48 ` Parav Pandit
2019-11-11 14:17 ` Jiri Pirko
2019-11-11 14:58 ` Parav Pandit
2019-11-11 15:06 ` Jiri Pirko
2019-11-19 4:51 ` Parav Pandit
2019-11-09 0:12 ` Jason Gunthorpe
2019-11-09 0:45 ` Parav Pandit
2019-11-11 2:19 ` Jason Wang
2019-11-08 21:45 ` Jakub Kicinski
2019-11-09 0:44 ` Jason Gunthorpe
2019-11-09 8:46 ` gregkh
2019-11-09 11:18 ` Jiri Pirko
2019-11-09 17:28 ` Jakub Kicinski
2019-11-10 9:16 ` gregkh
2019-11-09 17:27 ` Jakub Kicinski
2019-11-10 9:18 ` gregkh
2019-11-11 3:46 ` Jakub Kicinski
2019-11-11 5:18 ` Parav Pandit
2019-11-11 13:30 ` Jiri Pirko
2019-11-11 14:14 ` gregkh
2019-11-11 14:37 ` Jiri Pirko
2019-11-10 19:37 ` Jason Gunthorpe
2019-11-11 3:57 ` Jakub Kicinski
2019-11-08 16:06 ` Parav Pandit
2019-11-08 19:06 ` Jakub Kicinski
2019-11-08 19:34 ` Parav Pandit
2019-11-08 19:48 ` Jakub Kicinski
2019-11-08 19:41 ` Jiri Pirko
2019-11-08 20:40 ` Parav Pandit
2019-11-08 21:21 ` Jakub Kicinski
2019-11-08 21:39 ` Jiri Pirko
2019-11-08 21:51 ` Jakub Kicinski
2019-11-08 22:21 ` Jiri Pirko
2019-11-07 23:57 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=AM0PR05MB48668E29F1811F496602079ED17B0@AM0PR05MB4866.eurprd05.prod.outlook.com \
--to=parav@mellanox.com \
--cc=alex.williamson@redhat.com \
--cc=cohuck@redhat.com \
--cc=davem@davemloft.net \
--cc=gerlitz.or@gmail.com \
--cc=jakub.kicinski@netronome.com \
--cc=jiri@mellanox.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).