From: Parav Pandit <parav@mellanox.com>
To: alex.williamson@redhat.com, davem@davemloft.net,
kvm@vger.kernel.org, netdev@vger.kernel.org
Cc: saeedm@mellanox.com, kwankhede@nvidia.com, leon@kernel.org,
cohuck@redhat.com, jiri@mellanox.com, linux-rdma@vger.kernel.org,
Parav Pandit <parav@mellanox.com>
Subject: [PATCH net-next 18/19] Documentation: net: mlx5: Add mdev usage documentation
Date: Thu, 7 Nov 2019 10:08:33 -0600 [thread overview]
Message-ID: <20191107160834.21087-18-parav@mellanox.com> (raw)
In-Reply-To: <20191107160834.21087-1-parav@mellanox.com>
Reviewed-by: Saeed Mahameed <saeedm@mellanox.com>
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
.../device_drivers/mellanox/mlx5.rst | 122 ++++++++++++++++++
1 file changed, 122 insertions(+)
diff --git a/Documentation/networking/device_drivers/mellanox/mlx5.rst b/Documentation/networking/device_drivers/mellanox/mlx5.rst
index d071c6b49e1f..cbdf0a37205b 100644
--- a/Documentation/networking/device_drivers/mellanox/mlx5.rst
+++ b/Documentation/networking/device_drivers/mellanox/mlx5.rst
@@ -14,6 +14,7 @@ Contents
- `Devlink parameters`_
- `Devlink health reporters`_
- `mlx5 tracepoints`_
+- `Mediated devices`_
Enabling the driver and kconfig options
================================================
@@ -97,6 +98,10 @@ Enabling the driver and kconfig options
| Provides low-level InfiniBand/RDMA and `RoCE <https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment>`_ support.
+**CONFIG_MLX5_MDEV(y/n)** (module mlx5_core.ko)
+
+| Provides support for Sub Functions using mediated devices.
+
**External options** ( Choose if the corresponding mlx5 feature is required )
@@ -298,3 +303,120 @@ tc and eswitch offloads tracepoints:
$ cat /sys/kernel/debug/tracing/trace
...
kworker/u48:7-2221 [009] ...1 1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1
+
+Mediated devices
+================
+
+Overview
+--------
+mlx5 mediated device (mdev) enables users to create multiple netdevices
+and/or RDMA devices from single PCI function.
+
+Each mdev maps to a mlx5 sub function.
+mlx5 sub function is similar to PCI VF. However it doesn't have its own
+PCI function and MSI-X vectors.
+mlx5 sub function has several less low level device capabilities
+as compare to PCI function.
+
+Each mlx5 sub function has its own resource namespace for RDMA resources.
+
+mlx5 mdevs share common PCI resources such as PCI BAR region,
+MSI-X interrupts.
+
+Each mdev has its own window in the PCI BAR region, which is
+accessible only to that mdev and applications using it.
+
+mdevs are supported when eswitch mode of the devlink instance
+is in switchdev mode described in 'http://man7.org/linux/man-pages/man8/devlink-dev.8.html'.
+
+mdev uses mediated device subsystem 'https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt' of the kernel for its life cycle.
+
+mdev is identified using a UUID defined by RFC 4122.
+
+Each created mdev has unique 12 letters alias. This alias is used to
+derive phys_port_name attribute of the corresponding representor
+netdevice.
+
+User commands examples
+----------------------
+
+- Set eswitch mode as switchdev mode::
+
+ $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
+
+- Create a mdev::
+
+ Generate a UUID
+ $ UUID=$(uuidgen)
+ Create the mdev using UUID
+ $ echo $UUID > /sys/class/net/ens2f0_p0/device/mdev_supported_types/mlx5_core-local/create
+
+- Unbind a mdev from vfio_mdev driver::
+
+ $ echo $UUID > /sys/bus/mdev/drivers/vfio_mdev/unbind
+
+- Bind a mdev to mlx5_core driver::
+
+ $ echo $UUID > /sys/bus/mdev/drivers/mlx5_core/bind
+
+- View netdevice and (optionally) RDMA device in sysfs tree::
+
+ $ ls -l /sys/bus/mdev/devices/$UUID/net/
+ $ ls -l /sys/bus/mdev/devices/$UUID/infiniband/
+
+- View netdevice and (optionally) RDMA device using iproute2 tools::
+
+ $ ip link show
+ $ rdma dev show
+
+- Query maximum number of mdevs that can be created::
+
+ $ cat /sys/class/net/ens2f0_p0/device/mdev_supported_types/mlx5_core-local/max_mdevs
+
+- Query remaining number of mdevs that can be created::
+
+ $ cat /sys/class/net/ens2f0_p0/device/mdev_supported_types/mlx5_core-local/available_instances
+
+- Query an alias of the mdev::
+
+ $ cat /sys/bus/mdev/devices/$UUID/alias
+
+Security model
+--------------
+This section covers security aspects of mlx5 mediated devices at
+host level and at network level.
+
+Host side:
+- At present mlx5 mdev is meant to be used only in a host.
+It is not meant to be mapped to a VM or access by userspace application
+using VFIO framework.
+Hence, mlx5_core driver doesn't implement any of the VFIO device specific
+callback routines.
+Hence, mlx5 mediated device cannot be mapped to a VM or to a userspace
+application via VFIO framework.
+
+- At present an mlx5 mdev can be accessed by an application through
+its netdevice and/or RDMA device.
+
+- mlx5 mdev does not share PCI BAR with its parent PCI function.
+
+- All mlx5 mdevs of a given parent device share a single PCI BAR.
+However each mdev device has a small dedicated window of the PCI BAR.
+Hence, one mdev device cannot access PCI BAR or any of the resources
+of another mdev device.
+
+- Each mlx5 mdev has its own dedicated event queue through which interrupt
+notifications are delivered. Hence, one mlx5 mdev cannot enable/disable
+interrupts of other mlx5 mdev. mlx5 mdev cannot enable/disable interrupts
+of the parent PCI function.
+
+Network side:
+- By default the netdevice and the rdma device of mlx5 mdev cannot send or
+receive any packets over the network or to any other mlx5 mdev.
+
+- mlx5 mdev follows devlink eswitch and vport model of PCI SR-IOV PF and VFs.
+All traffic is dropped by default in this eswitch model.
+
+- Each mlx5 mdev has one eswitch vport representor netdevice and rdma port.
+The user must do necessary configuration through such representor to enable
+mlx5 mdev to send and/or receive packets.
--
2.19.2
next prev parent reply other threads:[~2019-11-07 16:09 UTC|newest]
Thread overview: 132+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-07 16:04 [PATCH net-next 00/19] Mellanox, mlx5 sub function support Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 01/19] net/mlx5: E-switch, Move devlink port close to eswitch port Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 02/19] net/mlx5: E-Switch, Add SF vport, vport-rep support Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 03/19] net/mlx5: Introduce SF table framework Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 04/19] net/mlx5: Introduce SF life cycle APIs to allocate/free Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 05/19] net/mlx5: E-Switch, Enable/disable SF's vport during SF life cycle Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 06/19] net/mlx5: Add support for mediated devices in switchdev mode Parav Pandit
2019-11-08 10:32 ` Jiri Pirko
2019-11-08 16:03 ` Parav Pandit
2019-11-08 16:22 ` Jiri Pirko
2019-11-08 16:29 ` Parav Pandit
2019-11-08 18:01 ` Jiri Pirko
2019-11-08 18:04 ` Jiri Pirko
2019-11-08 18:21 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 07/19] vfio/mdev: Introduce sha1 based mdev alias Parav Pandit
2019-11-08 11:04 ` Jiri Pirko
2019-11-08 15:59 ` Parav Pandit
2019-11-08 16:28 ` Jiri Pirko
2019-11-08 11:10 ` Cornelia Huck
2019-11-08 16:03 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 08/19] vfio/mdev: Make mdev alias unique among all mdevs Parav Pandit
2019-11-08 10:49 ` Jiri Pirko
2019-11-08 15:13 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 09/19] vfio/mdev: Expose mdev alias in sysfs tree Parav Pandit
2019-11-08 13:22 ` Jiri Pirko
2019-11-08 18:03 ` Alex Williamson
2019-11-08 18:16 ` Jiri Pirko
2019-11-07 16:08 ` [PATCH net-next 10/19] vfio/mdev: Introduce an API mdev_alias Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 11/19] vfio/mdev: Improvise mdev life cycle and parent removal scheme Parav Pandit
2019-11-08 13:01 ` Cornelia Huck
2019-11-08 16:12 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 12/19] devlink: Introduce mdev port flavour Parav Pandit
2019-11-07 20:38 ` Jakub Kicinski
2019-11-07 21:03 ` Parav Pandit
2019-11-08 1:17 ` Jakub Kicinski
2019-11-08 1:44 ` Parav Pandit
2019-11-08 2:20 ` Jakub Kicinski
2019-11-08 2:31 ` Parav Pandit
2019-11-08 9:46 ` Jiri Pirko
2019-11-08 15:45 ` Parav Pandit
2019-11-08 16:31 ` Jiri Pirko
2019-11-08 16:43 ` Parav Pandit
2019-11-08 18:11 ` Jiri Pirko
2019-11-08 18:23 ` Parav Pandit
2019-11-08 18:34 ` Jiri Pirko
2019-11-08 18:56 ` Parav Pandit
2019-11-08 9:30 ` Jiri Pirko
2019-11-08 15:41 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 13/19] net/mlx5: Register SF devlink port Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 14/19] net/mlx5: Share irqs between SFs and parent PCI device Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 15/19] net/mlx5: Add load/unload routines for SF driver binding Parav Pandit
2019-11-08 9:48 ` Jiri Pirko
2019-11-08 11:13 ` Jiri Pirko
2019-11-07 16:08 ` [PATCH net-next 16/19] net/mlx5: Implement dma ops and params for mediated device Parav Pandit
2019-11-07 20:42 ` Jakub Kicinski
2019-11-07 21:30 ` Parav Pandit
2019-11-08 1:16 ` Jakub Kicinski
2019-11-08 6:37 ` Christoph Hellwig
2019-11-08 15:29 ` Parav Pandit
2019-11-07 16:08 ` [PATCH net-next 17/19] net/mlx5: Add mdev driver to bind to mdev devices Parav Pandit
2019-11-07 16:08 ` Parav Pandit [this message]
2019-11-07 16:08 ` [PATCH net-next 19/19] mtty: Optionally support mtty alias Parav Pandit
2019-11-08 6:26 ` Leon Romanovsky
2019-11-08 10:45 ` Jiri Pirko
2019-11-08 15:08 ` Parav Pandit
2019-11-08 15:15 ` Jiri Pirko
2019-11-08 13:46 ` Cornelia Huck
2019-11-08 15:10 ` Parav Pandit
2019-11-08 15:28 ` Cornelia Huck
2019-11-08 15:30 ` Parav Pandit
2019-11-08 17:54 ` Alex Williamson
2019-11-08 9:51 ` [PATCH net-next 01/19] net/mlx5: E-switch, Move devlink port close to eswitch port Jiri Pirko
2019-11-08 15:50 ` Parav Pandit
2019-11-07 17:03 ` [PATCH net-next 00/19] Mellanox, mlx5 sub function support Leon Romanovsky
2019-11-07 20:10 ` Parav Pandit
2019-11-08 6:20 ` Leon Romanovsky
2019-11-08 15:01 ` Parav Pandit
2019-11-07 20:32 ` Jakub Kicinski
2019-11-07 20:52 ` Parav Pandit
2019-11-08 1:16 ` Jakub Kicinski
2019-11-08 1:49 ` Parav Pandit
2019-11-08 2:12 ` Jakub Kicinski
2019-11-08 12:12 ` Jiri Pirko
2019-11-08 14:40 ` Jason Gunthorpe
2019-11-08 15:40 ` Parav Pandit
2019-11-08 19:12 ` Jakub Kicinski
2019-11-08 20:12 ` Jason Gunthorpe
2019-11-08 20:20 ` Parav Pandit
2019-11-08 20:32 ` Jason Gunthorpe
2019-11-08 20:52 ` gregkh
2019-11-08 20:34 ` Alex Williamson
2019-11-08 21:05 ` Jason Gunthorpe
2019-11-08 21:19 ` gregkh
2019-11-08 21:52 ` Alex Williamson
2019-11-08 22:48 ` Parav Pandit
2019-11-09 0:57 ` Jason Gunthorpe
2019-11-09 17:41 ` Jakub Kicinski
2019-11-10 19:04 ` Jason Gunthorpe
2019-11-10 19:48 ` Parav Pandit
2019-11-11 14:17 ` Jiri Pirko
2019-11-11 14:58 ` Parav Pandit
2019-11-11 15:06 ` Jiri Pirko
2019-11-19 4:51 ` Parav Pandit
2019-11-09 0:12 ` Jason Gunthorpe
2019-11-09 0:45 ` Parav Pandit
2019-11-11 2:19 ` Jason Wang
2019-11-08 21:45 ` Jakub Kicinski
2019-11-09 0:44 ` Jason Gunthorpe
2019-11-09 8:46 ` gregkh
2019-11-09 11:18 ` Jiri Pirko
2019-11-09 17:28 ` Jakub Kicinski
2019-11-10 9:16 ` gregkh
2019-11-09 17:27 ` Jakub Kicinski
2019-11-10 9:18 ` gregkh
2019-11-11 3:46 ` Jakub Kicinski
2019-11-11 5:18 ` Parav Pandit
2019-11-11 13:30 ` Jiri Pirko
2019-11-11 14:14 ` gregkh
2019-11-11 14:37 ` Jiri Pirko
2019-11-10 19:37 ` Jason Gunthorpe
2019-11-11 3:57 ` Jakub Kicinski
2019-11-08 16:06 ` Parav Pandit
2019-11-08 19:06 ` Jakub Kicinski
2019-11-08 19:34 ` Parav Pandit
2019-11-08 19:48 ` Jakub Kicinski
2019-11-08 19:41 ` Jiri Pirko
2019-11-08 20:40 ` Parav Pandit
2019-11-08 21:21 ` Jakub Kicinski
2019-11-08 21:39 ` Jiri Pirko
2019-11-08 21:51 ` Jakub Kicinski
2019-11-08 22:21 ` Jiri Pirko
2019-11-07 23:57 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191107160834.21087-18-parav@mellanox.com \
--to=parav@mellanox.com \
--cc=alex.williamson@redhat.com \
--cc=cohuck@redhat.com \
--cc=davem@davemloft.net \
--cc=jiri@mellanox.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).