[PATCH] doc: enic SR-IOV configurations

[PATCH] doc: enic SR-IOV configurations

[John Daley]

Document SR-IOV passthrough setup and limitations
for enic PMD.

Signed-off-by: John Daley <johndale@cisco.com>
---
 doc/guides/nics/enic.rst | 94 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)

diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst
index c535b589c..8f8ae1f9c 100644
--- a/doc/guides/nics/enic.rst
+++ b/doc/guides/nics/enic.rst
@@ -140,6 +140,83 @@ Masking of these feilds for partial match is also supported.
 Without advanced filter support, the flow director is limited to IPv4
 perfect filtering of the 5-tuple with no masking of fields supported.
 
+SR-IOV mode utilization
+-----------------------
+
+UCS blade servers configured with dynamic vNIC connection policies in UCS
+manager are capable of supporting assigned devices on virtual machines (VMs)
+through a KVM hypervisor. Assigned devices, also known as 'passthrough'
+devices, are SR-IOV virtual functions (VFs) on the host which are exposed
+to VM instances.
+
+The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a dedicated
+interface on the Fabric Interconnect (FI). Layer 2 switching is done at
+the FI. This may eliminate the requirement for software switching on the
+host to route intra-host VM traffic.
+
+Please refer to `Creating a Dynamic vNIC Connection Policy
+<http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/vmware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E01651F69464783A68E66DA8A47A5>`_
+for information on configuring SR-IOV Adapter policies using UCS manager.
+
+Once the policies are in place and the host OS is reboot, VFs should be visible
+on the host, E.g.:
+
+.. code-block:: c
+
+     # lspci | grep Cisco | grep Ethernet
+     0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+     0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+
+Enable Intel IOMMU on the host and install KVM and libvirt. A VM instance should
+be created with an assigned device. When using libvirt, this configuration can
+be done within the domain (i.e. VM) config file. For example this entry maps
+host VF 0d:00:01 into the VM.
+
+.. code-block:: c
+
+    <interface type='hostdev' managed='yes'>
+      <mac address='52:54:00:ac:ff:b6'/>
+      <source>
+        <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
+      </source>
+
+This configuration method is explained in more detail
+`here <https://libvirt.org/formatdomain.html#elementsNICS>`_.
+Alternatively, the configuration can be done in a separate file using the
+``network`` keyword. This method is described here:
+`https://libvirt.org/formatnetwork.html <https://libvirt.org/formatnetwork.html>`_
+
+When the VM instance is started, the enic KVM driver will bind the host VF to
+vfio, complete provisioning on the FI and bring up the link.
+
+.. note::
+
+    It is not possible to use a VF directly from the host because it is not
+    fully provisioned until the hypervisor brings up the VM that it is assigned
+    to.
+
+In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is
+seen on the VM instance and should be available for binding to a DPDK.
+
+.. code-block:: c
+
+     # lspci | grep Ether
+     00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+
+Follow the normal DPDK install proceedure, binding the VF to either ``igb_uio``
+or ``vfio`` in non-IOMMU mode.
+
+Please see :ref:`Limitations <enic_limitations>` for limitations in
+the use of SR-IOV.
+
+.. _enic_limitations:
+
 Limitations
 -----------
 
@@ -169,6 +246,23 @@ Limitations
 - Flow director features are not supported on generation 1 Cisco VIC adapters
   (M81KR and P81E)
 
+- **SR-IOV**
+
+  - KVM hypervisor support only. VMware has not been tested.
+  - Requires VM-FEX, and so is only available on UCS managed servers connected
+    to Fabric Interrconnects. It is not on standalone C-Series servers.
+  - VF devices are not usable directly from the host. They can  only be used
+    as assigned devices on VM instances.
+  - Currently, unbind of the enic kernel mode driver 'enic' on the VM instance
+    may hang. As a workaround, enic.ko should blacklisted or removed from the
+    boot process.
+  - pci_generic cannot be used as the uio module in the VM. igb_uio or
+    vfio in non-IOMMU mode can be used.
+  - The number of RQs in UCSM dynamic vNIC configurations must be at least 2.
+  - The number of SR-IOV devices is limited to 256. Components on target system
+    might limit this number to fewer than 256.
+
+
 How to build the suite?
 -----------------------
 The build instructions for the DPDK suite should be followed. By default
-- 
2.12.0

Re: [PATCH] doc: enic SR-IOV configurations

[Ferruh Yigit]

On 4/26/2017 6:56 AM, John Daley wrote:
> Document SR-IOV passthrough setup and limitations
> for enic PMD.
> 
> Signed-off-by: John Daley <johndale@cisco.com>

Hi John,

Thank you for the documentation update, just I am getting following
warning [1], can you please check.

Also enic feature table (enic.ini) is not announcing SR-IOV support, can
you please fix it?

Thanks,
ferruh

[1]
doc/guides/nics/enic.rst:181: WARNING: Could not lex literal_block as
"c". Highlighting skipped.

Re: [PATCH] doc: enic SR-IOV configurations

[Mcnamara, John]

> -----Original Message-----
> From: John Daley [mailto:johndale@cisco.com]
> Sent: Wednesday, April 26, 2017 6:56 AM
> To: Mcnamara, John <john.mcnamara@intel.com>
> Cc: dev@dpdk.org; John Daley <johndale@cisco.com>
> Subject: [PATCH] doc: enic SR-IOV configurations
> 
> Document SR-IOV passthrough setup and limitations for enic PMD.
> 


Hi John,

Thanks for the doc updates. Some comments below.


>  Without advanced filter support, the flow director is limited to IPv4
> perfect filtering of the 5-tuple with no masking of fields supported.
> 
> +SR-IOV mode utilization
> +-----------------------
> +
> +UCS blade servers configured with dynamic vNIC connection policies in
> +UCS manager are capable of supporting assigned devices on virtual
> +machines (VMs) through a KVM hypervisor. Assigned devices, also known as
> 'passthrough'
> +devices, are SR-IOV virtual functions (VFs) on the host which are
> +exposed to VM instances.
> +
> +The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a
> +dedicated interface on the Fabric Interconnect (FI). Layer 2 switching
> +is done at the FI. This may eliminate the requirement for software
> +switching on the host to route intra-host VM traffic.
> +
> +Please refer to `Creating a Dynamic vNIC Connection Policy
> +<http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/v
> +mware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_G
> +UI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E016
> +51F69464783A68E66DA8A47A5>`_ for information on configuring SR-IOV
> +Adapter policies using UCS manager.
> +
> +Once the policies are in place and the host OS is reboot, VFs should be

s/reboot/rebooted/

> +visible on the host, E.g.:
> +
> +.. code-block:: c

This should be "code-block:: console" or just plain "::" since it defaults
to console.


> +
> +.. code-block:: c
> +
> +    <interface type='hostdev' managed='yes'>
> +      <mac address='52:54:00:ac:ff:b6'/>
> +      <source>
> +        <address type='pci' domain='0x0000' bus='0x0d' slot='0x00'
> function='0x1'/>
> +      </source>
> +


This should be "code-block:: xml". Using "c" causes the parse warning
that Ferruh pointed out.
 
> +This configuration method is explained in more detail `here
> +<https://libvirt.org/formatdomain.html#elementsNICS>`_.
> +Alternatively, the configuration can be done in a separate file using
> +the ``network`` keyword. This method is described here:
> +`https://libvirt.org/formatnetwork.html
> +<https://libvirt.org/formatnetwork.html>`_

If the url target and text are the same then you don't have to repeat it:

    `<https://libvirt.org/formatnetwork.html>`_


> +
> +When the VM instance is started, the enic KVM driver will bind the host

s/enic/ENIC/ ? Here and elsewhere.


> +Follow the normal DPDK install proceedure, binding the VF to either
> +``igb_uio`` or ``vfio`` in non-IOMMU mode.

s/procedure/procedure/



> 
> +- **SR-IOV**
> +
> +  - KVM hypervisor support only. VMware has not been tested.
> +  - Requires VM-FEX, and so is only available on UCS managed servers
> connected
> +    to Fabric Interrconnects. It is not on standalone C-Series servers.


s/Interrconnects/Interconnects/


> +  - VF devices are not usable directly from the host. They can  only be
> used
> +    as assigned devices on VM instances.
> +  - Currently, unbind of the enic kernel mode driver 'enic' on the VM
> instance


> +    may hang. As a workaround, enic.ko should blacklisted or removed from
> the
> +    boot process.
> +  - pci_generic cannot be used as the uio module in the VM. igb_uio or
> +    vfio in non-IOMMU mode can be used.
> +  - The number of RQs in UCSM dynamic vNIC configurations must be at
> least 2.
> +  - The number of SR-IOV devices is limited to 256. Components on target
> system
> +    might limit this number to fewer than 256.
> +
> +
>  How to build the suite?
>  -----------------------
>  The build instructions for the DPDK suite should be followed. By default


There was a new chapter added recently in the NICs section that has generic
build and testing instructions for PMDs. So you could change this section to
something like the following:

 
How to build the suite
----------------------

Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
for details.

By default the ENIC PMD library will be built into the DPDK library.

For configuring and using UIO and VFIO frameworks, please refer the
documentation that comes with DPDK suite.



Regards,

John

Re: [PATCH] doc: enic SR-IOV configurations

[Mcnamara, John]

> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Wednesday, April 26, 2017 10:31 AM
> To: John Daley <johndale@cisco.com>; Mcnamara, John
> <john.mcnamara@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] doc: enic SR-IOV configurations
> 
> On 4/26/2017 6:56 AM, John Daley wrote:
> > Document SR-IOV passthrough setup and limitations for enic PMD.
> >
> > Signed-off-by: John Daley <johndale@cisco.com>
> 
> Hi John,
> 
> Thank you for the documentation update, just I am getting following
> warning [1], can you please check.
> 
> ...
> doc/guides/nics/enic.rst:181: WARNING: Could not lex literal_block as "c".
> Highlighting skipped.

Hi,

To catch issues like this could we add a "make doc" test to the automated patch tests.

Thanks,

John

[PATCH v2] doc: enic SR-IOV configurations

[John Daley]

Document SR-IOV passthrough setup and limitations
for enic PMD.

Signed-off-by: John Daley <johndale@cisco.com>
---
v2 - fix typos, URL targets and formatting in SR-IOV section.

 doc/guides/nics/enic.rst          | 93 +++++++++++++++++++++++++++++++++++++++
 doc/guides/nics/features/enic.ini |  1 +
 2 files changed, 94 insertions(+)

diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst
index c535b589c..f744151f5 100644
--- a/doc/guides/nics/enic.rst
+++ b/doc/guides/nics/enic.rst
@@ -140,6 +140,81 @@ Masking of these feilds for partial match is also supported.
 Without advanced filter support, the flow director is limited to IPv4
 perfect filtering of the 5-tuple with no masking of fields supported.
 
+SR-IOV mode utilization
+-----------------------
+
+UCS blade servers configured with dynamic vNIC connection policies in UCS
+manager are capable of supporting assigned devices on virtual machines (VMs)
+through a KVM hypervisor. Assigned devices, also known as 'passthrough'
+devices, are SR-IOV virtual functions (VFs) on the host which are exposed
+to VM instances.
+
+The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a dedicated
+interface on the Fabric Interconnect (FI). Layer 2 switching is done at
+the FI. This may eliminate the requirement for software switching on the
+host to route intra-host VM traffic.
+
+Please refer to `Creating a Dynamic vNIC Connection Policy
+<http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/vmware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E01651F69464783A68E66DA8A47A5>`_
+for information on configuring SR-IOV Adapter policies using UCS manager.
+
+Once the policies are in place and the host OS is rebooted, VFs should be
+visible on the host, E.g.:
+
+.. code-block:: console
+
+     # lspci | grep Cisco | grep Ethernet
+     0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+     0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+
+Enable Intel IOMMU on the host and install KVM and libvirt. A VM instance should
+be created with an assigned device. When using libvirt, this configuration can
+be done within the domain (i.e. VM) config file. For example this entry maps
+host VF 0d:00:01 into the VM.
+
+.. code-block:: console
+
+    <interface type='hostdev' managed='yes'>
+      <mac address='52:54:00:ac:ff:b6'/>
+      <source>
+        <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
+      </source>
+
+Alternatively, the configuration can be done in a separate file using the
+``network`` keyword. These methods are described in the libvirt documentation for
+`Network XML format <https://libvirt.org/formatnetwork.html>`_.
+
+When the VM instance is started, the enic KVM driver will bind the host VF to
+vfio, complete provisioning on the FI and bring up the link.
+
+.. note::
+
+    It is not possible to use a VF directly from the host because it is not
+    fully provisioned until the hypervisor brings up the VM that it is assigned
+    to.
+
+In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is
+seen on the VM instance and should be available for binding to a DPDK.
+
+.. code-block:: console
+
+     # lspci | grep Ether
+     00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+
+Follow the normal DPDK install procedure, binding the VF to either ``igb_uio``
+or ``vfio`` in non-IOMMU mode.
+
+Please see :ref:`Limitations <enic_limitations>` for limitations in
+the use of SR-IOV.
+
+.. _enic_limitations:
+
 Limitations
 -----------
 
@@ -169,6 +244,23 @@ Limitations
 - Flow director features are not supported on generation 1 Cisco VIC adapters
   (M81KR and P81E)
 
+- **SR-IOV**
+
+  - KVM hypervisor support only. VMware has not been tested.
+  - Requires VM-FEX, and so is only available on UCS managed servers connected
+    to Fabric Interconnects. It is not on standalone C-Series servers.
+  - VF devices are not usable directly from the host. They can  only be used
+    as assigned devices on VM instances.
+  - Currently, unbind of the enic kernel mode driver 'enic' on the VM instance
+    may hang. As a workaround, enic.ko should blacklisted or removed from the
+    boot process.
+  - pci_generic cannot be used as the uio module in the VM. igb_uio or
+    vfio in non-IOMMU mode can be used.
+  - The number of RQs in UCSM dynamic vNIC configurations must be at least 2.
+  - The number of SR-IOV devices is limited to 256. Components on target system
+    might limit this number to fewer than 256.
+
+
 How to build the suite?
 -----------------------
 The build instructions for the DPDK suite should be followed. By default
@@ -216,6 +308,7 @@ Supported features
 - IPV4, IPV6 and TCP RSS hashing
 - Scattered Rx
 - MTU update
+- SR-IOV on UCS managed servers connected to Fabric Interconnects.
 
 Known bugs and Unsupported features in this release
 ---------------------------------------------------
diff --git a/doc/guides/nics/features/enic.ini b/doc/guides/nics/features/enic.ini
index 51b330fa8..94e7f3cba 100644
--- a/doc/guides/nics/features/enic.ini
+++ b/doc/guides/nics/features/enic.ini
@@ -15,6 +15,7 @@ Promiscuous mode     = Y
 Unicast MAC filter   = Y
 Multicast MAC filter = Y
 RSS hash             = Y
+SR-IOV               = Y
 VLAN filter          = Y
 CRC offload          = Y
 VLAN offload         = Y
-- 
2.12.0

Re: [PATCH v2] doc: enic SR-IOV configurations

[Mcnamara, John]

> -----Original Message-----
> From: John Daley [mailto:johndale@cisco.com]
> Sent: Thursday, April 27, 2017 1:52 AM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>
> Cc: dev@dpdk.org; John Daley <johndale@cisco.com>
> Subject: [PATCH v2] doc: enic SR-IOV configurations
> 
> Document SR-IOV passthrough setup and limitations for enic PMD.
> 
> Signed-off-by: John Daley <johndale@cisco.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2] doc: enic SR-IOV configurations

[Ferruh Yigit]

On 4/27/2017 11:12 AM, Mcnamara, John wrote:
> 
> 
>> -----Original Message-----
>> From: John Daley [mailto:johndale@cisco.com]
>> Sent: Thursday, April 27, 2017 1:52 AM
>> To: Yigit, Ferruh <ferruh.yigit@intel.com>; Mcnamara, John
>> <john.mcnamara@intel.com>
>> Cc: dev@dpdk.org; John Daley <johndale@cisco.com>
>> Subject: [PATCH v2] doc: enic SR-IOV configurations
>>
>> Document SR-IOV passthrough setup and limitations for enic PMD.
>>
>> Signed-off-by: John Daley <johndale@cisco.com>
> 
> Acked-by: John McNamara <john.mcnamara@intel.com>

Applied to dpdk-next-net/master, thanks.