From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jingjing Wu Subject: =?utf-8?q?=5BPATCH_v2_1/2=5D_doc=3A_add_bifurcated_dri?= =?utf-8?q?ver_guide_on_ixgbe_nic?= Date: Thu, 23 Jun 2016 11:02:26 +0800 Message-ID: <1466650946-22523-1-git-send-email-jingjing.wu@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: dev@dpdk.org, jingjing.wu@intel.com, yong.liu@intel.com, helin.zhang@intel.com To: john.mcnamara@intel.com Return-path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 737E49AD3 for ; Thu, 23 Jun 2016 05:02:35 +0200 (CEST) List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Bifurcated driver is a mechanism which depends the advanced Ethernet device to split traffic between queues. It provides the capability to let the kernel driver and DPDK driver co-exist and take their advantage. It is achieved by using SRIOV and NIC's advanced filtering. This patch describes it and adds the user guide on ixgbe NICs. Signed-off-by: Jingjing Wu --- doc/guides/nics/img/bifurcated_driver_overview.svg | 544 +++++++++++++++= ++++++ doc/guides/nics/img/ixgbe_bifu_queue_idx.svg | 101 ++++ doc/guides/nics/ixgbe.rst | 119 +++++ 3 files changed, 764 insertions(+) create mode 100644 doc/guides/nics/img/bifurcated_driver_overview.svg create mode 100644 doc/guides/nics/img/ixgbe_bifu_queue_idx.svg diff --git a/doc/guides/nics/img/bifurcated_driver_overview.svg b/doc/gui= des/nics/img/bifurcated_driver_overview.svg new file mode 100644 index 0000000..4fa2764 --- /dev/null +++ b/doc/guides/nics/img/bifurcated_driver_overview.svg @@ -0,0 +1,544 @@ + + + +image/svg+xmlPage-1Sheet.85NICNIC +NIC +Rounded Rectangle.20LINUXLINUX +LINUX +Rounded Rectangle.8Kernel pf driverKernel pf driver +Rounded RectangleFilters support traffic steering to VFFilters support traffic +steering to VF +Rectangle.3Rx Queues (0-N) PFRx Queues +( 0-N ) + PF +Rectangle.4Rx Queues (0-M) VF(vf 0)Rx Queues +( 0-M ) +VF(vf0) +Rectangle.5filtersfilters +Rounded Rectangle.6Tools to program filtersTools to +program filters +2-D word balloonDirector flows to queue index in specified VFinspecified VF +Director flows +to queue index +in specified VF +Rounded Rectangle.24DPDKDPDK +Rounded Rectangle.25SocketSocket +Simple Arrow.44Single arrowheadDynamic connector.70Dynamic connector.81Dynamic connector.83Dynamic connector.84Sheet.98Sheet.109Sheet.110 \ No newline at end of file diff --git a/doc/guides/nics/img/ixgbe_bifu_queue_idx.svg b/doc/guides/ni= cs/img/ixgbe_bifu_queue_idx.svg new file mode 100644 index 0000000..f7e2bd8 --- /dev/null +++ b/doc/guides/nics/img/ixgbe_bifu_queue_idx.svg @@ -0,0 +1,101 @@ + + + + + + + + + + + + + + + Page-1 + + + Rectangle + 0x000000 + + + + + + + + + + 0x000000 + + Rectangle.2 + VF ID + 1 + + + + + + + + + + VF ID + 1 + + Rectangle.3 + Queue Index + + + + + + + + + + Queue Index + + Sheet.4 + 0 + + + + 0 + + Sheet.6 + 31 + + + + 31 + + Sheet.7 + 39 + + + + 39 + + Sheet.8 + 63 + + + + 63 + + diff --git a/doc/guides/nics/ixgbe.rst b/doc/guides/nics/ixgbe.rst index 3dc6b00..74ec4b7 100644 --- a/doc/guides/nics/ixgbe.rst +++ b/doc/guides/nics/ixgbe.rst @@ -224,3 +224,122 @@ be calculated as follows: max_read_interval =3D ~4 mins 48 sec. =20 In order to ensure valid results, it is recommended to poll every 4 minu= tes. + +.. _bifurcated_driver: + +Bifurcated driver guide +----------------------- + +Bifurcated driver is a mechanism which depends the advanced Ethernet dev= ice to +split traffic between Linux user space and kernel space. Because it is h= ardware +assisted design, this approach can provide the line rate processing capa= bility. +Other than KNI, the SW is just required to device configuration, no need= to +take care of the packet movement during the traffic split. This can get = more +performance with less CPU overhead. + +The Bifurcated driver take advantage of Ethernet device feature to split= the +incoming data traffic to user space application (Such as DPDK applicatio= n) +and/or kernel space program (Linux kernel stack). It can direct some tra= ffic +(e.g data plane traffic) to DPDK, while direct some other traffic (e.g c= ontrol +plane traffic) to the traditional Linux networking stack. + +There are a number of technical options to achieve this. A typical examp= le is +to combine the technology of SR-IOV and packet classification filtering. + +SR-IOV is a PCI standard that allows the same physical adapter to split = as +multiple virtual functions. Each virtual function has separated queues w= ith +physical function. Traffic with a virtual function=E2=80=99s destination= MAC address +from network adapter will be directed to it. In a sense, SR-IOV has the +capability on queue division. + +Packet classification filtering is the hardware capability available on = most +network adapters. Filters can be configured to direct specific flows to = a given +receive queue by hardware. Different NIC may have different filter types= to +direct flow to a Virtual Function or a queue belong to it. + +Linux network can receive the specific traffic through kernel driver, wh= ile +DPDK can receive the specific traffic bypassing the Linux kernel by usin= g +drivers like VFIO or DPDK igb_uio module. + +.. _figure_bifurcated_driver_overview: + +.. figure:: img/bifurcated_driver_overview.* + + Bifurcated Driver Overview + +Use Bifurcated driver on IXGBE in Linux +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On Intel=C2=AE 82599 10 Gigabit Ethernet Controller series NICs, Bifurca= ted driver +can be achieved by SR-IOV and flow director technologies. So the traffic= can +be directed to queues by flow director capability, typically by matching= 5-tuple +of UDP/TCP packets. + +The step procedure is as following: + +#. Boot system without iommu, or with =E2=80=9Ciommu=3Dpt=E2=80=9D. + +#. Create Virtual Functions: + + .. code-block:: console + + echo 2 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs + +#. Enable and set flow filters: + + .. code-block:: console + + ethtool -K eth1 ntuple on + ethtool -N eth1 flow-type udp4 src-ip 192.0.2.2 dst-ip 198.51.10= 0.2 \ + action $queue_index_in_VF0 + ethtool -N eth1 flow-type udp4 src-ip 198.51.100.2 dst-ip 192.0.= 2.2 \ + action $queue_index_in_VF1 + + where: + + * $queue_index_in_PF: [queue index] + + * $queue_index_in_VFn: Bits 39:32 of the variable defines VF i= d + 1; lower 32 bits indicates the queue index of VF. + + * $queue_index_in_VF0 =3D (0x1 & 0xFF) << 32 + [queue inde= x]; + + * $queue_index_in_VF1 =3D (0x2 & 0xFF) << 32 + [queue inde= x]; + + .. _figure_ixgbe_bifu_queue_idx: + + .. figure:: img/ixgbe_bifu_queue_idx.* + +#. Compile the DPDK and insert igb_uio or probe vfio-pci kernel modules= as normal. + +#. Bind virtual function: + + .. code-block:: console + + modprobe vfio-pci + dpdk_nic_bind.py -b vfio-pci 01:10.0 + dpdk_nic_bind.py -b vfio-pci 01:10.1 + +#. run DPDK application on VFs: + + .. code-block:: console + + testpmd -c 0xff -n 4 -- -i -w 01:10.0 -w 01:10.1 --forward-mode=3D= mac + +In this example, traffic matching the rules will go through VF by matchi= ng the +filter rule. All other traffic which mismatching the rules, will go thro= ugh +the default queue or scaling on queues in PF. That is to say UDP packets= with +those IP source and destination addresses will go through the DPDK. All = other +traffic, with different hosts or different protocols, will go through th= e Linux +networking stack. + +.. note:: + + * The above steps work on the Linux kernel v4.2. + + * The Bifurcated driver is implemented in Linux kernel and ixgbe k= ernel driver by following patches: + + * `ethtool: Add helper routines to pass vf to rx_flow_spec `_ + + * `ixgbe: Allow flow director to use entire queue space `_ + + * Ethtool's version used in this example is 3.18. \ No newline at end of file --=20 2.4.0