[PATCH v2 00/18] net/nfp: add PF support

[PATCH v2 00/18] net/nfp: add PF support

[Alejandro Lucero]

NFP PMD has just had support for SRIOV VFs until now. This patch set adds
support for the PF, but just for being used as another DPDK port. No VF
management is added by now.

NFP is a programmable device and it supports virtual NICs (vNICs) through
firmware implementation. Different firmware applications implement vNICs
for PF devices and VF devices, being number of vNICs dependent on the
firmware and the NFP card available. PF vNIC (virtual) BARs are a subset
of PF PCI device BARs while VF vNIC BARs are same than VF PCI BARs.

Working with VF vNICs requires a PF driver uploading the firmware, doing
some NFP configuration and creating/destroying VFs. This can be only done
with the kernel NFP PF netdev driver by now.

Working with PF vNIC requires the PMD doing the NFP configuration and for
accessing the NFP a specific user space interface is created. NFP Service
Processor Userspace (NSPU) interface allows to create specific PCI BAR
windows for accessing different parts of the NFP device, including the
Network Service Processor (NSP) itself. The NSPU interface is implemented
as the base for working with the PF.

v2:
 - fix commits headers
 - update nic guide
 - add info to release notes
 - creating different features files for PF and VF
 - fix typo inside nfp_nspu.c

Alejandro Lucero (18):
  net/nfp: add NSP user space interface
  net/nfp: add specific PF probe function
  net/nfp: add support for new PCI id
  net/nfp: add NSP support for commands
  net/nfp: add NSP FW upload command
  net/nfp: add NSP symbol resolution command
  net/nfp: add FW upload logic
  net/nfp: add support for vnic config bar mapping
  net/nfp: add support for vNIC Rx/Tx bar mappings
  net/nfp: support PF devices inside PMD initialization
  net/nfp: allocate ethernet device from PF probe function
  net/nfp: support PF multiport
  net/nfp: add NSP support for HW link configuration
  net/nfp: add support for HW port link configuration
  net/nfp: read PF port MAC addr using NSP
  doc: update NFP with PF support information
  doc: update release notes with NFP PF support
  doc: create different features files for NFP drivers

 doc/guides/nics/features/nfp.ini       |  29 --
 doc/guides/nics/features/nfp_pf.ini    |  28 ++
 doc/guides/nics/features/nfp_vf.ini    |  28 ++
 doc/guides/nics/nfp.rst                |  77 +++-
 doc/guides/rel_notes/release_17_11.rst |   9 +
 drivers/net/nfp/Makefile               |   2 +
 drivers/net/nfp/nfp_net.c              | 382 +++++++++++++++++++-
 drivers/net/nfp/nfp_net_ctrl.h         |   3 +
 drivers/net/nfp/nfp_net_eth.h          |  82 +++++
 drivers/net/nfp/nfp_net_pmd.h          |   8 +
 drivers/net/nfp/nfp_nfpu.c             | 103 ++++++
 drivers/net/nfp/nfp_nfpu.h             |  55 +++
 drivers/net/nfp/nfp_nspu.c             | 623 +++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h             |  83 +++++
 14 files changed, 1445 insertions(+), 67 deletions(-)
 delete mode 100644 doc/guides/nics/features/nfp.ini
 create mode 100644 doc/guides/nics/features/nfp_pf.ini
 create mode 100644 doc/guides/nics/features/nfp_vf.ini
 create mode 100644 drivers/net/nfp/nfp_net_eth.h
 create mode 100644 drivers/net/nfp/nfp_nfpu.c
 create mode 100644 drivers/net/nfp/nfp_nfpu.h
 create mode 100644 drivers/net/nfp/nfp_nspu.c
 create mode 100644 drivers/net/nfp/nfp_nspu.h

-- 
1.9.1

[PATCH v2 01/18] net/nfp: add NSP user space interface

[Alejandro Lucero]

Working with the PF requires access to the NFP for basic configuration.
NSP is the NFP Service Processor helping with hardware and firmware
configuration. NSPU is the NSP user space interface for working with the
NSP.

Configuration through NSPU allows to create PCI BAR windows for accessing
different NFP hardware units, including the BAR window for the NSPU
interface access itself. NFP expansion bar registers are used for creating
those PCI BAR windows. NSPU uses a specific expansion bar which is
reprogrammed for accessing/doing different things.

Other expansion bars will be configured later for configuring the PF vNIC
bars, a subset of PF PCI BARs.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/Makefile   |   2 +
 drivers/net/nfp/nfp_nfpu.c | 103 ++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nfpu.h |  55 +++++++++++++++++++
 drivers/net/nfp/nfp_nspu.c | 129 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h |  71 +++++++++++++++++++++++++
 5 files changed, 360 insertions(+)
 create mode 100644 drivers/net/nfp/nfp_nfpu.c
 create mode 100644 drivers/net/nfp/nfp_nfpu.h
 create mode 100644 drivers/net/nfp/nfp_nspu.c
 create mode 100644 drivers/net/nfp/nfp_nspu.h

diff --git a/drivers/net/nfp/Makefile b/drivers/net/nfp/Makefile
index 4ee2c2d..3e4c6f4 100644
--- a/drivers/net/nfp/Makefile
+++ b/drivers/net/nfp/Makefile
@@ -49,5 +49,7 @@ LIBABIVER := 1
 # all source are stored in SRCS-y
 #
 SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_net.c
+SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_nfpu.c
+SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_nspu.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/net/nfp/nfp_nfpu.c b/drivers/net/nfp/nfp_nfpu.c
new file mode 100644
index 0000000..556ded3
--- /dev/null
+++ b/drivers/net/nfp/nfp_nfpu.c
@@ -0,0 +1,103 @@
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <sys/file.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <sys/types.h>
+
+#include <rte_pci.h>
+#include <rte_malloc.h>
+
+#include "nfp_nfpu.h"
+
+/* PF BAR and expansion BAR for the NSP interface */
+#define NFP_CFG_PCIE_BAR        0
+#define NFP_CFG_EXP_BAR         7
+
+#define NFP_CFG_EXP_BAR_CFG_BASE	0x30000
+
+/* There could be other NFP userspace tools using the NSP interface.
+ * Make sure there is no other process using it and locking the access for
+ * avoiding problems.
+ */
+static int
+nspv_aquire_process_lock(nfpu_desc_t *desc)
+{
+	int rc;
+	struct flock lock;
+	char lockname[30];
+
+	memset(&lock, 0, sizeof(lock));
+
+	snprintf(lockname, sizeof(lockname), "/var/lock/nfp%d", desc->nfp);
+
+	/* Using S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH */
+	desc->lock = open(lockname, O_RDWR | O_CREAT, 0666);
+
+	if (desc->lock < 0)
+		return desc->lock;
+
+	lock.l_type = F_WRLCK;
+	lock.l_whence = SEEK_SET;
+	rc = -1;
+	while (rc != 0) {
+		rc = fcntl(desc->lock, F_SETLK, &lock);
+		if (rc < 0) {
+			if ((errno != EAGAIN) && (errno != EACCES)) {
+				close(desc->lock);
+				return rc;
+			}
+		}
+	}
+
+	return 0;
+}
+
+int
+nfpu_open(struct rte_pci_device *pci_dev, nfpu_desc_t *desc, int nfp)
+{
+	void *cfg_base, *mem_base;
+	size_t barsz;
+	int ret = 0;
+	int i = 0;
+
+	desc->nfp = nfp;
+
+	ret = nspv_aquire_process_lock(desc);
+	if (ret)
+		return -1;
+
+	barsz = pci_dev->mem_resource[0].len;
+
+	/* barsz in log2 */
+	while (barsz >>= 1)
+		i++;
+	barsz = i;
+
+	/* Getting address for NFP expansion BAR registers */
+	cfg_base = pci_dev->mem_resource[0].addr;
+	cfg_base = (uint8_t *)cfg_base + NFP_CFG_EXP_BAR_CFG_BASE;
+
+	/* Getting address for NFP NSP interface registers */
+	mem_base = pci_dev->mem_resource[0].addr;
+	mem_base = (uint8_t *)mem_base + (NFP_CFG_EXP_BAR << (barsz - 3));
+
+
+	desc->nspu = rte_malloc("nfp nspu", sizeof(nspu_desc_t), 0);
+	nfp_nspu_init(desc->nspu, desc->nfp, NFP_CFG_PCIE_BAR, barsz,
+		      NFP_CFG_EXP_BAR, cfg_base, mem_base);
+
+	return ret;
+}
+
+int
+nfpu_close(nfpu_desc_t *desc)
+{
+	rte_free(desc->nspu);
+	close(desc->lock);
+	unlink("/var/lock/nfp0");
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nfpu.h b/drivers/net/nfp/nfp_nfpu.h
new file mode 100644
index 0000000..31511b3
--- /dev/null
+++ b/drivers/net/nfp/nfp_nfpu.h
@@ -0,0 +1,55 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_nfpu.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+/*
+ * NFP User interface creates a window for talking with NFP NSP processor
+ */
+
+
+#include <rte_pci.h>
+#include "nfp_nspu.h"
+
+typedef struct {
+	int nfp;
+	int lock;
+	nspu_desc_t *nspu;
+} nfpu_desc_t;
+
+int nfpu_open(struct rte_pci_device *pci_dev, nfpu_desc_t *desc, int nfp);
+int nfpu_close(nfpu_desc_t *desc);
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
new file mode 100644
index 0000000..a157915
--- /dev/null
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -0,0 +1,129 @@
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/types.h>
+
+#include <rte_log.h>
+
+#include "nfp_nfpu.h"
+
+#define CFG_EXP_BAR_ADDR_SZ     1
+#define CFG_EXP_BAR_MAP_TYPE	1
+
+#define EXP_BAR_TARGET_SHIFT     23
+#define EXP_BAR_LENGTH_SHIFT     27 /* 0=32, 1=64 bit increment */
+#define EXP_BAR_MAP_TYPE_SHIFT   29 /* Bulk BAR map */
+
+/* NFP target for NSP access */
+#define NFP_NSP_TARGET   7
+
+/*
+ * This is an NFP internal address used for configuring properly an NFP
+ * expansion BAR.
+ */
+#define MEM_CMD_BASE_ADDR       0x8100000000
+
+/* NSP interface registers */
+#define NSP_BASE                (MEM_CMD_BASE_ADDR + 0x22100)
+#define NSP_STATUS              0x00
+#define NSP_COMMAND             0x08
+#define NSP_BUFFER		0x10
+#define NSP_DEFAULT_BUFFER      0x18
+#define NSP_DEFAULT_BUFFER_CFG  0x20
+
+#define NSP_MAGIC                0xab10
+#define NSP_STATUS_MAGIC(x)      (((x) >> 48) & 0xffff)
+#define NSP_STATUS_MAJOR(x)      (int)(((x) >> 44) & 0xf)
+#define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
+
+#define NSP_REG_ADDR(d, off, reg) ((uint8_t *)(d)->mem_base + (off) + (reg))
+#define NSP_REG_VAL(p) (*(uint64_t *)(p))
+
+/*
+ * An NFP expansion BAR is configured for allowing access to a specific NFP
+ * target:
+ *
+ *  IN:
+ *	desc: struct with basic NSP addresses to work with
+ *	expbar: NFP PF expansion BAR index to configure
+ *	tgt: NFP target to configure access
+ *	addr: NFP target address
+ *
+ *  OUT:
+ *	pcie_offset: NFP PCI BAR offset to work with
+ */
+static void
+nfp_nspu_mem_bar_cfg(nspu_desc_t *desc, int expbar, int tgt,
+		     uint64_t addr, uint64_t *pcie_offset)
+{
+	uint64_t x, y, barsz;
+	uint32_t *expbar_ptr;
+
+	barsz = desc->barsz;
+
+	/*
+	 * NFP CPP address to configure. This comes from NFP 6000
+	 * datasheet document based on Bulk mapping.
+	 */
+	x = (addr >> (barsz - 3)) << (21 - (40 - (barsz - 3)));
+	x |= CFG_EXP_BAR_MAP_TYPE << EXP_BAR_MAP_TYPE_SHIFT;
+	x |= CFG_EXP_BAR_ADDR_SZ << EXP_BAR_LENGTH_SHIFT;
+	x |= tgt << EXP_BAR_TARGET_SHIFT;
+
+	/* Getting expansion bar configuration register address */
+	expbar_ptr = (uint32_t *)desc->cfg_base;
+	/* Each physical PCI BAR has 8 NFP expansion BARs */
+	expbar_ptr += (desc->pcie_bar * 8) + expbar;
+
+	/* Writing to the expansion BAR register */
+	*expbar_ptr = (uint32_t)x;
+
+	/* Getting the pcie offset to work with from userspace */
+	y = addr & ((uint64_t)(1 << (barsz - 3)) - 1);
+	*pcie_offset = y;
+}
+
+/*
+ * Configuring an expansion bar for accessing NSP userspace interface. This
+ * function configures always the same expansion bar, which implies access to
+ * previously configured NFP target is lost.
+ */
+static void
+nspu_xlate(nspu_desc_t *desc, uint64_t addr, uint64_t *pcie_offset)
+{
+	nfp_nspu_mem_bar_cfg(desc, desc->exp_bar, NFP_NSP_TARGET, addr,
+			     pcie_offset);
+}
+
+int
+nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor)
+{
+	uint64_t pcie_offset;
+	uint64_t nsp_reg;
+
+	nspu_xlate(desc, NSP_BASE, &pcie_offset);
+	nsp_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, pcie_offset, NSP_STATUS));
+
+	if (NSP_STATUS_MAGIC(nsp_reg) != NSP_MAGIC)
+		return -1;
+
+	*major = NSP_STATUS_MAJOR(nsp_reg);
+	*minor = NSP_STATUS_MINOR(nsp_reg);
+
+	return 0;
+}
+
+int
+nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
+	      int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap)
+{
+	desc->nfp = nfp;
+	desc->pcie_bar = pcie_bar;
+	desc->exp_bar = exp_bar;
+	desc->barsz = pcie_barsz;
+	desc->cfg_base = exp_bar_cfg_base;
+	desc->mem_base = exp_bar_mmap;
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
new file mode 100644
index 0000000..7a1ac91
--- /dev/null
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -0,0 +1,71 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_nspu.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+/*
+ * NSP is the NFP Service Processor. NSPU is NSP Userspace interface.
+ *
+ * NFP NSP helps with firmware/hardware configuration. NSP is another component
+ * in NFP programmable processor and accessing it from host requires to firstly
+ * configure a specific NFP PCI expansion BAR.
+ *
+ * Once access is ready, configuration can be done reading and writing
+ * from/to a specific PF PCI BAR window. This same interface will allow to
+ * create other PCI BAR windows for accessing other NFP components.
+ *
+ * This file includes low-level functions, using the NSPU interface, and high
+ * level functions, invoked by the PMD for using NSP services. This allows
+ * firmware upload, vNIC PCI BARs mapping and other low-level configurations
+ * like link setup.
+ *
+ * NSP access is done during initialization and it is not involved at all with
+ * the fast path.
+ */
+
+typedef struct {
+	int nfp;        /* NFP device */
+	int pcie_bar;   /* PF PCI BAR to work with */
+	int exp_bar;    /* Expansion BAR number used by NSPU */
+	int barsz;      /* PCIE BAR log2 size */
+	void *cfg_base; /* Expansion BARs address */
+	void *mem_base; /* NSP interface */
+} nspu_desc_t;
+
+int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
+		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
+int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
-- 
1.9.1

[PATCH v2 02/18] net/nfp: add specific PF probe function

[Alejandro Lucero]

Configuring the NFP PMD for using the PF requires access through the
NSPU interface for device configuration. This patch adds a specific probe
function for the PF which uses the NSPU interface. Just basic NSPU access
is done by now reading the NSPU ABI version.

NSPU ABI version needs to be greater than 0.19.

No ethernet port is created yet.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 75 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 69 insertions(+), 6 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index a3bf5e1..cb48fdc 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -59,6 +59,8 @@
 #include "nfp_net_logs.h"
 #include "nfp_net_ctrl.h"
 
+#include "nfp_nfpu.h"
+
 /* Prototypes */
 static void nfp_net_close(struct rte_eth_dev *dev);
 static int nfp_net_configure(struct rte_eth_dev *dev);
@@ -2632,12 +2634,63 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	return 0;
 }
 
-static const struct rte_pci_id pci_id_nfp_net_map[] = {
+static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+			    struct rte_pci_device *dev)
+{
+	nfpu_desc_t *nfpu_desc;
+	nspu_desc_t *nspu_desc;
+	int major, minor;
+
+	if (!dev)
+		return -ENODEV;
+
+	nfpu_desc = rte_malloc("nfp nfpu", sizeof(nfpu_desc_t), 0);
+	if (!nfpu_desc)
+		return -ENOMEM;
+
+	if (nfpu_open(dev, nfpu_desc, 0) < 0) {
+		RTE_LOG(ERR, PMD,
+			"nfpu_open failed\n");
+		goto nfpu_error;
+	}
+
+	nspu_desc = nfpu_desc->nspu;
+
+
+	/* Check NSP ABI version */
+	if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
+		RTE_LOG(INFO, PMD, "NFP NSP not present\n");
+		goto no_abi;
+	}
+	PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
+
+	if ((major == 0) && (minor < 20)) {
+		RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required 0.20 or higher\n");
+		goto no_abi;
+	}
+
+	/* No port is created yet */
+
+no_abi:
+	nfpu_close(nfpu_desc);
+nfpu_error:
+	rte_free(nfpu_desc);
+
+	return -ENODEV;
+}
+
+static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_PF_NIC)
 	},
 	{
+		.vendor_id = 0,
+	},
+};
+
+static const struct rte_pci_id pci_id_nfp_vf_net_map[] = {
+	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_VF_NIC)
 	},
@@ -2658,16 +2711,26 @@ static int eth_nfp_pci_remove(struct rte_pci_device *pci_dev)
 	return rte_eth_dev_pci_generic_remove(pci_dev, NULL);
 }
 
-static struct rte_pci_driver rte_nfp_net_pmd = {
-	.id_table = pci_id_nfp_net_map,
+static struct rte_pci_driver rte_nfp_net_pf_pmd = {
+	.id_table = pci_id_nfp_pf_net_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.probe = nfp_pf_pci_probe,
+	.remove = eth_nfp_pci_remove,
+};
+
+static struct rte_pci_driver rte_nfp_net_vf_pmd = {
+	.id_table = pci_id_nfp_vf_net_map,
 	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
 	.probe = eth_nfp_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
 
-RTE_PMD_REGISTER_PCI(net_nfp, rte_nfp_net_pmd);
-RTE_PMD_REGISTER_PCI_TABLE(net_nfp, pci_id_nfp_net_map);
-RTE_PMD_REGISTER_KMOD_DEP(net_nfp, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PCI(net_nfp_pf, rte_nfp_net_pf_pmd);
+RTE_PMD_REGISTER_PCI(net_nfp_vf, rte_nfp_net_vf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_nfp_pf, pci_id_nfp_pf_net_map);
+RTE_PMD_REGISTER_PCI_TABLE(net_nfp_vf, pci_id_nfp_vf_net_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_nfp_pf, "* igb_uio | uio_pci_generic | vfio");
+RTE_PMD_REGISTER_KMOD_DEP(net_nfp_vf, "* igb_uio | uio_pci_generic | vfio");
 
 /*
  * Local variables:
-- 
1.9.1

[PATCH v2 03/18] net/nfp: add support for new PCI id

[Alejandro Lucero]

A NFP PF PCI devices can have PCI ID 4000 or 6000.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 4 ++++
 drivers/net/nfp/nfp_net_pmd.h | 1 +
 2 files changed, 5 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index cb48fdc..ca481e4 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
+			       PCI_DEVICE_ID_NFP4000_PF_NIC)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_PF_NIC)
 	},
 	{
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index c6bddaa..3818130 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -42,6 +42,7 @@
 
 #define NFP_NET_PMD_VERSION "0.1"
 #define PCI_VENDOR_ID_NETRONOME         0x19ee
+#define PCI_DEVICE_ID_NFP4000_PF_NIC    0x4000
 #define PCI_DEVICE_ID_NFP6000_PF_NIC    0x6000
 #define PCI_DEVICE_ID_NFP6000_VF_NIC    0x6003
 
-- 
1.9.1

[PATCH v2 04/18] net/nfp: add NSP support for commands

[Alejandro Lucero]

NSPU interface declares a buffer controlled by the NFP NSP service
processor. It is possible to send commands to the NSP using the NSPU
and this buffer for data related to the command. A command can imply
buffer read, buffer write, both or none.

Initial command for resetting the firmware is added as well which
does not require the buffer at all.

Commands will allow firmware upload, symbol resolution and ethernet
link configuration. Future commands will allow specific offloads like
flow offloads and eBPF offload.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 181 ++++++++++++++++++++++++++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |   4 +
 2 files changed, 183 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index a157915..dbb5305 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -29,14 +29,19 @@
 #define NSP_STATUS              0x00
 #define NSP_COMMAND             0x08
 #define NSP_BUFFER		0x10
-#define NSP_DEFAULT_BUFFER      0x18
-#define NSP_DEFAULT_BUFFER_CFG  0x20
+#define NSP_DEFAULT_BUF         0x18
+#define NSP_DEFAULT_BUF_CFG  0x20
 
 #define NSP_MAGIC                0xab10
 #define NSP_STATUS_MAGIC(x)      (((x) >> 48) & 0xffff)
 #define NSP_STATUS_MAJOR(x)      (int)(((x) >> 44) & 0xf)
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
+/* NSP commands */
+#define NSP_CMD_RESET			1
+
+#define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
+
 #define NSP_REG_ADDR(d, off, reg) ((uint8_t *)(d)->mem_base + (off) + (reg))
 #define NSP_REG_VAL(p) (*(uint64_t *)(p))
 
@@ -118,12 +123,184 @@
 nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 	      int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap)
 {
+	uint64_t offset, buffaddr;
+	uint64_t nsp_reg;
+
 	desc->nfp = nfp;
 	desc->pcie_bar = pcie_bar;
 	desc->exp_bar = exp_bar;
 	desc->barsz = pcie_barsz;
+	desc->windowsz = 1 << (desc->barsz - 3);
 	desc->cfg_base = exp_bar_cfg_base;
 	desc->mem_base = exp_bar_mmap;
 
+	nspu_xlate(desc, NSP_BASE, &offset);
+
+	/*
+	 * Other NSPU clients can use other buffers. Let's tell NSPU we use the
+	 * default buffer.
+	 */
+	buffaddr = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_DEFAULT_BUF));
+	NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_BUFFER)) = buffaddr;
+
+	/* NFP internal addresses are 40 bits. Clean all other bits here */
+	buffaddr = buffaddr & (((uint64_t)1 << 40) - 1);
+	desc->bufaddr = buffaddr;
+
+	/* Lets get information about the buffer */
+	nsp_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_DEFAULT_BUF_CFG));
+
+	/* Buffer size comes in MBs. Coversion to bytes */
+	desc->buf_size = ((size_t)nsp_reg & NSP_BUFFER_CFG_SIZE_MASK) << 20;
+
 	return 0;
 }
+
+#define NSPU_NFP_BUF(addr, base, off) \
+	(*(uint64_t *)((uint8_t *)(addr)->mem_base + ((base) | (off))))
+
+#define NSPU_HOST_BUF(base, off) (*(uint64_t *)((uint8_t *)(base) + (off)))
+
+static int
+nspu_buff_write(nspu_desc_t *desc, void *buffer, size_t size)
+{
+	uint64_t pcie_offset, pcie_window_base, pcie_window_offset;
+	uint64_t windowsz = desc->windowsz;
+	uint64_t buffaddr, j, i = 0;
+	int ret = 0;
+
+	if (size > desc->buf_size)
+		return -1;
+
+	buffaddr = desc->bufaddr;
+	windowsz = desc->windowsz;
+
+	while (i < size) {
+		/* Expansion bar reconfiguration per window size */
+		nspu_xlate(desc, buffaddr + i, &pcie_offset);
+		pcie_window_base = pcie_offset & (~(windowsz - 1));
+		pcie_window_offset = pcie_offset & (windowsz - 1);
+		for (j = pcie_window_offset; ((j < windowsz) && (i < size));
+		     j += 8) {
+			NSPU_NFP_BUF(desc, pcie_window_base, j) =
+				NSPU_HOST_BUF(buffer, i);
+			i += 8;
+		}
+	}
+
+	return ret;
+}
+
+static int
+nspu_buff_read(nspu_desc_t *desc, void *buffer, size_t size)
+{
+	uint64_t pcie_offset, pcie_window_base, pcie_window_offset;
+	uint64_t windowsz, i = 0, j;
+	uint64_t buffaddr;
+	int ret = 0;
+
+	if (size > desc->buf_size)
+		return -1;
+
+	buffaddr = desc->bufaddr;
+	windowsz = desc->windowsz;
+
+	while (i < size) {
+		/* Expansion bar reconfiguration per window size */
+		nspu_xlate(desc, buffaddr + i, &pcie_offset);
+		pcie_window_base = pcie_offset & (~(windowsz - 1));
+		pcie_window_offset = pcie_offset & (windowsz - 1);
+		for (j = pcie_window_offset; ((j < windowsz) && (i < size));
+		     j += 8) {
+			NSPU_HOST_BUF(buffer, i) =
+				NSPU_NFP_BUF(desc, pcie_window_base, j);
+			i += 8;
+		}
+	}
+
+	return ret;
+}
+
+static int
+nspu_command(nspu_desc_t *desc, uint16_t cmd, int read, int write,
+		 void *buffer, size_t rsize, size_t wsize)
+{
+	uint64_t status, cmd_reg;
+	uint64_t offset;
+	int retry = 0;
+	int retries = 120;
+	int ret = 0;
+
+	/* Same expansion BAR is used for different things */
+	nspu_xlate(desc, NSP_BASE, &offset);
+
+	status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+
+	while ((status & 0x1) && (retry < retries)) {
+		status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+		retry++;
+		sleep(1);
+	}
+
+	if (retry == retries)
+		return -1;
+
+	if (write) {
+		ret = nspu_buff_write(desc, buffer, wsize);
+		if (ret)
+			return ret;
+
+		/* Expansion BAR changes when writing the buffer */
+		nspu_xlate(desc, NSP_BASE, &offset);
+	}
+
+	NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND)) =
+		(uint64_t)wsize << 32 | (uint64_t)cmd << 16 | 1;
+
+	retry = 0;
+
+	cmd_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND));
+	while ((cmd_reg & 0x1) && (retry < retries)) {
+		cmd_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND));
+		retry++;
+		sleep(1);
+	}
+	if (retry == retries)
+		return -1;
+
+	retry = 0;
+	status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+	while ((status & 0x1) && (retry < retries)) {
+		status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+		retry++;
+		sleep(1);
+	}
+
+	if (retry == retries)
+		return -1;
+
+	ret = status & (0xff << 8);
+	if (ret)
+		return ret;
+
+	if (read) {
+		ret = nspu_buff_read(desc, buffer, rsize);
+		if (ret)
+			return ret;
+	}
+
+	return ret;
+}
+
+int
+nfp_fw_reset(nspu_desc_t *nspu_desc)
+{
+	int res;
+
+	res = nspu_command(nspu_desc, NSP_CMD_RESET, 0, 0, 0, 0, 0);
+
+	if (res < 0)
+		RTE_LOG(INFO, PMD, "fw reset failed: error %d", res);
+
+	return res;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 7a1ac91..a142eb3 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -62,6 +62,9 @@
 	int pcie_bar;   /* PF PCI BAR to work with */
 	int exp_bar;    /* Expansion BAR number used by NSPU */
 	int barsz;      /* PCIE BAR log2 size */
+	uint64_t bufaddr;  /* commands buffer address */
+	size_t buf_size;   /* commands buffer size */
+	uint64_t windowsz; /* NSPU BAR window size */
 	void *cfg_base; /* Expansion BARs address */
 	void *mem_base; /* NSP interface */
 } nspu_desc_t;
@@ -69,3 +72,4 @@
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
+int nfp_fw_reset(nspu_desc_t *nspu_desc);
-- 
1.9.1

[PATCH v2 05/18] net/nfp: add NSP FW upload command

[Alejandro Lucero]

Using NSPU interface for fw upload. Firmware file needs to be
installed in specific path inside system firmware directory.

NSPU buffer is used for writing the firmware before sending the
command.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 66 +++++++++++++++++++++++++++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index dbb5305..57ee45f 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -3,6 +3,9 @@
 #include <string.h>
 #include <unistd.h>
 #include <sys/types.h>
+#include <sys/file.h>
+#include <sys/stat.h>
+#include <fcntl.h>
 
 #include <rte_log.h>
 
@@ -38,7 +41,8 @@
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
 /* NSP commands */
-#define NSP_CMD_RESET			1
+#define NSP_CMD_RESET          1
+#define NSP_CMD_FW_LOAD        6
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -304,3 +308,63 @@
 
 	return res;
 }
+
+#define DEFAULT_FW_PATH       "/lib/firmware/netronome"
+#define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"
+
+int
+nfp_fw_upload(nspu_desc_t *nspu_desc)
+{
+	int fw_f;
+	char *fw_buf;
+	char filename[100];
+	struct stat file_stat;
+	off_t fsize, bytes;
+	ssize_t size;
+	int ret;
+
+	size = nspu_desc->buf_size;
+
+	sprintf(filename, "%s/%s", DEFAULT_FW_PATH, DEFAULT_FW_FILENAME);
+	fw_f = open(filename, O_RDONLY);
+	if (fw_f < 0) {
+		RTE_LOG(INFO, PMD, "Firmware file %s/%s not found.",
+			DEFAULT_FW_PATH, DEFAULT_FW_FILENAME);
+		return -ENOENT;
+	}
+
+	fstat(fw_f, &file_stat);
+
+	fsize = file_stat.st_size;
+	RTE_LOG(DEBUG, PMD, "Firmware file with size: %" PRIu64 "\n",
+			    (uint64_t)fsize);
+
+	if (fsize > (off_t)size) {
+		RTE_LOG(INFO, PMD, "fw file too big: %" PRIu64
+				   " bytes (%" PRIu64 " max)",
+				  (uint64_t)fsize, (uint64_t)size);
+		return -EINVAL;
+	}
+
+	fw_buf = malloc((size_t)size);
+	if (!fw_buf) {
+		RTE_LOG(INFO, PMD, "malloc failed for fw buffer");
+		return -ENOMEM;
+	}
+	memset(fw_buf, 0, size);
+
+	bytes = read(fw_f, fw_buf, fsize);
+	if (bytes != fsize) {
+		RTE_LOG(INFO, PMD, "Reading fw to buffer failed.\n"
+				   "Just %" PRIu64 " of %" PRIu64 " bytes read.",
+				   (uint64_t)bytes, (uint64_t)fsize);
+		free(fw_buf);
+		return -EIO;
+	}
+
+	ret = nspu_command(nspu_desc, NSP_CMD_FW_LOAD, 0, 1, fw_buf, 0, bytes);
+
+	free(fw_buf);
+
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index a142eb3..6e1c25f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -73,3 +73,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_fw_reset(nspu_desc_t *nspu_desc);
+int nfp_fw_upload(nspu_desc_t *nspu_desc);
-- 
1.9.1

[PATCH v2 06/18] net/nfp: add NSP symbol resolution command

[Alejandro Lucero]

Firmware has symbols helping to configure things like number of
PF ports, vNIC BARs addresses inside NFP memories, or ethernet
link state. Different firmware apps have different things to map
and likely different internal NFP addresses to use.

Host drivers can use the NSPU interface for getting symbol data
regarding different hardware configurations. Once the driver has
the information about a specific object, a mapping is required
configuring an NFP expansion bar creating a device PCI bar window.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h |  3 ++
 2 files changed, 89 insertions(+)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 57ee45f..d80da80 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -43,6 +43,7 @@
 /* NSP commands */
 #define NSP_CMD_RESET          1
 #define NSP_CMD_FW_LOAD        6
+#define NSP_CMD_GET_SYMBOL     14
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -368,3 +369,88 @@
 
 	return ret;
 }
+
+/* Firmware symbol descriptor size */
+#define NFP_SYM_DESC_LEN 40
+
+#define SYMBOL_DATA(b, off)     (*(int64_t *)((b) + (off)))
+#define SYMBOL_UDATA(b, off)     (*(uint64_t *)((b) + (off)))
+
+/* Firmware symbols contain information about how to access what they
+ * represent. It can be as simple as an numeric variable declared at a
+ * specific NFP memory, but it can also be more complex structures and
+ * related to specific hardware functionalities or components. Target,
+ * domain and address allow to create the BAR window for accessing such
+ * hw object and size defines the length to map.
+ *
+ * A vNIC is a network interface implemented inside the NFP and using a
+ * subset of device PCI BARs. Specific firmware symbols allow to map those
+ * vNIC bars by host drivers like the NFP PMD.
+ *
+ * Accessing what the symbol represents implies to map the access through
+ * a PCI BAR window. NFP expansion BARs are used in this regard through
+ * the NSPU interface.
+ */
+int
+nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
+			    uint32_t expbar, uint64_t *pcie_offset,
+			    ssize_t *size)
+{
+	int64_t type;
+	int64_t target;
+	int64_t domain;
+	uint64_t addr;
+	char *sym_buf;
+	int ret = 0;
+
+	sym_buf = malloc(desc->buf_size);
+	strncpy(sym_buf, symbl, strlen(symbl));
+	ret = nspu_command(desc, NSP_CMD_GET_SYMBOL, 1, 1, sym_buf,
+			   NFP_SYM_DESC_LEN, strlen(symbl));
+	if (ret) {
+		RTE_LOG(DEBUG, PMD, "symbol resolution (%s) failed\n", symbl);
+		goto clean;
+	}
+
+	/* Reading symbol information */
+	type = SYMBOL_DATA(sym_buf, 0);
+	target = SYMBOL_DATA(sym_buf, 8);
+	domain =  SYMBOL_DATA(sym_buf, 16);
+	addr = SYMBOL_UDATA(sym_buf, 24);
+	*size = (ssize_t)SYMBOL_UDATA(sym_buf, 32);
+
+	if (type != 1) {
+		RTE_LOG(INFO, PMD, "wrong symbol type\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+	if (!(target == 7 || target == -7)) {
+		RTE_LOG(INFO, PMD, "wrong symbol target\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+	if (domain == 8 || domain == 9) {
+		RTE_LOG(INFO, PMD, "wrong symbol domain\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+
+	/* Adjusting address based on symbol location */
+	if ((domain >= 24) && (domain < 28) && (target == 7)) {
+		addr = 1ULL << 37 | addr | ((uint64_t)domain & 0x3) << 35;
+	} else {
+		addr = 1ULL << 39 | addr | ((uint64_t)domain & 0x3f) << 32;
+		if (target == -7)
+			target = 7;
+	}
+
+	/* Configuring NFP expansion bar for mapping specific PCI BAR window */
+	nfp_nspu_mem_bar_cfg(desc, expbar, target, addr, pcie_offset);
+
+	/* This is the PCI BAR offset to use by the host */
+	*pcie_offset |= ((expbar & 0x7) << (desc->barsz - 3));
+
+clean:
+	free(sym_buf);
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 6e1c25f..7734b4f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -74,3 +74,6 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_fw_reset(nspu_desc_t *nspu_desc);
 int nfp_fw_upload(nspu_desc_t *nspu_desc);
+int nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
+				uint32_t expbar, uint64_t *pcie_offset,
+				ssize_t *size);
-- 
1.9.1

[PATCH v2 07/18] net/nfp: add FW upload logic

[Alejandro Lucero]

PMD will use this function for uploading the firmware. First, a
symbol resolution is done for finding out if there is a firmware
already there. If not, a NFP reset is called before using NSPU
fw upload code.

PMD PF probe function is now using this logic.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c  |  3 +++
 drivers/net/nfp/nfp_nspu.c | 48 +++++++++++++++++++++++++++++++++++++++++++---
 drivers/net/nfp/nfp_nspu.h |  6 +-----
 3 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index ca481e4..506743d 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2639,6 +2639,7 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
+	uint64_t offset_symbol;
 	int major, minor;
 
 	if (!dev)
@@ -2669,6 +2670,8 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto no_abi;
 	}
 
+	nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports", &offset_symbol);
+
 	/* No port is created yet */
 
 no_abi:
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index d80da80..5a3ae0e 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -21,6 +21,9 @@
 /* NFP target for NSP access */
 #define NFP_NSP_TARGET   7
 
+/* Expansion BARs for mapping PF vnic BARs */
+#define NFP_NET_PF_CFG_EXP_BAR    6
+
 /*
  * This is an NFP internal address used for configuring properly an NFP
  * expansion BAR.
@@ -297,7 +300,7 @@
 	return ret;
 }
 
-int
+static int
 nfp_fw_reset(nspu_desc_t *nspu_desc)
 {
 	int res;
@@ -313,7 +316,7 @@
 #define DEFAULT_FW_PATH       "/lib/firmware/netronome"
 #define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"
 
-int
+static int
 nfp_fw_upload(nspu_desc_t *nspu_desc)
 {
 	int fw_f;
@@ -391,7 +394,7 @@
  * a PCI BAR window. NFP expansion BARs are used in this regard through
  * the NSPU interface.
  */
-int
+static int
 nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
 			    uint32_t expbar, uint64_t *pcie_offset,
 			    ssize_t *size)
@@ -454,3 +457,42 @@
 	free(sym_buf);
 	return ret;
 }
+
+int
+nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset)
+{
+	ssize_t bar0_sym_size;
+
+	/* If the symbol resolution works, it implies a firmware app
+	 * is already there.
+	 */
+	if (!nfp_nspu_set_bar_from_symbl(desc, sym, NFP_NET_PF_CFG_EXP_BAR,
+					 pcie_offset, &bar0_sym_size))
+		return 0;
+
+	/* No firmware app detected or not the right one */
+	RTE_LOG(INFO, PMD, "No firmware detected. Resetting NFP...\n");
+	if (nfp_fw_reset(desc) < 0) {
+		RTE_LOG(ERR, PMD, "nfp fw reset failed\n");
+		return -ENODEV;
+	}
+
+	RTE_LOG(INFO, PMD, "Reset done.\n");
+	RTE_LOG(INFO, PMD, "Uploading firmware...\n");
+
+	if (nfp_fw_upload(desc) < 0) {
+		RTE_LOG(ERR, PMD, "nfp fw upload failed\n");
+		return -ENODEV;
+	}
+
+	RTE_LOG(INFO, PMD, "Done.\n");
+
+	/* Now the symbol should be there */
+	if (nfp_nspu_set_bar_from_symbl(desc, sym, NFP_NET_PF_CFG_EXP_BAR,
+					pcie_offset, &bar0_sym_size)) {
+		RTE_LOG(ERR, PMD, "nfp PF BAR symbol resolution failed\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 7734b4f..c439700 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -72,8 +72,4 @@
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
-int nfp_fw_reset(nspu_desc_t *nspu_desc);
-int nfp_fw_upload(nspu_desc_t *nspu_desc);
-int nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
-				uint32_t expbar, uint64_t *pcie_offset,
-				ssize_t *size);
+int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
-- 
1.9.1

[PATCH v2 08/18] net/nfp: add support for vnic config bar mapping

[Alejandro Lucero]

NFP vNICs use a subset of PCI device BARs. vNIC config bar depends on
firmware symbol defining how to map it through a NFP expansion bar.

This patch adds a NSPU API function for getting a vNIC config bar
mapped through a expansion bar giving a firmware symbol. The PMD will
use the PCI bar offset returned for accessing the vNIC bar.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 13 +++++++++++++
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 5a3ae0e..827f0b2 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -496,3 +496,16 @@
 
 	return 0;
 }
+
+int
+nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset)
+{
+	ssize_t bar0_sym_size;
+
+	if (nfp_nspu_set_bar_from_symbl(desc, "_pf0_net_bar0",
+					NFP_NET_PF_CFG_EXP_BAR,
+					pcie_offset, &bar0_sym_size))
+		return -ENODEV;
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index c439700..8211f92 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -73,3 +73,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
+int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
-- 
1.9.1

[PATCH v2 09/18] net/nfp: add support for vNIC Rx/Tx bar mappings

[Alejandro Lucero]

NFP vNICs use a subset of PCI device BARs. vNIC rx/tx bars point to
NFP hardware queues unit. Unline vNIC config bar, the NFP address is
always the same so the NFP expansion bar configuration always uses
the same hardcoded physical address.

This patch adds a NSPU API function for getting vNIC rx/tx bars
mapped through a expansion bar using that specific physical address.

The PMD will use the PCI bar offset returned for mapping the vNIC
rx/tx bars.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 21 ++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 827f0b2..56fd063 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -22,7 +22,8 @@
 #define NFP_NSP_TARGET   7
 
 /* Expansion BARs for mapping PF vnic BARs */
-#define NFP_NET_PF_CFG_EXP_BAR    6
+#define NFP_NET_PF_CFG_EXP_BAR          6
+#define NFP_NET_PF_HW_QUEUES_EXP_BAR    5
 
 /*
  * This is an NFP internal address used for configuring properly an NFP
@@ -509,3 +510,21 @@
 
 	return 0;
 }
+
+/*
+ * This is a hardcoded fixed NFP internal CPP bus address for the hw queues unit
+ * inside the PCIE island.
+ */
+#define NFP_CPP_PCIE_QUEUES ((uint64_t)(1ULL << 39) |  0x80000 | \
+			     ((uint64_t)0x4 & 0x3f) << 32)
+
+/* Configure a specific NFP expansion bar for accessing the vNIC rx/tx BARs */
+void
+nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset)
+{
+	nfp_nspu_mem_bar_cfg(desc, NFP_NET_PF_HW_QUEUES_EXP_BAR, 0,
+			     NFP_CPP_PCIE_QUEUES, pcie_offset);
+
+	/* This is the pcie offset to use by the host */
+	*pcie_offset |= ((NFP_NET_PF_HW_QUEUES_EXP_BAR & 0x7) << (27 - 3));
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 8211f92..4b09d4f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -74,3 +74,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
+void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
-- 
1.9.1

[PATCH v2 10/18] net/nfp: support PF devices inside PMD initialization

[Alejandro Lucero]

nfp_net_init is where a dpdk port related to a eth_dev is initialized.
NFP VF vNICs use VF PCI BARs as they come after SRIOV is enabled. But for
NFP PF vNIC just a subset of PF PCI BARs are used.

This patch adds support for mapping the right PCI BAR subsets for the PF
vNIC. It uses the NSPU API functions introduced previously for configuring
NFP expansion bars.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 56 ++++++++++++++++++++++++++++++++++++-------
 drivers/net/nfp/nfp_net_pmd.h |  4 ++++
 2 files changed, 52 insertions(+), 8 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 506743d..021b0ea 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -55,12 +55,11 @@
 #include <rte_alarm.h>
 #include <rte_spinlock.h>
 
+#include "nfp_nfpu.h"
 #include "nfp_net_pmd.h"
 #include "nfp_net_logs.h"
 #include "nfp_net_ctrl.h"
 
-#include "nfp_nfpu.h"
-
 /* Prototypes */
 static void nfp_net_close(struct rte_eth_dev *dev);
 static int nfp_net_configure(struct rte_eth_dev *dev);
@@ -101,7 +100,7 @@ static uint16_t nfp_net_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
  * happen to be at the same offset on the NFP6000 and the NFP3200 so
  * we use a single macro here.
  */
-#define NFP_PCIE_QUEUE(_q)	(0x80000 + (0x800 * ((_q) & 0xff)))
+#define NFP_PCIE_QUEUE(_q)	(0x800 * ((_q) & 0xff))
 
 /* Maximum value which can be added to a queue with one transaction */
 #define NFP_QCP_MAX_ADD	0x7f
@@ -2496,10 +2495,13 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	struct rte_pci_device *pci_dev;
 	struct nfp_net_hw *hw;
 
-	uint32_t tx_bar_off, rx_bar_off;
+	uint64_t tx_bar_off = 0, rx_bar_off = 0;
 	uint32_t start_q;
 	int stride = 4;
 
+	nspu_desc_t *nspu_desc = NULL;
+	uint64_t bar_offset;
+
 	PMD_INIT_FUNC_TRACE();
 
 	hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -2532,11 +2534,33 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 			"hw->ctrl_bar is NULL. BAR0 not configured\n");
 		return -ENODEV;
 	}
+
+	/* Is this a PF device? */
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		nspu_desc = hw->nspu_desc;
+
+		if (nfp_nsp_map_ctrl_bar(nspu_desc, &bar_offset) != 0) {
+			/*
+			 * A firmware should be there after PF probe so this
+			 * should not happen.
+			 */
+			RTE_LOG(ERR, PMD, "PF BAR symbol resolution failed\n");
+			return -ENODEV;
+		}
+
+		/* vNIC PF control BAR is a subset of PF PCI device BAR */
+		hw->ctrl_bar += bar_offset;
+		PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
+	}
+
 	hw->max_rx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_RXRINGS);
 	hw->max_tx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_TXRINGS);
 
 	/* Work out where in the BAR the queues start. */
 	switch (pci_dev->id.device_id) {
+	case PCI_DEVICE_ID_NFP4000_PF_NIC:
+	case PCI_DEVICE_ID_NFP6000_PF_NIC:
 	case PCI_DEVICE_ID_NFP6000_VF_NIC:
 		start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
 		tx_bar_off = NFP_PCIE_QUEUE(start_q);
@@ -2548,11 +2572,27 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENODEV;
 	}
 
-	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%08x", tx_bar_off);
-	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%08x", rx_bar_off);
+	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%" PRIx64 "\n", tx_bar_off);
+	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%" PRIx64 "\n", rx_bar_off);
 
-	hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
-	hw->rx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + rx_bar_off;
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		/* configure access to tx/rx vNIC BARs */
+		nfp_nsp_map_queues_bar(nspu_desc, &bar_offset);
+		PMD_INIT_LOG(DEBUG, "tx/rx bar_offset: %" PRIx64 "\n",
+				    bar_offset);
+		hw->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
+
+		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
+		hw->hw_queues += bar_offset;
+		hw->tx_bar = hw->hw_queues + tx_bar_off;
+		hw->rx_bar = hw->hw_queues + rx_bar_off;
+	} else {
+		hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
+			     tx_bar_off;
+		hw->rx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
+			     rx_bar_off;
+	}
 
 	PMD_INIT_LOG(DEBUG, "ctrl_bar: %p, tx_bar: %p, rx_bar: %p",
 		     hw->ctrl_bar, hw->tx_bar, hw->rx_bar);
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index 3818130..0f902fc 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -437,6 +437,10 @@ struct nfp_net_hw {
 	struct nfp_cpp_area *rx_area;
 	struct nfp_cpp_area *msix_area;
 #endif
+	uint8_t *hw_queues;
+	uint8_t is_pf;
+	nspu_desc_t *nspu_desc;
+	nfpu_desc_t *nfpu_desc;
 };
 
 struct nfp_net_adapter {
-- 
1.9.1

[PATCH v2 11/18] net/nfp: allocate ethernet device from PF probe function

[Alejandro Lucero]

NFP can support several physical ports per PF device. Depending on
firmware info, one or more eth_dev objects will need to be created.

This patch adds the call to create just one eth_dev by now with future
commits supporting the multiport option. Once the eth_dev has been
created, probe function invokes pmd initialization with the new eth_dev.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 49 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 42 insertions(+), 7 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 021b0ea..5b4108a 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2677,13 +2677,16 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			    struct rte_pci_device *dev)
 {
+	struct rte_eth_dev *eth_dev;
+	struct nfp_net_hw *hw;
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
 	uint64_t offset_symbol;
 	int major, minor;
+	int ret = -ENODEV;
 
 	if (!dev)
-		return -ENODEV;
+		return ret;
 
 	nfpu_desc = rte_malloc("nfp nfpu", sizeof(nfpu_desc_t), 0);
 	if (!nfpu_desc)
@@ -2701,25 +2704,57 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	/* Check NSP ABI version */
 	if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
 		RTE_LOG(INFO, PMD, "NFP NSP not present\n");
-		goto no_abi;
+		goto error;
 	}
 	PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
 
 	if ((major == 0) && (minor < 20)) {
 		RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required 0.20 or higher\n");
-		goto no_abi;
+		goto error;
+	}
+
+	ret = nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports",
+			       &offset_symbol);
+	if (ret)
+		goto error;
+
+	eth_dev = rte_eth_dev_allocate(dev->device.name);
+	if (!eth_dev) {
+		ret = -ENODEV;
+		goto error;
 	}
 
-	nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports", &offset_symbol);
+	eth_dev->data->dev_private = rte_zmalloc("nfp_pf_port",
+						 sizeof(struct nfp_net_adapter),
+						 RTE_CACHE_LINE_SIZE);
+	if (!eth_dev->data->dev_private) {
+		rte_eth_dev_release_port(eth_dev);
+		ret = -ENODEV;
+		goto error;
+	}
+
+	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private);
+	hw->nspu_desc = nspu_desc;
+	hw->nfpu_desc = nfpu_desc;
+	hw->is_pf = 1;
 
-	/* No port is created yet */
+	eth_dev->device = &dev->device;
+	rte_eth_copy_pci_info(eth_dev, dev);
 
-no_abi:
+	ret = nfp_net_init(eth_dev);
+
+	if (!ret)
+		return 0;
+
+	/* something went wrong */
+	rte_eth_dev_release_port(eth_dev);
+
+error:
 	nfpu_close(nfpu_desc);
 nfpu_error:
 	rte_free(nfpu_desc);
 
-	return -ENODEV;
+	return ret;
 }
 
 static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
-- 
1.9.1

[PATCH v2 12/18] net/nfp: support PF multiport

[Alejandro Lucero]

A NFP PF PCI device can have several physical ports, up to 8. Because
DPDK core creates one eth_dev per PCI device, nfp pf probe function
is used. Number of PF ports is obtained from firmware symbol using
NSPU API. Inside PF probe function an eth_dev per port is created and
nfp_net_init invoked for each port.

There are some limitations reagarding multiport: rx interrupts and
device hotplug are not supported.

Interrupts are handled with the VFIO or UIO drivers help. Those
drivers just know about PCI devices, so it is not possible, without
changing how DPDK handles interrupts, manage interrupts assigned to
different PF ports.

About hotplug, the problem is this functionality is based on a PCI
device, and although device pluging is possible, which would add as
many ports as supported by firmware, unpluging is based on device name
linked to a eth_dev, and device name has a suffix now (_portX, with X
being the port index) which DPDK core is not aware of. While rx
interrupts with multiport could be likely solved with some layer of
indirection, hotplug would require changes to DPDK core.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c      | 199 +++++++++++++++++++++++++++++++++--------
 drivers/net/nfp/nfp_net_ctrl.h |   3 +
 drivers/net/nfp/nfp_net_pmd.h  |   2 +
 3 files changed, 165 insertions(+), 39 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 5b4108a..5bfb18e 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -687,6 +687,11 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 
 	/* check and configure queue intr-vector mapping */
 	if (dev->data->dev_conf.intr_conf.rxq != 0) {
+		if (hw->pf_multiport_enabled) {
+			PMD_INIT_LOG(ERR, "PMD rx interrupt is not supported "
+					  "with NFP multiport PF");
+				return -EINVAL;
+		}
 		if (intr_handle->type == RTE_INTR_HANDLE_UIO) {
 			/*
 			 * Better not to share LSC with RX interrupts.
@@ -2489,11 +2494,43 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	.rx_queue_intr_disable  = nfp_rx_queue_intr_disable,
 };
 
+/*
+ * All eth_dev created got its private data, but before nfp_net_init, that
+ * private data is referencing private data for all the PF ports. This is due
+ * to how the vNIC bars are mapped based on first port, so all ports need info
+ * about port 0 private data. Inside nfp_net_init the private data pointer is
+ * changed to the right address for each port once the bars have been mapped.
+ *
+ * This functions helps to find out which port and therefore which offset
+ * inside the private data array to use.
+ */
+static int
+get_pf_port_number(char *name)
+{
+	char *pf_str = name;
+	int size = 0;
+
+	while ((*pf_str != '_') && (*pf_str != '\0') && (size++ < 30))
+		pf_str++;
+
+	if (size == 30)
+		/*
+		 * This should not happen at all and it would mean major
+		 * implementation fault.
+		 */
+		rte_panic("nfp_net: problem with pf device name\n");
+
+	/* Expecting _portX with X within [0,7] */
+	pf_str += 5;
+
+	return (int)strtol(pf_str, NULL, 10);
+}
+
 static int
 nfp_net_init(struct rte_eth_dev *eth_dev)
 {
 	struct rte_pci_device *pci_dev;
-	struct nfp_net_hw *hw;
+	struct nfp_net_hw *hw, *hwport0;
 
 	uint64_t tx_bar_off = 0, rx_bar_off = 0;
 	uint32_t start_q;
@@ -2501,10 +2538,32 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 	nspu_desc_t *nspu_desc = NULL;
 	uint64_t bar_offset;
+	int port = 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		port = get_pf_port_number(eth_dev->data->name);
+		if (port < 0 || port > 7) {
+			RTE_LOG(ERR, PMD, "Port value is wrong\n");
+			return -ENODEV;
+		}
+
+		PMD_INIT_LOG(DEBUG, "Working with PF port value %d\n", port);
+
+		/* This points to port 0 private data */
+		hwport0 = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+
+		/* This points to the specific port private data */
+		hw = &hwport0[port];
+		hw->pf_port_idx = port;
+	} else {
+		hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+		hwport0 = 0;
+	}
 
 	eth_dev->dev_ops = &nfp_net_eth_dev_ops;
 	eth_dev->rx_pkt_burst = &nfp_net_recv_pkts;
@@ -2514,9 +2573,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
-	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
-	eth_dev->data->dev_flags |= RTE_ETH_DEV_DETACHABLE;
+	/* hotplug is not possible with multiport PF */
+	if (!hw->pf_multiport_enabled)
+		eth_dev->data->dev_flags |= RTE_ETH_DEV_DETACHABLE;
 
 	hw->device_id = pci_dev->id.device_id;
 	hw->vendor_id = pci_dev->id.vendor_id;
@@ -2535,9 +2595,7 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENODEV;
 	}
 
-	/* Is this a PF device? */
-	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
-	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+	if (hw->is_pf && port == 0) {
 		nspu_desc = hw->nspu_desc;
 
 		if (nfp_nsp_map_ctrl_bar(nspu_desc, &bar_offset) != 0) {
@@ -2554,6 +2612,17 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
 	}
 
+	if (port > 0) {
+		if (!hwport0->ctrl_bar)
+			return -ENODEV;
+
+		/* address based on port0 offset */
+		hw->ctrl_bar = hwport0->ctrl_bar +
+			       (port * NFP_PF_CSR_SLICE_SIZE);
+	}
+
+	PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
+
 	hw->max_rx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_RXRINGS);
 	hw->max_tx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_TXRINGS);
 
@@ -2575,18 +2644,21 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%" PRIx64 "\n", tx_bar_off);
 	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%" PRIx64 "\n", rx_bar_off);
 
-	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
-	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+	if (hw->is_pf && port == 0) {
 		/* configure access to tx/rx vNIC BARs */
 		nfp_nsp_map_queues_bar(nspu_desc, &bar_offset);
 		PMD_INIT_LOG(DEBUG, "tx/rx bar_offset: %" PRIx64 "\n",
 				    bar_offset);
-		hw->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
+		hwport0->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
 
 		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
-		hw->hw_queues += bar_offset;
-		hw->tx_bar = hw->hw_queues + tx_bar_off;
-		hw->rx_bar = hw->hw_queues + rx_bar_off;
+		hwport0->hw_queues += bar_offset;
+	}
+
+	if (hw->is_pf) {
+		hw->tx_bar = hwport0->hw_queues + tx_bar_off;
+		hw->rx_bar = hwport0->hw_queues + rx_bar_off;
+		eth_dev->data->dev_private = hw;
 	} else {
 		hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
 			     tx_bar_off;
@@ -2674,16 +2746,77 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	return 0;
 }
 
-static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
-			    struct rte_pci_device *dev)
+static int
+nfp_pf_create_dev(struct rte_pci_device *dev, int port, int ports,
+		  nfpu_desc_t *nfpu_desc, void **priv)
 {
 	struct rte_eth_dev *eth_dev;
 	struct nfp_net_hw *hw;
+	char *port_name;
+	int ret;
+
+	port_name = rte_zmalloc("nfp_pf_port_name", 100, 0);
+	if (!port_name)
+		return -ENOMEM;
+
+	if (ports > 1)
+		sprintf(port_name, "%s_port%d", dev->device.name, port);
+	else
+		sprintf(port_name, "%s", dev->device.name);
+
+	eth_dev = rte_eth_dev_allocate(port_name);
+	if (!eth_dev)
+		return -ENOMEM;
+
+	if (port == 0) {
+		*priv = rte_zmalloc(port_name,
+				    sizeof(struct nfp_net_adapter) * ports,
+				    RTE_CACHE_LINE_SIZE);
+		if (!*priv) {
+			rte_eth_dev_release_port(eth_dev);
+			return -ENOMEM;
+		}
+	}
+
+	eth_dev->data->dev_private = *priv;
+
+	/*
+	 * dev_private pointing to port0 dev_private because we need
+	 * to configure vNIC bars based on port0 at nfp_net_init.
+	 * Then dev_private is adjusted per port.
+	 */
+	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private) + port;
+	hw->nspu_desc = nfpu_desc->nspu;
+	hw->nfpu_desc = nfpu_desc;
+	hw->is_pf = 1;
+	if (ports > 1)
+		hw->pf_multiport_enabled = 1;
+
+	eth_dev->device = &dev->device;
+	rte_eth_copy_pci_info(eth_dev, dev);
+
+	ret = nfp_net_init(eth_dev);
+
+	if (ret)
+		rte_eth_dev_release_port(eth_dev);
+
+	rte_free(port_name);
+
+	return ret;
+}
+
+static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+			    struct rte_pci_device *dev)
+{
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
 	uint64_t offset_symbol;
+	uint8_t *bar_offset;
 	int major, minor;
+	int total_ports;
+	void *priv = 0;
 	int ret = -ENODEV;
+	int i;
 
 	if (!dev)
 		return ret;
@@ -2718,36 +2851,24 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret)
 		goto error;
 
-	eth_dev = rte_eth_dev_allocate(dev->device.name);
-	if (!eth_dev) {
-		ret = -ENODEV;
-		goto error;
-	}
+	bar_offset = (uint8_t *)dev->mem_resource[0].addr;
+	bar_offset += offset_symbol;
+	total_ports = (uint32_t)*bar_offset;
+	PMD_INIT_LOG(INFO, "Total pf ports: %d\n", total_ports);
 
-	eth_dev->data->dev_private = rte_zmalloc("nfp_pf_port",
-						 sizeof(struct nfp_net_adapter),
-						 RTE_CACHE_LINE_SIZE);
-	if (!eth_dev->data->dev_private) {
-		rte_eth_dev_release_port(eth_dev);
+	if (total_ports <= 0 || total_ports > 8) {
+		RTE_LOG(ERR, PMD, "nfd_cfg_pf0_num_ports symbol with wrong value");
 		ret = -ENODEV;
 		goto error;
 	}
 
-	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private);
-	hw->nspu_desc = nspu_desc;
-	hw->nfpu_desc = nfpu_desc;
-	hw->is_pf = 1;
-
-	eth_dev->device = &dev->device;
-	rte_eth_copy_pci_info(eth_dev, dev);
-
-	ret = nfp_net_init(eth_dev);
-
-	if (!ret)
-		return 0;
+	for (i = 0; i < total_ports; i++) {
+		ret = nfp_pf_create_dev(dev, i, total_ports, nfpu_desc, &priv);
+		if (ret)
+			goto error;
+	}
 
-	/* something went wrong */
-	rte_eth_dev_release_port(eth_dev);
+	return 0;
 
 error:
 	nfpu_close(nfpu_desc);
diff --git a/drivers/net/nfp/nfp_net_ctrl.h b/drivers/net/nfp/nfp_net_ctrl.h
index c1cba0e..1ebd99c 100644
--- a/drivers/net/nfp/nfp_net_ctrl.h
+++ b/drivers/net/nfp/nfp_net_ctrl.h
@@ -334,6 +334,9 @@
 #define NFP_NET_CFG_RXR_STATS(_x)       (NFP_NET_CFG_RXR_STATS_BASE + \
 					 ((_x) * 0x10))
 
+/* PF multiport offset */
+#define NFP_PF_CSR_SLICE_SIZE	(32 * 1024)
+
 #endif /* _NFP_NET_CTRL_H_ */
 /*
  * Local variables:
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index 0f902fc..d7e38d4 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -439,6 +439,8 @@ struct nfp_net_hw {
 #endif
 	uint8_t *hw_queues;
 	uint8_t is_pf;
+	uint8_t pf_port_idx;
+	uint8_t pf_multiport_enabled;
 	nspu_desc_t *nspu_desc;
 	nfpu_desc_t *nfpu_desc;
 };
-- 
1.9.1

[PATCH v2 13/18] net/nfp: add NSP support for HW link configuration

[Alejandro Lucero]

Adding a new NSPU command for being able to read and write the ethernet
port table from/to the NFP. This will allow the PMD to put the Link up
or down when a port is started or stopped. Until now, this was performed
by the firmware independently of PMD functionality.

The ethernet port table has also some other useful information that will
be used in further commits.

Usually NSPU is used at device probe time and that is sequential code
execution. However, reading and writing the NFP eth table can be done at
different times and from different cores, and it implies it could happen
a concurrent access. A spinlock is added to the global nspu object for
protecting the NFP and avoiding the concurrent access.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net_eth.h | 82 +++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.c    | 79 +++++++++++++++++++++++++++++++++++++++--
 drivers/net/nfp/nfp_nspu.h    |  4 +++
 3 files changed, 162 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/nfp/nfp_net_eth.h

diff --git a/drivers/net/nfp/nfp_net_eth.h b/drivers/net/nfp/nfp_net_eth.h
new file mode 100644
index 0000000..af57f03
--- /dev/null
+++ b/drivers/net/nfp/nfp_net_eth.h
@@ -0,0 +1,82 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_net_eth.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+union eth_table_entry {
+	struct {
+		uint64_t port;
+		uint64_t state;
+		uint8_t mac_addr[6];
+		uint8_t resv[2];
+		uint64_t control;
+	};
+	uint64_t raw[4];
+};
+
+#ifndef BIT_ULL
+#define BIT_ULL(a) (1ULL << (a))
+#endif
+
+#define NSP_ETH_NBI_PORT_COUNT          24
+#define NSP_ETH_MAX_COUNT               (2 * NSP_ETH_NBI_PORT_COUNT)
+#define NSP_ETH_TABLE_SIZE   (NSP_ETH_MAX_COUNT * sizeof(union eth_table_entry))
+
+#define NSP_ETH_PORT_LANES              0xf
+#define NSP_ETH_PORT_INDEX              0xff00
+#define NSP_ETH_PORT_LABEL              0x3f000000000000
+#define NSP_ETH_PORT_PHYLABEL           0xfc0000000000000
+
+#define NSP_ETH_PORT_LANES_MASK         rte_cpu_to_le_64(NSP_ETH_PORT_LANES)
+
+#define NSP_ETH_STATE_CONFIGURED        BIT_ULL(0)
+#define NSP_ETH_STATE_ENABLED           BIT_ULL(1)
+#define NSP_ETH_STATE_TX_ENABLED        BIT_ULL(2)
+#define NSP_ETH_STATE_RX_ENABLED        BIT_ULL(3)
+#define NSP_ETH_STATE_RATE              0xf00
+#define NSP_ETH_STATE_INTERFACE         0xff000
+#define NSP_ETH_STATE_MEDIA             0x300000
+#define NSP_ETH_STATE_OVRD_CHNG         BIT_ULL(22)
+#define NSP_ETH_STATE_ANEG              0x3800000
+
+#define NSP_ETH_CTRL_CONFIGURED         BIT_ULL(0)
+#define NSP_ETH_CTRL_ENABLED            BIT_ULL(1)
+#define NSP_ETH_CTRL_TX_ENABLED         BIT_ULL(2)
+#define NSP_ETH_CTRL_RX_ENABLED         BIT_ULL(3)
+#define NSP_ETH_CTRL_SET_RATE           BIT_ULL(4)
+#define NSP_ETH_CTRL_SET_LANES          BIT_ULL(5)
+#define NSP_ETH_CTRL_SET_ANEG           BIT_ULL(6)
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 56fd063..2f5632d 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -8,8 +8,10 @@
 #include <fcntl.h>
 
 #include <rte_log.h>
+#include <rte_byteorder.h>
 
 #include "nfp_nfpu.h"
+#include "nfp_net_eth.h"
 
 #define CFG_EXP_BAR_ADDR_SZ     1
 #define CFG_EXP_BAR_MAP_TYPE	1
@@ -45,9 +47,11 @@
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
 /* NSP commands */
-#define NSP_CMD_RESET          1
-#define NSP_CMD_FW_LOAD        6
-#define NSP_CMD_GET_SYMBOL     14
+#define NSP_CMD_RESET                   1
+#define NSP_CMD_FW_LOAD                 6
+#define NSP_CMD_READ_ETH_TABLE          7
+#define NSP_CMD_WRITE_ETH_TABLE         8
+#define NSP_CMD_GET_SYMBOL             14
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -528,3 +532,72 @@
 	/* This is the pcie offset to use by the host */
 	*pcie_offset |= ((NFP_NET_PF_HW_QUEUES_EXP_BAR & 0x7) << (27 - 3));
 }
+
+int
+nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up)
+{
+	union eth_table_entry *entries, *entry;
+	int modified;
+	int ret, idx;
+	int i;
+
+	idx = port;
+
+	RTE_LOG(INFO, PMD, "Hw ethernet port %d configure...\n", port);
+	rte_spinlock_lock(&desc->nsp_lock);
+	entries = malloc(NSP_ETH_TABLE_SIZE);
+	if (!entries) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return -ENOMEM;
+	}
+
+	ret = nspu_command(desc, NSP_CMD_READ_ETH_TABLE, 1, 0, entries,
+			   NSP_ETH_TABLE_SIZE, 0);
+	if (ret) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return ret;
+	}
+
+	entry = entries;
+
+	for (i = 0; i < NSP_ETH_MAX_COUNT; i++) {
+		/* ports in use do not appear sequentially in the table */
+		if (!(entry->port & NSP_ETH_PORT_LANES_MASK)) {
+			/* entry not in use */
+			entry++;
+			continue;
+		}
+		if (idx == 0)
+			break;
+		idx--;
+		entry++;
+	}
+
+	if (i == NSP_ETH_MAX_COUNT) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return -EINVAL;
+	}
+
+	if (up && !(entry->state & NSP_ETH_STATE_CONFIGURED)) {
+		entry->control |= NSP_ETH_STATE_CONFIGURED;
+		modified = 1;
+	}
+
+	if (!up && (entry->state & NSP_ETH_STATE_CONFIGURED)) {
+		entry->control &= ~NSP_ETH_STATE_CONFIGURED;
+		modified = 1;
+	}
+
+	if (modified) {
+		ret = nspu_command(desc, NSP_CMD_WRITE_ETH_TABLE, 0, 1, entries,
+				   0, NSP_ETH_TABLE_SIZE);
+		if (!ret)
+			RTE_LOG(INFO, PMD,
+				"Hw ethernet port %d configure done\n", port);
+		else
+			RTE_LOG(INFO, PMD,
+				"Hw ethernet port %d configure failed\n", port);
+	}
+	rte_spinlock_unlock(&desc->nsp_lock);
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 4b09d4f..4e58986 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -57,6 +57,8 @@
  * the fast path.
  */
 
+#include <rte_spinlock.h>
+
 typedef struct {
 	int nfp;        /* NFP device */
 	int pcie_bar;   /* PF PCI BAR to work with */
@@ -67,6 +69,7 @@
 	uint64_t windowsz; /* NSPU BAR window size */
 	void *cfg_base; /* Expansion BARs address */
 	void *mem_base; /* NSP interface */
+	rte_spinlock_t nsp_lock;
 } nspu_desc_t;
 
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
@@ -75,3 +78,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
+int nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up);
-- 
1.9.1

[PATCH v2 14/18] net/nfp: add support for HW port link configuration

[Alejandro Lucero]

PMD has to configure the hardware port: link up when port started and
link down when port stopped. This is not required for VFs but it is
for PF ports.

A minor refactoring in PMD stop and close functions is done because the
Link down needs to happen just when device is stopped.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 5bfb18e..251a1c6 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -737,6 +737,10 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 		goto error;
 	}
 
+	if (hw->is_pf)
+		/* Configure the physical port up */
+		nfp_nsp_eth_config(hw->nspu_desc, hw->pf_port_idx, 1);
+
 	hw->ctrl = new_ctrl;
 
 	return 0;
@@ -765,9 +769,12 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 nfp_net_stop(struct rte_eth_dev *dev)
 {
 	int i;
+	struct nfp_net_hw *hw;
 
 	PMD_INIT_LOG(DEBUG, "Stop");
 
+	hw = NFP_NET_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+
 	nfp_net_disable_queues(dev);
 
 	/* Clear queues */
@@ -780,6 +787,10 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 		nfp_net_reset_rx_queue(
 			(struct nfp_net_rxq *)dev->data->rx_queues[i]);
 	}
+
+	if (hw->is_pf)
+		/* Configure the physical port down */
+		nfp_nsp_eth_config(hw->nspu_desc, hw->pf_port_idx, 0);
 }
 
 /* Reset and stop device. The device can not be restarted. */
@@ -788,6 +799,7 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 {
 	struct nfp_net_hw *hw;
 	struct rte_pci_device *pci_dev;
+	int i;
 
 	PMD_INIT_LOG(DEBUG, "Close");
 
@@ -799,7 +811,18 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 	 * threads/queues before calling the device close function.
 	 */
 
-	nfp_net_stop(dev);
+	nfp_net_disable_queues(dev);
+
+	/* Clear queues */
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		nfp_net_reset_tx_queue(
+			(struct nfp_net_txq *)dev->data->tx_queues[i]);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		nfp_net_reset_rx_queue(
+			(struct nfp_net_rxq *)dev->data->rx_queues[i]);
+	}
 
 	rte_intr_disable(&pci_dev->intr_handle);
 	nn_cfg_writeb(hw, NFP_NET_CFG_LSC, 0xff);
-- 
1.9.1

[PATCH v2 15/18] net/nfp: read PF port MAC addr using NSP

[Alejandro Lucero]

During initialization, mac address is read from configuration bar. This is
the default option when using VFs.

This patch adds support for reading the mac address using the NSPU
interface when PMD works with the PF.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 59 +++++++++++++++++++++++++++++++++++++++++--
 drivers/net/nfp/nfp_net_pmd.h |  1 +
 drivers/net/nfp/nfp_nspu.c    | 22 +++++++++++++++-
 drivers/net/nfp/nfp_nspu.h    |  2 ++
 4 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 251a1c6..5d35ce1 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -593,7 +593,55 @@ enum nfp_qcp_ptr {
 	hw->qcp_cfg = hw->tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 }
 
-static void nfp_net_read_mac(struct nfp_net_hw *hw)
+#define ETH_ADDR_LEN	6
+
+static void
+nfp_eth_copy_mac_reverse(uint8_t *dst, const uint8_t *src)
+{
+	int i;
+
+	for (i = 0; i < ETH_ADDR_LEN; i++)
+		dst[ETH_ADDR_LEN - i - 1] = src[i];
+}
+
+static int
+nfp_net_pf_read_mac(struct nfp_net_hw *hw, int port)
+{
+	union eth_table_entry *entry;
+	int idx, i;
+
+	idx = port;
+	entry = hw->eth_table;
+
+	/* Reading NFP ethernet table obtained before */
+	for (i = 0; i < NSP_ETH_MAX_COUNT; i++) {
+		if (!(entry->port & NSP_ETH_PORT_LANES_MASK)) {
+			/* port not in use */
+			entry++;
+			continue;
+		}
+		if (idx == 0)
+			break;
+		idx--;
+		entry++;
+	}
+
+	if (i == NSP_ETH_MAX_COUNT)
+		return -EINVAL;
+
+	/*
+	 * hw points to port0 private data. We need hw now pointing to
+	 * right port.
+	 */
+	hw += port;
+	nfp_eth_copy_mac_reverse((uint8_t *)&hw->mac_addr,
+				 (uint8_t *)&entry->mac_addr);
+
+	return 0;
+}
+
+static void
+nfp_net_vf_read_mac(struct nfp_net_hw *hw)
 {
 	uint32_t tmp;
 
@@ -2676,6 +2724,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
 		hwport0->hw_queues += bar_offset;
+
+		/* Lets seize the chance to read eth table from hw */
+		if (nfp_nsp_eth_read_table(nspu_desc, &hw->eth_table))
+			return -ENODEV;
 	}
 
 	if (hw->is_pf) {
@@ -2736,7 +2788,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENOMEM;
 	}
 
-	nfp_net_read_mac(hw);
+	if (hw->is_pf)
+		nfp_net_pf_read_mac(hwport0, port);
+	else
+		nfp_net_vf_read_mac(hw);
 
 	if (!is_valid_assigned_ether_addr((struct ether_addr *)&hw->mac_addr)) {
 		/* Using random mac addresses for VFs */
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index d7e38d4..20ade1a 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -441,6 +441,7 @@ struct nfp_net_hw {
 	uint8_t is_pf;
 	uint8_t pf_port_idx;
 	uint8_t pf_multiport_enabled;
+	union eth_table_entry *eth_table;
 	nspu_desc_t *nspu_desc;
 	nfpu_desc_t *nfpu_desc;
 };
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 2f5632d..6ba940c 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -11,7 +11,6 @@
 #include <rte_byteorder.h>
 
 #include "nfp_nfpu.h"
-#include "nfp_net_eth.h"
 
 #define CFG_EXP_BAR_ADDR_SZ     1
 #define CFG_EXP_BAR_MAP_TYPE	1
@@ -601,3 +600,24 @@
 	rte_spinlock_unlock(&desc->nsp_lock);
 	return ret;
 }
+
+int
+nfp_nsp_eth_read_table(nspu_desc_t *desc, union eth_table_entry **table)
+{
+	int ret;
+
+	RTE_LOG(INFO, PMD, "Reading hw ethernet table...\n");
+	/* port 0 allocates the eth table and read it using NSPU */
+	*table = malloc(NSP_ETH_TABLE_SIZE);
+	if (!table)
+		return -ENOMEM;
+
+	ret = nspu_command(desc, NSP_CMD_READ_ETH_TABLE, 1, 0, *table,
+			   NSP_ETH_TABLE_SIZE, 0);
+	if (ret)
+		return ret;
+
+	RTE_LOG(INFO, PMD, "Done\n");
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 4e58986..8c33835 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -58,6 +58,7 @@
  */
 
 #include <rte_spinlock.h>
+#include "nfp_net_eth.h"
 
 typedef struct {
 	int nfp;        /* NFP device */
@@ -79,3 +80,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 int nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up);
+int nfp_nsp_eth_read_table(nspu_desc_t *desc, union eth_table_entry **table);
-- 
1.9.1

[PATCH v2 16/18] doc: update NFP with PF support information

[Alejandro Lucero]

NFP PMD has now support for both, PF and VFs. This patch updates
the guide and give some information about implications.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 doc/guides/nics/nfp.rst | 77 +++++++++++++++++++++++++++++++++++++------------
 1 file changed, 58 insertions(+), 19 deletions(-)

diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
index c732fb1..45dbbc3 100644
--- a/doc/guides/nics/nfp.rst
+++ b/doc/guides/nics/nfp.rst
@@ -1,5 +1,5 @@
 ..  BSD LICENSE
-    Copyright(c) 2015 Netronome Systems, Inc. All rights reserved.
+    Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved.
     All rights reserved.
 
     Redistribution and use in source and binary forms, with or without
@@ -38,31 +38,32 @@ up to 400 Gbps.
 
 This document explains how to use DPDK with the Netronome Poll Mode
 Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
-(NFP-6xxx).
+(NFP-6xxx) and Netronome's Flow Processor 4xxx (NFP-4xxx).
 
-Currently the driver supports virtual functions (VFs) only.
+NFP is a SRIOV capable device and the PMD driver supports the physical
+function (PF) and virtual functions (VFs).
 
 Dependencies
 ------------
 
-Before using the Netronome's DPDK PMD some NFP-6xxx configuration,
+Before using the Netronome's DPDK PMD some NFP configuration,
 which is not related to DPDK, is required. The system requires
-installation of **Netronome's BSP (Board Support Package)** which includes
-Linux drivers, programs and libraries.
+installation of **Netronome's BSP (Board Support Package)** along
+with some specific NFP firmware application. Netronome's NSP ABI
+version should be 0.20 or higher.
 
-If you have a NFP-6xxx device you should already have the code and
-documentation for doing this configuration. Contact
+If you have a NFP device you should already have the code and
+documentation for doing all this configuration. Contact
 **support@netronome.com** to obtain the latest available firmware.
 
-The NFP Linux kernel drivers (including the required PF driver for the
-NFP) are available on Github at
+The NFP Linux netdev kernel driver for VFs is part of vanilla kernel
+since kernel vesion 4.5, and support for the PF since kernel version
+4.11. Support for older kernels can be obtained on Github at
 **https://github.com/Netronome/nfp-drv-kmods** along with build
 instructions.
 
-DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to
-allow access to physical devices from userspace. The NFP PMD requires
-the **igb_uio** UIO driver, available with DPDK, to perform correct
-initialization.
+NFP PMD needs to be used along with UIO **igb_uio** or VFIO (vfio-pci)
+Linux kernel driver.
 
 Building the software
 ---------------------
@@ -71,7 +72,7 @@ Netronome's PMD code is provided in the **drivers/net/nfp** directory.
 Although NFP PMD has Netronome´s BSP dependencies, it is possible to
 compile it along with other DPDK PMDs even if no BSP was installed before.
 Of course, a DPDK app will require such a BSP installed for using the
-NFP PMD.
+NFP PMD, along with a specific NFP firmware application.
 
 Default PMD configuration is at **common_linuxapp configuration** file:
 
@@ -88,13 +89,51 @@ Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and
 for details.
 
 
+Using the PF
+------------
+
+NFP PMD has support for using the NFP PF as another DPDK port, but it has not
+any functionality for controlling VFs. In fact, it is not possible to use the
+PMD with the VFs if the PF is being used by DPDK, this is, NFP PF bound to
+igb_uio or vfio-pci kernel drivers. Future DPDK version will have a PMD able
+to work with the PF and VFs at the same time and with the PF implementing VF
+management along wih other PF-only functionalities/offloads.
+
+The PMD PF has extra work to do which will delay the DPDK app initialization
+like checking if a firmware is already available in the device, uploading the
+firmware if necessary, and configure the Link state properly when starting or
+stopping a PF port. Note that firmware upload is not always necessary which is
+the main delay for NFP PF PMD initialization.
+
+Depending on the Netronome's product installed in the system, firmware files
+should be available under /lib/firmware/netronome. DPDK PMD supporting the PF
+requires a specific link, /lib/firmware/netronome/nic_dpdk_default.nffw, which
+should be created automatically with Netronome's Agilio products installation.
+
+PF multiport support
+--------------------
+
+Some NFP cards support several physical ports with just one single PCI device.
+DPDK core is designed with the 1:1 relationship between PCI devices and DPDK
+ports, so NFP PMD PF support requires to handle the multiport case specifically.
+During NFP PF initialization, the PMD will extract the information about the
+number of PF ports from the firmware and will create as many DPDK ports as
+needed.
+
+Because the unusual relationship between a single PCI device and several DPDK
+ports, there are some limitations when using more than one PF DPDK ports: there
+is no support for RX interrrupts and it is not possible either to use those PF
+ports with the device hotplug functionality.
+
 System configuration
 --------------------
 
-#. **Enable SR-IOV on the NFP-6xxx device:** The current NFP PMD works with
-   Virtual Functions (VFs) on a NFP device. Make sure that one of the Physical
-   Function (PF) drivers from the above Github repository is installed and
-   loaded.
+#. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and
+   the VFs on a NFP device. However, it is not possible to work with both at the
+   same time because the VFs require the PF being bound to the NFP PF Linux
+   netdev driver.  Make sure you are working with a kernel with NFP PF support or
+   get the drivers from the above Github repository and follow the instructions
+   for building and installing it.
 
    Virtual Functions need to be enabled before they can be used with the PMD.
    Before enabling the VFs it is useful to obtain information about the
-- 
1.9.1

[PATCH v2 17/18] doc: update release notes with NFP PF support

[Alejandro Lucero]

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 doc/guides/rel_notes/release_17_11.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst
index 170f4f9..8773b08 100644
--- a/doc/guides/rel_notes/release_17_11.rst
+++ b/doc/guides/rel_notes/release_17_11.rst
@@ -69,6 +69,15 @@ EAL
 Drivers
 ~~~~~~~
 
+* **nfp: Added PF support.**
+
+  Previously Netronome's NFP PMD had just support for VFs. PF support is
+  just as a basic DPDK port and has no VF management yet.
+
+  PF support comes with firmware upload support which allows the PMD to
+  independently work from kernel netdev NFP drivers.
+
+  NFP 4000 devices are also now supported along with previous 6000 devices.
 
 Libraries
 ~~~~~~~~~
-- 
1.9.1

[PATCH v2 18/18] doc: create different features files for NFP drivers

[Alejandro Lucero]

NFP PMD implement now PF and VF drivers. Although the driver
functionality is the same by now, except for initialization, it
will change with future PF additions.

A new feature is required for describing the firmware upload
capability coming with the NFP PF now, so the PF file will be
updated soon in another patch.

SRIOV is not supported by the PF yet, and it is wrong to include it
as a VF driver feature, so none of the files have such a feature.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 doc/guides/nics/features/nfp.ini    | 29 -----------------------------
 doc/guides/nics/features/nfp_pf.ini | 28 ++++++++++++++++++++++++++++
 doc/guides/nics/features/nfp_vf.ini | 28 ++++++++++++++++++++++++++++
 3 files changed, 56 insertions(+), 29 deletions(-)
 delete mode 100644 doc/guides/nics/features/nfp.ini
 create mode 100644 doc/guides/nics/features/nfp_pf.ini
 create mode 100644 doc/guides/nics/features/nfp_vf.ini

diff --git a/doc/guides/nics/features/nfp.ini b/doc/guides/nics/features/nfp.ini
deleted file mode 100644
index a1281d2..0000000
--- a/doc/guides/nics/features/nfp.ini
+++ /dev/null
@@ -1,29 +0,0 @@
-;
-; Supported features of the 'nfp' network poll mode driver.
-;
-; Refer to default.ini for the full list of available PMD features.
-;
-[Features]
-Speed capabilities   = Y
-Link status          = Y
-Link status event    = Y
-Rx interrupt         = Y
-Queue start/stop     = Y
-MTU update           = Y
-Jumbo frame          = Y
-Promiscuous mode     = Y
-TSO                  = Y
-RSS hash             = Y
-RSS key update       = Y
-RSS reta update      = Y
-SR-IOV               = Y
-Flow control         = Y
-VLAN offload         = Y
-L3 checksum offload  = Y
-L4 checksum offload  = Y
-Basic stats          = Y
-Stats per queue      = Y
-Linux UIO            = Y
-Linux VFIO           = Y
-x86-64               = Y
-Usage doc            = Y
diff --git a/doc/guides/nics/features/nfp_pf.ini b/doc/guides/nics/features/nfp_pf.ini
new file mode 100644
index 0000000..d2899e7
--- /dev/null
+++ b/doc/guides/nics/features/nfp_pf.ini
@@ -0,0 +1,28 @@
+;
+; Supported features of the 'nfp' network poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities   = Y
+Link status          = Y
+Link status event    = Y
+Rx interrupt         = Y
+Queue start/stop     = Y
+MTU update           = Y
+Jumbo frame          = Y
+Promiscuous mode     = Y
+TSO                  = Y
+RSS hash             = Y
+RSS key update       = Y
+RSS reta update      = Y
+Flow control         = Y
+VLAN offload         = Y
+L3 checksum offload  = Y
+L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
+Linux UIO            = Y
+Linux VFIO           = Y
+x86-64               = Y
+Usage doc            = Y
diff --git a/doc/guides/nics/features/nfp_vf.ini b/doc/guides/nics/features/nfp_vf.ini
new file mode 100644
index 0000000..d2899e7
--- /dev/null
+++ b/doc/guides/nics/features/nfp_vf.ini
@@ -0,0 +1,28 @@
+;
+; Supported features of the 'nfp' network poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities   = Y
+Link status          = Y
+Link status event    = Y
+Rx interrupt         = Y
+Queue start/stop     = Y
+MTU update           = Y
+Jumbo frame          = Y
+Promiscuous mode     = Y
+TSO                  = Y
+RSS hash             = Y
+RSS key update       = Y
+RSS reta update      = Y
+Flow control         = Y
+VLAN offload         = Y
+L3 checksum offload  = Y
+L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
+Linux UIO            = Y
+Linux VFIO           = Y
+x86-64               = Y
+Usage doc            = Y
-- 
1.9.1

Re: [PATCH v2 16/18] doc: update NFP with PF support information

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> Sent: Friday, September 1, 2017 3:12 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 16/18] doc: update NFP with PF support
> information
> 
> NFP PMD has now support for both, PF and VFs. This patch updates the guide
> and give some information about implications.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>


Minor comments below.

>..
> -The NFP Linux kernel drivers (including the required PF driver for the
> -NFP) are available on Github at
> +The NFP Linux netdev kernel driver for VFs is part of vanilla kernel
> +since kernel vesion 4.5, and support for the PF since kernel version

s/vesion/version/


> -DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to -
> allow access to physical devices from userspace. The NFP PMD requires -the
> **igb_uio** UIO driver, available with DPDK, to perform correct -
> initialization.
> +NFP PMD needs to be used along with UIO **igb_uio** or VFIO (vfio-pci)
> +Linux kernel driver.


Probably better to fixed-width quote these as``igb_uio`` or ``vfio-pci``,
here and in other places.


> +Using the PF
> +------------
> +
> +NFP PMD has support for using the NFP PF as another DPDK port, but it
> +has not

s/but it has not/but it does not have/


> +possible to use the PMD with the VFs if the PF is being used by DPDK,
> +this is, NFP PF bound to igb_uio or vfio-pci kernel drivers. Future


s/this is, NFP PF/that is, with the NFP PF/

> +DPDK version will have a PMD able to work with the PF and VFs at the
> +same time and with the PF implementing VF management along wih other PF-

s/wih/with/


> +Depending on the Netronome's product installed in the system, firmware

s/Netronome's/Netronome/

> +files should be available under /lib/firmware/netronome. DPDK PMD
> +supporting the PF requires a specific link,
> +/lib/firmware/netronome/nic_dpdk_default.nffw, which should be created
> automatically with Netronome's Agilio products installation.

Fixed-width quote the file paths: ``/lib/firmware/netronome``, etc.

> +
> +PF multiport support
> +--------------------
> +
> +Some NFP cards support several physical ports with just one single PCI
> device.
> +DPDK core is designed with the 1:1 relationship between PCI devices and
> +DPDK ports, so NFP PMD PF support requires to handle the multiport case

s/to handle/handling/

> specifically.
> +During NFP PF initialization, the PMD will extract the information
> +about the number of PF ports from the firmware and will create as many
> +DPDK ports as needed.
> +
> +Because the unusual relationship between a single PCI device and
> +several DPDK ports, there are some limitations when using more than one
> +PF DPDK ports: there is no support for RX interrrupts and it is not

s/interrrupts/interrupts/


Reviewed-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 16/18] doc: update NFP with PF support information

[Alejandro Lucero]

Hi John,

Thanks for your review. I will modify the doc in next patch set version.

I'll wait until Ferruh give me the green light for sending the new version.


On Mon, Sep 4, 2017 at 3:20 PM, Mcnamara, John <john.mcnamara@intel.com>
wrote:

>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> > Sent: Friday, September 1, 2017 3:12 PM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [PATCH v2 16/18] doc: update NFP with PF support
> > information
> >
> > NFP PMD has now support for both, PF and VFs. This patch updates the
> guide
> > and give some information about implications.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
>
> Minor comments below.
>
> >..
> > -The NFP Linux kernel drivers (including the required PF driver for the
> > -NFP) are available on Github at
> > +The NFP Linux netdev kernel driver for VFs is part of vanilla kernel
> > +since kernel vesion 4.5, and support for the PF since kernel version
>
> s/vesion/version/
>
>
> > -DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to -
> > allow access to physical devices from userspace. The NFP PMD requires
> -the
> > **igb_uio** UIO driver, available with DPDK, to perform correct -
> > initialization.
> > +NFP PMD needs to be used along with UIO **igb_uio** or VFIO (vfio-pci)
> > +Linux kernel driver.
>
>
> Probably better to fixed-width quote these as``igb_uio`` or ``vfio-pci``,
> here and in other places.
>
>
> > +Using the PF
> > +------------
> > +
> > +NFP PMD has support for using the NFP PF as another DPDK port, but it
> > +has not
>
> s/but it has not/but it does not have/
>
>
> > +possible to use the PMD with the VFs if the PF is being used by DPDK,
> > +this is, NFP PF bound to igb_uio or vfio-pci kernel drivers. Future
>
>
> s/this is, NFP PF/that is, with the NFP PF/
>
> > +DPDK version will have a PMD able to work with the PF and VFs at the
> > +same time and with the PF implementing VF management along wih other PF-
>
> s/wih/with/
>
>
> > +Depending on the Netronome's product installed in the system, firmware
>
> s/Netronome's/Netronome/
>
> > +files should be available under /lib/firmware/netronome. DPDK PMD
> > +supporting the PF requires a specific link,
> > +/lib/firmware/netronome/nic_dpdk_default.nffw, which should be created
> > automatically with Netronome's Agilio products installation.
>
> Fixed-width quote the file paths: ``/lib/firmware/netronome``, etc.
>
> > +
> > +PF multiport support
> > +--------------------
> > +
> > +Some NFP cards support several physical ports with just one single PCI
> > device.
> > +DPDK core is designed with the 1:1 relationship between PCI devices and
> > +DPDK ports, so NFP PMD PF support requires to handle the multiport case
>
> s/to handle/handling/
>
> > specifically.
> > +During NFP PF initialization, the PMD will extract the information
> > +about the number of PF ports from the firmware and will create as many
> > +DPDK ports as needed.
> > +
> > +Because the unusual relationship between a single PCI device and
> > +several DPDK ports, there are some limitations when using more than one
> > +PF DPDK ports: there is no support for RX interrrupts and it is not
>
> s/interrrupts/interrupts/
>
>
> Reviewed-by: John McNamara <john.mcnamara@intel.com>
>
>
>

Re: [PATCH v2 17/18] doc: update release notes with NFP PF support

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> Sent: Friday, September 1, 2017 3:12 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 17/18] doc: update release notes with NFP PF
> support
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 18/18] doc: create different features files for NFP drivers

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> Sent: Friday, September 1, 2017 3:12 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2 18/18] doc: create different features files
> for NFP drivers
> 
> NFP PMD implement now PF and VF drivers. Although the driver functionality
> is the same by now, except for initialization, it will change with future
> PF additions.
> 
> A new feature is required for describing the firmware upload capability
> coming with the NFP PF now, so the PF file will be updated soon in another
> patch.
> 
> SRIOV is not supported by the PF yet, and it is wrong to include it as a
> VF driver feature, so none of the files have such a feature.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>


Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 16/18] doc: update NFP with PF support information

[Ferruh Yigit]

On 9/4/2017 5:03 PM, Alejandro Lucero wrote:
> Hi John,
> 
> Thanks for your review. I will modify the doc in next patch set version.
> 
> I'll wait until Ferruh give me the green light for sending the new version.

Patchset looks good to me, I can get it after mentioned documentation
corrections done.

If you prefer you can send only the documentation patch, instead of all
patchset, keeping John's ack.

Thanks,
ferruh

<...>

Re: [PATCH v2 16/18] doc: update NFP with PF support information

[Alejandro Lucero]

On Wed, Sep 6, 2017 at 10:53 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 9/4/2017 5:03 PM, Alejandro Lucero wrote:
> > Hi John,
> >
> > Thanks for your review. I will modify the doc in next patch set version.
> >
> > I'll wait until Ferruh give me the green light for sending the new
> version.
>
> Patchset looks good to me, I can get it after mentioned documentation
> corrections done.
>
> If you prefer you can send only the documentation patch, instead of all
> patchset, keeping John's ack.
>
>
Great.

I will send the v3 for this patch then.

Thank you


> Thanks,
> ferruh
>
> <...>
>
>

Re: [PATCH v2 00/18] net/nfp: add PF support

[Ferruh Yigit]

On 9/1/2017 3:12 PM, Alejandro Lucero wrote:
> NFP PMD has just had support for SRIOV VFs until now. This patch set adds
> support for the PF, but just for being used as another DPDK port. No VF
> management is added by now.
> 
> NFP is a programmable device and it supports virtual NICs (vNICs) through
> firmware implementation. Different firmware applications implement vNICs
> for PF devices and VF devices, being number of vNICs dependent on the
> firmware and the NFP card available. PF vNIC (virtual) BARs are a subset
> of PF PCI device BARs while VF vNIC BARs are same than VF PCI BARs.
> 
> Working with VF vNICs requires a PF driver uploading the firmware, doing
> some NFP configuration and creating/destroying VFs. This can be only done
> with the kernel NFP PF netdev driver by now.
> 
> Working with PF vNIC requires the PMD doing the NFP configuration and for
> accessing the NFP a specific user space interface is created. NFP Service
> Processor Userspace (NSPU) interface allows to create specific PCI BAR
> windows for accessing different parts of the NFP device, including the
> Network Service Processor (NSP) itself. The NSPU interface is implemented
> as the base for working with the PF.
> 
> v2:
>  - fix commits headers
>  - update nic guide
>  - add info to release notes
>  - creating different features files for PF and VF
>  - fix typo inside nfp_nspu.c
> 
> Alejandro Lucero (18):
>   net/nfp: add NSP user space interface
>   net/nfp: add specific PF probe function
>   net/nfp: add support for new PCI id
>   net/nfp: add NSP support for commands
>   net/nfp: add NSP FW upload command
>   net/nfp: add NSP symbol resolution command
>   net/nfp: add FW upload logic
>   net/nfp: add support for vnic config bar mapping
>   net/nfp: add support for vNIC Rx/Tx bar mappings
>   net/nfp: support PF devices inside PMD initialization
>   net/nfp: allocate ethernet device from PF probe function
>   net/nfp: support PF multiport
>   net/nfp: add NSP support for HW link configuration
>   net/nfp: add support for HW port link configuration
>   net/nfp: read PF port MAC addr using NSP
>   doc: update NFP with PF support information
>   doc: update release notes with NFP PF support
>   doc: create different features files for NFP drivers

Series applied to dpdk-next-net/master, thanks.

(Except patch 16/18, instead of it applied patch dpdk.org/patch/28425/)

Re: [PATCH v2 17/18] doc: update release notes with NFP PF support

[Ferruh Yigit]

On 9/1/2017 3:12 PM, Alejandro Lucero wrote:
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---
>  doc/guides/rel_notes/release_17_11.rst | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst
> index 170f4f9..8773b08 100644
> --- a/doc/guides/rel_notes/release_17_11.rst
> +++ b/doc/guides/rel_notes/release_17_11.rst
> @@ -69,6 +69,15 @@ EAL
>  Drivers
>  ~~~~~~~

Note seems added into "Resolved Issues" section, I have moved it to the
"New Features" section, can you please confirm patch from latest next-net?

>  
> +* **nfp: Added PF support.**
> +
> +  Previously Netronome's NFP PMD had just support for VFs. PF support is
> +  just as a basic DPDK port and has no VF management yet.
> +
> +  PF support comes with firmware upload support which allows the PMD to
> +  independently work from kernel netdev NFP drivers.
> +
> +  NFP 4000 devices are also now supported along with previous 6000 devices.
>  
>  Libraries
>  ~~~~~~~~~
> 

Re: [PATCH v2 17/18] doc: update release notes with NFP PF support

[Alejandro Lucero]

On Tue, Sep 19, 2017 at 10:53 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 9/1/2017 3:12 PM, Alejandro Lucero wrote:
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> > ---
> >  doc/guides/rel_notes/release_17_11.rst | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/release_17_11.rst
> b/doc/guides/rel_notes/release_17_11.rst
> > index 170f4f9..8773b08 100644
> > --- a/doc/guides/rel_notes/release_17_11.rst
> > +++ b/doc/guides/rel_notes/release_17_11.rst
> > @@ -69,6 +69,15 @@ EAL
> >  Drivers
> >  ~~~~~~~
>
> Note seems added into "Resolved Issues" section, I have moved it to the
> "New Features" section, can you please confirm patch from latest next-net?
>
>
It should be inside the "New features" section.

Thanks!


> >
> > +* **nfp: Added PF support.**
> > +
> > +  Previously Netronome's NFP PMD had just support for VFs. PF support is
> > +  just as a basic DPDK port and has no VF management yet.
> > +
> > +  PF support comes with firmware upload support which allows the PMD to
> > +  independently work from kernel netdev NFP drivers.
> > +
> > +  NFP 4000 devices are also now supported along with previous 6000
> devices.
> >
> >  Libraries
> >  ~~~~~~~~~
> >
>
>