[PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver

[PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver

[Kamil Rytarowski]

This series of patches adds support for secondary queue set in nicvf thunderx
driver

There are two types of VFs:
 - Primary VF
 - Secondary VF

Each port consist of a primary VF and n secondary VF(s). Each VF provides 8
Tx/Rx queues to a port. In case port is configured to use more than 8 queues,
then it requires one (or more) secondary VF. Each secondary VF adds additional
8 queues to the queue set.

During PMD driver initialization, the primary VF's are enumerated by checking the
specific flag (see READY message). They are at the beginning of  VF list (the remain
ones are secondary VF's).

The primary VFs are used as master queue sets. Secondary VFs provide
additional queue sets for primary ones. If a port is configured for more then
8 queues then it will request for additional queues from secondary VFs.

Secondary VFs cannot be shared between primary VFs.

Primary VFs are present on the tail of the 'Network devices using kernel
driver' list, secondary VFs are on the remaining tail of the list.

The VNIC driver in the multiqueue setup works differently than other drivers
like `ixgbe`. We need to bind separately each specific queue set device with
the ``tools/dpdk-devbind.py`` utility.

Depending on the hardware used, the kernel driver sets a threshold ``vf_id``.
VFs that try to attach with an id below or equal to this boundary are
considered primary VFs. VFs that try to attach with an id above this boundary
are considered secondary VFs.

This patchset also contains other cleanups and improvements like fixing
erroneous checksum calculation and preparing the thunderx driver for the multi
queue set feature support.


These changes base on the following pending patches:

[dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver
http://dpdk.org/dev/patchwork/patch/14963/

[dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag
http://dpdk.org/dev/patchwork/patch/14964/

[dpdk-dev,3/3] net/thunderx: add 81xx SoC support
http://dpdk.org/dev/patchwork/patch/14965/

Kamil Rytarowski (13):
  net/thunderx: cleanup the driver before adding new features
  net/thunderx: correct transmit checksum handling
  net/thunderx/base: add family of functions to store qsets
  net/thunderx/base: add secondary queue set support
  net/thunderx: add family of functions to store DPDK qsets
  net/thunderx: add secondary queue set in interrupt functions
  net/thunderx: fix multiprocess support in stats
  net/thunderx: add helper utils for secondary qset support
  net/thunderx: add secondary qset support in dev stop/close
  net/thunderx: add secondary qset support in device start
  net/thunderx: add secondary qset support in device configure
  net/thunderx: add final bits for secondary queue support
  net/thunderx: document secondary queue set support

 doc/guides/nics/thunderx.rst              | 114 ++++-
 drivers/net/thunderx/Makefile             |   2 +
 drivers/net/thunderx/base/nicvf_bsvf.c    |  72 +++
 drivers/net/thunderx/base/nicvf_bsvf.h    |  76 +++
 drivers/net/thunderx/base/nicvf_hw.c      |  10 +-
 drivers/net/thunderx/base/nicvf_hw.h      |   6 +-
 drivers/net/thunderx/base/nicvf_hw_defs.h |   1 +
 drivers/net/thunderx/base/nicvf_mbox.c    |  34 +-
 drivers/net/thunderx/base/nicvf_mbox.h    |  21 +-
 drivers/net/thunderx/nicvf_ethdev.c       | 753 +++++++++++++++++++++---------
 drivers/net/thunderx/nicvf_ethdev.h       |  39 ++
 drivers/net/thunderx/nicvf_rxtx.c         |  14 +-
 drivers/net/thunderx/nicvf_struct.h       |   6 +-
 drivers/net/thunderx/nicvf_svf.c          |  78 ++++
 drivers/net/thunderx/nicvf_svf.h          |  66 +++
 15 files changed, 1046 insertions(+), 246 deletions(-)
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
 create mode 100644 drivers/net/thunderx/nicvf_svf.c
 create mode 100644 drivers/net/thunderx/nicvf_svf.h

-- 
1.9.1

[PATCH 01/13] net/thunderx: cleanup the driver before adding new features

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Refactored features:
 - enable nicvf_qset_rbdr_precharge to handle handle secondary queue sets
 - rte_free already handles NULL pointer
 - check mempool flags to predict being contiguous in memory
 - allow to use mempool with multiple memory chunks
 - simplify local construct of accessing nb_rx_queus

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/base/nicvf_hw.c | 10 +++++-----
 drivers/net/thunderx/base/nicvf_hw.h |  6 +++---
 drivers/net/thunderx/nicvf_ethdev.c  | 32 ++++++++++++--------------------
 3 files changed, 20 insertions(+), 28 deletions(-)

diff --git a/drivers/net/thunderx/base/nicvf_hw.c b/drivers/net/thunderx/base/nicvf_hw.c
index 4bdd183..1f08ef2 100644
--- a/drivers/net/thunderx/base/nicvf_hw.c
+++ b/drivers/net/thunderx/base/nicvf_hw.c
@@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
 		return NICVF_ERR_BASE_INIT;
 
 	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
-		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
+		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
 
 	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
 		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
@@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
 }
 
 int
-nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
-			  rbdr_pool_get_handler handler,
-			  void *opaque, uint32_t max_buffs)
+nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+			  uint16_t ridx, rbdr_pool_get_handler handler,
+			  uint32_t max_buffs)
 {
 	struct rbdr_entry_t *desc, *desc0;
 	struct nicvf_rbdr *rbdr = nic->rbdr;
@@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
 		if (count >= max_buffs)
 			break;
 		desc0 = desc + count;
-		phy = handler(opaque);
+		phy = handler(dev, nic);
 		if (phy) {
 			desc0->full_addr = phy;
 			count++;
diff --git a/drivers/net/thunderx/base/nicvf_hw.h b/drivers/net/thunderx/base/nicvf_hw.h
index a6cda82..2b8738b 100644
--- a/drivers/net/thunderx/base/nicvf_hw.h
+++ b/drivers/net/thunderx/base/nicvf_hw.h
@@ -85,7 +85,7 @@ enum nicvf_err_e {
 	NICVF_ERR_RSS_GET_SZ,    /* -8171 */
 };
 
-typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
+typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
 
 struct nicvf_hw_rx_qstats {
 	uint64_t q_rx_bytes;
@@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
 
 int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
 int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
-int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
-			      rbdr_pool_get_handler handler, void *opaque,
+int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+			      uint16_t ridx, rbdr_pool_get_handler handler,
 			      uint32_t max_buffs);
 int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
 
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 4402f6a..48f2cd2 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
 	int ret;
 
 	/* Count started rx queues */
-	for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+	for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
 		if (dev->data->rx_queue_state[qidx] ==
 		    RTE_ETH_QUEUE_STATE_STARTED)
 			qcnt++;
@@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
 static void
 nicvf_dev_rx_queue_release(void *rx_queue)
 {
-	struct nicvf_rxq *rxq = rx_queue;
-
 	PMD_INIT_FUNC_TRACE();
 
-	if (rxq)
-		rte_free(rxq);
+	rte_free(rx_queue);
 }
 
 static int
@@ -1087,9 +1084,9 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
 		socket_id, nic->node);
 
-	/* Mempool memory should be contiguous */
-	if (mp->nb_mem_chunks != 1) {
-		PMD_INIT_LOG(ERR, "Non contiguous mempool, check huge page sz");
+	/* Mempool memory must be contiguous */
+	if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG) {
+		PMD_INIT_LOG(ERR, "Mempool memory must be contiguous");
 		return -EINVAL;
 	}
 
@@ -1212,15 +1209,16 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 }
 
 static nicvf_phys_addr_t
-rbdr_rte_mempool_get(void *opaque)
+rbdr_rte_mempool_get(void *dev, void *opaque)
 {
 	uint16_t qidx;
 	uintptr_t mbuf;
 	struct nicvf_rxq *rxq;
-	struct nicvf *nic = nicvf_pmd_priv((struct rte_eth_dev *)opaque);
+	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
+	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
 
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
-		rxq = nic->eth_dev->data->rx_queues[qidx];
+	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
+		rxq = eth_dev->data->rx_queues[qidx];
 		/* Maintain equal buffer count across all pools */
 		if (rxq->precharge_cnt >= rxq->qlen_mask)
 			continue;
@@ -1354,8 +1352,8 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	}
 
 	/* Fill rte_mempool buffers in RBDR pool and precharge it */
-	ret = nicvf_qset_rbdr_precharge(nic, 0, rbdr_rte_mempool_get,
-					dev, total_rxq_desc);
+	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
+					total_rxq_desc);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
 		goto qset_rbdr_reclaim;
@@ -1721,12 +1719,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 		goto malloc_fail;
 	}
 
-	ret = nicvf_mbox_get_rss_size(nic);
-	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to get rss table size");
-		goto malloc_fail;
-	}
-
 	PMD_INIT_LOG(INFO, "Port %d (%x:%x) mac=%02x:%02x:%02x:%02x:%02x:%02x",
 		eth_dev->data->port_id, nic->vendor_id, nic->device_id,
 		nic->mac_addr[0], nic->mac_addr[1], nic->mac_addr[2],
-- 
1.9.1

[PATCH 02/13] net/thunderx: correct transmit checksum handling

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

The symbols PKT_TX_TCP_CKSUM and PKT_TX_UDP_CKSUM are not bits on a
bitmask. Set l3_offset always for TX offloads, not just for PKT_TX_IP_CKSUM
being true.

Fixes: 1c421f18e095 ("net/thunderx: add single and multi-segment Tx")

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_rxtx.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_rxtx.c b/drivers/net/thunderx/nicvf_rxtx.c
index eb51a72..e15c730 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -70,19 +70,20 @@ fill_sq_desc_header(union sq_entry_t *entry, struct rte_mbuf *pkt)
 	ol_flags = pkt->ol_flags & NICVF_TX_OFFLOAD_MASK;
 	if (unlikely(ol_flags)) {
 		/* L4 cksum */
-		if (ol_flags & PKT_TX_TCP_CKSUM)
+		uint64_t l4_flags = ol_flags & PKT_TX_L4_MASK;
+		if (l4_flags == PKT_TX_TCP_CKSUM)
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_TCP;
-		else if (ol_flags & PKT_TX_UDP_CKSUM)
+		else if (l4_flags == PKT_TX_UDP_CKSUM)
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_UDP;
 		else
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_DISABLE;
+
+		sqe.hdr.l3_offset = pkt->l2_len;
 		sqe.hdr.l4_offset = pkt->l3_len + pkt->l2_len;
 
 		/* L3 cksum */
-		if (ol_flags & PKT_TX_IP_CKSUM) {
+		if (ol_flags & PKT_TX_IP_CKSUM)
 			sqe.hdr.csum_l3 = 1;
-			sqe.hdr.l3_offset = pkt->l2_len;
-		}
 	}
 
 	entry->buff[0] = sqe.buff[0];
-- 
1.9.1

[PATCH 03/13] net/thunderx/base: add family of functions to store qsets

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

This interface (nicvf_bsvf) will be used for secondary queue set support.

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/Makefile          |  1 +
 drivers/net/thunderx/base/nicvf_bsvf.c | 72 ++++++++++++++++++++++++++++++++
 drivers/net/thunderx/base/nicvf_bsvf.h | 76 ++++++++++++++++++++++++++++++++++
 3 files changed, 149 insertions(+)
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h

diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 8ea6b45..711400a 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -57,6 +57,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_rxtx.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
 
 ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
 CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.c b/drivers/net/thunderx/base/nicvf_bsvf.c
new file mode 100644
index 0000000..9e028a3
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.c
@@ -0,0 +1,72 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <assert.h>
+#include <stddef.h>
+#include <err.h>
+
+#include "nicvf_bsvf.h"
+#include "nicvf_plat.h"
+
+static SIMPLEQ_HEAD(, svf_entry) head = SIMPLEQ_HEAD_INITIALIZER(head);
+
+void
+nicvf_bsvf_push(struct svf_entry *entry)
+{
+	assert(entry != NULL);
+	assert(entry->vf != NULL);
+
+	SIMPLEQ_INSERT_TAIL(&head, entry, next);
+}
+
+struct svf_entry *
+nicvf_bsvf_pop(void)
+{
+	struct svf_entry *entry;
+
+	assert(!SIMPLEQ_EMPTY(&head));
+
+	entry = SIMPLEQ_FIRST(&head);
+
+	assert(entry != NULL);
+	assert(entry->vf != NULL);
+
+	SIMPLEQ_REMOVE_HEAD(&head, next);
+
+	return entry;
+}
+
+int
+nicvf_bsvf_empty(void)
+{
+	return SIMPLEQ_EMPTY(&head);
+}
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.h b/drivers/net/thunderx/base/nicvf_bsvf.h
new file mode 100644
index 0000000..5d5a25e
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.h
@@ -0,0 +1,76 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __THUNDERX_NICVF_BSVF_H__
+#define __THUNDERX_NICVF_BSVF_H__
+
+#include <sys/queue.h>
+
+struct nicvf;
+
+/**
+ * The base queue structure to hold secondary qsets.
+ */
+struct svf_entry {
+	SIMPLEQ_ENTRY(svf_entry) next; /**< Next element's pointer */
+	struct nicvf *vf;              /**< Holder of a secondary qset */
+};
+
+/**
+ * Enqueue new entry to secondary qsets.
+ *
+ * @param entry
+ *   Entry to be enqueued.
+ */
+void
+nicvf_bsvf_push(struct svf_entry *entry);
+
+/**
+ * Dequeue an entry from secondary qsets.
+ *
+ * @return
+ *   Dequeued entry.
+ */
+struct svf_entry *
+nicvf_bsvf_pop(void);
+
+/**
+ * Check if the queue of secondary qsets is empty.
+ *
+ * @return
+ *   0 on non-empty
+ *   otherwise empty
+ */
+int
+nicvf_bsvf_empty(void);
+
+#endif /* __THUNDERX_NICVF_BSVF_H__  */
-- 
1.9.1

[PATCH 04/13] net/thunderx/base: add secondary queue set support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Changes:
 - add new message sqs_alloc in mailbox
 - add a queue container to hold secondary qsets.
 - add nicvf_mbox_request_sqs
 - handle new mailbox messages for secondary queue set support
 - register secondary queue sets for furthe reuse
 - register the number secondary queue sets in MSG_QS_CFG

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/base/nicvf_hw_defs.h |  1 +
 drivers/net/thunderx/base/nicvf_mbox.c    | 34 ++++++++++++++++++++++++++++++-
 drivers/net/thunderx/base/nicvf_mbox.h    | 21 +++++++++++++++++--
 drivers/net/thunderx/nicvf_struct.h       |  5 +++++
 4 files changed, 58 insertions(+), 3 deletions(-)

diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h b/drivers/net/thunderx/base/nicvf_hw_defs.h
index 2f2b225..3b947e0 100644
--- a/drivers/net/thunderx/base/nicvf_hw_defs.h
+++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
@@ -207,6 +207,7 @@
 #define NICVF_CQE_RX2_RBPTR_WORD        (7)
 
 #define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
+#define assert_if_secondary(nic) assert((nic)->sqs_mode == 0)
 
 typedef uint64_t nicvf_phys_addr_t;
 
diff --git a/drivers/net/thunderx/base/nicvf_mbox.c b/drivers/net/thunderx/base/nicvf_mbox.c
index 9c5cd83..974ce97 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.c
+++ b/drivers/net/thunderx/base/nicvf_mbox.c
@@ -62,6 +62,9 @@ static const char *mbox_message[NIC_MBOX_MSG_MAX] =  {
 	[NIC_MBOX_MSG_RESET_STAT_COUNTER] = "NIC_MBOX_MSG_RESET_STAT_COUNTER",
 	[NIC_MBOX_MSG_CFG_DONE]           = "NIC_MBOX_MSG_CFG_DONE",
 	[NIC_MBOX_MSG_SHUTDOWN]           = "NIC_MBOX_MSG_SHUTDOWN",
+	[NIC_MBOX_MSG_RES_BIT]            = "NIC_MBOX_MSG_RES_BIT",
+	[NIC_MBOX_MSG_RSS_SIZE_RES_BIT]   = "NIC_MBOX_MSG_RSS_SIZE",
+	[NIC_MBOX_MSG_ALLOC_SQS_RES_BIT]  = "NIC_MBOX_MSG_ALLOC_SQS",
 };
 
 static inline const char * __attribute__((unused))
@@ -173,7 +176,7 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
 	case NIC_MBOX_MSG_NACK:
 		nic->pf_nacked = true;
 		break;
-	case NIC_MBOX_MSG_RSS_SIZE:
+	case NIC_MBOX_MSG_RSS_SIZE_RES_BIT:
 		nic->rss_info.rss_size = mbx.rss_size.ind_tbl_size;
 		nic->pf_acked = true;
 		break;
@@ -183,6 +186,13 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
 		nic->speed = mbx.link_status.speed;
 		nic->pf_acked = true;
 		break;
+	case NIC_MBOX_MSG_ALLOC_SQS_RES_BIT:
+		assert_if_secondary(nic);
+		assert(mbx.sqs_alloc.qs_count == nic->sqs_count);
+		for (i = 0; i < mbx.sqs_alloc.qs_count; i++)
+			assert(mbx.sqs_alloc.svf[i] == nic->snicvf[i]->vf_id);
+		nic->pf_acked = true;
+		break;
 	default:
 		nicvf_log_error("Invalid message from PF, msg_id=0x%hhx %s",
 				mbx.msg.msg, nicvf_mbox_msg_str(mbx.msg.msg));
@@ -314,11 +324,33 @@ nicvf_mbox_qset_config(struct nicvf *nic, struct pf_qs_cfg *qs_cfg)
 	/* Send a mailbox msg to PF to config Qset */
 	mbx.msg.msg = NIC_MBOX_MSG_QS_CFG;
 	mbx.qs.num = nic->vf_id;
+	mbx.qs.sqs_count = nic->sqs_count;
 	mbx.qs.cfg = qs_cfg->value;
 	return nicvf_mbox_send_msg_to_pf(nic, &mbx);
 }
 
 int
+nicvf_mbox_request_sqs(struct nicvf *nic)
+{
+	struct nic_mbx mbx = { .msg = { 0 } };
+	size_t i;
+
+	assert_if_secondary(nic);
+	assert(nic->sqs_count > 0);
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	mbx.sqs_alloc.msg = NIC_MBOX_MSG_ALLOC_SQS;
+	mbx.sqs_alloc.spec = 1;
+	mbx.sqs_alloc.qs_count = nic->sqs_count;
+
+	/* Set no of Rx/Tx queues in each of the SQsets */
+	for (i = 0; i < nic->sqs_count; i++)
+		mbx.sqs_alloc.svf[i] = nic->snicvf[i]->vf_id;
+
+	return nicvf_mbox_send_msg_to_pf(nic, &mbx);
+}
+
+int
 nicvf_mbox_rq_drop_config(struct nicvf *nic, uint16_t qidx, bool enable)
 {
 	struct nic_mbx mbx = { .msg = { 0 } };
diff --git a/drivers/net/thunderx/base/nicvf_mbox.h b/drivers/net/thunderx/base/nicvf_mbox.h
index 7c0c6a9..084f3a7 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.h
+++ b/drivers/net/thunderx/base/nicvf_mbox.h
@@ -36,6 +36,7 @@
 #include <stdint.h>
 
 #include "nicvf_plat.h"
+#include "../nicvf_struct.h"
 
 /* PF <--> VF Mailbox communication
  * Two 64bit registers are shared between PF and VF for each VF
@@ -67,10 +68,16 @@
 #define	NIC_MBOX_MSG_ALLOC_SQS		0x12	/* Allocate secondary Qset */
 #define	NIC_MBOX_MSG_LOOPBACK		0x16	/* Set interface in loopback */
 #define	NIC_MBOX_MSG_RESET_STAT_COUNTER 0x17	/* Reset statistics counters */
-#define	NIC_MBOX_MSG_CFG_DONE		0xF0	/* VF configuration done */
-#define	NIC_MBOX_MSG_SHUTDOWN		0xF1	/* VF is being shutdown */
+#define	NIC_MBOX_MSG_CFG_DONE		0x7E	/* VF configuration done */
+#define	NIC_MBOX_MSG_SHUTDOWN		0x7F	/* VF is being shutdown */
+#define	NIC_MBOX_MSG_RES_BIT		0x80	/* Reset bit from PF */
 #define	NIC_MBOX_MSG_MAX		0x100	/* Maximum number of messages */
 
+#define NIC_MBOX_MSG_RSS_SIZE_RES_BIT \
+	(NIC_MBOX_MSG_RSS_SIZE | NIC_MBOX_MSG_RES_BIT)
+#define NIC_MBOX_MSG_ALLOC_SQS_RES_BIT \
+	(NIC_MBOX_MSG_ALLOC_SQS | NIC_MBOX_MSG_RES_BIT)
+
 /* Get vNIC VF configuration */
 struct nic_cfg_msg {
 	uint8_t    msg;
@@ -155,6 +162,14 @@ struct bgx_link_status {
 	uint32_t   speed;
 };
 
+/* Allocate additional SQS to VF */
+struct sqs_alloc {
+	uint8_t    msg;
+	uint8_t    spec;
+	uint8_t    qs_count;
+	uint8_t    svf[MAX_SQS_PER_VF];
+};
+
 /* Set interface in loopback mode */
 struct set_loopback {
 	uint8_t    msg;
@@ -201,6 +216,7 @@ union {
 	struct rss_sz_msg	rss_size;
 	struct rss_cfg_msg	rss_cfg;
 	struct bgx_link_status  link_status;
+	struct sqs_alloc	sqs_alloc;
 	struct set_loopback	lbk;
 	struct reset_stat_cfg	reset_stat;
 };
@@ -211,6 +227,7 @@ NICVF_STATIC_ASSERT(sizeof(struct nic_mbx) <= 16);
 int nicvf_handle_mbx_intr(struct nicvf *nic);
 int nicvf_mbox_check_pf_ready(struct nicvf *nic);
 int nicvf_mbox_qset_config(struct nicvf *nic, struct pf_qs_cfg *qs_cfg);
+int nicvf_mbox_request_sqs(struct nicvf *nic);
 int nicvf_mbox_rq_config(struct nicvf *nic, uint16_t qidx,
 			 struct pf_rq_cfg *pf_rq_cfg);
 int nicvf_mbox_sq_config(struct nicvf *nic, uint16_t qidx);
diff --git a/drivers/net/thunderx/nicvf_struct.h b/drivers/net/thunderx/nicvf_struct.h
index c52545d..a72f752 100644
--- a/drivers/net/thunderx/nicvf_struct.h
+++ b/drivers/net/thunderx/nicvf_struct.h
@@ -119,6 +119,11 @@ struct nicvf {
 	uint16_t mtu;
 	bool vlan_filter_en;
 	uint8_t mac_addr[ETHER_ADDR_LEN];
+	/* secondary queue set support */
+	uint8_t sqs_id;
+	uint8_t sqs_count;
+#define MAX_SQS_PER_VF 11
+	struct nicvf *snicvf[MAX_SQS_PER_VF];
 } __rte_cache_aligned;
 
 #endif /* _THUNDERX_NICVF_STRUCT_H */
-- 
1.9.1

[PATCH 05/13] net/thunderx: add family of functions to store DPDK qsets

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

These functions (nicvf_svf) are DPDK specialization of base/nicvf_bsvf.[ch]
ones.

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/Makefile    |  1 +
 drivers/net/thunderx/nicvf_svf.c | 78 ++++++++++++++++++++++++++++++++++++++++
 drivers/net/thunderx/nicvf_svf.h | 66 ++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/net/thunderx/nicvf_svf.c
 create mode 100644 drivers/net/thunderx/nicvf_svf.h

diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 711400a..bcab5f9 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -58,6 +58,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_svf.c
 
 ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
 CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/nicvf_svf.c b/drivers/net/thunderx/nicvf_svf.c
new file mode 100644
index 0000000..f746e94
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.c
@@ -0,0 +1,78 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <assert.h>
+#include <stddef.h>
+
+#include <rte_debug.h>
+#include <rte_malloc.h>
+
+#include "base/nicvf_bsvf.h"
+
+#include "nicvf_svf.h"
+
+void
+nicvf_svf_push(struct nicvf *vf)
+{
+	struct svf_entry *entry = NULL;
+
+	assert(vf != NULL);
+
+	entry = rte_zmalloc("nicvf", sizeof(*entry), RTE_CACHE_LINE_SIZE);
+	if (entry == NULL)
+		rte_panic("Cannoc allocate memory for svf_entry\n");
+
+	entry->vf = vf;
+
+	nicvf_bsvf_push(entry);
+}
+
+struct nicvf *
+nicvf_svf_pop(void)
+{
+	struct nicvf *vf;
+	struct svf_entry *entry;
+
+	entry = nicvf_bsvf_pop();
+
+	vf = entry->vf;
+
+	rte_free(entry);
+
+	return vf;
+}
+
+int
+nicvf_svf_empty(void)
+{
+	return nicvf_bsvf_empty();
+}
diff --git a/drivers/net/thunderx/nicvf_svf.h b/drivers/net/thunderx/nicvf_svf.h
new file mode 100644
index 0000000..6471aa5
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.h
@@ -0,0 +1,66 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __THUNDERX_NICVF_SVF_H__
+#define __THUNDERX_NICVF_SVF_H__
+
+struct nicvf;
+
+/**
+ * Enqueue new VF to secondary qsets.
+ *
+ * @param entry
+ *   Entry to be enqueued.
+ */
+void
+nicvf_svf_push(struct nicvf *vf);
+
+/**
+ * Dequeue a VF from secondary qsets.
+ *
+ * @return
+ *   Dequeued entry.
+ */
+struct nicvf *
+nicvf_svf_pop(void);
+
+/**
+ * Check if the queue of secondary qsets is empty.
+ *
+ * @return
+ *   0 on non-empty
+ *   otherwise empty
+ */
+int
+nicvf_svf_empty(void);
+
+#endif /* __THUNDERX_NICVF_SVF_H__  */
-- 
1.9.1

[PATCH 06/13] net/thunderx: add secondary queue set in interrupt functions

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 41 ++++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 17 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 48f2cd2..55f6480 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -101,31 +101,40 @@ nicvf_set_eth_link_status(struct nicvf *nic, struct rte_eth_link *link)
 static void
 nicvf_interrupt(void *arg)
 {
-	struct nicvf *nic = arg;
+	struct rte_eth_dev *dev = arg;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 
 	if (nicvf_reg_poll_interrupts(nic) == NIC_MBOX_MSG_BGX_LINK_CHANGE) {
-		if (nic->eth_dev->data->dev_conf.intr_conf.lsc)
-			nicvf_set_eth_link_status(nic,
-					&nic->eth_dev->data->dev_link);
-		_rte_eth_dev_callback_process(nic->eth_dev,
-				RTE_ETH_EVENT_INTR_LSC);
+		if (dev->data->dev_conf.intr_conf.lsc)
+			nicvf_set_eth_link_status(nic, &dev->data->dev_link);
+		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC);
 	}
 
 	rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
-				nicvf_interrupt, nic);
+				nicvf_interrupt, dev);
+}
+
+static void __rte_unused
+nicvf_vf_interrupt(void *arg)
+{
+	struct nicvf *nic = arg;
+
+	nicvf_reg_poll_interrupts(nic);
+
+	rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
+				nicvf_vf_interrupt, nic);
 }
 
 static int
-nicvf_periodic_alarm_start(struct nicvf *nic)
+nicvf_periodic_alarm_start(void (fn)(void *), void *arg)
 {
-	return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
-					nicvf_interrupt, nic);
+	return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000, fn, arg);
 }
 
 static int
-nicvf_periodic_alarm_stop(struct nicvf *nic)
+nicvf_periodic_alarm_stop(void (fn)(void *), void *arg)
 {
-	return rte_eal_alarm_cancel(nicvf_interrupt, nic);
+	return rte_eal_alarm_cancel(fn, arg);
 }
 
 /*
@@ -1512,12 +1521,10 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 static void
 nicvf_dev_close(struct rte_eth_dev *dev)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
-
 	PMD_INIT_FUNC_TRACE();
 
 	nicvf_dev_stop(dev);
-	nicvf_periodic_alarm_stop(nic);
+	nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
 }
 
 static int
@@ -1668,7 +1675,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 
 	nicvf_disable_all_interrupts(nic);
 
-	ret = nicvf_periodic_alarm_start(nic);
+	ret = nicvf_periodic_alarm_start(nicvf_interrupt, eth_dev);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to start period alarm");
 		goto fail;
@@ -1729,7 +1736,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 malloc_fail:
 	rte_free(eth_dev->data->mac_addrs);
 alarm_fail:
-	nicvf_periodic_alarm_stop(nic);
+	nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
 fail:
 	return ret;
 }
-- 
1.9.1

[PATCH 07/13] net/thunderx: fix multiprocess support in stats

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

In case of the multiprocess mode a shared nicvf struct between processes
cannot point with the eth_dev pointer to master device, therefore remove it
allong with references to it refactoring the code where needed.

Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 69 ++++++++++++++++++-------------------
 drivers/net/thunderx/nicvf_rxtx.c   |  3 +-
 drivers/net/thunderx/nicvf_struct.h |  1 -
 3 files changed, 36 insertions(+), 37 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 55f6480..b4ab505 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -497,14 +497,14 @@ nicvf_dev_rss_hash_update(struct rte_eth_dev *dev,
 }
 
 static int
-nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
-		    uint32_t desc_cnt)
+nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		    struct nicvf_rxq *rxq, uint16_t qidx, uint32_t desc_cnt)
 {
 	const struct rte_memzone *rz;
 	uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "cq_ring", qidx, ring_size,
-					NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+				      NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
 		return -ENOMEM;
@@ -520,13 +520,13 @@ nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
 }
 
 static int
-nicvf_qset_sq_alloc(struct nicvf *nic,  struct nicvf_txq *sq, uint16_t qidx,
-		    uint32_t desc_cnt)
+nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		    struct nicvf_txq *sq, uint16_t qidx, uint32_t desc_cnt)
 {
 	const struct rte_memzone *rz;
 	uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "sq", qidx, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
 				NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
@@ -543,7 +543,8 @@ nicvf_qset_sq_alloc(struct nicvf *nic,  struct nicvf_txq *sq, uint16_t qidx,
 }
 
 static int
-nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
+nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		      uint32_t desc_cnt, uint32_t buffsz)
 {
 	struct nicvf_rbdr *rbdr;
 	const struct rte_memzone *rz;
@@ -558,7 +559,7 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
 	}
 
 	ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "rbdr", 0, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
 				   NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
@@ -583,14 +584,15 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
 }
 
 static void
-nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic __rte_unused,
+			nicvf_phys_addr_t phy)
 {
 	uint16_t qidx;
 	void *obj;
 	struct nicvf_rxq *rxq;
 
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
-		rxq = nic->eth_dev->data->rx_queues[qidx];
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+		rxq = dev->data->rx_queues[qidx];
 		if (rxq->precharge_cnt) {
 			obj = (void *)nicvf_mbuff_phy2virt(phy,
 							   rxq->mbuf_phys_off);
@@ -602,7 +604,7 @@ nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
 }
 
 static inline void
-nicvf_rbdr_release_mbufs(struct nicvf *nic)
+nicvf_rbdr_release_mbufs(struct rte_eth_dev *dev, struct nicvf *nic)
 {
 	uint32_t qlen_mask, head;
 	struct rbdr_entry_t *entry;
@@ -612,7 +614,7 @@ nicvf_rbdr_release_mbufs(struct nicvf *nic)
 	head = rbdr->head;
 	while (head != rbdr->tail) {
 		entry = rbdr->desc + head;
-		nicvf_rbdr_release_mbuf(nic, entry->full_addr);
+		nicvf_rbdr_release_mbuf(dev, nic, entry->full_addr);
 		head++;
 		head = head & qlen_mask;
 	}
@@ -724,14 +726,13 @@ nicvf_configure_rss(struct rte_eth_dev *dev)
 			dev->data->dev_conf.rx_adv_conf.rss_conf.rss_hf);
 	PMD_DRV_LOG(INFO, "mode=%d rx_queues=%d loopback=%d rsshf=0x%" PRIx64,
 		    dev->data->dev_conf.rxmode.mq_mode,
-		    nic->eth_dev->data->nb_rx_queues,
-		    nic->eth_dev->data->dev_conf.lpbk_mode, rsshf);
+		    dev->data->nb_rx_queues,
+		    dev->data->dev_conf.lpbk_mode, rsshf);
 
 	if (dev->data->dev_conf.rxmode.mq_mode == ETH_MQ_RX_NONE)
 		ret = nicvf_rss_term(nic);
 	else if (dev->data->dev_conf.rxmode.mq_mode == ETH_MQ_RX_RSS)
-		ret = nicvf_rss_config(nic,
-				       nic->eth_dev->data->nb_rx_queues, rsshf);
+		ret = nicvf_rss_config(nic, dev->data->nb_rx_queues, rsshf);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to configure RSS %d", ret);
 
@@ -915,7 +916,7 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 		return -ENOMEM;
 	}
 
-	if (nicvf_qset_sq_alloc(nic, txq, qidx, nb_desc)) {
+	if (nicvf_qset_sq_alloc(dev, nic, txq, qidx, nb_desc)) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for sq %d", qidx);
 		nicvf_dev_tx_queue_release(txq);
 		return -ENOMEM;
@@ -932,12 +933,11 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 }
 
 static inline void
-nicvf_rx_queue_release_mbufs(struct nicvf_rxq *rxq)
+nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 {
 	uint32_t rxq_cnt;
 	uint32_t nb_pkts, released_pkts = 0;
 	uint32_t refill_cnt = 0;
-	struct rte_eth_dev *dev = rxq->nic->eth_dev;
 	struct rte_mbuf *rx_pkts[NICVF_MAX_RX_FREE_THRESH];
 
 	if (dev->rx_pkt_burst == NULL)
@@ -1017,7 +1017,7 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
 
 	other_error = ret;
 	rxq = dev->data->rx_queues[qidx];
-	nicvf_rx_queue_release_mbufs(rxq);
+	nicvf_rx_queue_release_mbufs(dev, rxq);
 	nicvf_rx_queue_reset(rxq);
 
 	ret = nicvf_qset_cq_reclaim(nic, qidx);
@@ -1156,7 +1156,7 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 
 	/* Alloc completion queue */
-	if (nicvf_qset_cq_alloc(nic, rxq, rxq->queue_id, nb_desc)) {
+	if (nicvf_qset_cq_alloc(dev, nic, rxq, rxq->queue_id, nb_desc)) {
 		PMD_INIT_LOG(ERR, "failed to allocate cq %u", rxq->queue_id);
 		nicvf_dev_rx_queue_release(rxq);
 		return -ENOMEM;
@@ -1274,7 +1274,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	 */
 
 	/* Validate RBDR buff size */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		mbp_priv = rte_mempool_get_priv(rxq->pool);
 		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
@@ -1292,7 +1292,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	}
 
 	/* Validate mempool attributes */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		rxq->mbuf_phys_off = nicvf_mempool_phy_offset(rxq->pool);
 		mbuf = rte_pktmbuf_alloc(rxq->pool);
@@ -1316,12 +1316,12 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 
 	/* Check the level of buffers in the pool */
 	total_rxq_desc = 0;
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		/* Count total numbers of rxq descs */
 		total_rxq_desc += rxq->qlen_mask + 1;
 		exp_buffs = RTE_MEMPOOL_CACHE_MAX_SIZE + rxq->rx_free_thresh;
-		exp_buffs *= nic->eth_dev->data->nb_rx_queues;
+		exp_buffs *= dev->data->nb_rx_queues;
 		if (rte_mempool_avail_count(rxq->pool) < exp_buffs) {
 			PMD_INIT_LOG(ERR, "Buff shortage in pool=%s (%d/%d)",
 				     rxq->pool->name,
@@ -1347,7 +1347,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 
 	/* Allocate RBDR and RBDR ring desc */
 	nb_rbdr_desc = nicvf_qsize_rbdr_roundup(total_rxq_desc);
-	ret = nicvf_qset_rbdr_alloc(nic, nb_rbdr_desc, rbdrsz);
+	ret = nicvf_qset_rbdr_alloc(dev, nic, nb_rbdr_desc, rbdrsz);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc");
 		goto qset_reclaim;
@@ -1372,7 +1372,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		     nic->rbdr->tail, nb_rbdr_desc);
 
 	/* Configure RX queues */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		ret = nicvf_start_rx_queue(dev, qidx);
 		if (ret)
 			goto start_rxq_error;
@@ -1382,7 +1382,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
 
 	/* Configure TX queues */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_tx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
 		ret = nicvf_start_tx_queue(dev, qidx);
 		if (ret)
 			goto start_txq_error;
@@ -1441,14 +1441,14 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 qset_rss_error:
 	nicvf_rss_term(nic);
 start_txq_error:
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_tx_queues; qidx++)
+	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
 		nicvf_stop_tx_queue(dev, qidx);
 start_rxq_error:
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
 		nicvf_stop_rx_queue(dev, qidx);
 qset_rbdr_reclaim:
 	nicvf_qset_rbdr_reclaim(nic, 0);
-	nicvf_rbdr_release_mbufs(nic);
+	nicvf_rbdr_release_mbufs(dev, nic);
 qset_rbdr_free:
 	if (nic->rbdr) {
 		rte_free(nic->rbdr);
@@ -1494,7 +1494,7 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 
 	/* Move all charged buffers in RBDR back to pool */
 	if (nic->rbdr != NULL)
-		nicvf_rbdr_release_mbufs(nic);
+		nicvf_rbdr_release_mbufs(dev, nic);
 
 	/* Reclaim CPI configuration */
 	if (!nic->sqs_mode) {
@@ -1659,7 +1659,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 	nic->vendor_id = pci_dev->id.vendor_id;
 	nic->subsystem_device_id = pci_dev->id.subsystem_device_id;
 	nic->subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
-	nic->eth_dev = eth_dev;
 
 	PMD_INIT_LOG(DEBUG, "nicvf: device (%x:%x) %u:%u:%u:%u",
 			pci_dev->id.vendor_id, pci_dev->id.device_id,
diff --git a/drivers/net/thunderx/nicvf_rxtx.c b/drivers/net/thunderx/nicvf_rxtx.c
index e15c730..fc43b74 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -368,7 +368,8 @@ nicvf_fill_rbdr(struct nicvf_rxq *rxq, int to_fill)
 	void *obj_p[NICVF_MAX_RX_FREE_THRESH] __rte_cache_aligned;
 
 	if (unlikely(rte_mempool_get_bulk(rxq->pool, obj_p, to_fill) < 0)) {
-		rxq->nic->eth_dev->data->rx_mbuf_alloc_failed += to_fill;
+		rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed +=
+			to_fill;
 		return 0;
 	}
 
diff --git a/drivers/net/thunderx/nicvf_struct.h b/drivers/net/thunderx/nicvf_struct.h
index a72f752..c900e12 100644
--- a/drivers/net/thunderx/nicvf_struct.h
+++ b/drivers/net/thunderx/nicvf_struct.h
@@ -113,7 +113,6 @@ struct nicvf {
 	uint16_t subsystem_vendor_id;
 	struct nicvf_rbdr *rbdr;
 	struct nicvf_rss_reta_info rss_info;
-	struct rte_eth_dev *eth_dev;
 	struct rte_intr_handle intr_handle;
 	uint8_t cpi_alg;
 	uint16_t mtu;
-- 
1.9.1

[PATCH 08/13] net/thunderx: add helper utils for secondary qset support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.h | 39 +++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/net/thunderx/nicvf_ethdev.h b/drivers/net/thunderx/nicvf_ethdev.h
index 34447e0..e162364 100644
--- a/drivers/net/thunderx/nicvf_ethdev.h
+++ b/drivers/net/thunderx/nicvf_ethdev.h
@@ -87,6 +87,17 @@ nicvf_mbuff_meta_length(struct rte_mbuf *mbuf)
 	return (uint16_t)((uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf);
 }
 
+static inline uint16_t
+nicvf_netdev_qidx(struct nicvf *nic, uint8_t local_qidx)
+{
+	uint16_t global_qidx = local_qidx;
+
+	if (nic->sqs_mode)
+		global_qidx += ((nic->sqs_id + 1) * MAX_CMP_QUEUES_PER_QS);
+
+	return global_qidx;
+}
+
 /*
  * Simple phy2virt functions assuming mbufs are in a single huge page
  * V = P + offset
@@ -104,4 +115,32 @@ nicvf_mbuff_virt2phy(uintptr_t virt, uint64_t mbuf_phys_off)
 	return (phys_addr_t)(virt - mbuf_phys_off);
 }
 
+static inline void
+nicvf_tx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *tx_start,
+	       uint16_t *tx_end)
+{
+	uint16_t tmp;
+
+	*tx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+				    MAX_SND_QUEUES_PER_QS);
+	tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+			     MAX_SND_QUEUES_PER_QS) - 1;
+	*tx_end = dev->data->nb_tx_queues ?
+		RTE_MIN(tmp, dev->data->nb_tx_queues - 1) : 0;
+}
+
+static inline void
+nicvf_rx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *rx_start,
+	       uint16_t *rx_end)
+{
+	uint16_t tmp;
+
+	*rx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+				    MAX_RCV_QUEUES_PER_QS);
+	tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+			     MAX_RCV_QUEUES_PER_QS) - 1;
+	*rx_end = dev->data->nb_rx_queues ?
+		RTE_MIN(tmp, dev->data->nb_rx_queues - 1) : 0;
+}
+
 #endif /* __THUNDERX_NICVF_ETHDEV_H__  */
-- 
1.9.1

[PATCH 09/13] net/thunderx: add secondary qset support in dev stop/close

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 142 +++++++++++++++++++++++++++---------
 1 file changed, 107 insertions(+), 35 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index b4ab505..c9c5b78 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -67,9 +67,13 @@
 
 #include "nicvf_ethdev.h"
 #include "nicvf_rxtx.h"
+#include "nicvf_svf.h"
 #include "nicvf_logs.h"
 
 static void nicvf_dev_stop(struct rte_eth_dev *dev);
+static void nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup);
+static void nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic,
+			  bool cleanup);
 
 static inline int
 nicvf_atomic_write_link_status(struct rte_eth_dev *dev,
@@ -674,23 +678,29 @@ config_sq_error:
 }
 
 static inline int
-nicvf_stop_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
 	struct nicvf_txq *txq;
 	int ret;
 
-	if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+	if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
-	ret = nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+	ret = nicvf_qset_sq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim sq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim sq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
-	txq = dev->data->tx_queues[qidx];
+	txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_tx_queue_release_mbufs(txq);
 	nicvf_tx_queue_reset(txq);
 
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return ret;
 }
 
@@ -1002,30 +1012,34 @@ config_rq_error:
 }
 
 static inline int
-nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
 	struct nicvf_rxq *rxq;
 	int ret, other_error;
 
-	if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
 	ret = nicvf_qset_rq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim rq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim rq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error = ret;
-	rxq = dev->data->rx_queues[qidx];
+	rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_rx_queue_release_mbufs(dev, rxq);
 	nicvf_rx_queue_reset(rxq);
 
 	ret = nicvf_qset_cq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim cq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim cq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error |= ret;
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return other_error;
 }
 
@@ -1057,8 +1071,14 @@ static int
 nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
 	int ret;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 
-	ret = nicvf_stop_rx_queue(dev, qidx);
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+	ret = nicvf_vf_stop_rx_queue(dev, nic, qidx);
 	ret |= nicvf_configure_cpi(dev);
 	ret |= nicvf_configure_rss_reta(dev);
 	return ret;
@@ -1073,7 +1093,14 @@ nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 static int
 nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
-	return nicvf_stop_tx_queue(dev, qidx);
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+	return nicvf_vf_stop_tx_queue(dev, nic, qidx);
 }
 
 static int
@@ -1442,10 +1469,10 @@ qset_rss_error:
 	nicvf_rss_term(nic);
 start_txq_error:
 	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+		nicvf_vf_stop_tx_queue(dev, nic, qidx);
 start_rxq_error:
 	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+		nicvf_vf_stop_rx_queue(dev, nic, qidx);
 qset_rbdr_reclaim:
 	nicvf_qset_rbdr_reclaim(nic, 0);
 	nicvf_rbdr_release_mbufs(dev, nic);
@@ -1460,32 +1487,74 @@ qset_reclaim:
 }
 
 static void
-nicvf_dev_stop(struct rte_eth_dev *dev)
+nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup)
 {
+	size_t i;
 	int ret;
-	uint16_t qidx;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Let PF make the BGX's RX and TX switches to OFF position */
-	nicvf_mbox_shutdown(nic);
+	/* Teardown secondary vf first */
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_vf_stop(dev, nic->snicvf[i], cleanup);
+	}
+
+	/* Stop the primary VF now */
+	nicvf_vf_stop(dev, nic, cleanup);
 
 	/* Disable loopback */
 	ret = nicvf_loopback_config(nic, 0);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable loopback %d", ret);
 
+	/* Reclaim CPI configuration */
+	ret = nicvf_mbox_config_cpi(nic, 0);
+	if (ret)
+		PMD_INIT_LOG(ERR, "Failed to reclaim CPI config %d", ret);
+}
+
+static void
+nicvf_dev_stop(struct rte_eth_dev *dev)
+{
+	PMD_INIT_FUNC_TRACE();
+
+	nicvf_dev_stop_cleanup(dev, false);
+}
+
+static void
+nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic, bool cleanup)
+{
+	int ret;
+	uint16_t qidx;
+	uint16_t tx_start, tx_end;
+	uint16_t rx_start, rx_end;
+
+	PMD_INIT_FUNC_TRACE();
+
+	if (cleanup) {
+		/* Let PF make the BGX's RX and TX switches to OFF position */
+		nicvf_mbox_shutdown(nic);
+	}
+
 	/* Disable VLAN Strip */
 	nicvf_vlan_hw_strip(nic, 0);
 
-	/* Reclaim sq */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+	/* Get queue ranges for this VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
+	for (qidx = tx_start; qidx <= tx_end; qidx++)
+		nicvf_vf_stop_tx_queue(dev, nic, qidx % MAX_SND_QUEUES_PER_QS);
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/* Reclaim rq */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+	for (qidx = rx_start; qidx <= rx_end; qidx++)
+		nicvf_vf_stop_rx_queue(dev, nic, qidx % MAX_RCV_QUEUES_PER_QS);
 
 	/* Reclaim RBDR */
 	ret = nicvf_qset_rbdr_reclaim(nic, 0);
@@ -1496,15 +1565,8 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 	if (nic->rbdr != NULL)
 		nicvf_rbdr_release_mbufs(dev, nic);
 
-	/* Reclaim CPI configuration */
-	if (!nic->sqs_mode) {
-		ret = nicvf_mbox_config_cpi(nic, 0);
-		if (ret)
-			PMD_INIT_LOG(ERR, "Failed to reclaim CPI config");
-	}
-
 	/* Disable qset */
-	ret = nicvf_qset_config(nic);
+	ret = nicvf_qset_reclaim(nic);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable qset %d", ret);
 
@@ -1521,10 +1583,20 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 static void
 nicvf_dev_close(struct rte_eth_dev *dev)
 {
+	size_t i;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
 	PMD_INIT_FUNC_TRACE();
 
-	nicvf_dev_stop(dev);
+	nicvf_dev_stop_cleanup(dev, true);
 	nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
+
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_periodic_alarm_stop(nicvf_vf_interrupt, nic->snicvf[i]);
+	}
 }
 
 static int
-- 
1.9.1

[PATCH 10/13] net/thunderx: add secondary qset support in device start

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 266 +++++++++++++++++++++++-------------
 1 file changed, 172 insertions(+), 94 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index c9c5b78..8510675 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -653,27 +653,33 @@ nicvf_tx_queue_reset(struct nicvf_txq *txq)
 }
 
 static inline int
-nicvf_start_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+			uint16_t qidx)
 {
 	struct nicvf_txq *txq;
 	int ret;
 
-	if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+	assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+	if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STARTED)
 		return 0;
 
-	txq = dev->data->tx_queues[qidx];
+	txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
 	txq->pool = NULL;
-	ret = nicvf_qset_sq_config(nicvf_pmd_priv(dev), qidx, txq);
+	ret = nicvf_qset_sq_config(nic, qidx, txq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure sq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure sq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_sq_error;
 	}
 
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STARTED;
 	return ret;
 
 config_sq_error:
-	nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+	nicvf_qset_sq_reclaim(nic, qidx);
 	return ret;
 }
 
@@ -977,31 +983,37 @@ nicvf_rx_queue_reset(struct nicvf_rxq *rxq)
 }
 
 static inline int
-nicvf_start_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+			uint16_t qidx)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
 	struct nicvf_rxq *rxq;
 	int ret;
 
-	if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+	assert(qidx < MAX_RCV_QUEUES_PER_QS);
+
+	if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STARTED)
 		return 0;
 
 	/* Update rbdr pointer to all rxq */
-	rxq = dev->data->rx_queues[qidx];
+	rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
 	rxq->shared_rbdr = nic->rbdr;
 
 	ret = nicvf_qset_rq_config(nic, qidx, rxq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure rq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure rq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_rq_error;
 	}
 	ret = nicvf_qset_cq_config(nic, qidx, rxq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure cq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure cq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_cq_error;
 	}
 
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STARTED;
 	return 0;
 
 config_cq_error:
@@ -1054,9 +1066,15 @@ nicvf_dev_rx_queue_release(void *rx_queue)
 static int
 nicvf_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 {
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 	int ret;
 
-	ret = nicvf_start_rx_queue(dev, qidx);
+	if (qidx >= MAX_RCV_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_RCV_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+	ret = nicvf_vf_start_rx_queue(dev, nic, qidx);
 	if (ret)
 		return ret;
 
@@ -1087,7 +1105,14 @@ nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 static int
 nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 {
-	return nicvf_start_tx_queue(dev, qidx);
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+	return nicvf_vf_start_tx_queue(dev, nic, qidx);
 }
 
 static int
@@ -1267,25 +1292,25 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
 }
 
 static int
-nicvf_dev_start(struct rte_eth_dev *dev)
+nicvf_vf_start(struct rte_eth_dev *dev, struct nicvf *nic, uint32_t rbdrsz)
 {
 	int ret;
 	uint16_t qidx;
-	uint32_t buffsz = 0, rbdrsz = 0;
 	uint32_t total_rxq_desc, nb_rbdr_desc, exp_buffs;
 	uint64_t mbuf_phys_off = 0;
 	struct nicvf_rxq *rxq;
-	struct rte_pktmbuf_pool_private *mbp_priv;
 	struct rte_mbuf *mbuf;
-	struct nicvf *nic = nicvf_pmd_priv(dev);
-	struct rte_eth_rxmode *rx_conf = &dev->data->dev_conf.rxmode;
-	uint16_t mtu;
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
 
 	PMD_INIT_FUNC_TRACE();
 
 	/* Userspace process exited without proper shutdown in last run */
 	if (nicvf_qset_rbdr_active(nic, 0))
-		nicvf_dev_stop(dev);
+		nicvf_vf_stop(dev, nic, false);
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/*
 	 * Thunderx nicvf PMD can support more than one pool per port only when
@@ -1300,32 +1325,15 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	 *
 	 */
 
-	/* Validate RBDR buff size */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
-		rxq = dev->data->rx_queues[qidx];
-		mbp_priv = rte_mempool_get_priv(rxq->pool);
-		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
-		if (buffsz % 128) {
-			PMD_INIT_LOG(ERR, "rxbuf size must be multiply of 128");
-			return -EINVAL;
-		}
-		if (rbdrsz == 0)
-			rbdrsz = buffsz;
-		if (rbdrsz != buffsz) {
-			PMD_INIT_LOG(ERR, "buffsz not same, qid=%d (%d/%d)",
-				     qidx, rbdrsz, buffsz);
-			return -EINVAL;
-		}
-	}
-
 	/* Validate mempool attributes */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		rxq->mbuf_phys_off = nicvf_mempool_phy_offset(rxq->pool);
 		mbuf = rte_pktmbuf_alloc(rxq->pool);
 		if (mbuf == NULL) {
-			PMD_INIT_LOG(ERR, "Failed allocate mbuf qid=%d pool=%s",
-				     qidx, rxq->pool->name);
+			PMD_INIT_LOG(ERR, "Failed allocate mbuf VF%d qid=%d "
+				     "pool=%s",
+				     nic->vf_id, qidx, rxq->pool->name);
 			return -ENOMEM;
 		}
 		rxq->mbuf_phys_off -= nicvf_mbuff_meta_length(mbuf);
@@ -1335,15 +1343,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		if (mbuf_phys_off == 0)
 			mbuf_phys_off = rxq->mbuf_phys_off;
 		if (mbuf_phys_off != rxq->mbuf_phys_off) {
-			PMD_INIT_LOG(ERR, "pool params not same,%s %" PRIx64,
-				     rxq->pool->name, mbuf_phys_off);
+			PMD_INIT_LOG(ERR, "pool params not same,%s VF%d %"
+				     PRIx64, rxq->pool->name, nic->vf_id,
+				     mbuf_phys_off);
 			return -EINVAL;
 		}
 	}
 
 	/* Check the level of buffers in the pool */
 	total_rxq_desc = 0;
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		/* Count total numbers of rxq descs */
 		total_rxq_desc += rxq->qlen_mask + 1;
@@ -1361,14 +1370,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	/* Check RBDR desc overflow */
 	ret = nicvf_qsize_rbdr_roundup(total_rxq_desc);
 	if (ret == 0) {
-		PMD_INIT_LOG(ERR, "Reached RBDR desc limit, reduce nr desc");
+		PMD_INIT_LOG(ERR, "Reached RBDR desc limit, reduce nr desc "
+			     "VF%d", nic->vf_id);
 		return -ENOMEM;
 	}
 
 	/* Enable qset */
 	ret = nicvf_qset_config(nic);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to enable qset %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to enable qset %d VF%d", ret,
+			     nic->vf_id);
 		return ret;
 	}
 
@@ -1376,14 +1387,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	nb_rbdr_desc = nicvf_qsize_rbdr_roundup(total_rxq_desc);
 	ret = nicvf_qset_rbdr_alloc(dev, nic, nb_rbdr_desc, rbdrsz);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc");
+		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc "
+			     "VF%d", nic->vf_id);
 		goto qset_reclaim;
 	}
 
 	/* Enable and configure RBDR registers */
 	ret = nicvf_qset_rbdr_config(nic, 0);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure rbdr %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to configure rbdr %d VF%d", ret,
+			     nic->vf_id);
 		goto qset_rbdr_free;
 	}
 
@@ -1391,52 +1404,127 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
 					total_rxq_desc);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d VF%d", ret,
+			     nic->vf_id);
 		goto qset_rbdr_reclaim;
 	}
 
-	PMD_DRV_LOG(INFO, "Filled %d out of %d entries in RBDR",
-		     nic->rbdr->tail, nb_rbdr_desc);
+	PMD_DRV_LOG(INFO, "Filled %d out of %d entries in RBDR VF%d",
+		     nic->rbdr->tail, nb_rbdr_desc, nic->vf_id);
+
+	/* Configure VLAN Strip */
+	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
+
+	/* Get queue ranges for this VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
+	/* Configure TX queues */
+	for (qidx = tx_start; qidx <= tx_end; qidx++) {
+		ret = nicvf_vf_start_tx_queue(dev, nic,
+			qidx % MAX_SND_QUEUES_PER_QS);
+		if (ret)
+			goto start_txq_error;
+	}
 
 	/* Configure RX queues */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
-		ret = nicvf_start_rx_queue(dev, qidx);
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
+		ret = nicvf_vf_start_rx_queue(dev, nic,
+			qidx % MAX_RCV_QUEUES_PER_QS);
 		if (ret)
 			goto start_rxq_error;
 	}
 
-	/* Configure VLAN Strip */
-	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
-
-	/* Configure TX queues */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
-		ret = nicvf_start_tx_queue(dev, qidx);
+	if (!nic->sqs_mode) {
+		/* Configure CPI algorithm */
+		ret = nicvf_configure_cpi(dev);
 		if (ret)
 			goto start_txq_error;
+
+		ret = nicvf_mbox_get_rss_size(nic);
+		if (ret) {
+			PMD_INIT_LOG(ERR, "Failed to get rss table size");
+			goto qset_rss_error;
+		}
+
+		/* Configure RSS */
+		ret = nicvf_configure_rss(dev);
+		if (ret)
+			goto qset_rss_error;
 	}
 
-	/* Configure CPI algorithm */
-	ret = nicvf_configure_cpi(dev);
-	if (ret)
-		goto start_txq_error;
+	/* Done; Let PF make the BGX's RX and TX switches to ON position */
+	nicvf_mbox_cfg_done(nic);
+	return 0;
 
-	/* Configure RSS */
-	ret = nicvf_configure_rss(dev);
-	if (ret)
-		goto qset_rss_error;
+qset_rss_error:
+	nicvf_rss_term(nic);
+start_rxq_error:
+	for (qidx = rx_start; qidx <= rx_end; qidx++)
+		nicvf_vf_stop_rx_queue(dev, nic, qidx % MAX_RCV_QUEUES_PER_QS);
+start_txq_error:
+	for (qidx = tx_start; qidx <= tx_end; qidx++)
+		nicvf_vf_stop_tx_queue(dev, nic, qidx % MAX_SND_QUEUES_PER_QS);
+qset_rbdr_reclaim:
+	nicvf_qset_rbdr_reclaim(nic, 0);
+	nicvf_rbdr_release_mbufs(dev, nic);
+qset_rbdr_free:
+	if (nic->rbdr) {
+		rte_free(nic->rbdr);
+		nic->rbdr = NULL;
+	}
+qset_reclaim:
+	nicvf_qset_reclaim(nic);
+	return ret;
+}
+
+static int
+nicvf_dev_start(struct rte_eth_dev *dev)
+{
+	uint16_t qidx;
+	int ret;
+	size_t i;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+	struct rte_eth_rxmode *rx_conf = &dev->data->dev_conf.rxmode;
+	uint16_t mtu;
+	uint32_t buffsz = 0, rbdrsz = 0;
+	struct rte_pktmbuf_pool_private *mbp_priv;
+	struct nicvf_rxq *rxq;
+
+	PMD_INIT_FUNC_TRACE();
+
+	/* This function must be called for a primary device */
+	assert_if_secondary(nic);
+
+	/* Validate RBDR buff size */
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+		rxq = dev->data->rx_queues[qidx];
+		mbp_priv = rte_mempool_get_priv(rxq->pool);
+		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
+		if (buffsz % 128) {
+			PMD_INIT_LOG(ERR, "rxbuf size must be multiply of 128");
+			return -EINVAL;
+		}
+		if (rbdrsz == 0)
+			rbdrsz = buffsz;
+		if (rbdrsz != buffsz) {
+			PMD_INIT_LOG(ERR, "buffsz not same, qidx=%d (%d/%d)",
+				     qidx, rbdrsz, buffsz);
+			return -EINVAL;
+		}
+	}
 
 	/* Configure loopback */
 	ret = nicvf_loopback_config(nic, dev->data->dev_conf.lpbk_mode);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to configure loopback %d", ret);
-		goto qset_rss_error;
+		return ret;
 	}
 
 	/* Reset all statistics counters attached to this port */
 	ret = nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, 0xFFFF, 0xFFFF);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to reset stat counters %d", ret);
-		goto qset_rss_error;
+		return ret;
 	}
 
 	/* Setup scatter mode if needed by jumbo */
@@ -1457,33 +1545,23 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		return -EBUSY;
 	}
 
+	ret = nicvf_vf_start(dev, nic, rbdrsz);
+	if (ret != 0)
+		return ret;
+
+	for (i = 0; i < nic->sqs_count; i++) {
+		assert(nic->snicvf[i]);
+
+		ret = nicvf_vf_start(dev, nic->snicvf[i], rbdrsz);
+		if (ret != 0)
+			return ret;
+	}
+
 	/* Configure callbacks based on scatter mode */
 	nicvf_set_tx_function(dev);
 	nicvf_set_rx_function(dev);
 
-	/* Done; Let PF make the BGX's RX and TX switches to ON position */
-	nicvf_mbox_cfg_done(nic);
 	return 0;
-
-qset_rss_error:
-	nicvf_rss_term(nic);
-start_txq_error:
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_vf_stop_tx_queue(dev, nic, qidx);
-start_rxq_error:
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_vf_stop_rx_queue(dev, nic, qidx);
-qset_rbdr_reclaim:
-	nicvf_qset_rbdr_reclaim(nic, 0);
-	nicvf_rbdr_release_mbufs(dev, nic);
-qset_rbdr_free:
-	if (nic->rbdr) {
-		rte_free(nic->rbdr);
-		nic->rbdr = NULL;
-	}
-qset_reclaim:
-	nicvf_qset_reclaim(nic);
-	return ret;
 }
 
 static void
-- 
1.9.1

[PATCH 11/13] net/thunderx: add secondary qset support in device configure

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 65 +++++++++++++++++++++++++++++++++----
 1 file changed, 59 insertions(+), 6 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 8510675..fcdbebf 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -163,6 +163,7 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct nicvf *nic = nicvf_pmd_priv(dev);
 	uint32_t buffsz, frame_size = mtu + ETHER_HDR_LEN + ETHER_CRC_LEN;
+	size_t i;
 
 	PMD_INIT_FUNC_TRACE();
 
@@ -198,6 +199,10 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
 	/* Update max frame size */
 	dev->data->dev_conf.rxmode.max_rx_pkt_len = (uint32_t)frame_size;
 	nic->mtu = mtu;
+
+	for (i = 0; i < nic->sqs_count; i++)
+		nic->snicvf[i]->mtu = mtu;
+
 	return 0;
 }
 
@@ -507,7 +512,8 @@ nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	const struct rte_memzone *rz;
 	uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "cq_ring",
+				      nicvf_netdev_qidx(nic, qidx), ring_size,
 				      NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
@@ -530,8 +536,9 @@ nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	const struct rte_memzone *rz;
 	uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
-				NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "sq",
+				      nicvf_netdev_qidx(nic, qidx), ring_size,
+				      NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
 		return -ENOMEM;
@@ -563,8 +570,9 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	}
 
 	ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
-	rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
-				   NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "rbdr",
+				      nicvf_netdev_qidx(nic, 0), ring_size,
+				      NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
 		return -ENOMEM;
@@ -1678,12 +1686,37 @@ nicvf_dev_close(struct rte_eth_dev *dev)
 }
 
 static int
+nicvf_request_sqs(struct nicvf *nic)
+{
+	size_t i;
+
+	assert_if_secondary(nic);
+	assert(nic->sqs_count > 0);
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	/* Set no of Rx/Tx queues in each of the SQsets */
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (nicvf_svf_empty())
+			rte_panic("Cannot assign sufficient number of "
+				  "secondary queues to primary VF%" PRIu8 "\n",
+				  nic->vf_id);
+
+		nic->snicvf[i] = nicvf_svf_pop();
+		nic->snicvf[i]->sqs_id = i;
+	}
+
+	return nicvf_mbox_request_sqs(nic);
+}
+
+static int
 nicvf_dev_configure(struct rte_eth_dev *dev)
 {
-	struct rte_eth_conf *conf = &dev->data->dev_conf;
+	struct rte_eth_dev_data *data = dev->data;
+	struct rte_eth_conf *conf = &data->dev_conf;
 	struct rte_eth_rxmode *rxmode = &conf->rxmode;
 	struct rte_eth_txmode *txmode = &conf->txmode;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint8_t cqcount;
 
 	PMD_INIT_FUNC_TRACE();
 
@@ -1748,6 +1781,26 @@ nicvf_dev_configure(struct rte_eth_dev *dev)
 		return -EINVAL;
 	}
 
+	assert_if_secondary(nic);
+	NICVF_STATIC_ASSERT(MAX_RCV_QUEUES_PER_QS == MAX_SND_QUEUES_PER_QS);
+	cqcount = RTE_MAX(data->nb_tx_queues, data->nb_rx_queues);
+	if (cqcount > MAX_RCV_QUEUES_PER_QS) {
+		nic->sqs_count = RTE_ALIGN_CEIL(cqcount, MAX_RCV_QUEUES_PER_QS);
+		nic->sqs_count = (nic->sqs_count / MAX_RCV_QUEUES_PER_QS) - 1;
+	} else {
+		nic->sqs_count = 0;
+	}
+
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	if (nic->sqs_count > 0) {
+		if (nicvf_request_sqs(nic)) {
+			rte_panic("Cannot assign sufficient number of "
+				  "secondary queues to PORT%d VF%" PRIu8 "\n",
+				  dev->data->port_id, nic->vf_id);
+		}
+	}
+
 	PMD_INIT_LOG(DEBUG, "Configured ethdev port%d hwcap=0x%" PRIx64,
 		dev->data->port_id, nicvf_hw_cap(nic));
 
-- 
1.9.1

[PATCH 12/13] net/thunderx: add final bits for secondary queue support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 178 +++++++++++++++++++++++++++++-------
 1 file changed, 144 insertions(+), 34 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index fcdbebf..04bfde0 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -118,7 +118,7 @@ nicvf_interrupt(void *arg)
 				nicvf_interrupt, dev);
 }
 
-static void __rte_unused
+static void
 nicvf_vf_interrupt(void *arg)
 {
 	struct nicvf *nic = arg;
@@ -236,9 +236,15 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	struct nicvf_hw_tx_qstats tx_qstats;
 	struct nicvf_hw_stats port_stats;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
+	size_t i;
+
+	/* RX queue indices for the first VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/* Reading per RX ring stats */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
 			break;
 
@@ -247,8 +253,11 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 		stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
 	}
 
+	/* TX queue indices for the first VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
 	/* Reading per TX ring stats */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
+	for (qidx = tx_start; qidx <= tx_end; qidx++) {
 		if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
 			break;
 
@@ -257,6 +266,40 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 		stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
 	}
 
+	for (i = 0; i < nic->sqs_count; i++) {
+		struct nicvf *snic = nic->snicvf[i];
+
+		if (snic == NULL)
+			break;
+
+		/* RX queue indices for a secondary VF */
+		nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+
+		/* Reading per RX ring stats */
+		for (qidx = rx_start; qidx <= rx_end; qidx++) {
+			if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+				break;
+
+			nicvf_hw_get_rx_qstats(snic, &rx_qstats,
+					       qidx % MAX_RCV_QUEUES_PER_QS);
+			stats->q_ibytes[qidx] = rx_qstats.q_rx_bytes;
+			stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
+		}
+
+		/* TX queue indices for a secondary VF */
+		nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+		/* Reading per TX ring stats */
+		for (qidx = tx_start; qidx <= tx_end; qidx++) {
+			if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+				break;
+
+			nicvf_hw_get_tx_qstats(snic, &tx_qstats,
+					       qidx % MAX_SND_QUEUES_PER_QS);
+			stats->q_obytes[qidx] = tx_qstats.q_tx_bytes;
+			stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
+		}
+	}
+
 	nicvf_hw_get_stats(nic, &port_stats);
 	stats->ibytes = port_stats.rx_bytes;
 	stats->ipackets = port_stats.rx_ucast_frames;
@@ -322,13 +365,36 @@ nicvf_dev_stats_reset(struct rte_eth_dev *dev)
 	int i;
 	uint16_t rxqs = 0, txqs = 0;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
 
-	for (i = 0; i < dev->data->nb_rx_queues; i++)
+	/* Reset all primary nic counters */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+	for (i = rx_start; i <= rx_end; i++)
 		rxqs |= (0x3 << (i * 2));
-	for (i = 0; i < dev->data->nb_tx_queues; i++)
+
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+	for (i = tx_start; i <= tx_end; i++)
 		txqs |= (0x3 << (i * 2));
 
 	nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, rxqs, txqs);
+
+	/* Reset secondary nic queue counters */
+	for (i = 0; i < nic->sqs_count; i++) {
+		struct nicvf *snic = nic->snicvf[i];
+		if (snic == NULL)
+			break;
+
+		nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+		for (i = rx_start; i <= rx_end; i++)
+			rxqs |= (0x3 << ((i % MAX_CMP_QUEUES_PER_QS) * 2));
+
+		nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+		for (i = tx_start; i <= tx_end; i++)
+			txqs |= (0x3 << ((i % MAX_SND_QUEUES_PER_QS) * 2));
+
+		nicvf_mbox_reset_stat_counters(snic, 0, 0, rxqs, txqs);
+	}
 }
 
 /* Promiscuous mode enabled by default in LMAC to VF 1:1 map configuration */
@@ -596,14 +662,18 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 }
 
 static void
-nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic __rte_unused,
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic,
 			nicvf_phys_addr_t phy)
 {
 	uint16_t qidx;
 	void *obj;
 	struct nicvf_rxq *rxq;
+	uint16_t rx_start, rx_end;
 
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		if (rxq->precharge_cnt) {
 			obj = (void *)nicvf_mbuff_phy2virt(phy,
@@ -861,6 +931,11 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[qidx / MAX_SND_QUEUES_PER_QS - 1];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
 	/* Socket id check */
 	if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node)
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
@@ -895,18 +970,20 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	}
 
 	/* Free memory prior to re-allocation if needed. */
-	if (dev->data->tx_queues[qidx] != NULL) {
+	if (dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) {
 		PMD_TX_LOG(DEBUG, "Freeing memory prior to re-allocation %d",
-				qidx);
-		nicvf_dev_tx_queue_release(dev->data->tx_queues[qidx]);
-		dev->data->tx_queues[qidx] = NULL;
+				nicvf_netdev_qidx(nic, qidx));
+		nicvf_dev_tx_queue_release(
+			dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)]);
+		dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL;
 	}
 
 	/* Allocating tx queue data structure */
 	txq = rte_zmalloc_socket("ethdev TX queue", sizeof(struct nicvf_txq),
 					RTE_CACHE_LINE_SIZE, nic->node);
 	if (txq == NULL) {
-		PMD_INIT_LOG(ERR, "Failed to allocate txq=%d", qidx);
+		PMD_INIT_LOG(ERR, "Failed to allocate txq=%d",
+			     nicvf_netdev_qidx(nic, qidx));
 		return -ENOMEM;
 	}
 
@@ -949,10 +1026,12 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	nicvf_tx_queue_reset(txq);
 
 	PMD_TX_LOG(DEBUG, "[%d] txq=%p nb_desc=%d desc=%p phys=0x%" PRIx64,
-			qidx, txq, nb_desc, txq->desc, txq->phys);
+			nicvf_netdev_qidx(nic, qidx), txq, nb_desc, txq->desc,
+			txq->phys);
 
-	dev->data->tx_queues[qidx] = txq;
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = txq;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return 0;
 }
 
@@ -967,7 +1046,8 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 	if (dev->rx_pkt_burst == NULL)
 		return;
 
-	while ((rxq_cnt = nicvf_dev_rx_queue_count(dev, rxq->queue_id))) {
+	while ((rxq_cnt = nicvf_dev_rx_queue_count(dev,
+				nicvf_netdev_qidx(rxq->nic, rxq->queue_id)))) {
 		nb_pkts = dev->rx_pkt_burst(rxq, rx_pkts,
 					NICVF_MAX_RX_FREE_THRESH);
 		PMD_DRV_LOG(INFO, "nb_pkts=%d  rxq_cnt=%d", nb_pkts, rxq_cnt);
@@ -977,7 +1057,10 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 		}
 	}
 
-	refill_cnt += nicvf_dev_rbdr_refill(dev, rxq->queue_id);
+
+	refill_cnt += nicvf_dev_rbdr_refill(dev,
+			nicvf_netdev_qidx(rxq->nic, rxq->queue_id));
+
 	PMD_DRV_LOG(INFO, "free_cnt=%d  refill_cnt=%d",
 		    released_pkts, refill_cnt);
 }
@@ -1148,6 +1231,11 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (qidx >= MAX_RCV_QUEUES_PER_QS)
+		nic = nic->snicvf[qidx / MAX_RCV_QUEUES_PER_QS - 1];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
 	/* Socket id check */
 	if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node)
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
@@ -1184,18 +1272,20 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	}
 
 	/* Free memory prior to re-allocation if needed */
-	if (dev->data->rx_queues[qidx] != NULL) {
+	if (dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) {
 		PMD_RX_LOG(DEBUG, "Freeing memory prior to re-allocation %d",
-				qidx);
-		nicvf_dev_rx_queue_release(dev->data->rx_queues[qidx]);
-		dev->data->rx_queues[qidx] = NULL;
+				nicvf_netdev_qidx(nic, qidx));
+		nicvf_dev_rx_queue_release(
+			dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)]);
+		dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL;
 	}
 
 	/* Allocate rxq memory */
 	rxq = rte_zmalloc_socket("ethdev rx queue", sizeof(struct nicvf_rxq),
 					RTE_CACHE_LINE_SIZE, nic->node);
 	if (rxq == NULL) {
-		PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d", qidx);
+		PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d",
+			     nicvf_netdev_qidx(nic, qidx));
 		return -ENOMEM;
 	}
 
@@ -1225,11 +1315,12 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	nicvf_rx_queue_reset(rxq);
 
 	PMD_RX_LOG(DEBUG, "[%d] rxq=%p pool=%s nb_desc=(%d/%d) phy=%" PRIx64,
-			qidx, rxq, mp->name, nb_desc,
+			nicvf_netdev_qidx(nic, qidx), rxq, mp->name, nb_desc,
 			rte_mempool_avail_count(mp), rxq->phys);
 
-	dev->data->rx_queues[qidx] = rxq;
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = rxq;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return 0;
 }
 
@@ -1242,8 +1333,10 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 
 	dev_info->min_rx_bufsize = ETHER_MIN_MTU;
 	dev_info->max_rx_pktlen = NIC_HW_MAX_FRS;
-	dev_info->max_rx_queues = (uint16_t)MAX_RCV_QUEUES_PER_QS;
-	dev_info->max_tx_queues = (uint16_t)MAX_SND_QUEUES_PER_QS;
+	dev_info->max_rx_queues =
+			(uint16_t)MAX_RCV_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1);
+	dev_info->max_tx_queues =
+			(uint16_t)MAX_SND_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_vfs = dev->pci_dev->max_vfs;
 
@@ -1284,9 +1377,13 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
 	uintptr_t mbuf;
 	struct nicvf_rxq *rxq;
 	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
-	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
+	struct nicvf *nic = (struct nicvf *)opaque;
+	uint16_t rx_start, rx_end;
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(eth_dev, nic, &rx_start, &rx_end);
 
-	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = eth_dev->data->rx_queues[qidx];
 		/* Maintain equal buffer count across all pools */
 		if (rxq->precharge_cnt >= rxq->qlen_mask)
@@ -1852,7 +1949,9 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 		/* Setup callbacks for secondary process */
 		nicvf_set_tx_function(eth_dev);
 		nicvf_set_rx_function(eth_dev);
-		return 0;
+		/* If nic == NULL than it is secondary function
+		 * so ethdev need to be released by caller */
+		return nic != NULL ? 0 : ENOTSUP;
 	}
 
 	pci_dev = eth_dev->pci_dev;
@@ -1898,10 +1997,21 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 	}
 
 	if (nic->sqs_mode) {
-		PMD_INIT_LOG(INFO, "Unsupported SQS VF detected, Detaching...");
-		/* Detach port by returning Positive error number */
-		ret = ENOTSUP;
-		goto alarm_fail;
+		/* Push nic to stack of secondary vfs */
+		nicvf_svf_push(nic);
+
+		/* Steal nic pointer from the device for further reuse */
+		eth_dev->data->dev_private = NULL;
+
+		nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
+		ret = nicvf_periodic_alarm_start(nicvf_vf_interrupt, nic);
+		if (ret) {
+			PMD_INIT_LOG(ERR, "Failed to start period alarm");
+			goto fail;
+		}
+
+		/* Detach port by returning postive error number */
+		return ENOTSUP;
 	}
 
 	eth_dev->data->mac_addrs = rte_zmalloc("mac_addr", ETHER_ADDR_LEN, 0);
-- 
1.9.1

[PATCH 13/13] net/thunderx: document secondary queue set support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 doc/guides/nics/thunderx.rst | 114 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 107 insertions(+), 7 deletions(-)

diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst
index acc05a4..2b6f1f0 100644
--- a/doc/guides/nics/thunderx.rst
+++ b/doc/guides/nics/thunderx.rst
@@ -56,6 +56,7 @@ Features of the ThunderX PMD are:
 - VLAN stripping
 - SR-IOV VF
 - NUMA support
+- Multi queue set support (up to 96 queues (12 queue sets)) per port
 
 Supported ThunderX SoCs
 -----------------------
@@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS.
 #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
    how to launch ``testpmd`` application.
 
+Multiple Queue Set per DPDK port configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two types of VFs:
+
+- Primary VF
+- Secondary VF
+
+Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
+In case port is configured to use more than 8 queues, then it requires one (or more)
+secondary VF. Each secondary VF adds additional 8 queues to the queue set.
+
+During PMD driver initialization, the primary VF's are enumerated by checking the
+specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
+They are at the beginning of VF list (the remain ones are secondary VF's).
+
+The primary VFs are used as master queue sets. Secondary VFs provid
+additional queue sets for primary ones. If a port is configured for more then
+8 queues than it will request for additional queues from secondary VFs.
+
+Secondary VFs cannot be shared between primary VFs.
+
+Primary VFs are present on the beginning of the 'Network devices using kernel
+driver' list, secondary VFs are on the remaining on the remaining part of the list.
+
+   .. note::
+
+      The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
+      We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility.
+
+   .. note::
+
+      Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
+      this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
+
+
+Example device binding
+~~~~~~~~~~~~~~~~~~~~~~
+
+If a system has three interfaces, a total of 18 VF devices will be created
+on a non-NUMA machine.
+
+   .. note::
+
+      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
+
+   .. code-block:: console
+
+      # tools/dpdk-devbind.py --status
+      
+      Network devices using DPDK-compatible driver
+      ============================================
+      <none>
+      
+      Network devices using kernel driver
+      ===================================
+      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
+      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      
+      Other network devices
+      =====================
+      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
+
+
+We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
+and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
+We will chose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
+
+
+#. Bind two primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
+
+#. Bind four primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
+
+The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
+
 Limitations
 -----------
 
@@ -346,10 +453,3 @@ Maximum packet segments
 The ThunderX SoC family NICs support up to 12 segments per packet when working
 in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
 frame size does not fit in the maximum number of segments.
-
-Limited VFs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues
-for RX/TX respectively. Current driver implementation has one to one mapping
-between physical port and VF hence only limited VFs can be used.
-- 
1.9.1

Re: [PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver

[Kamil Rytarowski]

ping


W dniu 26.08.2016 o 18:53, Kamil Rytarowski pisze:
> This series of patches adds support for secondary queue set in nicvf thunderx
> driver
>
> There are two types of VFs:
>   - Primary VF
>   - Secondary VF
>
> Each port consist of a primary VF and n secondary VF(s). Each VF provides 8
> Tx/Rx queues to a port. In case port is configured to use more than 8 queues,
> then it requires one (or more) secondary VF. Each secondary VF adds additional
> 8 queues to the queue set.
>
> During PMD driver initialization, the primary VF's are enumerated by checking the
> specific flag (see READY message). They are at the beginning of  VF list (the remain
> ones are secondary VF's).
>
> The primary VFs are used as master queue sets. Secondary VFs provide
> additional queue sets for primary ones. If a port is configured for more then
> 8 queues then it will request for additional queues from secondary VFs.
>
> Secondary VFs cannot be shared between primary VFs.
>
> Primary VFs are present on the tail of the 'Network devices using kernel
> driver' list, secondary VFs are on the remaining tail of the list.
>
> The VNIC driver in the multiqueue setup works differently than other drivers
> like `ixgbe`. We need to bind separately each specific queue set device with
> the ``tools/dpdk-devbind.py`` utility.
>
> Depending on the hardware used, the kernel driver sets a threshold ``vf_id``.
> VFs that try to attach with an id below or equal to this boundary are
> considered primary VFs. VFs that try to attach with an id above this boundary
> are considered secondary VFs.
>
> This patchset also contains other cleanups and improvements like fixing
> erroneous checksum calculation and preparing the thunderx driver for the multi
> queue set feature support.
>
>
> These changes base on the following pending patches:
>
> [dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver
> http://dpdk.org/dev/patchwork/patch/14963/
>
> [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag
> http://dpdk.org/dev/patchwork/patch/14964/
>
> [dpdk-dev,3/3] net/thunderx: add 81xx SoC support
> http://dpdk.org/dev/patchwork/patch/14965/
>
> Kamil Rytarowski (13):
>    net/thunderx: cleanup the driver before adding new features
>    net/thunderx: correct transmit checksum handling
>    net/thunderx/base: add family of functions to store qsets
>    net/thunderx/base: add secondary queue set support
>    net/thunderx: add family of functions to store DPDK qsets
>    net/thunderx: add secondary queue set in interrupt functions
>    net/thunderx: fix multiprocess support in stats
>    net/thunderx: add helper utils for secondary qset support
>    net/thunderx: add secondary qset support in dev stop/close
>    net/thunderx: add secondary qset support in device start
>    net/thunderx: add secondary qset support in device configure
>    net/thunderx: add final bits for secondary queue support
>    net/thunderx: document secondary queue set support
>
>   doc/guides/nics/thunderx.rst              | 114 ++++-
>   drivers/net/thunderx/Makefile             |   2 +
>   drivers/net/thunderx/base/nicvf_bsvf.c    |  72 +++
>   drivers/net/thunderx/base/nicvf_bsvf.h    |  76 +++
>   drivers/net/thunderx/base/nicvf_hw.c      |  10 +-
>   drivers/net/thunderx/base/nicvf_hw.h      |   6 +-
>   drivers/net/thunderx/base/nicvf_hw_defs.h |   1 +
>   drivers/net/thunderx/base/nicvf_mbox.c    |  34 +-
>   drivers/net/thunderx/base/nicvf_mbox.h    |  21 +-
>   drivers/net/thunderx/nicvf_ethdev.c       | 753 +++++++++++++++++++++---------
>   drivers/net/thunderx/nicvf_ethdev.h       |  39 ++
>   drivers/net/thunderx/nicvf_rxtx.c         |  14 +-
>   drivers/net/thunderx/nicvf_struct.h       |   6 +-
>   drivers/net/thunderx/nicvf_svf.c          |  78 ++++
>   drivers/net/thunderx/nicvf_svf.h          |  66 +++
>   15 files changed, 1046 insertions(+), 246 deletions(-)
>   create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
>   create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
>   create mode 100644 drivers/net/thunderx/nicvf_svf.c
>   create mode 100644 drivers/net/thunderx/nicvf_svf.h
>

Re: [PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver

[Kamil Rytarowski]

ping


W dniu 12.09.2016 o 12:59, Kamil Rytarowski pisze:
> ping
>
>
> W dniu 26.08.2016 o 18:53, Kamil Rytarowski pisze:
>> This series of patches adds support for secondary queue set in nicvf 
>> thunderx
>> driver
>>
>> There are two types of VFs:
>>   - Primary VF
>>   - Secondary VF
>>
>> Each port consist of a primary VF and n secondary VF(s). Each VF 
>> provides 8
>> Tx/Rx queues to a port. In case port is configured to use more than 8 
>> queues,
>> then it requires one (or more) secondary VF. Each secondary VF adds 
>> additional
>> 8 queues to the queue set.
>>
>> During PMD driver initialization, the primary VF's are enumerated by 
>> checking the
>> specific flag (see READY message). They are at the beginning of VF 
>> list (the remain
>> ones are secondary VF's).
>>
>> The primary VFs are used as master queue sets. Secondary VFs provide
>> additional queue sets for primary ones. If a port is configured for 
>> more then
>> 8 queues then it will request for additional queues from secondary VFs.
>>
>> Secondary VFs cannot be shared between primary VFs.
>>
>> Primary VFs are present on the tail of the 'Network devices using kernel
>> driver' list, secondary VFs are on the remaining tail of the list.
>>
>> The VNIC driver in the multiqueue setup works differently than other 
>> drivers
>> like `ixgbe`. We need to bind separately each specific queue set 
>> device with
>> the ``tools/dpdk-devbind.py`` utility.
>>
>> Depending on the hardware used, the kernel driver sets a threshold 
>> ``vf_id``.
>> VFs that try to attach with an id below or equal to this boundary are
>> considered primary VFs. VFs that try to attach with an id above this 
>> boundary
>> are considered secondary VFs.
>>
>> This patchset also contains other cleanups and improvements like fixing
>> erroneous checksum calculation and preparing the thunderx driver for 
>> the multi
>> queue set feature support.
>>
>>
>> These changes base on the following pending patches:
>>
>> [dpdk-dev,1/3] net/thunderx: remove generic passx references from the 
>> driver
>> http://dpdk.org/dev/patchwork/patch/14963/
>>
>> [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag
>> http://dpdk.org/dev/patchwork/patch/14964/
>>
>> [dpdk-dev,3/3] net/thunderx: add 81xx SoC support
>> http://dpdk.org/dev/patchwork/patch/14965/
>>
>> Kamil Rytarowski (13):
>>    net/thunderx: cleanup the driver before adding new features
>>    net/thunderx: correct transmit checksum handling
>>    net/thunderx/base: add family of functions to store qsets
>>    net/thunderx/base: add secondary queue set support
>>    net/thunderx: add family of functions to store DPDK qsets
>>    net/thunderx: add secondary queue set in interrupt functions
>>    net/thunderx: fix multiprocess support in stats
>>    net/thunderx: add helper utils for secondary qset support
>>    net/thunderx: add secondary qset support in dev stop/close
>>    net/thunderx: add secondary qset support in device start
>>    net/thunderx: add secondary qset support in device configure
>>    net/thunderx: add final bits for secondary queue support
>>    net/thunderx: document secondary queue set support
>>
>>   doc/guides/nics/thunderx.rst              | 114 ++++-
>>   drivers/net/thunderx/Makefile             |   2 +
>>   drivers/net/thunderx/base/nicvf_bsvf.c    |  72 +++
>>   drivers/net/thunderx/base/nicvf_bsvf.h    |  76 +++
>>   drivers/net/thunderx/base/nicvf_hw.c      |  10 +-
>>   drivers/net/thunderx/base/nicvf_hw.h      |   6 +-
>>   drivers/net/thunderx/base/nicvf_hw_defs.h |   1 +
>>   drivers/net/thunderx/base/nicvf_mbox.c    |  34 +-
>>   drivers/net/thunderx/base/nicvf_mbox.h    |  21 +-
>>   drivers/net/thunderx/nicvf_ethdev.c       | 753 
>> +++++++++++++++++++++---------
>>   drivers/net/thunderx/nicvf_ethdev.h       |  39 ++
>>   drivers/net/thunderx/nicvf_rxtx.c         |  14 +-
>>   drivers/net/thunderx/nicvf_struct.h       |   6 +-
>>   drivers/net/thunderx/nicvf_svf.c          |  78 ++++
>>   drivers/net/thunderx/nicvf_svf.h          |  66 +++
>>   15 files changed, 1046 insertions(+), 246 deletions(-)
>>   create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
>>   create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
>>   create mode 100644 drivers/net/thunderx/nicvf_svf.c
>>   create mode 100644 drivers/net/thunderx/nicvf_svf.h
>>
>

Re: [PATCH 01/13] net/thunderx: cleanup the driver before adding new features

[Ferruh Yigit]

On 8/26/2016 5:53 PM, Kamil Rytarowski wrote:
> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> 
> Refactored features:
>  - enable nicvf_qset_rbdr_precharge to handle handle secondary queue sets
double "handle"

>  - rte_free already handles NULL pointer
>  - check mempool flags to predict being contiguous in memory
>  - allow to use mempool with multiple memory chunks

I am not able to find the implementation for this.

>  - simplify local construct of accessing nb_rx_queus
s/nb_rx_queus/nb_rx_queues

> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
>  drivers/net/thunderx/base/nicvf_hw.c | 10 +++++-----
>  drivers/net/thunderx/base/nicvf_hw.h |  6 +++---
>  drivers/net/thunderx/nicvf_ethdev.c  | 32 ++++++++++++--------------------
>  3 files changed, 20 insertions(+), 28 deletions(-)
> 
> diff --git a/drivers/net/thunderx/base/nicvf_hw.c b/drivers/net/thunderx/base/nicvf_hw.c
> index 4bdd183..1f08ef2 100644
> --- a/drivers/net/thunderx/base/nicvf_hw.c
> +++ b/drivers/net/thunderx/base/nicvf_hw.c
> @@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
>  		return NICVF_ERR_BASE_INIT;
>  
>  	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
> -		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
> +		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;

is this new flag NICVF_CAP_CQE_RX2 flag described in commit log?

>  
>  	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
>  		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
> @@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
>  }
>  
>  int
> -nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
> -			  rbdr_pool_get_handler handler,
> -			  void *opaque, uint32_t max_buffs)
> +nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
> +			  uint16_t ridx, rbdr_pool_get_handler handler,
> +			  uint32_t max_buffs)
>  {
>  	struct rbdr_entry_t *desc, *desc0;
>  	struct nicvf_rbdr *rbdr = nic->rbdr;
> @@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
>  		if (count >= max_buffs)
>  			break;
>  		desc0 = desc + count;
> -		phy = handler(opaque);
> +		phy = handler(dev, nic);
>  		if (phy) {
>  			desc0->full_addr = phy;
>  			count++;
> diff --git a/drivers/net/thunderx/base/nicvf_hw.h b/drivers/net/thunderx/base/nicvf_hw.h
> index a6cda82..2b8738b 100644
> --- a/drivers/net/thunderx/base/nicvf_hw.h
> +++ b/drivers/net/thunderx/base/nicvf_hw.h
> @@ -85,7 +85,7 @@ enum nicvf_err_e {
>  	NICVF_ERR_RSS_GET_SZ,    /* -8171 */
>  };
>  
> -typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
> +typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
>  
>  struct nicvf_hw_rx_qstats {
>  	uint64_t q_rx_bytes;
> @@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
>  
>  int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
>  int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
> -int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
> -			      rbdr_pool_get_handler handler, void *opaque,
> +int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
> +			      uint16_t ridx, rbdr_pool_get_handler handler,
>  			      uint32_t max_buffs);
>  int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
>  
> diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
> index 4402f6a..48f2cd2 100644
> --- a/drivers/net/thunderx/nicvf_ethdev.c
> +++ b/drivers/net/thunderx/nicvf_ethdev.c
> @@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
>  	int ret;
>  
>  	/* Count started rx queues */
> -	for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
> +	for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
>  		if (dev->data->rx_queue_state[qidx] ==
>  		    RTE_ETH_QUEUE_STATE_STARTED)
>  			qcnt++;
> @@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
>  static void
>  nicvf_dev_rx_queue_release(void *rx_queue)
>  {
> -	struct nicvf_rxq *rxq = rx_queue;
> -
>  	PMD_INIT_FUNC_TRACE();
>  
> -	if (rxq)
> -		rte_free(rxq);
> +	rte_free(rx_queue);
>  }
>  
>  static int
> @@ -1087,9 +1084,9 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
>  		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
>  		socket_id, nic->node);
>  
> -	/* Mempool memory should be contiguous */
> -	if (mp->nb_mem_chunks != 1) {
> -		PMD_INIT_LOG(ERR, "Non contiguous mempool, check huge page sz");
> +	/* Mempool memory must be contiguous */
> +	if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG) {

If you need continuous memory, this check is not enough.
Not having this flag doesn't guaranties that memory is continuous, this
flag can be set but still can have multiple mem_chunks. And there is no
guarantee that mem_chunks are continuous.

> +		PMD_INIT_LOG(ERR, "Mempool memory must be contiguous");
>  		return -EINVAL;
>  	}
>  
> @@ -1212,15 +1209,16 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
>  }
>  
>  static nicvf_phys_addr_t
> -rbdr_rte_mempool_get(void *opaque)
> +rbdr_rte_mempool_get(void *dev, void *opaque)
>  {
>  	uint16_t qidx;
>  	uintptr_t mbuf;
>  	struct nicvf_rxq *rxq;
> -	struct nicvf *nic = nicvf_pmd_priv((struct rte_eth_dev *)opaque);
> +	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
> +	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
>  
> -	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
> -		rxq = nic->eth_dev->data->rx_queues[qidx];
> +	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
> +		rxq = eth_dev->data->rx_queues[qidx];
>  		/* Maintain equal buffer count across all pools */
>  		if (rxq->precharge_cnt >= rxq->qlen_mask)
>  			continue;
> @@ -1354,8 +1352,8 @@ nicvf_dev_start(struct rte_eth_dev *dev)
>  	}
>  
>  	/* Fill rte_mempool buffers in RBDR pool and precharge it */
> -	ret = nicvf_qset_rbdr_precharge(nic, 0, rbdr_rte_mempool_get,
> -					dev, total_rxq_desc);
> +	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
> +					total_rxq_desc);
>  	if (ret) {
>  		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
>  		goto qset_rbdr_reclaim;
> @@ -1721,12 +1719,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
>  		goto malloc_fail;
>  	}
>  
> -	ret = nicvf_mbox_get_rss_size(nic);
> -	if (ret) {
> -		PMD_INIT_LOG(ERR, "Failed to get rss table size");
> -		goto malloc_fail;
> -	}
> -

Is removing mbox_get_rss_size() mentioned in commit log?

>  	PMD_INIT_LOG(INFO, "Port %d (%x:%x) mac=%02x:%02x:%02x:%02x:%02x:%02x",
>  		eth_dev->data->port_id, nic->vendor_id, nic->device_id,
>  		nic->mac_addr[0], nic->mac_addr[1], nic->mac_addr[2],
> 

Re: [PATCH 04/13] net/thunderx/base: add secondary queue set support

[Ferruh Yigit]

On 8/26/2016 5:53 PM, Kamil Rytarowski wrote:
> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> 
> Changes:
>  - add new message sqs_alloc in mailbox
>  - add a queue container to hold secondary qsets.
>  - add nicvf_mbox_request_sqs
>  - handle new mailbox messages for secondary queue set support
>  - register secondary queue sets for furthe reuse
>  - register the number secondary queue sets in MSG_QS_CFG
> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
>  drivers/net/thunderx/base/nicvf_hw_defs.h |  1 +
>  drivers/net/thunderx/base/nicvf_mbox.c    | 34 ++++++++++++++++++++++++++++++-
>  drivers/net/thunderx/base/nicvf_mbox.h    | 21 +++++++++++++++++--
>  drivers/net/thunderx/nicvf_struct.h       |  5 +++++
>  4 files changed, 58 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h b/drivers/net/thunderx/base/nicvf_hw_defs.h
> index 2f2b225..3b947e0 100644
> --- a/drivers/net/thunderx/base/nicvf_hw_defs.h
> +++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
> @@ -207,6 +207,7 @@
>  #define NICVF_CQE_RX2_RBPTR_WORD        (7)
>  
>  #define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
> +#define assert_if_secondary(nic) assert((nic)->sqs_mode == 0)

assert_if_not_secondary?

Re: [PATCH 07/13] net/thunderx: fix multiprocess support in stats

[Ferruh Yigit]

On 8/26/2016 5:54 PM, Kamil Rytarowski wrote:
> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> 
> In case of the multiprocess mode a shared nicvf struct between processes
> cannot point with the eth_dev pointer to master device, therefore remove it
> allong with references to it refactoring the code where needed.

Patch subject says fix multiprocess support in stat, but it seems doing
more than just for stats fix.

Overall eliminates private_data->eht_dev link.

I guess this is because eth_dev->data is shared for primary and
secondary processes, and this makes impossible to use separate
private_data for primary and secondaries.

So this patch looks like keeping separate copy of private_data (nic) and
using eth_dev and nic structs for functions instead of using
eth_dev->data->private_data.

If above correct, can you please updated patch subject?

Also another approach can be allocating "data" independently and
overwrite eth_dev->data with this per each process, this also makes
eth_dev->data->private_data usable for each process.

> 
> Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")
> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---

Re: [PATCH 12/13] net/thunderx: add final bits for secondary queue support

[Ferruh Yigit]

On 8/26/2016 5:54 PM, Kamil Rytarowski wrote:
> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---

Is it possible to give more details what has been done in this patch for
secondary queue support?

thanks,
ferruh

Re: [PATCH 13/13] net/thunderx: document secondary queue set support

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Kamil Rytarowski
> Sent: Friday, August 26, 2016 5:54 PM
> To: dev@dpdk.org
> Cc: maciej.czekaj@caviumnetworks.com; zyta.szpak@semihalf.com;
> slawomir.rosek@semihalf.com; rad@semihalf.com;
> jerin.jacob@caviumnetworks.com; Kamil Rytarowski
> <kamil.rytarowski@caviumnetworks.com>
> Subject: [dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue
> set support
> 

There are some whitespace errors in the docs:

    Applying patch #15435 using 'git am'
    Description: [dpdk-dev,13/13] net/thunderx: document secondary queue set support
    Applying: net/thunderx: document secondary queue set support
    .git/rebase-apply/patch:70: trailing whitespace.
          
    .git/rebase-apply/patch:74: trailing whitespace.
          
    .git/rebase-apply/patch:98: trailing whitespace.



Some other minor comments below.



> 
>  Supported ThunderX SoCs
>  -----------------------
> @@ -322,6 +323,112 @@ This section provides instructions to configure SR-
> IOV with Linux OS.
>  #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for
> instruction
>     how to launch ``testpmd`` application.
> 
> +Multiple Queue Set per DPDK port configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are two types of VFs:
> +
> +- Primary VF
> +- Secondary VF
> +
> +Each port consist of a primary VF and n secondary VF(s). Each VF provides

s/consist/consists/


> 8 Tx/Rx queues to a port.
> +In case port is configured to use more than 8 queues, then it requires
> +one (or more) secondary VF. Each secondary VF adds additional 8 queues to
> the queue set.

There are a few missing definite and indefinite articles missing in the text.


> +
> +During PMD driver initialization, the primary VF's are enumerated by
> +checking the specific flag (see sqs message in DPDK boot log - sqs
> indicates secondary queue set).
> +They are at the beginning of VF list (the remain ones are secondary
> VF's).
> +
> +The primary VFs are used as master queue sets. Secondary VFs provid

s/provid/provide/

> +additional queue sets for primary ones. If a port is configured for
> +more then
> +8 queues than it will request for additional queues from secondary VFs.
> +
> +Secondary VFs cannot be shared between primary VFs.
> +
> +Primary VFs are present on the beginning of the 'Network devices using
> +kernel driver' list, secondary VFs are on the remaining on the remaining
> part of the list.
> +
> +   .. note::
> +

This note and the following one are indented too far. They should be aligned with the margin.



> ...


> +Example device binding
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +If a system has three interfaces, a total of 18 VF devices will be
> +created on a non-NUMA machine.
> +
> +   .. note::
> +
> +      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
> +
> +   .. code-block:: console

This note and code block are indented too far. They should be aligned with the margin.


> +
> +      # tools/dpdk-devbind.py --status
> +
> +      Network devices using DPDK-compatible driver
> +      ============================================
> +      <none>
> +
> +      Network devices using kernel driver
> +      ===================================
> +      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-
> pci,uio_pci_generic
> +      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf
> + unused=vfio-pci,uio_pci_generic
> +
> +      Other network devices
> +      =====================
> +      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
> +
> +
> +We want to bind two physical interfaces with 24 queues each device, we
> +attach two primary VFs and four secondary queues. In our example we
> choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
> +We will chose four secondary queue sets from the ending of the list
> (0002:01:01.7-0002:01:02.2).

s/chose/choose/

> +
> +
> +#. Bind two primary VFs to the ``vfio-pci`` driver:
> +
> +   .. code-block:: console
> +

These code blocks are indented correctly.


John.
-- 

Re: [PATCH 01/13] net/thunderx: cleanup the driver before adding new features

[Maciej Czekaj]

> On 8/26/2016 5:53 PM, Kamil Rytarowski wrote:
>> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>
>> Refactored features:
>>  - enable nicvf_qset_rbdr_precharge to handle handle secondary queue sets
> double "handle"

Will fix comment in v2.

>
>>  - rte_free already handles NULL pointer	
>>  - check mempool flags to predict being contiguous in memory
>>  - allow to use mempool with multiple memory chunks
>
> I am not able to find the implementation for this.

As commented below, it is not true so will remove that point in v2.


>
>>  - simplify local construct of accessing nb_rx_queus
> s/nb_rx_queus/nb_rx_queues


Will fix comment in v2.


>
>>
>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> ---
>>  drivers/net/thunderx/base/nicvf_hw.c | 10 +++++-----
>>  drivers/net/thunderx/base/nicvf_hw.h |  6 +++---
>>  drivers/net/thunderx/nicvf_ethdev.c  | 32 ++++++++++++--------------------
>>  3 files changed, 20 insertions(+), 28 deletions(-)
>>
>> diff --git a/drivers/net/thunderx/base/nicvf_hw.c b/drivers/net/thunderx/base/nicvf_hw.c
>> index 4bdd183..1f08ef2 100644
>> --- a/drivers/net/thunderx/base/nicvf_hw.c
>> +++ b/drivers/net/thunderx/base/nicvf_hw.c
>> @@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
>>  		return NICVF_ERR_BASE_INIT;
>>
>>  	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
>> -		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
>> +		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
>
> is this new flag NICVF_CAP_CQE_RX2 flag described in commit log?
>
>>
>>  	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
>>  		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
>> @@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
>>  }
>>
>>  int
>> -nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
>> -			  rbdr_pool_get_handler handler,
>> -			  void *opaque, uint32_t max_buffs)
>> +nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
>> +			  uint16_t ridx, rbdr_pool_get_handler handler,
>> +			  uint32_t max_buffs)
>>  {
>>  	struct rbdr_entry_t *desc, *desc0;
>>  	struct nicvf_rbdr *rbdr = nic->rbdr;
>> @@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
>>  		if (count >= max_buffs)
>>  			break;
>>  		desc0 = desc + count;
>> -		phy = handler(opaque);
>> +		phy = handler(dev, nic);
>>  		if (phy) {
>>  			desc0->full_addr = phy;
>>  			count++;
>> diff --git a/drivers/net/thunderx/base/nicvf_hw.h b/drivers/net/thunderx/base/nicvf_hw.h
>> index a6cda82..2b8738b 100644
>> --- a/drivers/net/thunderx/base/nicvf_hw.h
>> +++ b/drivers/net/thunderx/base/nicvf_hw.h
>> @@ -85,7 +85,7 @@ enum nicvf_err_e {
>>  	NICVF_ERR_RSS_GET_SZ,    /* -8171 */
>>  };
>>
>> -typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
>> +typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
>>
>>  struct nicvf_hw_rx_qstats {
>>  	uint64_t q_rx_bytes;
>> @@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
>>
>>  int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
>>  int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
>> -int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
>> -			      rbdr_pool_get_handler handler, void *opaque,
>> +int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
>> +			      uint16_t ridx, rbdr_pool_get_handler handler,
>>  			      uint32_t max_buffs);
>>  int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
>>
>> diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
>> index 4402f6a..48f2cd2 100644
>> --- a/drivers/net/thunderx/nicvf_ethdev.c
>> +++ b/drivers/net/thunderx/nicvf_ethdev.c
>> @@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
>>  	int ret;
>>
>>  	/* Count started rx queues */
>> -	for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
>> +	for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
>>  		if (dev->data->rx_queue_state[qidx] ==
>>  		    RTE_ETH_QUEUE_STATE_STARTED)
>>  			qcnt++;
>> @@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
>>  static void
>>  nicvf_dev_rx_queue_release(void *rx_queue)
>>  {
>> -	struct nicvf_rxq *rxq = rx_queue;
>> -
>>  	PMD_INIT_FUNC_TRACE();
>>
>> -	if (rxq)
>> -		rte_free(rxq);
>> +	rte_free(rx_queue);
>>  }
>>
>>  static int
>> @@ -1087,9 +1084,9 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
>>  		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
>>  		socket_id, nic->node);
>>
>> -	/* Mempool memory should be contiguous */
>> -	if (mp->nb_mem_chunks != 1) {
>> -		PMD_INIT_LOG(ERR, "Non contiguous mempool, check huge page sz");
>> +	/* Mempool memory must be contiguous */
>> +	if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG) {
>
> If you need continuous memory, this check is not enough.
> Not having this flag doesn't guaranties that memory is continuous, this
> flag can be set but still can have multiple mem_chunks. And there is no
> guarantee that mem_chunks are continuous.

True, we need both checks:
	
	mp->nb_mem_chunks == 1 && !MEMPOOL_F_NO_PHYS_CONTIG.

Will fix in v2.

>
>> +		PMD_INIT_LOG(ERR, "Mempool memory must be contiguous");
>>  		return -EINVAL;
>>  	}
>>
>> @@ -1212,15 +1209,16 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
>>  }
>>
>>  static nicvf_phys_addr_t
>> -rbdr_rte_mempool_get(void *opaque)
>> +rbdr_rte_mempool_get(void *dev, void *opaque)
>>  {
>>  	uint16_t qidx;
>>  	uintptr_t mbuf;
>>  	struct nicvf_rxq *rxq;
>> -	struct nicvf *nic = nicvf_pmd_priv((struct rte_eth_dev *)opaque);
>> +	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
>> +	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
>>
>> -	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
>> -		rxq = nic->eth_dev->data->rx_queues[qidx];
>> +	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
>> +		rxq = eth_dev->data->rx_queues[qidx];
>>  		/* Maintain equal buffer count across all pools */
>>  		if (rxq->precharge_cnt >= rxq->qlen_mask)
>>  			continue;
>> @@ -1354,8 +1352,8 @@ nicvf_dev_start(struct rte_eth_dev *dev)
>>  	}
>>
>>  	/* Fill rte_mempool buffers in RBDR pool and precharge it */
>> -	ret = nicvf_qset_rbdr_precharge(nic, 0, rbdr_rte_mempool_get,
>> -					dev, total_rxq_desc);
>> +	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
>> +					total_rxq_desc);
>>  	if (ret) {
>>  		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
>>  		goto qset_rbdr_reclaim;
>> @@ -1721,12 +1719,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
>>  		goto malloc_fail;
>>  	}
>>
>> -	ret = nicvf_mbox_get_rss_size(nic);
>> -	if (ret) {
>> -		PMD_INIT_LOG(ERR, "Failed to get rss table size");
>> -		goto malloc_fail;
>> -	}
>> -
>
> Is removing mbox_get_rss_size() mentioned in commit log?

This chage remove spare function call but is not mentioned.
Will add a comment in v2.


>>  	PMD_INIT_LOG(INFO, "Port %d (%x:%x) mac=%02x:%02x:%02x:%02x:%02x:%02x",
>>  		eth_dev->data->port_id, nic->vendor_id, nic->device_id,
>>  		nic->mac_addr[0], nic->mac_addr[1], nic->mac_addr[2],
>>
>
>
>

Re: [PATCH 04/13] net/thunderx/base: add secondary queue set support

[Maciej Czekaj]

> On 8/26/2016 5:53 PM, Kamil Rytarowski wrote:
>> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>
>> Changes:
>>  - add new message sqs_alloc in mailbox
>>  - add a queue container to hold secondary qsets.
>>  - add nicvf_mbox_request_sqs
>>  - handle new mailbox messages for secondary queue set support
>>  - register secondary queue sets for furthe reuse
>>  - register the number secondary queue sets in MSG_QS_CFG
>>
>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> ---
>>  drivers/net/thunderx/base/nicvf_hw_defs.h |  1 +
>>  drivers/net/thunderx/base/nicvf_mbox.c    | 34 ++++++++++++++++++++++++++++++-
>>  drivers/net/thunderx/base/nicvf_mbox.h    | 21 +++++++++++++++++--
>>  drivers/net/thunderx/nicvf_struct.h       |  5 +++++
>>  4 files changed, 58 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h b/drivers/net/thunderx/base/nicvf_hw_defs.h
>> index 2f2b225..3b947e0 100644
>> --- a/drivers/net/thunderx/base/nicvf_hw_defs.h
>> +++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
>> @@ -207,6 +207,7 @@
>>  #define NICVF_CQE_RX2_RBPTR_WORD        (7)
>>
>>  #define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
>> +#define assert_if_secondary(nic) assert((nic)->sqs_mode == 0)
>
> assert_if_not_secondary?
>
>

Will be refactored to macro nic_is_primary in v2.

Re: [PATCH 07/13] net/thunderx: fix multiprocess support in stats

[Maciej Czekaj]

> On 8/26/2016 5:54 PM, Kamil Rytarowski wrote:
>> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>
>> In case of the multiprocess mode a shared nicvf struct between processes
>> cannot point with the eth_dev pointer to master device, therefore remove it
>> allong with references to it refactoring the code where needed.
>
> Patch subject says fix multiprocess support in stat, but it seems doing
> more than just for stats fix.
>
> Overall eliminates private_data->eht_dev link.
>
> I guess this is because eth_dev->data is shared for primary and
> secondary processes, and this makes impossible to use separate
> private_data for primary and secondaries.
>
> So this patch looks like keeping separate copy of private_data (nic) and
> using eth_dev and nic structs for functions instead of using
> eth_dev->data->private_data.
>
> If above correct, can you please updated patch subject?
>
> Also another approach can be allocating "data" independently and
> overwrite eth_dev->data with this per each process, this also makes
> eth_dev->data->private_data usable for each process.
>
>>
>> Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")
>>
>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> ---
>
>
>

Yes, this patch enables multi-process support for primary and secondary 
functions by eliminating the pointer from private struct to eth_dev.

Will extend the comment in v2.

Re: [PATCH 12/13] net/thunderx: add final bits for secondary queue support

[Maciej Czekaj]

> On 8/26/2016 5:54 PM, Kamil Rytarowski wrote:
>> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>
>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> ---
>
> Is it possible to give more details what has been done in this patch for
> secondary queue support?
>
> thanks,
> ferruh
>
>
>

Will add more details in v2.

Re: [PATCH 13/13] net/thunderx: document secondary queue set support

[Maciej Czekaj]

We will address all the issues in v2.


>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Kamil Rytarowski
>> Sent: Friday, August 26, 2016 5:54 PM
>> To: dev@dpdk.org
>> Cc: maciej.czekaj@caviumnetworks.com; zyta.szpak@semihalf.com;
>> slawomir.rosek@semihalf.com; rad@semihalf.com;
>> jerin.jacob@caviumnetworks.com; Kamil Rytarowski
>> <kamil.rytarowski@caviumnetworks.com>
>> Subject: [dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue
>> set support
>>
>
> There are some whitespace errors in the docs:
>
>     Applying patch #15435 using 'git am'
>     Description: [dpdk-dev,13/13] net/thunderx: document secondary queue set support
>     Applying: net/thunderx: document secondary queue set support
>     .git/rebase-apply/patch:70: trailing whitespace.
>
>     .git/rebase-apply/patch:74: trailing whitespace.
>
>     .git/rebase-apply/patch:98: trailing whitespace.
>
>
>
> Some other minor comments below.
>
>
>
>>
>>  Supported ThunderX SoCs
>>  -----------------------
>> @@ -322,6 +323,112 @@ This section provides instructions to configure SR-
>> IOV with Linux OS.
>>  #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for
>> instruction
>>     how to launch ``testpmd`` application.
>>
>> +Multiple Queue Set per DPDK port configuration
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +There are two types of VFs:
>> +
>> +- Primary VF
>> +- Secondary VF
>> +
>> +Each port consist of a primary VF and n secondary VF(s). Each VF provides
>
> s/consist/consists/
>
>
>> 8 Tx/Rx queues to a port.
>> +In case port is configured to use more than 8 queues, then it requires
>> +one (or more) secondary VF. Each secondary VF adds additional 8 queues to
>> the queue set.
>
> There are a few missing definite and indefinite articles missing in the text.
>
>
>> +
>> +During PMD driver initialization, the primary VF's are enumerated by
>> +checking the specific flag (see sqs message in DPDK boot log - sqs
>> indicates secondary queue set).
>> +They are at the beginning of VF list (the remain ones are secondary
>> VF's).
>> +
>> +The primary VFs are used as master queue sets. Secondary VFs provid
>
> s/provid/provide/
>
>> +additional queue sets for primary ones. If a port is configured for
>> +more then
>> +8 queues than it will request for additional queues from secondary VFs.
>> +
>> +Secondary VFs cannot be shared between primary VFs.
>> +
>> +Primary VFs are present on the beginning of the 'Network devices using
>> +kernel driver' list, secondary VFs are on the remaining on the remaining
>> part of the list.
>> +
>> +   .. note::
>> +
>
> This note and the following one are indented too far. They should be aligned with the margin.
>
>
>
>> ...
>
>
>> +Example device binding
>> +~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +If a system has three interfaces, a total of 18 VF devices will be
>> +created on a non-NUMA machine.
>> +
>> +   .. note::
>> +
>> +      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
>> +
>> +   .. code-block:: console
>
> This note and code block are indented too far. They should be aligned with the margin.
>
>
>> +
>> +      # tools/dpdk-devbind.py --status
>> +
>> +      Network devices using DPDK-compatible driver
>> +      ============================================
>> +      <none>
>> +
>> +      Network devices using kernel driver
>> +      ===================================
>> +      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-
>> pci,uio_pci_generic
>> +      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf
>> + unused=vfio-pci,uio_pci_generic
>> +
>> +      Other network devices
>> +      =====================
>> +      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
>> +
>> +
>> +We want to bind two physical interfaces with 24 queues each device, we
>> +attach two primary VFs and four secondary queues. In our example we
>> choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
>> +We will chose four secondary queue sets from the ending of the list
>> (0002:01:01.7-0002:01:02.2).
>
> s/chose/choose/
>
>> +
>> +
>> +#. Bind two primary VFs to the ``vfio-pci`` driver:
>> +
>> +   .. code-block:: console
>> +
>
> These code blocks are indented correctly.
>
>
> John.
>

[PATCH v2 00/15] Add support for secondary queue set in nicvf thunderx driver

[Kamil Rytarowski]

This series of patches adds support for secondary queue set in nicvf thunderx
driver

There are two types of VFs:
 - Primary VF
 - Secondary VF

Each port consist of a primary VF and n secondary VF(s). Each VF provides 8
Tx/Rx queues to a port. In case port is configured to use more than 8 queues,
then it requires one (or more) secondary VF. Each secondary VF adds additional
8 queues to the queue set.

During PMD driver initialization, the primary VF's are enumerated by checking the
specific flag (see READY message). They are at the beginning of  VF list (the remain
ones are secondary VF's).

The primary VFs are used as master queue sets. Secondary VFs provide
additional queue sets for primary ones. If a port is configured for more then
8 queues then it will request for additional queues from secondary VFs.

Secondary VFs cannot be shared between primary VFs.

Primary VFs are present on the tail of the 'Network devices using kernel
driver' list, secondary VFs are on the remaining tail of the list.

The VNIC driver in the multiqueue setup works differently than other drivers
like `ixgbe`. We need to bind separately each specific queue set device with
the ``tools/dpdk-devbind.py`` utility.

Depending on the hardware used, the kernel driver sets a threshold ``vf_id``.
VFs that try to attach with an id below or equal to this boundary are
considered primary VFs. VFs that try to attach with an id above this boundary
are considered secondary VFs.

This patchset also contains other cleanups and improvements like fixing
erroneous checksum calculation and preparing the thunderx driver for the multi
queue set feature support.


These changes base on the following pending patches:

[dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver
http://dpdk.org/dev/patchwork/patch/14963/

[dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag
http://dpdk.org/dev/patchwork/patch/14964/

[dpdk-dev,3/3] net/thunderx: add 81xx SoC support
http://dpdk.org/dev/patchwork/patch/14965/

Kamil Rytarowski (15):
  net/thunderx: cleanup the driver before adding new features
  net/thunderx: correct transmit checksum handling
  net/thunderx/base: add family of functions to store qsets
  net/thunderx/base: add secondary queue set support
  net/thunderx: add family of functions to store DPDK qsets
  net/thunderx: add secondary queue set in interrupt functions
  net/thunderx: remove problematic private_data->eth_dev link
  net/thunderx: add helper utils for secondary qset support
  net/thunderx: add secondary qset support in dev stop/close
  net/thunderx: add secondary qset support in device start
  net/thunderx: add secondary qset support in device configure
  net/thunderx: add final bits for secondary queue support
  net/thunderx: document secondary queue set support
  ethdev: Support VFs on the different PCI domains
  net/thunderx: Bump driver version to 2.0

 doc/guides/nics/thunderx.rst              | 148 +++++-
 drivers/net/thunderx/Makefile             |   2 +
 drivers/net/thunderx/base/nicvf_bsvf.c    |  72 +++
 drivers/net/thunderx/base/nicvf_bsvf.h    |  76 +++
 drivers/net/thunderx/base/nicvf_hw.c      |  10 +-
 drivers/net/thunderx/base/nicvf_hw.h      |   6 +-
 drivers/net/thunderx/base/nicvf_hw_defs.h |   1 +
 drivers/net/thunderx/base/nicvf_mbox.c    |  47 +-
 drivers/net/thunderx/base/nicvf_mbox.h    |  21 +-
 drivers/net/thunderx/nicvf_ethdev.c       | 780 +++++++++++++++++++++---------
 drivers/net/thunderx/nicvf_ethdev.h       |  41 +-
 drivers/net/thunderx/nicvf_rxtx.c         |  14 +-
 drivers/net/thunderx/nicvf_struct.h       |   6 +-
 drivers/net/thunderx/nicvf_svf.c          |  78 +++
 drivers/net/thunderx/nicvf_svf.h          |  66 +++
 lib/librte_ether/rte_ethdev.c             |   2 +-
 16 files changed, 1097 insertions(+), 273 deletions(-)
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
 create mode 100644 drivers/net/thunderx/nicvf_svf.c
 create mode 100644 drivers/net/thunderx/nicvf_svf.h

-- 
1.9.1

[PATCH v2 01/15] net/thunderx: cleanup the driver before adding new features

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Refactored features:
 - enable nicvf_qset_rbdr_precharge to handle secondary queue sets
 - rte_free already handles NULL pointer
 - check mempool flags to predict being contiguous in memory
 - prohibit to use mempool with multiple memory chunks
 - simplify local construct of accessing nb_rx_queues
 - enable NICVF_CAP_CQE_RX2 on CN88XX PASS2.0 hardware.
 - remove redundant check for RSS size in nicvf_eth_dev_init

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/base/nicvf_hw.c | 10 +++++-----
 drivers/net/thunderx/base/nicvf_hw.h |  6 +++---
 drivers/net/thunderx/nicvf_ethdev.c  | 37 ++++++++++++++++++------------------
 3 files changed, 26 insertions(+), 27 deletions(-)

diff --git a/drivers/net/thunderx/base/nicvf_hw.c b/drivers/net/thunderx/base/nicvf_hw.c
index 4bdd183..1f08ef2 100644
--- a/drivers/net/thunderx/base/nicvf_hw.c
+++ b/drivers/net/thunderx/base/nicvf_hw.c
@@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
 		return NICVF_ERR_BASE_INIT;
 
 	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
-		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
+		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
 
 	if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
 		nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
@@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
 }
 
 int
-nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
-			  rbdr_pool_get_handler handler,
-			  void *opaque, uint32_t max_buffs)
+nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+			  uint16_t ridx, rbdr_pool_get_handler handler,
+			  uint32_t max_buffs)
 {
 	struct rbdr_entry_t *desc, *desc0;
 	struct nicvf_rbdr *rbdr = nic->rbdr;
@@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
 		if (count >= max_buffs)
 			break;
 		desc0 = desc + count;
-		phy = handler(opaque);
+		phy = handler(dev, nic);
 		if (phy) {
 			desc0->full_addr = phy;
 			count++;
diff --git a/drivers/net/thunderx/base/nicvf_hw.h b/drivers/net/thunderx/base/nicvf_hw.h
index a6cda82..2b8738b 100644
--- a/drivers/net/thunderx/base/nicvf_hw.h
+++ b/drivers/net/thunderx/base/nicvf_hw.h
@@ -85,7 +85,7 @@ enum nicvf_err_e {
 	NICVF_ERR_RSS_GET_SZ,    /* -8171 */
 };
 
-typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
+typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
 
 struct nicvf_hw_rx_qstats {
 	uint64_t q_rx_bytes;
@@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
 
 int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
 int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
-int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
-			      rbdr_pool_get_handler handler, void *opaque,
+int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+			      uint16_t ridx, rbdr_pool_get_handler handler,
 			      uint32_t max_buffs);
 int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
 
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 914035e..3cf32bf 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
 	int ret;
 
 	/* Count started rx queues */
-	for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+	for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
 		if (dev->data->rx_queue_state[qidx] ==
 		    RTE_ETH_QUEUE_STATE_STARTED)
 			qcnt++;
@@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
 static void
 nicvf_dev_rx_queue_release(void *rx_queue)
 {
-	struct nicvf_rxq *rxq = rx_queue;
-
 	PMD_INIT_FUNC_TRACE();
 
-	if (rxq)
-		rte_free(rxq);
+	rte_free(rx_queue);
 }
 
 static int
@@ -1070,6 +1067,7 @@ nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 	return nicvf_stop_tx_queue(dev, qidx);
 }
 
+
 static int
 nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 			 uint16_t nb_desc, unsigned int socket_id,
@@ -1087,9 +1085,15 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
 		socket_id, nic->node);
 
-	/* Mempool memory should be contiguous */
+	/* Mempool memory must be contiguous, so must be one memory segment*/
 	if (mp->nb_mem_chunks != 1) {
-		PMD_INIT_LOG(ERR, "Non contiguous mempool, check huge page sz");
+		PMD_INIT_LOG(ERR, "Non-contiguous mempool, add more huge pages");
+		return -EINVAL;
+	}
+
+	/* Mempool memory must be physically contiguous */
+	if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG) {
+		PMD_INIT_LOG(ERR, "Mempool memory must be physically contiguous");
 		return -EINVAL;
 	}
 
@@ -1212,15 +1216,16 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 }
 
 static nicvf_phys_addr_t
-rbdr_rte_mempool_get(void *opaque)
+rbdr_rte_mempool_get(void *dev, void *opaque)
 {
 	uint16_t qidx;
 	uintptr_t mbuf;
 	struct nicvf_rxq *rxq;
-	struct nicvf *nic = nicvf_pmd_priv((struct rte_eth_dev *)opaque);
+	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
+	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
 
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
-		rxq = nic->eth_dev->data->rx_queues[qidx];
+	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
+		rxq = eth_dev->data->rx_queues[qidx];
 		/* Maintain equal buffer count across all pools */
 		if (rxq->precharge_cnt >= rxq->qlen_mask)
 			continue;
@@ -1354,8 +1359,8 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	}
 
 	/* Fill rte_mempool buffers in RBDR pool and precharge it */
-	ret = nicvf_qset_rbdr_precharge(nic, 0, rbdr_rte_mempool_get,
-					dev, total_rxq_desc);
+	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
+					total_rxq_desc);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
 		goto qset_rbdr_reclaim;
@@ -1721,12 +1726,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 		goto malloc_fail;
 	}
 
-	ret = nicvf_mbox_get_rss_size(nic);
-	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to get rss table size");
-		goto malloc_fail;
-	}
-
 	PMD_INIT_LOG(INFO, "Port %d (%x:%x) mac=%02x:%02x:%02x:%02x:%02x:%02x",
 		eth_dev->data->port_id, nic->vendor_id, nic->device_id,
 		nic->mac_addr[0], nic->mac_addr[1], nic->mac_addr[2],
-- 
1.9.1

[PATCH v2 02/15] net/thunderx: correct transmit checksum handling

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

The symbols PKT_TX_TCP_CKSUM and PKT_TX_UDP_CKSUM are not bits on a
bitmask. Set l3_offset always for TX offloads, not just for PKT_TX_IP_CKSUM
being true.

Fixes: 1c421f18e095 ("net/thunderx: add single and multi-segment Tx")

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_rxtx.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_rxtx.c b/drivers/net/thunderx/nicvf_rxtx.c
index eb51a72..e15c730 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -70,19 +70,20 @@ fill_sq_desc_header(union sq_entry_t *entry, struct rte_mbuf *pkt)
 	ol_flags = pkt->ol_flags & NICVF_TX_OFFLOAD_MASK;
 	if (unlikely(ol_flags)) {
 		/* L4 cksum */
-		if (ol_flags & PKT_TX_TCP_CKSUM)
+		uint64_t l4_flags = ol_flags & PKT_TX_L4_MASK;
+		if (l4_flags == PKT_TX_TCP_CKSUM)
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_TCP;
-		else if (ol_flags & PKT_TX_UDP_CKSUM)
+		else if (l4_flags == PKT_TX_UDP_CKSUM)
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_UDP;
 		else
 			sqe.hdr.csum_l4 = SEND_L4_CSUM_DISABLE;
+
+		sqe.hdr.l3_offset = pkt->l2_len;
 		sqe.hdr.l4_offset = pkt->l3_len + pkt->l2_len;
 
 		/* L3 cksum */
-		if (ol_flags & PKT_TX_IP_CKSUM) {
+		if (ol_flags & PKT_TX_IP_CKSUM)
 			sqe.hdr.csum_l3 = 1;
-			sqe.hdr.l3_offset = pkt->l2_len;
-		}
 	}
 
 	entry->buff[0] = sqe.buff[0];
-- 
1.9.1

[PATCH v2 03/15] net/thunderx/base: add family of functions to store qsets

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

This interface (nicvf_bsvf) will be used for secondary queue set support.

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/Makefile          |  1 +
 drivers/net/thunderx/base/nicvf_bsvf.c | 72 ++++++++++++++++++++++++++++++++
 drivers/net/thunderx/base/nicvf_bsvf.h | 76 ++++++++++++++++++++++++++++++++++
 3 files changed, 149 insertions(+)
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
 create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h

diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 8ea6b45..711400a 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -57,6 +57,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_rxtx.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
 
 ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
 CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.c b/drivers/net/thunderx/base/nicvf_bsvf.c
new file mode 100644
index 0000000..9e028a3
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.c
@@ -0,0 +1,72 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <assert.h>
+#include <stddef.h>
+#include <err.h>
+
+#include "nicvf_bsvf.h"
+#include "nicvf_plat.h"
+
+static SIMPLEQ_HEAD(, svf_entry) head = SIMPLEQ_HEAD_INITIALIZER(head);
+
+void
+nicvf_bsvf_push(struct svf_entry *entry)
+{
+	assert(entry != NULL);
+	assert(entry->vf != NULL);
+
+	SIMPLEQ_INSERT_TAIL(&head, entry, next);
+}
+
+struct svf_entry *
+nicvf_bsvf_pop(void)
+{
+	struct svf_entry *entry;
+
+	assert(!SIMPLEQ_EMPTY(&head));
+
+	entry = SIMPLEQ_FIRST(&head);
+
+	assert(entry != NULL);
+	assert(entry->vf != NULL);
+
+	SIMPLEQ_REMOVE_HEAD(&head, next);
+
+	return entry;
+}
+
+int
+nicvf_bsvf_empty(void)
+{
+	return SIMPLEQ_EMPTY(&head);
+}
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.h b/drivers/net/thunderx/base/nicvf_bsvf.h
new file mode 100644
index 0000000..5d5a25e
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.h
@@ -0,0 +1,76 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __THUNDERX_NICVF_BSVF_H__
+#define __THUNDERX_NICVF_BSVF_H__
+
+#include <sys/queue.h>
+
+struct nicvf;
+
+/**
+ * The base queue structure to hold secondary qsets.
+ */
+struct svf_entry {
+	SIMPLEQ_ENTRY(svf_entry) next; /**< Next element's pointer */
+	struct nicvf *vf;              /**< Holder of a secondary qset */
+};
+
+/**
+ * Enqueue new entry to secondary qsets.
+ *
+ * @param entry
+ *   Entry to be enqueued.
+ */
+void
+nicvf_bsvf_push(struct svf_entry *entry);
+
+/**
+ * Dequeue an entry from secondary qsets.
+ *
+ * @return
+ *   Dequeued entry.
+ */
+struct svf_entry *
+nicvf_bsvf_pop(void);
+
+/**
+ * Check if the queue of secondary qsets is empty.
+ *
+ * @return
+ *   0 on non-empty
+ *   otherwise empty
+ */
+int
+nicvf_bsvf_empty(void);
+
+#endif /* __THUNDERX_NICVF_BSVF_H__  */
-- 
1.9.1

[PATCH v2 04/15] net/thunderx/base: add secondary queue set support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Changes:
 - add new message sqs_alloc in mailbox
 - add a queue container to hold secondary qsets.
 - add nicvf_mbox_request_sqs
 - handle new mailbox messages for secondary queue set support
 - register secondary queue sets for furthe reuse
 - register the number secondary queue sets in MSG_QS_CFG

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/base/nicvf_hw_defs.h |  1 +
 drivers/net/thunderx/base/nicvf_mbox.c    | 47 ++++++++++++++++++++++++++++++-
 drivers/net/thunderx/base/nicvf_mbox.h    | 21 ++++++++++++--
 drivers/net/thunderx/nicvf_struct.h       |  5 ++++
 4 files changed, 71 insertions(+), 3 deletions(-)

diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h b/drivers/net/thunderx/base/nicvf_hw_defs.h
index 2f2b225..e144d44 100644
--- a/drivers/net/thunderx/base/nicvf_hw_defs.h
+++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
@@ -207,6 +207,7 @@
 #define NICVF_CQE_RX2_RBPTR_WORD        (7)
 
 #define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
+#define assert_primary(nic) assert((nic)->sqs_mode == 0)
 
 typedef uint64_t nicvf_phys_addr_t;
 
diff --git a/drivers/net/thunderx/base/nicvf_mbox.c b/drivers/net/thunderx/base/nicvf_mbox.c
index 9c5cd83..3b7b8a5 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.c
+++ b/drivers/net/thunderx/base/nicvf_mbox.c
@@ -62,6 +62,9 @@ static const char *mbox_message[NIC_MBOX_MSG_MAX] =  {
 	[NIC_MBOX_MSG_RESET_STAT_COUNTER] = "NIC_MBOX_MSG_RESET_STAT_COUNTER",
 	[NIC_MBOX_MSG_CFG_DONE]           = "NIC_MBOX_MSG_CFG_DONE",
 	[NIC_MBOX_MSG_SHUTDOWN]           = "NIC_MBOX_MSG_SHUTDOWN",
+	[NIC_MBOX_MSG_RES_BIT]            = "NIC_MBOX_MSG_RES_BIT",
+	[NIC_MBOX_MSG_RSS_SIZE_RES_BIT]   = "NIC_MBOX_MSG_RSS_SIZE",
+	[NIC_MBOX_MSG_ALLOC_SQS_RES_BIT]  = "NIC_MBOX_MSG_ALLOC_SQS",
 };
 
 static inline const char * __attribute__((unused))
@@ -173,7 +176,7 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
 	case NIC_MBOX_MSG_NACK:
 		nic->pf_nacked = true;
 		break;
-	case NIC_MBOX_MSG_RSS_SIZE:
+	case NIC_MBOX_MSG_RSS_SIZE_RES_BIT:
 		nic->rss_info.rss_size = mbx.rss_size.ind_tbl_size;
 		nic->pf_acked = true;
 		break;
@@ -183,6 +186,26 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
 		nic->speed = mbx.link_status.speed;
 		nic->pf_acked = true;
 		break;
+	case NIC_MBOX_MSG_ALLOC_SQS_RES_BIT:
+		assert_primary(nic);
+		if (mbx.sqs_alloc.qs_count != nic->sqs_count) {
+			nicvf_log_error("Received %" PRIu8 "/%" PRIu8
+			                " secondary qsets",
+			                mbx.sqs_alloc.qs_count,
+			                nic->sqs_count);
+			abort();
+		}
+		for (i = 0; i < mbx.sqs_alloc.qs_count; i++) {
+			if (mbx.sqs_alloc.svf[i] != nic->snicvf[i]->vf_id) {
+				nicvf_log_error("Received secondary qset[%zu] "
+				                "ID %" PRIu8 " expected %"
+				                PRIu8, i, mbx.sqs_alloc.svf[i],
+				                nic->snicvf[i]->vf_id);
+				abort();
+			}
+		}
+		nic->pf_acked = true;
+		break;
 	default:
 		nicvf_log_error("Invalid message from PF, msg_id=0x%hhx %s",
 				mbx.msg.msg, nicvf_mbox_msg_str(mbx.msg.msg));
@@ -314,11 +337,33 @@ nicvf_mbox_qset_config(struct nicvf *nic, struct pf_qs_cfg *qs_cfg)
 	/* Send a mailbox msg to PF to config Qset */
 	mbx.msg.msg = NIC_MBOX_MSG_QS_CFG;
 	mbx.qs.num = nic->vf_id;
+	mbx.qs.sqs_count = nic->sqs_count;
 	mbx.qs.cfg = qs_cfg->value;
 	return nicvf_mbox_send_msg_to_pf(nic, &mbx);
 }
 
 int
+nicvf_mbox_request_sqs(struct nicvf *nic)
+{
+	struct nic_mbx mbx = { .msg = { 0 } };
+	size_t i;
+
+	assert_primary(nic);
+	assert(nic->sqs_count > 0);
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	mbx.sqs_alloc.msg = NIC_MBOX_MSG_ALLOC_SQS;
+	mbx.sqs_alloc.spec = 1;
+	mbx.sqs_alloc.qs_count = nic->sqs_count;
+
+	/* Set no of Rx/Tx queues in each of the SQsets */
+	for (i = 0; i < nic->sqs_count; i++)
+		mbx.sqs_alloc.svf[i] = nic->snicvf[i]->vf_id;
+
+	return nicvf_mbox_send_msg_to_pf(nic, &mbx);
+}
+
+int
 nicvf_mbox_rq_drop_config(struct nicvf *nic, uint16_t qidx, bool enable)
 {
 	struct nic_mbx mbx = { .msg = { 0 } };
diff --git a/drivers/net/thunderx/base/nicvf_mbox.h b/drivers/net/thunderx/base/nicvf_mbox.h
index 7c0c6a9..084f3a7 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.h
+++ b/drivers/net/thunderx/base/nicvf_mbox.h
@@ -36,6 +36,7 @@
 #include <stdint.h>
 
 #include "nicvf_plat.h"
+#include "../nicvf_struct.h"
 
 /* PF <--> VF Mailbox communication
  * Two 64bit registers are shared between PF and VF for each VF
@@ -67,10 +68,16 @@
 #define	NIC_MBOX_MSG_ALLOC_SQS		0x12	/* Allocate secondary Qset */
 #define	NIC_MBOX_MSG_LOOPBACK		0x16	/* Set interface in loopback */
 #define	NIC_MBOX_MSG_RESET_STAT_COUNTER 0x17	/* Reset statistics counters */
-#define	NIC_MBOX_MSG_CFG_DONE		0xF0	/* VF configuration done */
-#define	NIC_MBOX_MSG_SHUTDOWN		0xF1	/* VF is being shutdown */
+#define	NIC_MBOX_MSG_CFG_DONE		0x7E	/* VF configuration done */
+#define	NIC_MBOX_MSG_SHUTDOWN		0x7F	/* VF is being shutdown */
+#define	NIC_MBOX_MSG_RES_BIT		0x80	/* Reset bit from PF */
 #define	NIC_MBOX_MSG_MAX		0x100	/* Maximum number of messages */
 
+#define NIC_MBOX_MSG_RSS_SIZE_RES_BIT \
+	(NIC_MBOX_MSG_RSS_SIZE | NIC_MBOX_MSG_RES_BIT)
+#define NIC_MBOX_MSG_ALLOC_SQS_RES_BIT \
+	(NIC_MBOX_MSG_ALLOC_SQS | NIC_MBOX_MSG_RES_BIT)
+
 /* Get vNIC VF configuration */
 struct nic_cfg_msg {
 	uint8_t    msg;
@@ -155,6 +162,14 @@ struct bgx_link_status {
 	uint32_t   speed;
 };
 
+/* Allocate additional SQS to VF */
+struct sqs_alloc {
+	uint8_t    msg;
+	uint8_t    spec;
+	uint8_t    qs_count;
+	uint8_t    svf[MAX_SQS_PER_VF];
+};
+
 /* Set interface in loopback mode */
 struct set_loopback {
 	uint8_t    msg;
@@ -201,6 +216,7 @@ union {
 	struct rss_sz_msg	rss_size;
 	struct rss_cfg_msg	rss_cfg;
 	struct bgx_link_status  link_status;
+	struct sqs_alloc	sqs_alloc;
 	struct set_loopback	lbk;
 	struct reset_stat_cfg	reset_stat;
 };
@@ -211,6 +227,7 @@ NICVF_STATIC_ASSERT(sizeof(struct nic_mbx) <= 16);
 int nicvf_handle_mbx_intr(struct nicvf *nic);
 int nicvf_mbox_check_pf_ready(struct nicvf *nic);
 int nicvf_mbox_qset_config(struct nicvf *nic, struct pf_qs_cfg *qs_cfg);
+int nicvf_mbox_request_sqs(struct nicvf *nic);
 int nicvf_mbox_rq_config(struct nicvf *nic, uint16_t qidx,
 			 struct pf_rq_cfg *pf_rq_cfg);
 int nicvf_mbox_sq_config(struct nicvf *nic, uint16_t qidx);
diff --git a/drivers/net/thunderx/nicvf_struct.h b/drivers/net/thunderx/nicvf_struct.h
index c52545d..a72f752 100644
--- a/drivers/net/thunderx/nicvf_struct.h
+++ b/drivers/net/thunderx/nicvf_struct.h
@@ -119,6 +119,11 @@ struct nicvf {
 	uint16_t mtu;
 	bool vlan_filter_en;
 	uint8_t mac_addr[ETHER_ADDR_LEN];
+	/* secondary queue set support */
+	uint8_t sqs_id;
+	uint8_t sqs_count;
+#define MAX_SQS_PER_VF 11
+	struct nicvf *snicvf[MAX_SQS_PER_VF];
 } __rte_cache_aligned;
 
 #endif /* _THUNDERX_NICVF_STRUCT_H */
-- 
1.9.1

[PATCH v2 05/15] net/thunderx: add family of functions to store DPDK qsets

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

These functions (nicvf_svf) are DPDK specialization of base/nicvf_bsvf.[ch]
ones.

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/Makefile    |  1 +
 drivers/net/thunderx/nicvf_svf.c | 78 ++++++++++++++++++++++++++++++++++++++++
 drivers/net/thunderx/nicvf_svf.h | 66 ++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/net/thunderx/nicvf_svf.c
 create mode 100644 drivers/net/thunderx/nicvf_svf.h

diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 711400a..bcab5f9 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -58,6 +58,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
 SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_svf.c
 
 ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
 CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/nicvf_svf.c b/drivers/net/thunderx/nicvf_svf.c
new file mode 100644
index 0000000..f746e94
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.c
@@ -0,0 +1,78 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <assert.h>
+#include <stddef.h>
+
+#include <rte_debug.h>
+#include <rte_malloc.h>
+
+#include "base/nicvf_bsvf.h"
+
+#include "nicvf_svf.h"
+
+void
+nicvf_svf_push(struct nicvf *vf)
+{
+	struct svf_entry *entry = NULL;
+
+	assert(vf != NULL);
+
+	entry = rte_zmalloc("nicvf", sizeof(*entry), RTE_CACHE_LINE_SIZE);
+	if (entry == NULL)
+		rte_panic("Cannoc allocate memory for svf_entry\n");
+
+	entry->vf = vf;
+
+	nicvf_bsvf_push(entry);
+}
+
+struct nicvf *
+nicvf_svf_pop(void)
+{
+	struct nicvf *vf;
+	struct svf_entry *entry;
+
+	entry = nicvf_bsvf_pop();
+
+	vf = entry->vf;
+
+	rte_free(entry);
+
+	return vf;
+}
+
+int
+nicvf_svf_empty(void)
+{
+	return nicvf_bsvf_empty();
+}
diff --git a/drivers/net/thunderx/nicvf_svf.h b/drivers/net/thunderx/nicvf_svf.h
new file mode 100644
index 0000000..6471aa5
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.h
@@ -0,0 +1,66 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright (C) Cavium networks Ltd. 2016.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Cavium networks nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __THUNDERX_NICVF_SVF_H__
+#define __THUNDERX_NICVF_SVF_H__
+
+struct nicvf;
+
+/**
+ * Enqueue new VF to secondary qsets.
+ *
+ * @param entry
+ *   Entry to be enqueued.
+ */
+void
+nicvf_svf_push(struct nicvf *vf);
+
+/**
+ * Dequeue a VF from secondary qsets.
+ *
+ * @return
+ *   Dequeued entry.
+ */
+struct nicvf *
+nicvf_svf_pop(void);
+
+/**
+ * Check if the queue of secondary qsets is empty.
+ *
+ * @return
+ *   0 on non-empty
+ *   otherwise empty
+ */
+int
+nicvf_svf_empty(void);
+
+#endif /* __THUNDERX_NICVF_SVF_H__  */
-- 
1.9.1

[PATCH v2 06/15] net/thunderx: add secondary queue set in interrupt functions

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 41 ++++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 17 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 3cf32bf..72e6667 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -101,31 +101,40 @@ nicvf_set_eth_link_status(struct nicvf *nic, struct rte_eth_link *link)
 static void
 nicvf_interrupt(void *arg)
 {
-	struct nicvf *nic = arg;
+	struct rte_eth_dev *dev = arg;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 
 	if (nicvf_reg_poll_interrupts(nic) == NIC_MBOX_MSG_BGX_LINK_CHANGE) {
-		if (nic->eth_dev->data->dev_conf.intr_conf.lsc)
-			nicvf_set_eth_link_status(nic,
-					&nic->eth_dev->data->dev_link);
-		_rte_eth_dev_callback_process(nic->eth_dev,
-				RTE_ETH_EVENT_INTR_LSC);
+		if (dev->data->dev_conf.intr_conf.lsc)
+			nicvf_set_eth_link_status(nic, &dev->data->dev_link);
+		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC);
 	}
 
 	rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
-				nicvf_interrupt, nic);
+				nicvf_interrupt, dev);
+}
+
+static void __rte_unused
+nicvf_vf_interrupt(void *arg)
+{
+	struct nicvf *nic = arg;
+
+	nicvf_reg_poll_interrupts(nic);
+
+	rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
+				nicvf_vf_interrupt, nic);
 }
 
 static int
-nicvf_periodic_alarm_start(struct nicvf *nic)
+nicvf_periodic_alarm_start(void (fn)(void *), void *arg)
 {
-	return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
-					nicvf_interrupt, nic);
+	return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000, fn, arg);
 }
 
 static int
-nicvf_periodic_alarm_stop(struct nicvf *nic)
+nicvf_periodic_alarm_stop(void (fn)(void *), void *arg)
 {
-	return rte_eal_alarm_cancel(nicvf_interrupt, nic);
+	return rte_eal_alarm_cancel(fn, arg);
 }
 
 /*
@@ -1519,12 +1528,10 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 static void
 nicvf_dev_close(struct rte_eth_dev *dev)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
-
 	PMD_INIT_FUNC_TRACE();
 
 	nicvf_dev_stop(dev);
-	nicvf_periodic_alarm_stop(nic);
+	nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
 }
 
 static int
@@ -1675,7 +1682,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 
 	nicvf_disable_all_interrupts(nic);
 
-	ret = nicvf_periodic_alarm_start(nic);
+	ret = nicvf_periodic_alarm_start(nicvf_interrupt, eth_dev);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to start period alarm");
 		goto fail;
@@ -1736,7 +1743,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 malloc_fail:
 	rte_free(eth_dev->data->mac_addrs);
 alarm_fail:
-	nicvf_periodic_alarm_stop(nic);
+	nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
 fail:
 	return ret;
 }
-- 
1.9.1

[PATCH v2 07/15] net/thunderx: remove problematic private_data->eth_dev link

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

In case of the multiprocess mode a shared nicvf struct between processes
cannot point with the eth_dev pointer to master device, therefore remove it
allong with references to it refactoring the code where needed.

This change fixes multiprocess issues detected in stats.

Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 69 ++++++++++++++++++-------------------
 drivers/net/thunderx/nicvf_rxtx.c   |  3 +-
 drivers/net/thunderx/nicvf_struct.h |  1 -
 3 files changed, 36 insertions(+), 37 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 72e6667..c8b42f7 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -497,14 +497,14 @@ nicvf_dev_rss_hash_update(struct rte_eth_dev *dev,
 }
 
 static int
-nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
-		    uint32_t desc_cnt)
+nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		    struct nicvf_rxq *rxq, uint16_t qidx, uint32_t desc_cnt)
 {
 	const struct rte_memzone *rz;
 	uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "cq_ring", qidx, ring_size,
-					NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+				      NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
 		return -ENOMEM;
@@ -520,13 +520,13 @@ nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
 }
 
 static int
-nicvf_qset_sq_alloc(struct nicvf *nic,  struct nicvf_txq *sq, uint16_t qidx,
-		    uint32_t desc_cnt)
+nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		    struct nicvf_txq *sq, uint16_t qidx, uint32_t desc_cnt)
 {
 	const struct rte_memzone *rz;
 	uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "sq", qidx, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
 				NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
@@ -543,7 +543,8 @@ nicvf_qset_sq_alloc(struct nicvf *nic,  struct nicvf_txq *sq, uint16_t qidx,
 }
 
 static int
-nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
+nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+		      uint32_t desc_cnt, uint32_t buffsz)
 {
 	struct nicvf_rbdr *rbdr;
 	const struct rte_memzone *rz;
@@ -558,7 +559,7 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
 	}
 
 	ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
-	rz = rte_eth_dma_zone_reserve(nic->eth_dev, "rbdr", 0, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
 				   NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
@@ -583,14 +584,15 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
 }
 
 static void
-nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic __rte_unused,
+			nicvf_phys_addr_t phy)
 {
 	uint16_t qidx;
 	void *obj;
 	struct nicvf_rxq *rxq;
 
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
-		rxq = nic->eth_dev->data->rx_queues[qidx];
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+		rxq = dev->data->rx_queues[qidx];
 		if (rxq->precharge_cnt) {
 			obj = (void *)nicvf_mbuff_phy2virt(phy,
 							   rxq->mbuf_phys_off);
@@ -602,7 +604,7 @@ nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
 }
 
 static inline void
-nicvf_rbdr_release_mbufs(struct nicvf *nic)
+nicvf_rbdr_release_mbufs(struct rte_eth_dev *dev, struct nicvf *nic)
 {
 	uint32_t qlen_mask, head;
 	struct rbdr_entry_t *entry;
@@ -612,7 +614,7 @@ nicvf_rbdr_release_mbufs(struct nicvf *nic)
 	head = rbdr->head;
 	while (head != rbdr->tail) {
 		entry = rbdr->desc + head;
-		nicvf_rbdr_release_mbuf(nic, entry->full_addr);
+		nicvf_rbdr_release_mbuf(dev, nic, entry->full_addr);
 		head++;
 		head = head & qlen_mask;
 	}
@@ -724,14 +726,13 @@ nicvf_configure_rss(struct rte_eth_dev *dev)
 			dev->data->dev_conf.rx_adv_conf.rss_conf.rss_hf);
 	PMD_DRV_LOG(INFO, "mode=%d rx_queues=%d loopback=%d rsshf=0x%" PRIx64,
 		    dev->data->dev_conf.rxmode.mq_mode,
-		    nic->eth_dev->data->nb_rx_queues,
-		    nic->eth_dev->data->dev_conf.lpbk_mode, rsshf);
+		    dev->data->nb_rx_queues,
+		    dev->data->dev_conf.lpbk_mode, rsshf);
 
 	if (dev->data->dev_conf.rxmode.mq_mode == ETH_MQ_RX_NONE)
 		ret = nicvf_rss_term(nic);
 	else if (dev->data->dev_conf.rxmode.mq_mode == ETH_MQ_RX_RSS)
-		ret = nicvf_rss_config(nic,
-				       nic->eth_dev->data->nb_rx_queues, rsshf);
+		ret = nicvf_rss_config(nic, dev->data->nb_rx_queues, rsshf);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to configure RSS %d", ret);
 
@@ -915,7 +916,7 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 		return -ENOMEM;
 	}
 
-	if (nicvf_qset_sq_alloc(nic, txq, qidx, nb_desc)) {
+	if (nicvf_qset_sq_alloc(dev, nic, txq, qidx, nb_desc)) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for sq %d", qidx);
 		nicvf_dev_tx_queue_release(txq);
 		return -ENOMEM;
@@ -932,12 +933,11 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 }
 
 static inline void
-nicvf_rx_queue_release_mbufs(struct nicvf_rxq *rxq)
+nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 {
 	uint32_t rxq_cnt;
 	uint32_t nb_pkts, released_pkts = 0;
 	uint32_t refill_cnt = 0;
-	struct rte_eth_dev *dev = rxq->nic->eth_dev;
 	struct rte_mbuf *rx_pkts[NICVF_MAX_RX_FREE_THRESH];
 
 	if (dev->rx_pkt_burst == NULL)
@@ -1017,7 +1017,7 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
 
 	other_error = ret;
 	rxq = dev->data->rx_queues[qidx];
-	nicvf_rx_queue_release_mbufs(rxq);
+	nicvf_rx_queue_release_mbufs(dev, rxq);
 	nicvf_rx_queue_reset(rxq);
 
 	ret = nicvf_qset_cq_reclaim(nic, qidx);
@@ -1163,7 +1163,7 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 
 	/* Alloc completion queue */
-	if (nicvf_qset_cq_alloc(nic, rxq, rxq->queue_id, nb_desc)) {
+	if (nicvf_qset_cq_alloc(dev, nic, rxq, rxq->queue_id, nb_desc)) {
 		PMD_INIT_LOG(ERR, "failed to allocate cq %u", rxq->queue_id);
 		nicvf_dev_rx_queue_release(rxq);
 		return -ENOMEM;
@@ -1281,7 +1281,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	 */
 
 	/* Validate RBDR buff size */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		mbp_priv = rte_mempool_get_priv(rxq->pool);
 		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
@@ -1299,7 +1299,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	}
 
 	/* Validate mempool attributes */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		rxq->mbuf_phys_off = nicvf_mempool_phy_offset(rxq->pool);
 		mbuf = rte_pktmbuf_alloc(rxq->pool);
@@ -1323,12 +1323,12 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 
 	/* Check the level of buffers in the pool */
 	total_rxq_desc = 0;
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		/* Count total numbers of rxq descs */
 		total_rxq_desc += rxq->qlen_mask + 1;
 		exp_buffs = RTE_MEMPOOL_CACHE_MAX_SIZE + rxq->rx_free_thresh;
-		exp_buffs *= nic->eth_dev->data->nb_rx_queues;
+		exp_buffs *= dev->data->nb_rx_queues;
 		if (rte_mempool_avail_count(rxq->pool) < exp_buffs) {
 			PMD_INIT_LOG(ERR, "Buff shortage in pool=%s (%d/%d)",
 				     rxq->pool->name,
@@ -1354,7 +1354,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 
 	/* Allocate RBDR and RBDR ring desc */
 	nb_rbdr_desc = nicvf_qsize_rbdr_roundup(total_rxq_desc);
-	ret = nicvf_qset_rbdr_alloc(nic, nb_rbdr_desc, rbdrsz);
+	ret = nicvf_qset_rbdr_alloc(dev, nic, nb_rbdr_desc, rbdrsz);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc");
 		goto qset_reclaim;
@@ -1379,7 +1379,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		     nic->rbdr->tail, nb_rbdr_desc);
 
 	/* Configure RX queues */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
 		ret = nicvf_start_rx_queue(dev, qidx);
 		if (ret)
 			goto start_rxq_error;
@@ -1389,7 +1389,7 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
 
 	/* Configure TX queues */
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_tx_queues; qidx++) {
+	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
 		ret = nicvf_start_tx_queue(dev, qidx);
 		if (ret)
 			goto start_txq_error;
@@ -1448,14 +1448,14 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 qset_rss_error:
 	nicvf_rss_term(nic);
 start_txq_error:
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_tx_queues; qidx++)
+	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
 		nicvf_stop_tx_queue(dev, qidx);
 start_rxq_error:
-	for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
 		nicvf_stop_rx_queue(dev, qidx);
 qset_rbdr_reclaim:
 	nicvf_qset_rbdr_reclaim(nic, 0);
-	nicvf_rbdr_release_mbufs(nic);
+	nicvf_rbdr_release_mbufs(dev, nic);
 qset_rbdr_free:
 	if (nic->rbdr) {
 		rte_free(nic->rbdr);
@@ -1501,7 +1501,7 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 
 	/* Move all charged buffers in RBDR back to pool */
 	if (nic->rbdr != NULL)
-		nicvf_rbdr_release_mbufs(nic);
+		nicvf_rbdr_release_mbufs(dev, nic);
 
 	/* Reclaim CPI configuration */
 	if (!nic->sqs_mode) {
@@ -1666,7 +1666,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 	nic->vendor_id = pci_dev->id.vendor_id;
 	nic->subsystem_device_id = pci_dev->id.subsystem_device_id;
 	nic->subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
-	nic->eth_dev = eth_dev;
 
 	PMD_INIT_LOG(DEBUG, "nicvf: device (%x:%x) %u:%u:%u:%u",
 			pci_dev->id.vendor_id, pci_dev->id.device_id,
diff --git a/drivers/net/thunderx/nicvf_rxtx.c b/drivers/net/thunderx/nicvf_rxtx.c
index e15c730..fc43b74 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -368,7 +368,8 @@ nicvf_fill_rbdr(struct nicvf_rxq *rxq, int to_fill)
 	void *obj_p[NICVF_MAX_RX_FREE_THRESH] __rte_cache_aligned;
 
 	if (unlikely(rte_mempool_get_bulk(rxq->pool, obj_p, to_fill) < 0)) {
-		rxq->nic->eth_dev->data->rx_mbuf_alloc_failed += to_fill;
+		rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed +=
+			to_fill;
 		return 0;
 	}
 
diff --git a/drivers/net/thunderx/nicvf_struct.h b/drivers/net/thunderx/nicvf_struct.h
index a72f752..c900e12 100644
--- a/drivers/net/thunderx/nicvf_struct.h
+++ b/drivers/net/thunderx/nicvf_struct.h
@@ -113,7 +113,6 @@ struct nicvf {
 	uint16_t subsystem_vendor_id;
 	struct nicvf_rbdr *rbdr;
 	struct nicvf_rss_reta_info rss_info;
-	struct rte_eth_dev *eth_dev;
 	struct rte_intr_handle intr_handle;
 	uint8_t cpi_alg;
 	uint16_t mtu;
-- 
1.9.1

[PATCH v2 08/15] net/thunderx: add helper utils for secondary qset support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.h | 39 +++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/net/thunderx/nicvf_ethdev.h b/drivers/net/thunderx/nicvf_ethdev.h
index 34447e0..e162364 100644
--- a/drivers/net/thunderx/nicvf_ethdev.h
+++ b/drivers/net/thunderx/nicvf_ethdev.h
@@ -87,6 +87,17 @@ nicvf_mbuff_meta_length(struct rte_mbuf *mbuf)
 	return (uint16_t)((uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf);
 }
 
+static inline uint16_t
+nicvf_netdev_qidx(struct nicvf *nic, uint8_t local_qidx)
+{
+	uint16_t global_qidx = local_qidx;
+
+	if (nic->sqs_mode)
+		global_qidx += ((nic->sqs_id + 1) * MAX_CMP_QUEUES_PER_QS);
+
+	return global_qidx;
+}
+
 /*
  * Simple phy2virt functions assuming mbufs are in a single huge page
  * V = P + offset
@@ -104,4 +115,32 @@ nicvf_mbuff_virt2phy(uintptr_t virt, uint64_t mbuf_phys_off)
 	return (phys_addr_t)(virt - mbuf_phys_off);
 }
 
+static inline void
+nicvf_tx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *tx_start,
+	       uint16_t *tx_end)
+{
+	uint16_t tmp;
+
+	*tx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+				    MAX_SND_QUEUES_PER_QS);
+	tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+			     MAX_SND_QUEUES_PER_QS) - 1;
+	*tx_end = dev->data->nb_tx_queues ?
+		RTE_MIN(tmp, dev->data->nb_tx_queues - 1) : 0;
+}
+
+static inline void
+nicvf_rx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *rx_start,
+	       uint16_t *rx_end)
+{
+	uint16_t tmp;
+
+	*rx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+				    MAX_RCV_QUEUES_PER_QS);
+	tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+			     MAX_RCV_QUEUES_PER_QS) - 1;
+	*rx_end = dev->data->nb_rx_queues ?
+		RTE_MIN(tmp, dev->data->nb_rx_queues - 1) : 0;
+}
+
 #endif /* __THUNDERX_NICVF_ETHDEV_H__  */
-- 
1.9.1

[PATCH v2 09/15] net/thunderx: add secondary qset support in dev stop/close

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 142 +++++++++++++++++++++++++++---------
 1 file changed, 107 insertions(+), 35 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index c8b42f7..7ca7d07 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -67,9 +67,13 @@
 
 #include "nicvf_ethdev.h"
 #include "nicvf_rxtx.h"
+#include "nicvf_svf.h"
 #include "nicvf_logs.h"
 
 static void nicvf_dev_stop(struct rte_eth_dev *dev);
+static void nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup);
+static void nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic,
+			  bool cleanup);
 
 static inline int
 nicvf_atomic_write_link_status(struct rte_eth_dev *dev,
@@ -674,23 +678,29 @@ config_sq_error:
 }
 
 static inline int
-nicvf_stop_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
 	struct nicvf_txq *txq;
 	int ret;
 
-	if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+	if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
-	ret = nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+	ret = nicvf_qset_sq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim sq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim sq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
-	txq = dev->data->tx_queues[qidx];
+	txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_tx_queue_release_mbufs(txq);
 	nicvf_tx_queue_reset(txq);
 
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return ret;
 }
 
@@ -1002,30 +1012,34 @@ config_rq_error:
 }
 
 static inline int
-nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+		       uint16_t qidx)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
 	struct nicvf_rxq *rxq;
 	int ret, other_error;
 
-	if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+	if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STOPPED)
 		return 0;
 
 	ret = nicvf_qset_rq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim rq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim rq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error = ret;
-	rxq = dev->data->rx_queues[qidx];
+	rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
 	nicvf_rx_queue_release_mbufs(dev, rxq);
 	nicvf_rx_queue_reset(rxq);
 
 	ret = nicvf_qset_cq_reclaim(nic, qidx);
 	if (ret)
-		PMD_INIT_LOG(ERR, "Failed to reclaim cq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to reclaim cq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 
 	other_error |= ret;
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return other_error;
 }
 
@@ -1057,8 +1071,14 @@ static int
 nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
 	int ret;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 
-	ret = nicvf_stop_rx_queue(dev, qidx);
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+	ret = nicvf_vf_stop_rx_queue(dev, nic, qidx);
 	ret |= nicvf_configure_cpi(dev);
 	ret |= nicvf_configure_rss_reta(dev);
 	return ret;
@@ -1073,7 +1093,14 @@ nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 static int
 nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 {
-	return nicvf_stop_tx_queue(dev, qidx);
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+	return nicvf_vf_stop_tx_queue(dev, nic, qidx);
 }
 
 
@@ -1449,10 +1476,10 @@ qset_rss_error:
 	nicvf_rss_term(nic);
 start_txq_error:
 	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+		nicvf_vf_stop_tx_queue(dev, nic, qidx);
 start_rxq_error:
 	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+		nicvf_vf_stop_rx_queue(dev, nic, qidx);
 qset_rbdr_reclaim:
 	nicvf_qset_rbdr_reclaim(nic, 0);
 	nicvf_rbdr_release_mbufs(dev, nic);
@@ -1467,32 +1494,74 @@ qset_reclaim:
 }
 
 static void
-nicvf_dev_stop(struct rte_eth_dev *dev)
+nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup)
 {
+	size_t i;
 	int ret;
-	uint16_t qidx;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Let PF make the BGX's RX and TX switches to OFF position */
-	nicvf_mbox_shutdown(nic);
+	/* Teardown secondary vf first */
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_vf_stop(dev, nic->snicvf[i], cleanup);
+	}
+
+	/* Stop the primary VF now */
+	nicvf_vf_stop(dev, nic, cleanup);
 
 	/* Disable loopback */
 	ret = nicvf_loopback_config(nic, 0);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable loopback %d", ret);
 
+	/* Reclaim CPI configuration */
+	ret = nicvf_mbox_config_cpi(nic, 0);
+	if (ret)
+		PMD_INIT_LOG(ERR, "Failed to reclaim CPI config %d", ret);
+}
+
+static void
+nicvf_dev_stop(struct rte_eth_dev *dev)
+{
+	PMD_INIT_FUNC_TRACE();
+
+	nicvf_dev_stop_cleanup(dev, false);
+}
+
+static void
+nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic, bool cleanup)
+{
+	int ret;
+	uint16_t qidx;
+	uint16_t tx_start, tx_end;
+	uint16_t rx_start, rx_end;
+
+	PMD_INIT_FUNC_TRACE();
+
+	if (cleanup) {
+		/* Let PF make the BGX's RX and TX switches to OFF position */
+		nicvf_mbox_shutdown(nic);
+	}
+
 	/* Disable VLAN Strip */
 	nicvf_vlan_hw_strip(nic, 0);
 
-	/* Reclaim sq */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_stop_tx_queue(dev, qidx);
+	/* Get queue ranges for this VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
+	for (qidx = tx_start; qidx <= tx_end; qidx++)
+		nicvf_vf_stop_tx_queue(dev, nic, qidx % MAX_SND_QUEUES_PER_QS);
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/* Reclaim rq */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_stop_rx_queue(dev, qidx);
+	for (qidx = rx_start; qidx <= rx_end; qidx++)
+		nicvf_vf_stop_rx_queue(dev, nic, qidx % MAX_RCV_QUEUES_PER_QS);
 
 	/* Reclaim RBDR */
 	ret = nicvf_qset_rbdr_reclaim(nic, 0);
@@ -1503,15 +1572,8 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 	if (nic->rbdr != NULL)
 		nicvf_rbdr_release_mbufs(dev, nic);
 
-	/* Reclaim CPI configuration */
-	if (!nic->sqs_mode) {
-		ret = nicvf_mbox_config_cpi(nic, 0);
-		if (ret)
-			PMD_INIT_LOG(ERR, "Failed to reclaim CPI config");
-	}
-
 	/* Disable qset */
-	ret = nicvf_qset_config(nic);
+	ret = nicvf_qset_reclaim(nic);
 	if (ret)
 		PMD_INIT_LOG(ERR, "Failed to disable qset %d", ret);
 
@@ -1528,10 +1590,20 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
 static void
 nicvf_dev_close(struct rte_eth_dev *dev)
 {
+	size_t i;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
 	PMD_INIT_FUNC_TRACE();
 
-	nicvf_dev_stop(dev);
+	nicvf_dev_stop_cleanup(dev, true);
 	nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
+
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (!nic->snicvf[i])
+			continue;
+
+		nicvf_periodic_alarm_stop(nicvf_vf_interrupt, nic->snicvf[i]);
+	}
 }
 
 static int
-- 
1.9.1

[PATCH v2 10/15] net/thunderx: add secondary qset support in device start

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 266 +++++++++++++++++++++++-------------
 1 file changed, 172 insertions(+), 94 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 7ca7d07..f7d5188 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -653,27 +653,33 @@ nicvf_tx_queue_reset(struct nicvf_txq *txq)
 }
 
 static inline int
-nicvf_start_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+			uint16_t qidx)
 {
 	struct nicvf_txq *txq;
 	int ret;
 
-	if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+	assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+	if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STARTED)
 		return 0;
 
-	txq = dev->data->tx_queues[qidx];
+	txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
 	txq->pool = NULL;
-	ret = nicvf_qset_sq_config(nicvf_pmd_priv(dev), qidx, txq);
+	ret = nicvf_qset_sq_config(nic, qidx, txq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure sq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure sq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_sq_error;
 	}
 
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STARTED;
 	return ret;
 
 config_sq_error:
-	nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+	nicvf_qset_sq_reclaim(nic, qidx);
 	return ret;
 }
 
@@ -977,31 +983,37 @@ nicvf_rx_queue_reset(struct nicvf_rxq *rxq)
 }
 
 static inline int
-nicvf_start_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+			uint16_t qidx)
 {
-	struct nicvf *nic = nicvf_pmd_priv(dev);
 	struct nicvf_rxq *rxq;
 	int ret;
 
-	if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+	assert(qidx < MAX_RCV_QUEUES_PER_QS);
+
+	if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+		RTE_ETH_QUEUE_STATE_STARTED)
 		return 0;
 
 	/* Update rbdr pointer to all rxq */
-	rxq = dev->data->rx_queues[qidx];
+	rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
 	rxq->shared_rbdr = nic->rbdr;
 
 	ret = nicvf_qset_rq_config(nic, qidx, rxq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure rq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure rq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_rq_error;
 	}
 	ret = nicvf_qset_cq_config(nic, qidx, rxq);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure cq %d %d", qidx, ret);
+		PMD_INIT_LOG(ERR, "Failed to configure cq VF%d %d %d",
+			     nic->vf_id, qidx, ret);
 		goto config_cq_error;
 	}
 
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STARTED;
 	return 0;
 
 config_cq_error:
@@ -1054,9 +1066,15 @@ nicvf_dev_rx_queue_release(void *rx_queue)
 static int
 nicvf_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 {
+	struct nicvf *nic = nicvf_pmd_priv(dev);
 	int ret;
 
-	ret = nicvf_start_rx_queue(dev, qidx);
+	if (qidx >= MAX_RCV_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_RCV_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+	ret = nicvf_vf_start_rx_queue(dev, nic, qidx);
 	if (ret)
 		return ret;
 
@@ -1087,7 +1105,14 @@ nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
 static int
 nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
 {
-	return nicvf_start_tx_queue(dev, qidx);
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+	return nicvf_vf_start_tx_queue(dev, nic, qidx);
 }
 
 static int
@@ -1274,25 +1299,25 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
 }
 
 static int
-nicvf_dev_start(struct rte_eth_dev *dev)
+nicvf_vf_start(struct rte_eth_dev *dev, struct nicvf *nic, uint32_t rbdrsz)
 {
 	int ret;
 	uint16_t qidx;
-	uint32_t buffsz = 0, rbdrsz = 0;
 	uint32_t total_rxq_desc, nb_rbdr_desc, exp_buffs;
 	uint64_t mbuf_phys_off = 0;
 	struct nicvf_rxq *rxq;
-	struct rte_pktmbuf_pool_private *mbp_priv;
 	struct rte_mbuf *mbuf;
-	struct nicvf *nic = nicvf_pmd_priv(dev);
-	struct rte_eth_rxmode *rx_conf = &dev->data->dev_conf.rxmode;
-	uint16_t mtu;
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
 
 	PMD_INIT_FUNC_TRACE();
 
 	/* Userspace process exited without proper shutdown in last run */
 	if (nicvf_qset_rbdr_active(nic, 0))
-		nicvf_dev_stop(dev);
+		nicvf_vf_stop(dev, nic, false);
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/*
 	 * Thunderx nicvf PMD can support more than one pool per port only when
@@ -1307,32 +1332,15 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	 *
 	 */
 
-	/* Validate RBDR buff size */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
-		rxq = dev->data->rx_queues[qidx];
-		mbp_priv = rte_mempool_get_priv(rxq->pool);
-		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
-		if (buffsz % 128) {
-			PMD_INIT_LOG(ERR, "rxbuf size must be multiply of 128");
-			return -EINVAL;
-		}
-		if (rbdrsz == 0)
-			rbdrsz = buffsz;
-		if (rbdrsz != buffsz) {
-			PMD_INIT_LOG(ERR, "buffsz not same, qid=%d (%d/%d)",
-				     qidx, rbdrsz, buffsz);
-			return -EINVAL;
-		}
-	}
-
 	/* Validate mempool attributes */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		rxq->mbuf_phys_off = nicvf_mempool_phy_offset(rxq->pool);
 		mbuf = rte_pktmbuf_alloc(rxq->pool);
 		if (mbuf == NULL) {
-			PMD_INIT_LOG(ERR, "Failed allocate mbuf qid=%d pool=%s",
-				     qidx, rxq->pool->name);
+			PMD_INIT_LOG(ERR, "Failed allocate mbuf VF%d qid=%d "
+				     "pool=%s",
+				     nic->vf_id, qidx, rxq->pool->name);
 			return -ENOMEM;
 		}
 		rxq->mbuf_phys_off -= nicvf_mbuff_meta_length(mbuf);
@@ -1342,15 +1350,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		if (mbuf_phys_off == 0)
 			mbuf_phys_off = rxq->mbuf_phys_off;
 		if (mbuf_phys_off != rxq->mbuf_phys_off) {
-			PMD_INIT_LOG(ERR, "pool params not same,%s %" PRIx64,
-				     rxq->pool->name, mbuf_phys_off);
+			PMD_INIT_LOG(ERR, "pool params not same,%s VF%d %"
+				     PRIx64, rxq->pool->name, nic->vf_id,
+				     mbuf_phys_off);
 			return -EINVAL;
 		}
 	}
 
 	/* Check the level of buffers in the pool */
 	total_rxq_desc = 0;
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		/* Count total numbers of rxq descs */
 		total_rxq_desc += rxq->qlen_mask + 1;
@@ -1368,14 +1377,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	/* Check RBDR desc overflow */
 	ret = nicvf_qsize_rbdr_roundup(total_rxq_desc);
 	if (ret == 0) {
-		PMD_INIT_LOG(ERR, "Reached RBDR desc limit, reduce nr desc");
+		PMD_INIT_LOG(ERR, "Reached RBDR desc limit, reduce nr desc "
+			     "VF%d", nic->vf_id);
 		return -ENOMEM;
 	}
 
 	/* Enable qset */
 	ret = nicvf_qset_config(nic);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to enable qset %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to enable qset %d VF%d", ret,
+			     nic->vf_id);
 		return ret;
 	}
 
@@ -1383,14 +1394,16 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	nb_rbdr_desc = nicvf_qsize_rbdr_roundup(total_rxq_desc);
 	ret = nicvf_qset_rbdr_alloc(dev, nic, nb_rbdr_desc, rbdrsz);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc");
+		PMD_INIT_LOG(ERR, "Failed to allocate memory for rbdr alloc "
+			     "VF%d", nic->vf_id);
 		goto qset_reclaim;
 	}
 
 	/* Enable and configure RBDR registers */
 	ret = nicvf_qset_rbdr_config(nic, 0);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to configure rbdr %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to configure rbdr %d VF%d", ret,
+			     nic->vf_id);
 		goto qset_rbdr_free;
 	}
 
@@ -1398,52 +1411,127 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 	ret = nicvf_qset_rbdr_precharge(dev, nic, 0, rbdr_rte_mempool_get,
 					total_rxq_desc);
 	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d", ret);
+		PMD_INIT_LOG(ERR, "Failed to fill rbdr %d VF%d", ret,
+			     nic->vf_id);
 		goto qset_rbdr_reclaim;
 	}
 
-	PMD_DRV_LOG(INFO, "Filled %d out of %d entries in RBDR",
-		     nic->rbdr->tail, nb_rbdr_desc);
+	PMD_DRV_LOG(INFO, "Filled %d out of %d entries in RBDR VF%d",
+		     nic->rbdr->tail, nb_rbdr_desc, nic->vf_id);
+
+	/* Configure VLAN Strip */
+	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
+
+	/* Get queue ranges for this VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
+	/* Configure TX queues */
+	for (qidx = tx_start; qidx <= tx_end; qidx++) {
+		ret = nicvf_vf_start_tx_queue(dev, nic,
+			qidx % MAX_SND_QUEUES_PER_QS);
+		if (ret)
+			goto start_txq_error;
+	}
 
 	/* Configure RX queues */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
-		ret = nicvf_start_rx_queue(dev, qidx);
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
+		ret = nicvf_vf_start_rx_queue(dev, nic,
+			qidx % MAX_RCV_QUEUES_PER_QS);
 		if (ret)
 			goto start_rxq_error;
 	}
 
-	/* Configure VLAN Strip */
-	nicvf_vlan_hw_strip(nic, dev->data->dev_conf.rxmode.hw_vlan_strip);
-
-	/* Configure TX queues */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
-		ret = nicvf_start_tx_queue(dev, qidx);
+	if (!nic->sqs_mode) {
+		/* Configure CPI algorithm */
+		ret = nicvf_configure_cpi(dev);
 		if (ret)
 			goto start_txq_error;
+
+		ret = nicvf_mbox_get_rss_size(nic);
+		if (ret) {
+			PMD_INIT_LOG(ERR, "Failed to get rss table size");
+			goto qset_rss_error;
+		}
+
+		/* Configure RSS */
+		ret = nicvf_configure_rss(dev);
+		if (ret)
+			goto qset_rss_error;
 	}
 
-	/* Configure CPI algorithm */
-	ret = nicvf_configure_cpi(dev);
-	if (ret)
-		goto start_txq_error;
+	/* Done; Let PF make the BGX's RX and TX switches to ON position */
+	nicvf_mbox_cfg_done(nic);
+	return 0;
 
-	/* Configure RSS */
-	ret = nicvf_configure_rss(dev);
-	if (ret)
-		goto qset_rss_error;
+qset_rss_error:
+	nicvf_rss_term(nic);
+start_rxq_error:
+	for (qidx = rx_start; qidx <= rx_end; qidx++)
+		nicvf_vf_stop_rx_queue(dev, nic, qidx % MAX_RCV_QUEUES_PER_QS);
+start_txq_error:
+	for (qidx = tx_start; qidx <= tx_end; qidx++)
+		nicvf_vf_stop_tx_queue(dev, nic, qidx % MAX_SND_QUEUES_PER_QS);
+qset_rbdr_reclaim:
+	nicvf_qset_rbdr_reclaim(nic, 0);
+	nicvf_rbdr_release_mbufs(dev, nic);
+qset_rbdr_free:
+	if (nic->rbdr) {
+		rte_free(nic->rbdr);
+		nic->rbdr = NULL;
+	}
+qset_reclaim:
+	nicvf_qset_reclaim(nic);
+	return ret;
+}
+
+static int
+nicvf_dev_start(struct rte_eth_dev *dev)
+{
+	uint16_t qidx;
+	int ret;
+	size_t i;
+	struct nicvf *nic = nicvf_pmd_priv(dev);
+	struct rte_eth_rxmode *rx_conf = &dev->data->dev_conf.rxmode;
+	uint16_t mtu;
+	uint32_t buffsz = 0, rbdrsz = 0;
+	struct rte_pktmbuf_pool_private *mbp_priv;
+	struct nicvf_rxq *rxq;
+
+	PMD_INIT_FUNC_TRACE();
+
+	/* This function must be called for a primary device */
+	assert_primary(nic);
+
+	/* Validate RBDR buff size */
+	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+		rxq = dev->data->rx_queues[qidx];
+		mbp_priv = rte_mempool_get_priv(rxq->pool);
+		buffsz = mbp_priv->mbuf_data_room_size - RTE_PKTMBUF_HEADROOM;
+		if (buffsz % 128) {
+			PMD_INIT_LOG(ERR, "rxbuf size must be multiply of 128");
+			return -EINVAL;
+		}
+		if (rbdrsz == 0)
+			rbdrsz = buffsz;
+		if (rbdrsz != buffsz) {
+			PMD_INIT_LOG(ERR, "buffsz not same, qidx=%d (%d/%d)",
+				     qidx, rbdrsz, buffsz);
+			return -EINVAL;
+		}
+	}
 
 	/* Configure loopback */
 	ret = nicvf_loopback_config(nic, dev->data->dev_conf.lpbk_mode);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to configure loopback %d", ret);
-		goto qset_rss_error;
+		return ret;
 	}
 
 	/* Reset all statistics counters attached to this port */
 	ret = nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, 0xFFFF, 0xFFFF);
 	if (ret) {
 		PMD_INIT_LOG(ERR, "Failed to reset stat counters %d", ret);
-		goto qset_rss_error;
+		return ret;
 	}
 
 	/* Setup scatter mode if needed by jumbo */
@@ -1464,33 +1552,23 @@ nicvf_dev_start(struct rte_eth_dev *dev)
 		return -EBUSY;
 	}
 
+	ret = nicvf_vf_start(dev, nic, rbdrsz);
+	if (ret != 0)
+		return ret;
+
+	for (i = 0; i < nic->sqs_count; i++) {
+		assert(nic->snicvf[i]);
+
+		ret = nicvf_vf_start(dev, nic->snicvf[i], rbdrsz);
+		if (ret != 0)
+			return ret;
+	}
+
 	/* Configure callbacks based on scatter mode */
 	nicvf_set_tx_function(dev);
 	nicvf_set_rx_function(dev);
 
-	/* Done; Let PF make the BGX's RX and TX switches to ON position */
-	nicvf_mbox_cfg_done(nic);
 	return 0;
-
-qset_rss_error:
-	nicvf_rss_term(nic);
-start_txq_error:
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++)
-		nicvf_vf_stop_tx_queue(dev, nic, qidx);
-start_rxq_error:
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++)
-		nicvf_vf_stop_rx_queue(dev, nic, qidx);
-qset_rbdr_reclaim:
-	nicvf_qset_rbdr_reclaim(nic, 0);
-	nicvf_rbdr_release_mbufs(dev, nic);
-qset_rbdr_free:
-	if (nic->rbdr) {
-		rte_free(nic->rbdr);
-		nic->rbdr = NULL;
-	}
-qset_reclaim:
-	nicvf_qset_reclaim(nic);
-	return ret;
 }
 
 static void
-- 
1.9.1

[PATCH v2 11/15] net/thunderx: add secondary qset support in device configure

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 65 +++++++++++++++++++++++++++++++++----
 1 file changed, 59 insertions(+), 6 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index f7d5188..5f12b91 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -163,6 +163,7 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
 {
 	struct nicvf *nic = nicvf_pmd_priv(dev);
 	uint32_t buffsz, frame_size = mtu + ETHER_HDR_LEN + ETHER_CRC_LEN;
+	size_t i;
 
 	PMD_INIT_FUNC_TRACE();
 
@@ -198,6 +199,10 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
 	/* Update max frame size */
 	dev->data->dev_conf.rxmode.max_rx_pkt_len = (uint32_t)frame_size;
 	nic->mtu = mtu;
+
+	for (i = 0; i < nic->sqs_count; i++)
+		nic->snicvf[i]->mtu = mtu;
+
 	return 0;
 }
 
@@ -507,7 +512,8 @@ nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	const struct rte_memzone *rz;
 	uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+	rz = rte_eth_dma_zone_reserve(dev, "cq_ring",
+				      nicvf_netdev_qidx(nic, qidx), ring_size,
 				      NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
@@ -530,8 +536,9 @@ nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	const struct rte_memzone *rz;
 	uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
 
-	rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
-				NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "sq",
+				      nicvf_netdev_qidx(nic, qidx), ring_size,
+				      NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
 		return -ENOMEM;
@@ -563,8 +570,9 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 	}
 
 	ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
-	rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
-				   NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
+	rz = rte_eth_dma_zone_reserve(dev, "rbdr",
+				      nicvf_netdev_qidx(nic, 0), ring_size,
+				      NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
 	if (rz == NULL) {
 		PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
 		return -ENOMEM;
@@ -1685,12 +1693,37 @@ nicvf_dev_close(struct rte_eth_dev *dev)
 }
 
 static int
+nicvf_request_sqs(struct nicvf *nic)
+{
+	size_t i;
+
+	assert_primary(nic);
+	assert(nic->sqs_count > 0);
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	/* Set no of Rx/Tx queues in each of the SQsets */
+	for (i = 0; i < nic->sqs_count; i++) {
+		if (nicvf_svf_empty())
+			rte_panic("Cannot assign sufficient number of "
+				  "secondary queues to primary VF%" PRIu8 "\n",
+				  nic->vf_id);
+
+		nic->snicvf[i] = nicvf_svf_pop();
+		nic->snicvf[i]->sqs_id = i;
+	}
+
+	return nicvf_mbox_request_sqs(nic);
+}
+
+static int
 nicvf_dev_configure(struct rte_eth_dev *dev)
 {
-	struct rte_eth_conf *conf = &dev->data->dev_conf;
+	struct rte_eth_dev_data *data = dev->data;
+	struct rte_eth_conf *conf = &data->dev_conf;
 	struct rte_eth_rxmode *rxmode = &conf->rxmode;
 	struct rte_eth_txmode *txmode = &conf->txmode;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint8_t cqcount;
 
 	PMD_INIT_FUNC_TRACE();
 
@@ -1755,6 +1788,26 @@ nicvf_dev_configure(struct rte_eth_dev *dev)
 		return -EINVAL;
 	}
 
+	assert_primary(nic);
+	NICVF_STATIC_ASSERT(MAX_RCV_QUEUES_PER_QS == MAX_SND_QUEUES_PER_QS);
+	cqcount = RTE_MAX(data->nb_tx_queues, data->nb_rx_queues);
+	if (cqcount > MAX_RCV_QUEUES_PER_QS) {
+		nic->sqs_count = RTE_ALIGN_CEIL(cqcount, MAX_RCV_QUEUES_PER_QS);
+		nic->sqs_count = (nic->sqs_count / MAX_RCV_QUEUES_PER_QS) - 1;
+	} else {
+		nic->sqs_count = 0;
+	}
+
+	assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+	if (nic->sqs_count > 0) {
+		if (nicvf_request_sqs(nic)) {
+			rte_panic("Cannot assign sufficient number of "
+				  "secondary queues to PORT%d VF%" PRIu8 "\n",
+				  dev->data->port_id, nic->vf_id);
+		}
+	}
+
 	PMD_INIT_LOG(DEBUG, "Configured ethdev port%d hwcap=0x%" PRIx64,
 		dev->data->port_id, nicvf_hw_cap(nic));
 
-- 
1.9.1

[PATCH v2 12/15] net/thunderx: add final bits for secondary queue support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Summary:
 - add secondary qset support in device stats
 - add support for releasing mbufs from RBDR for >8 queues
 - add support for releasing mbufs from RX queues for >8 queues
 - support >8 queues in tx_queue_setup
 - support >8 queues in rx_queue_setup
 - support up to 96 queues per device (dev_info->max_rx_queues)
 - add secondary qset support in rbdr_rte_mempool_get
 - support >8 queues in multiprocess mode (do not reconfigure VFs)
 - setup periodic alarm accordingly for type of VFs:
   * primary VF   - handle events on queues and link status
   * secondary VF - handle events on queues
 - initialize hardware capabilities in secondary qsets

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.c | 200 ++++++++++++++++++++++++++++--------
 1 file changed, 157 insertions(+), 43 deletions(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 5f12b91..9dde75d 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -118,7 +118,7 @@ nicvf_interrupt(void *arg)
 				nicvf_interrupt, dev);
 }
 
-static void __rte_unused
+static void
 nicvf_vf_interrupt(void *arg)
 {
 	struct nicvf *nic = arg;
@@ -236,9 +236,15 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	struct nicvf_hw_tx_qstats tx_qstats;
 	struct nicvf_hw_stats port_stats;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
+	size_t i;
+
+	/* RX queue indices for the first VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
 
 	/* Reading per RX ring stats */
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
 			break;
 
@@ -247,8 +253,11 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 		stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
 	}
 
+	/* TX queue indices for the first VF */
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
 	/* Reading per TX ring stats */
-	for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
+	for (qidx = tx_start; qidx <= tx_end; qidx++) {
 		if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
 			break;
 
@@ -257,6 +266,40 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 		stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
 	}
 
+	for (i = 0; i < nic->sqs_count; i++) {
+		struct nicvf *snic = nic->snicvf[i];
+
+		if (snic == NULL)
+			break;
+
+		/* RX queue indices for a secondary VF */
+		nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+
+		/* Reading per RX ring stats */
+		for (qidx = rx_start; qidx <= rx_end; qidx++) {
+			if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+				break;
+
+			nicvf_hw_get_rx_qstats(snic, &rx_qstats,
+					       qidx % MAX_RCV_QUEUES_PER_QS);
+			stats->q_ibytes[qidx] = rx_qstats.q_rx_bytes;
+			stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
+		}
+
+		/* TX queue indices for a secondary VF */
+		nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+		/* Reading per TX ring stats */
+		for (qidx = tx_start; qidx <= tx_end; qidx++) {
+			if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+				break;
+
+			nicvf_hw_get_tx_qstats(snic, &tx_qstats,
+					       qidx % MAX_SND_QUEUES_PER_QS);
+			stats->q_obytes[qidx] = tx_qstats.q_tx_bytes;
+			stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
+		}
+	}
+
 	nicvf_hw_get_stats(nic, &port_stats);
 	stats->ibytes = port_stats.rx_bytes;
 	stats->ipackets = port_stats.rx_ucast_frames;
@@ -322,13 +365,36 @@ nicvf_dev_stats_reset(struct rte_eth_dev *dev)
 	int i;
 	uint16_t rxqs = 0, txqs = 0;
 	struct nicvf *nic = nicvf_pmd_priv(dev);
+	uint16_t rx_start, rx_end;
+	uint16_t tx_start, tx_end;
 
-	for (i = 0; i < dev->data->nb_rx_queues; i++)
+	/* Reset all primary nic counters */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+	for (i = rx_start; i <= rx_end; i++)
 		rxqs |= (0x3 << (i * 2));
-	for (i = 0; i < dev->data->nb_tx_queues; i++)
+
+	nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+	for (i = tx_start; i <= tx_end; i++)
 		txqs |= (0x3 << (i * 2));
 
 	nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, rxqs, txqs);
+
+	/* Reset secondary nic queue counters */
+	for (i = 0; i < nic->sqs_count; i++) {
+		struct nicvf *snic = nic->snicvf[i];
+		if (snic == NULL)
+			break;
+
+		nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+		for (i = rx_start; i <= rx_end; i++)
+			rxqs |= (0x3 << ((i % MAX_CMP_QUEUES_PER_QS) * 2));
+
+		nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+		for (i = tx_start; i <= tx_end; i++)
+			txqs |= (0x3 << ((i % MAX_SND_QUEUES_PER_QS) * 2));
+
+		nicvf_mbox_reset_stat_counters(snic, 0, 0, rxqs, txqs);
+	}
 }
 
 /* Promiscuous mode enabled by default in LMAC to VF 1:1 map configuration */
@@ -596,14 +662,18 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
 }
 
 static void
-nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic __rte_unused,
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic,
 			nicvf_phys_addr_t phy)
 {
 	uint16_t qidx;
 	void *obj;
 	struct nicvf_rxq *rxq;
+	uint16_t rx_start, rx_end;
 
-	for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = dev->data->rx_queues[qidx];
 		if (rxq->precharge_cnt) {
 			obj = (void *)nicvf_mbuff_phy2virt(phy,
@@ -861,6 +931,11 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (qidx >= MAX_SND_QUEUES_PER_QS)
+		nic = nic->snicvf[qidx / MAX_SND_QUEUES_PER_QS - 1];
+
+	qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
 	/* Socket id check */
 	if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node)
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
@@ -895,18 +970,20 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	}
 
 	/* Free memory prior to re-allocation if needed. */
-	if (dev->data->tx_queues[qidx] != NULL) {
+	if (dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) {
 		PMD_TX_LOG(DEBUG, "Freeing memory prior to re-allocation %d",
-				qidx);
-		nicvf_dev_tx_queue_release(dev->data->tx_queues[qidx]);
-		dev->data->tx_queues[qidx] = NULL;
+				nicvf_netdev_qidx(nic, qidx));
+		nicvf_dev_tx_queue_release(
+			dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)]);
+		dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL;
 	}
 
 	/* Allocating tx queue data structure */
 	txq = rte_zmalloc_socket("ethdev TX queue", sizeof(struct nicvf_txq),
 					RTE_CACHE_LINE_SIZE, nic->node);
 	if (txq == NULL) {
-		PMD_INIT_LOG(ERR, "Failed to allocate txq=%d", qidx);
+		PMD_INIT_LOG(ERR, "Failed to allocate txq=%d",
+			     nicvf_netdev_qidx(nic, qidx));
 		return -ENOMEM;
 	}
 
@@ -949,10 +1026,12 @@ nicvf_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	nicvf_tx_queue_reset(txq);
 
 	PMD_TX_LOG(DEBUG, "[%d] txq=%p nb_desc=%d desc=%p phys=0x%" PRIx64,
-			qidx, txq, nb_desc, txq->desc, txq->phys);
+			nicvf_netdev_qidx(nic, qidx), txq, nb_desc, txq->desc,
+			txq->phys);
 
-	dev->data->tx_queues[qidx] = txq;
-	dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)] = txq;
+	dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return 0;
 }
 
@@ -967,7 +1046,8 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 	if (dev->rx_pkt_burst == NULL)
 		return;
 
-	while ((rxq_cnt = nicvf_dev_rx_queue_count(dev, rxq->queue_id))) {
+	while ((rxq_cnt = nicvf_dev_rx_queue_count(dev,
+				nicvf_netdev_qidx(rxq->nic, rxq->queue_id)))) {
 		nb_pkts = dev->rx_pkt_burst(rxq, rx_pkts,
 					NICVF_MAX_RX_FREE_THRESH);
 		PMD_DRV_LOG(INFO, "nb_pkts=%d  rxq_cnt=%d", nb_pkts, rxq_cnt);
@@ -977,7 +1057,10 @@ nicvf_rx_queue_release_mbufs(struct rte_eth_dev *dev, struct nicvf_rxq *rxq)
 		}
 	}
 
-	refill_cnt += nicvf_dev_rbdr_refill(dev, rxq->queue_id);
+
+	refill_cnt += nicvf_dev_rbdr_refill(dev,
+			nicvf_netdev_qidx(rxq->nic, rxq->queue_id));
+
 	PMD_DRV_LOG(INFO, "free_cnt=%d  refill_cnt=%d",
 		    released_pkts, refill_cnt);
 }
@@ -1149,6 +1232,11 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (qidx >= MAX_RCV_QUEUES_PER_QS)
+		nic = nic->snicvf[qidx / MAX_RCV_QUEUES_PER_QS - 1];
+
+	qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
 	/* Socket id check */
 	if (socket_id != (unsigned int)SOCKET_ID_ANY && socket_id != nic->node)
 		PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
@@ -1191,18 +1279,20 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	}
 
 	/* Free memory prior to re-allocation if needed */
-	if (dev->data->rx_queues[qidx] != NULL) {
+	if (dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] != NULL) {
 		PMD_RX_LOG(DEBUG, "Freeing memory prior to re-allocation %d",
-				qidx);
-		nicvf_dev_rx_queue_release(dev->data->rx_queues[qidx]);
-		dev->data->rx_queues[qidx] = NULL;
+				nicvf_netdev_qidx(nic, qidx));
+		nicvf_dev_rx_queue_release(
+			dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)]);
+		dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = NULL;
 	}
 
 	/* Allocate rxq memory */
 	rxq = rte_zmalloc_socket("ethdev rx queue", sizeof(struct nicvf_rxq),
 					RTE_CACHE_LINE_SIZE, nic->node);
 	if (rxq == NULL) {
-		PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d", qidx);
+		PMD_INIT_LOG(ERR, "Failed to allocate rxq=%d",
+			     nicvf_netdev_qidx(nic, qidx));
 		return -ENOMEM;
 	}
 
@@ -1232,11 +1322,12 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
 	nicvf_rx_queue_reset(rxq);
 
 	PMD_RX_LOG(DEBUG, "[%d] rxq=%p pool=%s nb_desc=(%d/%d) phy=%" PRIx64,
-			qidx, rxq, mp->name, nb_desc,
+			nicvf_netdev_qidx(nic, qidx), rxq, mp->name, nb_desc,
 			rte_mempool_avail_count(mp), rxq->phys);
 
-	dev->data->rx_queues[qidx] = rxq;
-	dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+	dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)] = rxq;
+	dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+		RTE_ETH_QUEUE_STATE_STOPPED;
 	return 0;
 }
 
@@ -1249,8 +1340,10 @@ nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 
 	dev_info->min_rx_bufsize = ETHER_MIN_MTU;
 	dev_info->max_rx_pktlen = NIC_HW_MAX_FRS;
-	dev_info->max_rx_queues = (uint16_t)MAX_RCV_QUEUES_PER_QS;
-	dev_info->max_tx_queues = (uint16_t)MAX_SND_QUEUES_PER_QS;
+	dev_info->max_rx_queues =
+			(uint16_t)MAX_RCV_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1);
+	dev_info->max_tx_queues =
+			(uint16_t)MAX_SND_QUEUES_PER_QS * (MAX_SQS_PER_VF + 1);
 	dev_info->max_mac_addrs = 1;
 	dev_info->max_vfs = dev->pci_dev->max_vfs;
 
@@ -1291,9 +1384,13 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
 	uintptr_t mbuf;
 	struct nicvf_rxq *rxq;
 	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)dev;
-	struct nicvf *nic __rte_unused = (struct nicvf *)opaque;
+	struct nicvf *nic = (struct nicvf *)opaque;
+	uint16_t rx_start, rx_end;
+
+	/* Get queue ranges for this VF */
+	nicvf_rx_range(eth_dev, nic, &rx_start, &rx_end);
 
-	for (qidx = 0; qidx < eth_dev->data->nb_rx_queues; qidx++) {
+	for (qidx = rx_start; qidx <= rx_end; qidx++) {
 		rxq = eth_dev->data->rx_queues[qidx];
 		/* Maintain equal buffer count across all pools */
 		if (rxq->precharge_cnt >= rxq->qlen_mask)
@@ -1856,10 +1953,16 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 
 	/* For secondary processes, the primary has done all the work */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
-		/* Setup callbacks for secondary process */
-		nicvf_set_tx_function(eth_dev);
-		nicvf_set_rx_function(eth_dev);
-		return 0;
+		if (nic) {
+			/* Setup callbacks for secondary process */
+			nicvf_set_tx_function(eth_dev);
+			nicvf_set_rx_function(eth_dev);
+			return 0;
+		} else {
+			/* If nic == NULL than it is secondary function
+			 * so ethdev need to be released by caller */
+			return ENOTSUP;
+		}
 	}
 
 	pci_dev = eth_dev->pci_dev;
@@ -1904,11 +2007,28 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 			);
 	}
 
+	ret = nicvf_base_init(nic);
+	if (ret) {
+		PMD_INIT_LOG(ERR, "Failed to execute nicvf_base_init");
+		goto malloc_fail;
+	}
+
 	if (nic->sqs_mode) {
-		PMD_INIT_LOG(INFO, "Unsupported SQS VF detected, Detaching...");
-		/* Detach port by returning Positive error number */
-		ret = ENOTSUP;
-		goto alarm_fail;
+		/* Push nic to stack of secondary vfs */
+		nicvf_svf_push(nic);
+
+		/* Steal nic pointer from the device for further reuse */
+		eth_dev->data->dev_private = NULL;
+
+		nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
+		ret = nicvf_periodic_alarm_start(nicvf_vf_interrupt, nic);
+		if (ret) {
+			PMD_INIT_LOG(ERR, "Failed to start period alarm");
+			goto fail;
+		}
+
+		/* Detach port by returning postive error number */
+		return ENOTSUP;
 	}
 
 	eth_dev->data->mac_addrs = rte_zmalloc("mac_addr", ETHER_ADDR_LEN, 0);
@@ -1929,12 +2049,6 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
 		goto malloc_fail;
 	}
 
-	ret = nicvf_base_init(nic);
-	if (ret) {
-		PMD_INIT_LOG(ERR, "Failed to execute nicvf_base_init");
-		goto malloc_fail;
-	}
-
 	PMD_INIT_LOG(INFO, "Port %d (%x:%x) mac=%02x:%02x:%02x:%02x:%02x:%02x",
 		eth_dev->data->port_id, nic->vendor_id, nic->device_id,
 		nic->mac_addr[0], nic->mac_addr[1], nic->mac_addr[2],
-- 
1.9.1

[PATCH v2 13/15] net/thunderx: document secondary queue set support

[Kamil Rytarowski]

From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 doc/guides/nics/thunderx.rst | 148 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 124 insertions(+), 24 deletions(-)

diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst
index acc05a4..9763bb6 100644
--- a/doc/guides/nics/thunderx.rst
+++ b/doc/guides/nics/thunderx.rst
@@ -56,6 +56,7 @@ Features of the ThunderX PMD are:
 - VLAN stripping
 - SR-IOV VF
 - NUMA support
+- Multi queue set support (up to 96 queues (12 queue sets)) per port
 
 Supported ThunderX SoCs
 -----------------------
@@ -207,13 +208,13 @@ This section provides instructions to configure SR-IOV with Linux OS.
    .. code-block:: console
 
       0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01)
-              ...
-              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
-              ...
-              Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
-              ...
-              Kernel driver in use: thunder-nic
-              ...
+      ...
+      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
+      ...
+      Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
+      ...
+      Kernel driver in use: thunder-nic
+      ...
 
    .. note::
 
@@ -230,18 +231,18 @@ This section provides instructions to configure SR-IOV with Linux OS.
    .. code-block:: console
 
       0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01)
-              ...
-              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
-              ...
-              Kernel driver in use: thunder-nicvf
-              ...
+      ...
+      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
+      ...
+      Kernel driver in use: thunder-nicvf
+      ...
 
       0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01)
-              ...
-              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
-              ...
-              Kernel driver in use: thunder-nicvf
-              ...
+      ...
+      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
+      ...
+      Kernel driver in use: thunder-nicvf
+      ...
 
    .. note::
 
@@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS.
 #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
    how to launch ``testpmd`` application.
 
+Multiple Queue Set per DPDK port configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two types of VFs:
+
+- Primary VF
+- Secondary VF
+
+Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
+When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF.
+Each secondary VF adds 8 additional queues to the queue set.
+
+During PMD driver initialization, the primary VF's are enumerated by checking the
+specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
+They are at the beginning of VF list (the remain ones are secondary VF's).
+
+The primary VFs are used as master queue sets. Secondary VFs provide
+additional queue sets for primary ones. If a port is configured for more then
+8 queues than it will request for additional queues from secondary VFs.
+
+Secondary VFs cannot be shared between primary VFs.
+
+Primary VFs are present on the beginning of the 'Network devices using kernel
+driver' list, secondary VFs are on the remaining on the remaining part of the list.
+
+   .. note::
+
+      The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
+      We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility.
+
+   .. note::
+
+      Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
+      this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
+
+
+Example device binding
+~~~~~~~~~~~~~~~~~~~~~~
+
+If a system has three interfaces, a total of 18 VF devices will be created
+on a non-NUMA machine.
+
+   .. note::
+
+      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
+
+   .. code-block:: console
+
+      # tools/dpdk-devbind.py --status
+
+      Network devices using DPDK-compatible driver
+      ============================================
+      <none>
+
+      Network devices using kernel driver
+      ===================================
+      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
+      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+
+      Other network devices
+      =====================
+      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
+
+
+We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
+and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
+We will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
+
+
+#. Bind two primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
+
+#. Bind four primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
+
+The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
+
 Limitations
 -----------
 
@@ -346,10 +453,3 @@ Maximum packet segments
 The ThunderX SoC family NICs support up to 12 segments per packet when working
 in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
 frame size does not fit in the maximum number of segments.
-
-Limited VFs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues
-for RX/TX respectively. Current driver implementation has one to one mapping
-between physical port and VF hence only limited VFs can be used.
-- 
1.9.1

[PATCH v2 14/15] ethdev: Support VFs on the different PCI domains

[Kamil Rytarowski]

It's possible to have the same numbers for bus, device id and function,
therefore we need to differentiate on domain.

This enables DPDK with multiple VFs on ThunderX 2-socket hardware.

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 lib/librte_ether/rte_ethdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 382c959..01d5fb0 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t size,
 {
 	int ret;
 
-	ret = snprintf(name, size, "%d:%d.%d",
+	ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
 			pci_dev->addr.bus, pci_dev->addr.devid,
 			pci_dev->addr.function);
 	if (ret < 0)
-- 
1.9.1

[PATCH v2 15/15] net/thunderx: Bump driver version to 2.0

[Kamil Rytarowski]

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 drivers/net/thunderx/nicvf_ethdev.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/thunderx/nicvf_ethdev.h b/drivers/net/thunderx/nicvf_ethdev.h
index e162364..a74219f 100644
--- a/drivers/net/thunderx/nicvf_ethdev.h
+++ b/drivers/net/thunderx/nicvf_ethdev.h
@@ -35,7 +35,7 @@
 
 #include <rte_ethdev.h>
 
-#define THUNDERX_NICVF_PMD_VERSION      "1.0"
+#define THUNDERX_NICVF_PMD_VERSION      "2.0"
 #define THUNDERX_REG_BYTES		8
 
 #define NICVF_INTR_POLL_INTERVAL_MS	50
-- 
1.9.1

Re: [PATCH v2 13/15] net/thunderx: document secondary queue set support

[Mcnamara, John]

> -----Original Message-----
> From: Kamil Rytarowski [mailto:krytarowski@caviumnetworks.com]
> Sent: Friday, September 30, 2016 1:06 PM
> To: dev@dpdk.org
> Cc: maciej.czekaj@caviumnetworks.com; zyta.szpak@semihalf.com;
> slawomir.rosek@semihalf.com; rad@semihalf.com;
> jerin.jacob@caviumnetworks.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> Mcnamara, John <john.mcnamara@intel.com>; Kamil Rytarowski
> <kamil.rytarowski@caviumnetworks.com>
> Subject: [PATCH v2 13/15] net/thunderx: document secondary queue set
> support
> 
> From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains

[Ferruh Yigit]

Hi Kamil,

On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
> It's possible to have the same numbers for bus, device id and function,
> therefore we need to differentiate on domain.
> 
> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
> 
> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
>  lib/librte_ether/rte_ethdev.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
> index 382c959..01d5fb0 100644
> --- a/lib/librte_ether/rte_ethdev.c
> +++ b/lib/librte_ether/rte_ethdev.c
> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t size,
>  {
>  	int ret;
>  
> -	ret = snprintf(name, size, "%d:%d.%d",
> +	ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>  			pci_dev->addr.bus, pci_dev->addr.devid,
>  			pci_dev->addr.function);
>  	if (ret < 0)
> 

Is it possible to separate this patch from patchset, this is a ethdev
patch and it seems not directly related to the rest of the patchset?

Thanks,
ferruh

Re: [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains

[Kamil Rytarowski]

W dniu 10.10.2016 o 12:19, Ferruh Yigit pisze:
> Hi Kamil,
>
> On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
>> It's possible to have the same numbers for bus, device id and function,
>> therefore we need to differentiate on domain.
>>
>> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
>>
>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> ---
>>   lib/librte_ether/rte_ethdev.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>> index 382c959..01d5fb0 100644
>> --- a/lib/librte_ether/rte_ethdev.c
>> +++ b/lib/librte_ether/rte_ethdev.c
>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t size,
>>   {
>>   	int ret;
>>   
>> -	ret = snprintf(name, size, "%d:%d.%d",
>> +	ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>>   			pci_dev->addr.bus, pci_dev->addr.devid,
>>   			pci_dev->addr.function);
>>   	if (ret < 0)
>>
> Is it possible to separate this patch from patchset, this is a ethdev
> patch and it seems not directly related to the rest of the patchset?
>
> Thanks,
> ferruh

This patch is directly related with secondary queue set support on 
ThunderX, but it can be skipped in this chain of patches and applied as 
a standalone diff.

Is disabling this one on patch work sufficient? Of course unless there 
are no more comments to produce v3 of the original patch chain "Add 
support for secondary queue set in nicvf thunderx driver".

Should I resubmit it as a new standalone patch?

Re: [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains

[Ferruh Yigit]

On 10/10/2016 2:01 PM, Kamil Rytarowski wrote:
> 
> 
> W dniu 10.10.2016 o 12:19, Ferruh Yigit pisze:
>> Hi Kamil,
>>
>> On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
>>> It's possible to have the same numbers for bus, device id and function,
>>> therefore we need to differentiate on domain.
>>>
>>> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
>>>
>>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>>> ---
>>>   lib/librte_ether/rte_ethdev.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>>> index 382c959..01d5fb0 100644
>>> --- a/lib/librte_ether/rte_ethdev.c
>>> +++ b/lib/librte_ether/rte_ethdev.c
>>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t size,
>>>   {
>>>   	int ret;
>>>   
>>> -	ret = snprintf(name, size, "%d:%d.%d",
>>> +	ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>>>   			pci_dev->addr.bus, pci_dev->addr.devid,
>>>   			pci_dev->addr.function);
>>>   	if (ret < 0)
>>>
>> Is it possible to separate this patch from patchset, this is a ethdev
>> patch and it seems not directly related to the rest of the patchset?
>>
>> Thanks,
>> ferruh
> 
> This patch is directly related with secondary queue set support on 
> ThunderX, but it can be skipped in this chain of patches and applied as 
> a standalone diff.
> 
> Is disabling this one on patch work sufficient? Of course unless there 
> are no more comments to produce v3 of the original patch chain "Add 
> support for secondary queue set in nicvf thunderx driver".

I think it is sufficient, at least I don't have any more comment for
rest of the patchset and it looks good to me.

> 
> Should I resubmit it as a new standalone patch?

Can you please resubmit just this one patch, so it can be properly reviewed.

Thanks,
ferruh

Re: [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains

[Kamil Rytarowski]

W dniu 10.10.2016 o 15:27, Ferruh Yigit pisze:
> On 10/10/2016 2:01 PM, Kamil Rytarowski wrote:
>>
>> W dniu 10.10.2016 o 12:19, Ferruh Yigit pisze:
>>> Hi Kamil,
>>>
>>> On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
>>>> It's possible to have the same numbers for bus, device id and function,
>>>> therefore we need to differentiate on domain.
>>>>
>>>> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
>>>>
>>>> Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
>>>> Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
>>>> Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
>>>> Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
>>>> Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
>>>> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>>>> ---
>>>>    lib/librte_ether/rte_ethdev.c | 2 +-
>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>>>> index 382c959..01d5fb0 100644
>>>> --- a/lib/librte_ether/rte_ethdev.c
>>>> +++ b/lib/librte_ether/rte_ethdev.c
>>>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t size,
>>>>    {
>>>>    	int ret;
>>>>    
>>>> -	ret = snprintf(name, size, "%d:%d.%d",
>>>> +	ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>>>>    			pci_dev->addr.bus, pci_dev->addr.devid,
>>>>    			pci_dev->addr.function);
>>>>    	if (ret < 0)
>>>>
>>> Is it possible to separate this patch from patchset, this is a ethdev
>>> patch and it seems not directly related to the rest of the patchset?
>>>
>>> Thanks,
>>> ferruh
>> This patch is directly related with secondary queue set support on
>> ThunderX, but it can be skipped in this chain of patches and applied as
>> a standalone diff.
>>
>> Is disabling this one on patch work sufficient? Of course unless there
>> are no more comments to produce v3 of the original patch chain "Add
>> support for secondary queue set in nicvf thunderx driver".
> I think it is sufficient, at least I don't have any more comment for
> rest of the patchset and it looks good to me.
>
>> Should I resubmit it as a new standalone patch?
> Can you please resubmit just this one patch, so it can be properly reviewed.
>
> Thanks,
> ferruh
>

Hi,

I've performed the needed actions.

This patch should be disabled in the patch-chain and has been resent as 
a new one.

Re: [PATCH v2 00/15] Add support for secondary queue set in nicvf thunderx driver

[Bruce Richardson]

On Fri, Sep 30, 2016 at 02:05:39PM +0200, Kamil Rytarowski wrote:
> This series of patches adds support for secondary queue set in nicvf thunderx
> driver
> 
> There are two types of VFs:
>  - Primary VF
>  - Secondary VF
> 
> Each port consist of a primary VF and n secondary VF(s). Each VF provides 8
> Tx/Rx queues to a port. In case port is configured to use more than 8 queues,
> then it requires one (or more) secondary VF. Each secondary VF adds additional
> 8 queues to the queue set.
> 
> During PMD driver initialization, the primary VF's are enumerated by checking the
> specific flag (see READY message). They are at the beginning of  VF list (the remain
> ones are secondary VF's).
> 
> The primary VFs are used as master queue sets. Secondary VFs provide
> additional queue sets for primary ones. If a port is configured for more then
> 8 queues then it will request for additional queues from secondary VFs.
> 
> Secondary VFs cannot be shared between primary VFs.
> 
> Primary VFs are present on the tail of the 'Network devices using kernel
> driver' list, secondary VFs are on the remaining tail of the list.
> 
> The VNIC driver in the multiqueue setup works differently than other drivers
> like `ixgbe`. We need to bind separately each specific queue set device with
> the ``tools/dpdk-devbind.py`` utility.
> 
> Depending on the hardware used, the kernel driver sets a threshold ``vf_id``.
> VFs that try to attach with an id below or equal to this boundary are
> considered primary VFs. VFs that try to attach with an id above this boundary
> are considered secondary VFs.
> 
> This patchset also contains other cleanups and improvements like fixing
> erroneous checksum calculation and preparing the thunderx driver for the multi
> queue set feature support.
> 
> 
> These changes base on the following pending patches:
> 
> [dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver
> http://dpdk.org/dev/patchwork/patch/14963/
> 
> [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag
> http://dpdk.org/dev/patchwork/patch/14964/
> 
> [dpdk-dev,3/3] net/thunderx: add 81xx SoC support
> http://dpdk.org/dev/patchwork/patch/14965/
> 
> Kamil Rytarowski (15):
>   net/thunderx: cleanup the driver before adding new features
>   net/thunderx: correct transmit checksum handling
>   net/thunderx/base: add family of functions to store qsets
>   net/thunderx/base: add secondary queue set support
>   net/thunderx: add family of functions to store DPDK qsets
>   net/thunderx: add secondary queue set in interrupt functions
>   net/thunderx: remove problematic private_data->eth_dev link
>   net/thunderx: add helper utils for secondary qset support
>   net/thunderx: add secondary qset support in dev stop/close
>   net/thunderx: add secondary qset support in device start
>   net/thunderx: add secondary qset support in device configure
>   net/thunderx: add final bits for secondary queue support
>   net/thunderx: document secondary queue set support
>   ethdev: Support VFs on the different PCI domains
>   net/thunderx: Bump driver version to 2.0
> 
Series applied to dpdk-next-net/rel_16_11, with the exception of patch 14 which
has been resubmitted as a separate ethdev patch: 
	http://dpdk.org/dev/patchwork/patch/16480/

	/Bruce