* [PATCH net-next 00/16] nfp: ctrl vNIC
@ 2017-06-06 0:01 Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 01/16] nfp: reorder open and close functions Jakub Kicinski
` (17 more replies)
0 siblings, 18 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
Jakub Kicinski (16):
nfp: reorder open and close functions
nfp: split out the allocation part of open
nfp: reuse ring free code on close
nfp: move nfp_net_vecs_init()
nfp: prepare print macros for use without netdev
nfp: make sure debug accesses don't depend on netdevs
nfp: allow allocation and initialization of netdev-less vNICs
nfp: prepare config and enable for working without netdevs
nfp: add control vNIC datapath
nfp: make vNIC ctrl memory mapping function reusable
nfp: map all queue controllers at once
nfp: don't clutter init code passing fw_ver around
nfp: slice the netdev spawning function
nfp: allow non-equal distribution of IRQs
nfp: create control vNICs and wire up rx/tx
nfp: advertise support for NFD ABI 0.5
drivers/net/ethernet/netronome/nfp/nfp_app.c | 18 +
drivers/net/ethernet/netronome/nfp/nfp_app.h | 55 ++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 13 +-
drivers/net/ethernet/netronome/nfp/nfp_net.h | 64 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 709 +++++++++++++++------
drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h | 3 +
.../net/ethernet/netronome/nfp/nfp_net_debugfs.c | 9 +-
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 365 ++++++-----
.../net/ethernet/netronome/nfp/nfp_netvf_main.c | 4 +-
9 files changed, 872 insertions(+), 368 deletions(-)
--
2.11.0
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH net-next 01/16] nfp: reorder open and close functions
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 02/16] nfp: split out the allocation part of open Jakub Kicinski
` (16 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
We will soon reuse parts of .ndo_stop() for clean up after errors
in .ndo_open(). Reorder the associated functions to make that possible.
No functional changes.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 140 ++++++++++-----------
1 file changed, 70 insertions(+), 70 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index c3235d03b8eb..770ef28c0380 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -2276,6 +2276,76 @@ static int nfp_net_set_config_and_enable(struct nfp_net *nn)
}
/**
+ * nfp_net_close_stack() - Quiesce the stack (part of close)
+ * @nn: NFP Net device to reconfigure
+ */
+static void nfp_net_close_stack(struct nfp_net *nn)
+{
+ unsigned int r;
+
+ disable_irq(nn->irq_entries[NFP_NET_IRQ_LSC_IDX].vector);
+ netif_carrier_off(nn->dp.netdev);
+ nn->link_up = false;
+
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
+ disable_irq(nn->r_vecs[r].irq_vector);
+ napi_disable(&nn->r_vecs[r].napi);
+ }
+
+ netif_tx_disable(nn->dp.netdev);
+}
+
+/**
+ * nfp_net_close_free_all() - Free all runtime resources
+ * @nn: NFP Net device to reconfigure
+ */
+static void nfp_net_close_free_all(struct nfp_net *nn)
+{
+ unsigned int r;
+
+ for (r = 0; r < nn->dp.num_rx_rings; r++) {
+ nfp_net_rx_ring_bufs_free(&nn->dp, &nn->dp.rx_rings[r]);
+ nfp_net_rx_ring_free(&nn->dp.rx_rings[r]);
+ }
+ for (r = 0; r < nn->dp.num_tx_rings; r++) {
+ nfp_net_tx_ring_bufs_free(&nn->dp, &nn->dp.tx_rings[r]);
+ nfp_net_tx_ring_free(&nn->dp.tx_rings[r]);
+ }
+ for (r = 0; r < nn->dp.num_r_vecs; r++)
+ nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
+
+ kfree(nn->dp.rx_rings);
+ kfree(nn->dp.tx_rings);
+
+ nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
+ nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
+}
+
+/**
+ * nfp_net_netdev_close() - Called when the device is downed
+ * @netdev: netdev structure
+ */
+static int nfp_net_netdev_close(struct net_device *netdev)
+{
+ struct nfp_net *nn = netdev_priv(netdev);
+
+ /* Step 1: Disable RX and TX rings from the Linux kernel perspective
+ */
+ nfp_net_close_stack(nn);
+
+ /* Step 2: Tell NFP
+ */
+ nfp_net_clear_config_and_disable(nn);
+
+ /* Step 3: Free resources
+ */
+ nfp_net_close_free_all(nn);
+
+ nn_dbg(nn, "%s down", netdev->name);
+ return 0;
+}
+
+/**
* nfp_net_open_stack() - Start the device from stack's perspective
* @nn: NFP Net device to reconfigure
*/
@@ -2377,76 +2447,6 @@ static int nfp_net_netdev_open(struct net_device *netdev)
return err;
}
-/**
- * nfp_net_close_stack() - Quiescent the stack (part of close)
- * @nn: NFP Net device to reconfigure
- */
-static void nfp_net_close_stack(struct nfp_net *nn)
-{
- unsigned int r;
-
- disable_irq(nn->irq_entries[NFP_NET_IRQ_LSC_IDX].vector);
- netif_carrier_off(nn->dp.netdev);
- nn->link_up = false;
-
- for (r = 0; r < nn->dp.num_r_vecs; r++) {
- disable_irq(nn->r_vecs[r].irq_vector);
- napi_disable(&nn->r_vecs[r].napi);
- }
-
- netif_tx_disable(nn->dp.netdev);
-}
-
-/**
- * nfp_net_close_free_all() - Free all runtime resources
- * @nn: NFP Net device to reconfigure
- */
-static void nfp_net_close_free_all(struct nfp_net *nn)
-{
- unsigned int r;
-
- for (r = 0; r < nn->dp.num_rx_rings; r++) {
- nfp_net_rx_ring_bufs_free(&nn->dp, &nn->dp.rx_rings[r]);
- nfp_net_rx_ring_free(&nn->dp.rx_rings[r]);
- }
- for (r = 0; r < nn->dp.num_tx_rings; r++) {
- nfp_net_tx_ring_bufs_free(&nn->dp, &nn->dp.tx_rings[r]);
- nfp_net_tx_ring_free(&nn->dp.tx_rings[r]);
- }
- for (r = 0; r < nn->dp.num_r_vecs; r++)
- nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
-
- kfree(nn->dp.rx_rings);
- kfree(nn->dp.tx_rings);
-
- nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
- nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
-}
-
-/**
- * nfp_net_netdev_close() - Called when the device is downed
- * @netdev: netdev structure
- */
-static int nfp_net_netdev_close(struct net_device *netdev)
-{
- struct nfp_net *nn = netdev_priv(netdev);
-
- /* Step 1: Disable RX and TX rings from the Linux kernel perspective
- */
- nfp_net_close_stack(nn);
-
- /* Step 2: Tell NFP
- */
- nfp_net_clear_config_and_disable(nn);
-
- /* Step 3: Free resources
- */
- nfp_net_close_free_all(nn);
-
- nn_dbg(nn, "%s down", netdev->name);
- return 0;
-}
-
static void nfp_net_set_rx_mode(struct net_device *netdev)
{
struct nfp_net *nn = netdev_priv(netdev);
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 02/16] nfp: split out the allocation part of open
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 01/16] nfp: reorder open and close functions Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 03/16] nfp: reuse ring free code on close Jakub Kicinski
` (15 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Our open/close implementations have 3 stages:
- allocation/freeing of ring resources, irqs etc.,
- device config,
- device/stack enable (can't fail).
Right now all of those stages are placed in separate functions,
apart from allocation during open. Fix that. It will make it
easier for us to allocate resources for netdev-less vNICs.
Because we want to reuse allocation code in netdev-less vNICs
leave the netif_set_real_num_[rt]x_queues() calls inside open.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 57 +++++++++++++---------
1 file changed, 35 insertions(+), 22 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index 770ef28c0380..bec51f4a9299 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -2364,16 +2364,10 @@ static void nfp_net_open_stack(struct nfp_net *nn)
nfp_net_read_link_status(nn);
}
-static int nfp_net_netdev_open(struct net_device *netdev)
+static int nfp_net_open_alloc_all(struct nfp_net *nn)
{
- struct nfp_net *nn = netdev_priv(netdev);
int err, r;
- /* Step 1: Allocate resources for rings and the like
- * - Request interrupts
- * - Allocate RX and TX ring resources
- * - Setup initial RSS table
- */
err = nfp_net_aux_irq_request(nn, NFP_NET_CFG_EXN, "%s-exn",
nn->exn_name, sizeof(nn->exn_name),
NFP_NET_IRQ_EXN_IDX, nn->exn_handler);
@@ -2403,13 +2397,42 @@ static int nfp_net_netdev_open(struct net_device *netdev)
for (r = 0; r < nn->max_r_vecs; r++)
nfp_net_vector_assign_rings(&nn->dp, &nn->r_vecs[r], r);
+ return 0;
+
+err_free_rx_rings:
+ nfp_net_rx_rings_free(&nn->dp);
+err_cleanup_vec:
+ r = nn->dp.num_r_vecs;
+err_cleanup_vec_p:
+ while (r--)
+ nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
+ nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
+err_free_exn:
+ nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
+ return err;
+}
+
+static int nfp_net_netdev_open(struct net_device *netdev)
+{
+ struct nfp_net *nn = netdev_priv(netdev);
+ int err;
+
+ /* Step 1: Allocate resources for rings and the like
+ * - Request interrupts
+ * - Allocate RX and TX ring resources
+ * - Setup initial RSS table
+ */
+ err = nfp_net_open_alloc_all(nn);
+ if (err)
+ return err;
+
err = netif_set_real_num_tx_queues(netdev, nn->dp.num_stack_tx_rings);
if (err)
- goto err_free_rings;
+ goto err_free_all;
err = netif_set_real_num_rx_queues(netdev, nn->dp.num_rx_rings);
if (err)
- goto err_free_rings;
+ goto err_free_all;
/* Step 2: Configure the NFP
* - Enable rings from 0 to tx_rings/rx_rings - 1.
@@ -2420,7 +2443,7 @@ static int nfp_net_netdev_open(struct net_device *netdev)
*/
err = nfp_net_set_config_and_enable(nn);
if (err)
- goto err_free_rings;
+ goto err_free_all;
/* Step 3: Enable for kernel
* - put some freelist descriptors on each RX ring
@@ -2432,18 +2455,8 @@ static int nfp_net_netdev_open(struct net_device *netdev)
return 0;
-err_free_rings:
- nfp_net_tx_rings_free(&nn->dp);
-err_free_rx_rings:
- nfp_net_rx_rings_free(&nn->dp);
-err_cleanup_vec:
- r = nn->dp.num_r_vecs;
-err_cleanup_vec_p:
- while (r--)
- nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
- nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
-err_free_exn:
- nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
+err_free_all:
+ nfp_net_close_free_all(nn);
return err;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 03/16] nfp: reuse ring free code on close
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 01/16] nfp: reorder open and close functions Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 02/16] nfp: split out the allocation part of open Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 04/16] nfp: move nfp_net_vecs_init() Jakub Kicinski
` (14 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
On the close path reuse the ring free helpers introduced for runtime
reconfiguration.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index bec51f4a9299..23419883cfd4 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -2303,20 +2303,12 @@ static void nfp_net_close_free_all(struct nfp_net *nn)
{
unsigned int r;
- for (r = 0; r < nn->dp.num_rx_rings; r++) {
- nfp_net_rx_ring_bufs_free(&nn->dp, &nn->dp.rx_rings[r]);
- nfp_net_rx_ring_free(&nn->dp.rx_rings[r]);
- }
- for (r = 0; r < nn->dp.num_tx_rings; r++) {
- nfp_net_tx_ring_bufs_free(&nn->dp, &nn->dp.tx_rings[r]);
- nfp_net_tx_ring_free(&nn->dp.tx_rings[r]);
- }
+ nfp_net_tx_rings_free(&nn->dp);
+ nfp_net_rx_rings_free(&nn->dp);
+
for (r = 0; r < nn->dp.num_r_vecs; r++)
nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
- kfree(nn->dp.rx_rings);
- kfree(nn->dp.tx_rings);
-
nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
}
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 04/16] nfp: move nfp_net_vecs_init()
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (2 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 03/16] nfp: reuse ring free code on close Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 05/16] nfp: prepare print macros for use without netdev Jakub Kicinski
` (13 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Move nfp_net_vecs_init() after all datapath functions. We will need
to init poll() callbacks from this function soon.
No functional changes.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 54 +++++++++++-----------
1 file changed, 27 insertions(+), 27 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index 23419883cfd4..f8dba793a8fe 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -504,33 +504,6 @@ nfp_net_rx_ring_init(struct nfp_net_rx_ring *rx_ring,
}
/**
- * nfp_net_vecs_init() - Assign IRQs and setup rvecs.
- * @nn: NFP Network structure
- */
-static void nfp_net_vecs_init(struct nfp_net *nn)
-{
- struct nfp_net_r_vector *r_vec;
- int r;
-
- nn->lsc_handler = nfp_net_irq_lsc;
- nn->exn_handler = nfp_net_irq_exn;
-
- for (r = 0; r < nn->max_r_vecs; r++) {
- struct msix_entry *entry;
-
- entry = &nn->irq_entries[NFP_NET_NON_Q_VECTORS + r];
-
- r_vec = &nn->r_vecs[r];
- r_vec->nfp_net = nn;
- r_vec->handler = nfp_net_irq_rxtx;
- r_vec->irq_entry = entry->entry;
- r_vec->irq_vector = entry->vector;
-
- cpumask_set_cpu(r, &r_vec->affinity_mask);
- }
-}
-
-/**
* nfp_net_aux_irq_request() - Request an auxiliary interrupt (LSC or EXN)
* @nn: NFP Network structure
* @ctrl_offset: Control BAR offset where IRQ configuration should be written
@@ -1773,6 +1746,33 @@ static int nfp_net_poll(struct napi_struct *napi, int budget)
*/
/**
+ * nfp_net_vecs_init() - Assign IRQs and setup rvecs.
+ * @nn: NFP Network structure
+ */
+static void nfp_net_vecs_init(struct nfp_net *nn)
+{
+ struct nfp_net_r_vector *r_vec;
+ int r;
+
+ nn->lsc_handler = nfp_net_irq_lsc;
+ nn->exn_handler = nfp_net_irq_exn;
+
+ for (r = 0; r < nn->max_r_vecs; r++) {
+ struct msix_entry *entry;
+
+ entry = &nn->irq_entries[NFP_NET_NON_Q_VECTORS + r];
+
+ r_vec = &nn->r_vecs[r];
+ r_vec->nfp_net = nn;
+ r_vec->handler = nfp_net_irq_rxtx;
+ r_vec->irq_entry = entry->entry;
+ r_vec->irq_vector = entry->vector;
+
+ cpumask_set_cpu(r, &r_vec->affinity_mask);
+ }
+}
+
+/**
* nfp_net_tx_ring_free() - Free resources allocated to a TX ring
* @tx_ring: TX ring to free
*/
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 05/16] nfp: prepare print macros for use without netdev
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (3 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 04/16] nfp: move nfp_net_vecs_init() Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 06/16] nfp: make sure debug accesses don't depend on netdevs Jakub Kicinski
` (12 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
To be able to reuse print macros easily with control vNICs make the
macros check if netdev pointer is populated and use dev_* print
functions otherwise.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 33 +++++++++++++++++++++-------
1 file changed, 25 insertions(+), 8 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index 2e526338f678..b14aa31d494a 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -50,15 +50,32 @@
#include "nfp_net_ctrl.h"
-#define nn_err(nn, fmt, args...) netdev_err((nn)->dp.netdev, fmt, ## args)
-#define nn_warn(nn, fmt, args...) netdev_warn((nn)->dp.netdev, fmt, ## args)
-#define nn_info(nn, fmt, args...) netdev_info((nn)->dp.netdev, fmt, ## args)
-#define nn_dbg(nn, fmt, args...) netdev_dbg((nn)->dp.netdev, fmt, ## args)
+#define nn_pr(nn, lvl, fmt, args...) \
+ ({ \
+ struct nfp_net *__nn = (nn); \
+ \
+ if (__nn->dp.netdev) \
+ netdev_printk(lvl, __nn->dp.netdev, fmt, ## args); \
+ else \
+ dev_printk(lvl, __nn->dp.dev, "ctrl: " fmt, ## args); \
+ })
+
+#define nn_err(nn, fmt, args...) nn_pr(nn, KERN_ERR, fmt, ## args)
+#define nn_warn(nn, fmt, args...) nn_pr(nn, KERN_WARNING, fmt, ## args)
+#define nn_info(nn, fmt, args...) nn_pr(nn, KERN_INFO, fmt, ## args)
+#define nn_dbg(nn, fmt, args...) nn_pr(nn, KERN_DEBUG, fmt, ## args)
+
#define nn_dp_warn(dp, fmt, args...) \
- do { \
- if (unlikely(net_ratelimit())) \
- netdev_warn((dp)->netdev, fmt, ## args); \
- } while (0)
+ ({ \
+ struct nfp_net_dp *__dp = (dp); \
+ \
+ if (unlikely(net_ratelimit())) { \
+ if (__dp->netdev) \
+ netdev_warn(__dp->netdev, fmt, ## args); \
+ else \
+ dev_warn(__dp->dev, fmt, ## args); \
+ } \
+ })
/* Max time to wait for NFP to respond on updates (in seconds) */
#define NFP_NET_POLL_TIMEOUT 5
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 06/16] nfp: make sure debug accesses don't depend on netdevs
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (4 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 05/16] nfp: prepare print macros for use without netdev Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 07/16] nfp: allow allocation and initialization of netdev-less vNICs Jakub Kicinski
` (11 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
We want to be able to inspect the state of descriptor rings of
the control vNIC, so it will use the same interface as data vNICs.
Make sure the code doesn't use netdevs to determine state
of the rings and names things appropriately.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 11 +++++++++++
drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c | 9 ++++++---
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index b14aa31d494a..3eec4195c155 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -805,6 +805,17 @@ static inline u32 nfp_qcp_wr_ptr_read(u8 __iomem *q)
return _nfp_qcp_read(q, NFP_QCP_WRITE_PTR);
}
+static inline bool nfp_net_is_data_vnic(struct nfp_net *nn)
+{
+ WARN_ON_ONCE(!nn->dp.netdev && nn->port);
+ return !!nn->dp.netdev;
+}
+
+static inline bool nfp_net_running(struct nfp_net *nn)
+{
+ return nn->dp.ctrl & NFP_NET_CFG_CTRL_ENABLE;
+}
+
/* Globals */
extern const char nfp_driver_version[];
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c b/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
index 8c52c0e8379c..40217ece5fcb 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
@@ -54,7 +54,7 @@ static int nfp_net_debugfs_rx_q_read(struct seq_file *file, void *data)
goto out;
nn = r_vec->nfp_net;
rx_ring = r_vec->rx_ring;
- if (!netif_running(nn->dp.netdev))
+ if (!nfp_net_running(nn))
goto out;
rxd_cnt = rx_ring->cnt;
@@ -138,7 +138,7 @@ static int nfp_net_debugfs_tx_q_read(struct seq_file *file, void *data)
if (!r_vec->nfp_net || !tx_ring)
goto out;
nn = r_vec->nfp_net;
- if (!netif_running(nn->dp.netdev))
+ if (!nfp_net_running(nn))
goto out;
txd_cnt = tx_ring->cnt;
@@ -209,7 +209,10 @@ void nfp_net_debugfs_vnic_add(struct nfp_net *nn, struct dentry *ddir, int id)
if (IS_ERR_OR_NULL(nfp_dir))
return;
- sprintf(name, "vnic%d", id);
+ if (nfp_net_is_data_vnic(nn))
+ sprintf(name, "vnic%d", id);
+ else
+ strcpy(name, "ctrl-vnic");
nn->debugfs_dir = debugfs_create_dir(name, ddir);
if (IS_ERR_OR_NULL(nn->debugfs_dir))
return;
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 07/16] nfp: allow allocation and initialization of netdev-less vNICs
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (5 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 06/16] nfp: make sure debug accesses don't depend on netdevs Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 08/16] nfp: prepare config and enable for working without netdevs Jakub Kicinski
` (10 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
vNICs used for sending and receiving control messages shouldn't
really have a netdev. Add the ability to initialize vNICs for
netdev-less operation.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 2 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 170 ++++++++++++---------
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 7 +-
.../net/ethernet/netronome/nfp/nfp_netvf_main.c | 2 +-
4 files changed, 105 insertions(+), 76 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index 3eec4195c155..6b21c4d0ccfa 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -831,7 +831,7 @@ void nfp_net_get_fw_version(struct nfp_net_fw_version *fw_ver,
void __iomem *ctrl_bar);
struct nfp_net *
-nfp_net_alloc(struct pci_dev *pdev,
+nfp_net_alloc(struct pci_dev *pdev, bool needs_netdev,
unsigned int max_tx_rings, unsigned int max_rx_rings);
void nfp_net_free(struct nfp_net *nn);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index f8dba793a8fe..1cc7425ffd27 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -61,7 +61,7 @@
#include <linux/log2.h>
#include <linux/if_vlan.h>
#include <linux/random.h>
-
+#include <linux/vmalloc.h>
#include <linux/ktime.h>
#include <net/vxlan.h>
@@ -1820,7 +1820,7 @@ nfp_net_tx_ring_alloc(struct nfp_net_dp *dp, struct nfp_net_tx_ring *tx_ring)
if (!tx_ring->txbufs)
goto err_alloc;
- if (!tx_ring->is_xdp)
+ if (!tx_ring->is_xdp && dp->netdev)
netif_set_xps_queue(dp->netdev, &r_vec->affinity_mask,
tx_ring->idx);
@@ -3034,30 +3034,39 @@ void nfp_net_info(struct nfp_net *nn)
/**
* nfp_net_alloc() - Allocate netdev and related structure
* @pdev: PCI device
+ * @needs_netdev: Whether to allocate a netdev for this vNIC
* @max_tx_rings: Maximum number of TX rings supported by device
* @max_rx_rings: Maximum number of RX rings supported by device
*
* This function allocates a netdev device and fills in the initial
- * part of the @struct nfp_net structure.
+ * part of the @struct nfp_net structure. In case of control device
+ * nfp_net structure is allocated without the netdev.
*
* Return: NFP Net device structure, or ERR_PTR on error.
*/
-struct nfp_net *nfp_net_alloc(struct pci_dev *pdev,
+struct nfp_net *nfp_net_alloc(struct pci_dev *pdev, bool needs_netdev,
unsigned int max_tx_rings,
unsigned int max_rx_rings)
{
- struct net_device *netdev;
struct nfp_net *nn;
- netdev = alloc_etherdev_mqs(sizeof(struct nfp_net),
- max_tx_rings, max_rx_rings);
- if (!netdev)
- return ERR_PTR(-ENOMEM);
+ if (needs_netdev) {
+ struct net_device *netdev;
- SET_NETDEV_DEV(netdev, &pdev->dev);
- nn = netdev_priv(netdev);
+ netdev = alloc_etherdev_mqs(sizeof(struct nfp_net),
+ max_tx_rings, max_rx_rings);
+ if (!netdev)
+ return ERR_PTR(-ENOMEM);
+
+ SET_NETDEV_DEV(netdev, &pdev->dev);
+ nn = netdev_priv(netdev);
+ nn->dp.netdev = netdev;
+ } else {
+ nn = vzalloc(sizeof(*nn));
+ if (!nn)
+ return ERR_PTR(-ENOMEM);
+ }
- nn->dp.netdev = netdev;
nn->dp.dev = &pdev->dev;
nn->pdev = pdev;
@@ -3091,7 +3100,10 @@ struct nfp_net *nfp_net_alloc(struct pci_dev *pdev,
*/
void nfp_net_free(struct nfp_net *nn)
{
- free_netdev(nn->dp.netdev);
+ if (nn->dp.netdev)
+ free_netdev(nn->dp.netdev);
+ else
+ vfree(nn);
}
/**
@@ -3162,52 +3174,13 @@ static void nfp_net_irqmod_init(struct nfp_net *nn)
nn->tx_coalesce_max_frames = 64;
}
-/**
- * nfp_net_init() - Initialise/finalise the nfp_net structure
- * @nn: NFP Net device structure
- *
- * Return: 0 on success or negative errno on error.
- */
-int nfp_net_init(struct nfp_net *nn)
+static void nfp_net_netdev_init(struct nfp_net *nn)
{
struct net_device *netdev = nn->dp.netdev;
- int err;
-
- nn->dp.rx_dma_dir = DMA_FROM_DEVICE;
-
- /* Get some of the read-only fields from the BAR */
- nn->cap = nn_readl(nn, NFP_NET_CFG_CAP);
- nn->max_mtu = nn_readl(nn, NFP_NET_CFG_MAX_MTU);
-
- /* Chained metadata is signalled by capabilities except in version 4 */
- nn->dp.chained_metadata_format = nn->fw_ver.major == 4 ||
- nn->cap & NFP_NET_CFG_CTRL_CHAIN_META;
- if (nn->dp.chained_metadata_format && nn->fw_ver.major != 4)
- nn->cap &= ~NFP_NET_CFG_CTRL_RSS;
nfp_net_write_mac_addr(nn, nn->dp.netdev->dev_addr);
- /* Determine RX packet/metadata boundary offset */
- if (nn->fw_ver.major >= 2) {
- u32 reg;
-
- reg = nn_readl(nn, NFP_NET_CFG_RX_OFFSET);
- if (reg > NFP_NET_MAX_PREPEND) {
- nn_err(nn, "Invalid rx offset: %d\n", reg);
- return -EINVAL;
- }
- nn->dp.rx_offset = reg;
- } else {
- nn->dp.rx_offset = NFP_NET_RX_OFFSET;
- }
-
- /* Set default MTU and Freelist buffer size */
- if (nn->max_mtu < NFP_NET_DEFAULT_MTU)
- netdev->mtu = nn->max_mtu;
- else
- netdev->mtu = NFP_NET_DEFAULT_MTU;
- nn->dp.mtu = netdev->mtu;
- nn->dp.fl_bufsz = nfp_net_calc_fl_bufsz(&nn->dp);
+ netdev->mtu = nn->dp.mtu;
/* Advertise/enable offloads based on capabilities
*
@@ -3237,12 +3210,8 @@ int nfp_net_init(struct nfp_net *nn)
nn->dp.ctrl |= nn->cap & NFP_NET_CFG_CTRL_LSO2 ?:
NFP_NET_CFG_CTRL_LSO;
}
- if (nn->cap & NFP_NET_CFG_CTRL_RSS_ANY) {
+ if (nn->cap & NFP_NET_CFG_CTRL_RSS_ANY)
netdev->hw_features |= NETIF_F_RXHASH;
- nfp_net_rss_init(nn);
- nn->dp.ctrl |= nn->cap & NFP_NET_CFG_CTRL_RSS2 ?:
- NFP_NET_CFG_CTRL_RSS;
- }
if (nn->cap & NFP_NET_CFG_CTRL_VXLAN &&
nn->cap & NFP_NET_CFG_CTRL_NVGRE) {
if (nn->cap & NFP_NET_CFG_CTRL_LSO)
@@ -3277,6 +3246,68 @@ int nfp_net_init(struct nfp_net *nn)
netdev->features &= ~(NETIF_F_TSO | NETIF_F_TSO6);
nn->dp.ctrl &= ~NFP_NET_CFG_CTRL_LSO_ANY;
+ /* Finalise the netdev setup */
+ netdev->netdev_ops = &nfp_net_netdev_ops;
+ netdev->watchdog_timeo = msecs_to_jiffies(5 * 1000);
+
+ /* MTU range: 68 - hw-specific max */
+ netdev->min_mtu = ETH_MIN_MTU;
+ netdev->max_mtu = nn->max_mtu;
+
+ netif_carrier_off(netdev);
+
+ nfp_net_set_ethtool_ops(netdev);
+}
+
+/**
+ * nfp_net_init() - Initialise/finalise the nfp_net structure
+ * @nn: NFP Net device structure
+ *
+ * Return: 0 on success or negative errno on error.
+ */
+int nfp_net_init(struct nfp_net *nn)
+{
+ int err;
+
+ nn->dp.rx_dma_dir = DMA_FROM_DEVICE;
+
+ /* Get some of the read-only fields from the BAR */
+ nn->cap = nn_readl(nn, NFP_NET_CFG_CAP);
+ nn->max_mtu = nn_readl(nn, NFP_NET_CFG_MAX_MTU);
+
+ /* Chained metadata is signalled by capabilities except in version 4 */
+ nn->dp.chained_metadata_format = nn->fw_ver.major == 4 ||
+ nn->cap & NFP_NET_CFG_CTRL_CHAIN_META;
+ if (nn->dp.chained_metadata_format && nn->fw_ver.major != 4)
+ nn->cap &= ~NFP_NET_CFG_CTRL_RSS;
+
+ /* Determine RX packet/metadata boundary offset */
+ if (nn->fw_ver.major >= 2) {
+ u32 reg;
+
+ reg = nn_readl(nn, NFP_NET_CFG_RX_OFFSET);
+ if (reg > NFP_NET_MAX_PREPEND) {
+ nn_err(nn, "Invalid rx offset: %d\n", reg);
+ return -EINVAL;
+ }
+ nn->dp.rx_offset = reg;
+ } else {
+ nn->dp.rx_offset = NFP_NET_RX_OFFSET;
+ }
+
+ /* Set default MTU and Freelist buffer size */
+ if (nn->max_mtu < NFP_NET_DEFAULT_MTU)
+ nn->dp.mtu = nn->max_mtu;
+ else
+ nn->dp.mtu = NFP_NET_DEFAULT_MTU;
+ nn->dp.fl_bufsz = nfp_net_calc_fl_bufsz(&nn->dp);
+
+ if (nn->cap & NFP_NET_CFG_CTRL_RSS_ANY) {
+ nfp_net_rss_init(nn);
+ nn->dp.ctrl |= nn->cap & NFP_NET_CFG_CTRL_RSS2 ?:
+ NFP_NET_CFG_CTRL_RSS;
+ }
+
/* Allow L2 Broadcast and Multicast through by default, if supported */
if (nn->cap & NFP_NET_CFG_CTRL_L2BC)
nn->dp.ctrl |= NFP_NET_CFG_CTRL_L2BC;
@@ -3289,6 +3320,9 @@ int nfp_net_init(struct nfp_net *nn)
nn->dp.ctrl |= NFP_NET_CFG_CTRL_IRQMOD;
}
+ if (nn->dp.netdev)
+ nfp_net_netdev_init(nn);
+
/* Stash the re-configuration queue away. First odd queue in TX Bar */
nn->qcp_cfg = nn->tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
@@ -3301,20 +3335,11 @@ int nfp_net_init(struct nfp_net *nn)
if (err)
return err;
- /* Finalise the netdev setup */
- netdev->netdev_ops = &nfp_net_netdev_ops;
- netdev->watchdog_timeo = msecs_to_jiffies(5 * 1000);
-
- /* MTU range: 68 - hw-specific max */
- netdev->min_mtu = ETH_MIN_MTU;
- netdev->max_mtu = nn->max_mtu;
-
- netif_carrier_off(netdev);
-
- nfp_net_set_ethtool_ops(netdev);
nfp_net_vecs_init(nn);
- return register_netdev(netdev);
+ if (!nn->dp.netdev)
+ return 0;
+ return register_netdev(nn->dp.netdev);
}
/**
@@ -3323,6 +3348,9 @@ int nfp_net_init(struct nfp_net *nn)
*/
void nfp_net_clean(struct nfp_net *nn)
{
+ if (!nn->dp.netdev)
+ return;
+
unregister_netdev(nn->dp.netdev);
if (nn->dp.xdp_prog)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 7dd310911d9f..dd2a99fca716 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -303,7 +303,8 @@ static void nfp_net_pf_free_vnics(struct nfp_pf *pf)
}
static struct nfp_net *
-nfp_net_pf_alloc_vnic(struct nfp_pf *pf, void __iomem *ctrl_bar,
+nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
+ void __iomem *ctrl_bar,
void __iomem *tx_bar, void __iomem *rx_bar,
int stride, struct nfp_net_fw_version *fw_ver,
unsigned int eth_id)
@@ -316,7 +317,7 @@ nfp_net_pf_alloc_vnic(struct nfp_pf *pf, void __iomem *ctrl_bar,
n_rx_rings = readl(ctrl_bar + NFP_NET_CFG_MAX_RXRINGS);
/* Allocate and initialise the vNIC */
- nn = nfp_net_alloc(pf->pdev, n_tx_rings, n_rx_rings);
+ nn = nfp_net_alloc(pf->pdev, needs_netdev, n_tx_rings, n_rx_rings);
if (IS_ERR(nn))
return nn;
@@ -395,7 +396,7 @@ nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
prev_tx_base = tgt_tx_base;
prev_rx_base = tgt_rx_base;
- nn = nfp_net_pf_alloc_vnic(pf, ctrl_bar, tx_bar, rx_bar,
+ nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, tx_bar, rx_bar,
stride, fw_ver, i);
if (IS_ERR(nn)) {
err = PTR_ERR(nn);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
index 3f1c7f0f392e..0bf3b0febd07 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
@@ -202,7 +202,7 @@ static int nfp_netvf_pci_probe(struct pci_dev *pdev,
rx_bar_off = NFP_PCIE_QUEUE(startq);
/* Allocate and initialise the netdev */
- nn = nfp_net_alloc(pdev, max_tx_rings, max_rx_rings);
+ nn = nfp_net_alloc(pdev, true, max_tx_rings, max_rx_rings);
if (IS_ERR(nn)) {
err = PTR_ERR(nn);
goto err_ctrl_unmap;
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 08/16] nfp: prepare config and enable for working without netdevs
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (6 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 07/16] nfp: allow allocation and initialization of netdev-less vNICs Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 09/16] nfp: add control vNIC datapath Jakub Kicinski
` (9 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Out of the three stages of ifup/ifdown (allocate, configure, start)
- this commit prepares the configuration stage for working with
control vNICs.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index 1cc7425ffd27..c47705861a81 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -2236,9 +2236,10 @@ static int nfp_net_set_config_and_enable(struct nfp_net *nn)
nn_writeq(nn, NFP_NET_CFG_RXRS_ENABLE, nn->dp.num_rx_rings == 64 ?
0xffffffffffffffffULL : ((u64)1 << nn->dp.num_rx_rings) - 1);
- nfp_net_write_mac_addr(nn, nn->dp.netdev->dev_addr);
+ if (nn->dp.netdev)
+ nfp_net_write_mac_addr(nn, nn->dp.netdev->dev_addr);
- nn_writel(nn, NFP_NET_CFG_MTU, nn->dp.netdev->mtu);
+ nn_writel(nn, NFP_NET_CFG_MTU, nn->dp.mtu);
bufsz = nn->dp.fl_bufsz - nn->dp.rx_dma_off - NFP_NET_RX_BUF_NON_DATA;
nn_writel(nn, NFP_NET_CFG_FLBUFSZ, bufsz);
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 09/16] nfp: add control vNIC datapath
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (7 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 08/16] nfp: prepare config and enable for working without netdevs Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 10/16] nfp: make vNIC ctrl memory mapping function reusable Jakub Kicinski
` (8 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Since control vNICs don't have a netdev, they can't use napi and
queuing stack provides. Add simple tasklet-based data receive
and send of control messages with queuing on a skb_list.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_app.h | 11 +
drivers/net/ethernet/netronome/nfp/nfp_net.h | 17 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 323 ++++++++++++++++++++-
drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h | 3 +
4 files changed, 345 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.h b/drivers/net/ethernet/netronome/nfp/nfp_app.h
index 13efdefffa1a..f6091ad0a9a9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_app.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_app.h
@@ -38,6 +38,7 @@ struct bpf_prog;
struct net_device;
struct pci_dev;
struct tc_to_netdev;
+struct sk_buff;
struct nfp_app;
struct nfp_cpp;
struct nfp_pf;
@@ -55,6 +56,7 @@ extern const struct nfp_app_type app_bpf;
* struct nfp_app_type - application definition
* @id: application ID
* @name: application name
+ * @ctrl_has_meta: control messages have prepend of type:5/port:CTRL
*
* Callbacks
* @init: perform basic app checks
@@ -69,6 +71,8 @@ struct nfp_app_type {
enum nfp_app_id id;
const char *name;
+ bool ctrl_has_meta;
+
int (*init)(struct nfp_app *app);
const char *(*extra_cap)(struct nfp_app *app, struct nfp_net *nn);
@@ -99,6 +103,8 @@ struct nfp_app {
const struct nfp_app_type *type;
};
+bool nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb);
+
static inline int nfp_app_init(struct nfp_app *app)
{
if (!app->type->init)
@@ -125,6 +131,11 @@ static inline const char *nfp_app_name(struct nfp_app *app)
return app->type->name;
}
+static inline bool nfp_app_ctrl_has_meta(struct nfp_app *app)
+{
+ return app->type->ctrl_has_meta;
+}
+
static inline const char *nfp_app_extra_cap(struct nfp_app *app,
struct nfp_net *nn)
{
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index 6b21c4d0ccfa..eb849d26f4dd 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -405,7 +405,14 @@ struct nfp_net_rx_ring {
*/
struct nfp_net_r_vector {
struct nfp_net *nfp_net;
- struct napi_struct napi;
+ union {
+ struct napi_struct napi;
+ struct {
+ struct tasklet_struct tasklet;
+ struct sk_buff_head queue;
+ struct spinlock lock;
+ };
+ };
struct nfp_net_tx_ring *tx_ring;
struct nfp_net_rx_ring *rx_ring;
@@ -816,6 +823,11 @@ static inline bool nfp_net_running(struct nfp_net *nn)
return nn->dp.ctrl & NFP_NET_CFG_CTRL_ENABLE;
}
+static inline const char *nfp_net_name(struct nfp_net *nn)
+{
+ return nn->dp.netdev ? nn->dp.netdev->name : "ctrl";
+}
+
/* Globals */
extern const char nfp_driver_version[];
@@ -838,6 +850,9 @@ void nfp_net_free(struct nfp_net *nn);
int nfp_net_init(struct nfp_net *nn);
void nfp_net_clean(struct nfp_net *nn);
+int nfp_ctrl_open(struct nfp_net *nn);
+void nfp_ctrl_close(struct nfp_net *nn);
+
void nfp_net_set_ethtool_ops(struct net_device *netdev);
void nfp_net_info(struct nfp_net *nn);
int nfp_net_reconfig(struct nfp_net *nn, u32 update);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index c47705861a81..59f1764242a0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -392,6 +392,15 @@ static irqreturn_t nfp_net_irq_rxtx(int irq, void *data)
return IRQ_HANDLED;
}
+static irqreturn_t nfp_ctrl_irq_rxtx(int irq, void *data)
+{
+ struct nfp_net_r_vector *r_vec = data;
+
+ tasklet_schedule(&r_vec->tasklet);
+
+ return IRQ_HANDLED;
+}
+
/**
* nfp_net_read_link_status() - Reread link status from control BAR
* @nn: NFP Network structure
@@ -523,7 +532,7 @@ nfp_net_aux_irq_request(struct nfp_net *nn, u32 ctrl_offset,
entry = &nn->irq_entries[vector_idx];
- snprintf(name, name_sz, format, netdev_name(nn->dp.netdev));
+ snprintf(name, name_sz, format, nfp_net_name(nn));
err = request_irq(entry->vector, handler, 0, name, nn);
if (err) {
nn_err(nn, "Failed to request IRQ %d (err=%d).\n",
@@ -943,6 +952,9 @@ static void nfp_net_tx_complete(struct nfp_net_tx_ring *tx_ring)
r_vec->tx_pkts += done_pkts;
u64_stats_update_end(&r_vec->tx_sync);
+ if (!dp->netdev)
+ return;
+
nd_q = netdev_get_tx_queue(dp->netdev, tx_ring->idx);
netdev_tx_completed_queue(nd_q, done_pkts, done_bytes);
if (nfp_net_tx_ring_should_wake(tx_ring)) {
@@ -1052,7 +1064,7 @@ nfp_net_tx_ring_reset(struct nfp_net_dp *dp, struct nfp_net_tx_ring *tx_ring)
tx_ring->qcp_rd_p = 0;
tx_ring->wr_ptr_add = 0;
- if (tx_ring->is_xdp)
+ if (tx_ring->is_xdp || !dp->netdev)
return;
nd_q = netdev_get_tx_queue(dp->netdev, tx_ring->idx);
@@ -1742,6 +1754,231 @@ static int nfp_net_poll(struct napi_struct *napi, int budget)
return pkts_polled;
}
+/* Control device data path
+ */
+
+static bool
+nfp_ctrl_tx_one(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
+ struct sk_buff *skb, bool old)
+{
+ unsigned int real_len = skb->len, meta_len = 0;
+ struct nfp_net_tx_ring *tx_ring;
+ struct nfp_net_tx_buf *txbuf;
+ struct nfp_net_tx_desc *txd;
+ struct nfp_net_dp *dp;
+ dma_addr_t dma_addr;
+ int wr_idx;
+
+ dp = &r_vec->nfp_net->dp;
+ tx_ring = r_vec->tx_ring;
+
+ if (WARN_ON_ONCE(skb_shinfo(skb)->nr_frags)) {
+ nn_dp_warn(dp, "Driver's CTRL TX does not implement gather\n");
+ goto err_free;
+ }
+
+ if (unlikely(nfp_net_tx_full(tx_ring, 1))) {
+ u64_stats_update_begin(&r_vec->tx_sync);
+ r_vec->tx_busy++;
+ u64_stats_update_end(&r_vec->tx_sync);
+ if (!old)
+ __skb_queue_tail(&r_vec->queue, skb);
+ else
+ __skb_queue_head(&r_vec->queue, skb);
+ return true;
+ }
+
+ if (nfp_app_ctrl_has_meta(nn->app)) {
+ if (unlikely(skb_headroom(skb) < 8)) {
+ nn_dp_warn(dp, "CTRL TX on skb without headroom\n");
+ goto err_free;
+ }
+ meta_len = 8;
+ put_unaligned_be32(NFP_META_PORT_ID_CTRL, skb_push(skb, 4));
+ put_unaligned_be32(NFP_NET_META_PORTID, skb_push(skb, 4));
+ }
+
+ /* Start with the head skbuf */
+ dma_addr = dma_map_single(dp->dev, skb->data, skb_headlen(skb),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(dp->dev, dma_addr))
+ goto err_dma_warn;
+
+ wr_idx = D_IDX(tx_ring, tx_ring->wr_p);
+
+ /* Stash the soft descriptor of the head then initialize it */
+ txbuf = &tx_ring->txbufs[wr_idx];
+ txbuf->skb = skb;
+ txbuf->dma_addr = dma_addr;
+ txbuf->fidx = -1;
+ txbuf->pkt_cnt = 1;
+ txbuf->real_len = real_len;
+
+ /* Build TX descriptor */
+ txd = &tx_ring->txds[wr_idx];
+ txd->offset_eop = meta_len | PCIE_DESC_TX_EOP;
+ txd->dma_len = cpu_to_le16(skb_headlen(skb));
+ nfp_desc_set_dma_addr(txd, dma_addr);
+ txd->data_len = cpu_to_le16(skb->len);
+
+ txd->flags = 0;
+ txd->mss = 0;
+ txd->lso_hdrlen = 0;
+
+ tx_ring->wr_p++;
+ tx_ring->wr_ptr_add++;
+ nfp_net_tx_xmit_more_flush(tx_ring);
+
+ return false;
+
+err_dma_warn:
+ nn_dp_warn(dp, "Failed to DMA map TX CTRL buffer\n");
+err_free:
+ u64_stats_update_begin(&r_vec->tx_sync);
+ r_vec->tx_errors++;
+ u64_stats_update_end(&r_vec->tx_sync);
+ dev_kfree_skb_any(skb);
+ return false;
+}
+
+bool nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb)
+{
+ struct nfp_net_r_vector *r_vec = &nn->r_vecs[0];
+ bool ret;
+
+ spin_lock_bh(&r_vec->lock);
+ ret = nfp_ctrl_tx_one(nn, r_vec, skb, false);
+ spin_unlock_bh(&r_vec->lock);
+
+ return ret;
+}
+
+static void __nfp_ctrl_tx_queued(struct nfp_net_r_vector *r_vec)
+{
+ struct sk_buff *skb;
+
+ while ((skb = __skb_dequeue(&r_vec->queue)))
+ if (nfp_ctrl_tx_one(r_vec->nfp_net, r_vec, skb, true))
+ return;
+}
+
+static bool
+nfp_ctrl_meta_ok(struct nfp_net *nn, void *data, unsigned int meta_len)
+{
+ u32 meta_type, meta_tag;
+
+ if (!nfp_app_ctrl_has_meta(nn->app))
+ return !meta_len;
+
+ if (meta_len != 8)
+ return false;
+
+ meta_type = get_unaligned_be32(data);
+ meta_tag = get_unaligned_be32(data + 4);
+
+ return (meta_type == NFP_NET_META_PORTID &&
+ meta_tag == NFP_META_PORT_ID_CTRL);
+}
+
+static bool
+nfp_ctrl_rx_one(struct nfp_net *nn, struct nfp_net_dp *dp,
+ struct nfp_net_r_vector *r_vec, struct nfp_net_rx_ring *rx_ring)
+{
+ unsigned int meta_len, data_len, meta_off, pkt_len, pkt_off;
+ struct nfp_net_rx_buf *rxbuf;
+ struct nfp_net_rx_desc *rxd;
+ dma_addr_t new_dma_addr;
+ struct sk_buff *skb;
+ void *new_frag;
+ int idx;
+
+ idx = D_IDX(rx_ring, rx_ring->rd_p);
+
+ rxd = &rx_ring->rxds[idx];
+ if (!(rxd->rxd.meta_len_dd & PCIE_DESC_RX_DD))
+ return false;
+
+ /* Memory barrier to ensure that we won't do other reads
+ * before the DD bit.
+ */
+ dma_rmb();
+
+ rx_ring->rd_p++;
+
+ rxbuf = &rx_ring->rxbufs[idx];
+ meta_len = rxd->rxd.meta_len_dd & PCIE_DESC_RX_META_LEN_MASK;
+ data_len = le16_to_cpu(rxd->rxd.data_len);
+ pkt_len = data_len - meta_len;
+
+ pkt_off = NFP_NET_RX_BUF_HEADROOM + dp->rx_dma_off;
+ if (dp->rx_offset == NFP_NET_CFG_RX_OFFSET_DYNAMIC)
+ pkt_off += meta_len;
+ else
+ pkt_off += dp->rx_offset;
+ meta_off = pkt_off - meta_len;
+
+ /* Stats update */
+ u64_stats_update_begin(&r_vec->rx_sync);
+ r_vec->rx_pkts++;
+ r_vec->rx_bytes += pkt_len;
+ u64_stats_update_end(&r_vec->rx_sync);
+
+ nfp_net_dma_sync_cpu_rx(dp, rxbuf->dma_addr + meta_off, data_len);
+
+ if (unlikely(!nfp_ctrl_meta_ok(nn, rxbuf->frag + meta_off, meta_len))) {
+ nn_dp_warn(dp, "incorrect metadata for ctrl packet (%d)\n",
+ meta_len);
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, NULL);
+ return true;
+ }
+
+ skb = build_skb(rxbuf->frag, dp->fl_bufsz);
+ if (unlikely(!skb)) {
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, NULL);
+ return true;
+ }
+ new_frag = nfp_net_napi_alloc_one(dp, &new_dma_addr);
+ if (unlikely(!new_frag)) {
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, skb);
+ return true;
+ }
+
+ nfp_net_dma_unmap_rx(dp, rxbuf->dma_addr);
+
+ nfp_net_rx_give_one(dp, rx_ring, new_frag, new_dma_addr);
+
+ skb_reserve(skb, pkt_off);
+ skb_put(skb, pkt_len);
+
+ dev_kfree_skb_any(skb);
+
+ return true;
+}
+
+static void nfp_ctrl_rx(struct nfp_net_r_vector *r_vec)
+{
+ struct nfp_net_rx_ring *rx_ring = r_vec->rx_ring;
+ struct nfp_net *nn = r_vec->nfp_net;
+ struct nfp_net_dp *dp = &nn->dp;
+
+ while (nfp_ctrl_rx_one(nn, dp, r_vec, rx_ring))
+ continue;
+}
+
+static void nfp_ctrl_poll(unsigned long arg)
+{
+ struct nfp_net_r_vector *r_vec = (void *)arg;
+
+ spin_lock_bh(&r_vec->lock);
+ nfp_net_tx_complete(r_vec->tx_ring);
+ __nfp_ctrl_tx_queued(r_vec);
+ spin_unlock_bh(&r_vec->lock);
+
+ nfp_ctrl_rx(r_vec);
+
+ nfp_net_irq_unmask(r_vec->nfp_net, r_vec->irq_entry);
+}
+
/* Setup and Configuration
*/
@@ -1764,10 +2001,21 @@ static void nfp_net_vecs_init(struct nfp_net *nn)
r_vec = &nn->r_vecs[r];
r_vec->nfp_net = nn;
- r_vec->handler = nfp_net_irq_rxtx;
r_vec->irq_entry = entry->entry;
r_vec->irq_vector = entry->vector;
+ if (nn->dp.netdev) {
+ r_vec->handler = nfp_net_irq_rxtx;
+ } else {
+ r_vec->handler = nfp_ctrl_irq_rxtx;
+
+ __skb_queue_head_init(&r_vec->queue);
+ spin_lock_init(&r_vec->lock);
+ tasklet_init(&r_vec->tasklet, nfp_ctrl_poll,
+ (unsigned long)r_vec);
+ tasklet_disable(&r_vec->tasklet);
+ }
+
cpumask_set_cpu(r, &r_vec->affinity_mask);
}
}
@@ -2034,15 +2282,22 @@ nfp_net_prepare_vector(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
int err;
/* Setup NAPI */
- netif_napi_add(nn->dp.netdev, &r_vec->napi,
- nfp_net_poll, NAPI_POLL_WEIGHT);
+ if (nn->dp.netdev)
+ netif_napi_add(nn->dp.netdev, &r_vec->napi,
+ nfp_net_poll, NAPI_POLL_WEIGHT);
+ else
+ tasklet_enable(&r_vec->tasklet);
snprintf(r_vec->name, sizeof(r_vec->name),
- "%s-rxtx-%d", nn->dp.netdev->name, idx);
+ "%s-rxtx-%d", nfp_net_name(nn), idx);
err = request_irq(r_vec->irq_vector, r_vec->handler, 0, r_vec->name,
r_vec);
if (err) {
- netif_napi_del(&r_vec->napi);
+ if (nn->dp.netdev)
+ netif_napi_del(&r_vec->napi);
+ else
+ tasklet_disable(&r_vec->tasklet);
+
nn_err(nn, "Error requesting IRQ %d\n", r_vec->irq_vector);
return err;
}
@@ -2060,7 +2315,11 @@ static void
nfp_net_cleanup_vector(struct nfp_net *nn, struct nfp_net_r_vector *r_vec)
{
irq_set_affinity_hint(r_vec->irq_vector, NULL);
- netif_napi_del(&r_vec->napi);
+ if (nn->dp.netdev)
+ netif_napi_del(&r_vec->napi);
+ else
+ tasklet_disable(&r_vec->tasklet);
+
free_irq(r_vec->irq_vector, r_vec);
}
@@ -2338,6 +2597,24 @@ static int nfp_net_netdev_close(struct net_device *netdev)
return 0;
}
+void nfp_ctrl_close(struct nfp_net *nn)
+{
+ int r;
+
+ rtnl_lock();
+
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
+ disable_irq(nn->r_vecs[r].irq_vector);
+ tasklet_disable(&nn->r_vecs[r].tasklet);
+ }
+
+ nfp_net_clear_config_and_disable(nn);
+
+ nfp_net_close_free_all(nn);
+
+ rtnl_unlock();
+}
+
/**
* nfp_net_open_stack() - Start the device from stack's perspective
* @nn: NFP Net device to reconfigure
@@ -2453,6 +2730,35 @@ static int nfp_net_netdev_open(struct net_device *netdev)
return err;
}
+int nfp_ctrl_open(struct nfp_net *nn)
+{
+ int err, r;
+
+ /* ring dumping depends on vNICs being opened/closed under rtnl */
+ rtnl_lock();
+
+ err = nfp_net_open_alloc_all(nn);
+ if (err)
+ goto err_unlock;
+
+ err = nfp_net_set_config_and_enable(nn);
+ if (err)
+ goto err_free_all;
+
+ for (r = 0; r < nn->dp.num_r_vecs; r++)
+ enable_irq(nn->r_vecs[r].irq_vector);
+
+ rtnl_unlock();
+
+ return 0;
+
+err_free_all:
+ nfp_net_close_free_all(nn);
+err_unlock:
+ rtnl_unlock();
+ return err;
+}
+
static void nfp_net_set_rx_mode(struct net_device *netdev)
{
struct nfp_net *nn = netdev_priv(netdev);
@@ -3278,6 +3584,7 @@ int nfp_net_init(struct nfp_net *nn)
/* Chained metadata is signalled by capabilities except in version 4 */
nn->dp.chained_metadata_format = nn->fw_ver.major == 4 ||
+ !nn->dp.netdev ||
nn->cap & NFP_NET_CFG_CTRL_CHAIN_META;
if (nn->dp.chained_metadata_format && nn->fw_ver.major != 4)
nn->cap &= ~NFP_NET_CFG_CTRL_RSS;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
index c8208bf370e0..48a8bf97645e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
@@ -71,8 +71,11 @@
#define NFP_NET_META_FIELD_SIZE 4
#define NFP_NET_META_HASH 1 /* next field carries hash type */
#define NFP_NET_META_MARK 2
+#define NFP_NET_META_PORTID 5
#define NFP_NET_META_CSUM 6 /* checksum complete type */
+#define NFP_META_PORT_ID_CTRL ~0U
+
/**
* Hash type pre-pended when a RSS hash was computed
*/
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 10/16] nfp: make vNIC ctrl memory mapping function reusable
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (8 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 09/16] nfp: add control vNIC datapath Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 11/16] nfp: map all queue controllers at once Jakub Kicinski
` (7 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
We will soon need to map control vNIC PCI memory as well as data vNIC
memory. Make the function for mapping areas pointed to by an RTsym
reusable.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 55 ++++++++++++-----------
1 file changed, 28 insertions(+), 27 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index dd2a99fca716..3644b12d93db 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -248,40 +248,37 @@ nfp_net_pf_total_qcs(struct nfp_pf *pf, void __iomem *ctrl_bar,
return max_qc - min_qc;
}
-static u8 __iomem *nfp_net_pf_map_ctrl_bar(struct nfp_pf *pf)
+static u8 __iomem *
+nfp_net_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+ unsigned int min_size, struct nfp_cpp_area **area)
{
- const struct nfp_rtsym *ctrl_sym;
- u8 __iomem *ctrl_bar;
+ const struct nfp_rtsym *sym;
char pf_symbol[256];
+ u8 __iomem *mem;
- snprintf(pf_symbol, sizeof(pf_symbol), "_pf%u_net_bar0",
+ snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt,
nfp_cppcore_pcie_unit(pf->cpp));
- ctrl_sym = nfp_rtsym_lookup(pf->cpp, pf_symbol);
- if (!ctrl_sym) {
- dev_err(&pf->pdev->dev,
- "Failed to find PF BAR0 symbol %s\n", pf_symbol);
- return NULL;
+ sym = nfp_rtsym_lookup(pf->cpp, pf_symbol);
+ if (!sym) {
+ nfp_err(pf->cpp, "Failed to find PF symbol %s\n", pf_symbol);
+ return (u8 __iomem *)ERR_PTR(-ENOENT);
}
- if (ctrl_sym->size < pf->max_data_vnics * NFP_PF_CSR_SLICE_SIZE) {
- dev_err(&pf->pdev->dev,
- "PF BAR0 too small to contain %d vNICs\n",
- pf->max_data_vnics);
- return NULL;
+ if (sym->size < min_size) {
+ nfp_err(pf->cpp, "PF symbol %s too small\n", pf_symbol);
+ return (u8 __iomem *)ERR_PTR(-EINVAL);
}
- ctrl_bar = nfp_net_map_area(pf->cpp, "net.ctrl",
- ctrl_sym->domain, ctrl_sym->target,
- ctrl_sym->addr, ctrl_sym->size,
- &pf->data_vnic_bar);
- if (IS_ERR(ctrl_bar)) {
- dev_err(&pf->pdev->dev, "Failed to map PF BAR0: %ld\n",
- PTR_ERR(ctrl_bar));
- return NULL;
+ mem = nfp_net_map_area(pf->cpp, name, sym->domain, sym->target,
+ sym->addr, sym->size, area);
+ if (IS_ERR(mem)) {
+ nfp_err(pf->cpp, "Failed to map PF symbol %s: %ld\n",
+ pf_symbol, PTR_ERR(mem));
+ return mem;
}
- return ctrl_bar;
+ return mem;
}
static void nfp_net_pf_free_vnic(struct nfp_pf *pf, struct nfp_net *nn)
@@ -662,10 +659,10 @@ int nfp_net_refresh_eth_port(struct nfp_port *port)
*/
int nfp_net_pci_probe(struct nfp_pf *pf)
{
+ u32 ctrl_bar_sz, tx_area_sz, rx_area_sz;
u8 __iomem *ctrl_bar, *tx_bar, *rx_bar;
u32 total_tx_qcs, total_rx_qcs;
struct nfp_net_fw_version fw_ver;
- u32 tx_area_sz, rx_area_sz;
u32 start_q;
int stride;
int err;
@@ -685,9 +682,13 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
goto err_unlock;
}
- ctrl_bar = nfp_net_pf_map_ctrl_bar(pf);
- if (!ctrl_bar) {
- err = pf->fw_loaded ? -EINVAL : -EPROBE_DEFER;
+ ctrl_bar_sz = pf->max_data_vnics * NFP_PF_CSR_SLICE_SIZE;
+ ctrl_bar = nfp_net_pf_map_rtsym(pf, "net.ctrl", "_pf%d_net_bar0",
+ ctrl_bar_sz, &pf->data_vnic_bar);
+ if (IS_ERR(ctrl_bar)) {
+ err = PTR_ERR(ctrl_bar);
+ if (!pf->fw_loaded && err == -ENOENT)
+ err = -EPROBE_DEFER;
goto err_unlock;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 11/16] nfp: map all queue controllers at once
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (9 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 10/16] nfp: make vNIC ctrl memory mapping function reusable Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 12/16] nfp: don't clutter init code passing fw_ver around Jakub Kicinski
` (6 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
RX and TX queue controllers are interleaved. Instead of creating
two mappings which map the same area at slightly different offset,
create only one mapping. Always map all queue controllers to simplify
the code and allow reusing the mapping for non-data vNICs.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.h | 6 +-
drivers/net/ethernet/netronome/nfp/nfp_net.h | 1 +
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 122 +++++-----------------
3 files changed, 28 insertions(+), 101 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index c46d00bbf19d..66b1e1490805 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -63,8 +63,7 @@ struct nfp_nsp_identify;
* @cpp: Pointer to the CPP handle
* @app: Pointer to the APP handle
* @data_vnic_bar: Pointer to the CPP area for the data vNICs' BARs
- * @tx_area: Pointer to the CPP area for the TX queues
- * @rx_area: Pointer to the CPP area for the FL/RX queues
+ * @qc_area: Pointer to the CPP area for the queues
* @irq_entries: Array of MSI-X entries for all vNICs
* @limit_vfs: Number of VFs supported by firmware (~0 for PCI limit)
* @num_vfs: Number of SR-IOV VFs enabled
@@ -88,8 +87,7 @@ struct nfp_pf {
struct nfp_app *app;
struct nfp_cpp_area *data_vnic_bar;
- struct nfp_cpp_area *tx_area;
- struct nfp_cpp_area *rx_area;
+ struct nfp_cpp_area *qc_area;
struct msix_entry *irq_entries;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index eb849d26f4dd..02fd8d4e253c 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -705,6 +705,7 @@ static inline void nn_pci_flush(struct nfp_net *nn)
* either add to a pointer or to read the pointer value.
*/
#define NFP_QCP_QUEUE_ADDR_SZ 0x800
+#define NFP_QCP_QUEUE_AREA_SZ 0x80000
#define NFP_QCP_QUEUE_OFF(_x) ((_x) * NFP_QCP_QUEUE_ADDR_SZ)
#define NFP_QCP_QUEUE_ADD_RPTR 0x0000
#define NFP_QCP_QUEUE_ADD_WPTR 0x0004
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 3644b12d93db..2a3b6deae607 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -223,31 +223,6 @@ static int nfp_net_pf_get_app_id(struct nfp_pf *pf)
NFP_APP_CORE_NIC);
}
-static unsigned int
-nfp_net_pf_total_qcs(struct nfp_pf *pf, void __iomem *ctrl_bar,
- unsigned int stride, u32 start_off, u32 num_off)
-{
- unsigned int i, min_qc, max_qc;
-
- min_qc = readl(ctrl_bar + start_off);
- max_qc = min_qc;
-
- for (i = 0; i < pf->max_data_vnics; i++) {
- /* To make our lives simpler only accept configuration where
- * queues are allocated to PFs in order (queues of PFn all have
- * indexes lower than PFn+1).
- */
- if (max_qc > readl(ctrl_bar + start_off))
- return 0;
-
- max_qc = readl(ctrl_bar + start_off);
- max_qc += readl(ctrl_bar + num_off) * stride;
- ctrl_bar += NFP_PF_CSR_SLICE_SIZE;
- }
-
- return max_qc - min_qc;
-}
-
static u8 __iomem *
nfp_net_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
unsigned int min_size, struct nfp_cpp_area **area)
@@ -301,15 +276,16 @@ static void nfp_net_pf_free_vnics(struct nfp_pf *pf)
static struct nfp_net *
nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
- void __iomem *ctrl_bar,
- void __iomem *tx_bar, void __iomem *rx_bar,
+ void __iomem *ctrl_bar, void __iomem *qc_bar,
int stride, struct nfp_net_fw_version *fw_ver,
unsigned int eth_id)
{
- u32 n_tx_rings, n_rx_rings;
+ u32 tx_base, rx_base, n_tx_rings, n_rx_rings;
struct nfp_net *nn;
int err;
+ tx_base = readl(ctrl_bar + NFP_NET_CFG_START_TXQ);
+ rx_base = readl(ctrl_bar + NFP_NET_CFG_START_RXQ);
n_tx_rings = readl(ctrl_bar + NFP_NET_CFG_MAX_TXRINGS);
n_rx_rings = readl(ctrl_bar + NFP_NET_CFG_MAX_RXRINGS);
@@ -321,8 +297,8 @@ nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
nn->app = pf->app;
nn->fw_ver = *fw_ver;
nn->dp.ctrl_bar = ctrl_bar;
- nn->tx_bar = tx_bar;
- nn->rx_bar = rx_bar;
+ nn->tx_bar = qc_bar + tx_base * NFP_QCP_QUEUE_ADDR_SZ;
+ nn->rx_bar = qc_bar + rx_base * NFP_QCP_QUEUE_ADDR_SZ;
nn->dp.is_vf = 0;
nn->stride_rx = stride;
nn->stride_tx = stride;
@@ -374,26 +350,15 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
static int
nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
- void __iomem *tx_bar, void __iomem *rx_bar,
- int stride, struct nfp_net_fw_version *fw_ver)
+ void __iomem *qc_bar, int stride,
+ struct nfp_net_fw_version *fw_ver)
{
- u32 prev_tx_base, prev_rx_base, tgt_tx_base, tgt_rx_base;
struct nfp_net *nn;
unsigned int i;
int err;
- prev_tx_base = readl(ctrl_bar + NFP_NET_CFG_START_TXQ);
- prev_rx_base = readl(ctrl_bar + NFP_NET_CFG_START_RXQ);
-
for (i = 0; i < pf->max_data_vnics; i++) {
- tgt_tx_base = readl(ctrl_bar + NFP_NET_CFG_START_TXQ);
- tgt_rx_base = readl(ctrl_bar + NFP_NET_CFG_START_RXQ);
- tx_bar += (tgt_tx_base - prev_tx_base) * NFP_QCP_QUEUE_ADDR_SZ;
- rx_bar += (tgt_rx_base - prev_rx_base) * NFP_QCP_QUEUE_ADDR_SZ;
- prev_tx_base = tgt_tx_base;
- prev_rx_base = tgt_rx_base;
-
- nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, tx_bar, rx_bar,
+ nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar,
stride, fw_ver, i);
if (IS_ERR(nn)) {
err = PTR_ERR(nn);
@@ -430,8 +395,7 @@ static void nfp_net_pf_clean_vnic(struct nfp_pf *pf, struct nfp_net *nn)
static int
nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
- void __iomem *ctrl_bar, void __iomem *tx_bar,
- void __iomem *rx_bar, int stride,
+ void __iomem *ctrl_bar, void __iomem *qc_bar, int stride,
struct nfp_net_fw_version *fw_ver)
{
unsigned int id, wanted_irqs, num_irqs, vnics_left, irqs_left;
@@ -439,8 +403,7 @@ nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
int err;
/* Allocate the vnics and do basic init */
- err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, tx_bar, rx_bar,
- stride, fw_ver);
+ err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, qc_bar, stride, fw_ver);
if (err)
return err;
@@ -534,8 +497,7 @@ static void nfp_net_pci_remove_finish(struct nfp_pf *pf)
nfp_net_pf_app_clean(pf);
- nfp_cpp_area_release_free(pf->rx_area);
- nfp_cpp_area_release_free(pf->tx_area);
+ nfp_cpp_area_release_free(pf->qc_area);
nfp_cpp_area_release_free(pf->data_vnic_bar);
}
@@ -659,11 +621,9 @@ int nfp_net_refresh_eth_port(struct nfp_port *port)
*/
int nfp_net_pci_probe(struct nfp_pf *pf)
{
- u32 ctrl_bar_sz, tx_area_sz, rx_area_sz;
- u8 __iomem *ctrl_bar, *tx_bar, *rx_bar;
- u32 total_tx_qcs, total_rx_qcs;
struct nfp_net_fw_version fw_ver;
- u32 start_q;
+ u8 __iomem *ctrl_bar, *qc_bar;
+ u32 ctrl_bar_sz;
int stride;
int err;
@@ -718,53 +678,23 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
}
}
- /* Find how many QC structs need to be mapped */
- total_tx_qcs = nfp_net_pf_total_qcs(pf, ctrl_bar, stride,
- NFP_NET_CFG_START_TXQ,
- NFP_NET_CFG_MAX_TXRINGS);
- total_rx_qcs = nfp_net_pf_total_qcs(pf, ctrl_bar, stride,
- NFP_NET_CFG_START_RXQ,
- NFP_NET_CFG_MAX_RXRINGS);
- if (!total_tx_qcs || !total_rx_qcs) {
- nfp_err(pf->cpp, "Invalid PF QC configuration [%d,%d]\n",
- total_tx_qcs, total_rx_qcs);
- err = -EINVAL;
- goto err_ctrl_unmap;
- }
-
- tx_area_sz = NFP_QCP_QUEUE_ADDR_SZ * total_tx_qcs;
- rx_area_sz = NFP_QCP_QUEUE_ADDR_SZ * total_rx_qcs;
-
- /* Map TX queues */
- start_q = readl(ctrl_bar + NFP_NET_CFG_START_TXQ);
- tx_bar = nfp_net_map_area(pf->cpp, "net.tx", 0, 0,
- NFP_PCIE_QUEUE(start_q),
- tx_area_sz, &pf->tx_area);
- if (IS_ERR(tx_bar)) {
- nfp_err(pf->cpp, "Failed to map TX area.\n");
- err = PTR_ERR(tx_bar);
+ /* Map queues */
+ qc_bar = nfp_net_map_area(pf->cpp, "net.qc", 0, 0,
+ NFP_PCIE_QUEUE(0), NFP_QCP_QUEUE_AREA_SZ,
+ &pf->qc_area);
+ if (IS_ERR(qc_bar)) {
+ nfp_err(pf->cpp, "Failed to map Queue Controller area.\n");
+ err = PTR_ERR(qc_bar);
goto err_ctrl_unmap;
}
- /* Map RX queues */
- start_q = readl(ctrl_bar + NFP_NET_CFG_START_RXQ);
- rx_bar = nfp_net_map_area(pf->cpp, "net.rx", 0, 0,
- NFP_PCIE_QUEUE(start_q),
- rx_area_sz, &pf->rx_area);
- if (IS_ERR(rx_bar)) {
- nfp_err(pf->cpp, "Failed to map RX area.\n");
- err = PTR_ERR(rx_bar);
- goto err_unmap_tx;
- }
-
err = nfp_net_pf_app_init(pf);
if (err)
- goto err_unmap_rx;
+ goto err_unmap_qc;
pf->ddir = nfp_net_debugfs_device_add(pf->pdev);
- err = nfp_net_pf_spawn_vnics(pf, ctrl_bar, tx_bar, rx_bar,
- stride, &fw_ver);
+ err = nfp_net_pf_spawn_vnics(pf, ctrl_bar, qc_bar, stride, &fw_ver);
if (err)
goto err_clean_ddir;
@@ -775,10 +705,8 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
err_clean_ddir:
nfp_net_debugfs_dir_clean(&pf->ddir);
nfp_net_pf_app_clean(pf);
-err_unmap_rx:
- nfp_cpp_area_release_free(pf->rx_area);
-err_unmap_tx:
- nfp_cpp_area_release_free(pf->tx_area);
+err_unmap_qc:
+ nfp_cpp_area_release_free(pf->qc_area);
err_ctrl_unmap:
nfp_cpp_area_release_free(pf->data_vnic_bar);
err_unlock:
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 12/16] nfp: don't clutter init code passing fw_ver around
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (10 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 11/16] nfp: map all queue controllers at once Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 13/16] nfp: slice the netdev spawning function Jakub Kicinski
` (5 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Reading fw version from the BAR is trivial. Don't pass it around
through layers of init functions, simply read it again where needed.
This commit has the side effect of each vNIC having the exact NFD
version from its own control memory, rather than all data vNICs
assuming the version of the first one. This should not result in
user-visible changes, though. Capabilities of data vNICs of trival
apps are identical.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 2a3b6deae607..82172665e023 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -277,8 +277,7 @@ static void nfp_net_pf_free_vnics(struct nfp_pf *pf)
static struct nfp_net *
nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
void __iomem *ctrl_bar, void __iomem *qc_bar,
- int stride, struct nfp_net_fw_version *fw_ver,
- unsigned int eth_id)
+ int stride, unsigned int eth_id)
{
u32 tx_base, rx_base, n_tx_rings, n_rx_rings;
struct nfp_net *nn;
@@ -295,7 +294,7 @@ nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
return nn;
nn->app = pf->app;
- nn->fw_ver = *fw_ver;
+ nfp_net_get_fw_version(&nn->fw_ver, ctrl_bar);
nn->dp.ctrl_bar = ctrl_bar;
nn->tx_bar = qc_bar + tx_base * NFP_QCP_QUEUE_ADDR_SZ;
nn->rx_bar = qc_bar + rx_base * NFP_QCP_QUEUE_ADDR_SZ;
@@ -350,8 +349,7 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
static int
nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
- void __iomem *qc_bar, int stride,
- struct nfp_net_fw_version *fw_ver)
+ void __iomem *qc_bar, int stride)
{
struct nfp_net *nn;
unsigned int i;
@@ -359,7 +357,7 @@ nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
for (i = 0; i < pf->max_data_vnics; i++) {
nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar,
- stride, fw_ver, i);
+ stride, i);
if (IS_ERR(nn)) {
err = PTR_ERR(nn);
goto err_free_prev;
@@ -395,15 +393,14 @@ static void nfp_net_pf_clean_vnic(struct nfp_pf *pf, struct nfp_net *nn)
static int
nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
- void __iomem *ctrl_bar, void __iomem *qc_bar, int stride,
- struct nfp_net_fw_version *fw_ver)
+ void __iomem *ctrl_bar, void __iomem *qc_bar, int stride)
{
unsigned int id, wanted_irqs, num_irqs, vnics_left, irqs_left;
struct nfp_net *nn;
int err;
/* Allocate the vnics and do basic init */
- err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, qc_bar, stride, fw_ver);
+ err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, qc_bar, stride);
if (err)
return err;
@@ -694,7 +691,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
pf->ddir = nfp_net_debugfs_device_add(pf->pdev);
- err = nfp_net_pf_spawn_vnics(pf, ctrl_bar, qc_bar, stride, &fw_ver);
+ err = nfp_net_pf_spawn_vnics(pf, ctrl_bar, qc_bar, stride);
if (err)
goto err_clean_ddir;
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 13/16] nfp: slice the netdev spawning function
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (11 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 12/16] nfp: don't clutter init code passing fw_ver around Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 14/16] nfp: allow non-equal distribution of IRQs Jakub Kicinski
` (4 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
We want to be able to create a special vNIC for control messages.
This vNIC should be created before any netdev is registered to allow
nfp_app logic to exchange messages with the FW app before any netdev
is visible to user space. Unfortunately we can't enable IRQs until
we know how many vNICs we will need to spawn.
Divide the function which spawns netdevs for vNICs into three parts:
- vNIC/memory allocation;
- IRQ allocation;
- netdev init and register.
This will help us insert the initialization of the control channel
after IRQ allocation but before netdev init and register.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 62 ++++++++++++++---------
1 file changed, 37 insertions(+), 25 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 82172665e023..98a99b199674 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -391,18 +391,10 @@ static void nfp_net_pf_clean_vnic(struct nfp_pf *pf, struct nfp_net *nn)
nfp_app_vnic_clean(pf->app, nn);
}
-static int
-nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
- void __iomem *ctrl_bar, void __iomem *qc_bar, int stride)
+static int nfp_net_pf_alloc_irqs(struct nfp_pf *pf)
{
- unsigned int id, wanted_irqs, num_irqs, vnics_left, irqs_left;
+ unsigned int wanted_irqs, num_irqs, vnics_left, irqs_left;
struct nfp_net *nn;
- int err;
-
- /* Allocate the vnics and do basic init */
- err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, qc_bar, stride);
- if (err)
- return err;
/* Get MSI-X vectors */
wanted_irqs = 0;
@@ -410,18 +402,16 @@ nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
wanted_irqs += NFP_NET_NON_Q_VECTORS + nn->dp.num_r_vecs;
pf->irq_entries = kcalloc(wanted_irqs, sizeof(*pf->irq_entries),
GFP_KERNEL);
- if (!pf->irq_entries) {
- err = -ENOMEM;
- goto err_nn_free;
- }
+ if (!pf->irq_entries)
+ return -ENOMEM;
num_irqs = nfp_net_irqs_alloc(pf->pdev, pf->irq_entries,
NFP_NET_MIN_VNIC_IRQS * pf->num_vnics,
wanted_irqs);
if (!num_irqs) {
- nn_warn(nn, "Unable to allocate MSI-X Vectors. Exiting\n");
- err = -ENOMEM;
- goto err_vec_free;
+ nfp_warn(pf->cpp, "Unable to allocate MSI-X vectors\n");
+ kfree(pf->irq_entries);
+ return -ENOMEM;
}
/* Distribute IRQs to vNICs */
@@ -437,6 +427,21 @@ nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
vnics_left--;
}
+ return 0;
+}
+
+static void nfp_net_pf_free_irqs(struct nfp_pf *pf)
+{
+ nfp_net_irqs_disable(pf->pdev);
+ kfree(pf->irq_entries);
+}
+
+static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
+{
+ struct nfp_net *nn;
+ unsigned int id;
+ int err;
+
/* Finish vNIC init and register */
id = 0;
list_for_each_entry(nn, &pf->vnics, vnic_list) {
@@ -452,11 +457,6 @@ nfp_net_pf_spawn_vnics(struct nfp_pf *pf,
err_prev_deinit:
list_for_each_entry_continue_reverse(nn, &pf->vnics, vnic_list)
nfp_net_pf_clean_vnic(pf, nn);
- nfp_net_irqs_disable(pf->pdev);
-err_vec_free:
- kfree(pf->irq_entries);
-err_nn_free:
- nfp_net_pf_free_vnics(pf);
return err;
}
@@ -489,8 +489,7 @@ static void nfp_net_pci_remove_finish(struct nfp_pf *pf)
{
nfp_net_debugfs_dir_clean(&pf->ddir);
- nfp_net_irqs_disable(pf->pdev);
- kfree(pf->irq_entries);
+ nfp_net_pf_free_irqs(pf);
nfp_net_pf_app_clean(pf);
@@ -691,14 +690,27 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
pf->ddir = nfp_net_debugfs_device_add(pf->pdev);
- err = nfp_net_pf_spawn_vnics(pf, ctrl_bar, qc_bar, stride);
+ /* Allocate the vnics and do basic init */
+ err = nfp_net_pf_alloc_vnics(pf, ctrl_bar, qc_bar, stride);
if (err)
goto err_clean_ddir;
+ err = nfp_net_pf_alloc_irqs(pf);
+ if (err)
+ goto err_free_vnics;
+
+ err = nfp_net_pf_init_vnics(pf);
+ if (err)
+ goto err_free_irqs;
+
mutex_unlock(&pf->lock);
return 0;
+err_free_irqs:
+ nfp_net_pf_free_irqs(pf);
+err_free_vnics:
+ nfp_net_pf_free_vnics(pf);
err_clean_ddir:
nfp_net_debugfs_dir_clean(&pf->ddir);
nfp_net_pf_app_clean(pf);
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 14/16] nfp: allow non-equal distribution of IRQs
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (12 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 13/16] nfp: slice the netdev spawning function Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 15/16] nfp: create control vNICs and wire up rx/tx Jakub Kicinski
` (3 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
Thus far the code assumed all vNICs will request similar number of IRQs.
This will be no longer true with control vNICs (where 1 IRQ will suffice).
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 98a99b199674..362dca38223b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -420,7 +420,8 @@ static int nfp_net_pf_alloc_irqs(struct nfp_pf *pf)
list_for_each_entry(nn, &pf->vnics, vnic_list) {
unsigned int n;
- n = DIV_ROUND_UP(irqs_left, vnics_left);
+ n = min(NFP_NET_NON_Q_VECTORS + nn->dp.num_r_vecs,
+ DIV_ROUND_UP(irqs_left, vnics_left));
nfp_net_irqs_assign(nn, &pf->irq_entries[num_irqs - irqs_left],
n);
irqs_left -= n;
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 15/16] nfp: create control vNICs and wire up rx/tx
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (13 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 14/16] nfp: allow non-equal distribution of IRQs Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 16/16] nfp: advertise support for NFD ABI 0.5 Jakub Kicinski
` (2 subsequent siblings)
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
When driver encounters an nfp_app which has a control message handler
defined, allocate a control vNIC. This control channel will be used
to exchange data with the application FW such as flow table programming,
statistics and global datapath control.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_app.c | 18 +++
drivers/net/ethernet/netronome/nfp/nfp_app.h | 44 ++++++++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 7 ++
.../net/ethernet/netronome/nfp/nfp_net_common.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 121 ++++++++++++++++++---
5 files changed, 177 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.c b/drivers/net/ethernet/netronome/nfp/nfp_app.c
index cea2090cf063..de07517da1bd 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_app.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_app.c
@@ -31,6 +31,7 @@
* SOFTWARE.
*/
+#include <linux/skbuff.h>
#include <linux/slab.h>
#include "nfpcore/nfp_cpp.h"
@@ -42,6 +43,23 @@ static const struct nfp_app_type *apps[] = {
&app_bpf,
};
+struct sk_buff *nfp_app_ctrl_msg_alloc(struct nfp_app *app, unsigned int size)
+{
+ struct sk_buff *skb;
+
+ if (nfp_app_ctrl_has_meta(app))
+ size += 8;
+
+ skb = alloc_skb(size, GFP_ATOMIC);
+ if (!skb)
+ return NULL;
+
+ if (nfp_app_ctrl_has_meta(app))
+ skb_reserve(skb, 8);
+
+ return skb;
+}
+
struct nfp_app *nfp_app_alloc(struct nfp_pf *pf, enum nfp_app_id id)
{
struct nfp_app *app;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.h b/drivers/net/ethernet/netronome/nfp/nfp_app.h
index f6091ad0a9a9..3fbf68f8577c 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_app.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_app.h
@@ -37,6 +37,7 @@
struct bpf_prog;
struct net_device;
struct pci_dev;
+struct sk_buff;
struct tc_to_netdev;
struct sk_buff;
struct nfp_app;
@@ -63,6 +64,9 @@ extern const struct nfp_app_type app_bpf;
* @extra_cap: extra capabilities string
* @vnic_init: init vNICs (assign port types, etc.)
* @vnic_clean: clean up app's vNIC state
+ * @start: start application logic
+ * @stop: stop application logic
+ * @ctrl_msg_rx: control message handler
* @setup_tc: setup TC ndo
* @tc_busy: TC HW offload busy (rules loaded)
* @xdp_offload: offload an XDP program
@@ -81,6 +85,11 @@ struct nfp_app_type {
unsigned int id);
void (*vnic_clean)(struct nfp_app *app, struct nfp_net *nn);
+ int (*start)(struct nfp_app *app);
+ void (*stop)(struct nfp_app *app);
+
+ void (*ctrl_msg_rx)(struct nfp_app *app, struct sk_buff *skb);
+
int (*setup_tc)(struct nfp_app *app, struct net_device *netdev,
u32 handle, __be16 proto, struct tc_to_netdev *tc);
bool (*tc_busy)(struct nfp_app *app, struct nfp_net *nn);
@@ -93,6 +102,7 @@ struct nfp_app_type {
* @pdev: backpointer to PCI device
* @pf: backpointer to NFP PF structure
* @cpp: pointer to the CPP handle
+ * @ctrl: pointer to ctrl vNIC struct
* @type: pointer to const application ops and info
*/
struct nfp_app {
@@ -100,6 +110,8 @@ struct nfp_app {
struct nfp_pf *pf;
struct nfp_cpp *cpp;
+ struct nfp_net *ctrl;
+
const struct nfp_app_type *type;
};
@@ -124,6 +136,21 @@ static inline void nfp_app_vnic_clean(struct nfp_app *app, struct nfp_net *nn)
app->type->vnic_clean(app, nn);
}
+static inline int nfp_app_start(struct nfp_app *app, struct nfp_net *ctrl)
+{
+ app->ctrl = ctrl;
+ if (!app->type->start)
+ return 0;
+ return app->type->start(app);
+}
+
+static inline void nfp_app_stop(struct nfp_app *app)
+{
+ if (!app->type->stop)
+ return;
+ app->type->stop(app);
+}
+
static inline const char *nfp_app_name(struct nfp_app *app)
{
if (!app)
@@ -131,6 +158,11 @@ static inline const char *nfp_app_name(struct nfp_app *app)
return app->type->name;
}
+static inline bool nfp_app_needs_ctrl_vnic(struct nfp_app *app)
+{
+ return app && app->type->ctrl_msg_rx;
+}
+
static inline bool nfp_app_ctrl_has_meta(struct nfp_app *app)
{
return app->type->ctrl_has_meta;
@@ -174,6 +206,18 @@ static inline int nfp_app_xdp_offload(struct nfp_app *app, struct nfp_net *nn,
return app->type->xdp_offload(app, nn, prog);
}
+static inline bool nfp_app_ctrl_tx(struct nfp_app *app, struct sk_buff *skb)
+{
+ return nfp_ctrl_tx(app->ctrl, skb);
+}
+
+static inline void nfp_app_ctrl_rx(struct nfp_app *app, struct sk_buff *skb)
+{
+ app->type->ctrl_msg_rx(app, skb);
+}
+
+struct sk_buff *nfp_app_ctrl_msg_alloc(struct nfp_app *app, unsigned int size);
+
struct nfp_app *nfp_app_alloc(struct nfp_pf *pf, enum nfp_app_id id);
void nfp_app_free(struct nfp_app *app);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 66b1e1490805..37832853b0b3 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -63,11 +63,13 @@ struct nfp_nsp_identify;
* @cpp: Pointer to the CPP handle
* @app: Pointer to the APP handle
* @data_vnic_bar: Pointer to the CPP area for the data vNICs' BARs
+ * @ctrl_vnic_bar: Pointer to the CPP area for the ctrl vNIC's BAR
* @qc_area: Pointer to the CPP area for the queues
* @irq_entries: Array of MSI-X entries for all vNICs
* @limit_vfs: Number of VFs supported by firmware (~0 for PCI limit)
* @num_vfs: Number of SR-IOV VFs enabled
* @fw_loaded: Is the firmware loaded?
+ * @ctrl_vnic: Pointer to the control vNIC if available
* @eth_tbl: NSP ETH table
* @nspi: NSP identification info
* @hwmon_dev: pointer to hwmon device
@@ -87,6 +89,7 @@ struct nfp_pf {
struct nfp_app *app;
struct nfp_cpp_area *data_vnic_bar;
+ struct nfp_cpp_area *ctrl_vnic_bar;
struct nfp_cpp_area *qc_area;
struct msix_entry *irq_entries;
@@ -96,6 +99,8 @@ struct nfp_pf {
bool fw_loaded;
+ struct nfp_net *ctrl_vnic;
+
struct nfp_eth_table *eth_tbl;
struct nfp_nsp_identify *nspi;
@@ -127,4 +132,6 @@ nfp_net_find_port(struct nfp_eth_table *eth_tbl, unsigned int id);
void
nfp_net_get_mac_addr(struct nfp_net *nn, struct nfp_cpp *cpp, unsigned int id);
+bool nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb);
+
#endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index 59f1764242a0..4f0df63de626 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -1950,7 +1950,7 @@ nfp_ctrl_rx_one(struct nfp_net *nn, struct nfp_net_dp *dp,
skb_reserve(skb, pkt_off);
skb_put(skb, pkt_len);
- dev_kfree_skb_any(skb);
+ nfp_app_ctrl_rx(nn->app, skb);
return true;
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 362dca38223b..db12700b5afc 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -266,12 +266,11 @@ static void nfp_net_pf_free_vnic(struct nfp_pf *pf, struct nfp_net *nn)
static void nfp_net_pf_free_vnics(struct nfp_pf *pf)
{
- struct nfp_net *nn;
+ struct nfp_net *nn, *next;
- while (!list_empty(&pf->vnics)) {
- nn = list_first_entry(&pf->vnics, struct nfp_net, vnic_list);
- nfp_net_pf_free_vnic(pf, nn);
- }
+ list_for_each_entry_safe(nn, next, &pf->vnics, vnic_list)
+ if (nfp_net_is_data_vnic(nn))
+ nfp_net_pf_free_vnic(pf, nn);
}
static struct nfp_net *
@@ -302,10 +301,12 @@ nfp_net_pf_alloc_vnic(struct nfp_pf *pf, bool needs_netdev,
nn->stride_rx = stride;
nn->stride_tx = stride;
- err = nfp_app_vnic_init(pf->app, nn, eth_id);
- if (err) {
- nfp_net_free(nn);
- return ERR_PTR(err);
+ if (needs_netdev) {
+ err = nfp_app_vnic_init(pf->app, nn, eth_id);
+ if (err) {
+ nfp_net_free(nn);
+ return ERR_PTR(err);
+ }
}
pf->num_vnics++;
@@ -446,6 +447,8 @@ static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
/* Finish vNIC init and register */
id = 0;
list_for_each_entry(nn, &pf->vnics, vnic_list) {
+ if (!nfp_net_is_data_vnic(nn))
+ continue;
err = nfp_net_pf_init_vnic(pf, nn, id);
if (err)
goto err_prev_deinit;
@@ -457,12 +460,15 @@ static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
err_prev_deinit:
list_for_each_entry_continue_reverse(nn, &pf->vnics, vnic_list)
- nfp_net_pf_clean_vnic(pf, nn);
+ if (nfp_net_is_data_vnic(nn))
+ nfp_net_pf_clean_vnic(pf, nn);
return err;
}
-static int nfp_net_pf_app_init(struct nfp_pf *pf)
+static int
+nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
{
+ u8 __iomem *ctrl_bar;
int err;
pf->app = nfp_app_alloc(pf, nfp_net_pf_get_app_id(pf));
@@ -473,8 +479,28 @@ static int nfp_net_pf_app_init(struct nfp_pf *pf)
if (err)
goto err_free;
+ if (!nfp_app_needs_ctrl_vnic(pf->app))
+ return 0;
+
+ ctrl_bar = nfp_net_pf_map_rtsym(pf, "net.ctrl", "_pf%u_net_ctrl_bar",
+ NFP_PF_CSR_SLICE_SIZE,
+ &pf->ctrl_vnic_bar);
+ if (IS_ERR(ctrl_bar)) {
+ err = PTR_ERR(ctrl_bar);
+ goto err_free;
+ }
+
+ pf->ctrl_vnic = nfp_net_pf_alloc_vnic(pf, false, ctrl_bar, qc_bar,
+ stride, 0);
+ if (IS_ERR(pf->ctrl_vnic)) {
+ err = PTR_ERR(pf->ctrl_vnic);
+ goto err_unmap;
+ }
+
return 0;
+err_unmap:
+ nfp_cpp_area_release_free(pf->ctrl_vnic_bar);
err_free:
nfp_app_free(pf->app);
return err;
@@ -482,12 +508,72 @@ static int nfp_net_pf_app_init(struct nfp_pf *pf)
static void nfp_net_pf_app_clean(struct nfp_pf *pf)
{
+ if (pf->ctrl_vnic) {
+ nfp_net_pf_free_vnic(pf, pf->ctrl_vnic);
+ nfp_cpp_area_release_free(pf->ctrl_vnic_bar);
+ }
nfp_app_free(pf->app);
pf->app = NULL;
}
+static int nfp_net_pf_app_start_ctrl(struct nfp_pf *pf)
+{
+ int err;
+
+ if (!pf->ctrl_vnic)
+ return 0;
+ err = nfp_net_pf_init_vnic(pf, pf->ctrl_vnic, 0);
+ if (err)
+ return err;
+
+ err = nfp_ctrl_open(pf->ctrl_vnic);
+ if (err)
+ goto err_clean_ctrl;
+
+ return 0;
+
+err_clean_ctrl:
+ nfp_net_pf_clean_vnic(pf, pf->ctrl_vnic);
+ return err;
+}
+
+static void nfp_net_pf_app_stop_ctrl(struct nfp_pf *pf)
+{
+ if (!pf->ctrl_vnic)
+ return;
+ nfp_ctrl_close(pf->ctrl_vnic);
+ nfp_net_pf_clean_vnic(pf, pf->ctrl_vnic);
+}
+
+static int nfp_net_pf_app_start(struct nfp_pf *pf)
+{
+ int err;
+
+ err = nfp_net_pf_app_start_ctrl(pf);
+ if (err)
+ return err;
+
+ err = nfp_app_start(pf->app, pf->ctrl_vnic);
+ if (err)
+ goto err_ctrl_stop;
+
+ return 0;
+
+err_ctrl_stop:
+ nfp_net_pf_app_stop_ctrl(pf);
+ return err;
+}
+
+static void nfp_net_pf_app_stop(struct nfp_pf *pf)
+{
+ nfp_app_stop(pf->app);
+ nfp_net_pf_app_stop_ctrl(pf);
+}
+
static void nfp_net_pci_remove_finish(struct nfp_pf *pf)
{
+ nfp_net_pf_app_stop(pf);
+ /* stop app first, to avoid double free of ctrl vNIC's ddir */
nfp_net_debugfs_dir_clean(&pf->ddir);
nfp_net_pf_free_irqs(pf);
@@ -685,7 +771,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
goto err_ctrl_unmap;
}
- err = nfp_net_pf_app_init(pf);
+ err = nfp_net_pf_app_init(pf, qc_bar, stride);
if (err)
goto err_unmap_qc;
@@ -700,14 +786,20 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
if (err)
goto err_free_vnics;
- err = nfp_net_pf_init_vnics(pf);
+ err = nfp_net_pf_app_start(pf);
if (err)
goto err_free_irqs;
+ err = nfp_net_pf_init_vnics(pf);
+ if (err)
+ goto err_stop_app;
+
mutex_unlock(&pf->lock);
return 0;
+err_stop_app:
+ nfp_net_pf_app_stop(pf);
err_free_irqs:
nfp_net_pf_free_irqs(pf);
err_free_vnics:
@@ -733,7 +825,8 @@ void nfp_net_pci_remove(struct nfp_pf *pf)
goto out;
list_for_each_entry(nn, &pf->vnics, vnic_list)
- nfp_net_pf_clean_vnic(pf, nn);
+ if (nfp_net_is_data_vnic(nn))
+ nfp_net_pf_clean_vnic(pf, nn);
nfp_net_pf_free_vnics(pf);
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH net-next 16/16] nfp: advertise support for NFD ABI 0.5
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (14 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 15/16] nfp: create control vNICs and wire up rx/tx Jakub Kicinski
@ 2017-06-06 0:01 ` Jakub Kicinski
2017-06-06 6:16 ` [PATCH net-next 00/16] nfp: ctrl vNIC Jiri Pirko
2017-06-07 16:52 ` David Miller
17 siblings, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 0:01 UTC (permalink / raw)
To: netdev; +Cc: oss-drivers, Jakub Kicinski
NFD ABI 0.5 is equivalent to NFD ABI 3.0 but requires that the
driver checks the APP id symbol and makes sure it can support
given app. Most advanced apps will likely require control vNIC
(ability to exchange control messages between the driver and
app FW). Detailed app version checking and capability exchange
is left to app-specific code.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index db12700b5afc..5f27703060c2 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -749,7 +749,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
nfp_warn(pf->cpp, "OBSOLETE Firmware detected - VF isolation not available\n");
} else {
switch (fw_ver.major) {
- case 1 ... 4:
+ case 1 ... 5:
stride = 4;
break;
default:
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
index 0bf3b0febd07..c879626e035b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
@@ -161,7 +161,7 @@ static int nfp_netvf_pci_probe(struct pci_dev *pdev,
dev_warn(&pdev->dev, "OBSOLETE Firmware detected - VF isolation not available\n");
} else {
switch (fw_ver.major) {
- case 1 ... 4:
+ case 1 ... 5:
stride = 4;
tx_bar_no = NFP_NET_Q0_BAR;
rx_bar_no = tx_bar_no;
--
2.11.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (15 preceding siblings ...)
2017-06-06 0:01 ` [PATCH net-next 16/16] nfp: advertise support for NFD ABI 0.5 Jakub Kicinski
@ 2017-06-06 6:16 ` Jiri Pirko
2017-06-06 7:21 ` Jakub Kicinski
2017-06-07 16:52 ` David Miller
17 siblings, 1 reply; 30+ messages in thread
From: Jiri Pirko @ 2017-06-06 6:16 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: netdev, oss-drivers
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
>Hi!
>
>This series adds the ability to use one vNIC as a control channel
>for passing messages to and from the application firmware. The
>implementation restructures the existing netdev vNIC code to be able
>to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
>are not visible to userspace (other than for dumping ring state), and
>since they don't have netdevs we use a tasklet for RX and simple skb
>list for TX queuing.
>
>Due to special status of the control vNIC we have to reshuffle the
>init code a bit to make sure control vNIC will be fully brought up
>(and therefore communication with app FW can happen) before any netdev
>or port is visible to user space.
>
>FW will designate which vNIC is supposed to be used as control one
>by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
>being prepended to control message, some prefer to look at queue ID
>to decide that something is a control message. Our implementation
>can cater to both.
>
>First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
I'm not really sure that doing it using one "control netdevice" is the
correct way to go. The configuration is asic-wide, should be done by a
devlink parent handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
Thanks.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 6:16 ` [PATCH net-next 00/16] nfp: ctrl vNIC Jiri Pirko
@ 2017-06-06 7:21 ` Jakub Kicinski
2017-06-06 8:23 ` Jiri Pirko
0 siblings, 1 reply; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 7:21 UTC (permalink / raw)
To: Jiri Pirko; +Cc: netdev, oss-drivers
On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
> Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
> >Hi!
> >
> >This series adds the ability to use one vNIC as a control channel
> >for passing messages to and from the application firmware. The
> >implementation restructures the existing netdev vNIC code to be able
> >to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
> >are not visible to userspace (other than for dumping ring state), and
> >since they don't have netdevs we use a tasklet for RX and simple skb
> >list for TX queuing.
> >
> >Due to special status of the control vNIC we have to reshuffle the
> >init code a bit to make sure control vNIC will be fully brought up
> >(and therefore communication with app FW can happen) before any netdev
> >or port is visible to user space.
> >
> >FW will designate which vNIC is supposed to be used as control one
> >by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
> >being prepended to control message, some prefer to look at queue ID
> >to decide that something is a control message. Our implementation
> >can cater to both.
> >
> >First two users of this code will be eBPF maps and flower offloads.
>
> How do you actually do the configuration from the userspace? I did not
> find it in the patches.
Yes, there is nothing interesting in those patches, really. It's all
internal to the driver. This set basically allows us to dedicate some
queue pairs to high-speed communication with the FW (sending commands,
populating/dumping eBPF and Flower tables).
The eBPF maps and Flower offload patches should follow in coming weeks.
> I'm not really sure that doing it using one "control netdevice" is the
> correct way to go.
The control queues are not exposed to user space at all. It's up to
the driver logic to generate and consume all messages passing on those
queues. I had to make this a separate series because of the amount
of code churn. I'm also not a fan of exposing control netdevices.
Reasons ranging from NetworkManager latching onto them (even if it's
configured not to) to someone inevitably trying to control the device
from user space with vendor commands and make the driver-kept state go
out of whack :/
> The configuration is asic-wide, should be done by a devlink parent
> handle which was introduced for that exact purpose.
>
> Am I missing something? We need to sync in this. In mlxsw we need to do
> some pre-netdev configuraton as well.
For programmable NICs we still need to come up with some API for setting
the target application/firmware name/adapter mode, but I don't think we
have any more fine-grained parameters to set per-device just yet.
I was thinking of either devlink, or perhaps, since I'm hearing people
want those to be persistent/written into flash, the recent MTD
discussion got me wondering if we should just expose the entire flash
and teach ethtool to modify the vendor-specific parameter table
directly in the flash. That seems like something that could quickly
get out of hand, though :S
What were your plans with pre-netdev config?
I hope this makes things slightly clearer, sorry for not doing a good
job on the cover letter :)
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 7:21 ` Jakub Kicinski
@ 2017-06-06 8:23 ` Jiri Pirko
2017-06-06 9:09 ` Jakub Kicinski
0 siblings, 1 reply; 30+ messages in thread
From: Jiri Pirko @ 2017-06-06 8:23 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: netdev, oss-drivers
Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
>On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
>> Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
>> >Hi!
>> >
>> >This series adds the ability to use one vNIC as a control channel
>> >for passing messages to and from the application firmware. The
>> >implementation restructures the existing netdev vNIC code to be able
>> >to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
>> >are not visible to userspace (other than for dumping ring state), and
>> >since they don't have netdevs we use a tasklet for RX and simple skb
>> >list for TX queuing.
>> >
>> >Due to special status of the control vNIC we have to reshuffle the
>> >init code a bit to make sure control vNIC will be fully brought up
>> >(and therefore communication with app FW can happen) before any netdev
>> >or port is visible to user space.
>> >
>> >FW will designate which vNIC is supposed to be used as control one
>> >by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
>> >being prepended to control message, some prefer to look at queue ID
>> >to decide that something is a control message. Our implementation
>> >can cater to both.
>> >
>> >First two users of this code will be eBPF maps and flower offloads.
>>
>> How do you actually do the configuration from the userspace? I did not
>> find it in the patches.
>
>Yes, there is nothing interesting in those patches, really. It's all
>internal to the driver. This set basically allows us to dedicate some
>queue pairs to high-speed communication with the FW (sending commands,
>populating/dumping eBPF and Flower tables).
>
>The eBPF maps and Flower offload patches should follow in coming weeks.
Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
he wants to use the nic, right?
>
>> I'm not really sure that doing it using one "control netdevice" is the
>> correct way to go.
>
>The control queues are not exposed to user space at all. It's up to
>the driver logic to generate and consume all messages passing on those
>queues. I had to make this a separate series because of the amount
>of code churn. I'm also not a fan of exposing control netdevices.
>Reasons ranging from NetworkManager latching onto them (even if it's
>configured not to) to someone inevitably trying to control the device
>from user space with vendor commands and make the driver-kept state go
>out of whack :/
Okay, that got me confused. I thought that you want the control
netdevice. Now that is clear we both don't want to see that, I'm fine :)
>
>> The configuration is asic-wide, should be done by a devlink parent
>> handle which was introduced for that exact purpose.
>>
>> Am I missing something? We need to sync in this. In mlxsw we need to do
>> some pre-netdev configuraton as well.
>
>For programmable NICs we still need to come up with some API for setting
>the target application/firmware name/adapter mode, but I don't think we
>have any more fine-grained parameters to set per-device just yet.
>
>I was thinking of either devlink, or perhaps, since I'm hearing people
>want those to be persistent/written into flash, the recent MTD
>discussion got me wondering if we should just expose the entire flash
>and teach ethtool to modify the vendor-specific parameter table
>directly in the flash. That seems like something that could quickly
>get out of hand, though :S
Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
for the config like yours (switching the operation mode of nic), we
should have well defined user api. And since this is per-device, not
per-netdev, devlink should be the vahicle to carry this.
>
>What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
>
>I hope this makes things slightly clearer, sorry for not doing a good
>job on the cover letter :)
No worries :)
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 8:23 ` Jiri Pirko
@ 2017-06-06 9:09 ` Jakub Kicinski
2017-06-06 9:17 ` Jiri Pirko
0 siblings, 1 reply; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 9:09 UTC (permalink / raw)
To: Jiri Pirko; +Cc: netdev, oss-drivers
On Tue, 6 Jun 2017 10:23:36 +0200, Jiri Pirko wrote:
> Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
> >On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
> >> Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
> >> >Hi!
> >> >
> >> >This series adds the ability to use one vNIC as a control channel
> >> >for passing messages to and from the application firmware. The
> >> >implementation restructures the existing netdev vNIC code to be able
> >> >to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
> >> >are not visible to userspace (other than for dumping ring state), and
> >> >since they don't have netdevs we use a tasklet for RX and simple skb
> >> >list for TX queuing.
> >> >
> >> >Due to special status of the control vNIC we have to reshuffle the
> >> >init code a bit to make sure control vNIC will be fully brought up
> >> >(and therefore communication with app FW can happen) before any netdev
> >> >or port is visible to user space.
> >> >
> >> >FW will designate which vNIC is supposed to be used as control one
> >> >by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
> >> >being prepended to control message, some prefer to look at queue ID
> >> >to decide that something is a control message. Our implementation
> >> >can cater to both.
> >> >
> >> >First two users of this code will be eBPF maps and flower offloads.
> >>
> >> How do you actually do the configuration from the userspace? I did not
> >> find it in the patches.
> >
> >Yes, there is nothing interesting in those patches, really. It's all
> >internal to the driver. This set basically allows us to dedicate some
> >queue pairs to high-speed communication with the FW (sending commands,
> >populating/dumping eBPF and Flower tables).
> >
> >The eBPF maps and Flower offload patches should follow in coming weeks.
>
> Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
> he wants to use the nic, right?
Yes, right now we just expect the user to drop the FW file he wants
to run into /lib/firmware/netronome. Driver will interrogate the
firmware once loaded to see which "app" it is.
> >> The configuration is asic-wide, should be done by a devlink parent
> >> handle which was introduced for that exact purpose.
> >>
> >> Am I missing something? We need to sync in this. In mlxsw we need to do
> >> some pre-netdev configuraton as well.
> >
> >For programmable NICs we still need to come up with some API for setting
> >the target application/firmware name/adapter mode, but I don't think we
> >have any more fine-grained parameters to set per-device just yet.
> >
> >I was thinking of either devlink, or perhaps, since I'm hearing people
> >want those to be persistent/written into flash, the recent MTD
> >discussion got me wondering if we should just expose the entire flash
> >and teach ethtool to modify the vendor-specific parameter table
> >directly in the flash. That seems like something that could quickly
> >get out of hand, though :S
>
> Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
> for the config like yours (switching the operation mode of nic), we
> should have well defined user api. And since this is per-device, not
> per-netdev, devlink should be the vahicle to carry this.
Well defined API would be nice. Perhaps we will just start with
something very simple and then extend it if need be. E.g. I'm hearing
people would like to select not only the app but also the version of
it... Yes, starting simple is probably the way forward.
> >What were your plans with pre-netdev config?
>
> We need to pass come initial resource division. Generally the consensus
> is to have these options exposed through devlink, let the user configure
> them all and then to have a trigger that would cause driver
> re-orchestration according to the new values. The flow would look like
> this:
>
> -driver loads with defaults, inits hw and instantiates netdevs
> -driver exposes config options via devlink
> -user sets up the options
> -user pushes the "go" trigger
> -upon the trigger command, devlink calls the driver re-init callback
> -driver shuts down the current instances, re-initializes hw,
> re-instantiates the netdevs
>
> Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 9:09 ` Jakub Kicinski
@ 2017-06-06 9:17 ` Jiri Pirko
2017-06-06 9:35 ` Mintz, Yuval
2017-06-06 19:54 ` Jakub Kicinski
0 siblings, 2 replies; 30+ messages in thread
From: Jiri Pirko @ 2017-06-06 9:17 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: netdev, oss-drivers
Tue, Jun 06, 2017 at 11:09:45AM CEST, kubakici@wp.pl wrote:
>On Tue, 6 Jun 2017 10:23:36 +0200, Jiri Pirko wrote:
>> Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
>> >On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
>> >> Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
>> >> >Hi!
>> >> >
>> >> >This series adds the ability to use one vNIC as a control channel
>> >> >for passing messages to and from the application firmware. The
>> >> >implementation restructures the existing netdev vNIC code to be able
>> >> >to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
>> >> >are not visible to userspace (other than for dumping ring state), and
>> >> >since they don't have netdevs we use a tasklet for RX and simple skb
>> >> >list for TX queuing.
>> >> >
>> >> >Due to special status of the control vNIC we have to reshuffle the
>> >> >init code a bit to make sure control vNIC will be fully brought up
>> >> >(and therefore communication with app FW can happen) before any netdev
>> >> >or port is visible to user space.
>> >> >
>> >> >FW will designate which vNIC is supposed to be used as control one
>> >> >by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
>> >> >being prepended to control message, some prefer to look at queue ID
>> >> >to decide that something is a control message. Our implementation
>> >> >can cater to both.
>> >> >
>> >> >First two users of this code will be eBPF maps and flower offloads.
>> >>
>> >> How do you actually do the configuration from the userspace? I did not
>> >> find it in the patches.
>> >
>> >Yes, there is nothing interesting in those patches, really. It's all
>> >internal to the driver. This set basically allows us to dedicate some
>> >queue pairs to high-speed communication with the FW (sending commands,
>> >populating/dumping eBPF and Flower tables).
>> >
>> >The eBPF maps and Flower offload patches should follow in coming weeks.
>>
>> Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
>> he wants to use the nic, right?
>
>Yes, right now we just expect the user to drop the FW file he wants
>to run into /lib/firmware/netronome. Driver will interrogate the
>firmware once loaded to see which "app" it is.
>
>> >> The configuration is asic-wide, should be done by a devlink parent
>> >> handle which was introduced for that exact purpose.
>> >>
>> >> Am I missing something? We need to sync in this. In mlxsw we need to do
>> >> some pre-netdev configuraton as well.
>> >
>> >For programmable NICs we still need to come up with some API for setting
>> >the target application/firmware name/adapter mode, but I don't think we
>> >have any more fine-grained parameters to set per-device just yet.
>> >
>> >I was thinking of either devlink, or perhaps, since I'm hearing people
>> >want those to be persistent/written into flash, the recent MTD
>> >discussion got me wondering if we should just expose the entire flash
>> >and teach ethtool to modify the vendor-specific parameter table
>> >directly in the flash. That seems like something that could quickly
>> >get out of hand, though :S
>>
>> Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
>> for the config like yours (switching the operation mode of nic), we
>> should have well defined user api. And since this is per-device, not
>> per-netdev, devlink should be the vahicle to carry this.
>
>Well defined API would be nice. Perhaps we will just start with
>something very simple and then extend it if need be. E.g. I'm hearing
>people would like to select not only the app but also the version of
>it... Yes, starting simple is probably the way forward.
>
>> >What were your plans with pre-netdev config?
>>
>> We need to pass come initial resource division. Generally the consensus
>> is to have these options exposed through devlink, let the user configure
>> them all and then to have a trigger that would cause driver
>> re-orchestration according to the new values. The flow would look like
>> this:
>>
>> -driver loads with defaults, inits hw and instantiates netdevs
>> -driver exposes config options via devlink
>> -user sets up the options
>> -user pushes the "go" trigger
>> -upon the trigger command, devlink calls the driver re-init callback
>> -driver shuts down the current instances, re-initializes hw,
>> re-instantiates the netdevs
>>
>> Makes sense?
>
>I like the idea of a "go"/apply/reload trigger and extending devlink.
>Do you plan on adding a way to persist the settings? I'm concerned NIC
>users may want to boot into the right mode once it's set, without
>reloads and reconfigs upon boot. Also is there going to be a way to
>query the pending/running config? Sounds like we may want to expose
>three value sets - persistent/default, running and pending/to be
>applied.
I don't think it is a good idea to introduce any kind of configuration
persistency in HW. I believe that user is the master and he has all
needed info. He can store it persistently, but it is up to him.
So basicaly during boot, we need the devlink configuration to happen
early on, before the netdevices get configured. udev? Not sure how
exactly to do this. Have to ask around :)
^ permalink raw reply [flat|nested] 30+ messages in thread
* RE: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 9:17 ` Jiri Pirko
@ 2017-06-06 9:35 ` Mintz, Yuval
2017-06-06 11:20 ` Jiri Pirko
2017-06-06 19:54 ` Jakub Kicinski
1 sibling, 1 reply; 30+ messages in thread
From: Mintz, Yuval @ 2017-06-06 9:35 UTC (permalink / raw)
To: Jiri Pirko, Jakub Kicinski; +Cc: netdev, oss-drivers
> >> >What were your plans with pre-netdev config?
> >>
> >> We need to pass come initial resource division. Generally the
> >> consensus is to have these options exposed through devlink, let the
> >> user configure them all and then to have a trigger that would cause
> >> driver re-orchestration according to the new values. The flow would
> >> look like
> >> this:
> >>
> >> -driver loads with defaults, inits hw and instantiates netdevs
> >> -driver exposes config options via devlink -user sets up the options
> >> -user pushes the "go" trigger -upon the trigger command, devlink
> >> calls the driver re-init callback -driver shuts down the current
> >> instances, re-initializes hw, re-instantiates the netdevs
> >>
> >> Makes sense?
> >
> >I like the idea of a "go"/apply/reload trigger and extending devlink.
> >Do you plan on adding a way to persist the settings? I'm concerned NIC
> >users may want to boot into the right mode once it's set, without
> >reloads and reconfigs upon boot. Also is there going to be a way to
> >query the pending/running config? Sounds like we may want to expose
> >three value sets - persistent/default, running and pending/to be
> >applied.
> I don't think it is a good idea to introduce any kind of configuration
> persistency in HW. I believe that user is the master and he has all needed
> info. He can store it persistently, but it is up to him.
>
> So basicaly during boot, we need the devlink configuration to happen early
> on, before the netdevices get configured. udev? Not sure how exactly to do
> this. Have to ask around :)
Thinking about use cases where we'd want information available at probe time,
it might have been even better to have it separated from the netdevice,
e.g., providing clients with node to configure (generic?) information on top of
their PCI nodes.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 9:35 ` Mintz, Yuval
@ 2017-06-06 11:20 ` Jiri Pirko
2017-06-07 17:48 ` Mintz, Yuval
0 siblings, 1 reply; 30+ messages in thread
From: Jiri Pirko @ 2017-06-06 11:20 UTC (permalink / raw)
To: Mintz, Yuval; +Cc: Jakub Kicinski, netdev, oss-drivers
Tue, Jun 06, 2017 at 11:35:08AM CEST, Yuval.Mintz@cavium.com wrote:
>
>> >> >What were your plans with pre-netdev config?
>> >>
>> >> We need to pass come initial resource division. Generally the
>> >> consensus is to have these options exposed through devlink, let the
>> >> user configure them all and then to have a trigger that would cause
>> >> driver re-orchestration according to the new values. The flow would
>> >> look like
>> >> this:
>> >>
>> >> -driver loads with defaults, inits hw and instantiates netdevs
>> >> -driver exposes config options via devlink -user sets up the options
>> >> -user pushes the "go" trigger -upon the trigger command, devlink
>> >> calls the driver re-init callback -driver shuts down the current
>> >> instances, re-initializes hw, re-instantiates the netdevs
>> >>
>> >> Makes sense?
>> >
>> >I like the idea of a "go"/apply/reload trigger and extending devlink.
>> >Do you plan on adding a way to persist the settings? I'm concerned NIC
>> >users may want to boot into the right mode once it's set, without
>> >reloads and reconfigs upon boot. Also is there going to be a way to
>> >query the pending/running config? Sounds like we may want to expose
>> >three value sets - persistent/default, running and pending/to be
>> >applied.
>
>> I don't think it is a good idea to introduce any kind of configuration
>> persistency in HW. I believe that user is the master and he has all needed
>> info. He can store it persistently, but it is up to him.
>>
>> So basicaly during boot, we need the devlink configuration to happen early
>> on, before the netdevices get configured. udev? Not sure how exactly to do
>> this. Have to ask around :)
>
>Thinking about use cases where we'd want information available at probe time,
>it might have been even better to have it separated from the netdevice,
>e.g., providing clients with node to configure (generic?) information on top of
>their PCI nodes.
Yuval, devlink is separated from the netdevices....
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 9:17 ` Jiri Pirko
2017-06-06 9:35 ` Mintz, Yuval
@ 2017-06-06 19:54 ` Jakub Kicinski
1 sibling, 0 replies; 30+ messages in thread
From: Jakub Kicinski @ 2017-06-06 19:54 UTC (permalink / raw)
To: Jiri Pirko; +Cc: netdev, oss-drivers
On Tue, 6 Jun 2017 11:17:57 +0200, Jiri Pirko wrote:
> >> >What were your plans with pre-netdev config?
> >>
> >> We need to pass come initial resource division. Generally the consensus
> >> is to have these options exposed through devlink, let the user configure
> >> them all and then to have a trigger that would cause driver
> >> re-orchestration according to the new values. The flow would look like
> >> this:
> >>
> >> -driver loads with defaults, inits hw and instantiates netdevs
> >> -driver exposes config options via devlink
> >> -user sets up the options
> >> -user pushes the "go" trigger
> >> -upon the trigger command, devlink calls the driver re-init callback
> >> -driver shuts down the current instances, re-initializes hw,
> >> re-instantiates the netdevs
> >>
> >> Makes sense?
> >
> >I like the idea of a "go"/apply/reload trigger and extending devlink.
> >Do you plan on adding a way to persist the settings? I'm concerned NIC
> >users may want to boot into the right mode once it's set, without
> >reloads and reconfigs upon boot. Also is there going to be a way to
> >query the pending/running config? Sounds like we may want to expose
> >three value sets - persistent/default, running and pending/to be
> >applied.
>
> I don't think it is a good idea to introduce any kind of configuration
> persistency in HW. I believe that user is the master and he has all
> needed info. He can store it persistently, but it is up to him.
>
> So basicaly during boot, we need the devlink configuration to happen
> early on, before the netdevices get configured. udev? Not sure how
> exactly to do this. Have to ask around :)
Happy to hear that. Now there is two of us, I'll try again with the
marketing dept :)
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
` (16 preceding siblings ...)
2017-06-06 6:16 ` [PATCH net-next 00/16] nfp: ctrl vNIC Jiri Pirko
@ 2017-06-07 16:52 ` David Miller
17 siblings, 0 replies; 30+ messages in thread
From: David Miller @ 2017-06-07 16:52 UTC (permalink / raw)
To: jakub.kicinski; +Cc: netdev, oss-drivers
From: Jakub Kicinski <jakub.kicinski@netronome.com>
Date: Mon, 5 Jun 2017 17:01:41 -0700
> This series adds the ability to use one vNIC as a control channel
> for passing messages to and from the application firmware. The
> implementation restructures the existing netdev vNIC code to be able
> to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
> are not visible to userspace (other than for dumping ring state), and
> since they don't have netdevs we use a tasklet for RX and simple skb
> list for TX queuing.
>
> Due to special status of the control vNIC we have to reshuffle the
> init code a bit to make sure control vNIC will be fully brought up
> (and therefore communication with app FW can happen) before any netdev
> or port is visible to user space.
>
> FW will designate which vNIC is supposed to be used as control one
> by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
> being prepended to control message, some prefer to look at queue ID
> to decide that something is a control message. Our implementation
> can cater to both.
>
> First two users of this code will be eBPF maps and flower offloads.
Ok, I read this over and also checked out your discussion with Jiri.
So far this looks OK to me, so series applied.
I look forward to seeing the eBPF maps and flower follow-on stuff.
^ permalink raw reply [flat|nested] 30+ messages in thread
* RE: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-06 11:20 ` Jiri Pirko
@ 2017-06-07 17:48 ` Mintz, Yuval
2017-06-08 5:38 ` Jiri Pirko
0 siblings, 1 reply; 30+ messages in thread
From: Mintz, Yuval @ 2017-06-07 17:48 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, netdev, oss-drivers
> >> >> >What were your plans with pre-netdev config?
> >> >>
> >> >> We need to pass come initial resource division. Generally the
> >> >> consensus is to have these options exposed through devlink, let
> >> >> the user configure them all and then to have a trigger that would
> >> >> cause driver re-orchestration according to the new values. The
> >> >> flow would look like
> >> >> this:
> >> >>
> >> >> -driver loads with defaults, inits hw and instantiates netdevs
> >> >> -driver exposes config options via devlink -user sets up the
> >> >> options -user pushes the "go" trigger -upon the trigger command,
> >> >> devlink calls the driver re-init callback -driver shuts down the
> >> >> current instances, re-initializes hw, re-instantiates the netdevs
> >> >>
> >> >> Makes sense?
> >> >
> >> >I like the idea of a "go"/apply/reload trigger and extending devlink.
> >> >Do you plan on adding a way to persist the settings? I'm concerned
> >> >NIC users may want to boot into the right mode once it's set,
> >> >without reloads and reconfigs upon boot. Also is there going to be
> >> >a way to query the pending/running config? Sounds like we may want
> >> >to expose three value sets - persistent/default, running and
> >> >pending/to be applied.
> >
> >> I don't think it is a good idea to introduce any kind of
> >> configuration persistency in HW. I believe that user is the master
> >> and he has all needed info. He can store it persistently, but it is up to
> him.
> >>
> >> So basicaly during boot, we need the devlink configuration to happen
> >> early on, before the netdevices get configured. udev? Not sure how
> >> exactly to do this. Have to ask around :)
> >
> >Thinking about use cases where we'd want information available at probe
> >time, it might have been even better to have it separated from the
> >netdevice, e.g., providing clients with node to configure (generic?)
> >information on top of their PCI nodes.
>
> Yuval, devlink is separated from the netdevices....
Separate from the netdevices, yes. But I think it's still a networking facility.
I.e., would it make sense creating devlink nodes for PCI devices?
Anyway, I don't think there's any *strong* need for what I was asking for;
It's simply that I was thinking of qed where there's quite a bit going on
during the pci probe, and thought how re-doing it can be avoided.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-07 17:48 ` Mintz, Yuval
@ 2017-06-08 5:38 ` Jiri Pirko
2017-06-08 6:33 ` Mintz, Yuval
0 siblings, 1 reply; 30+ messages in thread
From: Jiri Pirko @ 2017-06-08 5:38 UTC (permalink / raw)
To: Mintz, Yuval; +Cc: Jakub Kicinski, netdev, oss-drivers
Wed, Jun 07, 2017 at 07:48:08PM CEST, Yuval.Mintz@cavium.com wrote:
>> >> >> >What were your plans with pre-netdev config?
>> >> >>
>> >> >> We need to pass come initial resource division. Generally the
>> >> >> consensus is to have these options exposed through devlink, let
>> >> >> the user configure them all and then to have a trigger that would
>> >> >> cause driver re-orchestration according to the new values. The
>> >> >> flow would look like
>> >> >> this:
>> >> >>
>> >> >> -driver loads with defaults, inits hw and instantiates netdevs
>> >> >> -driver exposes config options via devlink -user sets up the
>> >> >> options -user pushes the "go" trigger -upon the trigger command,
>> >> >> devlink calls the driver re-init callback -driver shuts down the
>> >> >> current instances, re-initializes hw, re-instantiates the netdevs
>> >> >>
>> >> >> Makes sense?
>> >> >
>> >> >I like the idea of a "go"/apply/reload trigger and extending devlink.
>> >> >Do you plan on adding a way to persist the settings? I'm concerned
>> >> >NIC users may want to boot into the right mode once it's set,
>> >> >without reloads and reconfigs upon boot. Also is there going to be
>> >> >a way to query the pending/running config? Sounds like we may want
>> >> >to expose three value sets - persistent/default, running and
>> >> >pending/to be applied.
>> >
>> >> I don't think it is a good idea to introduce any kind of
>> >> configuration persistency in HW. I believe that user is the master
>> >> and he has all needed info. He can store it persistently, but it is up to
>> him.
>> >>
>> >> So basicaly during boot, we need the devlink configuration to happen
>> >> early on, before the netdevices get configured. udev? Not sure how
>> >> exactly to do this. Have to ask around :)
>> >
>> >Thinking about use cases where we'd want information available at probe
>> >time, it might have been even better to have it separated from the
>> >netdevice, e.g., providing clients with node to configure (generic?)
>> >information on top of their PCI nodes.
>>
>> Yuval, devlink is separated from the netdevices....
>
>Separate from the netdevices, yes. But I think it's still a networking facility.
>I.e., would it make sense creating devlink nodes for PCI devices?
True that devlink is placed in net/core. But from the very beginning, it
was designed to be multipurpose. So not only for net devices, but for
any device where is is suitable.
>
>Anyway, I don't think there's any *strong* need for what I was asking for;
>It's simply that I was thinking of qed where there's quite a bit going on
>during the pci probe, and thought how re-doing it can be avoided.
You can (should) register devlink early on way before netdevs.
^ permalink raw reply [flat|nested] 30+ messages in thread
* RE: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-08 5:38 ` Jiri Pirko
@ 2017-06-08 6:33 ` Mintz, Yuval
2017-06-08 6:38 ` Jiri Pirko
0 siblings, 1 reply; 30+ messages in thread
From: Mintz, Yuval @ 2017-06-08 6:33 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, netdev, oss-drivers
> >> >> >> >What were your plans with pre-netdev config?
> >> >> >>
> >> >> >> We need to pass come initial resource division. Generally the
> >> >> >> consensus is to have these options exposed through devlink, let
> >> >> >> the user configure them all and then to have a trigger that
> >> >> >> would cause driver re-orchestration according to the new
> >> >> >> values. The flow would look like
> >> >> >> this:
> >> >> >>
> >> >> >> -driver loads with defaults, inits hw and instantiates netdevs
> >> >> >> -driver exposes config options via devlink -user sets up the
> >> >> >> options -user pushes the "go" trigger -upon the trigger
> >> >> >> command, devlink calls the driver re-init callback -driver
> >> >> >> shuts down the current instances, re-initializes hw,
> >> >> >> re-instantiates the netdevs
> >> >> >>
> >> >> >> Makes sense?
> >> >> >
> >> >> >I like the idea of a "go"/apply/reload trigger and extending devlink.
> >> >> >Do you plan on adding a way to persist the settings? I'm
> >> >> >concerned NIC users may want to boot into the right mode once
> >> >> >it's set, without reloads and reconfigs upon boot. Also is there
> >> >> >going to be a way to query the pending/running config? Sounds
> >> >> >like we may want to expose three value sets - persistent/default,
> >> >> >running and pending/to be applied.
> >> >
> >> >> I don't think it is a good idea to introduce any kind of
> >> >> configuration persistency in HW. I believe that user is the master
> >> >> and he has all needed info. He can store it persistently, but it
> >> >> is up to
> >> him.
> >> >>
> >> >> So basicaly during boot, we need the devlink configuration to
> >> >> happen early on, before the netdevices get configured. udev? Not
> >> >> sure how exactly to do this. Have to ask around :)
> >> >
> >> >Thinking about use cases where we'd want information available at
> >> >probe time, it might have been even better to have it separated from
> >> >the netdevice, e.g., providing clients with node to configure
> >> >(generic?) information on top of their PCI nodes.
> >>
> >> Yuval, devlink is separated from the netdevices....
> >
> >Separate from the netdevices, yes. But I think it's still a networking facility.
> >I.e., would it make sense creating devlink nodes for PCI devices?
>
> True that devlink is placed in net/core. But from the very beginning, it was
> designed to be multipurpose. So not only for net devices, but for any device
> where is is suitable.
>
> >
> >Anyway, I don't think there's any *strong* need for what I was asking
> >for; It's simply that I was thinking of qed where there's quite a bit
> >going on during the pci probe, and thought how re-doing it can be avoided.
>
> You can (should) register devlink early on way before netdevs.
Given that I can't really register it sooner than the PCI probe,
how would shortly-before or way-before differ?
Assume you have some userspace daemon monitoring devlink and applying
user-provided configuration.
How could it reliably apply the configuration between devlink registration
and netdevice registeration?
The merit of the udev suggestion was that the sync. point was in userspace,
so user could always make sure that devlink rules apply before netdev state changes.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
2017-06-08 6:33 ` Mintz, Yuval
@ 2017-06-08 6:38 ` Jiri Pirko
0 siblings, 0 replies; 30+ messages in thread
From: Jiri Pirko @ 2017-06-08 6:38 UTC (permalink / raw)
To: Mintz, Yuval; +Cc: Jakub Kicinski, netdev, oss-drivers
Thu, Jun 08, 2017 at 08:33:11AM CEST, Yuval.Mintz@cavium.com wrote:
>> >> >> >> >What were your plans with pre-netdev config?
>> >> >> >>
>> >> >> >> We need to pass come initial resource division. Generally the
>> >> >> >> consensus is to have these options exposed through devlink, let
>> >> >> >> the user configure them all and then to have a trigger that
>> >> >> >> would cause driver re-orchestration according to the new
>> >> >> >> values. The flow would look like
>> >> >> >> this:
>> >> >> >>
>> >> >> >> -driver loads with defaults, inits hw and instantiates netdevs
>> >> >> >> -driver exposes config options via devlink -user sets up the
>> >> >> >> options -user pushes the "go" trigger -upon the trigger
>> >> >> >> command, devlink calls the driver re-init callback -driver
>> >> >> >> shuts down the current instances, re-initializes hw,
>> >> >> >> re-instantiates the netdevs
>> >> >> >>
>> >> >> >> Makes sense?
>> >> >> >
>> >> >> >I like the idea of a "go"/apply/reload trigger and extending devlink.
>> >> >> >Do you plan on adding a way to persist the settings? I'm
>> >> >> >concerned NIC users may want to boot into the right mode once
>> >> >> >it's set, without reloads and reconfigs upon boot. Also is there
>> >> >> >going to be a way to query the pending/running config? Sounds
>> >> >> >like we may want to expose three value sets - persistent/default,
>> >> >> >running and pending/to be applied.
>> >> >
>> >> >> I don't think it is a good idea to introduce any kind of
>> >> >> configuration persistency in HW. I believe that user is the master
>> >> >> and he has all needed info. He can store it persistently, but it
>> >> >> is up to
>> >> him.
>> >> >>
>> >> >> So basicaly during boot, we need the devlink configuration to
>> >> >> happen early on, before the netdevices get configured. udev? Not
>> >> >> sure how exactly to do this. Have to ask around :)
>> >> >
>> >> >Thinking about use cases where we'd want information available at
>> >> >probe time, it might have been even better to have it separated from
>> >> >the netdevice, e.g., providing clients with node to configure
>> >> >(generic?) information on top of their PCI nodes.
>> >>
>> >> Yuval, devlink is separated from the netdevices....
>> >
>> >Separate from the netdevices, yes. But I think it's still a networking facility.
>> >I.e., would it make sense creating devlink nodes for PCI devices?
>>
>> True that devlink is placed in net/core. But from the very beginning, it was
>> designed to be multipurpose. So not only for net devices, but for any device
>> where is is suitable.
>>
>> >
>> >Anyway, I don't think there's any *strong* need for what I was asking
>> >for; It's simply that I was thinking of qed where there's quite a bit
>> >going on during the pci probe, and thought how re-doing it can be avoided.
>>
>> You can (should) register devlink early on way before netdevs.
>
>Given that I can't really register it sooner than the PCI probe,
>how would shortly-before or way-before differ?
Yeah. Pci probe is the limit :)
>
>Assume you have some userspace daemon monitoring devlink and applying
>user-provided configuration.
>How could it reliably apply the configuration between devlink registration
>and netdevice registeration?
You cannot assume that. That is why I suggested devlink config and then
a trigger to re-initialize the hw and the netdevices.
>
>The merit of the udev suggestion was that the sync. point was in userspace,
>so user could always make sure that devlink rules apply before netdev state changes.
>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2017-06-08 6:38 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-06 0:01 [PATCH net-next 00/16] nfp: ctrl vNIC Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 01/16] nfp: reorder open and close functions Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 02/16] nfp: split out the allocation part of open Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 03/16] nfp: reuse ring free code on close Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 04/16] nfp: move nfp_net_vecs_init() Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 05/16] nfp: prepare print macros for use without netdev Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 06/16] nfp: make sure debug accesses don't depend on netdevs Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 07/16] nfp: allow allocation and initialization of netdev-less vNICs Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 08/16] nfp: prepare config and enable for working without netdevs Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 09/16] nfp: add control vNIC datapath Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 10/16] nfp: make vNIC ctrl memory mapping function reusable Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 11/16] nfp: map all queue controllers at once Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 12/16] nfp: don't clutter init code passing fw_ver around Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 13/16] nfp: slice the netdev spawning function Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 14/16] nfp: allow non-equal distribution of IRQs Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 15/16] nfp: create control vNICs and wire up rx/tx Jakub Kicinski
2017-06-06 0:01 ` [PATCH net-next 16/16] nfp: advertise support for NFD ABI 0.5 Jakub Kicinski
2017-06-06 6:16 ` [PATCH net-next 00/16] nfp: ctrl vNIC Jiri Pirko
2017-06-06 7:21 ` Jakub Kicinski
2017-06-06 8:23 ` Jiri Pirko
2017-06-06 9:09 ` Jakub Kicinski
2017-06-06 9:17 ` Jiri Pirko
2017-06-06 9:35 ` Mintz, Yuval
2017-06-06 11:20 ` Jiri Pirko
2017-06-07 17:48 ` Mintz, Yuval
2017-06-08 5:38 ` Jiri Pirko
2017-06-08 6:33 ` Mintz, Yuval
2017-06-08 6:38 ` Jiri Pirko
2017-06-06 19:54 ` Jakub Kicinski
2017-06-07 16:52 ` David Miller
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.