[PATCH 0/5] add device removal event

[PATCH 0/5] add device removal event

[Gaetan Rivet]

This new event represents the sudden removal of a device from its bus.
The underlying resources exposed by the bus are expected not to be available
anymore. The application should thus be able to react and possibly clean up
related resources that it reserved for the removed device.

This event is different from the current hotplug API available in the DPDK
for two reasons:

1. It is a reactive design: the application reacts to a device that has been
   removed instead of removing a device from its pool.

2. The event itself is going further than the current detaching of a device
   from a DPDK application. If the bus is a hardware one, it is expected of the
   underlying resources to not be available anymore.

This series adds a new event type to ethdev and implements it in mlx4.
Testpmd is also updated to report all asynchronous ethdev events including this
one for testing purposes and as a practical usage example.

This series depends on the series titled
[PATCH 1/2] net/mlx4: split the definitions to the header file

Gaetan Rivet (5):
  ethdev: introduce device removal event
  net/mlx4: device removal event support
  app/testpmd: generic event handler
  app/testpmd: request link status interrupt
  app/testpmd: request device removal interrupt

 app/test-pmd/parameters.c                       |   8 +
 app/test-pmd/testpmd.c                          | 103 ++++++++++
 app/test-pmd/testpmd.h                          |   2 +
 doc/guides/nics/features/default.ini            |   1 +
 doc/guides/nics/features/mlx4.ini               |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  21 +-
 drivers/net/mlx4/mlx4.c                         | 258 ++++++++++++++++++++----
 drivers/net/mlx4/mlx4.h                         |   1 +
 lib/librte_eal/common/include/rte_pci.h         |   2 +
 lib/librte_ether/rte_ethdev.c                   |  13 +-
 lib/librte_ether/rte_ethdev.h                   |   9 +-
 11 files changed, 375 insertions(+), 44 deletions(-)

-- 
2.1.4

[PATCH 1/5] ethdev: introduce device removal event

[Gaetan Rivet]

This new API allows reacting to a device removal.
A device removal is the sudden disappearance of a device from its
bus.

PMDs implementing support for this notification guarantee that the removal
of the underlying device does not incur a risk to the application.

In particular, Rx/Tx bursts and all other functions can still be called
(albeit likely returning errors) without triggering a crash, irrespective
of an application handling this event.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 doc/guides/nics/features/default.ini            |  1 +
 doc/guides/prog_guide/env_abstraction_layer.rst | 21 +++++++++++++++++++--
 lib/librte_eal/common/include/rte_pci.h         |  2 ++
 lib/librte_ether/rte_ethdev.c                   | 13 +++++++++----
 lib/librte_ether/rte_ethdev.h                   |  9 +++++++--
 5 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini
index 9e363ff..7bb56ec 100644
--- a/doc/guides/nics/features/default.ini
+++ b/doc/guides/nics/features/default.ini
@@ -9,6 +9,7 @@
 Speed capabilities   =
 Link status          =
 Link status event    =
+Removal event        =
 Queue status event   =
 Rx interrupt         =
 Queue start/stop     =
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 10a10a8..f72674f 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -178,8 +178,8 @@ The EAL also allows timed callbacks to be used in the same way as for NIC interr
 
 .. note::
 
-    In DPDK PMD, the only interrupts handled by the dedicated host thread are those for link status change,
-    i.e. link up and link down notification.
+    In DPDK PMD, the only interrupts handled by the dedicated host thread are those for link status change
+    (link up and link down notification) and for sudden device removal.
 
 
 + RX Interrupt Event
@@ -207,6 +207,23 @@ The eth_dev driver takes responsibility to program the latter mapping.
 The RX interrupt are controlled/enabled/disabled by ethdev APIs - 'rte_eth_dev_rx_intr_*'. They return failure if the PMD
 hasn't support them yet. The intr_conf.rxq flag is used to turn on the capability of RX interrupt per device.
 
++ Device Removal Event
+
+This event is triggered by a device being removed at a bus level. Its
+underlying resources may have been made unavailable (i.e. PCI mappings
+unmapped). The PMD must make sure that on such occurrence, the application can
+still safely use its callbacks.
+
+This event can be subscribed to in the same way one would subscribe to a link
+status change event. The execution context is thus the same, i.e. it is the
+dedicated interrupt host thread.
+
+Considering this, it is likely that an application would want to close a
+device having emitted a Device Removal Event. In such case, calling
+``rte_eth_dev_close()`` can trigger it to unregister its own Device Removal Event
+callback. Care must be taken not to close the device from the interrupt handler
+context. It is necessary to reschedule such closing operation.
+
 Blacklisting
 ~~~~~~~~~~~~
 
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index 8557e47..4a67883 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -216,6 +216,8 @@ struct rte_pci_driver {
 #define RTE_PCI_DRV_NEED_MAPPING 0x0001
 /** Device driver supports link state interrupt */
 #define RTE_PCI_DRV_INTR_LSC	0x0008
+/** Device driver supports device removal interrupt */
+#define RTE_PCI_DRV_INTR_RMV 0x0010
 
 /**
  * A structure describing a PCI mapping.
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 3a52d0a..6c4b796 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -848,16 +848,19 @@ rte_eth_dev_configure(uint8_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
 		return -EINVAL;
 	}
 
-	/*
-	 * If link state interrupt is enabled, check that the
-	 * device supports it.
-	 */
+	/* Check that the device supports requested interrupts */
 	if ((dev_conf->intr_conf.lsc == 1) &&
 		(!(dev->data->dev_flags & RTE_ETH_DEV_INTR_LSC))) {
 			RTE_PMD_DEBUG_TRACE("driver %s does not support lsc\n",
 					dev->data->drv_name);
 			return -EINVAL;
 	}
+	if ((dev_conf->intr_conf.rmv == 1) &&
+	    (!(dev->data->dev_flags & RTE_ETH_DEV_INTR_RMV))) {
+		RTE_PMD_DEBUG_TRACE("driver %s does not support rmv\n",
+				    dev->data->drv_name);
+		return -EINVAL;
+	}
 
 	/*
 	 * If jumbo frames are enabled, check that the maximum RX packet
@@ -3222,6 +3225,8 @@ rte_eth_copy_pci_info(struct rte_eth_dev *eth_dev, struct rte_pci_device *pci_de
 	eth_dev->data->dev_flags = 0;
 	if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_LSC)
 		eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC;
+	if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_RMV)
+		eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV;
 
 	eth_dev->data->kdrv = pci_dev->kdrv;
 	eth_dev->data->numa_node = pci_dev->device.numa_node;
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index bdad81b..ed99660 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -816,9 +816,11 @@ struct rte_eth_udp_tunnel {
  */
 struct rte_intr_conf {
 	/** enable/disable lsc interrupt. 0 (default) - disable, 1 enable */
-	uint16_t lsc;
+	uint32_t lsc:1;
 	/** enable/disable rxq interrupt. 0 (default) - disable, 1 enable */
-	uint16_t rxq;
+	uint32_t rxq:1;
+	/** enable/disable rmv interrupt. 0 (default) - disable, 1 enable */
+	uint32_t rmv:1;
 };
 
 /**
@@ -1722,6 +1724,8 @@ struct rte_eth_dev_data {
 #define RTE_ETH_DEV_INTR_LSC     0x0002
 /** Device is a bonded slave */
 #define RTE_ETH_DEV_BONDED_SLAVE 0x0004
+/** Device supports device removal interrupt */
+#define RTE_ETH_DEV_INTR_RMV     0x0008
 
 /**
  * @internal
@@ -3212,6 +3216,7 @@ enum rte_eth_event_type {
 			/**< reset interrupt event, sent to VF on PF reset */
 	RTE_ETH_EVENT_VF_MBOX,  /**< message from the VF received by PF */
 	RTE_ETH_EVENT_MACSEC,   /**< MACsec offload related event */
+	RTE_ETH_EVENT_INTR_RMV, /**< device removal event */
 	RTE_ETH_EVENT_MAX       /**< max value of this enum */
 };
 
-- 
2.1.4

[PATCH 2/5] net/mlx4: device removal event support

[Gaetan Rivet]

Extend the LSC event handling to support the device removal as well. The
Verbs library will send several related events, that can conflict
with the LSC event itself.

The event handling has thus been made capable of receiving and signaling
several event types at once.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 doc/guides/nics/features/mlx4.ini |   1 +
 drivers/net/mlx4/mlx4.c           | 258 ++++++++++++++++++++++++++++++++------
 drivers/net/mlx4/mlx4.h           |   1 +
 3 files changed, 224 insertions(+), 36 deletions(-)

diff --git a/doc/guides/nics/features/mlx4.ini b/doc/guides/nics/features/mlx4.ini
index c9828f7..285f0ec 100644
--- a/doc/guides/nics/features/mlx4.ini
+++ b/doc/guides/nics/features/mlx4.ini
@@ -6,6 +6,7 @@
 [Features]
 Link status          = Y
 Link status event    = Y
+Removal event        = Y
 Queue start/stop     = Y
 MTU update           = Y
 Jumbo frame          = Y
diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index c4b9fbd..db37711 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -3931,9 +3931,15 @@ mlx4_rx_queue_release(void *dpdk_rxq)
 	priv_unlock(priv);
 }
 
-static void
+static int
 priv_dev_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
 
+static int
+priv_dev_removal_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
+
+static int
+priv_dev_link_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
+
 /**
  * DPDK callback to start the device.
  *
@@ -3986,7 +3992,18 @@ mlx4_dev_start(struct rte_eth_dev *dev)
 		     (void *)dev, strerror(ret));
 		goto err;
 	} while ((--r) && ((rxq = (*priv->rxqs)[++i]), i));
-	priv_dev_interrupt_handler_install(priv, dev);
+	ret = priv_dev_link_interrupt_handler_install(priv, dev);
+	if (ret) {
+		ERROR("%p: LSC handler install failed",
+		     (void *)dev);
+		goto err;
+	}
+	ret = priv_dev_removal_interrupt_handler_install(priv, dev);
+	if (ret) {
+		ERROR("%p: RMV handler install failed",
+		     (void *)dev);
+		goto err;
+	}
 	ret = mlx4_priv_flow_start(priv);
 	if (ret) {
 		ERROR("%p: flow start failed: %s",
@@ -4105,9 +4122,16 @@ removed_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
 	return 0;
 }
 
-static void
+static int
 priv_dev_interrupt_handler_uninstall(struct priv *, struct rte_eth_dev *);
 
+static int
+priv_dev_removal_interrupt_handler_uninstall(struct priv *,
+					     struct rte_eth_dev *);
+
+static int
+priv_dev_link_interrupt_handler_uninstall(struct priv *, struct rte_eth_dev *);
+
 /**
  * DPDK callback to close the device.
  *
@@ -4170,7 +4194,8 @@ mlx4_dev_close(struct rte_eth_dev *dev)
 		claim_zero(ibv_close_device(priv->ctx));
 	} else
 		assert(priv->ctx == NULL);
-	priv_dev_interrupt_handler_uninstall(priv, dev);
+	priv_dev_removal_interrupt_handler_uninstall(priv, dev);
+	priv_dev_link_interrupt_handler_uninstall(priv, dev);
 	priv_unlock(priv);
 	memset(priv, 0, sizeof(*priv));
 }
@@ -4730,6 +4755,10 @@ mlx4_link_update(struct rte_eth_dev *dev, int wait_to_complete)
 	return -1;
 }
 
+static int
+mlx4_ibv_device_to_pci_addr(const struct ibv_device *device,
+			    struct rte_pci_addr *pci_addr);
+
 /**
  * DPDK callback to change the MTU.
  *
@@ -5261,32 +5290,41 @@ static void
 mlx4_dev_interrupt_handler(struct rte_intr_handle *, void *);
 
 /**
- * Link status handler.
+ * Link/device status handler.
  *
  * @param priv
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @param events
+ *   Pointer to event flags holder.
  *
  * @return
- *   Nonzero if the callback process can be called immediately.
+ *   Number of events
  */
 static int
-priv_dev_link_status_handler(struct priv *priv, struct rte_eth_dev *dev)
+priv_dev_status_handler(struct priv *priv, struct rte_eth_dev *dev,
+			uint32_t *events)
 {
 	struct ibv_async_event event;
 	int port_change = 0;
 	int ret = 0;
 
+	*events = 0;
 	/* Read all message and acknowledge them. */
 	for (;;) {
 		if (ibv_get_async_event(priv->ctx, &event))
 			break;
-
-		if (event.event_type == IBV_EVENT_PORT_ACTIVE ||
-		    event.event_type == IBV_EVENT_PORT_ERR)
+		if ((event.event_type == IBV_EVENT_PORT_ACTIVE ||
+		     event.event_type == IBV_EVENT_PORT_ERR) &&
+		    (priv->intr_conf.lsc == 1)) {
 			port_change = 1;
-		else
+			ret++;
+		} else if (event.event_type == IBV_EVENT_DEVICE_FATAL &&
+			   priv->intr_conf.rmv == 1) {
+			*events |= (1 << RTE_ETH_EVENT_INTR_RMV);
+			ret++;
+		} else
 			DEBUG("event type %d on port %d not handled",
 			      event.event_type, event.element.port_num);
 		ibv_ack_async_event(&event);
@@ -5304,8 +5342,9 @@ priv_dev_link_status_handler(struct priv *priv, struct rte_eth_dev *dev)
 			rte_eal_alarm_set(MLX4_ALARM_TIMEOUT_US,
 					  mlx4_dev_link_status_handler,
 					  dev);
-		} else
-			ret = 1;
+		} else {
+			*events |= (1 << RTE_ETH_EVENT_INTR_LSC);
+		}
 	}
 	return ret;
 }
@@ -5321,13 +5360,14 @@ mlx4_dev_link_status_handler(void *arg)
 {
 	struct rte_eth_dev *dev = arg;
 	struct priv *priv = dev->data->dev_private;
+	uint32_t events;
 	int ret;
 
 	priv_lock(priv);
 	assert(priv->pending_alarm == 1);
-	ret = priv_dev_link_status_handler(priv, dev);
+	ret = priv_dev_status_handler(priv, dev, &events);
 	priv_unlock(priv);
-	if (ret)
+	if (ret > 0 && events & (1 << RTE_ETH_EVENT_INTR_LSC))
 		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
 }
 
@@ -5345,13 +5385,27 @@ mlx4_dev_interrupt_handler(struct rte_intr_handle *intr_handle, void *cb_arg)
 	struct rte_eth_dev *dev = cb_arg;
 	struct priv *priv = dev->data->dev_private;
 	int ret;
+	uint32_t ev;
+	int i;
 
 	(void)intr_handle;
 	priv_lock(priv);
-	ret = priv_dev_link_status_handler(priv, dev);
+	ret = priv_dev_status_handler(priv, dev, &ev);
 	priv_unlock(priv);
-	if (ret)
-		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	if (ret > 0) {
+		for (i = RTE_ETH_EVENT_UNKNOWN;
+		     i < RTE_ETH_EVENT_MAX;
+		     i++) {
+			if (ev & (1 << i)) {
+				ev &= ~(1 << i);
+				_rte_eth_dev_callback_process(dev, i, NULL);
+				ret--;
+			}
+		}
+		if (ret)
+			WARN("%d event%s not processed", ret,
+			     (ret > 1 ? "s were" : " was"));
+	}
 }
 
 /**
@@ -5361,20 +5415,30 @@ mlx4_dev_interrupt_handler(struct rte_intr_handle *intr_handle, void *cb_arg)
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative errno value on failure.
  */
-static void
+static int
 priv_dev_interrupt_handler_uninstall(struct priv *priv, struct rte_eth_dev *dev)
 {
-	if (!dev->data->dev_conf.intr_conf.lsc)
-		return;
-	rte_intr_callback_unregister(&priv->intr_handle,
-				     mlx4_dev_interrupt_handler,
-				     dev);
-	if (priv->pending_alarm)
-		rte_eal_alarm_cancel(mlx4_dev_link_status_handler, dev);
-	priv->pending_alarm = 0;
+	int ret;
+
+	if (priv->intr_conf.lsc ||
+	    priv->intr_conf.rmv)
+		return 0;
+	ret = rte_intr_callback_unregister(&priv->intr_handle,
+					   mlx4_dev_interrupt_handler,
+					   dev);
+	if (ret < 0) {
+		ERROR("rte_intr_callback_unregister failed with %d"
+		      "%s%s%s", ret,
+		      (errno ? " (errno: " : ""),
+		      (errno ? strerror(errno) : ""),
+		      (errno ? ")" : ""));
+	}
 	priv->intr_handle.fd = 0;
 	priv->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
+	return ret;
 }
 
 /**
@@ -5384,27 +5448,148 @@ priv_dev_interrupt_handler_uninstall(struct priv *priv, struct rte_eth_dev *dev)
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative errno value on failure.
  */
-static void
-priv_dev_interrupt_handler_install(struct priv *priv, struct rte_eth_dev *dev)
+static int
+priv_dev_interrupt_handler_install(struct priv *priv,
+				   struct rte_eth_dev *dev)
 {
-	int rc, flags;
+	int flags;
+	int rc;
 
-	if (!dev->data->dev_conf.intr_conf.lsc)
-		return;
+	/* Check whether the interrupt handler has already been installed
+	 * for either type of interrupt
+	 */
+	if (priv->intr_conf.lsc &&
+	    priv->intr_conf.rmv &&
+	    priv->intr_handle.fd)
+		return 0;
 	assert(priv->ctx->async_fd > 0);
 	flags = fcntl(priv->ctx->async_fd, F_GETFL);
 	rc = fcntl(priv->ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
 	if (rc < 0) {
 		INFO("failed to change file descriptor async event queue");
 		dev->data->dev_conf.intr_conf.lsc = 0;
+		dev->data->dev_conf.intr_conf.rmv = 0;
+		return -errno;
 	} else {
 		priv->intr_handle.fd = priv->ctx->async_fd;
 		priv->intr_handle.type = RTE_INTR_HANDLE_EXT;
-		rte_intr_callback_register(&priv->intr_handle,
-					   mlx4_dev_interrupt_handler,
-					   dev);
+		rc = rte_intr_callback_register(&priv->intr_handle,
+						 mlx4_dev_interrupt_handler,
+						 dev);
+		if (rc) {
+			ERROR("rte_intr_callback_register failed "
+			      " (errno: %s)", strerror(errno));
+			return rc;
+		}
 	}
+	return 0;
+}
+
+/**
+ * Uninstall interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_removal_interrupt_handler_uninstall(struct priv *priv,
+					    struct rte_eth_dev *dev)
+{
+	if (dev->data->dev_conf.intr_conf.rmv) {
+		priv->intr_conf.rmv = 0;
+		return priv_dev_interrupt_handler_uninstall(priv, dev);
+	}
+	return 0;
+}
+
+/**
+ * Uninstall interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error,
+ */
+static int
+priv_dev_link_interrupt_handler_uninstall(struct priv *priv,
+					  struct rte_eth_dev *dev)
+{
+	int ret = 0;
+
+	if (dev->data->dev_conf.intr_conf.lsc) {
+		priv->intr_conf.lsc = 0;
+		ret = priv_dev_interrupt_handler_uninstall(priv, dev);
+		if (ret)
+			return ret;
+	}
+	if (priv->pending_alarm)
+		if (rte_eal_alarm_cancel(mlx4_dev_link_status_handler,
+					 dev)) {
+			ERROR("rte_eal_alarm_cancel failed "
+			      " (errno: %s)", strerror(rte_errno));
+			return -rte_errno;
+		}
+	priv->pending_alarm = 0;
+	return 0;
+}
+
+/**
+ * Install link interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_link_interrupt_handler_install(struct priv *priv,
+					struct rte_eth_dev *dev)
+{
+	int ret;
+
+	if (dev->data->dev_conf.intr_conf.lsc) {
+		ret = priv_dev_interrupt_handler_install(priv, dev);
+		if (ret)
+			return ret;
+		priv->intr_conf.lsc = 1;
+	}
+	return 0;
+}
+
+/**
+ * Install removal interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_removal_interrupt_handler_install(struct priv *priv,
+					   struct rte_eth_dev *dev)
+{
+	int ret;
+
+	if (dev->data->dev_conf.intr_conf.rmv) {
+		ret = priv_dev_interrupt_handler_install(priv, dev);
+		if (ret)
+			return ret;
+		priv->intr_conf.rmv = 1;
+	}
+	return 0;
 }
 
 /**
@@ -5889,7 +6074,8 @@ static struct eth_driver mlx4_driver = {
 		},
 		.id_table = mlx4_pci_id_map,
 		.probe = mlx4_pci_probe,
-		.drv_flags = RTE_PCI_DRV_INTR_LSC,
+		.drv_flags = RTE_PCI_DRV_INTR_LSC |
+			     RTE_PCI_DRV_INTR_RMV,
 	},
 	.dev_private_size = sizeof(struct priv)
 };
diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
index e9659eb..063df5f 100644
--- a/drivers/net/mlx4/mlx4.h
+++ b/drivers/net/mlx4/mlx4.h
@@ -356,6 +356,7 @@ struct priv {
 	struct rxq *(*rxqs)[]; /* RX queues. */
 	struct txq *(*txqs)[]; /* TX queues. */
 	struct rte_intr_handle intr_handle; /* Interrupt handler. */
+	struct rte_intr_conf intr_conf; /* Active interrupt configuration. */
 	LIST_HEAD(mlx4_flows, rte_flow) flows;
 	rte_spinlock_t lock; /* Lock for control functions. */
 };
-- 
2.1.4

[PATCH 3/5] app/testpmd: generic event handler

[Gaetan Rivet]

This is a rather simple handler that prints a message with the name of
the current event. It can be used to check PMD callback registration and
triggers.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 app/test-pmd/testpmd.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index b8ec1fe..f2b67d5 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -326,6 +326,9 @@ unsigned max_socket = 0;
 /* Forward function declarations */
 static void map_port_queue_stats_mapping_registers(uint8_t pi, struct rte_port *port);
 static void check_all_ports_link_status(uint32_t port_mask);
+static void eth_event_callback(uint8_t port_id,
+			       enum rte_eth_event_type type,
+			       void *param);
 
 /*
  * Check if all the ports are started.
@@ -1299,6 +1302,7 @@ start_port(portid_t pid)
 	queueid_t qi;
 	struct rte_port *port;
 	struct ether_addr mac_addr;
+	enum rte_eth_event_type event_type;
 
 	if (port_id_is_invalid(pid, ENABLED_WARN))
 		return 0;
@@ -1410,6 +1414,21 @@ start_port(portid_t pid)
 				return -1;
 			}
 		}
+
+		for (event_type = RTE_ETH_EVENT_UNKNOWN;
+		     event_type < RTE_ETH_EVENT_MAX;
+		     event_type++) {
+			diag = rte_eth_dev_callback_register(pi,
+							event_type,
+							eth_event_callback,
+							NULL);
+			if (diag) {
+				printf("Failed to setup even callback for event %d\n",
+					event_type);
+				return -1;
+			}
+		}
+
 		/* start port */
 		if (rte_eth_dev_start(pi) < 0) {
 			printf("Fail to start port %d\n", pi);
@@ -1682,6 +1701,34 @@ check_all_ports_link_status(uint32_t port_mask)
 	}
 }
 
+/* This function is used by the interrupt thread */
+static void
+eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
+{
+	static const char * const event_desc[] = {
+		[RTE_ETH_EVENT_UNKNOWN] = "Unknown",
+		[RTE_ETH_EVENT_INTR_LSC] = "LSC",
+		[RTE_ETH_EVENT_QUEUE_STATE] = "Queue state",
+		[RTE_ETH_EVENT_INTR_RESET] = "Interrupt reset",
+		[RTE_ETH_EVENT_VF_MBOX] = "VF Mbox",
+		[RTE_ETH_EVENT_MACSEC] = "MACsec",
+		[RTE_ETH_EVENT_INTR_RMV] = "device removal",
+		[RTE_ETH_EVENT_MAX] = NULL,
+	};
+
+	RTE_SET_USED(param);
+
+	if (type >= RTE_ETH_EVENT_MAX) {
+		fprintf(stderr, "\nPort %" PRIu8 ": %s called upon invalid event %d\n",
+			port_id, __func__, type);
+		fflush(stderr);
+	} else {
+		printf("\nPort %" PRIu8 ": %s event\n", port_id,
+			event_desc[type]);
+		fflush(stdout);
+	}
+}
+
 static int
 set_tx_queue_stats_mapping_registers(uint8_t port_id, struct rte_port *port)
 {
-- 
2.1.4

[PATCH 4/5] app/testpmd: request link status interrupt

[Gaetan Rivet]

For drivers supporting the LSC event, enable it.
This allows to test LSC event support.

Add the --no-lsc-interrupt parameter to explicitly disable the link status
change interrupt.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 app/test-pmd/parameters.c |  4 ++++
 app/test-pmd/testpmd.c    | 13 +++++++++++++
 app/test-pmd/testpmd.h    |  1 +
 3 files changed, 18 insertions(+)

diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 67d8bf2..89231be 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -196,6 +196,7 @@ usage(char* progname)
 		" or total packet length.\n");
 	printf("  --disable-link-check: disable check on link status when "
 	       "starting/stopping ports.\n");
+	printf("  --no-lsc-interrupt: disable link status change interrupt.\n");
 }
 
 #ifdef RTE_LIBRTE_CMDLINE
@@ -561,6 +562,7 @@ launch_args_parse(int argc, char** argv)
 		{ "no-flush-rx",	0, 0, 0 },
 		{ "txpkts",			1, 0, 0 },
 		{ "disable-link-check",		0, 0, 0 },
+		{ "no-lsc-interrupt",		0, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -978,6 +980,8 @@ launch_args_parse(int argc, char** argv)
 				no_flush_rx = 1;
 			if (!strcmp(lgopts[opt_idx].name, "disable-link-check"))
 				no_link_check = 1;
+			if (!strcmp(lgopts[opt_idx].name, "no-lsc-interrupt"))
+				lsc_interrupt = 0;
 
 			break;
 		case 'h':
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index f2b67d5..8bd1285 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -263,6 +263,11 @@ uint8_t no_flush_rx = 0; /* flush by default */
 uint8_t no_link_check = 0; /* check by default */
 
 /*
+ * Enable link status change notification
+ */
+uint8_t lsc_interrupt = 1; /* enabled by default */
+
+/*
  * NIC bypass mode configuration options.
  */
 #ifdef RTE_NIC_BYPASS
@@ -1698,6 +1703,9 @@ check_all_ports_link_status(uint32_t port_mask)
 		if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
 			print_flag = 1;
 		}
+
+		if (lsc_interrupt)
+			break;
 	}
 }
 
@@ -1892,6 +1900,11 @@ init_port_config(void)
 #ifdef RTE_NIC_BYPASS
 		rte_eth_dev_bypass_init(pid);
 #endif
+
+		if (lsc_interrupt &&
+		    (rte_eth_devices[pid].data->dev_flags &
+		     RTE_ETH_DEV_INTR_LSC))
+			port->dev_conf.intr_conf.lsc = 1;
 	}
 }
 
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index e1e4939..77fa7ee 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -305,6 +305,7 @@ extern uint8_t no_flush_rx; /**<set by "--no-flush-rx" parameter */
 extern uint8_t  mp_anon; /**< set by "--mp-anon" parameter */
 extern uint8_t no_link_check; /**<set by "--disable-link-check" parameter */
 extern volatile int test_done; /* stop packet forwarding when set to 1. */
+extern uint8_t lsc_interrupt; /**< disabled by "--no-lsc-interrupt" parameter */
 
 #ifdef RTE_NIC_BYPASS
 extern uint32_t bypass_timeout; /**< Store the NIC bypass watchdog timeout */
-- 
2.1.4

[PATCH 5/5] app/testpmd: request device removal interrupt

[Gaetan Rivet]

Enable device removal event for PMD supporting it.
Add the --no-rmv-interrupt parameter to explicitly disable it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 app/test-pmd/parameters.c |  4 ++++
 app/test-pmd/testpmd.c    | 43 +++++++++++++++++++++++++++++++++++++++++++
 app/test-pmd/testpmd.h    |  1 +
 3 files changed, 48 insertions(+)

diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 89231be..792e2e9 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -197,6 +197,7 @@ usage(char* progname)
 	printf("  --disable-link-check: disable check on link status when "
 	       "starting/stopping ports.\n");
 	printf("  --no-lsc-interrupt: disable link status change interrupt.\n");
+	printf("  --no-rmv-interrupt: disable device removal interrupt.");
 }
 
 #ifdef RTE_LIBRTE_CMDLINE
@@ -563,6 +564,7 @@ launch_args_parse(int argc, char** argv)
 		{ "txpkts",			1, 0, 0 },
 		{ "disable-link-check",		0, 0, 0 },
 		{ "no-lsc-interrupt",		0, 0, 0 },
+		{ "no-rmv-interrupt",		0, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -982,6 +984,8 @@ launch_args_parse(int argc, char** argv)
 				no_link_check = 1;
 			if (!strcmp(lgopts[opt_idx].name, "no-lsc-interrupt"))
 				lsc_interrupt = 0;
+			if (!strcmp(lgopts[opt_idx].name, "no-rmv-interrupt"))
+				rmv_interrupt = 0;
 
 			break;
 		case 'h':
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 8bd1285..de1bb12 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -59,6 +59,7 @@
 #include <rte_memzone.h>
 #include <rte_launch.h>
 #include <rte_eal.h>
+#include <rte_alarm.h>
 #include <rte_per_lcore.h>
 #include <rte_lcore.h>
 #include <rte_atomic.h>
@@ -268,6 +269,11 @@ uint8_t no_link_check = 0; /* check by default */
 uint8_t lsc_interrupt = 1; /* enabled by default */
 
 /*
+ * Enable device removal notification.
+ */
+uint8_t rmv_interrupt = 1; /* enabled by default */
+
+/*
  * NIC bypass mode configuration options.
  */
 #ifdef RTE_NIC_BYPASS
@@ -1709,6 +1715,29 @@ check_all_ports_link_status(uint32_t port_mask)
 	}
 }
 
+static void
+rmv_event_callback(void *arg)
+{
+	struct rte_eth_dev *dev;
+	struct rte_devargs *da;
+	char name[32] = "";
+	uint8_t port_id = (intptr_t)arg;
+
+	RTE_ETH_VALID_PORTID_OR_RET(port_id);
+	dev = &rte_eth_devices[port_id];
+	da = dev->device->devargs;
+
+	stop_port(port_id);
+	close_port(port_id);
+	if (da->type == RTE_DEVTYPE_VIRTUAL)
+		snprintf(name, sizeof(name), "%s", da->virt.drv_name);
+	else if (da->type == RTE_DEVTYPE_WHITELISTED_PCI)
+		rte_eal_pci_device_name(&da->pci.addr, name, sizeof(name));
+	printf("removing device %s\n", name);
+	rte_eal_dev_detach(name);
+	dev->state = RTE_ETH_DEV_UNUSED;
+}
+
 /* This function is used by the interrupt thread */
 static void
 eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
@@ -1735,6 +1764,16 @@ eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
 			event_desc[type]);
 		fflush(stdout);
 	}
+
+	switch (type) {
+	case RTE_ETH_EVENT_INTR_RMV:
+		if (rte_eal_alarm_set(100000,
+				rmv_event_callback, (void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device removal\n");
+		break;
+	default:
+		break;
+	}
 }
 
 static int
@@ -1905,6 +1944,10 @@ init_port_config(void)
 		    (rte_eth_devices[pid].data->dev_flags &
 		     RTE_ETH_DEV_INTR_LSC))
 			port->dev_conf.intr_conf.lsc = 1;
+		if (rmv_interrupt &&
+		    (rte_eth_devices[pid].data->dev_flags &
+		     RTE_ETH_DEV_INTR_RMV))
+			port->dev_conf.intr_conf.rmv = 1;
 	}
 }
 
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 77fa7ee..85d3e82 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -306,6 +306,7 @@ extern uint8_t  mp_anon; /**< set by "--mp-anon" parameter */
 extern uint8_t no_link_check; /**<set by "--disable-link-check" parameter */
 extern volatile int test_done; /* stop packet forwarding when set to 1. */
 extern uint8_t lsc_interrupt; /**< disabled by "--no-lsc-interrupt" parameter */
+extern uint8_t rmv_interrupt; /**< disabled by "--no-rmv-interrupt" parameter */
 
 #ifdef RTE_NIC_BYPASS
 extern uint32_t bypass_timeout; /**< Store the NIC bypass watchdog timeout */
-- 
2.1.4

Re: [PATCH 0/5] add device removal event

[Gaetan Rivet]

Hi all,

Following the discussion that took place during the tech board meeting,
it seems that some clarification and some additional discussion is
needed about this series.

As of now, what I plan to do is to:

1. Rebase this series upon the bus framework [1][2][3].
2. Write further documentation to clarify the meaning of this event and
further identify the best approach to implement it.

What I expect from this is to spawn a discussion about the best
way to achieve a true hot-plug feature in DPDK.

Any thoughts, comments regarding this?
Regards,

[1] http://dpdk.org/ml/archives/dev/2017-January/054165.html
[2] http://dpdk.org/ml/archives/dev/2017-March/059423.html
[3] http://dpdk.org/ml/archives/dev/2017-March/059376.html

On Fri, Mar 3, 2017 at 4:40 PM, Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
>
> This new event represents the sudden removal of a device from its bus.
> The underlying resources exposed by the bus are expected not to be available
> anymore. The application should thus be able to react and possibly clean up
> related resources that it reserved for the removed device.
>
> This event is different from the current hotplug API available in the DPDK
> for two reasons:
>
> 1. It is a reactive design: the application reacts to a device that has been
>    removed instead of removing a device from its pool.
>
> 2. The event itself is going further than the current detaching of a device
>    from a DPDK application. If the bus is a hardware one, it is expected of the
>    underlying resources to not be available anymore.
>
> This series adds a new event type to ethdev and implements it in mlx4.
> Testpmd is also updated to report all asynchronous ethdev events including this
> one for testing purposes and as a practical usage example.
>
> This series depends on the series titled
> [PATCH 1/2] net/mlx4: split the definitions to the header file
>
> Gaetan Rivet (5):
>   ethdev: introduce device removal event
>   net/mlx4: device removal event support
>   app/testpmd: generic event handler
>   app/testpmd: request link status interrupt
>   app/testpmd: request device removal interrupt
>
>  app/test-pmd/parameters.c                       |   8 +
>  app/test-pmd/testpmd.c                          | 103 ++++++++++
>  app/test-pmd/testpmd.h                          |   2 +
>  doc/guides/nics/features/default.ini            |   1 +
>  doc/guides/nics/features/mlx4.ini               |   1 +
>  doc/guides/prog_guide/env_abstraction_layer.rst |  21 +-
>  drivers/net/mlx4/mlx4.c                         | 258 ++++++++++++++++++++----
>  drivers/net/mlx4/mlx4.h                         |   1 +
>  lib/librte_eal/common/include/rte_pci.h         |   2 +
>  lib/librte_ether/rte_ethdev.c                   |  13 +-
>  lib/librte_ether/rte_ethdev.h                   |   9 +-
>  11 files changed, 375 insertions(+), 44 deletions(-)
>
> --
> 2.1.4
>

[PATCH v2 0/5] add device removal event

[Gaetan Rivet]

This new event represents the sudden removal of a device from its bus.
The underlying resources exposed by the bus are expected not to be available
anymore. The application should thus be able to react and possibly clean up
related resources that it reserved for the removed device.

This event is different from the current hotplug API available in the DPDK
for two reasons:

1. It is a reactive design: the application reacts to a device that has been
   removed instead of removing a device from its pool.

2. The event itself is going further than the current detaching of a device
   from a DPDK application. If the bus is a hardware one, it is expected of the
   underlying resources to not be available anymore.

This series adds a new event type to ethdev and implements it in mlx4.
Testpmd is also updated to report all asynchronous ethdev events including this
one for testing purposes and as a practical usage example.

This series depends on the series titled
[PATCH 1/2] net/mlx4: split the definitions to the header file

v1 --> v2:
  * integrated the series with the new PCI rte_bus implementation.

  I planned on working out a more generic implementation of the RMV event for
  the v17.05, however while writing it I found that I had to evolve the support
  of interrupts in the PCI rte_bus, which did not seem wise at this point of the
  release cycle.

  I consider that this event should be generalized along the LSC event, going from
  pure PCI events to generic rte_bus events, given that they can be relevant to other
  busses (vdev at least). I would reserve this evolution for future release however,
  once a real discussion has taken place.

Gaetan Rivet (5):
  ethdev: introduce device removal event
  net/mlx4: device removal event support
  app/testpmd: generic event handler
  app/testpmd: request link status interrupt
  app/testpmd: request device removal interrupt

 app/test-pmd/parameters.c                       |   8 +
 app/test-pmd/testpmd.c                          | 103 ++++++++++
 app/test-pmd/testpmd.h                          |   2 +
 doc/guides/nics/features/default.ini            |   1 +
 doc/guides/nics/features/mlx4.ini               |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  21 +-
 drivers/net/mlx4/mlx4.c                         | 258 ++++++++++++++++++++----
 drivers/net/mlx4/mlx4.h                         |   1 +
 lib/librte_eal/common/include/rte_pci.h         |   2 +
 lib/librte_ether/rte_ethdev.c                   |  11 +-
 lib/librte_ether/rte_ethdev.h                   |   9 +-
 lib/librte_ether/rte_ethdev_pci.h               |   2 +
 12 files changed, 375 insertions(+), 44 deletions(-)

-- 
2.1.4

[PATCH v2 1/5] ethdev: introduce device removal event

[Gaetan Rivet]

This new API allows reacting to a device removal.
A device removal is the sudden disappearance of a device from its
bus.

PMDs implementing support for this notification guarantee that the removal
of the underlying device does not incur a risk to the application.

In particular, Rx/Tx bursts and all other functions can still be called
(albeit likely returning errors) without triggering a crash, irrespective
of an application handling this event.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 doc/guides/nics/features/default.ini            |  1 +
 doc/guides/prog_guide/env_abstraction_layer.rst | 21 +++++++++++++++++++--
 lib/librte_eal/common/include/rte_pci.h         |  2 ++
 lib/librte_ether/rte_ethdev.c                   | 11 +++++++----
 lib/librte_ether/rte_ethdev.h                   |  9 +++++++--
 lib/librte_ether/rte_ethdev_pci.h               |  2 ++
 6 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini
index b1b9114..cafc6c7 100644
--- a/doc/guides/nics/features/default.ini
+++ b/doc/guides/nics/features/default.ini
@@ -10,6 +10,7 @@
 Speed capabilities   =
 Link status          =
 Link status event    =
+Removal event        =
 Queue status event   =
 Rx interrupt         =
 Free Tx mbuf on demand =
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 7c39cd2..fff1c06 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -178,8 +178,8 @@ The EAL also allows timed callbacks to be used in the same way as for NIC interr
 
 .. note::
 
-    In DPDK PMD, the only interrupts handled by the dedicated host thread are those for link status change,
-    i.e. link up and link down notification.
+    In DPDK PMD, the only interrupts handled by the dedicated host thread are those for link status change
+    (link up and link down notification) and for sudden device removal.
 
 
 + RX Interrupt Event
@@ -207,6 +207,23 @@ The eth_dev driver takes responsibility to program the latter mapping.
 The RX interrupt are controlled/enabled/disabled by ethdev APIs - 'rte_eth_dev_rx_intr_*'. They return failure if the PMD
 hasn't support them yet. The intr_conf.rxq flag is used to turn on the capability of RX interrupt per device.
 
++ Device Removal Event
+
+This event is triggered by a device being removed at a bus level. Its
+underlying resources may have been made unavailable (i.e. PCI mappings
+unmapped). The PMD must make sure that on such occurrence, the application can
+still safely use its callbacks.
+
+This event can be subscribed to in the same way one would subscribe to a link
+status change event. The execution context is thus the same, i.e. it is the
+dedicated interrupt host thread.
+
+Considering this, it is likely that an application would want to close a
+device having emitted a Device Removal Event. In such case, calling
+``rte_eth_dev_close()`` can trigger it to unregister its own Device Removal Event
+callback. Care must be taken not to close the device from the interrupt handler
+context. It is necessary to reschedule such closing operation.
+
 Blacklisting
 ~~~~~~~~~~~~
 
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index fedb197..6effb7a 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -239,6 +239,8 @@ struct rte_pci_bus {
 #define RTE_PCI_DRV_NEED_MAPPING 0x0001
 /** Device driver supports link state interrupt */
 #define RTE_PCI_DRV_INTR_LSC	0x0008
+/** Device driver supports device removal interrupt */
+#define RTE_PCI_DRV_INTR_RMV 0x0010
 
 /**
  * A structure describing a PCI mapping.
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 474188c..ed261c2 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -749,16 +749,19 @@ rte_eth_dev_configure(uint8_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
 		return -EINVAL;
 	}
 
-	/*
-	 * If link state interrupt is enabled, check that the
-	 * device supports it.
-	 */
+	/* Check that the device supports requested interrupts */
 	if ((dev_conf->intr_conf.lsc == 1) &&
 		(!(dev->data->dev_flags & RTE_ETH_DEV_INTR_LSC))) {
 			RTE_PMD_DEBUG_TRACE("driver %s does not support lsc\n",
 					dev->data->drv_name);
 			return -EINVAL;
 	}
+	if ((dev_conf->intr_conf.rmv == 1) &&
+	    (!(dev->data->dev_flags & RTE_ETH_DEV_INTR_RMV))) {
+		RTE_PMD_DEBUG_TRACE("driver %s does not support rmv\n",
+				    dev->data->drv_name);
+		return -EINVAL;
+	}
 
 	/*
 	 * If jumbo frames are enabled, check that the maximum RX packet
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 68a91e2..b8381d6 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -815,9 +815,11 @@ struct rte_eth_udp_tunnel {
  */
 struct rte_intr_conf {
 	/** enable/disable lsc interrupt. 0 (default) - disable, 1 enable */
-	uint16_t lsc;
+	uint32_t lsc:1;
 	/** enable/disable rxq interrupt. 0 (default) - disable, 1 enable */
-	uint16_t rxq;
+	uint32_t rxq:1;
+	/** enable/disable rmv interrupt. 0 (default) - disable, 1 enable */
+	uint32_t rmv:1;
 };
 
 /**
@@ -1737,6 +1739,8 @@ struct rte_eth_dev_data {
 #define RTE_ETH_DEV_INTR_LSC     0x0002
 /** Device is a bonded slave */
 #define RTE_ETH_DEV_BONDED_SLAVE 0x0004
+/** Device supports device removal interrupt */
+#define RTE_ETH_DEV_INTR_RMV     0x0008
 
 /**
  * @internal
@@ -3313,6 +3317,7 @@ enum rte_eth_event_type {
 			/**< reset interrupt event, sent to VF on PF reset */
 	RTE_ETH_EVENT_VF_MBOX,  /**< message from the VF received by PF */
 	RTE_ETH_EVENT_MACSEC,   /**< MACsec offload related event */
+	RTE_ETH_EVENT_INTR_RMV, /**< device removal event */
 	RTE_ETH_EVENT_MAX       /**< max value of this enum */
 };
 
diff --git a/lib/librte_ether/rte_ethdev_pci.h b/lib/librte_ether/rte_ethdev_pci.h
index 2953579..d3bc03c 100644
--- a/lib/librte_ether/rte_ethdev_pci.h
+++ b/lib/librte_ether/rte_ethdev_pci.h
@@ -64,6 +64,8 @@ rte_eth_copy_pci_info(struct rte_eth_dev *eth_dev,
 	eth_dev->data->dev_flags = 0;
 	if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_LSC)
 		eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC;
+	if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_RMV)
+		eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV;
 
 	eth_dev->data->kdrv = pci_dev->kdrv;
 	eth_dev->data->numa_node = pci_dev->device.numa_node;
-- 
2.1.4

Re: [PATCH v2 1/5] ethdev: introduce device removal event

[Ferruh Yigit]

On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
> This new API allows reacting to a device removal.
> A device removal is the sudden disappearance of a device from its
> bus.
> 
> PMDs implementing support for this notification guarantee that the removal
> of the underlying device does not incur a risk to the application.
> 
> In particular, Rx/Tx bursts and all other functions can still be called
> (albeit likely returning errors) without triggering a crash, irrespective
> of an application handling this event.
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> Signed-off-by: Elad Persiko <eladpe@mellanox.com>

<...>

> diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini
> index b1b9114..cafc6c7 100644
> --- a/doc/guides/nics/features/default.ini
> +++ b/doc/guides/nics/features/default.ini
> @@ -10,6 +10,7 @@
>  Speed capabilities   =
>  Link status          =
>  Link status event    =
> +Removal event        =
>  Queue status event   =
>  Rx interrupt         =
>  Free Tx mbuf on demand =

This release a few NIC features added, and it is hard to follow them if
you are particularly looking for these features.

So do you think does it make sense to put those new PMD features into
release notes to make it more visible?

Thanks,
ferruh

Re: [PATCH v2 1/5] ethdev: introduce device removal event

[Gaëtan Rivet]

Hi Ferruh,

On Fri, Apr 21, 2017 at 03:59:24PM +0100, Ferruh Yigit wrote:
>On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
>> This new API allows reacting to a device removal.
>> A device removal is the sudden disappearance of a device from its
>> bus.
>>
>> PMDs implementing support for this notification guarantee that the removal
>> of the underlying device does not incur a risk to the application.
>>
>> In particular, Rx/Tx bursts and all other functions can still be called
>> (albeit likely returning errors) without triggering a crash, irrespective
>> of an application handling this event.
>>
>> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>> Signed-off-by: Elad Persiko <eladpe@mellanox.com>
>
><...>
>
>> diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini
>> index b1b9114..cafc6c7 100644
>> --- a/doc/guides/nics/features/default.ini
>> +++ b/doc/guides/nics/features/default.ini
>> @@ -10,6 +10,7 @@
>>  Speed capabilities   =
>>  Link status          =
>>  Link status event    =
>> +Removal event        =
>>  Queue status event   =
>>  Rx interrupt         =
>>  Free Tx mbuf on demand =
>
>This release a few NIC features added, and it is hard to follow them if
>you are particularly looking for these features.
>
>So do you think does it make sense to put those new PMD features into
>release notes to make it more visible?
>

Yes it seems like a good idea to announce this evolution. I will
send the relevant patch for this soon.

>Thanks,
>ferruh

-- 
Gaëtan Rivet
6WIND

Re: [PATCH v2 1/5] ethdev: introduce device removal event

[Jan Blunck]

Am 25.04.2017 11:06 schrieb "Gaëtan Rivet" <gaetan.rivet@6wind.com>:

Hi Ferruh,


On Fri, Apr 21, 2017 at 03:59:24PM +0100, Ferruh Yigit wrote:

> On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
>
>> This new API allows reacting to a device removal.
>> A device removal is the sudden disappearance of a device from its
>> bus.
>>
>
I don't think this belongs into ethdev. If it is bus related we need to
expose this from it so that apps can register for the low level device
being unplugged.

Jan


>> PMDs implementing support for this notification guarantee that the removal
>> of the underlying device does not incur a risk to the application.
>>
>> In particular, Rx/Tx bursts and all other functions can still be called
>> (albeit likely returning errors) without triggering a crash, irrespective
>> of an application handling this event.
>>
>> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>> Signed-off-by: Elad Persiko <eladpe@mellanox.com>
>>
>
> <...>
>
> diff --git a/doc/guides/nics/features/default.ini
>> b/doc/guides/nics/features/default.ini
>> index b1b9114..cafc6c7 100644
>> --- a/doc/guides/nics/features/default.ini
>> +++ b/doc/guides/nics/features/default.ini
>> @@ -10,6 +10,7 @@
>>  Speed capabilities   =
>>  Link status          =
>>  Link status event    =
>> +Removal event        =
>>  Queue status event   =
>>  Rx interrupt         =
>>  Free Tx mbuf on demand =
>>
>
> This release a few NIC features added, and it is hard to follow them if
> you are particularly looking for these features.
>
> So do you think does it make sense to put those new PMD features into
> release notes to make it more visible?
>
>
Yes it seems like a good idea to announce this evolution. I will
send the relevant patch for this soon.

Thanks,
> ferruh
>

-- 
Gaėtan Rivet
6WIND

Re: [PATCH v2 1/5] ethdev: introduce device removal event

[Thomas Monjalon]

02/05/2017 09:35, Jan Blunck:
> Am 25.04.2017 11:06 schrieb "Gaëtan Rivet" <gaetan.rivet@6wind.com>:
> 
> Hi Ferruh,
> 
> 
> On Fri, Apr 21, 2017 at 03:59:24PM +0100, Ferruh Yigit wrote:
> 
> > On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
> >
> >> This new API allows reacting to a device removal.
> >> A device removal is the sudden disappearance of a device from its
> >> bus.
> >>
> >
> I don't think this belongs into ethdev. If it is bus related we need to
> expose this from it so that apps can register for the low level device
> being unplugged.

Yes it sounds right.
We could work on device notifications.
We need to find a way of notifying the application that there is a
device event and that it affects one or more port at
ethdev/cryptodev/eventdev level.

Re: [PATCH v2 1/5] ethdev: introduce device removal event

[Gaëtan Rivet]

On Tue, May 02, 2017 at 11:18:06AM +0200, Thomas Monjalon wrote:
>02/05/2017 09:35, Jan Blunck:
>> Am 25.04.2017 11:06 schrieb "Gaëtan Rivet" <gaetan.rivet@6wind.com>:
>>
>> Hi Ferruh,
>>
>>
>> On Fri, Apr 21, 2017 at 03:59:24PM +0100, Ferruh Yigit wrote:
>>
>> > On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
>> >
>> >> This new API allows reacting to a device removal.
>> >> A device removal is the sudden disappearance of a device from its
>> >> bus.
>> >>
>> >
>> I don't think this belongs into ethdev. If it is bus related we need to
>> expose this from it so that apps can register for the low level device
>> being unplugged.
>
>Yes it sounds right.
>We could work on device notifications.
>We need to find a way of notifying the application that there is a
>device event and that it affects one or more port at
>ethdev/cryptodev/eventdev level.

This is interesting.
I developed this event with an easier integration in v17.05 in mind.
It needs a proper generic implementation however (as suggested in [1]).

I tried to have this discussion earlier[2], but without much interest.

However, even with a bus-level event framework, we still need a way for drivers
to advertize their support for specific events, and we still need to
differentiate devices that are ready for specific events from those that
do not.

So I agree that it would be interesting to have a generic rte_device
level interrupt framework to support generic events accross the whole
board, but I'm not sure it would make the dichotomy between the *driver
support* flag and the *device enabled* flag disappear.

Regards,
--
[1]: http://dpdk.org/ml/archives/dev/2017-April/064190.html
[2]: http://dpdk.org/ml/archives/dev/2017-March/060998.html
-- 
Gaëtan Rivet
6WIND

[PATCH v2 2/5] net/mlx4: device removal event support

[Gaetan Rivet]

Extend the LSC event handling to support the device removal as well. The
Verbs library will send several related events, that can conflict
with the LSC event itself.

The event handling has thus been made capable of receiving and signaling
several event types at once.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 doc/guides/nics/features/mlx4.ini |   1 +
 drivers/net/mlx4/mlx4.c           | 258 ++++++++++++++++++++++++++++++++------
 drivers/net/mlx4/mlx4.h           |   1 +
 3 files changed, 224 insertions(+), 36 deletions(-)

diff --git a/doc/guides/nics/features/mlx4.ini b/doc/guides/nics/features/mlx4.ini
index c9828f7..285f0ec 100644
--- a/doc/guides/nics/features/mlx4.ini
+++ b/doc/guides/nics/features/mlx4.ini
@@ -6,6 +6,7 @@
 [Features]
 Link status          = Y
 Link status event    = Y
+Removal event        = Y
 Queue start/stop     = Y
 MTU update           = Y
 Jumbo frame          = Y
diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index a1363c8..1e86a7d 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -3930,9 +3930,15 @@ mlx4_rx_queue_release(void *dpdk_rxq)
 	priv_unlock(priv);
 }
 
-static void
+static int
 priv_dev_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
 
+static int
+priv_dev_removal_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
+
+static int
+priv_dev_link_interrupt_handler_install(struct priv *, struct rte_eth_dev *);
+
 /**
  * DPDK callback to start the device.
  *
@@ -3985,7 +3991,18 @@ mlx4_dev_start(struct rte_eth_dev *dev)
 		     (void *)dev, strerror(ret));
 		goto err;
 	} while ((--r) && ((rxq = (*priv->rxqs)[++i]), i));
-	priv_dev_interrupt_handler_install(priv, dev);
+	ret = priv_dev_link_interrupt_handler_install(priv, dev);
+	if (ret) {
+		ERROR("%p: LSC handler install failed",
+		     (void *)dev);
+		goto err;
+	}
+	ret = priv_dev_removal_interrupt_handler_install(priv, dev);
+	if (ret) {
+		ERROR("%p: RMV handler install failed",
+		     (void *)dev);
+		goto err;
+	}
 	ret = mlx4_priv_flow_start(priv);
 	if (ret) {
 		ERROR("%p: flow start failed: %s",
@@ -4104,9 +4121,16 @@ removed_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
 	return 0;
 }
 
-static void
+static int
 priv_dev_interrupt_handler_uninstall(struct priv *, struct rte_eth_dev *);
 
+static int
+priv_dev_removal_interrupt_handler_uninstall(struct priv *,
+					     struct rte_eth_dev *);
+
+static int
+priv_dev_link_interrupt_handler_uninstall(struct priv *, struct rte_eth_dev *);
+
 /**
  * DPDK callback to close the device.
  *
@@ -4169,7 +4193,8 @@ mlx4_dev_close(struct rte_eth_dev *dev)
 		claim_zero(ibv_close_device(priv->ctx));
 	} else
 		assert(priv->ctx == NULL);
-	priv_dev_interrupt_handler_uninstall(priv, dev);
+	priv_dev_removal_interrupt_handler_uninstall(priv, dev);
+	priv_dev_link_interrupt_handler_uninstall(priv, dev);
 	priv_unlock(priv);
 	memset(priv, 0, sizeof(*priv));
 }
@@ -4729,6 +4754,10 @@ mlx4_link_update(struct rte_eth_dev *dev, int wait_to_complete)
 	return -1;
 }
 
+static int
+mlx4_ibv_device_to_pci_addr(const struct ibv_device *device,
+			    struct rte_pci_addr *pci_addr);
+
 /**
  * DPDK callback to change the MTU.
  *
@@ -5255,32 +5284,41 @@ static void
 mlx4_dev_interrupt_handler(void *);
 
 /**
- * Link status handler.
+ * Link/device status handler.
  *
  * @param priv
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @param events
+ *   Pointer to event flags holder.
  *
  * @return
- *   Nonzero if the callback process can be called immediately.
+ *   Number of events
  */
 static int
-priv_dev_link_status_handler(struct priv *priv, struct rte_eth_dev *dev)
+priv_dev_status_handler(struct priv *priv, struct rte_eth_dev *dev,
+			uint32_t *events)
 {
 	struct ibv_async_event event;
 	int port_change = 0;
 	int ret = 0;
 
+	*events = 0;
 	/* Read all message and acknowledge them. */
 	for (;;) {
 		if (ibv_get_async_event(priv->ctx, &event))
 			break;
-
-		if (event.event_type == IBV_EVENT_PORT_ACTIVE ||
-		    event.event_type == IBV_EVENT_PORT_ERR)
+		if ((event.event_type == IBV_EVENT_PORT_ACTIVE ||
+		     event.event_type == IBV_EVENT_PORT_ERR) &&
+		    (priv->intr_conf.lsc == 1)) {
 			port_change = 1;
-		else
+			ret++;
+		} else if (event.event_type == IBV_EVENT_DEVICE_FATAL &&
+			   priv->intr_conf.rmv == 1) {
+			*events |= (1 << RTE_ETH_EVENT_INTR_RMV);
+			ret++;
+		} else
 			DEBUG("event type %d on port %d not handled",
 			      event.event_type, event.element.port_num);
 		ibv_ack_async_event(&event);
@@ -5298,8 +5336,9 @@ priv_dev_link_status_handler(struct priv *priv, struct rte_eth_dev *dev)
 			rte_eal_alarm_set(MLX4_ALARM_TIMEOUT_US,
 					  mlx4_dev_link_status_handler,
 					  dev);
-		} else
-			ret = 1;
+		} else {
+			*events |= (1 << RTE_ETH_EVENT_INTR_LSC);
+		}
 	}
 	return ret;
 }
@@ -5315,13 +5354,14 @@ mlx4_dev_link_status_handler(void *arg)
 {
 	struct rte_eth_dev *dev = arg;
 	struct priv *priv = dev->data->dev_private;
+	uint32_t events;
 	int ret;
 
 	priv_lock(priv);
 	assert(priv->pending_alarm == 1);
-	ret = priv_dev_link_status_handler(priv, dev);
+	ret = priv_dev_status_handler(priv, dev, &events);
 	priv_unlock(priv);
-	if (ret)
+	if (ret > 0 && events & (1 << RTE_ETH_EVENT_INTR_LSC))
 		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
 }
 
@@ -5339,12 +5379,26 @@ mlx4_dev_interrupt_handler(void *cb_arg)
 	struct rte_eth_dev *dev = cb_arg;
 	struct priv *priv = dev->data->dev_private;
 	int ret;
+	uint32_t ev;
+	int i;
 
 	priv_lock(priv);
-	ret = priv_dev_link_status_handler(priv, dev);
+	ret = priv_dev_status_handler(priv, dev, &ev);
 	priv_unlock(priv);
-	if (ret)
-		_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	if (ret > 0) {
+		for (i = RTE_ETH_EVENT_UNKNOWN;
+		     i < RTE_ETH_EVENT_MAX;
+		     i++) {
+			if (ev & (1 << i)) {
+				ev &= ~(1 << i);
+				_rte_eth_dev_callback_process(dev, i, NULL);
+				ret--;
+			}
+		}
+		if (ret)
+			WARN("%d event%s not processed", ret,
+			     (ret > 1 ? "s were" : " was"));
+	}
 }
 
 /**
@@ -5354,20 +5408,30 @@ mlx4_dev_interrupt_handler(void *cb_arg)
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative errno value on failure.
  */
-static void
+static int
 priv_dev_interrupt_handler_uninstall(struct priv *priv, struct rte_eth_dev *dev)
 {
-	if (!dev->data->dev_conf.intr_conf.lsc)
-		return;
-	rte_intr_callback_unregister(&priv->intr_handle,
-				     mlx4_dev_interrupt_handler,
-				     dev);
-	if (priv->pending_alarm)
-		rte_eal_alarm_cancel(mlx4_dev_link_status_handler, dev);
-	priv->pending_alarm = 0;
+	int ret;
+
+	if (priv->intr_conf.lsc ||
+	    priv->intr_conf.rmv)
+		return 0;
+	ret = rte_intr_callback_unregister(&priv->intr_handle,
+					   mlx4_dev_interrupt_handler,
+					   dev);
+	if (ret < 0) {
+		ERROR("rte_intr_callback_unregister failed with %d"
+		      "%s%s%s", ret,
+		      (errno ? " (errno: " : ""),
+		      (errno ? strerror(errno) : ""),
+		      (errno ? ")" : ""));
+	}
 	priv->intr_handle.fd = 0;
 	priv->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
+	return ret;
 }
 
 /**
@@ -5377,27 +5441,148 @@ priv_dev_interrupt_handler_uninstall(struct priv *priv, struct rte_eth_dev *dev)
  *   Pointer to private structure.
  * @param dev
  *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative errno value on failure.
  */
-static void
-priv_dev_interrupt_handler_install(struct priv *priv, struct rte_eth_dev *dev)
+static int
+priv_dev_interrupt_handler_install(struct priv *priv,
+				   struct rte_eth_dev *dev)
 {
-	int rc, flags;
+	int flags;
+	int rc;
 
-	if (!dev->data->dev_conf.intr_conf.lsc)
-		return;
+	/* Check whether the interrupt handler has already been installed
+	 * for either type of interrupt
+	 */
+	if (priv->intr_conf.lsc &&
+	    priv->intr_conf.rmv &&
+	    priv->intr_handle.fd)
+		return 0;
 	assert(priv->ctx->async_fd > 0);
 	flags = fcntl(priv->ctx->async_fd, F_GETFL);
 	rc = fcntl(priv->ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
 	if (rc < 0) {
 		INFO("failed to change file descriptor async event queue");
 		dev->data->dev_conf.intr_conf.lsc = 0;
+		dev->data->dev_conf.intr_conf.rmv = 0;
+		return -errno;
 	} else {
 		priv->intr_handle.fd = priv->ctx->async_fd;
 		priv->intr_handle.type = RTE_INTR_HANDLE_EXT;
-		rte_intr_callback_register(&priv->intr_handle,
-					   mlx4_dev_interrupt_handler,
-					   dev);
+		rc = rte_intr_callback_register(&priv->intr_handle,
+						 mlx4_dev_interrupt_handler,
+						 dev);
+		if (rc) {
+			ERROR("rte_intr_callback_register failed "
+			      " (errno: %s)", strerror(errno));
+			return rc;
+		}
 	}
+	return 0;
+}
+
+/**
+ * Uninstall interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_removal_interrupt_handler_uninstall(struct priv *priv,
+					    struct rte_eth_dev *dev)
+{
+	if (dev->data->dev_conf.intr_conf.rmv) {
+		priv->intr_conf.rmv = 0;
+		return priv_dev_interrupt_handler_uninstall(priv, dev);
+	}
+	return 0;
+}
+
+/**
+ * Uninstall interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error,
+ */
+static int
+priv_dev_link_interrupt_handler_uninstall(struct priv *priv,
+					  struct rte_eth_dev *dev)
+{
+	int ret = 0;
+
+	if (dev->data->dev_conf.intr_conf.lsc) {
+		priv->intr_conf.lsc = 0;
+		ret = priv_dev_interrupt_handler_uninstall(priv, dev);
+		if (ret)
+			return ret;
+	}
+	if (priv->pending_alarm)
+		if (rte_eal_alarm_cancel(mlx4_dev_link_status_handler,
+					 dev)) {
+			ERROR("rte_eal_alarm_cancel failed "
+			      " (errno: %s)", strerror(rte_errno));
+			return -rte_errno;
+		}
+	priv->pending_alarm = 0;
+	return 0;
+}
+
+/**
+ * Install link interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_link_interrupt_handler_install(struct priv *priv,
+					struct rte_eth_dev *dev)
+{
+	int ret;
+
+	if (dev->data->dev_conf.intr_conf.lsc) {
+		ret = priv_dev_interrupt_handler_install(priv, dev);
+		if (ret)
+			return ret;
+		priv->intr_conf.lsc = 1;
+	}
+	return 0;
+}
+
+/**
+ * Install removal interrupt handler.
+ *
+ * @param priv
+ *   Pointer to private structure.
+ * @param dev
+ *   Pointer to the rte_eth_dev structure.
+ * @return
+ *   0 on success, negative value on error.
+ */
+static int
+priv_dev_removal_interrupt_handler_install(struct priv *priv,
+					   struct rte_eth_dev *dev)
+{
+	int ret;
+
+	if (dev->data->dev_conf.intr_conf.rmv) {
+		ret = priv_dev_interrupt_handler_install(priv, dev);
+		if (ret)
+			return ret;
+		priv->intr_conf.rmv = 1;
+	}
+	return 0;
 }
 
 /**
@@ -5879,7 +6064,8 @@ static struct rte_pci_driver mlx4_driver = {
 	},
 	.id_table = mlx4_pci_id_map,
 	.probe = mlx4_pci_probe,
-	.drv_flags = RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_INTR_RMV,
 };
 
 /**
diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
index 08af155..9a3bae9 100644
--- a/drivers/net/mlx4/mlx4.h
+++ b/drivers/net/mlx4/mlx4.h
@@ -347,6 +347,7 @@ struct priv {
 	struct rte_intr_handle intr_handle; /* Interrupt handler. */
 	struct rte_flow_drop *flow_drop_queue; /* Flow drop queue. */
 	LIST_HEAD(mlx4_flows, rte_flow) flows;
+	struct rte_intr_conf intr_conf; /* Active interrupt configuration. */
 	rte_spinlock_t lock; /* Lock for control functions. */
 };
 
-- 
2.1.4

[PATCH v2 3/5] app/testpmd: generic event handler

[Gaetan Rivet]

This is a rather simple handler that prints a message with the name of
the current event. It can be used to check PMD callback registration and
triggers.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 app/test-pmd/testpmd.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index d6bd2b2..221f0e9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -351,6 +351,9 @@ struct rte_stats_bitrates *bitrate_data;
 /* Forward function declarations */
 static void map_port_queue_stats_mapping_registers(uint8_t pi, struct rte_port *port);
 static void check_all_ports_link_status(uint32_t port_mask);
+static void eth_event_callback(uint8_t port_id,
+			       enum rte_eth_event_type type,
+			       void *param);
 
 /*
  * Check if all the ports are started.
@@ -1347,6 +1350,7 @@ start_port(portid_t pid)
 	queueid_t qi;
 	struct rte_port *port;
 	struct ether_addr mac_addr;
+	enum rte_eth_event_type event_type;
 
 	if (port_id_is_invalid(pid, ENABLED_WARN))
 		return 0;
@@ -1458,6 +1462,21 @@ start_port(portid_t pid)
 				return -1;
 			}
 		}
+
+		for (event_type = RTE_ETH_EVENT_UNKNOWN;
+		     event_type < RTE_ETH_EVENT_MAX;
+		     event_type++) {
+			diag = rte_eth_dev_callback_register(pi,
+							event_type,
+							eth_event_callback,
+							NULL);
+			if (diag) {
+				printf("Failed to setup even callback for event %d\n",
+					event_type);
+				return -1;
+			}
+		}
+
 		/* start port */
 		if (rte_eth_dev_start(pi) < 0) {
 			printf("Fail to start port %d\n", pi);
@@ -1730,6 +1749,34 @@ check_all_ports_link_status(uint32_t port_mask)
 	}
 }
 
+/* This function is used by the interrupt thread */
+static void
+eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
+{
+	static const char * const event_desc[] = {
+		[RTE_ETH_EVENT_UNKNOWN] = "Unknown",
+		[RTE_ETH_EVENT_INTR_LSC] = "LSC",
+		[RTE_ETH_EVENT_QUEUE_STATE] = "Queue state",
+		[RTE_ETH_EVENT_INTR_RESET] = "Interrupt reset",
+		[RTE_ETH_EVENT_VF_MBOX] = "VF Mbox",
+		[RTE_ETH_EVENT_MACSEC] = "MACsec",
+		[RTE_ETH_EVENT_INTR_RMV] = "device removal",
+		[RTE_ETH_EVENT_MAX] = NULL,
+	};
+
+	RTE_SET_USED(param);
+
+	if (type >= RTE_ETH_EVENT_MAX) {
+		fprintf(stderr, "\nPort %" PRIu8 ": %s called upon invalid event %d\n",
+			port_id, __func__, type);
+		fflush(stderr);
+	} else {
+		printf("\nPort %" PRIu8 ": %s event\n", port_id,
+			event_desc[type]);
+		fflush(stdout);
+	}
+}
+
 static int
 set_tx_queue_stats_mapping_registers(uint8_t port_id, struct rte_port *port)
 {
-- 
2.1.4

[PATCH v2 4/5] app/testpmd: request link status interrupt

[Gaetan Rivet]

For drivers supporting the LSC event, enable it.
This allows to test LSC event support.

Add the --no-lsc-interrupt parameter to explicitly disable the link status
change interrupt.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 app/test-pmd/parameters.c |  4 ++++
 app/test-pmd/testpmd.c    | 13 +++++++++++++
 app/test-pmd/testpmd.h    |  1 +
 3 files changed, 18 insertions(+)

diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c36bcf8..c79c349 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -200,6 +200,7 @@ usage(char* progname)
 		" or total packet length.\n");
 	printf("  --disable-link-check: disable check on link status when "
 	       "starting/stopping ports.\n");
+	printf("  --no-lsc-interrupt: disable link status change interrupt.\n");
 }
 
 #ifdef RTE_LIBRTE_CMDLINE
@@ -568,6 +569,7 @@ launch_args_parse(int argc, char** argv)
 		{ "no-flush-rx",	0, 0, 0 },
 		{ "txpkts",			1, 0, 0 },
 		{ "disable-link-check",		0, 0, 0 },
+		{ "no-lsc-interrupt",		0, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -998,6 +1000,8 @@ launch_args_parse(int argc, char** argv)
 				no_flush_rx = 1;
 			if (!strcmp(lgopts[opt_idx].name, "disable-link-check"))
 				no_link_check = 1;
+			if (!strcmp(lgopts[opt_idx].name, "no-lsc-interrupt"))
+				lsc_interrupt = 0;
 
 			break;
 		case 'h':
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 221f0e9..5f94393 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -271,6 +271,11 @@ uint8_t no_flush_rx = 0; /* flush by default */
 uint8_t no_link_check = 0; /* check by default */
 
 /*
+ * Enable link status change notification
+ */
+uint8_t lsc_interrupt = 1; /* enabled by default */
+
+/*
  * NIC bypass mode configuration options.
  */
 #ifdef RTE_NIC_BYPASS
@@ -1746,6 +1751,9 @@ check_all_ports_link_status(uint32_t port_mask)
 		if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
 			print_flag = 1;
 		}
+
+		if (lsc_interrupt)
+			break;
 	}
 }
 
@@ -1929,6 +1937,11 @@ init_port_config(void)
 #ifdef RTE_NIC_BYPASS
 		rte_eth_dev_bypass_init(pid);
 #endif
+
+		if (lsc_interrupt &&
+		    (rte_eth_devices[pid].data->dev_flags &
+		     RTE_ETH_DEV_INTR_LSC))
+			port->dev_conf.intr_conf.lsc = 1;
 	}
 }
 
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 642796e..62f89e6 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -305,6 +305,7 @@ extern uint8_t no_flush_rx; /**<set by "--no-flush-rx" parameter */
 extern uint8_t  mp_anon; /**< set by "--mp-anon" parameter */
 extern uint8_t no_link_check; /**<set by "--disable-link-check" parameter */
 extern volatile int test_done; /* stop packet forwarding when set to 1. */
+extern uint8_t lsc_interrupt; /**< disabled by "--no-lsc-interrupt" parameter */
 
 #ifdef RTE_NIC_BYPASS
 extern uint32_t bypass_timeout; /**< Store the NIC bypass watchdog timeout */
-- 
2.1.4

Re: [PATCH v2 4/5] app/testpmd: request link status interrupt

[Ferruh Yigit]

On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
> For drivers supporting the LSC event, enable it.
> This allows to test LSC event support.
> 
> Add the --no-lsc-interrupt parameter to explicitly disable the link status
> change interrupt.
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
>  app/test-pmd/parameters.c |  4 ++++
>  app/test-pmd/testpmd.c    | 13 +++++++++++++
>  app/test-pmd/testpmd.h    |  1 +
>  3 files changed, 18 insertions(+)

Hi Gaetan,

This patch adds new option to testpmd, can you please update testpmd
documentation to document new option?

Same is valid for next patch (no-rmv-interrupt option)

Thanks,
ferruh

Re: [PATCH v2 4/5] app/testpmd: request link status interrupt

[Gaëtan Rivet]

On Fri, Apr 21, 2017 at 03:55:43PM +0100, Ferruh Yigit wrote:
>On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
>> For drivers supporting the LSC event, enable it.
>> This allows to test LSC event support.
>>
>> Add the --no-lsc-interrupt parameter to explicitly disable the link status
>> change interrupt.
>>
>> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>> ---
>>  app/test-pmd/parameters.c |  4 ++++
>>  app/test-pmd/testpmd.c    | 13 +++++++++++++
>>  app/test-pmd/testpmd.h    |  1 +
>>  3 files changed, 18 insertions(+)
>
>Hi Gaetan,
>
>This patch adds new option to testpmd, can you please update testpmd
>documentation to document new option?
>
>Same is valid for next patch (no-rmv-interrupt option)
>

Sure, I will send a patch for this.

Should I send a new version of my previous series or a single
patch fixing the documentation for both options as a standalone?

>Thanks,
>ferruh
>

-- 
Gaëtan Rivet
6WIND

Re: [PATCH v2 4/5] app/testpmd: request link status interrupt

[Ferruh Yigit]

On 4/25/2017 10:07 AM, Gaëtan Rivet wrote:
> On Fri, Apr 21, 2017 at 03:55:43PM +0100, Ferruh Yigit wrote:
>> On 4/18/2017 1:17 PM, Gaetan Rivet wrote:
>>> For drivers supporting the LSC event, enable it.
>>> This allows to test LSC event support.
>>>
>>> Add the --no-lsc-interrupt parameter to explicitly disable the link status
>>> change interrupt.
>>>
>>> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>>> ---
>>>  app/test-pmd/parameters.c |  4 ++++
>>>  app/test-pmd/testpmd.c    | 13 +++++++++++++
>>>  app/test-pmd/testpmd.h    |  1 +
>>>  3 files changed, 18 insertions(+)
>>
>> Hi Gaetan,
>>
>> This patch adds new option to testpmd, can you please update testpmd
>> documentation to document new option?
>>
>> Same is valid for next patch (no-rmv-interrupt option)
>>
> 
> Sure, I will send a patch for this.
> 
> Should I send a new version of my previous series or a single
> patch fixing the documentation for both options as a standalone?

Single patch to fix documentation will be do it.

Thanks,
ferruh

[PATCH 1/3] doc: fix missing backquotes

[Gaetan Rivet]

Fixes: ea85e7d711b6 ("ethdev: retrieve xstats by ID")

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/rel_notes/release_17_05.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_17_05.rst b/doc/guides/rel_notes/release_17_05.rst
index ad20e86..eb5b30f 100644
--- a/doc/guides/rel_notes/release_17_05.rst
+++ b/doc/guides/rel_notes/release_17_05.rst
@@ -454,7 +454,7 @@ API Changes
 
   * Changed set of input parameters for ``rte_eth_xstats_get`` and ``rte_eth_xstats_get_names`` functions.
 
-  * Added new functions ``rte_eth_xstats_get_all`` and ``rte_eth_xstats_get_names_all to provide backward compatibility for
+  * Added new functions ``rte_eth_xstats_get_all`` and ``rte_eth_xstats_get_names_all`` to provide backward compatibility for
     ``rte_eth_xstats_get`` and ``rte_eth_xstats_get_names``
 
   * Added new function ``rte_eth_xstats_get_id_by_name``
-- 
2.1.4

[PATCH 2/3] doc: add device removal event to release note

[Gaetan Rivet]

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/rel_notes/release_17_05.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_17_05.rst b/doc/guides/rel_notes/release_17_05.rst
index eb5b30f..293d3fd 100644
--- a/doc/guides/rel_notes/release_17_05.rst
+++ b/doc/guides/rel_notes/release_17_05.rst
@@ -298,6 +298,11 @@ New Features
 
   * DES DOCSIS BPI algorithm.
 
+* **Added Device Removal Interrupt.**
+
+  * Added a new ethdev event ``RTE_ETH_DEV_INTR_RMV`` to signify the sudden removal of a device. This
+    event can be advertized by PCI drivers and enabled accordingly.
+
 
 Resolved Issues
 ---------------
@@ -459,7 +464,6 @@ API Changes
 
   * Added new function ``rte_eth_xstats_get_id_by_name``
 
-
 ABI Changes
 -----------
 
-- 
2.1.4

[PATCH 3/3] doc: add lsc and rmv interrupt to testpmd user guide

[Gaetan Rivet]

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/testpmd_app_ug/run_app.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index df5a0ee..a72c7e3 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -465,3 +465,11 @@ The commandline options are:
 *   ``--disable-link-check``
 
     Disable check on link status when starting/stopping ports.
+
+*  ``--no-lsc-interrupt``
+
+   Disable LSC interrupts for all ports, even those supporting it.
+
+*  ``--no-rmv-interrupt``
+
+  Disable RMV interrupts for all ports, even those supporting it.
-- 
2.1.4

[PATCH v2 1/4] doc: fix missing backquotes

[Gaetan Rivet]

Fixes: ea85e7d711b6 ("ethdev: retrieve xstats by ID")

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/rel_notes/release_17_05.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_17_05.rst b/doc/guides/rel_notes/release_17_05.rst
index ad20e86..eb5b30f 100644
--- a/doc/guides/rel_notes/release_17_05.rst
+++ b/doc/guides/rel_notes/release_17_05.rst
@@ -454,7 +454,7 @@ API Changes
 
   * Changed set of input parameters for ``rte_eth_xstats_get`` and ``rte_eth_xstats_get_names`` functions.
 
-  * Added new functions ``rte_eth_xstats_get_all`` and ``rte_eth_xstats_get_names_all to provide backward compatibility for
+  * Added new functions ``rte_eth_xstats_get_all`` and ``rte_eth_xstats_get_names_all`` to provide backward compatibility for
     ``rte_eth_xstats_get`` and ``rte_eth_xstats_get_names``
 
   * Added new function ``rte_eth_xstats_get_id_by_name``
-- 
2.1.4

[PATCH v2 2/4] doc: add device removal event to release note

[Gaetan Rivet]

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/rel_notes/release_17_05.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/guides/rel_notes/release_17_05.rst b/doc/guides/rel_notes/release_17_05.rst
index eb5b30f..602ef8e 100644
--- a/doc/guides/rel_notes/release_17_05.rst
+++ b/doc/guides/rel_notes/release_17_05.rst
@@ -298,6 +298,11 @@ New Features
 
   * DES DOCSIS BPI algorithm.
 
+* **Added Device Removal Interrupt.**
+
+  * Added a new ethdev event ``RTE_ETH_DEV_INTR_RMV`` to signify the sudden removal of a device. This
+    event can be advertized by PCI drivers and enabled accordingly.
+
 
 Resolved Issues
 ---------------
-- 
2.1.4

Re: [PATCH v2 2/4] doc: add device removal event to release note

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Gaetan Rivet
> Sent: Tuesday, April 25, 2017 11:18 AM
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: [dpdk-dev] [PATCH v2 2/4] doc: add device removal event to
> release note
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>


Acked-by: John McNamara <john.mcnamara@intel.com>

[PATCH v2 3/4] doc: add LSC and RMV interrupt to testpmd user guide

[Gaetan Rivet]

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 doc/guides/testpmd_app_ug/run_app.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index df5a0ee..a72c7e3 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -465,3 +465,11 @@ The commandline options are:
 *   ``--disable-link-check``
 
     Disable check on link status when starting/stopping ports.
+
+*  ``--no-lsc-interrupt``
+
+   Disable LSC interrupts for all ports, even those supporting it.
+
+*  ``--no-rmv-interrupt``
+
+  Disable RMV interrupts for all ports, even those supporting it.
-- 
2.1.4

Re: [PATCH v2 3/4] doc: add LSC and RMV interrupt to testpmd user guide

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Gaetan Rivet
> Sent: Tuesday, April 25, 2017 11:18 AM
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: [dpdk-dev] [PATCH v2 3/4] doc: add LSC and RMV interrupt to
> testpmd user guide
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

[PATCH v2 4/4] devtools: add git log checks for rmv

[Gaetan Rivet]

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---
 devtools/check-git-log.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/devtools/check-git-log.sh b/devtools/check-git-log.sh
index 8449d4f..f081ff5 100755
--- a/devtools/check-git-log.sh
+++ b/devtools/check-git-log.sh
@@ -132,6 +132,7 @@ bad=$(echo "$headlines" | grep -E --color=always \
 	-e ':.*\<numa\>' \
 	-e ':.*\<pci\>' \
 	-e ':.*\<pmd\>' \
+	-e ':.*\<rmv\>' \
 	-e ':.*\<rss\>' \
 	-e ':.*\<tso\>' \
 	-e ':.*\<[Vv]lan\>' \
-- 
2.1.4

Re: [PATCH v2 4/4] devtools: add git log checks for rmv

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Gaetan Rivet
> Sent: Tuesday, April 25, 2017 11:18 AM
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: [dpdk-dev] [PATCH v2 4/4] devtools: add git log checks for rmv
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 4/4] devtools: add git log checks for rmv

[Thomas Monjalon]

25/04/2017 12:18, Gaetan Rivet:
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
>  devtools/check-git-log.sh | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/devtools/check-git-log.sh b/devtools/check-git-log.sh
> index 8449d4f..f081ff5 100755
> --- a/devtools/check-git-log.sh
> +++ b/devtools/check-git-log.sh
> @@ -132,6 +132,7 @@ bad=$(echo "$headlines" | grep -E --color=always \
>  	-e ':.*\<numa\>' \
>  	-e ':.*\<pci\>' \
>  	-e ':.*\<pmd\>' \
> +	-e ':.*\<rmv\>' \
>  	-e ':.*\<rss\>' \
>  	-e ':.*\<tso\>' \
>  	-e ':.*\<[Vv]lan\>' \

rmv is not an acronym. I think there is no need to enforce uppercases.

Re: [PATCH v2 1/4] doc: fix missing backquotes

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Gaetan Rivet
> Sent: Tuesday, April 25, 2017 11:18 AM
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: [dpdk-dev] [PATCH v2 1/4] doc: fix missing backquotes
> 
> Fixes: ea85e7d711b6 ("ethdev: retrieve xstats by ID")
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>

Thanks. Could you also make the V1 patches as "superseded" in patchwork.

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [PATCH v2 1/4] doc: fix missing backquotes

[Thomas Monjalon]

26/04/2017 16:58, Mcnamara, John:
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Gaetan Rivet
> > Sent: Tuesday, April 25, 2017 11:18 AM
> > To: dev@dpdk.org
> > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>
> > Subject: [dpdk-dev] [PATCH v2 1/4] doc: fix missing backquotes
> > 
> > Fixes: ea85e7d711b6 ("ethdev: retrieve xstats by ID")
> > 
> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> 
> Thanks. Could you also make the V1 patches as "superseded" in patchwork.
> 
> Acked-by: John McNamara <john.mcnamara@intel.com>

Series applied, except patch 4/4 which is superfluous.

[PATCH v2 5/5] app/testpmd: request device removal interrupt

[Gaetan Rivet]

Enable device removal event for PMD supporting it.
Add the --no-rmv-interrupt parameter to explicitly disable it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
Signed-off-by: Elad Persiko <eladpe@mellanox.com>
---
 app/test-pmd/parameters.c |  4 ++++
 app/test-pmd/testpmd.c    | 43 +++++++++++++++++++++++++++++++++++++++++++
 app/test-pmd/testpmd.h    |  1 +
 3 files changed, 48 insertions(+)

diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c79c349..7efd718 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -201,6 +201,7 @@ usage(char* progname)
 	printf("  --disable-link-check: disable check on link status when "
 	       "starting/stopping ports.\n");
 	printf("  --no-lsc-interrupt: disable link status change interrupt.\n");
+	printf("  --no-rmv-interrupt: disable device removal interrupt.");
 }
 
 #ifdef RTE_LIBRTE_CMDLINE
@@ -570,6 +571,7 @@ launch_args_parse(int argc, char** argv)
 		{ "txpkts",			1, 0, 0 },
 		{ "disable-link-check",		0, 0, 0 },
 		{ "no-lsc-interrupt",		0, 0, 0 },
+		{ "no-rmv-interrupt",		0, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -1002,6 +1004,8 @@ launch_args_parse(int argc, char** argv)
 				no_link_check = 1;
 			if (!strcmp(lgopts[opt_idx].name, "no-lsc-interrupt"))
 				lsc_interrupt = 0;
+			if (!strcmp(lgopts[opt_idx].name, "no-rmv-interrupt"))
+				rmv_interrupt = 0;
 
 			break;
 		case 'h':
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 5f94393..a6f59ba 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -59,6 +59,7 @@
 #include <rte_memzone.h>
 #include <rte_launch.h>
 #include <rte_eal.h>
+#include <rte_alarm.h>
 #include <rte_per_lcore.h>
 #include <rte_lcore.h>
 #include <rte_atomic.h>
@@ -276,6 +277,11 @@ uint8_t no_link_check = 0; /* check by default */
 uint8_t lsc_interrupt = 1; /* enabled by default */
 
 /*
+ * Enable device removal notification.
+ */
+uint8_t rmv_interrupt = 1; /* enabled by default */
+
+/*
  * NIC bypass mode configuration options.
  */
 #ifdef RTE_NIC_BYPASS
@@ -1757,6 +1763,29 @@ check_all_ports_link_status(uint32_t port_mask)
 	}
 }
 
+static void
+rmv_event_callback(void *arg)
+{
+	struct rte_eth_dev *dev;
+	struct rte_devargs *da;
+	char name[32] = "";
+	uint8_t port_id = (intptr_t)arg;
+
+	RTE_ETH_VALID_PORTID_OR_RET(port_id);
+	dev = &rte_eth_devices[port_id];
+	da = dev->device->devargs;
+
+	stop_port(port_id);
+	close_port(port_id);
+	if (da->type == RTE_DEVTYPE_VIRTUAL)
+		snprintf(name, sizeof(name), "%s", da->virt.drv_name);
+	else if (da->type == RTE_DEVTYPE_WHITELISTED_PCI)
+		rte_eal_pci_device_name(&da->pci.addr, name, sizeof(name));
+	printf("removing device %s\n", name);
+	rte_eal_dev_detach(name);
+	dev->state = RTE_ETH_DEV_UNUSED;
+}
+
 /* This function is used by the interrupt thread */
 static void
 eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
@@ -1783,6 +1812,16 @@ eth_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
 			event_desc[type]);
 		fflush(stdout);
 	}
+
+	switch (type) {
+	case RTE_ETH_EVENT_INTR_RMV:
+		if (rte_eal_alarm_set(100000,
+				rmv_event_callback, (void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device removal\n");
+		break;
+	default:
+		break;
+	}
 }
 
 static int
@@ -1942,6 +1981,10 @@ init_port_config(void)
 		    (rte_eth_devices[pid].data->dev_flags &
 		     RTE_ETH_DEV_INTR_LSC))
 			port->dev_conf.intr_conf.lsc = 1;
+		if (rmv_interrupt &&
+		    (rte_eth_devices[pid].data->dev_flags &
+		     RTE_ETH_DEV_INTR_RMV))
+			port->dev_conf.intr_conf.rmv = 1;
 	}
 }
 
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 62f89e6..a9ff07e 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -306,6 +306,7 @@ extern uint8_t  mp_anon; /**< set by "--mp-anon" parameter */
 extern uint8_t no_link_check; /**<set by "--disable-link-check" parameter */
 extern volatile int test_done; /* stop packet forwarding when set to 1. */
 extern uint8_t lsc_interrupt; /**< disabled by "--no-lsc-interrupt" parameter */
+extern uint8_t rmv_interrupt; /**< disabled by "--no-rmv-interrupt" parameter */
 
 #ifdef RTE_NIC_BYPASS
 extern uint32_t bypass_timeout; /**< Store the NIC bypass watchdog timeout */
-- 
2.1.4

Re: [PATCH v2 0/5] add device removal event

[Thomas Monjalon]

18/04/2017 14:17, Gaetan Rivet:
> This new event represents the sudden removal of a device from its bus.
> The underlying resources exposed by the bus are expected not to be available
> anymore. The application should thus be able to react and possibly clean up
> related resources that it reserved for the removed device.
> 
> This event is different from the current hotplug API available in the DPDK
> for two reasons:
> 
> 1. It is a reactive design: the application reacts to a device that has been
> removed instead of removing a device from its pool.
> 
> 2. The event itself is going further than the current detaching of a device
>    from a DPDK application. If the bus is a hardware one, it is expected of
> the underlying resources to not be available anymore.
> 
> This series adds a new event type to ethdev and implements it in mlx4.
> Testpmd is also updated to report all asynchronous ethdev events including
> this one for testing purposes and as a practical usage example.
> 
> This series depends on the series titled
> [PATCH 1/2] net/mlx4: split the definitions to the header file
> 
> v1 --> v2:
>   * integrated the series with the new PCI rte_bus implementation.
> 
>   I planned on working out a more generic implementation of the RMV event
> for the v17.05, however while writing it I found that I had to evolve the
> support of interrupts in the PCI rte_bus, which did not seem wise at this
> point of the release cycle.
> 
>   I consider that this event should be generalized along the LSC event,
> going from pure PCI events to generic rte_bus events, given that they can
> be relevant to other busses (vdev at least). I would reserve this evolution
> for future release however, once a real discussion has taken place.
> 
> Gaetan Rivet (5):
>   ethdev: introduce device removal event
>   net/mlx4: device removal event support
>   app/testpmd: generic event handler
>   app/testpmd: request link status interrupt
>   app/testpmd: request device removal interrupt

Applied, thanks