[PATCH 1/2] net/failsafe: fix removed sub-device cleanup

[PATCH 1/2] net/failsafe: fix removed sub-device cleanup

[Matan Azrad]

The fail-safe PMD registers to RMV event for each removable sub-device
port in order to cleanup the sub-device resources and switch the Tx
sub-device directly when it is plugged-out.

In the removal time, the fail-safe PMD stops and closes the sub-device
but it doesn't unregister the LSC and RMV callbacks of the sub-device
port.

It can lead the callbacks to be called for a port which is no more
associated to the fail-safe sub-device, because there is not a
guaranty that a sub-device gets the same port ID for each plug-in
process. This port, for example, may belong to another sub-device of a
different fail-safe device.

Unregister the LSC and RMV callbacks for sub-devices which are not
used.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---
 drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
 drivers/net/failsafe/failsafe_ops.c     |  5 +++++
 drivers/net/failsafe/failsafe_private.h |  3 +++
 3 files changed, 30 insertions(+)

diff --git a/drivers/net/failsafe/failsafe_ether.c b/drivers/net/failsafe/failsafe_ether.c
index 733e95d..2bbee82 100644
--- a/drivers/net/failsafe/failsafe_ether.c
+++ b/drivers/net/failsafe/failsafe_ether.c
@@ -260,6 +260,7 @@
 		sdev->state = DEV_ACTIVE;
 		/* fallthrough */
 	case DEV_ACTIVE:
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_PROBED;
 		/* fallthrough */
@@ -321,6 +322,27 @@
 }
 
 void
+failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev)
+{
+	if (sdev == NULL)
+		return;
+	if (sdev->rmv_callback) {
+		rte_eth_dev_callback_unregister(PORT_ID(sdev),
+					RTE_ETH_EVENT_INTR_RMV,
+					failsafe_eth_rmv_event_callback,
+					sdev);
+		sdev->rmv_callback = 0;
+	}
+	if (sdev->lsc_callback) {
+		rte_eth_dev_callback_unregister(PORT_ID(sdev),
+					RTE_ETH_EVENT_INTR_LSC,
+					failsafe_eth_lsc_event_callback,
+					sdev);
+		sdev->lsc_callback = 0;
+	}
+}
+
+void
 failsafe_dev_remove(struct rte_eth_dev *dev)
 {
 	struct sub_device *sdev;
diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index d04277b..e0570b6 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -146,6 +146,8 @@
 			if (ret)
 				WARN("Failed to register RMV callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
 		if (lsc_interrupt) {
@@ -156,6 +158,8 @@
 			if (ret)
 				WARN("Failed to register LSC callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->lsc_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.lsc = lsc_enabled;
 		sdev->state = DEV_ACTIVE;
@@ -282,6 +286,7 @@
 	PRIV(dev)->state = DEV_ACTIVE - 1;
 	FOREACH_SUBDEV_STATE(sdev, i, dev, DEV_ACTIVE) {
 		DEBUG("Closing sub_device %d", i);
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_ACTIVE - 1;
 	}
diff --git a/drivers/net/failsafe/failsafe_private.h b/drivers/net/failsafe/failsafe_private.h
index 7e6a3f8..3222653 100644
--- a/drivers/net/failsafe/failsafe_private.h
+++ b/drivers/net/failsafe/failsafe_private.h
@@ -119,6 +119,8 @@ struct sub_device {
 	volatile unsigned int remove:1;
 	/* flow isolation state */
 	int flow_isolated:1;
+	unsigned int rmv_callback:1;
+	unsigned int lsc_callback:1;
 };
 
 struct fs_priv {
@@ -211,6 +213,7 @@ uint16_t failsafe_tx_burst_fast(void *txq,
 /* ETH_DEV */
 
 int failsafe_eth_dev_state_sync(struct rte_eth_dev *dev);
+void failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev);
 void failsafe_dev_remove(struct rte_eth_dev *dev);
 void failsafe_stats_increment(struct rte_eth_stats *to,
 				struct rte_eth_stats *from);
-- 
1.9.5

[PATCH 2/2] net/failsafe: fix duplicate event registraton

[Matan Azrad]

When the fail-safe device is reconfigured, it attempts to register
again for the sub-devices LSC and RMV events.

Prevent an event registration if it is already done.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---
 drivers/net/failsafe/failsafe_ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index e0570b6..24e91c9 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -138,7 +138,7 @@
 			fs_unlock(dev, 0);
 			return ret;
 		}
-		if (rmv_interrupt) {
+		if (rmv_interrupt && sdev->rmv_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 					RTE_ETH_EVENT_INTR_RMV,
 					failsafe_eth_rmv_event_callback,
@@ -150,7 +150,7 @@
 				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
-		if (lsc_interrupt) {
+		if (lsc_interrupt && sdev->lsc_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 						RTE_ETH_EVENT_INTR_LSC,
 						failsafe_eth_lsc_event_callback,
-- 
1.9.5

Re: [PATCH 1/2] net/failsafe: fix removed sub-device cleanup

[Ophir Munk]

Hi,
Please find comments inline.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matan Azrad
> Sent: Thursday, May 17, 2018 9:52 PM
> To: Gaetan Rivet <gaetan.rivet@6wind.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: [dpdk-dev] [PATCH 1/2] net/failsafe: fix removed sub-device cleanup
> 
> The fail-safe PMD registers to RMV event for each removable sub-device
> port in order to cleanup the sub-device resources and switch the Tx sub-
> device directly when it is plugged-out.
> 
> In the removal time, the fail-safe PMD stops and closes the sub-device but it

During removal time...

> doesn't unregister the LSC and RMV callbacks of the sub-device port.
> 
> It can lead the callbacks to be called for a port which is no more associated
> to the fail-safe sub-device, because there is not a guaranty that a sub-device

associated with........ guarantee that

> gets the same port ID for each plug-in process. This port, for example, may
> belong to another sub-device of a different fail-safe device.
> 
> Unregister the LSC and RMV callbacks for sub-devices which are not used.
> 
> Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> ---
>  drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
>  drivers/net/failsafe/failsafe_ops.c     |  5 +++++
>  drivers/net/failsafe/failsafe_private.h |  3 +++
>  3 files changed, 30 insertions(+)
> 
> diff --git a/drivers/net/failsafe/failsafe_ether.c
> b/drivers/net/failsafe/failsafe_ether.c
> index 733e95d..2bbee82 100644
> --- a/drivers/net/failsafe/failsafe_ether.c
> +++ b/drivers/net/failsafe/failsafe_ether.c
> @@ -260,6 +260,7 @@
>  		sdev->state = DEV_ACTIVE;
>  		/* fallthrough */
>  	case DEV_ACTIVE:
> +		failsafe_eth_dev_unregister_callbacks(sdev);
>  		rte_eth_dev_close(PORT_ID(sdev));
>  		sdev->state = DEV_PROBED;
>  		/* fallthrough */
> @@ -321,6 +322,27 @@
>  }
> 
>  void
> +failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev) {
> +	if (sdev == NULL)
> +		return;
> +	if (sdev->rmv_callback) {
> +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> +					RTE_ETH_EVENT_INTR_RMV,
> +					failsafe_eth_rmv_event_callback,
> +					sdev);

Please justify ignoring returned error from rte_eth_dev_callback_unregister call. 
I have noticed that this call returned error is ignored in other places in code (failsafe.c)...but if for example the callback returns with -EAGAIN shouldn't you try again to unregister?
Need to avoid a case where the callback is still registered, while rmv_callback is assigned to 0

> +		sdev->rmv_callback = 0;
> +	}
> +	if (sdev->lsc_callback) {
> +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> +					RTE_ETH_EVENT_INTR_LSC,
> +					failsafe_eth_lsc_event_callback,
> +					sdev);

Same comment here regarding the returned error from rte_eth_dev_callback_unregister call and lsc_event_callback.

> +		sdev->lsc_callback = 0;
> +	}
> +}
> +
> +void
>  failsafe_dev_remove(struct rte_eth_dev *dev)  {
>  	struct sub_device *sdev;
> diff --git a/drivers/net/failsafe/failsafe_ops.c
> b/drivers/net/failsafe/failsafe_ops.c
> index d04277b..e0570b6 100644
> --- a/drivers/net/failsafe/failsafe_ops.c
> +++ b/drivers/net/failsafe/failsafe_ops.c
> @@ -146,6 +146,8 @@
>  			if (ret)
>  				WARN("Failed to register RMV callback for
> sub_device %d",
>  				     SUB_ID(sdev));
> +			else
> +				sdev->rmv_callback = 1;
>  		}
>  		dev->data->dev_conf.intr_conf.rmv = 0;
>  		if (lsc_interrupt) {
> @@ -156,6 +158,8 @@
>  			if (ret)
>  				WARN("Failed to register LSC callback for
> sub_device %d",
>  				     SUB_ID(sdev));
> +			else
> +				sdev->lsc_callback = 1;
>  		}
>  		dev->data->dev_conf.intr_conf.lsc = lsc_enabled;
>  		sdev->state = DEV_ACTIVE;
> @@ -282,6 +286,7 @@
>  	PRIV(dev)->state = DEV_ACTIVE - 1;
>  	FOREACH_SUBDEV_STATE(sdev, i, dev, DEV_ACTIVE) {
>  		DEBUG("Closing sub_device %d", i);
> +		failsafe_eth_dev_unregister_callbacks(sdev);

DEBUG("Closing...") message should be in its right place just before closing sdev (next line).

>  		rte_eth_dev_close(PORT_ID(sdev));
>  		sdev->state = DEV_ACTIVE - 1;
>  	}
> diff --git a/drivers/net/failsafe/failsafe_private.h
> b/drivers/net/failsafe/failsafe_private.h
> index 7e6a3f8..3222653 100644
> --- a/drivers/net/failsafe/failsafe_private.h
> +++ b/drivers/net/failsafe/failsafe_private.h
> @@ -119,6 +119,8 @@ struct sub_device {
>  	volatile unsigned int remove:1;
>  	/* flow isolation state */
>  	int flow_isolated:1;
> +	unsigned int rmv_callback:1;
> +	unsigned int lsc_callback:1;

Nit-pick: please consider adding a description for rmv_callback and lsc_callback similar to the other fields in this struct.

>  };
> 
>  struct fs_priv {
> @@ -211,6 +213,7 @@ uint16_t failsafe_tx_burst_fast(void *txq,
>  /* ETH_DEV */
> 
>  int failsafe_eth_dev_state_sync(struct rte_eth_dev *dev);
> +void failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev);
>  void failsafe_dev_remove(struct rte_eth_dev *dev);  void
> failsafe_stats_increment(struct rte_eth_stats *to,
>  				struct rte_eth_stats *from);
> --
> 1.9.5

[PATCH v2 1/2] net/failsafe: fix removed sub-device cleanup

[Matan Azrad]

The fail-safe PMD registers to RMV event for each removable sub-device
port in order to cleanup the sub-device resources and switch the Tx
sub-device directly when it is plugged-out.

During removal time, the fail-safe PMD stops and closes the sub-device
but it doesn't unregister the LSC and RMV callbacks of the sub-device
port.

It can lead the callbacks to be called for a port which is no more
associated with the fail-safe sub-device, because there is not a
guarantee that a sub-device gets the same port ID for each plug-in
process. This port, for example, may belong to another sub-device of a
different fail-safe device.

Unregister the LSC and RMV callbacks for sub-devices which are not
used.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---
 drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
 drivers/net/failsafe/failsafe_ops.c     |  5 +++++
 drivers/net/failsafe/failsafe_private.h |  5 +++++
 3 files changed, 32 insertions(+)

V2:
Improve the commit log and add code comments for the new sub-dev fields (Ophir suggestion).


diff --git a/drivers/net/failsafe/failsafe_ether.c b/drivers/net/failsafe/failsafe_ether.c
index 733e95d..2bbee82 100644
--- a/drivers/net/failsafe/failsafe_ether.c
+++ b/drivers/net/failsafe/failsafe_ether.c
@@ -260,6 +260,7 @@
 		sdev->state = DEV_ACTIVE;
 		/* fallthrough */
 	case DEV_ACTIVE:
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_PROBED;
 		/* fallthrough */
@@ -321,6 +322,27 @@
 }
 
 void
+failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev)
+{
+	if (sdev == NULL)
+		return;
+	if (sdev->rmv_callback) {
+		rte_eth_dev_callback_unregister(PORT_ID(sdev),
+					RTE_ETH_EVENT_INTR_RMV,
+					failsafe_eth_rmv_event_callback,
+					sdev);
+		sdev->rmv_callback = 0;
+	}
+	if (sdev->lsc_callback) {
+		rte_eth_dev_callback_unregister(PORT_ID(sdev),
+					RTE_ETH_EVENT_INTR_LSC,
+					failsafe_eth_lsc_event_callback,
+					sdev);
+		sdev->lsc_callback = 0;
+	}
+}
+
+void
 failsafe_dev_remove(struct rte_eth_dev *dev)
 {
 	struct sub_device *sdev;
diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index d04277b..e0570b6 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -146,6 +146,8 @@
 			if (ret)
 				WARN("Failed to register RMV callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
 		if (lsc_interrupt) {
@@ -156,6 +158,8 @@
 			if (ret)
 				WARN("Failed to register LSC callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->lsc_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.lsc = lsc_enabled;
 		sdev->state = DEV_ACTIVE;
@@ -282,6 +286,7 @@
 	PRIV(dev)->state = DEV_ACTIVE - 1;
 	FOREACH_SUBDEV_STATE(sdev, i, dev, DEV_ACTIVE) {
 		DEBUG("Closing sub_device %d", i);
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_ACTIVE - 1;
 	}
diff --git a/drivers/net/failsafe/failsafe_private.h b/drivers/net/failsafe/failsafe_private.h
index 7e6a3f8..886af86 100644
--- a/drivers/net/failsafe/failsafe_private.h
+++ b/drivers/net/failsafe/failsafe_private.h
@@ -119,6 +119,10 @@ struct sub_device {
 	volatile unsigned int remove:1;
 	/* flow isolation state */
 	int flow_isolated:1;
+	/* RMV callback registration state */
+	unsigned int rmv_callback:1;
+	/* LSC callback registration state */
+	unsigned int lsc_callback:1;
 };
 
 struct fs_priv {
@@ -211,6 +215,7 @@ uint16_t failsafe_tx_burst_fast(void *txq,
 /* ETH_DEV */
 
 int failsafe_eth_dev_state_sync(struct rte_eth_dev *dev);
+void failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev);
 void failsafe_dev_remove(struct rte_eth_dev *dev);
 void failsafe_stats_increment(struct rte_eth_stats *to,
 				struct rte_eth_stats *from);
-- 
1.9.5

[PATCH v2 2/2] net/failsafe: fix duplicate event registration

[Matan Azrad]

When the fail-safe device is reconfigured, it attempts to register
again for the sub-devices LSC and RMV events.

Prevent an event registration if it is already done.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---
 drivers/net/failsafe/failsafe_ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index e0570b6..24e91c9 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -138,7 +138,7 @@
 			fs_unlock(dev, 0);
 			return ret;
 		}
-		if (rmv_interrupt) {
+		if (rmv_interrupt && sdev->rmv_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 					RTE_ETH_EVENT_INTR_RMV,
 					failsafe_eth_rmv_event_callback,
@@ -150,7 +150,7 @@
 				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
-		if (lsc_interrupt) {
+		if (lsc_interrupt && sdev->lsc_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 						RTE_ETH_EVENT_INTR_LSC,
 						failsafe_eth_lsc_event_callback,
-- 
1.9.5

Re: [PATCH v2 1/2] net/failsafe: fix removed sub-device cleanup

[Gaëtan Rivet]

Hello Matan,

On Mon, May 21, 2018 at 07:48:03PM +0000, Matan Azrad wrote:
> The fail-safe PMD registers to RMV event for each removable sub-device
> port in order to cleanup the sub-device resources and switch the Tx
> sub-device directly when it is plugged-out.
> 
> During removal time, the fail-safe PMD stops and closes the sub-device
> but it doesn't unregister the LSC and RMV callbacks of the sub-device
> port.
> 
> It can lead the callbacks to be called for a port which is no more
> associated with the fail-safe sub-device, because there is not a
> guarantee that a sub-device gets the same port ID for each plug-in
> process. This port, for example, may belong to another sub-device of a
> different fail-safe device.
> 
> Unregister the LSC and RMV callbacks for sub-devices which are not
> used.
> 
> Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> ---
>  drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
>  drivers/net/failsafe/failsafe_ops.c     |  5 +++++
>  drivers/net/failsafe/failsafe_private.h |  5 +++++
>  3 files changed, 32 insertions(+)
> 
> V2:
> Improve the commit log and add code comments for the new sub-dev fields (Ophir suggestion).
> 
> 
> diff --git a/drivers/net/failsafe/failsafe_ether.c b/drivers/net/failsafe/failsafe_ether.c
> index 733e95d..2bbee82 100644
> --- a/drivers/net/failsafe/failsafe_ether.c
> +++ b/drivers/net/failsafe/failsafe_ether.c
> @@ -260,6 +260,7 @@
>  		sdev->state = DEV_ACTIVE;
>  		/* fallthrough */
>  	case DEV_ACTIVE:
> +		failsafe_eth_dev_unregister_callbacks(sdev);
>  		rte_eth_dev_close(PORT_ID(sdev));
>  		sdev->state = DEV_PROBED;
>  		/* fallthrough */
> @@ -321,6 +322,27 @@
>  }
>  
>  void
> +failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev)
> +{
> +	if (sdev == NULL)
> +		return;
> +	if (sdev->rmv_callback) {
> +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> +					RTE_ETH_EVENT_INTR_RMV,
> +					failsafe_eth_rmv_event_callback,
> +					sdev);
> +		sdev->rmv_callback = 0;

I agree with Ophir here, either the return value should not be ignored,
and rmv_callback should only be set to 0 on success, or a proper
justification (and an accompanying comment) should be given.

The issue I could see is that even on error, there won't be a process to
try again unregistering the callback.

Maybe this could be added in failsafe_dev_remove()? Something like

FOREACH_SUBDEV(sdev, i, dev) {
    if (sdev->rmv_callback && sdev->state <= DEV_PROBED)
        if (rte_eth_dev_callback_unregister(...) == 0)
            sdev->rmv_callback = 0;
    /* same for lsc_callback */
}

Does it make sense to you? Do you think this is necessary, or should we
ignore this?

Thanks,
-- 
Gaëtan Rivet
6WIND

Re: [PATCH v2 1/2] net/failsafe: fix removed sub-device cleanup

[Matan Azrad]

Hi Gaetan

From: Gaëtan Rivet
> Hello Matan,
> 
> On Mon, May 21, 2018 at 07:48:03PM +0000, Matan Azrad wrote:
> > The fail-safe PMD registers to RMV event for each removable sub-device
> > port in order to cleanup the sub-device resources and switch the Tx
> > sub-device directly when it is plugged-out.
> >
> > During removal time, the fail-safe PMD stops and closes the sub-device
> > but it doesn't unregister the LSC and RMV callbacks of the sub-device
> > port.
> >
> > It can lead the callbacks to be called for a port which is no more
> > associated with the fail-safe sub-device, because there is not a
> > guarantee that a sub-device gets the same port ID for each plug-in
> > process. This port, for example, may belong to another sub-device of a
> > different fail-safe device.
> >
> > Unregister the LSC and RMV callbacks for sub-devices which are not
> > used.
> >
> > Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Matan Azrad <matan@mellanox.com>
> > ---
> >  drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
> >  drivers/net/failsafe/failsafe_ops.c     |  5 +++++
> >  drivers/net/failsafe/failsafe_private.h |  5 +++++
> >  3 files changed, 32 insertions(+)
> >
> > V2:
> > Improve the commit log and add code comments for the new sub-dev fields
> (Ophir suggestion).
> >
> >
> > diff --git a/drivers/net/failsafe/failsafe_ether.c
> > b/drivers/net/failsafe/failsafe_ether.c
> > index 733e95d..2bbee82 100644
> > --- a/drivers/net/failsafe/failsafe_ether.c
> > +++ b/drivers/net/failsafe/failsafe_ether.c
> > @@ -260,6 +260,7 @@
> >  		sdev->state = DEV_ACTIVE;
> >  		/* fallthrough */
> >  	case DEV_ACTIVE:
> > +		failsafe_eth_dev_unregister_callbacks(sdev);
> >  		rte_eth_dev_close(PORT_ID(sdev));
> >  		sdev->state = DEV_PROBED;
> >  		/* fallthrough */
> > @@ -321,6 +322,27 @@
> >  }
> >
> >  void
> > +failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev) {
> > +	if (sdev == NULL)
> > +		return;
> > +	if (sdev->rmv_callback) {
> > +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> > +					RTE_ETH_EVENT_INTR_RMV,
> > +					failsafe_eth_rmv_event_callback,
> > +					sdev);
> > +		sdev->rmv_callback = 0;
> 
> I agree with Ophir here, either the return value should not be ignored, and
> rmv_callback should only be set to 0 on success, or a proper justification (and
> an accompanying comment) should be given.
> 
> The issue I could see is that even on error, there won't be a process to try again
> unregistering the callback.
> 
> Maybe this could be added in failsafe_dev_remove()? Something like
> 
> FOREACH_SUBDEV(sdev, i, dev) {
>     if (sdev->rmv_callback && sdev->state <= DEV_PROBED)
>         if (rte_eth_dev_callback_unregister(...) == 0)
>             sdev->rmv_callback = 0;
>     /* same for lsc_callback */
> }
> 
> Does it make sense to you? Do you think this is necessary, or should we ignore
> this?

The RMV\LSC event callbacks are called from the host thread and also the removal process is running from the host thread so I think EAGAIN is not expected in the removal time.
Other error (EINVAL) may return again every attempt and probably points to another critical issue.

Is a code comment for the above enough? Or you think we still need to check it?


> Thanks,
> --
> Gaëtan Rivet
> 6WIND

Re: [PATCH v2 1/2] net/failsafe: fix removed sub-device cleanup

[Gaëtan Rivet]

On Tue, May 22, 2018 at 10:19:14AM +0000, Matan Azrad wrote:
> Hi Gaetan
> 
> From: Gaëtan Rivet
> > Hello Matan,
> > 
> > On Mon, May 21, 2018 at 07:48:03PM +0000, Matan Azrad wrote:
> > > The fail-safe PMD registers to RMV event for each removable sub-device
> > > port in order to cleanup the sub-device resources and switch the Tx
> > > sub-device directly when it is plugged-out.
> > >
> > > During removal time, the fail-safe PMD stops and closes the sub-device
> > > but it doesn't unregister the LSC and RMV callbacks of the sub-device
> > > port.
> > >
> > > It can lead the callbacks to be called for a port which is no more
> > > associated with the fail-safe sub-device, because there is not a
> > > guarantee that a sub-device gets the same port ID for each plug-in
> > > process. This port, for example, may belong to another sub-device of a
> > > different fail-safe device.
> > >
> > > Unregister the LSC and RMV callbacks for sub-devices which are not
> > > used.
> > >
> > > Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Matan Azrad <matan@mellanox.com>
> > > ---
> > >  drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
> > >  drivers/net/failsafe/failsafe_ops.c     |  5 +++++
> > >  drivers/net/failsafe/failsafe_private.h |  5 +++++
> > >  3 files changed, 32 insertions(+)
> > >
> > > V2:
> > > Improve the commit log and add code comments for the new sub-dev fields
> > (Ophir suggestion).
> > >
> > >
> > > diff --git a/drivers/net/failsafe/failsafe_ether.c
> > > b/drivers/net/failsafe/failsafe_ether.c
> > > index 733e95d..2bbee82 100644
> > > --- a/drivers/net/failsafe/failsafe_ether.c
> > > +++ b/drivers/net/failsafe/failsafe_ether.c
> > > @@ -260,6 +260,7 @@
> > >  		sdev->state = DEV_ACTIVE;
> > >  		/* fallthrough */
> > >  	case DEV_ACTIVE:
> > > +		failsafe_eth_dev_unregister_callbacks(sdev);
> > >  		rte_eth_dev_close(PORT_ID(sdev));
> > >  		sdev->state = DEV_PROBED;
> > >  		/* fallthrough */
> > > @@ -321,6 +322,27 @@
> > >  }
> > >
> > >  void
> > > +failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev) {
> > > +	if (sdev == NULL)
> > > +		return;
> > > +	if (sdev->rmv_callback) {
> > > +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> > > +					RTE_ETH_EVENT_INTR_RMV,
> > > +					failsafe_eth_rmv_event_callback,
> > > +					sdev);
> > > +		sdev->rmv_callback = 0;
> > 
> > I agree with Ophir here, either the return value should not be ignored, and
> > rmv_callback should only be set to 0 on success, or a proper justification (and
> > an accompanying comment) should be given.
> > 
> > The issue I could see is that even on error, there won't be a process to try again
> > unregistering the callback.
> > 
> > Maybe this could be added in failsafe_dev_remove()? Something like
> > 
> > FOREACH_SUBDEV(sdev, i, dev) {
> >     if (sdev->rmv_callback && sdev->state <= DEV_PROBED)
> >         if (rte_eth_dev_callback_unregister(...) == 0)
> >             sdev->rmv_callback = 0;
> >     /* same for lsc_callback */
> > }
> > 
> > Does it make sense to you? Do you think this is necessary, or should we ignore
> > this?
> 
> The RMV\LSC event callbacks are called from the host thread and also the removal process is running from the host thread so I think EAGAIN is not expected in the removal time.
> Other error (EINVAL) may return again every attempt and probably points to another critical issue.
> 
> Is a code comment for the above enough? Or you think we still need to check it?
> 
> 

Ok, that makes sense.

If EINVAL is possible however, I think a warning would be helpful for
the user to be aware of the issue. The callback flag would then be
meaningless anyway.

-- 
Gaëtan Rivet
6WIND

Re: [PATCH v2 1/2] net/failsafe: fix removed sub-device cleanup

[Matan Azrad]

From: Gaëtan Rivet
> On Tue, May 22, 2018 at 10:19:14AM +0000, Matan Azrad wrote:
> > Hi Gaetan
> >
> > From: Gaëtan Rivet
> > > Hello Matan,
> > >
> > > On Mon, May 21, 2018 at 07:48:03PM +0000, Matan Azrad wrote:
> > > > The fail-safe PMD registers to RMV event for each removable
> > > > sub-device port in order to cleanup the sub-device resources and
> > > > switch the Tx sub-device directly when it is plugged-out.
> > > >
> > > > During removal time, the fail-safe PMD stops and closes the
> > > > sub-device but it doesn't unregister the LSC and RMV callbacks of
> > > > the sub-device port.
> > > >
> > > > It can lead the callbacks to be called for a port which is no more
> > > > associated with the fail-safe sub-device, because there is not a
> > > > guarantee that a sub-device gets the same port ID for each plug-in
> > > > process. This port, for example, may belong to another sub-device
> > > > of a different fail-safe device.
> > > >
> > > > Unregister the LSC and RMV callbacks for sub-devices which are not
> > > > used.
> > > >
> > > > Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Matan Azrad <matan@mellanox.com>
> > > > ---
> > > >  drivers/net/failsafe/failsafe_ether.c   | 22 ++++++++++++++++++++++
> > > >  drivers/net/failsafe/failsafe_ops.c     |  5 +++++
> > > >  drivers/net/failsafe/failsafe_private.h |  5 +++++
> > > >  3 files changed, 32 insertions(+)
> > > >
> > > > V2:
> > > > Improve the commit log and add code comments for the new sub-dev
> > > > fields
> > > (Ophir suggestion).
> > > >
> > > >
> > > > diff --git a/drivers/net/failsafe/failsafe_ether.c
> > > > b/drivers/net/failsafe/failsafe_ether.c
> > > > index 733e95d..2bbee82 100644
> > > > --- a/drivers/net/failsafe/failsafe_ether.c
> > > > +++ b/drivers/net/failsafe/failsafe_ether.c
> > > > @@ -260,6 +260,7 @@
> > > >  		sdev->state = DEV_ACTIVE;
> > > >  		/* fallthrough */
> > > >  	case DEV_ACTIVE:
> > > > +		failsafe_eth_dev_unregister_callbacks(sdev);
> > > >  		rte_eth_dev_close(PORT_ID(sdev));
> > > >  		sdev->state = DEV_PROBED;
> > > >  		/* fallthrough */
> > > > @@ -321,6 +322,27 @@
> > > >  }
> > > >
> > > >  void
> > > > +failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev) {
> > > > +	if (sdev == NULL)
> > > > +		return;
> > > > +	if (sdev->rmv_callback) {
> > > > +		rte_eth_dev_callback_unregister(PORT_ID(sdev),
> > > > +					RTE_ETH_EVENT_INTR_RMV,
> > > > +					failsafe_eth_rmv_event_callback,
> > > > +					sdev);
> > > > +		sdev->rmv_callback = 0;
> > >
> > > I agree with Ophir here, either the return value should not be
> > > ignored, and rmv_callback should only be set to 0 on success, or a
> > > proper justification (and an accompanying comment) should be given.
> > >
> > > The issue I could see is that even on error, there won't be a
> > > process to try again unregistering the callback.
> > >
> > > Maybe this could be added in failsafe_dev_remove()? Something like
> > >
> > > FOREACH_SUBDEV(sdev, i, dev) {
> > >     if (sdev->rmv_callback && sdev->state <= DEV_PROBED)
> > >         if (rte_eth_dev_callback_unregister(...) == 0)
> > >             sdev->rmv_callback = 0;
> > >     /* same for lsc_callback */
> > > }
> > >
> > > Does it make sense to you? Do you think this is necessary, or should
> > > we ignore this?
> >
> > The RMV\LSC event callbacks are called from the host thread and also the
> removal process is running from the host thread so I think EAGAIN is not
> expected in the removal time.
> > Other error (EINVAL) may return again every attempt and probably points to
> another critical issue.
> >
> > Is a code comment for the above enough? Or you think we still need to check
> it?
> >
> >
> 
> Ok, that makes sense.
> 
> If EINVAL is possible however, I think a warning would be helpful for the user to
> be aware of the issue. The callback flag would then be meaningless anyway.

Ok, thanks, V3 is coming.

> 
> --
> Gaëtan Rivet
> 6WIND

[PATCH v3 1/2] net/failsafe: fix removed sub-device cleanup

[Matan Azrad]

The fail-safe PMD registers to RMV event for each removable sub-device
port in order to cleanup the sub-device resources and switch the Tx
sub-device directly when it is plugged-out.

During removal time, the fail-safe PMD stops and closes the sub-device
but it doesn't unregister the LSC and RMV callbacks of the sub-device
port.

It can lead the callbacks to be called for a port which is no more
associated with the fail-safe sub-device, because there is not a
guarantee that a sub-device gets the same port ID for each plug-in
process. This port, for example, may belong to another sub-device of a
different fail-safe device.

Unregister the LSC and RMV callbacks for sub-devices which are not
used.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---


V2:
Improve the commit log and add code comments for the new sub-dev fields (Ophir suggestion).

V3:
Add warning message if a sub-device event callback unregistration fails (Ophir + Gaetan suggestion). 


 drivers/net/failsafe/failsafe_ether.c   | 30 ++++++++++++++++++++++++++++++
 drivers/net/failsafe/failsafe_ops.c     |  5 +++++
 drivers/net/failsafe/failsafe_private.h |  5 +++++
 3 files changed, 40 insertions(+)

diff --git a/drivers/net/failsafe/failsafe_ether.c b/drivers/net/failsafe/failsafe_ether.c
index 733e95d..5b5cb3b 100644
--- a/drivers/net/failsafe/failsafe_ether.c
+++ b/drivers/net/failsafe/failsafe_ether.c
@@ -260,6 +260,7 @@
 		sdev->state = DEV_ACTIVE;
 		/* fallthrough */
 	case DEV_ACTIVE:
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_PROBED;
 		/* fallthrough */
@@ -321,6 +322,35 @@
 }
 
 void
+failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev)
+{
+	int ret;
+
+	if (sdev == NULL)
+		return;
+	if (sdev->rmv_callback) {
+		ret = rte_eth_dev_callback_unregister(PORT_ID(sdev),
+						RTE_ETH_EVENT_INTR_RMV,
+						failsafe_eth_rmv_event_callback,
+						sdev);
+		if (ret)
+			WARN("Failed to unregister RMV callback for sub_device"
+			     " %d", SUB_ID(sdev));
+		sdev->rmv_callback = 0;
+	}
+	if (sdev->lsc_callback) {
+		ret = rte_eth_dev_callback_unregister(PORT_ID(sdev),
+						RTE_ETH_EVENT_INTR_LSC,
+						failsafe_eth_lsc_event_callback,
+						sdev);
+		if (ret)
+			WARN("Failed to unregister LSC callback for sub_device"
+			     " %d", SUB_ID(sdev));
+		sdev->lsc_callback = 0;
+	}
+}
+
+void
 failsafe_dev_remove(struct rte_eth_dev *dev)
 {
 	struct sub_device *sdev;
diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index d04277b..e0570b6 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -146,6 +146,8 @@
 			if (ret)
 				WARN("Failed to register RMV callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
 		if (lsc_interrupt) {
@@ -156,6 +158,8 @@
 			if (ret)
 				WARN("Failed to register LSC callback for sub_device %d",
 				     SUB_ID(sdev));
+			else
+				sdev->lsc_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.lsc = lsc_enabled;
 		sdev->state = DEV_ACTIVE;
@@ -282,6 +286,7 @@
 	PRIV(dev)->state = DEV_ACTIVE - 1;
 	FOREACH_SUBDEV_STATE(sdev, i, dev, DEV_ACTIVE) {
 		DEBUG("Closing sub_device %d", i);
+		failsafe_eth_dev_unregister_callbacks(sdev);
 		rte_eth_dev_close(PORT_ID(sdev));
 		sdev->state = DEV_ACTIVE - 1;
 	}
diff --git a/drivers/net/failsafe/failsafe_private.h b/drivers/net/failsafe/failsafe_private.h
index 7e6a3f8..886af86 100644
--- a/drivers/net/failsafe/failsafe_private.h
+++ b/drivers/net/failsafe/failsafe_private.h
@@ -119,6 +119,10 @@ struct sub_device {
 	volatile unsigned int remove:1;
 	/* flow isolation state */
 	int flow_isolated:1;
+	/* RMV callback registration state */
+	unsigned int rmv_callback:1;
+	/* LSC callback registration state */
+	unsigned int lsc_callback:1;
 };
 
 struct fs_priv {
@@ -211,6 +215,7 @@ uint16_t failsafe_tx_burst_fast(void *txq,
 /* ETH_DEV */
 
 int failsafe_eth_dev_state_sync(struct rte_eth_dev *dev);
+void failsafe_eth_dev_unregister_callbacks(struct sub_device *sdev);
 void failsafe_dev_remove(struct rte_eth_dev *dev);
 void failsafe_stats_increment(struct rte_eth_stats *to,
 				struct rte_eth_stats *from);
-- 
1.9.5

[PATCH v3 2/2] net/failsafe: fix duplicate event registration

[Matan Azrad]

When the fail-safe device is reconfigured, it attempts to register
again for the sub-devices LSC and RMV events.

Prevent an event registration if it is already done.

Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
---
 drivers/net/failsafe/failsafe_ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/failsafe/failsafe_ops.c b/drivers/net/failsafe/failsafe_ops.c
index e0570b6..24e91c9 100644
--- a/drivers/net/failsafe/failsafe_ops.c
+++ b/drivers/net/failsafe/failsafe_ops.c
@@ -138,7 +138,7 @@
 			fs_unlock(dev, 0);
 			return ret;
 		}
-		if (rmv_interrupt) {
+		if (rmv_interrupt && sdev->rmv_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 					RTE_ETH_EVENT_INTR_RMV,
 					failsafe_eth_rmv_event_callback,
@@ -150,7 +150,7 @@
 				sdev->rmv_callback = 1;
 		}
 		dev->data->dev_conf.intr_conf.rmv = 0;
-		if (lsc_interrupt) {
+		if (lsc_interrupt && sdev->lsc_callback == 0) {
 			ret = rte_eth_dev_callback_register(PORT_ID(sdev),
 						RTE_ETH_EVENT_INTR_LSC,
 						failsafe_eth_lsc_event_callback,
-- 
1.9.5

Re: [PATCH v3 1/2] net/failsafe: fix removed sub-device cleanup

[Gaëtan Rivet]

On Tue, May 22, 2018 at 12:38:46PM +0000, Matan Azrad wrote:
> The fail-safe PMD registers to RMV event for each removable sub-device
> port in order to cleanup the sub-device resources and switch the Tx
> sub-device directly when it is plugged-out.
> 
> During removal time, the fail-safe PMD stops and closes the sub-device
> but it doesn't unregister the LSC and RMV callbacks of the sub-device
> port.
> 
> It can lead the callbacks to be called for a port which is no more
> associated with the fail-safe sub-device, because there is not a
> guarantee that a sub-device gets the same port ID for each plug-in
> process. This port, for example, may belong to another sub-device of a
> different fail-safe device.
> 
> Unregister the LSC and RMV callbacks for sub-devices which are not
> used.
> 
> Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>

Acked-by: Gaetan Rivet <gaetan.rivet@6wind.com>


-- 
Gaëtan Rivet
6WIND

Re: [PATCH v3 2/2] net/failsafe: fix duplicate event registration

[Gaëtan Rivet]

On Tue, May 22, 2018 at 12:38:47PM +0000, Matan Azrad wrote:
> When the fail-safe device is reconfigured, it attempts to register
> again for the sub-devices LSC and RMV events.
> 
> Prevent an event registration if it is already done.
> 
> Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Gaetan Rivet <gaetan.rivet@6wind.com>

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-stable] [PATCH v3 1/2] net/failsafe: fix removed sub-device cleanup

[Ferruh Yigit]

On 5/22/2018 2:14 PM, Gaëtan Rivet wrote:
> On Tue, May 22, 2018 at 12:38:46PM +0000, Matan Azrad wrote:
>> The fail-safe PMD registers to RMV event for each removable sub-device
>> port in order to cleanup the sub-device resources and switch the Tx
>> sub-device directly when it is plugged-out.
>>
>> During removal time, the fail-safe PMD stops and closes the sub-device
>> but it doesn't unregister the LSC and RMV callbacks of the sub-device
>> port.
>>
>> It can lead the callbacks to be called for a port which is no more
>> associated with the fail-safe sub-device, because there is not a
>> guarantee that a sub-device gets the same port ID for each plug-in
>> process. This port, for example, may belong to another sub-device of a
>> different fail-safe device.
>>
>> Unregister the LSC and RMV callbacks for sub-devices which are not
>> used.
>>
>> Fixes: 598fb8aec6f6 ("net/failsafe: support device removal")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Matan Azrad <matan@mellanox.com>
> 
> Acked-by: Gaetan Rivet <gaetan.rivet@6wind.com>

Series applied to dpdk-next-net/master, thanks.