[PATCH V2 0/6] [PATCH 0/6] app/testpmd: support attach and detach port for MP

[PATCH V2 0/6] [PATCH 0/6] app/testpmd: support attach and detach port for MP

[Huisong Li]

This patchset do some bugfix and add attach and detach port for MP.

---
 -v2:
  resend due to CI unexplained failure. 

Huisong Li (6):
  bus/pci: fix a segfault when call callback
  bus/vdev: fix a segfault when call callback
  ethdev: fix push new event
  app/testpmd: check the validity of the port
  app/testpmd: support attach and detach port for MP
  app/testpmd: stop packet forwarding in new and destroy event

 app/test-pmd/testpmd.c                | 85 +++++++++++++++++----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/bus/pci/pci_common.c          | 13 +++-
 drivers/bus/vdev/vdev.c               | 11 +++-
 drivers/net/bonding/bonding_testpmd.c |  1 -
 lib/ethdev/ethdev_driver.c            |  2 +-
 6 files changed, 75 insertions(+), 38 deletions(-)

-- 
2.22.0

[PATCH V2 1/6] bus/pci: fix a segfault when call callback

[Huisong Li]

After the driver probe is executed, the callback in application will
be called. The callback in application may call some APIs which access the
rte_pci_driver::driver by the device::driver pointer to get driver
information. If the rte_pci_device::device::driver pointer isn't pointed to
rte_pci_driver::driver in rte_pci_probe_one_driver, a segfault will occur.
For example, when ethdev driver probe completes, the callback in
application call rte_eth_dev_info_get which use dev->device->driver->name.
So rte_pci_device::device::driver should point to rte_pci_driver::driver
before executing the driver probe.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/pci/pci_common.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 37ab879779..831a9cd8c7 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -265,11 +265,22 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 			dr->driver.name, dev->id.vendor_id, dev->id.device_id,
 			loc->domain, loc->bus, loc->devid, loc->function,
 			dev->device.numa_node);
+
+	/*
+	 * After the driver probe is executed, the callback in application will
+	 * be called. The callback in application may call some APIs which use
+	 * dev->device.driver to get some driver information. If the driver
+	 * pointer isn't pointed to driver->driver here, a segfault will occur.
+	 */
+	if (!already_probed)
+		dev->device.driver = &dr->driver;
+
 	/* call the driver probe() function */
 	ret = dr->probe(dr, dev);
 	if (already_probed)
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
+		dev->device.driver = NULL;
 		dev->driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
@@ -282,8 +293,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH V2 2/6] bus/vdev: fix a segfault when call callback

[Huisong Li]

After the driver probe is executed, the callback in application will be
called. And this callback may call some APIs which access the driver in
struct rte_vdev_driver by the device::driver pointer to get some driver
information. If the rte_vdev_device::device::driver pointer isn't pointed
to the rte_vdev_driver::driver before executing driver probe, a segfault
will occur.

Fixes: e9d159c3d534 ("eal: allow probing a device again")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/vdev/vdev.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index a8d8b2327e..dea3937607 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -209,9 +209,16 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * After the driver probe is executed, the callback in application will
+	 * be called. The callback in application may call some APIs which use
+	 * dev->device.driver to get some driver information. If the driver
+	 * pointer isn't pointed to driver->driver here, a segfault will occur.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
-- 
2.22.0

[PATCH V2 3/6] ethdev: fix push new event

[Huisong Li]

The 'state' in struct rte_eth_dev may be used to update some information
when app receive these events. For example, when app receives a new event,
app may get the socket id of this port by calling rte_eth_dev_socket_id to
setup the attached port. The 'state' is used in rte_eth_dev_socket_id.

If the state isn't modified to RTE_ETH_DEV_ATTACHED before pushing the new
event, app will get the socket id failed. So this patch moves pushing event
operation after the state updated.

Fixes: 99a2dd955fba ("lib: remove librte_ prefix from directory names")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 lib/ethdev/ethdev_driver.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index a285f213f0..a6616f072b 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -206,9 +206,9 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
-	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
+	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 }
 
 int
-- 
2.22.0

[PATCH V2 4/6] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 77741fc41f..1605e34f35 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3673,14 +3673,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH V2 5/6] app/testpmd: support attach and detach port for MP

[Huisong Li]

This patch supports attach and detach port in primary and secondary
process.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 app/test-pmd/testpmd.c                | 76 +++++++++++++++++----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 47 insertions(+), 31 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1605e34f35..10b330e33c 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3323,9 +3323,6 @@ reset_port(portid_t pid)
 void
 attach_port(char *identifier)
 {
-	portid_t pi;
-	struct rte_dev_iterator iterator;
-
 	printf("Attaching a new port...\n");
 
 	if (identifier == NULL) {
@@ -3337,24 +3334,6 @@ attach_port(char *identifier)
 		TESTPMD_LOG(ERR, "Failed to attach port %s\n", identifier);
 		return;
 	}
-
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
-		return;
-	}
-
-	/* second attach mode: iterator */
-	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
-		/* setup ports matching the devargs used for probing */
-		if (port_is_forwarding(pi))
-			continue; /* port was already attached before */
-		setup_attached_port(pi);
-	}
 }
 
 static void
@@ -3377,7 +3356,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3411,11 +3389,10 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", dev->name);
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	return;
 }
 
@@ -3481,11 +3458,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3649,11 +3624,52 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void config_attached_port(portid_t port_id)
+{
+	struct rte_dev_iterator iterator;
+	struct rte_eth_dev_info dev_info;
+	portid_t pi;
+	int ret;
+
+	/* first attach mode: event */
+	if (setup_on_probe_event) {
+		/* new ports are detected on RTE_ETH_EVENT_NEW event */
+		setup_attached_port(port_id);
+		return;
+	}
+
+	ret = eth_dev_info_get_print_err(port_id, &dev_info);
+	if (ret != 0) {
+		TESTPMD_LOG(ERR,
+			"Failed to get device info for port %d, not detaching\n",
+			port_id);
+		return;
+	}
+	/* second attach mode: iterator */
+	RTE_ETH_FOREACH_MATCHING_DEV(pi, dev_info.device->name, &iterator) {
+		/* setup ports matching the devargs used for probing */
+		if (port_is_forwarding(port_id))
+			continue; /* port was already attached before */
+		setup_attached_port(port_id);
+	}
+}
+
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3678,8 +3694,7 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		config_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3690,6 +3705,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index ddf5e21849..22261f20a2 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -278,7 +278,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index 3941f4cf23..ed1a278a44 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -763,7 +763,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 			port_id, rte_strerror(-ret));
 
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH V2 6/6] app/testpmd: stop packet forwarding in new and destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 10b330e33c..faea6358f9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3694,6 +3694,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		config_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
@@ -3703,6 +3705,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

Re: [PATCH V2 4/6] app/testpmd: check the validity of the port

[Singh, Aman Deep]

On 9/15/2022 6:15 PM, Huisong Li wrote:
> This patch checks the validity of port id for all events in
> 'eth_event_callback()'.
> 
> Signed-off-by: Huisong Li <lihuisong@huawei.com>Acked-by: Aman Singh <aman.deep.singh@intel.com>
> ---

Re: [PATCH V2 3/6] ethdev: fix push new event

[Thomas Monjalon]

15/09/2022 14:45, Huisong Li:
> The 'state' in struct rte_eth_dev may be used to update some information
> when app receive these events. For example, when app receives a new event,
> app may get the socket id of this port by calling rte_eth_dev_socket_id to
> setup the attached port. The 'state' is used in rte_eth_dev_socket_id.
> 
> If the state isn't modified to RTE_ETH_DEV_ATTACHED before pushing the new
> event, app will get the socket id failed. So this patch moves pushing event
> operation after the state updated.
> 
> Fixes: 99a2dd955fba ("lib: remove librte_ prefix from directory names")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> ---
>  lib/ethdev/ethdev_driver.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
> index a285f213f0..a6616f072b 100644
> --- a/lib/ethdev/ethdev_driver.c
> +++ b/lib/ethdev/ethdev_driver.c
> @@ -206,9 +206,9 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
>  	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>  		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
>  
> -	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>  
>  	dev->state = RTE_ETH_DEV_ATTACHED;
> +	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>  }

As explained in the first patch, I don't think it is a good solution.
We should not allow the port to be used until the end of probing.
When RTE_ETH_EVENT_NEW is sent, the device is allocated but
not ready for use. If an entity like failsafe decides to take ownership
of the port, then the application should not consider it at all.
For these reasons, we should limit which operations can be done
during RTE_ETH_EVENT_NEW processing.
That's why I've proposed creating a new state RTE_ETH_DEV_ALLOCATED,
not sure why you didn't follow this advice.

Re: [PATCH V2 3/6] ethdev: fix push new event

[lihuisong (C)]

在 2022/9/27 18:49, Thomas Monjalon 写道:
> 15/09/2022 14:45, Huisong Li:
>> The 'state' in struct rte_eth_dev may be used to update some information
>> when app receive these events. For example, when app receives a new event,
>> app may get the socket id of this port by calling rte_eth_dev_socket_id to
>> setup the attached port. The 'state' is used in rte_eth_dev_socket_id.
>>
>> If the state isn't modified to RTE_ETH_DEV_ATTACHED before pushing the new
>> event, app will get the socket id failed. So this patch moves pushing event
>> operation after the state updated.
>>
>> Fixes: 99a2dd955fba ("lib: remove librte_ prefix from directory names")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>> ---
>>   lib/ethdev/ethdev_driver.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
>> index a285f213f0..a6616f072b 100644
>> --- a/lib/ethdev/ethdev_driver.c
>> +++ b/lib/ethdev/ethdev_driver.c
>> @@ -206,9 +206,9 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
>>   	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>>   		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
>>   
>> -	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>>   
>>   	dev->state = RTE_ETH_DEV_ATTACHED;
>> +	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>>   }
> As explained in the first patch, I don't think it is a good solution.
> We should not allow the port to be used until the end of probing.
> When RTE_ETH_EVENT_NEW is sent, the device is allocated but
> not ready for use. If an entity like failsafe decides to take ownership
> of the port, then the application should not consider it at all.
> For these reasons, we should limit which operations can be done
> during RTE_ETH_EVENT_NEW processing.
> That's why I've proposed creating a new state RTE_ETH_DEV_ALLOCATED,
> not sure why you didn't follow this advice.
I want to put this problem to this case this patchset mentions, so as to
have a better discussion. From the first patch, I know what you mean.
But I still have some confusion:
When a device taken by failsafe PMD push new event, all event callback
will be called under this device. As you suggested, the device is a valid
port if its state is ALLOCATED or ATTACHED. Now, the macro 
RTE_ETH_FOREACH_DEV
uses the condition that device is valid(state is ALLOCATED or ATTACHED)
and NOT owned.

Even if we add the new ALLOCATED state, this device taken by failsafe is
also a valid port in new event callback in application, and is in 'NO 
OWNER'.
if event callback of application is called before one of failsafe PMD.
As a result, application still can operate this device directly.
Can we make sure that event callback of failsafe PMD is before one of 
application?
>
>
>
>
> .

Re: [PATCH V2 1/6] bus/pci: fix a segfault when call callback

[Thomas Monjalon]

15/09/2022 14:45, Huisong Li:
> After the driver probe is executed, the callback in application will
> be called. The callback in application may call some APIs which access the
> rte_pci_driver::driver by the device::driver pointer to get driver
> information. If the rte_pci_device::device::driver pointer isn't pointed to
> rte_pci_driver::driver in rte_pci_probe_one_driver, a segfault will occur.
> For example, when ethdev driver probe completes, the callback in
> application call rte_eth_dev_info_get which use dev->device->driver->name.
> So rte_pci_device::device::driver should point to rte_pci_driver::driver
> before executing the driver probe.
> 
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org

To be more precise, it is reverting
391797f04208 ("drivers/bus: move driver assignment to end of probing")

dev->device.driver is used by rte_dev_is_probed():

int
rte_dev_is_probed(const struct rte_device *dev)
{
    /* The field driver should be set only when the probe is successful. */
    return dev->driver != NULL;
}

And the field comment is clear in rte_device:

const struct rte_driver *driver; /**< Driver assigned after probing */

That's why I am not enthusiastic about setting this pointer before probing.

I understand it is more convenient to use this pointer
in a probing callback.
We need to check it is not breaking rte_dev_is_probed() usage.
It may be OK if there is no parallel probing,
and rte_dev_is_probed() is not called inside probing.

At the very minimum, we need to update some comments in the code,
to mention that the pointer is set before probing,
and reset if probing failed.

Re: [PATCH V2 1/6] bus/pci: fix a segfault when call callback

[lihuisong (C)]

Hi Thomas,

I missed this e-mail, I'm sorry for late reply.

在 2022/10/11 3:49, Thomas Monjalon 写道:
> 15/09/2022 14:45, Huisong Li:
>> After the driver probe is executed, the callback in application will
>> be called. The callback in application may call some APIs which access the
>> rte_pci_driver::driver by the device::driver pointer to get driver
>> information. If the rte_pci_device::device::driver pointer isn't pointed to
>> rte_pci_driver::driver in rte_pci_probe_one_driver, a segfault will occur.
>> For example, when ethdev driver probe completes, the callback in
>> application call rte_eth_dev_info_get which use dev->device->driver->name.
>> So rte_pci_device::device::driver should point to rte_pci_driver::driver
>> before executing the driver probe.
>>
>> Fixes: c752998b5e2e ("pci: introduce library and driver")
>> Cc: stable@dpdk.org
> To be more precise, it is reverting
> 391797f04208 ("drivers/bus: move driver assignment to end of probing")
>
> dev->device.driver is used by rte_dev_is_probed():
>
> int
> rte_dev_is_probed(const struct rte_device *dev)
> {
>      /* The field driver should be set only when the probe is successful. */
>      return dev->driver != NULL;
> }
>
> And the field comment is clear in rte_device:
>
> const struct rte_driver *driver; /**< Driver assigned after probing */
>
> That's why I am not enthusiastic about setting this pointer before probing.
+1
>
> I understand it is more convenient to use this pointer
> in a probing callback.
> We need to check it is not breaking rte_dev_is_probed() usage.
> It may be OK if there is no parallel probing,
> and rte_dev_is_probed() is not called inside probing.
 From all the places where it's used, first of all, it is a internal 
interface
and only used when scan device on bus, remove device and hotplug device.

In process of scanning or probing device, the core code checks whether the
device is already probed by calling rte_dev_is_probed(). So I think it is ok
for parallel probing(if exist) to set dev->device.driver before probing 
successful.
The Concurrency between probing and removing should not be possible. The 
application
removes a device after device probing successful.

Currently, rte_dev_is_probed() isn't called inside probing.
>
> At the very minimum, we need to update some comments in the code,
> to mention that the pointer is set before probing,
> and reset if probing failed.
If it is ok for you, I will update some comments and add this 
modification to
other bus type in next version.
>
>
> .

Re: [PATCH V2 3/6] ethdev: fix push new event

[lihuisong (C)]

在 2022/10/8 12:09, lihuisong (C) 写道:
>
> 在 2022/9/27 18:49, Thomas Monjalon 写道:
>> 15/09/2022 14:45, Huisong Li:
>>> The 'state' in struct rte_eth_dev may be used to update some 
>>> information
>>> when app receive these events. For example, when app receives a new 
>>> event,
>>> app may get the socket id of this port by calling 
>>> rte_eth_dev_socket_id to
>>> setup the attached port. The 'state' is used in rte_eth_dev_socket_id.
>>>
>>> If the state isn't modified to RTE_ETH_DEV_ATTACHED before pushing 
>>> the new
>>> event, app will get the socket id failed. So this patch moves 
>>> pushing event
>>> operation after the state updated.
>>>
>>> Fixes: 99a2dd955fba ("lib: remove librte_ prefix from directory names")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>>> ---
>>>   lib/ethdev/ethdev_driver.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
>>> index a285f213f0..a6616f072b 100644
>>> --- a/lib/ethdev/ethdev_driver.c
>>> +++ b/lib/ethdev/ethdev_driver.c
>>> @@ -206,9 +206,9 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
>>>       if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>>>           eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, 
>>> dev);
>>>   -    rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>>>         dev->state = RTE_ETH_DEV_ATTACHED;
>>> +    rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>>>   }
>> As explained in the first patch, I don't think it is a good solution.
>> We should not allow the port to be used until the end of probing.
>> When RTE_ETH_EVENT_NEW is sent, the device is allocated but
>> not ready for use. If an entity like failsafe decides to take ownership
>> of the port, then the application should not consider it at all.
>> For these reasons, we should limit which operations can be done
>> during RTE_ETH_EVENT_NEW processing.
>> That's why I've proposed creating a new state RTE_ETH_DEV_ALLOCATED,
>> not sure why you didn't follow this advice.
> I want to put this problem to this case this patchset mentions, so as to
> have a better discussion. From the first patch, I know what you mean.
> But I still have some confusion:
> When a device taken by failsafe PMD push new event, all event callback
> will be called under this device. As you suggested, the device is a valid
> port if its state is ALLOCATED or ATTACHED. Now, the macro 
> RTE_ETH_FOREACH_DEV
> uses the condition that device is valid(state is ALLOCATED or ATTACHED)
> and NOT owned.
>
> Even if we add the new ALLOCATED state, this device taken by failsafe is
> also a valid port in new event callback in application, and is in 'NO 
> OWNER'.
> if event callback of application is called before one of failsafe PMD.
> As a result, application still can operate this device directly.
> Can we make sure that event callback of failsafe PMD is before one of 
> application?
Hi Thomas,
Can you look at my confusion?
>>
>>
>>
>>
>> .
> .

[PATCH V3 0/5] app/testpmd: support mulitple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v3:
   1) merge patch 1/6 and patch 2/6 into patch 1/5, and add some
      similar modifications for other bus type.
   2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 

 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.22.0

[PATCH V3 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff1369353a..13cb3fe0f8 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index bb943b58b5..5f23446f41 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -293,13 +293,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index bc3a7f39fe..efaa1792e9 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 41bc07dde7..2e3f0f2e12 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 8d32d66504..feb1651984 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH V3 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
RTE_ETH_DEV_UNUSED, this port will be skiped.

Currently, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and to
RTE_ETH_DEV_ATTACHED when definitely probed. And this port is valid if its
device state is ALLOCATED or ATTACHED. Naturally, a lot of places had to be
modified for this change.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index b3de490d36..7b08353ed5 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6003,7 +6003,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_in_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e55be8720e..949d8c1367 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3014,7 +3014,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_in_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index 0be1e8ca04..7868958dff 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_in_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_in_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	eth_dev_shared_data_prepare();
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_in_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 6a550cfc83..e04f88fe46 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1542,6 +1542,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is in used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_in_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 94b8fba5d7..362e5c2dec 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_in_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index 838b3a8f9f..a78d503e59 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_in_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 5d5e18db1e..a225141937 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -325,7 +325,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -372,7 +372,7 @@ int
 rte_eth_dev_is_valid_port(uint16_t port_id)
 {
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
 		return 0;
 	else
 		return 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index c129ca1eaf..459ad44021 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2000,7 +2000,9 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index 17201fbe0f..66d70f1aac 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -327,4 +327,5 @@ INTERNAL {
 	rte_eth_representor_id_get;
 	rte_eth_switch_domain_alloc;
 	rte_eth_switch_domain_free;
+	rte_eth_dev_in_used;
 };
-- 
2.22.0

[PATCH V3 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 134d79a555..bc25703490 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3798,14 +3798,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH V3 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

This patch supports attach and detach port in primary and secondary
process.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index bc25703490..2e6329c853 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3463,15 +3463,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3502,7 +3499,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3536,10 +3532,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3606,11 +3600,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3774,11 +3766,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3803,8 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3815,6 +3818,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 7d24d25970..080d3a1139 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -306,7 +306,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index 9529e16fb6..9216271314 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -765,7 +765,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH V3 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 2e6329c853..746f07652a 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3806,6 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3816,6 +3818,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

[PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v4: fix a misspelling. 
 -v3:
   1) merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
      for other bus type.
   2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 
 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.22.0

[PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff1369353a..13cb3fe0f8 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index bb943b58b5..5f23446f41 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -293,13 +293,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index bc3a7f39fe..efaa1792e9 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 41bc07dde7..2e3f0f2e12 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 8d32d66504..feb1651984 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH V4 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
RTE_ETH_DEV_UNUSED, this port will be skipped.

Currently, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and to
RTE_ETH_DEV_ATTACHED when definitely probed. And this port is valid if its
device state is ALLOCATED or ATTACHED. Naturally, a lot of places had to be
modified for this change.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index b3de490d36..7b08353ed5 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6003,7 +6003,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_in_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e55be8720e..949d8c1367 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3014,7 +3014,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_in_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index 0be1e8ca04..7868958dff 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_in_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_in_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	eth_dev_shared_data_prepare();
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_in_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 6a550cfc83..e04f88fe46 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1542,6 +1542,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is in used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_in_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 94b8fba5d7..362e5c2dec 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_in_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index 838b3a8f9f..a78d503e59 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_in_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 5d5e18db1e..a225141937 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -325,7 +325,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -372,7 +372,7 @@ int
 rte_eth_dev_is_valid_port(uint16_t port_id)
 {
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
 		return 0;
 	else
 		return 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index c129ca1eaf..459ad44021 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2000,7 +2000,9 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index 17201fbe0f..66d70f1aac 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -327,4 +327,5 @@ INTERNAL {
 	rte_eth_representor_id_get;
 	rte_eth_switch_domain_alloc;
 	rte_eth_switch_domain_free;
+	rte_eth_dev_in_used;
 };
-- 
2.22.0

[PATCH V4 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 134d79a555..bc25703490 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3798,14 +3798,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH V4 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

This patch supports attach and detach port in primary and secondary
process.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index bc25703490..2e6329c853 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3463,15 +3463,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3502,7 +3499,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3536,10 +3532,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3606,11 +3600,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3774,11 +3766,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3803,8 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3815,6 +3818,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 7d24d25970..080d3a1139 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -306,7 +306,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index 9529e16fb6..9216271314 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -765,7 +765,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH V4 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 2e6329c853..746f07652a 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3806,6 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3816,6 +3818,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[lihuisong (C)]

Hi Thomas and Ferruh,

Could you take a look at this patch series?
I have modified it according to our last discussion.

Best,
Huisong

在 2022/12/6 17:26, Huisong Li 写道:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
>
> ---
>   -v4: fix a misspelling.
>   -v3:
>     1) merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>        for other bus type.
>     2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>        the probelm in patch 2/5.
>   -v2: resend due to CI unexplained failure.
>
> Huisong Li (5):
>    drivers/bus: restore driver assignment at front of probing
>    ethdev: fix skip valid port in probing callback
>    app/testpmd: check the validity of the port
>    app/testpmd: add attach and detach port for multiple process
>    app/testpmd: stop forwarding in new or destroy event
>
>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>   app/test-pmd/testpmd.h                   |  1 -
>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>   drivers/bus/pci/pci_common.c             |  9 ++++-
>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>   drivers/net/mlx5/mlx5.c                  |  2 +-
>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>   lib/ethdev/ethdev_pci.h                  |  2 +-
>   lib/ethdev/rte_class_eth.c               |  2 +-
>   lib/ethdev/rte_ethdev.c                  |  4 +-
>   lib/ethdev/rte_ethdev.h                  |  4 +-
>   lib/ethdev/version.map                   |  1 +
>   19 files changed, 114 insertions(+), 44 deletions(-)
>

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Ferruh Yigit]

On 12/6/2022 9:26 AM, Huisong Li wrote:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
> 
> ---
>  -v4: fix a misspelling. 
>  -v3:
>    1) merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>       for other bus type.
>    2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>       the probelm in patch 2/5. 
>  -v2: resend due to CI unexplained failure.
> 
> Huisong Li (5):
>   drivers/bus: restore driver assignment at front of probing
>   ethdev: fix skip valid port in probing callback
>   app/testpmd: check the validity of the port
>   app/testpmd: add attach and detach port for multiple process
>   app/testpmd: stop forwarding in new or destroy event
> 

Hi Huisong,

I haven't checked the patch in detail yet, but I can see it gives some
ABI compatibility warnings, is this expected:


1 function with some indirect sub-type change:

  [C] 'function int dpaa_eth_eventq_attach(const rte_eth_dev*, int, u16,
const rte_event_eth_rx_adapter_queue_conf*)' at dpaa_ethdev.c:1149:1 has
some indirect sub-type changes:
    parameter 1 of type 'const rte_eth_dev*' has sub-type changes:
      in pointed to type 'const rte_eth_dev':
        in unqualified underlying type 'struct rte_eth_dev' at
ethdev_driver.h:50:1:
          type size hasn't changed
          1 data member change:
            type of 'rte_eth_dev_state state' changed:
              type size hasn't changed
              1 enumerator insertion:
                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
              2 enumerator changes:
                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
to '2' at rte_ethdev.h:2000:1
                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2'
to '3' at rte_ethdev.h:2000:1

1 function with some indirect sub-type change:

  [C] 'function int rte_pmd_i40e_set_switch_dev(uint16_t, rte_eth_dev*)'
at rte_pmd_i40e.c:3266:1 has some indirect sub-type changes:
    parameter 2 of type 'rte_eth_dev*' has sub-type changes:
      in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
        type size hasn't changed
        1 data member change:
          type of 'rte_eth_dev_state state' changed:
            type size hasn't changed
            1 enumerator insertion:
              'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
            2 enumerator changes:
              'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
to '2' at rte_ethdev.h:2000:1
              'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
'3' at rte_ethdev.h:2000:1

1 function with some indirect sub-type change:

  [C] 'function rte_eth_dev* rte_eth_dev_allocate(const char*)' at
ethdev_driver.c:72:1 has some indirect sub-type changes:
    return type changed:
      in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
        type size hasn't changed
        1 data member change:
          type of 'rte_eth_dev_state state' changed:
            type size hasn't changed
            1 enumerator insertion:
              'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
            2 enumerator changes:
              'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
to '2' at rte_ethdev.h:2000:1
              'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
'3' at rte_ethdev.h:2000:1

... there are more warnings for same issue ...

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[lihuisong (C)]

在 2023/1/11 0:51, Ferruh Yigit 写道:
> On 12/6/2022 9:26 AM, Huisong Li wrote:
>> This patchset fix some bugs and support attaching and detaching port
>> in primary and secondary.
>>
>> ---
>>   -v4: fix a misspelling.
>>   -v3:
>>     1) merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>>        for other bus type.
>>     2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>        the probelm in patch 2/5.
>>   -v2: resend due to CI unexplained failure.
>>
>> Huisong Li (5):
>>    drivers/bus: restore driver assignment at front of probing
>>    ethdev: fix skip valid port in probing callback
>>    app/testpmd: check the validity of the port
>>    app/testpmd: add attach and detach port for multiple process
>>    app/testpmd: stop forwarding in new or destroy event
>>
> Hi Huisong,
>
> I haven't checked the patch in detail yet, but I can see it gives some
> ABI compatibility warnings, is this expected:
This is to be expected. Because we insert a device state, 
RTE_ETH_DEV_ALLOCATED,
before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
We may have to announce it. What do you think?
>
> 1 function with some indirect sub-type change:
>
>    [C] 'function int dpaa_eth_eventq_attach(const rte_eth_dev*, int, u16,
> const rte_event_eth_rx_adapter_queue_conf*)' at dpaa_ethdev.c:1149:1 has
> some indirect sub-type changes:
>      parameter 1 of type 'const rte_eth_dev*' has sub-type changes:
>        in pointed to type 'const rte_eth_dev':
>          in unqualified underlying type 'struct rte_eth_dev' at
> ethdev_driver.h:50:1:
>            type size hasn't changed
>            1 data member change:
>              type of 'rte_eth_dev_state state' changed:
>                type size hasn't changed
>                1 enumerator insertion:
>                  'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>                2 enumerator changes:
>                  'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
> to '2' at rte_ethdev.h:2000:1
>                  'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2'
> to '3' at rte_ethdev.h:2000:1
>
> 1 function with some indirect sub-type change:
>
>    [C] 'function int rte_pmd_i40e_set_switch_dev(uint16_t, rte_eth_dev*)'
> at rte_pmd_i40e.c:3266:1 has some indirect sub-type changes:
>      parameter 2 of type 'rte_eth_dev*' has sub-type changes:
>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>          type size hasn't changed
>          1 data member change:
>            type of 'rte_eth_dev_state state' changed:
>              type size hasn't changed
>              1 enumerator insertion:
>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>              2 enumerator changes:
>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
> to '2' at rte_ethdev.h:2000:1
>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
> '3' at rte_ethdev.h:2000:1
>
> 1 function with some indirect sub-type change:
>
>    [C] 'function rte_eth_dev* rte_eth_dev_allocate(const char*)' at
> ethdev_driver.c:72:1 has some indirect sub-type changes:
>      return type changed:
>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>          type size hasn't changed
>          1 data member change:
>            type of 'rte_eth_dev_state state' changed:
>              type size hasn't changed
>              1 enumerator insertion:
>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>              2 enumerator changes:
>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
> to '2' at rte_ethdev.h:2000:1
>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
> '3' at rte_ethdev.h:2000:1
>
> ... there are more warnings for same issue ...
>
> .

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Ferruh Yigit]

On 1/11/2023 12:53 AM, lihuisong (C) wrote:
> 
> 在 2023/1/11 0:51, Ferruh Yigit 写道:
>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>> This patchset fix some bugs and support attaching and detaching port
>>> in primary and secondary.
>>>
>>> ---
>>>   -v4: fix a misspelling.
>>>   -v3:
>>>     1) merge patch 1/6 and patch 2/6 into patch 1/5, and add
>>> modification
>>>        for other bus type.
>>>     2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>>        the probelm in patch 2/5.
>>>   -v2: resend due to CI unexplained failure.
>>>
>>> Huisong Li (5):
>>>    drivers/bus: restore driver assignment at front of probing
>>>    ethdev: fix skip valid port in probing callback
>>>    app/testpmd: check the validity of the port
>>>    app/testpmd: add attach and detach port for multiple process
>>>    app/testpmd: stop forwarding in new or destroy event
>>>
>> Hi Huisong,
>>
>> I haven't checked the patch in detail yet, but I can see it gives some
>> ABI compatibility warnings, is this expected:
> This is to be expected. Because we insert a device state,
> RTE_ETH_DEV_ALLOCATED,
> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
> We may have to announce it. What do you think?

If there is an actual ABI break, it can't go in this release, need to
wait LTS release and yes needs deprecation notice in advance.

But not all enum value change warnings are real break, need to
investigate all warnings one by one.
Need to investigate if old application & new dpdk library may cause any
unexpected behavior for application.

>>
>> 1 function with some indirect sub-type change:
>>
>>    [C] 'function int dpaa_eth_eventq_attach(const rte_eth_dev*, int, u16,
>> const rte_event_eth_rx_adapter_queue_conf*)' at dpaa_ethdev.c:1149:1 has
>> some indirect sub-type changes:
>>      parameter 1 of type 'const rte_eth_dev*' has sub-type changes:
>>        in pointed to type 'const rte_eth_dev':
>>          in unqualified underlying type 'struct rte_eth_dev' at
>> ethdev_driver.h:50:1:
>>            type size hasn't changed
>>            1 data member change:
>>              type of 'rte_eth_dev_state state' changed:
>>                type size hasn't changed
>>                1 enumerator insertion:
>>                  'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>                2 enumerator changes:
>>                  'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>> to '2' at rte_ethdev.h:2000:1
>>                  'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2'
>> to '3' at rte_ethdev.h:2000:1
>>
>> 1 function with some indirect sub-type change:
>>
>>    [C] 'function int rte_pmd_i40e_set_switch_dev(uint16_t, rte_eth_dev*)'
>> at rte_pmd_i40e.c:3266:1 has some indirect sub-type changes:
>>      parameter 2 of type 'rte_eth_dev*' has sub-type changes:
>>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>          type size hasn't changed
>>          1 data member change:
>>            type of 'rte_eth_dev_state state' changed:
>>              type size hasn't changed
>>              1 enumerator insertion:
>>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>              2 enumerator changes:
>>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>> to '2' at rte_ethdev.h:2000:1
>>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>> '3' at rte_ethdev.h:2000:1
>>
>> 1 function with some indirect sub-type change:
>>
>>    [C] 'function rte_eth_dev* rte_eth_dev_allocate(const char*)' at
>> ethdev_driver.c:72:1 has some indirect sub-type changes:
>>      return type changed:
>>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>          type size hasn't changed
>>          1 data member change:
>>            type of 'rte_eth_dev_state state' changed:
>>              type size hasn't changed
>>              1 enumerator insertion:
>>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>              2 enumerator changes:
>>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>> to '2' at rte_ethdev.h:2000:1
>>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>> '3' at rte_ethdev.h:2000:1
>>
>> ... there are more warnings for same issue ...
>>
>> .

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Ferruh Yigit]

On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
> On 1/11/2023 12:53 AM, lihuisong (C) wrote:
>>
>> 在 2023/1/11 0:51, Ferruh Yigit 写道:
>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>> This patchset fix some bugs and support attaching and detaching port
>>>> in primary and secondary.
>>>>
>>>> ---
>>>>   -v4: fix a misspelling.
>>>>   -v3:
>>>>     1) merge patch 1/6 and patch 2/6 into patch 1/5, and add
>>>> modification
>>>>        for other bus type.
>>>>     2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>>>        the probelm in patch 2/5.
>>>>   -v2: resend due to CI unexplained failure.
>>>>
>>>> Huisong Li (5):
>>>>    drivers/bus: restore driver assignment at front of probing
>>>>    ethdev: fix skip valid port in probing callback
>>>>    app/testpmd: check the validity of the port
>>>>    app/testpmd: add attach and detach port for multiple process
>>>>    app/testpmd: stop forwarding in new or destroy event
>>>>
>>> Hi Huisong,
>>>
>>> I haven't checked the patch in detail yet, but I can see it gives some
>>> ABI compatibility warnings, is this expected:
>> This is to be expected. Because we insert a device state,
>> RTE_ETH_DEV_ALLOCATED,
>> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
>> We may have to announce it. What do you think?
> 
> If there is an actual ABI break, it can't go in this release, need to
> wait LTS release and yes needs deprecation notice in advance.
> 
> But not all enum value change warnings are real break, need to
> investigate all warnings one by one.
> Need to investigate if old application & new dpdk library may cause any
> unexpected behavior for application.
> 

OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
enum solves the issue, although logically it won't look nice.
Perhaps order can be fixed in next LTS, to have more logical order, but
not quite sure if order worth the disturbance may cause in application.

>>>
>>> 1 function with some indirect sub-type change:
>>>
>>>    [C] 'function int dpaa_eth_eventq_attach(const rte_eth_dev*, int, u16,
>>> const rte_event_eth_rx_adapter_queue_conf*)' at dpaa_ethdev.c:1149:1 has
>>> some indirect sub-type changes:
>>>      parameter 1 of type 'const rte_eth_dev*' has sub-type changes:
>>>        in pointed to type 'const rte_eth_dev':
>>>          in unqualified underlying type 'struct rte_eth_dev' at
>>> ethdev_driver.h:50:1:
>>>            type size hasn't changed
>>>            1 data member change:
>>>              type of 'rte_eth_dev_state state' changed:
>>>                type size hasn't changed
>>>                1 enumerator insertion:
>>>                  'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>                2 enumerator changes:
>>>                  'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>> to '2' at rte_ethdev.h:2000:1
>>>                  'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2'
>>> to '3' at rte_ethdev.h:2000:1
>>>
>>> 1 function with some indirect sub-type change:
>>>
>>>    [C] 'function int rte_pmd_i40e_set_switch_dev(uint16_t, rte_eth_dev*)'
>>> at rte_pmd_i40e.c:3266:1 has some indirect sub-type changes:
>>>      parameter 2 of type 'rte_eth_dev*' has sub-type changes:
>>>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>>          type size hasn't changed
>>>          1 data member change:
>>>            type of 'rte_eth_dev_state state' changed:
>>>              type size hasn't changed
>>>              1 enumerator insertion:
>>>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>              2 enumerator changes:
>>>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>> to '2' at rte_ethdev.h:2000:1
>>>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>>> '3' at rte_ethdev.h:2000:1
>>>
>>> 1 function with some indirect sub-type change:
>>>
>>>    [C] 'function rte_eth_dev* rte_eth_dev_allocate(const char*)' at
>>> ethdev_driver.c:72:1 has some indirect sub-type changes:
>>>      return type changed:
>>>        in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>>          type size hasn't changed
>>>          1 data member change:
>>>            type of 'rte_eth_dev_state state' changed:
>>>              type size hasn't changed
>>>              1 enumerator insertion:
>>>                'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>              2 enumerator changes:
>>>                'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>> to '2' at rte_ethdev.h:2000:1
>>>                'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>>> '3' at rte_ethdev.h:2000:1
>>>
>>> ... there are more warnings for same issue ...
>>>
>>> .
> 

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[Ferruh Yigit]

On 12/6/2022 9:26 AM, Huisong Li wrote:
> The driver assignment was moved back at the end of the device probing
> because there is no something to use rte_driver during the phase of
> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
> to end of probing")
> 
> However, it is necessary for probing callback to reference rte_driver
> before probing. For example, probing callback may call some APIs which
> access the rte_pci_driver::driver by the device::driver pointer to get
> driver information. In this case, a segment fault will occur in probing
> callback if there is not this assignment.
> 

Probing callback gets driver as parameter, so callback function can
access it via 'drv->driver', is there a specific usecase that
'dev->device->driver' needs to be accessed explicitly?

I assume this is related to coming patches that setting up device in
testpmd event callback, but can you please clarify exact need.

> Further, some comments in code need to be updated if we do that. The
> driver pointer in rte_device is set before probing and needs to be reset
> if probing failed. And rte_dev_is_probed can not be called inside probing.
> 
> Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> ---
>  drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
>  drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
>  drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
>  drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
>  drivers/bus/pci/pci_common.c             |  9 +++++++--
>  drivers/bus/vdev/vdev.c                  | 10 ++++++++--
>  drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
>  7 files changed, 52 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
> index ff1369353a..13cb3fe0f8 100644
> --- a/drivers/bus/auxiliary/auxiliary_common.c
> +++ b/drivers/bus/auxiliary/auxiliary_common.c
> @@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
>  	}
>  
>  	dev->driver = drv;
> +	/*
> +	 * Reference rte_driver before probing so as to this pointer can
> +	 * be used to get driver information in case of segment fault in
> +	 * probing callback.
> +	 */
> +	dev->device.driver = &drv->driver;
>  
>  	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
>  		      drv->driver.name, dev->name, dev->device.numa_node);
>  	ret = drv->probe(drv, dev);
>  	if (ret != 0) {
>  		dev->driver = NULL;
> +		dev->device.driver = NULL;
>  		rte_intr_instance_free(dev->intr_handle);
>  		dev->intr_handle = NULL;
> -	} else {
> -		dev->device.driver = &drv->driver;
>  	}
>  
>  	return ret;
> diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
> index e57159f5d8..f1b817e58c 100644
> --- a/drivers/bus/dpaa/dpaa_bus.c
> +++ b/drivers/bus/dpaa/dpaa_bus.c
> @@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
>  			    (dev->device.devargs &&
>  			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
>  				continue;
> -
> +			/*
> +			 * Reference rte_driver before probing so as to this
> +			 * pointer can be used to get driver information in case
> +			 * of segment fault in probing callback.
> +			 */
> +			dev->device.driver = &drv->driver;
>  			if (probe_all ||
>  			    (dev->device.devargs &&
>  			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
>  				ret = drv->probe(drv, dev);
>  				if (ret) {
> +					dev->device.driver = NULL;
>  					DPAA_BUS_ERR("unable to probe:%s",
>  						     dev->name);
>  				} else {
>  					dev->driver = drv;
> -					dev->device.driver = &drv->driver;
>  				}
>  			}
>  			break;
> diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
> index 57bfb5111a..4bc0c6d3d4 100644
> --- a/drivers/bus/fslmc/fslmc_bus.c
> +++ b/drivers/bus/fslmc/fslmc_bus.c
> @@ -471,15 +471,21 @@ rte_fslmc_probe(void)
>  				continue;
>  			}
>  
> +			/*
> +			 * Reference rte_driver before probing so as to this
> +			 * pointer can be used to get driver information in case
> +			 * of segment fault in probing callback.
> +			 */
> +			dev->device.driver = &drv->driver;
>  			if (probe_all ||
>  			   (dev->device.devargs &&
>  			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
>  				ret = drv->probe(drv, dev);
>  				if (ret) {
> +					dev->device.driver = NULL;
>  					DPAA2_BUS_ERR("Unable to probe");
>  				} else {
>  					dev->driver = drv;
> -					dev->device.driver = &drv->driver;
>  				}
>  			}
>  			break;
> diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
> index bb943b58b5..5f23446f41 100644
> --- a/drivers/bus/ifpga/ifpga_bus.c
> +++ b/drivers/bus/ifpga/ifpga_bus.c
> @@ -293,13 +293,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
>  
>  	/* reference driver structure */
>  	afu_dev->driver = drv;
> +	/*
> +	 * Reference rte_driver before probing so as to this pointer can
> +	 * be used to get driver information in case of segment fault in
> +	 * probing callback.
> +	 */
> +	afu_dev->device.driver = &drv->driver;
>  
>  	/* call the driver probe() function */
>  	ret = drv->probe(afu_dev);
> -	if (ret)
> +	if (ret) {
>  		afu_dev->driver = NULL;
> -	else
> -		afu_dev->device.driver = &drv->driver;
> +		afu_dev->device.driver = NULL;
> +	}
>  
>  	return ret;
>  }
> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
> index bc3a7f39fe..efaa1792e9 100644
> --- a/drivers/bus/pci/pci_common.c
> +++ b/drivers/bus/pci/pci_common.c
> @@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>  				return ret;
>  			}
>  		}
> +		/*
> +		 * Reference rte_driver before probing so as to this pointer can
> +		 * be used to get driver information in case of segment fault in
> +		 * probing callback.
> +		 */
> +		dev->device.driver = &dr->driver;
>  	}
>  
>  	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
> @@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>  		return ret; /* no rollback if already succeeded earlier */
>  	if (ret) {
>  		dev->driver = NULL;
> +		dev->device.driver = NULL;
>  		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
>  			/* Don't unmap if device is unsupported and
>  			 * driver needs mapped resources.
> @@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>  		dev->vfio_req_intr_handle = NULL;
>  		rte_intr_instance_free(dev->intr_handle);
>  		dev->intr_handle = NULL;
> -	} else {
> -		dev->device.driver = &dr->driver;
>  	}
>  
>  	return ret;
> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
> index 41bc07dde7..2e3f0f2e12 100644
> --- a/drivers/bus/vdev/vdev.c
> +++ b/drivers/bus/vdev/vdev.c
> @@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
>  		return -1;
>  	}
>  
> +	/*
> +	 * Reference rte_driver before probing so as to this pointer can
> +	 * be used to get driver information in case of segment fault in
> +	 * probing callback.
> +	 */
> +	dev->device.driver = &driver->driver;
>  	ret = driver->probe(dev);
> -	if (ret == 0)
> -		dev->device.driver = &driver->driver;
> +	if (ret != 0)
> +		dev->device.driver = NULL;
>  	return ret;
>  }
>  
> diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
> index 8d32d66504..feb1651984 100644
> --- a/drivers/bus/vmbus/vmbus_common.c
> +++ b/drivers/bus/vmbus/vmbus_common.c
> @@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
>  
>  	/* reference driver structure */
>  	dev->driver = dr;
> +	/*
> +	 * Reference rte_driver before probing so as to this pointer can
> +	 * be used to get driver information in case of segment fault in
> +	 * probing callback.
> +	 */
> +	dev->device.driver = &dr->driver;
>  
>  	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
>  		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
> @@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
>  	ret = dr->probe(dr, dev);
>  	if (ret) {
>  		dev->driver = NULL;
> +		dev->device.driver = NULL;
>  		rte_vmbus_unmap_device(dev);
> -	} else {
> -		dev->device.driver = &dr->driver;
>  	}
>  
>  	return ret;

Re: [PATCH V4 2/5] ethdev: fix skip valid port in probing callback

[Ferruh Yigit]

On 12/6/2022 9:26 AM, Huisong Li wrote:
> The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
> iterate over all enabled ports to do something(like, verifying the port id
> validity) when receive a probing event. If the ethdev state of a port is
> RTE_ETH_DEV_UNUSED, this port will be skipped.
> 
> Currently, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
> event. It means that probing callback will skip this port. But this
> assignment can not move to front of probing notification. See
> commit be8cd210379a ("ethdev: fix port probing notification")
> 

OK to abstract ethdev state check to a function, 'rte_eth_dev_in_used'.
BTW, I guess intention is to say 'is' instead of 'in', like
'rte_eth_dev_is_used'.

But not sure if new state is necessary.
I can see from next patches that if testpmd event callback wants to
validate port_id, it is using 'port_id_is_invalid()' function which is
using 'RTE_ETH_FOREACH_DEV' macro.
And using 'RTE_ETH_FOREACH_DEV' macro in callback function has the
problem you described above.

1) Do we really need to validate port_id in the event callback, callback
provided 'port_id' from 'rte_eth_dev_callback_process()' API, from
'dev->data->port_id'. I am not sure if there is case that
'dev->data->port_id' can be invalid.

2) is there any other specific usecase to use 'RTE_ETH_FOREACH_DEV' in
the 'RTE_ETH_EVENT_NEW' event callback function?



> So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
> state to RTE_ETH_DEV_ALLOCATED before pushing probing event and to
> RTE_ETH_DEV_ATTACHED when definitely probed. And this port is valid if its
> device state is ALLOCATED or ATTACHED. Naturally, a lot of places had to be
> modified for this change.
> 
> Fixes: be8cd210379a ("ethdev: fix port probing notification")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> ---
>  drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
>  drivers/net/mlx5/mlx5.c        |  2 +-
>  lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
>  lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
>  lib/ethdev/ethdev_pci.h        |  2 +-
>  lib/ethdev/rte_class_eth.c     |  2 +-
>  lib/ethdev/rte_ethdev.c        |  4 ++--
>  lib/ethdev/rte_ethdev.h        |  4 +++-
>  lib/ethdev/version.map         |  1 +
>  9 files changed, 33 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
> index b3de490d36..7b08353ed5 100644
> --- a/drivers/net/bnxt/bnxt_ethdev.c
> +++ b/drivers/net/bnxt/bnxt_ethdev.c
> @@ -6003,7 +6003,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
>  
>  	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
>  
> -	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
> +
> +	if (rte_eth_dev_in_used(eth_dev->state))
>  		bnxt_dev_close_op(eth_dev);
>  
>  	return 0;
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
> index e55be8720e..949d8c1367 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -3014,7 +3014,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
>  	while (port_id < RTE_MAX_ETHPORTS) {
>  		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
>  
> -		if (dev->state != RTE_ETH_DEV_UNUSED &&
> +		if (rte_eth_dev_in_used(dev->state) &&
>  		    dev->device &&
>  		    (dev->device == odev ||
>  		     (dev->device->driver &&
> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
> index 0be1e8ca04..7868958dff 100644
> --- a/lib/ethdev/ethdev_driver.c
> +++ b/lib/ethdev/ethdev_driver.c
> @@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
>  	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
>  		/* Using shared name field to find a free port. */
>  		if (eth_dev_shared_data->data[i].name[0] == '\0') {
> -			RTE_ASSERT(rte_eth_devices[i].state ==
> -				   RTE_ETH_DEV_UNUSED);
> +			RTE_ASSERT(!rte_eth_dev_in_used(
> +					rte_eth_devices[i].state));
>  			return i;
>  		}
>  	}
> @@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
>  	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>  		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
>  
> +	dev->state = RTE_ETH_DEV_ALLOCATED;
>  	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>  
>  	dev->state = RTE_ETH_DEV_ATTACHED;
>  }
>  
> +bool rte_eth_dev_in_used(uint16_t dev_state)
> +{
> +	return dev_state == RTE_ETH_DEV_ALLOCATED ||
> +		dev_state == RTE_ETH_DEV_ATTACHED;
> +}
> +
>  int
>  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>  {
> @@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>  
>  	eth_dev_shared_data_prepare();
>  
> -	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
> +	if (rte_eth_dev_in_used(eth_dev->state))
>  		rte_eth_dev_callback_process(eth_dev,
>  				RTE_ETH_EVENT_DESTROY, NULL);
>  
> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> index 6a550cfc83..e04f88fe46 100644
> --- a/lib/ethdev/ethdev_driver.h
> +++ b/lib/ethdev/ethdev_driver.h
> @@ -1542,6 +1542,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
>  __rte_internal
>  void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
>  
> +/**
> + * Check if a Ethernet device state is in used or not
> + *
> + * @param dev_state
> + *   The state of the Ethernet device
> + * @return
> + *   - true if the state of the Ethernet device is allocated or attached
> + *   - false if this state is neither allocated nor attached
> + */
> +__rte_internal
> +bool rte_eth_dev_in_used(uint16_t dev_state);
> +
>  /**
>   * Create memzone for HW rings.
>   * malloc can't be used as the physical address is needed.
> diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
> index 94b8fba5d7..362e5c2dec 100644
> --- a/lib/ethdev/ethdev_pci.h
> +++ b/lib/ethdev/ethdev_pci.h
> @@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
>  	 * eth device has been released.
>  	 */
>  	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
> -	    eth_dev->state == RTE_ETH_DEV_UNUSED)
> +	    !rte_eth_dev_in_used(eth_dev->state))
>  		return 0;
>  
>  	if (dev_uninit) {
> diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
> index 838b3a8f9f..a78d503e59 100644
> --- a/lib/ethdev/rte_class_eth.c
> +++ b/lib/ethdev/rte_class_eth.c
> @@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
>  	const struct rte_kvargs *kvlist = arg->kvlist;
>  	unsigned int pair;
>  
> -	if (edev->state == RTE_ETH_DEV_UNUSED)
> +	if (!rte_eth_dev_in_used(edev->state))
>  		return -1;
>  	if (arg->device != NULL && arg->device != edev->device)
>  		return -1;
> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> index 5d5e18db1e..a225141937 100644
> --- a/lib/ethdev/rte_ethdev.c
> +++ b/lib/ethdev/rte_ethdev.c
> @@ -325,7 +325,7 @@ uint16_t
>  rte_eth_find_next(uint16_t port_id)
>  {
>  	while (port_id < RTE_MAX_ETHPORTS &&
> -			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
> +	       !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
>  		port_id++;
>  
>  	if (port_id >= RTE_MAX_ETHPORTS)
> @@ -372,7 +372,7 @@ int
>  rte_eth_dev_is_valid_port(uint16_t port_id)
>  {
>  	if (port_id >= RTE_MAX_ETHPORTS ||
> -	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
> +	    !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
>  		return 0;
>  	else
>  		return 1;
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index c129ca1eaf..459ad44021 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -2000,7 +2000,9 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
>  enum rte_eth_dev_state {
>  	/** Device is unused before being probed. */
>  	RTE_ETH_DEV_UNUSED = 0,
> -	/** Device is attached when allocated in probing. */
> +	/** Device is allocated and is set before reporting new event. */
> +	RTE_ETH_DEV_ALLOCATED,
> +	/** Device is attached when definitely probed. */
>  	RTE_ETH_DEV_ATTACHED,
>  	/** Device is in removed state when plug-out is detected. */
>  	RTE_ETH_DEV_REMOVED,
> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> index 17201fbe0f..66d70f1aac 100644
> --- a/lib/ethdev/version.map
> +++ b/lib/ethdev/version.map
> @@ -327,4 +327,5 @@ INTERNAL {
>  	rte_eth_representor_id_get;
>  	rte_eth_switch_domain_alloc;
>  	rte_eth_switch_domain_free;
> +	rte_eth_dev_in_used;
>  };

Re: [PATCH V4 4/5] app/testpmd: add attach and detach port for multiple process

[Ferruh Yigit]

On 12/6/2022 9:26 AM, Huisong Li wrote:
> This patch supports attach and detach port in primary and secondary
> process.
> 

Hi Huisong,

This patch moves port setup and remove (via alarm callback) to event
callback,

1) I can see it is for MP but can you please give more details, what was
the problem before, how this solves the issue

2) I am concerned about doing more work on the event callback and
observe some unexpected side effects later, can't we handle this out of
event callback


> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
> ---
>  app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
>  app/test-pmd/testpmd.h                |  1 -
>  drivers/net/bonding/bonding_testpmd.c |  1 -
>  3 files changed, 22 insertions(+), 18 deletions(-)
> 
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index bc25703490..2e6329c853 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -3463,15 +3463,12 @@ attach_port(char *identifier)
>  		return;
>  	}
>  
> -	/* first attach mode: event */
> -	if (setup_on_probe_event) {
> -		/* new ports are detected on RTE_ETH_EVENT_NEW event */
> -		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
> -			if (ports[pi].port_status == RTE_PORT_HANDLING &&
> -					ports[pi].need_setup != 0)
> -				setup_attached_port(pi);
> +	/*
> +	 * first attach mode: event, setting up attached port is done in
> +	 * probing callback.
> +	 */
> +	if (setup_on_probe_event)
>  		return;
> -	}
>  
>  	/* second attach mode: iterator */
>  	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
> @@ -3502,7 +3499,6 @@ setup_attached_port(portid_t pi)
>  	ports_ids[nb_ports++] = pi;
>  	fwd_ports_ids[nb_fwd_ports++] = pi;
>  	nb_cfg_ports = nb_fwd_ports;
> -	ports[pi].need_setup = 0;
>  	ports[pi].port_status = RTE_PORT_STOPPED;
>  
>  	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
> @@ -3536,10 +3532,8 @@ detach_device(struct rte_device *dev)
>  		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
>  		return;
>  	}
> -	remove_invalid_ports();
>  
>  	printf("Device is detached\n");
> -	printf("Now total ports is %d\n", nb_ports);
>  	printf("Done\n");
>  	return;
>  }
> @@ -3606,11 +3600,9 @@ detach_devargs(char *identifier)
>  		return;
>  	}
>  
> -	remove_invalid_ports();
> -
>  	printf("Device %s is detached\n", identifier);
> -	printf("Now total ports is %d\n", nb_ports);
>  	printf("Done\n");
> +
>  	rte_devargs_reset(&da);
>  }
>  
> @@ -3774,11 +3766,22 @@ rmv_port_callback(void *arg)
>  		struct rte_device *device = dev_info.device;
>  		close_port(port_id);
>  		detach_device(device); /* might be already removed or have more ports */
> +		remove_invalid_ports();
> +		printf("Now total ports is %d\n", nb_ports);
>  	}
>  	if (need_to_start)
>  		start_packet_forwarding(0);
>  }
>  
> +static void
> +remove_invalid_ports_callback(void *arg)
> +{
> +	RTE_SET_USED(arg);
> +
> +	remove_invalid_ports();
> +	printf("Now total ports is %d\n", nb_ports);
> +}
> +
>  /* This function is used by the interrupt thread */
>  static int
>  eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
> @@ -3803,8 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>  
>  	switch (type) {
>  	case RTE_ETH_EVENT_NEW:
> -		ports[port_id].need_setup = 1;
> -		ports[port_id].port_status = RTE_PORT_HANDLING;
> +		if (setup_on_probe_event)
> +			setup_attached_port(port_id);
>  		break;
>  	case RTE_ETH_EVENT_INTR_RMV:
>  		if (rte_eal_alarm_set(100000,
> @@ -3815,6 +3818,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>  	case RTE_ETH_EVENT_DESTROY:
>  		ports[port_id].port_status = RTE_PORT_CLOSED;
>  		printf("Port %u is closed\n", port_id);
> +		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
> +			(void *)(intptr_t)port_id))
> +			fprintf(stderr, "Could not set up deferred device released\n");
>  		break;
>  	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
>  		uint16_t rxq_id;
> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
> index 7d24d25970..080d3a1139 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
> @@ -306,7 +306,6 @@ struct rte_port {
>  	uint16_t                tx_vlan_id;/**< The tag ID */
>  	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
>  	volatile uint16_t        port_status;    /**< port started or not */
> -	uint8_t                 need_setup;     /**< port just attached */
>  	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
>  	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
>  	uint8_t                 rss_flag;   /**< enable rss or not */
> diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
> index 9529e16fb6..9216271314 100644
> --- a/drivers/net/bonding/bonding_testpmd.c
> +++ b/drivers/net/bonding/bonding_testpmd.c
> @@ -765,7 +765,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
>  
>  	ports[port_id].update_conf = 1;
>  	ports[port_id].bond_flag = 1;
> -	ports[port_id].need_setup = 0;
>  	ports[port_id].port_status = RTE_PORT_STOPPED;
>  }
>  

Re: [PATCH V4 5/5] app/testpmd: stop forwarding in new or destroy event

[Ferruh Yigit]

On 12/6/2022 9:26 AM, Huisong Li wrote:
> When testpmd receives the new or destroy event, the port related
> information will be updated. Testpmd must stop packet forwarding
> before updating the information to avoid some serious problems.
> 
> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> ---
>  app/test-pmd/testpmd.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 2e6329c853..746f07652a 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -3806,6 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>  
>  	switch (type) {
>  	case RTE_ETH_EVENT_NEW:
> +		if (test_done == 0)
> +			stop_packet_forwarding();

testpmd is test application, why not prevent user to issue attach /
detach commands when packet forwarding is going on, and force user to
stop forwarding explicitly instead of doing this implicitly and silently?

Similar to previous comments, as we make things more complex for
specific use cases it will be very difficult to update testpmd without
hitting unexpected side effects everywhere, at least this is my concern.

>  		if (setup_on_probe_event)
>  			setup_attached_port(port_id);
>  		break;
> @@ -3816,6 +3818,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>  				"Could not set up deferred device removal\n");
>  		break;
>  	case RTE_ETH_EVENT_DESTROY:
> +		if (test_done == 0)
> +			stop_packet_forwarding();
>  		ports[port_id].port_status = RTE_PORT_CLOSED;
>  		printf("Port %u is closed\n", port_id);
>  		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[lihuisong (C)]

在 2023/1/11 18:46, Ferruh Yigit 写道:
> On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
>> On 1/11/2023 12:53 AM, lihuisong (C) wrote:
>>> 在 2023/1/11 0:51, Ferruh Yigit 写道:
>>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>>> This patchset fix some bugs and support attaching and detaching port
>>>>> in primary and secondary.
>>>>>
>>>>> ---
>>>>>    -v4: fix a misspelling.
>>>>>    -v3:
>>>>>      1) merge patch 1/6 and patch 2/6 into patch 1/5, and add
>>>>> modification
>>>>>         for other bus type.
>>>>>      2) add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>>>>         the probelm in patch 2/5.
>>>>>    -v2: resend due to CI unexplained failure.
>>>>>
>>>>> Huisong Li (5):
>>>>>     drivers/bus: restore driver assignment at front of probing
>>>>>     ethdev: fix skip valid port in probing callback
>>>>>     app/testpmd: check the validity of the port
>>>>>     app/testpmd: add attach and detach port for multiple process
>>>>>     app/testpmd: stop forwarding in new or destroy event
>>>>>
>>>> Hi Huisong,
>>>>
>>>> I haven't checked the patch in detail yet, but I can see it gives some
>>>> ABI compatibility warnings, is this expected:
>>> This is to be expected. Because we insert a device state,
>>> RTE_ETH_DEV_ALLOCATED,
>>> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
>>> We may have to announce it. What do you think?
>> If there is an actual ABI break, it can't go in this release, need to
>> wait LTS release and yes needs deprecation notice in advance.
>>
>> But not all enum value change warnings are real break, need to
>> investigate all warnings one by one.
>> Need to investigate if old application & new dpdk library may cause any
>> unexpected behavior for application.
>>
> OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
> enum solves the issue, although logically it won't look nice.
Thank you for your suggestion, Ferruh. It seems to be ok. I will fix it.
> Perhaps order can be fixed in next LTS, to have more logical order, but
> not quite sure if order worth the disturbance may cause in application.
Probably not worth it.
>>>> 1 function with some indirect sub-type change:
>>>>
>>>>     [C] 'function int dpaa_eth_eventq_attach(const rte_eth_dev*, int, u16,
>>>> const rte_event_eth_rx_adapter_queue_conf*)' at dpaa_ethdev.c:1149:1 has
>>>> some indirect sub-type changes:
>>>>       parameter 1 of type 'const rte_eth_dev*' has sub-type changes:
>>>>         in pointed to type 'const rte_eth_dev':
>>>>           in unqualified underlying type 'struct rte_eth_dev' at
>>>> ethdev_driver.h:50:1:
>>>>             type size hasn't changed
>>>>             1 data member change:
>>>>               type of 'rte_eth_dev_state state' changed:
>>>>                 type size hasn't changed
>>>>                 1 enumerator insertion:
>>>>                   'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>>                 2 enumerator changes:
>>>>                   'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>>> to '2' at rte_ethdev.h:2000:1
>>>>                   'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2'
>>>> to '3' at rte_ethdev.h:2000:1
>>>>
>>>> 1 function with some indirect sub-type change:
>>>>
>>>>     [C] 'function int rte_pmd_i40e_set_switch_dev(uint16_t, rte_eth_dev*)'
>>>> at rte_pmd_i40e.c:3266:1 has some indirect sub-type changes:
>>>>       parameter 2 of type 'rte_eth_dev*' has sub-type changes:
>>>>         in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>>>           type size hasn't changed
>>>>           1 data member change:
>>>>             type of 'rte_eth_dev_state state' changed:
>>>>               type size hasn't changed
>>>>               1 enumerator insertion:
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>>               2 enumerator changes:
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>>> to '2' at rte_ethdev.h:2000:1
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>>>> '3' at rte_ethdev.h:2000:1
>>>>
>>>> 1 function with some indirect sub-type change:
>>>>
>>>>     [C] 'function rte_eth_dev* rte_eth_dev_allocate(const char*)' at
>>>> ethdev_driver.c:72:1 has some indirect sub-type changes:
>>>>       return type changed:
>>>>         in pointed to type 'struct rte_eth_dev' at ethdev_driver.h:50:1:
>>>>           type size hasn't changed
>>>>           1 data member change:
>>>>             type of 'rte_eth_dev_state state' changed:
>>>>               type size hasn't changed
>>>>               1 enumerator insertion:
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_ALLOCATED' value '1'
>>>>               2 enumerator changes:
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_ATTACHED' from value '1'
>>>> to '2' at rte_ethdev.h:2000:1
>>>>                 'rte_eth_dev_state::RTE_ETH_DEV_REMOVED' from value '2' to
>>>> '3' at rte_ethdev.h:2000:1
>>>>
>>>> ... there are more warnings for same issue ...
>>>>
>>>> .
> .

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[lihuisong (C)]

在 2023/1/11 20:51, Ferruh Yigit 写道:
> On 12/6/2022 9:26 AM, Huisong Li wrote:
>> The driver assignment was moved back at the end of the device probing
>> because there is no something to use rte_driver during the phase of
>> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
>> to end of probing")
>>
>> However, it is necessary for probing callback to reference rte_driver
>> before probing. For example, probing callback may call some APIs which
>> access the rte_pci_driver::driver by the device::driver pointer to get
>> driver information. In this case, a segment fault will occur in probing
>> callback if there is not this assignment.
>>
> Probing callback gets driver as parameter, so callback function can
> access it via 'drv->driver', is there a specific usecase that
> 'dev->device->driver' needs to be accessed explicitly?
>
> I assume this is related to coming patches that setting up device in
> testpmd event callback, but can you please clarify exact need.
For example, rte_eth_dev_info_get is called in this event callback to get
driver name.
>
>> Further, some comments in code need to be updated if we do that. The
>> driver pointer in rte_device is set before probing and needs to be reset
>> if probing failed. And rte_dev_is_probed can not be called inside probing.
>>
>> Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>> ---
>>   drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
>>   drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
>>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
>>   drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
>>   drivers/bus/pci/pci_common.c             |  9 +++++++--
>>   drivers/bus/vdev/vdev.c                  | 10 ++++++++--
>>   drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
>>   7 files changed, 52 insertions(+), 14 deletions(-)
>>
>> diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
>> index ff1369353a..13cb3fe0f8 100644
>> --- a/drivers/bus/auxiliary/auxiliary_common.c
>> +++ b/drivers/bus/auxiliary/auxiliary_common.c
>> @@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
>>   	}
>>   
>>   	dev->driver = drv;
>> +	/*
>> +	 * Reference rte_driver before probing so as to this pointer can
>> +	 * be used to get driver information in case of segment fault in
>> +	 * probing callback.
>> +	 */
>> +	dev->device.driver = &drv->driver;
>>   
>>   	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
>>   		      drv->driver.name, dev->name, dev->device.numa_node);
>>   	ret = drv->probe(drv, dev);
>>   	if (ret != 0) {
>>   		dev->driver = NULL;
>> +		dev->device.driver = NULL;
>>   		rte_intr_instance_free(dev->intr_handle);
>>   		dev->intr_handle = NULL;
>> -	} else {
>> -		dev->device.driver = &drv->driver;
>>   	}
>>   
>>   	return ret;
>> diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
>> index e57159f5d8..f1b817e58c 100644
>> --- a/drivers/bus/dpaa/dpaa_bus.c
>> +++ b/drivers/bus/dpaa/dpaa_bus.c
>> @@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
>>   			    (dev->device.devargs &&
>>   			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
>>   				continue;
>> -
>> +			/*
>> +			 * Reference rte_driver before probing so as to this
>> +			 * pointer can be used to get driver information in case
>> +			 * of segment fault in probing callback.
>> +			 */
>> +			dev->device.driver = &drv->driver;
>>   			if (probe_all ||
>>   			    (dev->device.devargs &&
>>   			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
>>   				ret = drv->probe(drv, dev);
>>   				if (ret) {
>> +					dev->device.driver = NULL;
>>   					DPAA_BUS_ERR("unable to probe:%s",
>>   						     dev->name);
>>   				} else {
>>   					dev->driver = drv;
>> -					dev->device.driver = &drv->driver;
>>   				}
>>   			}
>>   			break;
>> diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
>> index 57bfb5111a..4bc0c6d3d4 100644
>> --- a/drivers/bus/fslmc/fslmc_bus.c
>> +++ b/drivers/bus/fslmc/fslmc_bus.c
>> @@ -471,15 +471,21 @@ rte_fslmc_probe(void)
>>   				continue;
>>   			}
>>   
>> +			/*
>> +			 * Reference rte_driver before probing so as to this
>> +			 * pointer can be used to get driver information in case
>> +			 * of segment fault in probing callback.
>> +			 */
>> +			dev->device.driver = &drv->driver;
>>   			if (probe_all ||
>>   			   (dev->device.devargs &&
>>   			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
>>   				ret = drv->probe(drv, dev);
>>   				if (ret) {
>> +					dev->device.driver = NULL;
>>   					DPAA2_BUS_ERR("Unable to probe");
>>   				} else {
>>   					dev->driver = drv;
>> -					dev->device.driver = &drv->driver;
>>   				}
>>   			}
>>   			break;
>> diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
>> index bb943b58b5..5f23446f41 100644
>> --- a/drivers/bus/ifpga/ifpga_bus.c
>> +++ b/drivers/bus/ifpga/ifpga_bus.c
>> @@ -293,13 +293,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
>>   
>>   	/* reference driver structure */
>>   	afu_dev->driver = drv;
>> +	/*
>> +	 * Reference rte_driver before probing so as to this pointer can
>> +	 * be used to get driver information in case of segment fault in
>> +	 * probing callback.
>> +	 */
>> +	afu_dev->device.driver = &drv->driver;
>>   
>>   	/* call the driver probe() function */
>>   	ret = drv->probe(afu_dev);
>> -	if (ret)
>> +	if (ret) {
>>   		afu_dev->driver = NULL;
>> -	else
>> -		afu_dev->device.driver = &drv->driver;
>> +		afu_dev->device.driver = NULL;
>> +	}
>>   
>>   	return ret;
>>   }
>> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
>> index bc3a7f39fe..efaa1792e9 100644
>> --- a/drivers/bus/pci/pci_common.c
>> +++ b/drivers/bus/pci/pci_common.c
>> @@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>>   				return ret;
>>   			}
>>   		}
>> +		/*
>> +		 * Reference rte_driver before probing so as to this pointer can
>> +		 * be used to get driver information in case of segment fault in
>> +		 * probing callback.
>> +		 */
>> +		dev->device.driver = &dr->driver;
>>   	}
>>   
>>   	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
>> @@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>>   		return ret; /* no rollback if already succeeded earlier */
>>   	if (ret) {
>>   		dev->driver = NULL;
>> +		dev->device.driver = NULL;
>>   		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
>>   			/* Don't unmap if device is unsupported and
>>   			 * driver needs mapped resources.
>> @@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
>>   		dev->vfio_req_intr_handle = NULL;
>>   		rte_intr_instance_free(dev->intr_handle);
>>   		dev->intr_handle = NULL;
>> -	} else {
>> -		dev->device.driver = &dr->driver;
>>   	}
>>   
>>   	return ret;
>> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
>> index 41bc07dde7..2e3f0f2e12 100644
>> --- a/drivers/bus/vdev/vdev.c
>> +++ b/drivers/bus/vdev/vdev.c
>> @@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
>>   		return -1;
>>   	}
>>   
>> +	/*
>> +	 * Reference rte_driver before probing so as to this pointer can
>> +	 * be used to get driver information in case of segment fault in
>> +	 * probing callback.
>> +	 */
>> +	dev->device.driver = &driver->driver;
>>   	ret = driver->probe(dev);
>> -	if (ret == 0)
>> -		dev->device.driver = &driver->driver;
>> +	if (ret != 0)
>> +		dev->device.driver = NULL;
>>   	return ret;
>>   }
>>   
>> diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
>> index 8d32d66504..feb1651984 100644
>> --- a/drivers/bus/vmbus/vmbus_common.c
>> +++ b/drivers/bus/vmbus/vmbus_common.c
>> @@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
>>   
>>   	/* reference driver structure */
>>   	dev->driver = dr;
>> +	/*
>> +	 * Reference rte_driver before probing so as to this pointer can
>> +	 * be used to get driver information in case of segment fault in
>> +	 * probing callback.
>> +	 */
>> +	dev->device.driver = &dr->driver;
>>   
>>   	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
>>   		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
>> @@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
>>   	ret = dr->probe(dr, dev);
>>   	if (ret) {
>>   		dev->driver = NULL;
>> +		dev->device.driver = NULL;
>>   		rte_vmbus_unmap_device(dev);
>> -	} else {
>> -		dev->device.driver = &dr->driver;
>>   	}
>>   
>>   	return ret;
> .

Re: [PATCH V4 2/5] ethdev: fix skip valid port in probing callback

[lihuisong (C)]

在 2023/1/11 20:51, Ferruh Yigit 写道:
> On 12/6/2022 9:26 AM, Huisong Li wrote:
>> The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
>> iterate over all enabled ports to do something(like, verifying the port id
>> validity) when receive a probing event. If the ethdev state of a port is
>> RTE_ETH_DEV_UNUSED, this port will be skipped.
>>
>> Currently, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
>> event. It means that probing callback will skip this port. But this
>> assignment can not move to front of probing notification. See
>> commit be8cd210379a ("ethdev: fix port probing notification")
>>
> OK to abstract ethdev state check to a function, 'rte_eth_dev_in_used'.
> BTW, I guess intention is to say 'is' instead of 'in', like
> 'rte_eth_dev_is_used'.
Ack
>
> But not sure if new state is necessary.
> I can see from next patches that if testpmd event callback wants to
> validate port_id, it is using 'port_id_is_invalid()' function which is
> using 'RTE_ETH_FOREACH_DEV' macro.
> And using 'RTE_ETH_FOREACH_DEV' macro in callback function has the
> problem you described above.
>
> 1) Do we really need to validate port_id in the event callback, callback
> provided 'port_id' from 'rte_eth_dev_callback_process()' API, from
> 'dev->data->port_id'. I am not sure if there is case that
> 'dev->data->port_id' can be invalid.
>
> 2) is there any other specific usecase to use 'RTE_ETH_FOREACH_DEV' in
> the 'RTE_ETH_EVENT_NEW' event callback function?
Reply 1) and 2): this check should be useful for failsafe.
>
>
>
>> So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
>> state to RTE_ETH_DEV_ALLOCATED before pushing probing event and to
>> RTE_ETH_DEV_ATTACHED when definitely probed. And this port is valid if its
>> device state is ALLOCATED or ATTACHED. Naturally, a lot of places had to be
>> modified for this change.
>>
>> Fixes: be8cd210379a ("ethdev: fix port probing notification")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>> ---
>>   drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
>>   drivers/net/mlx5/mlx5.c        |  2 +-
>>   lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
>>   lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
>>   lib/ethdev/ethdev_pci.h        |  2 +-
>>   lib/ethdev/rte_class_eth.c     |  2 +-
>>   lib/ethdev/rte_ethdev.c        |  4 ++--
>>   lib/ethdev/rte_ethdev.h        |  4 +++-
>>   lib/ethdev/version.map         |  1 +
>>   9 files changed, 33 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
>> index b3de490d36..7b08353ed5 100644
>> --- a/drivers/net/bnxt/bnxt_ethdev.c
>> +++ b/drivers/net/bnxt/bnxt_ethdev.c
>> @@ -6003,7 +6003,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
>>   
>>   	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
>>   
>> -	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
>> +
>> +	if (rte_eth_dev_in_used(eth_dev->state))
>>   		bnxt_dev_close_op(eth_dev);
>>   
>>   	return 0;
>> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
>> index e55be8720e..949d8c1367 100644
>> --- a/drivers/net/mlx5/mlx5.c
>> +++ b/drivers/net/mlx5/mlx5.c
>> @@ -3014,7 +3014,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
>>   	while (port_id < RTE_MAX_ETHPORTS) {
>>   		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
>>   
>> -		if (dev->state != RTE_ETH_DEV_UNUSED &&
>> +		if (rte_eth_dev_in_used(dev->state) &&
>>   		    dev->device &&
>>   		    (dev->device == odev ||
>>   		     (dev->device->driver &&
>> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
>> index 0be1e8ca04..7868958dff 100644
>> --- a/lib/ethdev/ethdev_driver.c
>> +++ b/lib/ethdev/ethdev_driver.c
>> @@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
>>   	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
>>   		/* Using shared name field to find a free port. */
>>   		if (eth_dev_shared_data->data[i].name[0] == '\0') {
>> -			RTE_ASSERT(rte_eth_devices[i].state ==
>> -				   RTE_ETH_DEV_UNUSED);
>> +			RTE_ASSERT(!rte_eth_dev_in_used(
>> +					rte_eth_devices[i].state));
>>   			return i;
>>   		}
>>   	}
>> @@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
>>   	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>>   		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
>>   
>> +	dev->state = RTE_ETH_DEV_ALLOCATED;
>>   	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
>>   
>>   	dev->state = RTE_ETH_DEV_ATTACHED;
>>   }
>>   
>> +bool rte_eth_dev_in_used(uint16_t dev_state)
>> +{
>> +	return dev_state == RTE_ETH_DEV_ALLOCATED ||
>> +		dev_state == RTE_ETH_DEV_ATTACHED;
>> +}
>> +
>>   int
>>   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>>   {
>> @@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>>   
>>   	eth_dev_shared_data_prepare();
>>   
>> -	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
>> +	if (rte_eth_dev_in_used(eth_dev->state))
>>   		rte_eth_dev_callback_process(eth_dev,
>>   				RTE_ETH_EVENT_DESTROY, NULL);
>>   
>> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
>> index 6a550cfc83..e04f88fe46 100644
>> --- a/lib/ethdev/ethdev_driver.h
>> +++ b/lib/ethdev/ethdev_driver.h
>> @@ -1542,6 +1542,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
>>   __rte_internal
>>   void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
>>   
>> +/**
>> + * Check if a Ethernet device state is in used or not
>> + *
>> + * @param dev_state
>> + *   The state of the Ethernet device
>> + * @return
>> + *   - true if the state of the Ethernet device is allocated or attached
>> + *   - false if this state is neither allocated nor attached
>> + */
>> +__rte_internal
>> +bool rte_eth_dev_in_used(uint16_t dev_state);
>> +
>>   /**
>>    * Create memzone for HW rings.
>>    * malloc can't be used as the physical address is needed.
>> diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
>> index 94b8fba5d7..362e5c2dec 100644
>> --- a/lib/ethdev/ethdev_pci.h
>> +++ b/lib/ethdev/ethdev_pci.h
>> @@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
>>   	 * eth device has been released.
>>   	 */
>>   	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
>> -	    eth_dev->state == RTE_ETH_DEV_UNUSED)
>> +	    !rte_eth_dev_in_used(eth_dev->state))
>>   		return 0;
>>   
>>   	if (dev_uninit) {
>> diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
>> index 838b3a8f9f..a78d503e59 100644
>> --- a/lib/ethdev/rte_class_eth.c
>> +++ b/lib/ethdev/rte_class_eth.c
>> @@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
>>   	const struct rte_kvargs *kvlist = arg->kvlist;
>>   	unsigned int pair;
>>   
>> -	if (edev->state == RTE_ETH_DEV_UNUSED)
>> +	if (!rte_eth_dev_in_used(edev->state))
>>   		return -1;
>>   	if (arg->device != NULL && arg->device != edev->device)
>>   		return -1;
>> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
>> index 5d5e18db1e..a225141937 100644
>> --- a/lib/ethdev/rte_ethdev.c
>> +++ b/lib/ethdev/rte_ethdev.c
>> @@ -325,7 +325,7 @@ uint16_t
>>   rte_eth_find_next(uint16_t port_id)
>>   {
>>   	while (port_id < RTE_MAX_ETHPORTS &&
>> -			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
>> +	       !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
>>   		port_id++;
>>   
>>   	if (port_id >= RTE_MAX_ETHPORTS)
>> @@ -372,7 +372,7 @@ int
>>   rte_eth_dev_is_valid_port(uint16_t port_id)
>>   {
>>   	if (port_id >= RTE_MAX_ETHPORTS ||
>> -	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
>> +	    !rte_eth_dev_in_used(rte_eth_devices[port_id].state))
>>   		return 0;
>>   	else
>>   		return 1;
>> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
>> index c129ca1eaf..459ad44021 100644
>> --- a/lib/ethdev/rte_ethdev.h
>> +++ b/lib/ethdev/rte_ethdev.h
>> @@ -2000,7 +2000,9 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
>>   enum rte_eth_dev_state {
>>   	/** Device is unused before being probed. */
>>   	RTE_ETH_DEV_UNUSED = 0,
>> -	/** Device is attached when allocated in probing. */
>> +	/** Device is allocated and is set before reporting new event. */
>> +	RTE_ETH_DEV_ALLOCATED,
>> +	/** Device is attached when definitely probed. */
>>   	RTE_ETH_DEV_ATTACHED,
>>   	/** Device is in removed state when plug-out is detected. */
>>   	RTE_ETH_DEV_REMOVED,
>> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
>> index 17201fbe0f..66d70f1aac 100644
>> --- a/lib/ethdev/version.map
>> +++ b/lib/ethdev/version.map
>> @@ -327,4 +327,5 @@ INTERNAL {
>>   	rte_eth_representor_id_get;
>>   	rte_eth_switch_domain_alloc;
>>   	rte_eth_switch_domain_free;
>> +	rte_eth_dev_in_used;
>>   };
> .

Re: [PATCH V4 4/5] app/testpmd: add attach and detach port for multiple process

[lihuisong (C)]

在 2023/1/11 20:51, Ferruh Yigit 写道:
> On 12/6/2022 9:26 AM, Huisong Li wrote:
>> This patch supports attach and detach port in primary and secondary
>> process.
>>
> Hi Huisong,
>
> This patch moves port setup and remove (via alarm callback) to event
> callback,
>
> 1) I can see it is for MP but can you please give more details, what was
> the problem before, how this solves the issue
The port information in testpmd needs to be updated due to attatching 
and detaching
port, which are done in the same thread as removing or probing device.
If testpmd don't support multiple process, it is ok. But it can not work 
again
for multiple process. For example, if primary process does attatch or 
detach a port,
primary and secondary process can receive 'new' or 'destroy' event. And 
primary can
update its port information, but secondary will never update its port 
information.
So we have to move updating port information to event callback.

The reason for adding an alarm callback is that the ethdev state is set 
from 'ATTACHED'
to 'UNUSED' only after the event callback finished. The 
remove_invalid_ports() function
removes invalid port according to ethdev state. If we don't add alarm 
callback, this
detached port information can not be removed.
>
> 2) I am concerned about doing more work on the event callback and
> observe some unexpected side effects later, can't we handle this out of
> event callback
There's no way, if you don't, the port information in testpmd is all 
messed up. That's very bad.
>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
>> ---
>>   app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
>>   app/test-pmd/testpmd.h                |  1 -
>>   drivers/net/bonding/bonding_testpmd.c |  1 -
>>   3 files changed, 22 insertions(+), 18 deletions(-)
>>
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index bc25703490..2e6329c853 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -3463,15 +3463,12 @@ attach_port(char *identifier)
>>   		return;
>>   	}
>>   
>> -	/* first attach mode: event */
>> -	if (setup_on_probe_event) {
>> -		/* new ports are detected on RTE_ETH_EVENT_NEW event */
>> -		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
>> -			if (ports[pi].port_status == RTE_PORT_HANDLING &&
>> -					ports[pi].need_setup != 0)
>> -				setup_attached_port(pi);
>> +	/*
>> +	 * first attach mode: event, setting up attached port is done in
>> +	 * probing callback.
>> +	 */
>> +	if (setup_on_probe_event)
>>   		return;
>> -	}
>>   
>>   	/* second attach mode: iterator */
>>   	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
>> @@ -3502,7 +3499,6 @@ setup_attached_port(portid_t pi)
>>   	ports_ids[nb_ports++] = pi;
>>   	fwd_ports_ids[nb_fwd_ports++] = pi;
>>   	nb_cfg_ports = nb_fwd_ports;
>> -	ports[pi].need_setup = 0;
>>   	ports[pi].port_status = RTE_PORT_STOPPED;
>>   
>>   	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
>> @@ -3536,10 +3532,8 @@ detach_device(struct rte_device *dev)
>>   		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
>>   		return;
>>   	}
>> -	remove_invalid_ports();
>>   
>>   	printf("Device is detached\n");
>> -	printf("Now total ports is %d\n", nb_ports);
>>   	printf("Done\n");
>>   	return;
>>   }
>> @@ -3606,11 +3600,9 @@ detach_devargs(char *identifier)
>>   		return;
>>   	}
>>   
>> -	remove_invalid_ports();
>> -
>>   	printf("Device %s is detached\n", identifier);
>> -	printf("Now total ports is %d\n", nb_ports);
>>   	printf("Done\n");
>> +
>>   	rte_devargs_reset(&da);
>>   }
>>   
>> @@ -3774,11 +3766,22 @@ rmv_port_callback(void *arg)
>>   		struct rte_device *device = dev_info.device;
>>   		close_port(port_id);
>>   		detach_device(device); /* might be already removed or have more ports */
>> +		remove_invalid_ports();
>> +		printf("Now total ports is %d\n", nb_ports);
>>   	}
>>   	if (need_to_start)
>>   		start_packet_forwarding(0);
>>   }
>>   
>> +static void
>> +remove_invalid_ports_callback(void *arg)
>> +{
>> +	RTE_SET_USED(arg);
>> +
>> +	remove_invalid_ports();
>> +	printf("Now total ports is %d\n", nb_ports);
>> +}
>> +
>>   /* This function is used by the interrupt thread */
>>   static int
>>   eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>> @@ -3803,8 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>>   
>>   	switch (type) {
>>   	case RTE_ETH_EVENT_NEW:
>> -		ports[port_id].need_setup = 1;
>> -		ports[port_id].port_status = RTE_PORT_HANDLING;
>> +		if (setup_on_probe_event)
>> +			setup_attached_port(port_id);
>>   		break;
>>   	case RTE_ETH_EVENT_INTR_RMV:
>>   		if (rte_eal_alarm_set(100000,
>> @@ -3815,6 +3818,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>>   	case RTE_ETH_EVENT_DESTROY:
>>   		ports[port_id].port_status = RTE_PORT_CLOSED;
>>   		printf("Port %u is closed\n", port_id);
>> +		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
>> +			(void *)(intptr_t)port_id))
>> +			fprintf(stderr, "Could not set up deferred device released\n");
>>   		break;
>>   	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
>>   		uint16_t rxq_id;
>> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
>> index 7d24d25970..080d3a1139 100644
>> --- a/app/test-pmd/testpmd.h
>> +++ b/app/test-pmd/testpmd.h
>> @@ -306,7 +306,6 @@ struct rte_port {
>>   	uint16_t                tx_vlan_id;/**< The tag ID */
>>   	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
>>   	volatile uint16_t        port_status;    /**< port started or not */
>> -	uint8_t                 need_setup;     /**< port just attached */
>>   	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
>>   	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
>>   	uint8_t                 rss_flag;   /**< enable rss or not */
>> diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
>> index 9529e16fb6..9216271314 100644
>> --- a/drivers/net/bonding/bonding_testpmd.c
>> +++ b/drivers/net/bonding/bonding_testpmd.c
>> @@ -765,7 +765,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
>>   
>>   	ports[port_id].update_conf = 1;
>>   	ports[port_id].bond_flag = 1;
>> -	ports[port_id].need_setup = 0;
>>   	ports[port_id].port_status = RTE_PORT_STOPPED;
>>   }
>>   
> .

Re: [PATCH V4 5/5] app/testpmd: stop forwarding in new or destroy event

[lihuisong (C)]

在 2023/1/11 20:52, Ferruh Yigit 写道:
> On 12/6/2022 9:26 AM, Huisong Li wrote:
>> When testpmd receives the new or destroy event, the port related
>> information will be updated. Testpmd must stop packet forwarding
>> before updating the information to avoid some serious problems.
>>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>> ---
>>   app/test-pmd/testpmd.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index 2e6329c853..746f07652a 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -3806,6 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>>   
>>   	switch (type) {
>>   	case RTE_ETH_EVENT_NEW:
>> +		if (test_done == 0)
>> +			stop_packet_forwarding();
> testpmd is test application, why not prevent user to issue attach /
> detach commands when packet forwarding is going on, and force user to
> stop forwarding explicitly instead of doing this implicitly and silently?
For primary process, maybe it is ok by forcing user to stop forwarding 
explicitly.
But for multiple process, it is better to do this before removing or adding.
otherwise there'll be a lot of weird prints.
>
> Similar to previous comments, as we make things more complex for
> specific use cases it will be very difficult to update testpmd without
> hitting unexpected side effects everywhere, at least this is my concern.
Understand your concern. But it's a problem.
>
>>   		if (setup_on_probe_event)
>>   			setup_attached_port(port_id);
>>   		break;
>> @@ -3816,6 +3818,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
>>   				"Could not set up deferred device removal\n");
>>   		break;
>>   	case RTE_ETH_EVENT_DESTROY:
>> +		if (test_done == 0)
>> +			stop_packet_forwarding();
>>   		ports[port_id].port_status = RTE_PORT_CLOSED;
>>   		printf("Port %u is closed\n", port_id);
>>   		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
> .

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Thomas Monjalon]

11/01/2023 11:46, Ferruh Yigit:
> On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
> > On 1/11/2023 12:53 AM, lihuisong (C) wrote:
> >> 在 2023/1/11 0:51, Ferruh Yigit 写道:
> >>> Hi Huisong,
> >>>
> >>> I haven't checked the patch in detail yet, but I can see it gives some
> >>> ABI compatibility warnings, is this expected:
> >> This is to be expected. Because we insert a device state,
> >> RTE_ETH_DEV_ALLOCATED,
> >> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
> >> We may have to announce it. What do you think?
> > 
> > If there is an actual ABI break, it can't go in this release, need to
> > wait LTS release and yes needs deprecation notice in advance.
> > 
> > But not all enum value change warnings are real break, need to
> > investigate all warnings one by one.
> > Need to investigate if old application & new dpdk library may cause any
> > unexpected behavior for application.
> > 
> 
> OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
> enum solves the issue, although logically it won't look nice.
> Perhaps order can be fixed in next LTS, to have more logical order, but
> not quite sure if order worth the disturbance may cause in application.

It is a state with a logical order, so it would be nice to be able to do
if (state > RTE_ETH_DEV_ALLOCATED)
but given there is RTE_ETH_DEV_REMOVED later in the enum, not sure it is useful.

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[lihuisong (C)]

在 2023/1/18 22:12, Thomas Monjalon 写道:
> 11/01/2023 11:46, Ferruh Yigit:
>> On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
>>> On 1/11/2023 12:53 AM, lihuisong (C) wrote:
>>>> 在 2023/1/11 0:51, Ferruh Yigit 写道:
>>>>> Hi Huisong,
>>>>>
>>>>> I haven't checked the patch in detail yet, but I can see it gives some
>>>>> ABI compatibility warnings, is this expected:
>>>> This is to be expected. Because we insert a device state,
>>>> RTE_ETH_DEV_ALLOCATED,
>>>> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
>>>> We may have to announce it. What do you think?
>>> If there is an actual ABI break, it can't go in this release, need to
>>> wait LTS release and yes needs deprecation notice in advance.
>>>
>>> But not all enum value change warnings are real break, need to
>>> investigate all warnings one by one.
>>> Need to investigate if old application & new dpdk library may cause any
>>> unexpected behavior for application.
>>>
>> OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
>> enum solves the issue, although logically it won't look nice.
>> Perhaps order can be fixed in next LTS, to have more logical order, but
>> not quite sure if order worth the disturbance may cause in application.
> It is a state with a logical order, so it would be nice to be able to do
> if (state > RTE_ETH_DEV_ALLOCATED)
> but given there is RTE_ETH_DEV_REMOVED later in the enum, not sure it is useful.
The device state is internel. Applications should not access it 
directly, right?
Currently, ethdev layer or PMD use it by enum value instead of the way like
'state > RTE_ETH_DEV_ALLOCATED'.

But, I encapsulated an API, rte_eth_dev_is_used(), for ethdev or PMD to 
call.
I'm not sure if it can help to eliminate our concerns.
>
>
> .

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[Thomas Monjalon]

19/01/2023 11:31, lihuisong (C):
> 在 2023/1/18 22:12, Thomas Monjalon 写道:
> > 11/01/2023 11:46, Ferruh Yigit:
> >> On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
> >>> On 1/11/2023 12:53 AM, lihuisong (C) wrote:
> >>>> 在 2023/1/11 0:51, Ferruh Yigit 写道:
> >>>>> Hi Huisong,
> >>>>>
> >>>>> I haven't checked the patch in detail yet, but I can see it gives some
> >>>>> ABI compatibility warnings, is this expected:
> >>>> This is to be expected. Because we insert a device state,
> >>>> RTE_ETH_DEV_ALLOCATED,
> >>>> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
> >>>> We may have to announce it. What do you think?
> >>> If there is an actual ABI break, it can't go in this release, need to
> >>> wait LTS release and yes needs deprecation notice in advance.
> >>>
> >>> But not all enum value change warnings are real break, need to
> >>> investigate all warnings one by one.
> >>> Need to investigate if old application & new dpdk library may cause any
> >>> unexpected behavior for application.
> >>>
> >> OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
> >> enum solves the issue, although logically it won't look nice.
> >> Perhaps order can be fixed in next LTS, to have more logical order, but
> >> not quite sure if order worth the disturbance may cause in application.
> > It is a state with a logical order, so it would be nice to be able to do
> > if (state > RTE_ETH_DEV_ALLOCATED)
> > but given there is RTE_ETH_DEV_REMOVED later in the enum, not sure it is useful.
> 
> The device state is internel. Applications should not access it 
> directly, right?

Right

> Currently, ethdev layer or PMD use it by enum value instead of the way like
> 'state > RTE_ETH_DEV_ALLOCATED'.

Right

> But, I encapsulated an API, rte_eth_dev_is_used(), for ethdev or PMD to 
> call.
> I'm not sure if it can help to eliminate our concerns.

Yes I think it's OK.

Re: [PATCH V4 0/5] app/testpmd: support mulitple process attach and detach port

[lihuisong (C)]

在 2023/1/19 22:35, Thomas Monjalon 写道:
> 19/01/2023 11:31, lihuisong (C):
>> 在 2023/1/18 22:12, Thomas Monjalon 写道:
>>> 11/01/2023 11:46, Ferruh Yigit:
>>>> On 1/11/2023 10:27 AM, Ferruh Yigit wrote:
>>>>> On 1/11/2023 12:53 AM, lihuisong (C) wrote:
>>>>>> 在 2023/1/11 0:51, Ferruh Yigit 写道:
>>>>>>> Hi Huisong,
>>>>>>>
>>>>>>> I haven't checked the patch in detail yet, but I can see it gives some
>>>>>>> ABI compatibility warnings, is this expected:
>>>>>> This is to be expected. Because we insert a device state,
>>>>>> RTE_ETH_DEV_ALLOCATED,
>>>>>> before RTE_ETH_DEV_ATTACHED for resolving the issue patch 2/5 mentioned.
>>>>>> We may have to announce it. What do you think?
>>>>> If there is an actual ABI break, it can't go in this release, need to
>>>>> wait LTS release and yes needs deprecation notice in advance.
>>>>>
>>>>> But not all enum value change warnings are real break, need to
>>>>> investigate all warnings one by one.
>>>>> Need to investigate if old application & new dpdk library may cause any
>>>>> unexpected behavior for application.
>>>>>
>>>> OR, appending new enum item, `RTE_ETH_DEV_ALLOCATED`, to the end of the
>>>> enum solves the issue, although logically it won't look nice.
>>>> Perhaps order can be fixed in next LTS, to have more logical order, but
>>>> not quite sure if order worth the disturbance may cause in application.
>>> It is a state with a logical order, so it would be nice to be able to do
>>> if (state > RTE_ETH_DEV_ALLOCATED)
>>> but given there is RTE_ETH_DEV_REMOVED later in the enum, not sure it is useful.
>> The device state is internel. Applications should not access it
>> directly, right?
> Right
>
>> Currently, ethdev layer or PMD use it by enum value instead of the way like
>> 'state > RTE_ETH_DEV_ALLOCATED'.
> Right
>
>> But, I encapsulated an API, rte_eth_dev_is_used(), for ethdev or PMD to
>> call.
>> I'm not sure if it can help to eliminate our concerns.
> Yes I think it's OK.
ok, I will fix it based on our discussion.
>
>
> .

[PATCH V5 0/5] app/testpmd: support multiple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
 -v4: fix a misspelling. 
 -v3:
   #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
      for other bus type.
   #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 
 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.22.0

[PATCH V5 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff1369353a..13cb3fe0f8 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index bb943b58b5..5f23446f41 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -293,13 +293,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index bc3a7f39fe..efaa1792e9 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 41bc07dde7..2e3f0f2e12 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 8d32d66504..feb1651984 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH V5 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.

However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and set it to
RTE_ETH_DEV_ATTACHED after definitely probed. And this port is valid if its
device state is 'ALLOCATED' or 'ATTACHED'.

In addition, the new state has to be placed behind 'REMOVED' to avoid ABI
break. Fortunately, this ethdev state is internal and applications can not
access it directly. So this patch encapsulates an API, rte_eth_dev_is_used,
for ethdev or PMD to call and eliminate concerns about using this state
enum value comparison.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index b3de490d36..be5c93010e 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6003,7 +6003,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_is_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e55be8720e..7f4dafaa22 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3014,7 +3014,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_is_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index 0be1e8ca04..29e9417bea 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_is_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_is_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	eth_dev_shared_data_prepare();
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_is_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 6a550cfc83..dde3ec84ef 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1542,6 +1542,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_is_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 94b8fba5d7..23270ccd73 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_is_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index 838b3a8f9f..504bfd99c9 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_is_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 5d5e18db1e..86ca303ab5 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -325,7 +325,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -372,7 +372,7 @@ int
 rte_eth_dev_is_valid_port(uint16_t port_id)
 {
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		return 0;
 	else
 		return 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index c129ca1eaf..d22de196db 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2000,10 +2000,12 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
 };
 
 struct rte_eth_dev_sriov {
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index 17201fbe0f..094c2a952e 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -327,4 +327,5 @@ INTERNAL {
 	rte_eth_representor_id_get;
 	rte_eth_switch_domain_alloc;
 	rte_eth_switch_domain_free;
+	rte_eth_dev_is_used;
 };
-- 
2.22.0

[PATCH V5 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index e366f81a0f..ee2383308d 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3798,14 +3798,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH V5 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

The port information needs to be updated due to attaching and detaching
port. Currently, it is done in the same thread as removing or probing
device, which doesn't satisfy the operation of attaching and detaching
device in multiple process.

If this operation is performed in one process, the other process can
receive 'new' or 'destroy' event. So we can move updating port information
to event callback to support attaching and detaching port in primary and
secondary process.

The reason for adding an alarm callback in 'destroy' event is that the
ethdev state is changed from 'ATTACHED' to 'UNUSED' only after the event
callback finished. But the remove_invalid_ports() function removes invalid
port only if ethdev state is 'UNUSED'. If we don't add alarm callback, this
detached port information can not be removed.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index ee2383308d..6210982f77 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3463,15 +3463,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3502,7 +3499,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3536,10 +3532,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3606,11 +3600,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3774,11 +3766,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3803,8 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3815,6 +3818,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 7d24d25970..080d3a1139 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -306,7 +306,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index b3c12cada0..160f638e3a 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -493,7 +493,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH V5 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 6210982f77..92a185695f 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3806,6 +3806,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3816,6 +3818,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[Ferruh Yigit]

On 1/12/2023 2:44 AM, lihuisong (C) wrote:
> 
> 在 2023/1/11 20:51, Ferruh Yigit 写道:
>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>> The driver assignment was moved back at the end of the device probing
>>> because there is no something to use rte_driver during the phase of
>>> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
>>> to end of probing")
>>>
>>> However, it is necessary for probing callback to reference rte_driver
>>> before probing. For example, probing callback may call some APIs which
>>> access the rte_pci_driver::driver by the device::driver pointer to get
>>> driver information. In this case, a segment fault will occur in probing
>>> callback if there is not this assignment.
>>>
>> Probing callback gets driver as parameter, so callback function can
>> access it via 'drv->driver', is there a specific usecase that
>> 'dev->device->driver' needs to be accessed explicitly?
>>
>> I assume this is related to coming patches that setting up device in
>> testpmd event callback, but can you please clarify exact need.
>
> For example, rte_eth_dev_info_get is called in this event callback to get
> driver name.
>

Why 'rte_eth_dev_info_get()' is called in the event called at first place?


This set updates multiple things to extend 'RTE_ETH_EVENT_NEW' event
callback function support, like:

- Assign device driver *before* probing completed, so that even callback
can run 'rte_eth_dev_info_get()'

- Add a new ethdev state so that port can be recognized as valid port in
the even callback.

- Stop forwarding implicitly in even callback in case event callback run
while forwarding is on.


All looks to me hack/complexity to make a specific case work, which is
make secondary *testmp* application work with attached/detached device.

And finally patch 4/5 adds port setup to testpmd event callback for this.


I understand the intention, but I disagree with bus and ethdev level
changes for this.



Event callback may not be only way to share port attach/detach
information between primary and secondary, there is a MP socket and
'rte_mp_handle' thread to handle communication between primary and
secondary process, this should be able to use carrying device
information, as far as I remember this is why it is introduced at first
place.

Did you consider using MP socket for your use case?



Following is a sample usage:

Primary:
started as:
sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 0-1
--log-level=*:debug -- -i --num-procs=2 --proc-id=0

``
testpmd> show port summary all
Number of available ports: 0
Port MAC Address       Name         Driver         Status   Link

testpmd> port attach net_null0
Attaching a new port...
dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
vdev_probe_all_drivers(): Search driver to probe device net_null0
rte_pmd_null_probe(): Initializing pmd_null for net_null0
rte_pmd_null_probe(): Configure pmd_null: packet size is 64, packet copy
is disabled
eth_dev_null_create(): Creating null ethdev on numa socket 0
EAL: request: eal_dev_mp_request
EAL: msg: bus_vdev_mp
vdev_action(): send vdev, net_null0
EAL: sendmsg: bus_vdev_mp
EAL: reply: bus_vdev_mp
EAL: msg: eal_dev_mp_request
Port 0 is attached. Now total ports is 1
Done

testpmd> show port summary all
Number of available ports: 1
Port MAC Address       Name         Driver         Status   Link
0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps

testpmd> port detach 0
Port was not closed
Removing a device...
EAL: request: eal_dev_mp_request
EAL: msg: eal_dev_mp_request
eth_dev_close(): Closing null ethdev on NUMA socket 0
Port 0 is closed
Device is detached
Now total ports is 0
Done

testpmd> show port summary all
Number of available ports: 0
Port MAC Address       Name         Driver         Status   Link
testpmd>

``

Secondary:
started as:
sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 2-3
--log-level=*:debug -- -i --num-procs=2 --proc-id=1

``
testpmd> show port summary all
Number of available ports: 0
Port MAC Address       Name         Driver         Status   Link

testpmd> EAL: msg: eal_dev_mp_request
dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
EAL: request: bus_vdev_mp
EAL: msg: bus_vdev_mp
vdev_action(): receive vdev, net_null0
EAL: msg: bus_vdev_mp
vdev_scan(): Received 1 vdevs
vdev_probe_all_drivers(): Search driver to probe device net_null0
rte_pmd_null_probe(): Initializing pmd_null for net_null0
EAL: reply: eal_dev_mp_request

testpmd> show port summary all
Number of available ports: 1
Port MAC Address       Name         Driver         Status   Link
0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps

testpmd> EAL: msg: eal_dev_mp_request
dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
eth_dev_close(): Closing null ethdev on NUMA socket 4294967295
Port 0 is closed
EAL: reply: eal_dev_mp_request

testpmd> show port summary all
Number of available ports: 0
Port MAC Address       Name         Driver         Status   Link
testpmd>
``

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[lihuisong (C)]

[-- Attachment #1: Type: text/plain, Size: 6868 bytes --]


在 2023/2/16 0:09, Ferruh Yigit 写道:
> On 1/12/2023 2:44 AM, lihuisong (C) wrote:
>> 在 2023/1/11 20:51, Ferruh Yigit 写道:
>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>> The driver assignment was moved back at the end of the device probing
>>>> because there is no something to use rte_driver during the phase of
>>>> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
>>>> to end of probing")
>>>>
>>>> However, it is necessary for probing callback to reference rte_driver
>>>> before probing. For example, probing callback may call some APIs which
>>>> access the rte_pci_driver::driver by the device::driver pointer to get
>>>> driver information. In this case, a segment fault will occur in probing
>>>> callback if there is not this assignment.
>>>>
>>> Probing callback gets driver as parameter, so callback function can
>>> access it via 'drv->driver', is there a specific usecase that
>>> 'dev->device->driver' needs to be accessed explicitly?
>>>
>>> I assume this is related to coming patches that setting up device in
>>> testpmd event callback, but can you please clarify exact need.
>> For example, rte_eth_dev_info_get is called in this event callback to get
>> driver name.
>>
Hi Ferruh,

Sorry for the delay. I missed this email.
Thanks for your review.
> Why 'rte_eth_dev_info_get()' is called in the event called at first place?
There is no limitation on rte_eth_dev_info_get() in event callback.
The upper layer is entirely possible to use.
After all, it is more convenient for app to use this pointer to get 
driver information in a probing callback.
>
> This set updates multiple things to extend 'RTE_ETH_EVENT_NEW' event
> callback function support, like:
>
> - Assign device driver *before* probing completed, so that even callback
> can run 'rte_eth_dev_info_get()'
>
> - Add a new ethdev state so that port can be recognized as valid port in
> the even callback.
Yes
>
> - Stop forwarding implicitly in even callback in case event callback run
> while forwarding is on.
But this is a modification for tesptmd to make it work well.
>
>
> All looks to me hack/complexity to make a specific case work, which is
> make secondary *testmp* application work with attached/detached device.
It is not just a testpmd application case, but, I think, still exists in 
actual case.
Because eal lib supports hotplugging device on primary and secondary 
process and the communication each other when attach or detach device.
The reason why no one has ever put forward this question, I think it may 
be attributed to the fact that the scene is not or rarely tested.
>
> And finally patch 4/5 adds port setup to testpmd event callback for this.
>
>
> I understand the intention, but I disagree with bus and ethdev level
> changes for this.
I'm just raising this issue, and we can discuss how to deal with it 
together.😁
>
>
>
> Event callback may not be only way to share port attach/detach
> information between primary and secondary, there is a MP socket and
> 'rte_mp_handle' thread to handle communication between primary and
> secondary process, this should be able to use carrying device
> information, as far as I remember this is why it is introduced at first
> place.
>
> Did you consider using MP socket for your use case?
Actually, the probing event callback called in patch 4/5 is the result 
of the MP socket communication (please see hogplug_mp.c).
>
>
>
> Following is a sample usage:
>
> Primary:
> started as:
> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 0-1
> --log-level=*:debug -- -i --num-procs=2 --proc-id=0
>
> ``
> testpmd> show port summary all
> Number of available ports: 0
> Port MAC Address       Name         Driver         Status   Link
>
> testpmd> port attach net_null0
> Attaching a new port...
> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
> vdev_probe_all_drivers(): Search driver to probe device net_null0
> rte_pmd_null_probe(): Initializing pmd_null for net_null0
> rte_pmd_null_probe(): Configure pmd_null: packet size is 64, packet copy
> is disabled
> eth_dev_null_create(): Creating null ethdev on numa socket 0
> EAL: request: eal_dev_mp_request
> EAL: msg: bus_vdev_mp
> vdev_action(): send vdev, net_null0
> EAL: sendmsg: bus_vdev_mp
> EAL: reply: bus_vdev_mp
> EAL: msg: eal_dev_mp_request
> Port 0 is attached. Now total ports is 1
> Done
>
> testpmd> show port summary all
> Number of available ports: 1
> Port MAC Address       Name         Driver         Status   Link
> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>
> testpmd> port detach 0
> Port was not closed
> Removing a device...
> EAL: request: eal_dev_mp_request
> EAL: msg: eal_dev_mp_request
> eth_dev_close(): Closing null ethdev on NUMA socket 0
> Port 0 is closed
> Device is detached
> Now total ports is 0
Please note the log *"Now total ports is 0"*,
which indicates the port number is updated if we detached device in this 
process.
> Done
>
> testpmd> show port summary all
> Number of available ports: 0
> Port MAC Address       Name         Driver         Status   Link
> testpmd>
>
> ``
>
> Secondary:
> started as:
> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 2-3
> --log-level=*:debug -- -i --num-procs=2 --proc-id=1
>
> ``
> testpmd> show port summary all
> Number of available ports: 0
> Port MAC Address       Name         Driver         Status   Link
>
> testpmd> EAL: msg: eal_dev_mp_request
> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
> EAL: request: bus_vdev_mp
> EAL: msg: bus_vdev_mp
> vdev_action(): receive vdev, net_null0
> EAL: msg: bus_vdev_mp
> vdev_scan(): Received 1 vdevs
> vdev_probe_all_drivers(): Search driver to probe device net_null0
> rte_pmd_null_probe(): Initializing pmd_null for net_null0
> EAL: reply: eal_dev_mp_request
>
> testpmd> show port summary all
> Number of available ports: 1
> Port MAC Address       Name         Driver         Status   Link
> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>
> testpmd> EAL: msg: eal_dev_mp_request
> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
> eth_dev_close(): Closing null ethdev on NUMA socket 4294967295
> Port 0 is closed
> EAL: reply: eal_dev_mp_request
But the port number in this process does not be updated after finishing 
the destroy event.
That's the problem.
> testpmd> show port summary all
> Number of available ports: 0
> Port MAC Address       Name         Driver         Status   Link
> testpmd>
> ``
>
>
> .

[-- Attachment #2: Type: text/html, Size: 8946 bytes --]

Re: [PATCH V5 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh and Thomas,

Can you continue to take a look at this series?
This work has been working on since August last year.

/Huisong


在 2023/1/31 11:33, Huisong Li 写道:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
>
> ---
>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>   -v4: fix a misspelling.
>   -v3:
>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>        for other bus type.
>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>        the probelm in patch 2/5.
>   -v2: resend due to CI unexplained failure.
>
> Huisong Li (5):
>    drivers/bus: restore driver assignment at front of probing
>    ethdev: fix skip valid port in probing callback
>    app/testpmd: check the validity of the port
>    app/testpmd: add attach and detach port for multiple process
>    app/testpmd: stop forwarding in new or destroy event
>
>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>   app/test-pmd/testpmd.h                   |  1 -
>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>   drivers/bus/pci/pci_common.c             |  9 ++++-
>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>   drivers/net/mlx5/mlx5.c                  |  2 +-
>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>   lib/ethdev/ethdev_pci.h                  |  2 +-
>   lib/ethdev/rte_class_eth.c               |  2 +-
>   lib/ethdev/rte_ethdev.c                  |  4 +-
>   lib/ethdev/rte_ethdev.h                  |  4 +-
>   lib/ethdev/version.map                   |  1 +
>   19 files changed, 114 insertions(+), 44 deletions(-)
>

Re: [PATCH V5 2/5] ethdev: fix skip valid port in probing callback

[fengchengwen]

On 2023/1/31 11:33, Huisong Li wrote:
> The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
> iterate over all enabled ports to do something(like, verifying the port id
> validity) when receive a probing event. If the ethdev state of a port is
> not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.
> 
> However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
> event. It means that probing callback will skip this port. But this
> assignment can not move to front of probing notification. See
> commit be8cd210379a ("ethdev: fix port probing notification")

...

>  
>  struct rte_eth_dev_sriov {
> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
> index 17201fbe0f..094c2a952e 100644
> --- a/lib/ethdev/version.map
> +++ b/lib/ethdev/version.map
> @@ -327,4 +327,5 @@ INTERNAL {
>  	rte_eth_representor_id_get;
>  	rte_eth_switch_domain_alloc;
>  	rte_eth_switch_domain_free;
> +	rte_eth_dev_is_used;

requires alphabetical order.

>  };
> 

Re: [PATCH V5 0/5] app/testpmd: support multiple process attach and detach port

[fengchengwen]

with 2/5 fixed,
Series-acked-by: Chengwen Feng <fengchengwen@huawei.com>

On 2023/1/31 11:33, Huisong Li wrote:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
> 
> ---
>  -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>  -v4: fix a misspelling. 
>  -v3:
>    #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>       for other bus type.
>    #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>       the probelm in patch 2/5. 
>  -v2: resend due to CI unexplained failure.
> 
> Huisong Li (5):
>   drivers/bus: restore driver assignment at front of probing
>   ethdev: fix skip valid port in probing callback
>   app/testpmd: check the validity of the port
>   app/testpmd: add attach and detach port for multiple process
>   app/testpmd: stop forwarding in new or destroy event
> 
>  app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>  app/test-pmd/testpmd.h                   |  1 -
>  drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>  drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>  drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>  drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>  drivers/bus/pci/pci_common.c             |  9 ++++-
>  drivers/bus/vdev/vdev.c                  | 10 ++++-
>  drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>  drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>  drivers/net/bonding/bonding_testpmd.c    |  1 -
>  drivers/net/mlx5/mlx5.c                  |  2 +-
>  lib/ethdev/ethdev_driver.c               | 13 +++++--
>  lib/ethdev/ethdev_driver.h               | 12 ++++++
>  lib/ethdev/ethdev_pci.h                  |  2 +-
>  lib/ethdev/rte_class_eth.c               |  2 +-
>  lib/ethdev/rte_ethdev.c                  |  4 +-
>  lib/ethdev/rte_ethdev.h                  |  4 +-
>  lib/ethdev/version.map                   |  1 +
>  19 files changed, 114 insertions(+), 44 deletions(-)
> 

Re: [PATCH V5 2/5] ethdev: fix skip valid port in probing callback

[lihuisong (C)]

在 2023/5/22 19:04, fengchengwen 写道:
> On 2023/1/31 11:33, Huisong Li wrote:
>> The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
>> iterate over all enabled ports to do something(like, verifying the port id
>> validity) when receive a probing event. If the ethdev state of a port is
>> not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.
>>
>> However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
>> event. It means that probing callback will skip this port. But this
>> assignment can not move to front of probing notification. See
>> commit be8cd210379a ("ethdev: fix port probing notification")
> ...
>
>>   
>>   struct rte_eth_dev_sriov {
>> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
>> index 17201fbe0f..094c2a952e 100644
>> --- a/lib/ethdev/version.map
>> +++ b/lib/ethdev/version.map
>> @@ -327,4 +327,5 @@ INTERNAL {
>>   	rte_eth_representor_id_get;
>>   	rte_eth_switch_domain_alloc;
>>   	rte_eth_switch_domain_free;
>> +	rte_eth_dev_is_used;
> requires alphabetical order.
Thanks, will fix it in v6.
>
>>   };
>>
> .

[PATCH V6 0/5] app/testpmd: support multiple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v6: adjust rte_eth_dev_is_used position based on alphabetical order
      in version.map
 -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
 -v4: fix a misspelling. 
 -v3:
   #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
      for other bus type.
   #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 
 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.22.0

[PATCH V6 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff1369353a..13cb3fe0f8 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index ffb0c61214..bfe7077645 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -294,13 +294,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index e32a9d517a..ad3ab9e797 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 7974b27295..286ba1fbf9 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 8d32d66504..feb1651984 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH V6 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.

However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and set it to
RTE_ETH_DEV_ATTACHED after definitely probed. And this port is valid if its
device state is 'ALLOCATED' or 'ATTACHED'.

In addition, the new state has to be placed behind 'REMOVED' to avoid ABI
break. Fortunately, this ethdev state is internal and applications can not
access it directly. So this patch encapsulates an API, rte_eth_dev_is_used,
for ethdev or PMD to call and eliminate concerns about using this state
enum value comparison.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index ef7b8859d9..74ec0c88fb 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6053,7 +6053,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_is_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a75fa1b7f0..881425bf83 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3145,7 +3145,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_is_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index 0be1e8ca04..29e9417bea 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_is_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_is_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	eth_dev_shared_data_prepare();
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_is_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 367c0c4878..d5fd6e19ba 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1583,6 +1583,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_is_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 94b8fba5d7..23270ccd73 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -164,7 +164,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_is_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index b61dae849d..88e56dd9a4 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_is_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index d46e74504e..c8f800bb12 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -338,7 +338,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -397,7 +397,7 @@ rte_eth_dev_is_valid_port(uint16_t port_id)
 	int is_valid;
 
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		is_valid = 0;
 	else
 		is_valid = 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index fe8f7466c8..d4de7942d0 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2002,10 +2002,12 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
 };
 
 struct rte_eth_dev_sriov {
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index 041f0da31f..673123dfb7 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -317,6 +317,7 @@ INTERNAL {
 	rte_eth_dev_get_by_name;
 	rte_eth_dev_is_rx_hairpin_queue;
 	rte_eth_dev_is_tx_hairpin_queue;
+	rte_eth_dev_is_used;
 	rte_eth_dev_probing_finish;
 	rte_eth_dev_release_port;
 	rte_eth_dev_internal_reset;
-- 
2.22.0

[PATCH V6 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 5cb6f92523..7e55b1ffc3 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3931,14 +3931,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH V6 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

The port information needs to be updated due to attaching and detaching
port. Currently, it is done in the same thread as removing or probing
device, which doesn't satisfy the operation of attaching and detaching
device in multiple process.

If this operation is performed in one process, the other process can
receive 'new' or 'destroy' event. So we can move updating port information
to event callback to support attaching and detaching port in primary and
secondary process.

The reason for adding an alarm callback in 'destroy' event is that the
ethdev state is changed from 'ATTACHED' to 'UNUSED' only after the event
callback finished. But the remove_invalid_ports() function removes invalid
port only if ethdev state is 'UNUSED'. If we don't add alarm callback, this
detached port information can not be removed.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 7e55b1ffc3..f03b3dfb84 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3596,15 +3596,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3635,7 +3632,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3669,10 +3665,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3739,11 +3733,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3907,11 +3899,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3936,8 +3939,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3948,6 +3951,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index bdfbfd36d3..2855dad69e 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -308,7 +308,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index b3c12cada0..160f638e3a 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -493,7 +493,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH V6 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index f03b3dfb84..7d9d6decc9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3939,6 +3939,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3949,6 +3951,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[Ferruh Yigit]

On 2/28/2023 2:21 AM, lihuisong (C) wrote:
> 
> 在 2023/2/16 0:09, Ferruh Yigit 写道:
>> On 1/12/2023 2:44 AM, lihuisong (C) wrote:
>>> 在 2023/1/11 20:51, Ferruh Yigit 写道:
>>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>>> The driver assignment was moved back at the end of the device probing
>>>>> because there is no something to use rte_driver during the phase of
>>>>> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
>>>>> to end of probing")
>>>>>
>>>>> However, it is necessary for probing callback to reference rte_driver
>>>>> before probing. For example, probing callback may call some APIs which
>>>>> access the rte_pci_driver::driver by the device::driver pointer to get
>>>>> driver information. In this case, a segment fault will occur in probing
>>>>> callback if there is not this assignment.
>>>>>
>>>> Probing callback gets driver as parameter, so callback function can
>>>> access it via 'drv->driver', is there a specific usecase that
>>>> 'dev->device->driver' needs to be accessed explicitly?
>>>>
>>>> I assume this is related to coming patches that setting up device in
>>>> testpmd event callback, but can you please clarify exact need.
>>> For example, rte_eth_dev_info_get is called in this event callback to get
>>> driver name.
>>>
> Hi Ferruh,
> 
> Sorry for the delay. I missed this email.
> Thanks for your review.
>> Why 'rte_eth_dev_info_get()' is called in the event called at first place?
> There is no limitation on rte_eth_dev_info_get() in event callback.
> The upper layer is entirely possible to use.
> After all, it is more convenient for app to use this pointer to get
> driver information in a probing callback.
>

I think there is a logical limit in calling 'rte_eth_dev_info_get()' in
NEW event callback, we may document this if it is missing.

The NEW event callback is to let application do proper tasks before port
addition finalized in the ethdev layer, but you are trying to call
'rte_eth_dev_info_get()' in the callback when it is not ready yet.


>> This set updates multiple things to extend 'RTE_ETH_EVENT_NEW' event
>> callback function support, like:
>>
>> - Assign device driver *before* probing completed, so that even callback
>> can run 'rte_eth_dev_info_get()'
>>
>> - Add a new ethdev state so that port can be recognized as valid port in
>> the even callback.
> Yes
>> - Stop forwarding implicitly in even callback in case event callback run
>> while forwarding is on.
> But this is a modification for tesptmd to make it work well.
>> All looks to me hack/complexity to make a specific case work, which is
>> make secondary *testmp* application work with attached/detached device.
> It is not just a testpmd application case, but, I think, still exists in
> actual case.
> Because eal lib supports hotplugging device on primary and secondary
> process and the communication each other when attach or detach device.
> The reason why no one has ever put forward this question, I think it may
> be attributed to the fact that the scene is not or rarely tested.
>> And finally patch 4/5 adds port setup to testpmd event callback for this.
>>
>>
>> I understand the intention, but I disagree with bus and ethdev level
>> changes for this.
> I'm just raising this issue, and we can discuss how to deal with it
> together.😁
>>
>> Event callback may not be only way to share port attach/detach
>> information between primary and secondary, there is a MP socket and
>> 'rte_mp_handle' thread to handle communication between primary and
>> secondary process, this should be able to use carrying device
>> information, as far as I remember this is why it is introduced at first
>> place.
>>
>> Did you consider using MP socket for your use case?
> Actually, the probing event callback called in patch 4/5 is the result
> of the MP socket communication (please see hogplug_mp.c).
>>
>> Following is a sample usage:
>>
>> Primary:
>> started as:
>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 0-1
>> --log-level=*:debug -- -i --num-procs=2 --proc-id=0
>>
>> ``
>> testpmd> show port summary all
>> Number of available ports: 0
>> Port MAC Address       Name         Driver         Status   Link
>>
>> testpmd> port attach net_null0
>> Attaching a new port...
>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>> rte_pmd_null_probe(): Configure pmd_null: packet size is 64, packet copy
>> is disabled
>> eth_dev_null_create(): Creating null ethdev on numa socket 0
>> EAL: request: eal_dev_mp_request
>> EAL: msg: bus_vdev_mp
>> vdev_action(): send vdev, net_null0
>> EAL: sendmsg: bus_vdev_mp
>> EAL: reply: bus_vdev_mp
>> EAL: msg: eal_dev_mp_request
>> Port 0 is attached. Now total ports is 1
>> Done
>>
>> testpmd> show port summary all
>> Number of available ports: 1
>> Port MAC Address       Name         Driver         Status   Link
>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>>
>> testpmd> port detach 0
>> Port was not closed
>> Removing a device...
>> EAL: request: eal_dev_mp_request
>> EAL: msg: eal_dev_mp_request
>> eth_dev_close(): Closing null ethdev on NUMA socket 0
>> Port 0 is closed
>> Device is detached
>> Now total ports is 0
> Please note the log *"Now total ports is 0"*,
> which indicates the port number is updated if we detached device in this
> process.
>

Yes, as expected.


>> Done
>>
>> testpmd> show port summary all
>> Number of available ports: 0
>> Port MAC Address       Name         Driver         Status   Link
>> testpmd>
>>
>> ``
>>
>> Secondary:
>> started as:
>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 2-3
>> --log-level=*:debug -- -i --num-procs=2 --proc-id=1
>>
>> ``
>> testpmd> show port summary all
>> Number of available ports: 0
>> Port MAC Address       Name         Driver         Status   Link
>>
>> testpmd> EAL: msg: eal_dev_mp_request
>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>> EAL: request: bus_vdev_mp
>> EAL: msg: bus_vdev_mp
>> vdev_action(): receive vdev, net_null0
>> EAL: msg: bus_vdev_mp
>> vdev_scan(): Received 1 vdevs
>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>> EAL: reply: eal_dev_mp_request
>>
>> testpmd> show port summary all
>> Number of available ports: 1
>> Port MAC Address       Name         Driver         Status   Link
>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>>
>> testpmd> EAL: msg: eal_dev_mp_request
>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>> eth_dev_close(): Closing null ethdev on NUMA socket 4294967295
>> Port 0 is closed
>> EAL: reply: eal_dev_mp_request
> But the port number in this process does not be updated after finishing
> the destroy event.
> That's the problem.
>

Please see following command output,
it says "Number of available ports: 0",
so that shows port number is updated after destroy event, isn't it?


>> testpmd> show port summary all
>> Number of available ports: 0
>> Port MAC Address       Name         Driver         Status   Link
>> testpmd>
>> ``
>>
>>
>> .

Re: [PATCH V6 0/5] app/testpmd: support multiple process attach and detach port

[Ferruh Yigit]

On 5/27/2023 3:11 AM, Huisong Li wrote:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
> 

Hi Huisong,

As commented on v4, I have some concerns on this set.

The set does multiple ethdev/testpmd change, but the main target of the
patch is not described clearly/simply.

It looks like intention is to be able to register NEW event callback in
the secondary process and be able to setup device in secondary when
primary attaches a device,
but my question is why not multi-process communication socket can't be
used for this?

MP socket/communication/thread is developed for this reason, I am not
convinced why it can't be used to sync primary and secondary for device
attach/detach.


> ---
>  -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>       in version.map
>  -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>  -v4: fix a misspelling. 
>  -v3:
>    #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>       for other bus type.
>    #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>       the probelm in patch 2/5. 
>  -v2: resend due to CI unexplained failure.
> 
> Huisong Li (5):
>   drivers/bus: restore driver assignment at front of probing
>   ethdev: fix skip valid port in probing callback
>   app/testpmd: check the validity of the port
>   app/testpmd: add attach and detach port for multiple process
>   app/testpmd: stop forwarding in new or destroy event
> 
>  app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>  app/test-pmd/testpmd.h                   |  1 -
>  drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>  drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>  drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>  drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>  drivers/bus/pci/pci_common.c             |  9 ++++-
>  drivers/bus/vdev/vdev.c                  | 10 ++++-
>  drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>  drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>  drivers/net/bonding/bonding_testpmd.c    |  1 -
>  drivers/net/mlx5/mlx5.c                  |  2 +-
>  lib/ethdev/ethdev_driver.c               | 13 +++++--
>  lib/ethdev/ethdev_driver.h               | 12 ++++++
>  lib/ethdev/ethdev_pci.h                  |  2 +-
>  lib/ethdev/rte_class_eth.c               |  2 +-
>  lib/ethdev/rte_ethdev.c                  |  4 +-
>  lib/ethdev/rte_ethdev.h                  |  4 +-
>  lib/ethdev/version.map                   |  1 +
>  19 files changed, 114 insertions(+), 44 deletions(-)
> 

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[lihuisong (C)]

在 2023/6/7 0:12, Ferruh Yigit 写道:
> On 2/28/2023 2:21 AM, lihuisong (C) wrote:
>> 在 2023/2/16 0:09, Ferruh Yigit 写道:
>>> On 1/12/2023 2:44 AM, lihuisong (C) wrote:
>>>> 在 2023/1/11 20:51, Ferruh Yigit 写道:
>>>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>>>> The driver assignment was moved back at the end of the device probing
>>>>>> because there is no something to use rte_driver during the phase of
>>>>>> probing. See commit 391797f04208 ("drivers/bus: move driver assignment
>>>>>> to end of probing")
>>>>>>
>>>>>> However, it is necessary for probing callback to reference rte_driver
>>>>>> before probing. For example, probing callback may call some APIs which
>>>>>> access the rte_pci_driver::driver by the device::driver pointer to get
>>>>>> driver information. In this case, a segment fault will occur in probing
>>>>>> callback if there is not this assignment.
>>>>>>
>>>>> Probing callback gets driver as parameter, so callback function can
>>>>> access it via 'drv->driver', is there a specific usecase that
>>>>> 'dev->device->driver' needs to be accessed explicitly?
>>>>>
>>>>> I assume this is related to coming patches that setting up device in
>>>>> testpmd event callback, but can you please clarify exact need.
>>>> For example, rte_eth_dev_info_get is called in this event callback to get
>>>> driver name.
>>>>
>> Hi Ferruh,
>>
>> Sorry for the delay. I missed this email.
>> Thanks for your review.
>>> Why 'rte_eth_dev_info_get()' is called in the event called at first place?
>> There is no limitation on rte_eth_dev_info_get() in event callback.
>> The upper layer is entirely possible to use.
>> After all, it is more convenient for app to use this pointer to get
>> driver information in a probing callback.
>>
> I think there is a logical limit in calling 'rte_eth_dev_info_get()' in
> NEW event callback, we may document this if it is missing.
>
> The NEW event callback is to let application do proper tasks before port
> addition finalized in the ethdev layer, but you are trying to call
> 'rte_eth_dev_info_get()' in the callback when it is not ready yet.
We probably shouldn't add this logical limit.
In terms of application perception, app may need to do something when a 
new or destory event is received, instead of just for printing a message.
>
>>> This set updates multiple things to extend 'RTE_ETH_EVENT_NEW' event
>>> callback function support, like:
>>>
>>> - Assign device driver *before* probing completed, so that even callback
>>> can run 'rte_eth_dev_info_get()'
>>>
>>> - Add a new ethdev state so that port can be recognized as valid port in
>>> the even callback.
>> Yes
>>> - Stop forwarding implicitly in even callback in case event callback run
>>> while forwarding is on.
>> But this is a modification for tesptmd to make it work well.
>>> All looks to me hack/complexity to make a specific case work, which is
>>> make secondary *testmp* application work with attached/detached device.
>> It is not just a testpmd application case, but, I think, still exists in
>> actual case.
>> Because eal lib supports hotplugging device on primary and secondary
>> process and the communication each other when attach or detach device.
>> The reason why no one has ever put forward this question, I think it may
>> be attributed to the fact that the scene is not or rarely tested.
>>> And finally patch 4/5 adds port setup to testpmd event callback for this.
>>>
>>>
>>> I understand the intention, but I disagree with bus and ethdev level
>>> changes for this.
>> I'm just raising this issue, and we can discuss how to deal with it
>> together.😁
>>> Event callback may not be only way to share port attach/detach
>>> information between primary and secondary, there is a MP socket and
>>> 'rte_mp_handle' thread to handle communication between primary and
>>> secondary process, this should be able to use carrying device
>>> information, as far as I remember this is why it is introduced at first
>>> place.
>>>
>>> Did you consider using MP socket for your use case?
>> Actually, the probing event callback called in patch 4/5 is the result
>> of the MP socket communication (please see hogplug_mp.c).
>>> Following is a sample usage:
>>>
>>> Primary:
>>> started as:
>>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 0-1
>>> --log-level=*:debug -- -i --num-procs=2 --proc-id=0
>>>
>>> ``
>>> testpmd> show port summary all
>>> Number of available ports: 0
>>> Port MAC Address       Name         Driver         Status   Link
>>>
>>> testpmd> port attach net_null0
>>> Attaching a new port...
>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>>> rte_pmd_null_probe(): Configure pmd_null: packet size is 64, packet copy
>>> is disabled
>>> eth_dev_null_create(): Creating null ethdev on numa socket 0
>>> EAL: request: eal_dev_mp_request
>>> EAL: msg: bus_vdev_mp
>>> vdev_action(): send vdev, net_null0
>>> EAL: sendmsg: bus_vdev_mp
>>> EAL: reply: bus_vdev_mp
>>> EAL: msg: eal_dev_mp_request
>>> Port 0 is attached. Now total ports is 1
>>> Done
>>>
>>> testpmd> show port summary all
>>> Number of available ports: 1
>>> Port MAC Address       Name         Driver         Status   Link
>>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>>>
>>> testpmd> port detach 0
>>> Port was not closed
>>> Removing a device...
>>> EAL: request: eal_dev_mp_request
>>> EAL: msg: eal_dev_mp_request
>>> eth_dev_close(): Closing null ethdev on NUMA socket 0
>>> Port 0 is closed
>>> Device is detached
>>> Now total ports is 0
>> Please note the log *"Now total ports is 0"*,
>> which indicates the port number is updated if we detached device in this
>> process.
>>
> Yes, as expected.
Yeah, it is just expected for the process did attatch or dettach operation.
Because the thread for updating the port number info (in 
setup_attached_port) and doing probe thread is the same.
please see attach_port().
This is ok for testpmd that does not support multiple processes. But 
testpmd support multiple processes now.
>
>
>>> Done
>>>
>>> testpmd> show port summary all
>>> Number of available ports: 0
>>> Port MAC Address       Name         Driver         Status   Link
>>> testpmd>
>>>
>>> ``
>>>
>>> Secondary:
>>> started as:
>>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 2-3
>>> --log-level=*:debug -- -i --num-procs=2 --proc-id=1
>>>
>>> ``
>>> testpmd> show port summary all
>>> Number of available ports: 0
>>> Port MAC Address       Name         Driver         Status   Link
>>>
>>> testpmd> EAL: msg: eal_dev_mp_request
>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>> EAL: request: bus_vdev_mp
>>> EAL: msg: bus_vdev_mp
>>> vdev_action(): receive vdev, net_null0
>>> EAL: msg: bus_vdev_mp
>>> vdev_scan(): Received 1 vdevs
>>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>>> EAL: reply: eal_dev_mp_request
>>>
>>> testpmd> show port summary all
>>> Number of available ports: 1
>>> Port MAC Address       Name         Driver         Status   Link
>>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down     10 Gbps
>>>
>>> testpmd> EAL: msg: eal_dev_mp_request
>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>> eth_dev_close(): Closing null ethdev on NUMA socket 4294967295
>>> Port 0 is closed
>>> EAL: reply: eal_dev_mp_request
>> But the port number in this process does not be updated after finishing
>> the destroy event.
>> That's the problem.
>>
> Please see following command output,
> it says "Number of available ports: 0",
Here comes from rte_eth_dev_count_avail().
> so that shows port number is updated after destroy event, isn't it?
no.
There are three global variables,  "fwd_ports_ids[]", "ports[]", 
"nb_ports" maintained in testpmd.
The secondary doesn't update these info after receiving a destory or new 
message in your test.
>
>
>>> testpmd> show port summary all
>>> Number of available ports: 0
>>> Port MAC Address       Name         Driver         Status   Link
>>> testpmd>
>>> ``
>>>
>>>
>>> .
> .

Re: [PATCH V6 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

在 2023/6/7 0:26, Ferruh Yigit 写道:
> On 5/27/2023 3:11 AM, Huisong Li wrote:
>> This patchset fix some bugs and support attaching and detaching port
>> in primary and secondary.
>>
> Hi Huisong,
>
> As commented on v4, I have some concerns on this set.
please see my reply.
>
> The set does multiple ethdev/testpmd change, but the main target of the
> patch is not described clearly/simply.
The main target is to support attaching and detaching port in primary 
and secondary.
Fixed some problems by the way.
>
> It looks like intention is to be able to register NEW event callback in
> the secondary process and be able to setup device in secondary when
> primary attaches a device,
> but my question is why not multi-process communication socket can't be
> used for this?
>
> MP socket/communication/thread is developed for this reason, I am not
> convinced why it can't be used to sync primary and secondary for device
> attach/detach.
The secondary process automatically probes the device when primary 
attaches a device.
The primary process automatically probes the device before doing probe 
phase in secondary when secondary attaches a device.
Above behavior itself is attributed to multi-process socket 
communication(see hogplug_mp.c).
This series are just to support this feature in testpmd.
But hogplug_mp cannot do something for application, like updating 
information, which is the duty of application.
>
>
>> ---
>>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>>        in version.map
>>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>>   -v4: fix a misspelling.
>>   -v3:
>>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>>        for other bus type.
>>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>        the probelm in patch 2/5.
>>   -v2: resend due to CI unexplained failure.
>>
>> Huisong Li (5):
>>    drivers/bus: restore driver assignment at front of probing
>>    ethdev: fix skip valid port in probing callback
>>    app/testpmd: check the validity of the port
>>    app/testpmd: add attach and detach port for multiple process
>>    app/testpmd: stop forwarding in new or destroy event
>>
>>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>>   app/test-pmd/testpmd.h                   |  1 -
>>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>>   drivers/bus/pci/pci_common.c             |  9 ++++-
>>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>>   drivers/net/mlx5/mlx5.c                  |  2 +-
>>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>>   lib/ethdev/ethdev_pci.h                  |  2 +-
>>   lib/ethdev/rte_class_eth.c               |  2 +-
>>   lib/ethdev/rte_ethdev.c                  |  4 +-
>>   lib/ethdev/rte_ethdev.h                  |  4 +-
>>   lib/ethdev/version.map                   |  1 +
>>   19 files changed, 114 insertions(+), 44 deletions(-)
>>
> .

Re: [PATCH V4 1/5] drivers/bus: restore driver assignment at front of probing

[lihuisong (C)]

Hi Ferruh,

add the call stack as belows.


在 2023/6/7 18:11, lihuisong (C) 写道:
>
> 在 2023/6/7 0:12, Ferruh Yigit 写道:
>> On 2/28/2023 2:21 AM, lihuisong (C) wrote:
>>> 在 2023/2/16 0:09, Ferruh Yigit 写道:
>>>> On 1/12/2023 2:44 AM, lihuisong (C) wrote:
>>>>> 在 2023/1/11 20:51, Ferruh Yigit 写道:
>>>>>> On 12/6/2022 9:26 AM, Huisong Li wrote:
>>>>>>> The driver assignment was moved back at the end of the device 
>>>>>>> probing
>>>>>>> because there is no something to use rte_driver during the phase of
>>>>>>> probing. See commit 391797f04208 ("drivers/bus: move driver 
>>>>>>> assignment
>>>>>>> to end of probing")
>>>>>>>
>>>>>>> However, it is necessary for probing callback to reference 
>>>>>>> rte_driver
>>>>>>> before probing. For example, probing callback may call some APIs 
>>>>>>> which
>>>>>>> access the rte_pci_driver::driver by the device::driver pointer 
>>>>>>> to get
>>>>>>> driver information. In this case, a segment fault will occur in 
>>>>>>> probing
>>>>>>> callback if there is not this assignment.
>>>>>>>
>>>>>> Probing callback gets driver as parameter, so callback function can
>>>>>> access it via 'drv->driver', is there a specific usecase that
>>>>>> 'dev->device->driver' needs to be accessed explicitly?
>>>>>>
>>>>>> I assume this is related to coming patches that setting up device in
>>>>>> testpmd event callback, but can you please clarify exact need.
>>>>> For example, rte_eth_dev_info_get is called in this event callback 
>>>>> to get
>>>>> driver name.
>>>>>
>>> Hi Ferruh,
>>>
>>> Sorry for the delay. I missed this email.
>>> Thanks for your review.
>>>> Why 'rte_eth_dev_info_get()' is called in the event called at first 
>>>> place?
>>> There is no limitation on rte_eth_dev_info_get() in event callback.
>>> The upper layer is entirely possible to use.
>>> After all, it is more convenient for app to use this pointer to get
>>> driver information in a probing callback.
>>>
>> I think there is a logical limit in calling 'rte_eth_dev_info_get()' in
>> NEW event callback, we may document this if it is missing.
>>
>> The NEW event callback is to let application do proper tasks before port
>> addition finalized in the ethdev layer, but you are trying to call
>> 'rte_eth_dev_info_get()' in the callback when it is not ready yet.
> We probably shouldn't add this logical limit.
> In terms of application perception, app may need to do something when 
> a new or destory event is received, instead of just for printing a 
> message.
In addition, if no this patch(1/5), a segment fault will occur when 
attach a new device. The call stack is as belows:
-->
testpmd> port attach 0000:7d:00.0
Attaching a new port...
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_hns3 (19e5:a222) device: 0000:7d:00.0 (socket 0)

Thread 1 "dpdk-testpmd" received signal SIGSEGV, Segmentation fault.
rte_eth_dev_info_get (port_id=0, dev_info=0x17f95ba80) at 
../lib/ethdev/rte_ethdev.c:3708
3708        dev_info->driver_name = dev->device->driver->name;
Missing separate debuginfos, use: dnf debuginfo-install 
libatomic-10.3.1-10.oe2203.aarch64 libnl3-3.5.0-4.oe2203.aarch64 
openssl-libs-1.1.1m-1.oe2203.aarch64 zlib-1.2.11-19.oe2203.aarch64
(gdb) bt
#0  rte_eth_dev_info_get (port_id=0, dev_info=0x17f95ba80)
     at ../lib/ethdev/rte_ethdev.c:3708
#1  0x00000000005e6038 in eth_dev_info_get_print_err (port_id=0, 
dev_info=0x17f95ba80)
     at ../app/test-pmd/util.c:441
#2  0x00000000005d5654 in init_config_port_offloads (pid=0, socket_id=0)
     at ../app/test-pmd/testpmd.c:1632
#3  0x00000000005d5ec0 in reconfig (new_port_id=0, socket_id=0)
     at ../app/test-pmd/testpmd.c:1828
#4  0x00000000005dafa4 in setup_attached_port (pi=0) at 
../app/test-pmd/testpmd.c:3639
#5  0x00000000005dbd14 in eth_event_callback (port_id=0, 
type=RTE_ETH_EVENT_NEW,
     param=0x0, ret_param=0x0) at ../app/test-pmd/testpmd.c:3959
#6  0x00000000008677dc in rte_eth_dev_callback_process (dev=0x1826fd80 
<rte_eth_devices>,
     event=RTE_ETH_EVENT_NEW, ret_param=0x0) at 
../lib/ethdev/ethdev_driver.c:188
#7  0x0000000000867898 in rte_eth_dev_probing_finish (dev=0x1826fd80 
<rte_eth_devices>)
     at ../lib/ethdev/ethdev_driver.c:212
#8  0x00000000055621c8 in rte_eth_dev_pci_generic_probe 
(pci_dev=0x18500470,
     private_data_size=8448, dev_init=0x5571454 <hns3_dev_init>)
     at ../lib/ethdev/ethdev_pci.h:139
#9  0x00000000055717bc in eth_hns3_pci_probe (pci_drv=0x17e46728 
<rte_hns3_pmd>,
     pci_dev=0x18500470) at ../drivers/net/hns3/hns3_ethdev.c:6577
#10 0x0000000000970474 in rte_pci_probe_one_driver (dr=0x17e46728 
<rte_hns3_pmd>,
     dev=0x18500470) at ../drivers/bus/pci/pci_common.c:312
#11 0x00000000009706e0 in pci_probe_all_drivers (dev=0x18500470)
     at ../drivers/bus/pci/pci_common.c:396
#12 0x0000000000970f78 in pci_plug (dev=0x18500480) at 
../drivers/bus/pci/pci_common.c:670
#13 0x00000000008e31d4 in local_dev_probe (devargs=0xffffffffb6f0 
"0000:7d:00.0",
     new_dev=0xffffffff9478) at ../lib/eal/common/eal_common_dev.c:212
#14 0x00000000008e3330 in rte_dev_probe (devargs=0xffffffffb6f0 
"0000:7d:00.0")
     at ../lib/eal/common/eal_common_dev.c:264
#15 0x00000000005daeb4 in attach_port (identifier=0xffffffffb6f0 
"0000:7d:00.0")
     at ../app/test-pmd/testpmd.c:3608
#16 0x0000000000579a74 in cmd_operate_attach_port_parsed 
(parsed_result=0xffffffffb5f0,
     cl=0x184f9010, data=0x0) at ../app/test-pmd/cmdline.c:1194
--Type <RET> for more, q to quit, c to continue without paging--
#17 0x00000000008624a4 in __cmdline_parse (cl=0x184f9010,
     buf=0x184f9058 "port attach 0000:7d:00.0\n", call_fn=true)
     at ../lib/cmdline/cmdline_parse.c:294
#18 0x00000000008624dc in cmdline_parse (cl=0x184f9010,
     buf=0x184f9058 "port attach 0000:7d:00.0\n") at 
../lib/cmdline/cmdline_parse.c:302
#19 0x0000000000860308 in cmdline_valid_buffer (rdl=0x184f9020,
     buf=0x184f9058 "port attach 0000:7d:00.0\n", size=26) at 
../lib/cmdline/cmdline.c:24
#20 0x000000000086588c in rdline_char_in (rdl=0x184f9020, c=10 '\n')
     at ../lib/cmdline/cmdline_rdline.c:444
#21 0x0000000000860750 in cmdline_in (cl=0x184f9010,
     buf=0xfffffffff77f "\n\220\367\377\377\377\377", size=1)
     at ../lib/cmdline/cmdline.c:146
#22 0x0000000000860a14 in cmdline_interact (cl=0x184f9010) at 
../lib/cmdline/cmdline.c:226
#23 0x0000000000587908 in prompt () at ../app/test-pmd/cmdline.c:13065
#24 0x00000000005ddaf4 in main (argc=6, argv=0xfffffffffad8)
     at ../app/test-pmd/testpmd.c:4690
>>
>>>> This set updates multiple things to extend 'RTE_ETH_EVENT_NEW' event
>>>> callback function support, like:
>>>>
>>>> - Assign device driver *before* probing completed, so that even 
>>>> callback
>>>> can run 'rte_eth_dev_info_get()'
>>>>
>>>> - Add a new ethdev state so that port can be recognized as valid 
>>>> port in
>>>> the even callback.
>>> Yes
>>>> - Stop forwarding implicitly in even callback in case event 
>>>> callback run
>>>> while forwarding is on.
>>> But this is a modification for tesptmd to make it work well.
>>>> All looks to me hack/complexity to make a specific case work, which is
>>>> make secondary *testmp* application work with attached/detached 
>>>> device.
>>> It is not just a testpmd application case, but, I think, still 
>>> exists in
>>> actual case.
>>> Because eal lib supports hotplugging device on primary and secondary
>>> process and the communication each other when attach or detach device.
>>> The reason why no one has ever put forward this question, I think it 
>>> may
>>> be attributed to the fact that the scene is not or rarely tested.
>>>> And finally patch 4/5 adds port setup to testpmd event callback for 
>>>> this.
>>>>
>>>>
>>>> I understand the intention, but I disagree with bus and ethdev level
>>>> changes for this.
>>> I'm just raising this issue, and we can discuss how to deal with it
>>> together.😁
>>>> Event callback may not be only way to share port attach/detach
>>>> information between primary and secondary, there is a MP socket and
>>>> 'rte_mp_handle' thread to handle communication between primary and
>>>> secondary process, this should be able to use carrying device
>>>> information, as far as I remember this is why it is introduced at 
>>>> first
>>>> place.
>>>>
>>>> Did you consider using MP socket for your use case?
>>> Actually, the probing event callback called in patch 4/5 is the result
>>> of the MP socket communication (please see hogplug_mp.c).
>>>> Following is a sample usage:
>>>>
>>>> Primary:
>>>> started as:
>>>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 0-1
>>>> --log-level=*:debug -- -i --num-procs=2 --proc-id=0
>>>>
>>>> ``
>>>> testpmd> show port summary all
>>>> Number of available ports: 0
>>>> Port MAC Address       Name         Driver         Status Link
>>>>
>>>> testpmd> port attach net_null0
>>>> Attaching a new port...
>>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>>>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>>>> rte_pmd_null_probe(): Configure pmd_null: packet size is 64, packet 
>>>> copy
>>>> is disabled
>>>> eth_dev_null_create(): Creating null ethdev on numa socket 0
>>>> EAL: request: eal_dev_mp_request
>>>> EAL: msg: bus_vdev_mp
>>>> vdev_action(): send vdev, net_null0
>>>> EAL: sendmsg: bus_vdev_mp
>>>> EAL: reply: bus_vdev_mp
>>>> EAL: msg: eal_dev_mp_request
>>>> Port 0 is attached. Now total ports is 1
>>>> Done
>>>>
>>>> testpmd> show port summary all
>>>> Number of available ports: 1
>>>> Port MAC Address       Name         Driver         Status Link
>>>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down 10 Gbps
>>>>
>>>> testpmd> port detach 0
>>>> Port was not closed
>>>> Removing a device...
>>>> EAL: request: eal_dev_mp_request
>>>> EAL: msg: eal_dev_mp_request
>>>> eth_dev_close(): Closing null ethdev on NUMA socket 0
>>>> Port 0 is closed
>>>> Device is detached
>>>> Now total ports is 0
>>> Please note the log *"Now total ports is 0"*,
>>> which indicates the port number is updated if we detached device in 
>>> this
>>> process.
>>>
>> Yes, as expected.
> Yeah, it is just expected for the process did attatch or dettach 
> operation.
> Because the thread for updating the port number info (in 
> setup_attached_port) and doing probe thread is the same.
> please see attach_port().
> This is ok for testpmd that does not support multiple processes. But 
> testpmd support multiple processes now.
>>
>>
>>>> Done
>>>>
>>>> testpmd> show port summary all
>>>> Number of available ports: 0
>>>> Port MAC Address       Name         Driver         Status Link
>>>> testpmd>
>>>>
>>>> ``
>>>>
>>>> Secondary:
>>>> started as:
>>>> sudo ./build/app/dpdk-testpmd --no-pci --proc-type=auto -l 2-3
>>>> --log-level=*:debug -- -i --num-procs=2 --proc-id=1
>>>>
>>>> ``
>>>> testpmd> show port summary all
>>>> Number of available ports: 0
>>>> Port MAC Address       Name         Driver         Status Link
>>>>
>>>> testpmd> EAL: msg: eal_dev_mp_request
>>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>>> EAL: request: bus_vdev_mp
>>>> EAL: msg: bus_vdev_mp
>>>> vdev_action(): receive vdev, net_null0
>>>> EAL: msg: bus_vdev_mp
>>>> vdev_scan(): Received 1 vdevs
>>>> vdev_probe_all_drivers(): Search driver to probe device net_null0
>>>> rte_pmd_null_probe(): Initializing pmd_null for net_null0
>>>> EAL: reply: eal_dev_mp_request
>>>>
>>>> testpmd> show port summary all
>>>> Number of available ports: 1
>>>> Port MAC Address       Name         Driver         Status Link
>>>> 0    DE:E5:79:00:A9:68 net_null0    net_null       down 10 Gbps
>>>>
>>>> testpmd> EAL: msg: eal_dev_mp_request
>>>> dpaa: rte_dpaa_bus_parse(): Parse device name (net_null0 )
>>>> fslmc: rte_fslmc_parse(): Parsing dev=(net_null0 )
>>>> fslmc: rte_fslmc_parse(): Unknown or unsupported device (net_null0 )
>>>> eth_dev_close(): Closing null ethdev on NUMA socket 4294967295
>>>> Port 0 is closed
>>>> EAL: reply: eal_dev_mp_request
>>> But the port number in this process does not be updated after finishing
>>> the destroy event.
>>> That's the problem.
>>>
>> Please see following command output,
>> it says "Number of available ports: 0",
> Here comes from rte_eth_dev_count_avail().
>> so that shows port number is updated after destroy event, isn't it?
> no.
> There are three global variables,  "fwd_ports_ids[]", "ports[]", 
> "nb_ports" maintained in testpmd.
> The secondary doesn't update these info after receiving a destory or 
> new message in your test.
>>
>>
>>>> testpmd> show port summary all
>>>> Number of available ports: 0
>>>> Port MAC Address       Name         Driver         Status Link
>>>> testpmd>
>>>> ``
>>>>
>>>>
>>>> .
>> .
> .

Re: [PATCH V6 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh,

Can you take a look at this series?
I added the call stack info for segment fault.

/Huisong


在 2023/5/27 10:11, Huisong Li 写道:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
>
> ---
>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>        in version.map
>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>   -v4: fix a misspelling.
>   -v3:
>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>        for other bus type.
>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>        the probelm in patch 2/5.
>   -v2: resend due to CI unexplained failure.
>
> Huisong Li (5):
>    drivers/bus: restore driver assignment at front of probing
>    ethdev: fix skip valid port in probing callback
>    app/testpmd: check the validity of the port
>    app/testpmd: add attach and detach port for multiple process
>    app/testpmd: stop forwarding in new or destroy event
>
>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>   app/test-pmd/testpmd.h                   |  1 -
>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>   drivers/bus/pci/pci_common.c             |  9 ++++-
>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>   drivers/net/mlx5/mlx5.c                  |  2 +-
>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>   lib/ethdev/ethdev_pci.h                  |  2 +-
>   lib/ethdev/rte_class_eth.c               |  2 +-
>   lib/ethdev/rte_ethdev.c                  |  4 +-
>   lib/ethdev/rte_ethdev.h                  |  4 +-
>   lib/ethdev/version.map                   |  1 +
>   19 files changed, 114 insertions(+), 44 deletions(-)
>

[PATCH RESEND v6 0/5] app/testpmd: support multiple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v6: adjust rte_eth_dev_is_used position based on alphabetical order
      in version.map
 -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
 -v4: fix a misspelling. 
 -v3:
   #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
      for other bus type.
   #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 
 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.22.0

[PATCH RESEND v6 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 29f99342a7..6313684b8f 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index ffb0c61214..bfe7077645 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -294,13 +294,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 52404ab0fe..c0d9e959ea 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 7974b27295..286ba1fbf9 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 95f3ad78bc..df63eaeaf9 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -110,6 +110,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -119,9 +125,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.22.0

[PATCH RESEND v6 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.

However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and set it to
RTE_ETH_DEV_ATTACHED after definitely probed. And this port is valid if its
device state is 'ALLOCATED' or 'ATTACHED'.

In addition, the new state has to be placed behind 'REMOVED' to avoid ABI
break. Fortunately, this ethdev state is internal and applications can not
access it directly. So this patch encapsulates an API, rte_eth_dev_is_used,
for ethdev or PMD to call and eliminate concerns about using this state
enum value comparison.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index ee1552452a..bf1910709b 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6108,7 +6108,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_is_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b373306f98..54c6fff889 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3152,7 +3152,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_is_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index 0be1e8ca04..29e9417bea 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -50,8 +50,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_is_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -208,11 +208,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_is_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -221,7 +228,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	eth_dev_shared_data_prepare();
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_is_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index 980f837ab6..5bd2780643 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1582,6 +1582,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_is_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 320e3e0093..efe20be1a7 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -165,7 +165,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_is_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index b61dae849d..88e56dd9a4 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_is_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 0840d2b594..ec44490cde 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -338,7 +338,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -397,7 +397,7 @@ rte_eth_dev_is_valid_port(uint16_t port_id)
 	int is_valid;
 
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		is_valid = 0;
 	else
 		is_valid = 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 04a2564f22..e7e521efc4 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2003,10 +2003,12 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
 };
 
 struct rte_eth_dev_sriov {
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index b965d6aa52..ad95329b57 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -326,6 +326,7 @@ INTERNAL {
 	rte_eth_dev_get_by_name;
 	rte_eth_dev_is_rx_hairpin_queue;
 	rte_eth_dev_is_tx_hairpin_queue;
+	rte_eth_dev_is_used;
 	rte_eth_dev_probing_finish;
 	rte_eth_dev_release_port;
 	rte_eth_dev_internal_reset;
-- 
2.22.0

[PATCH RESEND v6 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 938ca035d4..f934f6b418 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3945,14 +3945,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.22.0

[PATCH RESEND v6 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

The port information needs to be updated due to attaching and detaching
port. Currently, it is done in the same thread as removing or probing
device, which doesn't satisfy the operation of attaching and detaching
device in multiple process.

If this operation is performed in one process, the other process can
receive 'new' or 'destroy' event. So we can move updating port information
to event callback to support attaching and detaching port in primary and
secondary process.

The reason for adding an alarm callback in 'destroy' event is that the
ethdev state is changed from 'ATTACHED' to 'UNUSED' only after the event
callback finished. But the remove_invalid_ports() function removes invalid
port only if ethdev state is 'UNUSED'. If we don't add alarm callback, this
detached port information can not be removed.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index f934f6b418..a5124c3231 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3610,15 +3610,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3649,7 +3646,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3683,10 +3679,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3753,11 +3747,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3921,11 +3913,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3950,8 +3953,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3962,6 +3965,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index f1df6a8faf..daacdd7ee4 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -324,7 +324,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index b3c12cada0..160f638e3a 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -493,7 +493,6 @@ static void cmd_create_bonded_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.22.0

[PATCH RESEND v6 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index a5124c3231..15cbede69b 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3953,6 +3953,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3963,6 +3965,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.22.0

Re: [PATCH RESEND v6 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh and Thomas,

Can you take a look at this series? They've been over a year on 
disscussion.


在 2023/8/2 11:15, Huisong Li 写道:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
>
> ---
>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>        in version.map
>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>   -v4: fix a misspelling.
>   -v3:
>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>        for other bus type.
>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>        the probelm in patch 2/5.
>   -v2: resend due to CI unexplained failure.
>
> Huisong Li (5):
>    drivers/bus: restore driver assignment at front of probing
>    ethdev: fix skip valid port in probing callback
>    app/testpmd: check the validity of the port
>    app/testpmd: add attach and detach port for multiple process
>    app/testpmd: stop forwarding in new or destroy event
>
>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>   app/test-pmd/testpmd.h                   |  1 -
>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>   drivers/bus/pci/pci_common.c             |  9 ++++-
>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>   drivers/net/mlx5/mlx5.c                  |  2 +-
>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>   lib/ethdev/ethdev_pci.h                  |  2 +-
>   lib/ethdev/rte_class_eth.c               |  2 +-
>   lib/ethdev/rte_ethdev.c                  |  4 +-
>   lib/ethdev/rte_ethdev.h                  |  4 +-
>   lib/ethdev/version.map                   |  1 +
>   19 files changed, 114 insertions(+), 44 deletions(-)
>

Re: [PATCH RESEND v6 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh and Thomas,

This series have been discussing more than one year.
Kindly ping for reivew.


在 2023/10/9 18:34, lihuisong (C) 写道:
> Hi Ferruh and Thomas,
>
> Can you take a look at this series? They've been over a year on 
> disscussion.
>
>
> 在 2023/8/2 11:15, Huisong Li 写道:
>> This patchset fix some bugs and support attaching and detaching port
>> in primary and secondary.
>>
>> ---
>>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>>        in version.map
>>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi 
>> break.
>>   -v4: fix a misspelling.
>>   -v3:
>>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add 
>> modification
>>        for other bus type.
>>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>>        the probelm in patch 2/5.
>>   -v2: resend due to CI unexplained failure.
>>
>> Huisong Li (5):
>>    drivers/bus: restore driver assignment at front of probing
>>    ethdev: fix skip valid port in probing callback
>>    app/testpmd: check the validity of the port
>>    app/testpmd: add attach and detach port for multiple process
>>    app/testpmd: stop forwarding in new or destroy event
>>
>>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>>   app/test-pmd/testpmd.h                   |  1 -
>>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>>   drivers/bus/pci/pci_common.c             |  9 ++++-
>>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>>   drivers/net/mlx5/mlx5.c                  |  2 +-
>>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>>   lib/ethdev/ethdev_pci.h                  |  2 +-
>>   lib/ethdev/rte_class_eth.c               |  2 +-
>>   lib/ethdev/rte_ethdev.c                  |  4 +-
>>   lib/ethdev/rte_ethdev.h                  |  4 +-
>>   lib/ethdev/version.map                   |  1 +
>>   19 files changed, 114 insertions(+), 44 deletions(-)
>>
>
> .

Re: [PATCH RESEND v6 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh and Thomas,

This series have been discussing for over a year and a half.
Looking back on previous discussions, we have also made some progress 
and consensus.
I am sticking to track it. Because they resolve a real exist issue.
Can you take a look at it again?

BR,
/Huisong


在 2023/10/30 20:17, lihuisong (C) 写道:
> Hi Ferruh and Thomas,
>
> This series have been discussing more than one year.
> Kindly ping for reivew.
>
>
> 在 2023/10/9 18:34, lihuisong (C) 写道:
>> Hi Ferruh and Thomas,
>>
>> Can you take a look at this series? They've been over a year on 
>> disscussion.
>>
>>
>> 在 2023/8/2 11:15, Huisong Li 写道:
>>> This patchset fix some bugs and support attaching and detaching port
>>> in primary and secondary.
>>>
>>> ---
>>>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>>>        in version.map
>>>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi 
>>> break.
>>>   -v4: fix a misspelling.
>>>   -v3:
>>>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add 
>>> modification
>>>        for other bus type.
>>>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to 
>>> resolve
>>>        the probelm in patch 2/5.
>>>   -v2: resend due to CI unexplained failure.
>>>
>>> Huisong Li (5):
>>>    drivers/bus: restore driver assignment at front of probing
>>>    ethdev: fix skip valid port in probing callback
>>>    app/testpmd: check the validity of the port
>>>    app/testpmd: add attach and detach port for multiple process
>>>    app/testpmd: stop forwarding in new or destroy event
>>>
>>>   app/test-pmd/testpmd.c                   | 47 
>>> +++++++++++++++---------
>>>   app/test-pmd/testpmd.h                   |  1 -
>>>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>>>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>>>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>>>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>>>   drivers/bus/pci/pci_common.c             |  9 ++++-
>>>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>>>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>>>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>>>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>>>   drivers/net/mlx5/mlx5.c                  |  2 +-
>>>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>>>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>>>   lib/ethdev/ethdev_pci.h                  |  2 +-
>>>   lib/ethdev/rte_class_eth.c               |  2 +-
>>>   lib/ethdev/rte_ethdev.c                  |  4 +-
>>>   lib/ethdev/rte_ethdev.h                  |  4 +-
>>>   lib/ethdev/version.map                   |  1 +
>>>   19 files changed, 114 insertions(+), 44 deletions(-)
>>>
>>
>> .
>
> .

[PATCH v7 0/5] app/testpmd: support multiple process attach and detach port

[Huisong Li]

This patchset fix some bugs and support attaching and detaching port
in primary and secondary.

---
 -v7: fix conflicts
 -v6: adjust rte_eth_dev_is_used position based on alphabetical order
      in version.map
 -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
 -v4: fix a misspelling. 
 -v3:
   #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
      for other bus type.
   #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
      the probelm in patch 2/5. 
 -v2: resend due to CI unexplained failure.

Huisong Li (5):
  drivers/bus: restore driver assignment at front of probing
  ethdev: fix skip valid port in probing callback
  app/testpmd: check the validity of the port
  app/testpmd: add attach and detach port for multiple process
  app/testpmd: stop forwarding in new or destroy event

 app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
 app/test-pmd/testpmd.h                   |  1 -
 drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
 drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
 drivers/bus/pci/pci_common.c             |  9 ++++-
 drivers/bus/vdev/vdev.c                  | 10 ++++-
 drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
 drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
 drivers/net/bonding/bonding_testpmd.c    |  1 -
 drivers/net/mlx5/mlx5.c                  |  2 +-
 lib/ethdev/ethdev_driver.c               | 13 +++++--
 lib/ethdev/ethdev_driver.h               | 12 ++++++
 lib/ethdev/ethdev_pci.h                  |  2 +-
 lib/ethdev/rte_class_eth.c               |  2 +-
 lib/ethdev/rte_ethdev.c                  |  4 +-
 lib/ethdev/rte_ethdev.h                  |  4 +-
 lib/ethdev/version.map                   |  1 +
 19 files changed, 114 insertions(+), 44 deletions(-)

-- 
2.33.0

[PATCH v7 1/5] drivers/bus: restore driver assignment at front of probing

[Huisong Li]

The driver assignment was moved back at the end of the device probing
because there is no something to use rte_driver during the phase of
probing. See commit 391797f04208 ("drivers/bus: move driver assignment
to end of probing")

However, it is necessary for probing callback to reference rte_driver
before probing. For example, probing callback may call some APIs which
access the rte_pci_driver::driver by the device::driver pointer to get
driver information. In this case, a segment fault will occur in probing
callback if there is not this assignment.

Further, some comments in code need to be updated if we do that. The
driver pointer in rte_device is set before probing and needs to be reset
if probing failed. And rte_dev_is_probed can not be called inside probing.

Fixes: 391797f04208 ("drivers/bus: move driver assignment to end of probing")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/bus/auxiliary/auxiliary_common.c |  9 +++++++--
 drivers/bus/dpaa/dpaa_bus.c              |  9 +++++++--
 drivers/bus/fslmc/fslmc_bus.c            |  8 +++++++-
 drivers/bus/ifpga/ifpga_bus.c            | 12 +++++++++---
 drivers/bus/pci/pci_common.c             |  9 +++++++--
 drivers/bus/vdev/vdev.c                  | 10 ++++++++--
 drivers/bus/vmbus/vmbus_common.c         |  9 +++++++--
 7 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 29f99342a7..6313684b8f 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -132,16 +132,21 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
 	}
 
 	dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &drv->driver;
 
 	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
 		      drv->driver.name, dev->name, dev->device.numa_node);
 	ret = drv->probe(drv, dev);
 	if (ret != 0) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &drv->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index e57159f5d8..f1b817e58c 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -693,17 +693,22 @@ rte_dpaa_bus_probe(void)
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_BLOCKED))
 				continue;
-
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			    (dev->device.devargs &&
 			     dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA_BUS_ERR("unable to probe:%s",
 						     dev->name);
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 57bfb5111a..4bc0c6d3d4 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -471,15 +471,21 @@ rte_fslmc_probe(void)
 				continue;
 			}
 
+			/*
+			 * Reference rte_driver before probing so as to this
+			 * pointer can be used to get driver information in case
+			 * of segment fault in probing callback.
+			 */
+			dev->device.driver = &drv->driver;
 			if (probe_all ||
 			   (dev->device.devargs &&
 			    dev->device.devargs->policy == RTE_DEV_ALLOWED)) {
 				ret = drv->probe(drv, dev);
 				if (ret) {
+					dev->device.driver = NULL;
 					DPAA2_BUS_ERR("Unable to probe");
 				} else {
 					dev->driver = drv;
-					dev->device.driver = &drv->driver;
 				}
 			}
 			break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index ffb0c61214..bfe7077645 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -294,13 +294,19 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
 
 	/* reference driver structure */
 	afu_dev->driver = drv;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	afu_dev->device.driver = &drv->driver;
 
 	/* call the driver probe() function */
 	ret = drv->probe(afu_dev);
-	if (ret)
+	if (ret) {
 		afu_dev->driver = NULL;
-	else
-		afu_dev->device.driver = &drv->driver;
+		afu_dev->device.driver = NULL;
+	}
 
 	return ret;
 }
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 889a48d2af..404173e7cc 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -302,6 +302,12 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 				return ret;
 			}
 		}
+		/*
+		 * Reference rte_driver before probing so as to this pointer can
+		 * be used to get driver information in case of segment fault in
+		 * probing callback.
+		 */
+		dev->device.driver = &dr->driver;
 	}
 
 	RTE_LOG(INFO, EAL, "Probe PCI driver: %s (%x:%04x) device: "PCI_PRI_FMT" (socket %i)\n",
@@ -314,6 +320,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		return ret; /* no rollback if already succeeded earlier */
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
 			/* Don't unmap if device is unsupported and
 			 * driver needs mapped resources.
@@ -325,8 +332,6 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
 		dev->vfio_req_intr_handle = NULL;
 		rte_intr_instance_free(dev->intr_handle);
 		dev->intr_handle = NULL;
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 7974b27295..286ba1fbf9 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -207,9 +207,15 @@ vdev_probe_all_drivers(struct rte_vdev_device *dev)
 		return -1;
 	}
 
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &driver->driver;
 	ret = driver->probe(dev);
-	if (ret == 0)
-		dev->device.driver = &driver->driver;
+	if (ret != 0)
+		dev->device.driver = NULL;
 	return ret;
 }
 
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index b9139c6e6c..0238f15f4f 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -119,6 +119,12 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 
 	/* reference driver structure */
 	dev->driver = dr;
+	/*
+	 * Reference rte_driver before probing so as to this pointer can
+	 * be used to get driver information in case of segment fault in
+	 * probing callback.
+	 */
+	dev->device.driver = &dr->driver;
 
 	if (dev->device.numa_node < 0 && rte_socket_count() > 1)
 		VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
@@ -128,9 +134,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
 	ret = dr->probe(dr, dev);
 	if (ret) {
 		dev->driver = NULL;
+		dev->device.driver = NULL;
 		rte_vmbus_unmap_device(dev);
-	} else {
-		dev->device.driver = &dr->driver;
 	}
 
 	return ret;
-- 
2.33.0

[PATCH v7 2/5] ethdev: fix skip valid port in probing callback

[Huisong Li]

The event callback in application may use the macro RTE_ETH_FOREACH_DEV to
iterate over all enabled ports to do something(like, verifying the port id
validity) when receive a probing event. If the ethdev state of a port is
not RTE_ETH_DEV_UNUSED, this port will be considered as a valid port.

However, this state is set to RTE_ETH_DEV_ATTACHED after pushing probing
event. It means that probing callback will skip this port. But this
assignment can not move to front of probing notification. See
commit be8cd210379a ("ethdev: fix port probing notification")

So this patch has to add a new state, RTE_ETH_DEV_ALLOCATED. Set the ethdev
state to RTE_ETH_DEV_ALLOCATED before pushing probing event and set it to
RTE_ETH_DEV_ATTACHED after definitely probed. And this port is valid if its
device state is 'ALLOCATED' or 'ATTACHED'.

In addition, the new state has to be placed behind 'REMOVED' to avoid ABI
break. Fortunately, this ethdev state is internal and applications can not
access it directly. So this patch encapsulates an API, rte_eth_dev_is_used,
for ethdev or PMD to call and eliminate concerns about using this state
enum value comparison.

Fixes: be8cd210379a ("ethdev: fix port probing notification")
Cc: stable@dpdk.org

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/net/bnxt/bnxt_ethdev.c |  3 ++-
 drivers/net/mlx5/mlx5.c        |  2 +-
 lib/ethdev/ethdev_driver.c     | 13 ++++++++++---
 lib/ethdev/ethdev_driver.h     | 12 ++++++++++++
 lib/ethdev/ethdev_pci.h        |  2 +-
 lib/ethdev/rte_class_eth.c     |  2 +-
 lib/ethdev/rte_ethdev.c        |  4 ++--
 lib/ethdev/rte_ethdev.h        |  4 +++-
 lib/ethdev/version.map         |  1 +
 9 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index acf7e6e46e..cb4c4cfb0e 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -6109,7 +6109,8 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+
+	if (rte_eth_dev_is_used(eth_dev->state))
 		bnxt_dev_close_op(eth_dev);
 
 	return 0;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 3a182de248..1f1d9b7d96 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -3258,7 +3258,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	while (port_id < RTE_MAX_ETHPORTS) {
 		struct rte_eth_dev *dev = &rte_eth_devices[port_id];
 
-		if (dev->state != RTE_ETH_DEV_UNUSED &&
+		if (rte_eth_dev_is_used(dev->state) &&
 		    dev->device &&
 		    (dev->device == odev ||
 		     (dev->device->driver &&
diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
index bd917a15fc..2b4a6e2b2c 100644
--- a/lib/ethdev/ethdev_driver.c
+++ b/lib/ethdev/ethdev_driver.c
@@ -52,8 +52,8 @@ eth_dev_find_free_port(void)
 	for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
 		/* Using shared name field to find a free port. */
 		if (eth_dev_shared_data->data[i].name[0] == '\0') {
-			RTE_ASSERT(rte_eth_devices[i].state ==
-				   RTE_ETH_DEV_UNUSED);
+			RTE_ASSERT(!rte_eth_dev_is_used(
+					rte_eth_devices[i].state));
 			return i;
 		}
 	}
@@ -217,11 +217,18 @@ rte_eth_dev_probing_finish(struct rte_eth_dev *dev)
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
 		eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
 
+	dev->state = RTE_ETH_DEV_ALLOCATED;
 	rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_NEW, NULL);
 
 	dev->state = RTE_ETH_DEV_ATTACHED;
 }
 
+bool rte_eth_dev_is_used(uint16_t dev_state)
+{
+	return dev_state == RTE_ETH_DEV_ALLOCATED ||
+		dev_state == RTE_ETH_DEV_ATTACHED;
+}
+
 int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
@@ -239,7 +246,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (ret != 0)
 		return ret;
 
-	if (eth_dev->state != RTE_ETH_DEV_UNUSED)
+	if (rte_eth_dev_is_used(eth_dev->state))
 		rte_eth_dev_callback_process(eth_dev,
 				RTE_ETH_EVENT_DESTROY, NULL);
 
diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
index f05f68a67c..238bd72ab0 100644
--- a/lib/ethdev/ethdev_driver.h
+++ b/lib/ethdev/ethdev_driver.h
@@ -1596,6 +1596,18 @@ int rte_eth_dev_callback_process(struct rte_eth_dev *dev,
 __rte_internal
 void rte_eth_dev_probing_finish(struct rte_eth_dev *dev);
 
+/**
+ * Check if a Ethernet device state is used or not
+ *
+ * @param dev_state
+ *   The state of the Ethernet device
+ * @return
+ *   - true if the state of the Ethernet device is allocated or attached
+ *   - false if this state is neither allocated nor attached
+ */
+__rte_internal
+bool rte_eth_dev_is_used(uint16_t dev_state);
+
 /**
  * Create memzone for HW rings.
  * malloc can't be used as the physical address is needed.
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 737fff1833..c07a98e04f 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -165,7 +165,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	 * eth device has been released.
 	 */
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    eth_dev->state == RTE_ETH_DEV_UNUSED)
+	    !rte_eth_dev_is_used(eth_dev->state))
 		return 0;
 
 	if (dev_uninit) {
diff --git a/lib/ethdev/rte_class_eth.c b/lib/ethdev/rte_class_eth.c
index bc003db8af..31cabee525 100644
--- a/lib/ethdev/rte_class_eth.c
+++ b/lib/ethdev/rte_class_eth.c
@@ -118,7 +118,7 @@ eth_dev_match(const struct rte_eth_dev *edev,
 	const struct rte_kvargs *kvlist = arg->kvlist;
 	unsigned int pair;
 
-	if (edev->state == RTE_ETH_DEV_UNUSED)
+	if (!rte_eth_dev_is_used(edev->state))
 		return -1;
 	if (arg->device != NULL && arg->device != edev->device)
 		return -1;
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index f61dc7dd7b..b6b7a20e35 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -349,7 +349,7 @@ uint16_t
 rte_eth_find_next(uint16_t port_id)
 {
 	while (port_id < RTE_MAX_ETHPORTS &&
-			rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED)
+	       !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		port_id++;
 
 	if (port_id >= RTE_MAX_ETHPORTS)
@@ -408,7 +408,7 @@ rte_eth_dev_is_valid_port(uint16_t port_id)
 	int is_valid;
 
 	if (port_id >= RTE_MAX_ETHPORTS ||
-	    (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED))
+	    !rte_eth_dev_is_used(rte_eth_devices[port_id].state))
 		is_valid = 0;
 	else
 		is_valid = 1;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 2687c23fa6..5079fea8a6 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -2068,10 +2068,12 @@ typedef uint16_t (*rte_tx_callback_fn)(uint16_t port_id, uint16_t queue,
 enum rte_eth_dev_state {
 	/** Device is unused before being probed. */
 	RTE_ETH_DEV_UNUSED = 0,
-	/** Device is attached when allocated in probing. */
+	/** Device is attached when definitely probed. */
 	RTE_ETH_DEV_ATTACHED,
 	/** Device is in removed state when plug-out is detected. */
 	RTE_ETH_DEV_REMOVED,
+	/** Device is allocated and is set before reporting new event. */
+	RTE_ETH_DEV_ALLOCATED,
 };
 
 struct rte_eth_dev_sriov {
diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
index a050baab0f..26867eb6e5 100644
--- a/lib/ethdev/version.map
+++ b/lib/ethdev/version.map
@@ -333,6 +333,7 @@ INTERNAL {
 	rte_eth_dev_get_by_name;
 	rte_eth_dev_is_rx_hairpin_queue;
 	rte_eth_dev_is_tx_hairpin_queue;
+	rte_eth_dev_is_used;
 	rte_eth_dev_probing_finish;
 	rte_eth_dev_release_port;
 	rte_eth_dev_internal_reset;
-- 
2.33.0

[PATCH v7 3/5] app/testpmd: check the validity of the port

[Huisong Li]

This patch checks the validity of port id for all events in
'eth_event_callback()'.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Aman Singh <aman.deep.singh@intel.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 9e4e99e53b..859d4aad7d 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3960,14 +3960,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 		fflush(stdout);
 	}
 
+	if (port_id_is_invalid(port_id, DISABLED_WARN))
+		return 0;
+
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
 		ports[port_id].need_setup = 1;
 		ports[port_id].port_status = RTE_PORT_HANDLING;
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
-		if (port_id_is_invalid(port_id, DISABLED_WARN))
-			break;
 		if (rte_eal_alarm_set(100000,
 				rmv_port_callback, (void *)(intptr_t)port_id))
 			fprintf(stderr,
-- 
2.33.0

[PATCH v7 4/5] app/testpmd: add attach and detach port for multiple process

[Huisong Li]

The port information needs to be updated due to attaching and detaching
port. Currently, it is done in the same thread as removing or probing
device, which doesn't satisfy the operation of attaching and detaching
device in multiple process.

If this operation is performed in one process, the other process can
receive 'new' or 'destroy' event. So we can move updating port information
to event callback to support attaching and detaching port in primary and
secondary process.

The reason for adding an alarm callback in 'destroy' event is that the
ethdev state is changed from 'ATTACHED' to 'UNUSED' only after the event
callback finished. But the remove_invalid_ports() function removes invalid
port only if ethdev state is 'UNUSED'. If we don't add alarm callback, this
detached port information can not be removed.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c                | 38 ++++++++++++++++-----------
 app/test-pmd/testpmd.h                |  1 -
 drivers/net/bonding/bonding_testpmd.c |  1 -
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 859d4aad7d..db985d33a3 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3625,15 +3625,12 @@ attach_port(char *identifier)
 		return;
 	}
 
-	/* first attach mode: event */
-	if (setup_on_probe_event) {
-		/* new ports are detected on RTE_ETH_EVENT_NEW event */
-		for (pi = 0; pi < RTE_MAX_ETHPORTS; pi++)
-			if (ports[pi].port_status == RTE_PORT_HANDLING &&
-					ports[pi].need_setup != 0)
-				setup_attached_port(pi);
+	/*
+	 * first attach mode: event, setting up attached port is done in
+	 * probing callback.
+	 */
+	if (setup_on_probe_event)
 		return;
-	}
 
 	/* second attach mode: iterator */
 	RTE_ETH_FOREACH_MATCHING_DEV(pi, identifier, &iterator) {
@@ -3664,7 +3661,6 @@ setup_attached_port(portid_t pi)
 	ports_ids[nb_ports++] = pi;
 	fwd_ports_ids[nb_fwd_ports++] = pi;
 	nb_cfg_ports = nb_fwd_ports;
-	ports[pi].need_setup = 0;
 	ports[pi].port_status = RTE_PORT_STOPPED;
 
 	printf("Port %d is attached. Now total ports is %d\n", pi, nb_ports);
@@ -3698,10 +3694,8 @@ detach_device(struct rte_device *dev)
 		TESTPMD_LOG(ERR, "Failed to detach device %s\n", rte_dev_name(dev));
 		return;
 	}
-	remove_invalid_ports();
 
 	printf("Device is detached\n");
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
 	return;
 }
@@ -3768,11 +3762,9 @@ detach_devargs(char *identifier)
 		return;
 	}
 
-	remove_invalid_ports();
-
 	printf("Device %s is detached\n", identifier);
-	printf("Now total ports is %d\n", nb_ports);
 	printf("Done\n");
+
 	rte_devargs_reset(&da);
 }
 
@@ -3936,11 +3928,22 @@ rmv_port_callback(void *arg)
 		struct rte_device *device = dev_info.device;
 		close_port(port_id);
 		detach_device(device); /* might be already removed or have more ports */
+		remove_invalid_ports();
+		printf("Now total ports is %d\n", nb_ports);
 	}
 	if (need_to_start)
 		start_packet_forwarding(0);
 }
 
+static void
+remove_invalid_ports_callback(void *arg)
+{
+	RTE_SET_USED(arg);
+
+	remove_invalid_ports();
+	printf("Now total ports is %d\n", nb_ports);
+}
+
 /* This function is used by the interrupt thread */
 static int
 eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
@@ -3965,8 +3968,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
-		ports[port_id].need_setup = 1;
-		ports[port_id].port_status = RTE_PORT_HANDLING;
+		if (setup_on_probe_event)
+			setup_attached_port(port_id);
 		break;
 	case RTE_ETH_EVENT_INTR_RMV:
 		if (rte_eal_alarm_set(100000,
@@ -3977,6 +3980,9 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 	case RTE_ETH_EVENT_DESTROY:
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
+		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
+			(void *)(intptr_t)port_id))
+			fprintf(stderr, "Could not set up deferred device released\n");
 		break;
 	case RTE_ETH_EVENT_RX_AVAIL_THRESH: {
 		uint16_t rxq_id;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 9b10a9ea1c..0e860c2e67 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -326,7 +326,6 @@ struct rte_port {
 	uint16_t                tx_vlan_id;/**< The tag ID */
 	uint16_t                tx_vlan_id_outer;/**< The outer tag ID */
 	volatile uint16_t        port_status;    /**< port started or not */
-	uint8_t                 need_setup;     /**< port just attached */
 	uint8_t                 need_reconfig;  /**< need reconfiguring port or not */
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
diff --git a/drivers/net/bonding/bonding_testpmd.c b/drivers/net/bonding/bonding_testpmd.c
index 8fcd6cadd0..f4845206ad 100644
--- a/drivers/net/bonding/bonding_testpmd.c
+++ b/drivers/net/bonding/bonding_testpmd.c
@@ -493,7 +493,6 @@ static void cmd_create_bonding_device_parsed(void *parsed_result,
 
 	ports[port_id].update_conf = 1;
 	ports[port_id].bond_flag = 1;
-	ports[port_id].need_setup = 0;
 	ports[port_id].port_status = RTE_PORT_STOPPED;
 }
 
-- 
2.33.0

[PATCH v7 5/5] app/testpmd: stop forwarding in new or destroy event

[Huisong Li]

When testpmd receives the new or destroy event, the port related
information will be updated. Testpmd must stop packet forwarding
before updating the information to avoid some serious problems.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 app/test-pmd/testpmd.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index db985d33a3..dfdd7e723d 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -3968,6 +3968,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 
 	switch (type) {
 	case RTE_ETH_EVENT_NEW:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		if (setup_on_probe_event)
 			setup_attached_port(port_id);
 		break;
@@ -3978,6 +3980,8 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param,
 				"Could not set up deferred device removal\n");
 		break;
 	case RTE_ETH_EVENT_DESTROY:
+		if (test_done == 0)
+			stop_packet_forwarding();
 		ports[port_id].port_status = RTE_PORT_CLOSED;
 		printf("Port %u is closed\n", port_id);
 		if (rte_eal_alarm_set(100000, remove_invalid_ports_callback,
-- 
2.33.0

Re: [PATCH v7 0/5] app/testpmd: support multiple process attach and detach port

[lihuisong (C)]

Hi Ferruh and Thomas,

Kindly ping for review.


在 2024/1/30 14:36, Huisong Li 写道:
> This patchset fix some bugs and support attaching and detaching port
> in primary and secondary.
>
> ---
>   -v7: fix conflicts
>   -v6: adjust rte_eth_dev_is_used position based on alphabetical order
>        in version.map
>   -v5: move 'ALLOCATED' state to the back of 'REMOVED' to avoid abi break.
>   -v4: fix a misspelling.
>   -v3:
>     #1 merge patch 1/6 and patch 2/6 into patch 1/5, and add modification
>        for other bus type.
>     #2 add a RTE_ETH_DEV_ALLOCATED state in rte_eth_dev_state to resolve
>        the probelm in patch 2/5.
>   -v2: resend due to CI unexplained failure.
>
> Huisong Li (5):
>    drivers/bus: restore driver assignment at front of probing
>    ethdev: fix skip valid port in probing callback
>    app/testpmd: check the validity of the port
>    app/testpmd: add attach and detach port for multiple process
>    app/testpmd: stop forwarding in new or destroy event
>
>   app/test-pmd/testpmd.c                   | 47 +++++++++++++++---------
>   app/test-pmd/testpmd.h                   |  1 -
>   drivers/bus/auxiliary/auxiliary_common.c |  9 ++++-
>   drivers/bus/dpaa/dpaa_bus.c              |  9 ++++-
>   drivers/bus/fslmc/fslmc_bus.c            |  8 +++-
>   drivers/bus/ifpga/ifpga_bus.c            | 12 ++++--
>   drivers/bus/pci/pci_common.c             |  9 ++++-
>   drivers/bus/vdev/vdev.c                  | 10 ++++-
>   drivers/bus/vmbus/vmbus_common.c         |  9 ++++-
>   drivers/net/bnxt/bnxt_ethdev.c           |  3 +-
>   drivers/net/bonding/bonding_testpmd.c    |  1 -
>   drivers/net/mlx5/mlx5.c                  |  2 +-
>   lib/ethdev/ethdev_driver.c               | 13 +++++--
>   lib/ethdev/ethdev_driver.h               | 12 ++++++
>   lib/ethdev/ethdev_pci.h                  |  2 +-
>   lib/ethdev/rte_class_eth.c               |  2 +-
>   lib/ethdev/rte_ethdev.c                  |  4 +-
>   lib/ethdev/rte_ethdev.h                  |  4 +-
>   lib/ethdev/version.map                   |  1 +
>   19 files changed, 114 insertions(+), 44 deletions(-)
>