DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
* [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
@ 2019-12-02 15:35 David Marchand
  2019-12-02 15:41 ` [dpdk-dev] [PATCH 1/4] eal/windows: fix cpuset macro name David Marchand
                   ` (6 more replies)
  0 siblings, 7 replies; 10+ messages in thread
From: David Marchand @ 2019-12-02 15:35 UTC (permalink / raw)
  To: dev; +Cc: aconole

We are currently stuck with no option but recompile a DPDK if the system
has more cores than RTE_MAX_LCORE.
A bit of a pity when you get a system with more than 200+ cores and your
testpmd has been built and packaged with RTE_MAX_LCORE == 128.

The --lcores does not need to care about the underlying cores, remove
this limitation.

-- 
David Marchand

David Marchand (4):
  eal/windows: fix cpuset macro name
  eal: do not cache lcore detection state
  eal: display all detected cores at startup
  eal: remove limitation on cpuset with --lcores

 drivers/bus/fslmc/portal/dpaa2_hw_dpio.c   | 31 +++++----
 lib/librte_eal/common/eal_common_lcore.c   | 12 +++-
 lib/librte_eal/common/eal_common_options.c | 73 +++++++++++-----------
 lib/librte_eal/common/eal_common_thread.c  |  4 +-
 lib/librte_eal/common/eal_private.h        |  1 -
 lib/librte_eal/windows/eal/include/sched.h |  8 +--
 6 files changed, 68 insertions(+), 61 deletions(-)

-- 
2.23.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH 1/4] eal/windows: fix cpuset macro name
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
@ 2019-12-02 15:41 ` David Marchand
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 2/4] eal: do not cache lcore detection state David Marchand
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2019-12-02 15:41 UTC (permalink / raw)
  To: dev
  Cc: aconole, stable, Harini Ramakrishnan, Omar Cardona, Anand Rawat,
	Ranjit Menon

Fix the name of CPU_SETSIZE in hope we can reuse it in other parts of
the dpdk manipulating some rte_cpuset_t.

Fixes: 4dc2b4d2a4cd ("eal/windows: add headers for compatibility")
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/librte_eal/windows/eal/include/sched.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/windows/eal/include/sched.h b/lib/librte_eal/windows/eal/include/sched.h
index 257060594c..29868c93d1 100644
--- a/lib/librte_eal/windows/eal/include/sched.h
+++ b/lib/librte_eal/windows/eal/include/sched.h
@@ -14,8 +14,8 @@
 extern "C" {
 #endif
 
-#ifndef CPU_SET_SIZE
-#define CPU_SET_SIZE RTE_MAX_LCORE
+#ifndef CPU_SETSIZE
+#define CPU_SETSIZE RTE_MAX_LCORE
 #endif
 
 #define _BITS_PER_SET (sizeof(long long) * 8)
@@ -26,7 +26,7 @@ extern "C" {
 #define _WHICH_BIT(b) ((b) & (_BITS_PER_SET - 1))
 
 typedef struct _rte_cpuset_s {
-	long long _bits[_NUM_SETS(CPU_SET_SIZE)];
+	long long _bits[_NUM_SETS(CPU_SETSIZE)];
 } rte_cpuset_t;
 
 #define CPU_SET(b, s) ((s)->_bits[_WHICH_SET(b)] |= (1LL << _WHICH_BIT(b)))
@@ -35,7 +35,7 @@ typedef struct _rte_cpuset_s {
 	do {								\
 		unsigned int _i;					\
 									\
-		for (_i = 0; _i < _NUM_SETS(CPU_SET_SIZE); _i++)	\
+		for (_i = 0; _i < _NUM_SETS(CPU_SETSIZE); _i++)		\
 			(s)->_bits[_i] = 0LL;				\
 	} while (0)
 
-- 
2.23.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH 2/4] eal: do not cache lcore detection state
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
  2019-12-02 15:41 ` [dpdk-dev] [PATCH 1/4] eal/windows: fix cpuset macro name David Marchand
@ 2019-12-02 15:42 ` David Marchand
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 3/4] eal: display all detected cores at startup David Marchand
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2019-12-02 15:42 UTC (permalink / raw)
  To: dev; +Cc: aconole

We use this state in control path only for services cores and -c/-l
options.
The value is not updated when using --lcores.

Use the internal helper where needed.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/librte_eal/common/eal_common_lcore.c   |  4 +---
 lib/librte_eal/common/eal_common_options.c | 10 +++++-----
 lib/librte_eal/common/eal_private.h        |  1 -
 3 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index 39efadef1a..1d16fb2156 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -139,9 +139,7 @@ rte_eal_cpu_init(void)
 		socket_id = eal_cpu_socket_id(lcore_id);
 		lcore_to_socket_id[lcore_id] = socket_id;
 
-		/* in 1:1 mapping, record related cpu detected state */
-		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
-		if (lcore_config[lcore_id].detected == 0) {
+		if (eal_cpu_detected(lcore_id) == 0) {
 			config->lcore_role[lcore_id] = ROLE_OFF;
 			lcore_config[lcore_id].core_index = -1;
 			continue;
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index a7f9c5f9bd..68f7d1cd73 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -373,7 +373,7 @@ eal_parse_service_coremask(const char *coremask)
 					return -1;
 				}
 
-				if (!lcore_config[idx].detected) {
+				if (eal_cpu_detected(idx) == 0) {
 					RTE_LOG(ERR, EAL,
 						"lcore %u unavailable\n", idx);
 					return -1;
@@ -429,7 +429,7 @@ update_lcore_config(int *cores)
 
 	for (i = 0; i < RTE_MAX_LCORE; i++) {
 		if (cores[i] != -1) {
-			if (!lcore_config[i].detected) {
+			if (eal_cpu_detected(i) == 0) {
 				RTE_LOG(ERR, EAL, "lcore %u unavailable\n", i);
 				ret = -1;
 				continue;
@@ -785,7 +785,7 @@ convert_to_cpuset(rte_cpuset_t *cpusetp,
 		if (!set[idx])
 			continue;
 
-		if (!lcore_config[idx].detected) {
+		if (eal_cpu_detected(idx) == 0) {
 			RTE_LOG(ERR, EAL, "core %u "
 				"unavailable\n", idx);
 			return -1;
@@ -1138,7 +1138,7 @@ available_cores(void)
 
 	/* find the first available cpu */
 	for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
-		if (!lcore_config[idx].detected)
+		if (eal_cpu_detected(idx) == 0)
 			continue;
 		break;
 	}
@@ -1152,7 +1152,7 @@ available_cores(void)
 	sequence = 0;
 
 	for (idx++ ; idx < RTE_MAX_LCORE; idx++) {
-		if (!lcore_config[idx].detected)
+		if (eal_cpu_detected(idx) == 0)
 			continue;
 
 		if (idx == previous + 1) {
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 8a9d493f0c..ddcfbe2e44 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -29,7 +29,6 @@ struct lcore_config {
 	unsigned int core_id;      /**< core number on socket for this lcore */
 	int core_index;            /**< relative index, starting from 0 */
 	uint8_t core_role;         /**< role of core eg: OFF, RTE, SERVICE */
-	uint8_t detected;          /**< true if lcore was detected */
 
 	rte_cpuset_t cpuset;       /**< cpu set which the lcore affinity to */
 };
-- 
2.23.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH 3/4] eal: display all detected cores at startup
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
  2019-12-02 15:41 ` [dpdk-dev] [PATCH 1/4] eal/windows: fix cpuset macro name David Marchand
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 2/4] eal: do not cache lcore detection state David Marchand
@ 2019-12-02 15:42 ` David Marchand
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 4/4] eal: remove limitation on cpuset with --lcores David Marchand
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2019-12-02 15:42 UTC (permalink / raw)
  To: dev; +Cc: aconole

Add debug logs to have a trace of unused cores for -c/-l options on
systems with more cores than RTE_MAX_LCORE.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/librte_eal/common/eal_common_lcore.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index 1d16fb2156..5404922a87 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -159,6 +159,14 @@ rte_eal_cpu_init(void)
 				lcore_config[lcore_id].socket_id);
 		count++;
 	}
+	for (; lcore_id < CPU_SETSIZE; lcore_id++) {
+		if (eal_cpu_detected(lcore_id) == 0)
+			continue;
+		RTE_LOG(DEBUG, EAL, "Skipped lcore %u as core %u on socket %u\n",
+			lcore_id, eal_cpu_core_id(lcore_id),
+			eal_cpu_socket_id(lcore_id));
+	}
+
 	/* Set the count of enabled logical cores of the EAL configuration */
 	config->lcore_count = count;
 	RTE_LOG(DEBUG, EAL,
-- 
2.23.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH 4/4] eal: remove limitation on cpuset with --lcores
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
                   ` (2 preceding siblings ...)
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 3/4] eal: display all detected cores at startup David Marchand
@ 2019-12-02 15:42 ` David Marchand
  2020-01-14 12:58 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2019-12-02 15:42 UTC (permalink / raw)
  To: dev; +Cc: aconole, Hemant Agrawal, Sachin Saxena

Contrary to the -c/-l options, where a logical core runs on the same
physical core in a 1:1 fashion (example: lcore 0 runs on core 0, lcore
16 runs on core 16), the --lcores option makes it possible to select the
physical cores on which runs a logical core.

However the current parsing code still limits the cpuset to the
[0, RTE_MAX_LCORE] range.

Example, before the patch, on a 24 cores system with RTE_MAX_LCORE == 16:
$ ./master/app/testpmd --no-huge --no-pci -m 512 --log-level *:debug \
 --lcores 0@16,1@17 -- -i --total-num-mbufs 2048
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 8 on socket 0
EAL: Detected lcore 8 as core 9 on socket 0
EAL: Detected lcore 9 as core 10 on socket 0
EAL: Detected lcore 10 as core 11 on socket 0
EAL: Detected lcore 11 as core 12 on socket 0
EAL: Detected lcore 12 as core 13 on socket 0
EAL: Detected lcore 13 as core 14 on socket 0
EAL: Detected lcore 14 as core 0 on socket 0
EAL: Detected lcore 15 as core 1 on socket 0
EAL: Skipped lcore 16 as core 2 on socket 0
EAL: Skipped lcore 17 as core 3 on socket 0
EAL: Skipped lcore 18 as core 4 on socket 0
EAL: Skipped lcore 19 as core 5 on socket 0
EAL: Skipped lcore 20 as core 6 on socket 0
EAL: Skipped lcore 21 as core 8 on socket 0
EAL: Skipped lcore 22 as core 9 on socket 0
EAL: Skipped lcore 23 as core 10 on socket 0
EAL: Skipped lcore 24 as core 11 on socket 0
EAL: Skipped lcore 25 as core 12 on socket 0
EAL: Skipped lcore 26 as core 13 on socket 0
EAL: Skipped lcore 27 as core 14 on socket 0
EAL: Support maximum 16 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: invalid parameter for --lcores

We can remove this limitation by using a cpuset_t (which is a more
natural type since this is what gets passed to pthread_setaffinity*
in the end).

After the patch:
$ ./master/app/testpmd --no-huge --no-pci -m 512 --log-level *:debug \
 --lcores 0@16,1@17 -- -i --total-num-mbufs 2048
[...]
EAL: Master lcore 0 is ready (tid=7f94217bbc00;cpuset=[16])
EAL: lcore 1 is ready (tid=7f941f491700;cpuset=[17])

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 drivers/bus/fslmc/portal/dpaa2_hw_dpio.c   | 31 ++++++-----
 lib/librte_eal/common/eal_common_options.c | 63 +++++++++++-----------
 lib/librte_eal/common/eal_common_thread.c  |  4 +-
 3 files changed, 50 insertions(+), 48 deletions(-)

diff --git a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c b/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c
index 3ca3ae4f51..739ce434ba 100644
--- a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c
+++ b/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c
@@ -365,20 +365,25 @@ dpaa2_check_lcore_cpuset(void)
 		dpaa2_cpu[lcore_id] = 0xffffffff;
 
 	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
-		for (i = 0; i < RTE_MAX_LCORE; i++) {
-			rte_cpuset_t cpuset = rte_lcore_cpuset(lcore_id);
-
-			if (CPU_ISSET(i, &cpuset)) {
-				RTE_LOG(DEBUG, EAL, "lcore id = %u cpu=%u\n",
-					lcore_id, i);
-				if (dpaa2_cpu[lcore_id] != 0xffffffff) {
-					DPAA2_BUS_ERR(
-				    "ERR:lcore map to multi-cpu not supported");
-					ret = -1;
-				} else  {
-					dpaa2_cpu[lcore_id] = i;
-				}
+		rte_cpuset_t cpuset = rte_lcore_cpuset(lcore_id);
+
+		for (i = 0; i < CPU_SETSIZE; i++) {
+			if (!CPU_ISSET(i, &cpuset))
+				continue;
+			if (i >= RTE_MAX_LCORE) {
+				DPAA2_BUS_ERR("ERR:lcore map to core %u (>= %u) not supported",
+					i, RTE_MAX_LCORE);
+				ret = -1;
+				continue;
 			}
+			RTE_LOG(DEBUG, EAL, "lcore id = %u cpu=%u\n",
+				lcore_id, i);
+			if (dpaa2_cpu[lcore_id] != 0xffffffff) {
+				DPAA2_BUS_ERR("ERR:lcore map to multi-cpu not supported");
+				ret = -1;
+				continue;
+			}
+			dpaa2_cpu[lcore_id] = i;
 		}
 	}
 	return ret;
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 68f7d1cd73..5920233bcd 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -658,14 +658,14 @@ eal_parse_master_lcore(const char *arg)
  *                       ',' used for a single number.
  */
 static int
-eal_parse_set(const char *input, uint16_t set[], unsigned num)
+eal_parse_set(const char *input, rte_cpuset_t *set)
 {
 	unsigned idx;
 	const char *str = input;
 	char *end = NULL;
 	unsigned min, max;
 
-	memset(set, 0, num * sizeof(uint16_t));
+	CPU_ZERO(set);
 
 	while (isblank(*str))
 		str++;
@@ -678,7 +678,7 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 	if (*str != '(') {
 		errno = 0;
 		idx = strtoul(str, &end, 10);
-		if (errno || end == NULL || idx >= num)
+		if (errno || end == NULL || idx >= CPU_SETSIZE)
 			return -1;
 		else {
 			while (isblank(*end))
@@ -696,7 +696,7 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 
 				errno = 0;
 				idx = strtoul(end, &end, 10);
-				if (errno || end == NULL || idx >= num)
+				if (errno || end == NULL || idx >= CPU_SETSIZE)
 					return -1;
 				max = idx;
 				while (isblank(*end))
@@ -711,7 +711,7 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 
 			for (idx = RTE_MIN(min, max);
 			     idx <= RTE_MAX(min, max); idx++)
-				set[idx] = 1;
+				CPU_SET(idx, set);
 
 			return end - input;
 		}
@@ -736,7 +736,7 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 		/* get the digit value */
 		errno = 0;
 		idx = strtoul(str, &end, 10);
-		if (errno || end == NULL || idx >= num)
+		if (errno || end == NULL || idx >= CPU_SETSIZE)
 			return -1;
 
 		/* go ahead to separator '-',',' and ')' */
@@ -753,7 +753,7 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 				min = idx;
 			for (idx = RTE_MIN(min, max);
 			     idx <= RTE_MAX(min, max); idx++)
-				set[idx] = 1;
+				CPU_SET(idx, set);
 
 			min = RTE_MAX_LCORE;
 		} else
@@ -772,17 +772,13 @@ eal_parse_set(const char *input, uint16_t set[], unsigned num)
 	return str - input;
 }
 
-/* convert from set array to cpuset bitmap */
 static int
-convert_to_cpuset(rte_cpuset_t *cpusetp,
-	      uint16_t *set, unsigned num)
+check_cpuset(rte_cpuset_t *set)
 {
-	unsigned idx;
-
-	CPU_ZERO(cpusetp);
+	unsigned int idx;
 
-	for (idx = 0; idx < num; idx++) {
-		if (!set[idx])
+	for (idx = 0; idx < CPU_SETSIZE; idx++) {
+		if (!CPU_ISSET(idx, set))
 			continue;
 
 		if (eal_cpu_detected(idx) == 0) {
@@ -790,10 +786,7 @@ convert_to_cpuset(rte_cpuset_t *cpusetp,
 				"unavailable\n", idx);
 			return -1;
 		}
-
-		CPU_SET(idx, cpusetp);
 	}
-
 	return 0;
 }
 
@@ -815,7 +808,8 @@ static int
 eal_parse_lcores(const char *lcores)
 {
 	struct rte_config *cfg = rte_eal_get_configuration();
-	static uint16_t set[RTE_MAX_LCORE];
+	rte_cpuset_t lcore_set;
+	unsigned int set_count;
 	unsigned idx = 0;
 	unsigned count = 0;
 	const char *lcore_start = NULL;
@@ -864,18 +858,13 @@ eal_parse_lcores(const char *lcores)
 		lcores += strcspn(lcores, "@,");
 
 		if (*lcores == '@') {
-			/* explicit assign cpu_set */
-			offset = eal_parse_set(lcores + 1, set, RTE_DIM(set));
+			/* explicit assign cpuset and update the end cursor */
+			offset = eal_parse_set(lcores + 1, &cpuset);
 			if (offset < 0)
 				goto err;
-
-			/* prepare cpu_set and update the end cursor */
-			if (0 > convert_to_cpuset(&cpuset,
-						  set, RTE_DIM(set)))
-				goto err;
 			end = lcores + 1 + offset;
 		} else { /* ',' or '\0' */
-			/* haven't given cpu_set, current loop done */
+			/* haven't given cpuset, current loop done */
 			end = lcores;
 
 			/* go back to check <number>-<number> */
@@ -889,18 +878,19 @@ eal_parse_lcores(const char *lcores)
 			goto err;
 
 		/* parse lcore_set from start point */
-		if (0 > eal_parse_set(lcore_start, set, RTE_DIM(set)))
+		if (eal_parse_set(lcore_start, &lcore_set) < 0)
 			goto err;
 
-		/* without '@', by default using lcore_set as cpu_set */
-		if (*lcores != '@' &&
-		    0 > convert_to_cpuset(&cpuset, set, RTE_DIM(set)))
-			goto err;
+		/* without '@', by default using lcore_set as cpuset */
+		if (*lcores != '@')
+			rte_memcpy(&cpuset, &lcore_set, sizeof(cpuset));
 
+		set_count = CPU_COUNT(&lcore_set);
 		/* start to update lcore_set */
 		for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
-			if (!set[idx])
+			if (!CPU_ISSET(idx, &lcore_set))
 				continue;
+			set_count--;
 
 			if (cfg->lcore_role[idx] != ROLE_RTE) {
 				lcore_config[idx].core_index = count;
@@ -912,10 +902,17 @@ eal_parse_lcores(const char *lcores)
 				CPU_ZERO(&cpuset);
 				CPU_SET(idx, &cpuset);
 			}
+
+			if (check_cpuset(&cpuset) < 0)
+				goto err;
 			rte_memcpy(&lcore_config[idx].cpuset, &cpuset,
 				   sizeof(rte_cpuset_t));
 		}
 
+		/* some cores from the lcore_set can't be handled by EAL */
+		if (set_count != 0)
+			goto err;
+
 		lcores = end + 1;
 	} while (*end != '\0');
 
diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
index f9a8cf14d2..78581753c0 100644
--- a/lib/librte_eal/common/eal_common_thread.c
+++ b/lib/librte_eal/common/eal_common_thread.c
@@ -61,7 +61,7 @@ eal_cpuset_socket_id(rte_cpuset_t *cpusetp)
 			break;
 		}
 
-	} while (++cpu < RTE_MAX_LCORE);
+	} while (++cpu < CPU_SETSIZE);
 
 	return socket_id;
 }
@@ -118,7 +118,7 @@ eal_thread_dump_affinity(char *str, unsigned size)
 
 	rte_thread_get_affinity(&cpuset);
 
-	for (cpu = 0; cpu < RTE_MAX_LCORE; cpu++) {
+	for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
 		if (!CPU_ISSET(cpu, &cpuset))
 			continue;
 
-- 
2.23.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
                   ` (3 preceding siblings ...)
  2019-12-02 15:42 ` [dpdk-dev] [PATCH 4/4] eal: remove limitation on cpuset with --lcores David Marchand
@ 2020-01-14 12:58 ` David Marchand
  2020-01-14 15:32   ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores >RTE_MAX_LCORE Morten Brørup
  2020-01-20 18:37 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE Yigit, Ferruh
  2020-01-21  0:24 ` Thomas Monjalon
  6 siblings, 1 reply; 10+ messages in thread
From: David Marchand @ 2020-01-14 12:58 UTC (permalink / raw)
  To: dev; +Cc: Aaron Conole

On Mon, Dec 2, 2019 at 4:36 PM David Marchand <david.marchand@redhat.com> wrote:
>
> We are currently stuck with no option but recompile a DPDK if the system
> has more cores than RTE_MAX_LCORE.
> A bit of a pity when you get a system with more than 200+ cores and your
> testpmd has been built and packaged with RTE_MAX_LCORE == 128.
>
> The --lcores does not need to care about the underlying cores, remove
> this limitation.

Any comment?
Thanks.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores >RTE_MAX_LCORE
  2020-01-14 12:58 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
@ 2020-01-14 15:32   ` Morten Brørup
  0 siblings, 0 replies; 10+ messages in thread
From: Morten Brørup @ 2020-01-14 15:32 UTC (permalink / raw)
  To: David Marchand, dev, Aaron Conole; +Cc: Olivier Matz, Andrew Rybchenko

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Marchand
> Sent: Tuesday, January 14, 2020 1:59 PM
> 
> On Mon, Dec 2, 2019 at 4:36 PM David Marchand
> <david.marchand@redhat.com> wrote:
> >
> > We are currently stuck with no option but recompile a DPDK if the
> system
> > has more cores than RTE_MAX_LCORE.
> > A bit of a pity when you get a system with more than 200+ cores and
> your
> > testpmd has been built and packaged with RTE_MAX_LCORE == 128.
> >
> > The --lcores does not need to care about the underlying cores, remove
> > this limitation.
> 
> Any comment?
> Thanks.
> 
> 
> --
> David Marchand
> 

I haven't reviewed the patch, but recall that the Mempool library sets up caches per lcore using hardcoded loops going from 0 to RTE_MAX_LCORE, regardless how many lcores are present and assigned to DPDK.

Making rte_max_lcore dynamic (and possibly auto-detected by the EAL) instead of defined at compile time might also be an improvement for systems with fewer cores than RTE_MAX_LCORE. But there are probably a lot of considerations regarding multi-process applications.


Med venlig hilsen / kind regards
- Morten Brørup

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
                   ` (4 preceding siblings ...)
  2020-01-14 12:58 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
@ 2020-01-20 18:37 ` Yigit, Ferruh
  2020-01-20 19:35   ` David Marchand
  2020-01-21  0:24 ` Thomas Monjalon
  6 siblings, 1 reply; 10+ messages in thread
From: Yigit, Ferruh @ 2020-01-20 18:37 UTC (permalink / raw)
  To: David Marchand, dev; +Cc: aconole

On 12/2/2019 3:35 PM, David Marchand wrote:
> We are currently stuck with no option but recompile a DPDK if the system
> has more cores than RTE_MAX_LCORE.
> A bit of a pity when you get a system with more than 200+ cores and your
> testpmd has been built and packaged with RTE_MAX_LCORE == 128.

Just for clarification, in this case the DPDK application will work but will
only use RTE_MAX_LCORE cores, right? You need to compile to use all available cores.

Are cores more than RTE_MAX_LCORE usable after this patch?

> 
> The --lcores does not need to care about the underlying cores, remove
> this limitation.
> 

+1 to remove limit in --lcores, but I didn't make it work with this patchset, I
may be missing something, I tried by setting the "RTE_MAX_LCORE=32", in this
case should '--lcores "(30-40)@(10-20)"' work?



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
  2020-01-20 18:37 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE Yigit, Ferruh
@ 2020-01-20 19:35   ` David Marchand
  0 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2020-01-20 19:35 UTC (permalink / raw)
  To: Yigit, Ferruh; +Cc: dev, Aaron Conole

On Mon, Jan 20, 2020 at 7:37 PM Yigit, Ferruh
<ferruh.yigit@linux.intel.com> wrote:
>
> On 12/2/2019 3:35 PM, David Marchand wrote:
> > We are currently stuck with no option but recompile a DPDK if the system
> > has more cores than RTE_MAX_LCORE.
> > A bit of a pity when you get a system with more than 200+ cores and your
> > testpmd has been built and packaged with RTE_MAX_LCORE == 128.
>
> Just for clarification, in this case the DPDK application will work but will
> only use RTE_MAX_LCORE cores, right? You need to compile to use all available cores.

EAL starts up to RTE_MAX_LCORE lcores (as in EAL threads), that's true
before and after this patch.

By default (using the -c or -l options), a lcore X runs on the physical core X.
With the --lcores option, you can select on which physical cores a lcore runs.

But, before this series, the code limits the physical cores to the
same [0, RTE_MAX_LCORE[ range.

>
> Are cores more than RTE_MAX_LCORE usable after this patch?

Yes, see below.


>
> >
> > The --lcores does not need to care about the underlying cores, remove
> > this limitation.
> >
>
> +1 to remove limit in --lcores, but I didn't make it work with this patchset, I
> may be missing something, I tried by setting the "RTE_MAX_LCORE=32", in this
> case should '--lcores "(30-40)@(10-20)"' work?

Quoting the parser code:
 * The format pattern: --lcores='<lcores[@cpus]>[<,lcores[@cpus]>...]'
 * lcores, cpus could be a single digit/range or a group.
 * '(' and ')' are necessary if it's a group.
 * If not supply '@cpus', the value of cpus uses the same as lcores.
 * e.g. '1,2@(5-7),(3-5)@(0,2),(0,6),7-8' means start 9 EAL thread as below
 *   lcore 0 runs on cpuset 0x41 (cpu 0,6)
 *   lcore 1 runs on cpuset 0x2 (cpu 1)
 *   lcore 2 runs on cpuset 0xe0 (cpu 5,6,7)
 *   lcore 3,4,5 runs on cpuset 0x5 (cpu 0,2)
 *   lcore 6 runs on cpuset 0x41 (cpu 0,6)
 *   lcore 7 runs on cpuset 0x80 (cpu 7)
 *   lcore 8 runs on cpuset 0x100 (cpu 8)

With --lcores "(30-40)@(10-20)", you asked to start lcores 30 to 40 to
run on the cpuset 10 to 20.
So it will be refused, since the max lcore is 32.

If your intention was to start lcore 10 on physical core 30, lcore 11
on core 31 etc... then you can try --lcores 10@30,11@31,12@32...


-- 
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE
  2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
                   ` (5 preceding siblings ...)
  2020-01-20 18:37 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE Yigit, Ferruh
@ 2020-01-21  0:24 ` Thomas Monjalon
  6 siblings, 0 replies; 10+ messages in thread
From: Thomas Monjalon @ 2020-01-21  0:24 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, aconole, ferruh.yigit

02/12/2019 16:35, David Marchand:
> We are currently stuck with no option but recompile a DPDK if the system
> has more cores than RTE_MAX_LCORE.
> A bit of a pity when you get a system with more than 200+ cores and your
> testpmd has been built and packaged with RTE_MAX_LCORE == 128.
> 
> The --lcores does not need to care about the underlying cores, remove
> this limitation.

> David Marchand (4):
>   eal/windows: fix cpuset macro name
>   eal: do not cache lcore detection state
>   eal: display all detected cores at startup
>   eal: remove limitation on cpuset with --lcores

The patches look good but it is very hard to review parsing code (last patch).
We will better experience corner cases after merging.

Applied for -rc1, thanks





^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-02 15:35 [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
2019-12-02 15:41 ` [dpdk-dev] [PATCH 1/4] eal/windows: fix cpuset macro name David Marchand
2019-12-02 15:42 ` [dpdk-dev] [PATCH 2/4] eal: do not cache lcore detection state David Marchand
2019-12-02 15:42 ` [dpdk-dev] [PATCH 3/4] eal: display all detected cores at startup David Marchand
2019-12-02 15:42 ` [dpdk-dev] [PATCH 4/4] eal: remove limitation on cpuset with --lcores David Marchand
2020-01-14 12:58 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE David Marchand
2020-01-14 15:32   ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores >RTE_MAX_LCORE Morten Brørup
2020-01-20 18:37 ` [dpdk-dev] [PATCH 0/4] Extend --lcores to run on cores > RTE_MAX_LCORE Yigit, Ferruh
2020-01-20 19:35   ` David Marchand
2020-01-21  0:24 ` Thomas Monjalon

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git