[PATCH net-next] net: set default rss queues num to physical cores / 2

[PATCH net-next] net: set default rss queues num to physical cores / 2

[Íñigo Huguet]

Network drivers can call to netif_get_num_default_rss_queues to get the
default number of receive queues to use. Right now, this default number
is min(8, num_online_cpus()).

Instead, as suggested by Jakub, use the number of physical cores divided
by 2 as a way to avoid wasting CPU resources and to avoid using both CPU
threads, but still allowing to scale for high-end processors with many
cores.

As an exception, select 2 queues for processors with 2 cores, because
otherwise it won't take any advantage of RSS despite being SMP capable.

Tested: Processor Intel Xeon E5-2620 (2 sockets, 6 cores/socket, 2
threads/core). NIC Broadcom NetXtreme II BCM57810 (10GBps). Ran some
tests with `perf stat iperf3 -R`, with parallelisms of 1, 8 and 24,
getting the following results:
- Number of queues: 6 (instead of 8)
- Network throughput: not affected
- CPU usage: utilized 0.05-0.12 CPUs more than before (having 24 CPUs
  this is only 0.2-0.5% higher)
- Reduced the number of context switches by 7-50%, being more noticeable
  when using a higher number of parallel threads.

Suggested-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Íñigo Huguet <ihuguet@redhat.com>
---
 include/linux/netdevice.h |  1 -
 net/core/dev.c            | 20 ++++++++++++++++----
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0d994710b335..db9874ed79d9 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3664,7 +3664,6 @@ static inline unsigned int get_netdev_rx_queue_index(
 }
 #endif
 
-#define DEFAULT_MAX_NUM_RSS_QUEUES	(8)
 int netif_get_num_default_rss_queues(void);
 
 enum skb_free_reason {
diff --git a/net/core/dev.c b/net/core/dev.c
index 75bab5b0dbae..8e0cc5f2020d 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2990,13 +2990,25 @@ EXPORT_SYMBOL(netif_set_real_num_queues);
 /**
  * netif_get_num_default_rss_queues - default number of RSS queues
  *
- * This routine should set an upper limit on the number of RSS queues
- * used by default by multiqueue devices.
+ * Default value is the number of physical cores if there are only 1 or 2, or
+ * divided by 2 if there are more.
  */
 int netif_get_num_default_rss_queues(void)
 {
-	return is_kdump_kernel() ?
-		1 : min_t(int, DEFAULT_MAX_NUM_RSS_QUEUES, num_online_cpus());
+	cpumask_var_t cpus;
+	int cpu, count = 0;
+
+	if (unlikely(is_kdump_kernel() || !zalloc_cpumask_var(&cpus, GFP_KERNEL)))
+		return 1;
+
+	cpumask_copy(cpus, cpu_online_mask);
+	for_each_cpu(cpu, cpus) {
+		++count;
+		cpumask_andnot(cpus, cpus, topology_sibling_cpumask(cpu));
+	}
+	free_cpumask_var(cpus);
+
+	return count > 2 ? DIV_ROUND_UP(count, 2) : count;
 }
 EXPORT_SYMBOL(netif_get_num_default_rss_queues);
 
-- 
2.34.1

Re: [PATCH net-next] net: set default rss queues num to physical cores / 2

[Jakub Kicinski]

On Tue, 15 Mar 2022 10:18:32 +0100 Íñigo Huguet wrote:
> Network drivers can call to netif_get_num_default_rss_queues to get the
> default number of receive queues to use. Right now, this default number
> is min(8, num_online_cpus()).
> 
> Instead, as suggested by Jakub, use the number of physical cores divided
> by 2 as a way to avoid wasting CPU resources and to avoid using both CPU
> threads, but still allowing to scale for high-end processors with many
> cores.
> 
> As an exception, select 2 queues for processors with 2 cores, because
> otherwise it won't take any advantage of RSS despite being SMP capable.
> 
> Tested: Processor Intel Xeon E5-2620 (2 sockets, 6 cores/socket, 2
> threads/core). NIC Broadcom NetXtreme II BCM57810 (10GBps). Ran some
> tests with `perf stat iperf3 -R`, with parallelisms of 1, 8 and 24,
> getting the following results:
> - Number of queues: 6 (instead of 8)
> - Network throughput: not affected
> - CPU usage: utilized 0.05-0.12 CPUs more than before (having 24 CPUs
>   this is only 0.2-0.5% higher)
> - Reduced the number of context switches by 7-50%, being more noticeable
>   when using a higher number of parallel threads.

Thanks for following up, Inigo!

Heads up for the maintainers of drivers which use
netif_get_num_default_rss_queues() today - please note the above - 
the default number of Rx queues may change for you starting with 
the 5.18 kernel.

Re: [PATCH net-next] net: set default rss queues num to physical cores / 2

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (master)
by Jakub Kicinski <kuba@kernel.org>:

On Tue, 15 Mar 2022 10:18:32 +0100 you wrote:
> Network drivers can call to netif_get_num_default_rss_queues to get the
> default number of receive queues to use. Right now, this default number
> is min(8, num_online_cpus()).
> 
> Instead, as suggested by Jakub, use the number of physical cores divided
> by 2 as a way to avoid wasting CPU resources and to avoid using both CPU
> threads, but still allowing to scale for high-end processors with many
> cores.
> 
> [...]

Here is the summary with links:
  - [net-next] net: set default rss queues num to physical cores / 2
    https://git.kernel.org/netdev/net-next/c/046e1537a3cf

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html