[RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

The RPS code and "deferred skb free" both send IPI/ function call
to a remote CPU in which a softirq is raised. This leads to a warning on
PREEMPT_RT because raising softiqrs from function call led to undesired
behaviour in the past. I had duct tape in RT for the "deferred skb free"
and Wander Lairson Costa reported the RPS case.

Patch #1 creates per-CPU threads for the backlog NAPI. It follows the
	 threaded NAPI model and solves the issue and simplifies the
         code.
Patch #2 gets rid of the warning. Since the ksoftirqd changes the
         situtation isn't as bad as it was. Still, it would be better to
	 keep it in the context where it originated.

Sebastian

[RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

Backlog NAPI is a per-CPU NAPI struct only (with no device behind it)
used by drivers which don't do NAPI them self and RPS.
The non-NAPI driver use the CPU local backlog NAPI. If RPS is enabled
then a flow for the skb is computed and based on the flow the skb can be
enqueued on a remote CPU. Scheduling/ raising the softirq (for backlog's
NAPI) on the remote CPU isn't trivial because the softirq is only
scheduled on the local CPU and performed after the hardirq is done.
In order to schedule a softirq on the remote CPU, an IPI is sent to the
remote CPU which schedules the backlog-NAPI on the then local CPU.

On PREEMPT_RT interrupts are force-threaded. The soft interrupts are
raised within the interrupt thread and processed after the interrupt
handler completed still within the context of the interrupt thread. The
softirq is handled in the context where it originated.

With force-threaded interrupts enabled, ksoftirqd is woken up if a
softirq is raised from hardirq context. This is the case if it is raised
from an IPI. Additionally there is a warning on PREEMPT_RT if the
softirq is raised from the idle thread.
This was done for two reasons:
- With threaded interrupts the processing should happen in thread
  context (where it originated) and ksoftirqd is the only thread for
  this context if raised from hardirq. Using the currently running task
  instead would "punish" a random task.
- Once ksoftirqd is active it consumes all further softirqs until it
  stops running. This changed recently and is no longer the case.

Instead of keeping the backlog NAPI in ksoftirqd (in force-threaded/
PREEMPT_RT setups) I am proposing NAPI-threads for backlog.
The "proper" setup with threaded-NAPI is not doable because the threads
are not pinned to an individual CPU and can be modified by the user.
Additionally a dummy network device would have to be assigned. Also
CPU-hotplug has to be considered if additional CPUs show up.
All this can be probably done/ solved but the smpboot-threads already
provide this infrastructure.

Create NAPI-threads for backlog. The thread runs the inner loop from
napi_threaded_poll(), the wait part is different.
Since there are now per-CPU threads for backlog the remote IPI for
signaling is not needed and can be removed. The NAPI for backlog can
always be scheduled as it ends in waking the corresponding thread.
Since "deferred skb free" use a similar IPI mechanism for signaling, it
is also using the backlog threads.

This makes NAPI threads mandatory for backlog and it can not be
disabled. The other visibile part with RPS (or backlog usage in general)
is that it becomes now visible in `top' while earlier it would remain
unaccounted.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/netdevice.h |   8 --
 net/core/dev.c            | 226 +++++++++++++-------------------------
 net/core/net-procfs.c     |   2 +-
 net/core/skbuff.c         |   2 +-
 4 files changed, 79 insertions(+), 159 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0896aaa91dd7b..17e31a68e725e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3190,9 +3190,6 @@ struct softnet_data {
 	/* stats */
 	unsigned int		processed;
 	unsigned int		time_squeeze;
-#ifdef CONFIG_RPS
-	struct softnet_data	*rps_ipi_list;
-#endif
 
 	bool			in_net_rx_action;
 	bool			in_napi_threaded_poll;
@@ -3221,12 +3218,8 @@ struct softnet_data {
 	unsigned int		input_queue_head ____cacheline_aligned_in_smp;
 
 	/* Elements below can be accessed between CPUs for RPS/RFS */
-	call_single_data_t	csd ____cacheline_aligned_in_smp;
-	struct softnet_data	*rps_ipi_next;
-	unsigned int		cpu;
 	unsigned int		input_queue_tail;
 #endif
-	unsigned int		received_rps;
 	unsigned int		dropped;
 	struct sk_buff_head	input_pkt_queue;
 	struct napi_struct	backlog;
@@ -3236,7 +3229,6 @@ struct softnet_data {
 	int			defer_count;
 	int			defer_ipi_scheduled;
 	struct sk_buff		*defer_list;
-	call_single_data_t	defer_csd;
 };
 
 static inline void input_queue_head_incr(struct softnet_data *sd)
diff --git a/net/core/dev.c b/net/core/dev.c
index 636b41f0b32d6..40103238ac0a1 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -153,6 +153,7 @@
 #include <linux/prandom.h>
 #include <linux/once_lite.h>
 #include <net/netdev_rx_queue.h>
+#include <linux/smpboot.h>
 
 #include "dev.h"
 #include "net-sysfs.h"
@@ -4658,57 +4659,8 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
 EXPORT_SYMBOL(rps_may_expire_flow);
 
 #endif /* CONFIG_RFS_ACCEL */
-
-/* Called from hardirq (IPI) context */
-static void rps_trigger_softirq(void *data)
-{
-	struct softnet_data *sd = data;
-
-	____napi_schedule(sd, &sd->backlog);
-	sd->received_rps++;
-}
-
 #endif /* CONFIG_RPS */
 
-/* Called from hardirq (IPI) context */
-static void trigger_rx_softirq(void *data)
-{
-	struct softnet_data *sd = data;
-
-	__raise_softirq_irqoff(NET_RX_SOFTIRQ);
-	smp_store_release(&sd->defer_ipi_scheduled, 0);
-}
-
-/*
- * After we queued a packet into sd->input_pkt_queue,
- * we need to make sure this queue is serviced soon.
- *
- * - If this is another cpu queue, link it to our rps_ipi_list,
- *   and make sure we will process rps_ipi_list from net_rx_action().
- *
- * - If this is our own queue, NAPI schedule our backlog.
- *   Note that this also raises NET_RX_SOFTIRQ.
- */
-static void napi_schedule_rps(struct softnet_data *sd)
-{
-	struct softnet_data *mysd = this_cpu_ptr(&softnet_data);
-
-#ifdef CONFIG_RPS
-	if (sd != mysd) {
-		sd->rps_ipi_next = mysd->rps_ipi_list;
-		mysd->rps_ipi_list = sd;
-
-		/* If not called from net_rx_action() or napi_threaded_poll()
-		 * we have to raise NET_RX_SOFTIRQ.
-		 */
-		if (!mysd->in_net_rx_action && !mysd->in_napi_threaded_poll)
-			__raise_softirq_irqoff(NET_RX_SOFTIRQ);
-		return;
-	}
-#endif /* CONFIG_RPS */
-	__napi_schedule_irqoff(&mysd->backlog);
-}
-
 #ifdef CONFIG_NET_FLOW_LIMIT
 int netdev_flow_limit_table_len __read_mostly = (1 << 12);
 #endif
@@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
 		 * We can use non atomic operation since we own the queue lock
 		 */
 		if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state))
-			napi_schedule_rps(sd);
+			__napi_schedule_irqoff(&sd->backlog);
 		goto enqueue;
 	}
 	reason = SKB_DROP_REASON_CPU_BACKLOG;
@@ -5896,63 +5848,12 @@ static void flush_all_backlogs(void)
 	cpus_read_unlock();
 }
 
-static void net_rps_send_ipi(struct softnet_data *remsd)
-{
-#ifdef CONFIG_RPS
-	while (remsd) {
-		struct softnet_data *next = remsd->rps_ipi_next;
-
-		if (cpu_online(remsd->cpu))
-			smp_call_function_single_async(remsd->cpu, &remsd->csd);
-		remsd = next;
-	}
-#endif
-}
-
-/*
- * net_rps_action_and_irq_enable sends any pending IPI's for rps.
- * Note: called with local irq disabled, but exits with local irq enabled.
- */
-static void net_rps_action_and_irq_enable(struct softnet_data *sd)
-{
-#ifdef CONFIG_RPS
-	struct softnet_data *remsd = sd->rps_ipi_list;
-
-	if (remsd) {
-		sd->rps_ipi_list = NULL;
-
-		local_irq_enable();
-
-		/* Send pending IPI's to kick RPS processing on remote cpus. */
-		net_rps_send_ipi(remsd);
-	} else
-#endif
-		local_irq_enable();
-}
-
-static bool sd_has_rps_ipi_waiting(struct softnet_data *sd)
-{
-#ifdef CONFIG_RPS
-	return sd->rps_ipi_list != NULL;
-#else
-	return false;
-#endif
-}
-
 static int process_backlog(struct napi_struct *napi, int quota)
 {
 	struct softnet_data *sd = container_of(napi, struct softnet_data, backlog);
 	bool again = true;
 	int work = 0;
 
-	/* Check if we have pending ipi, its better to send them now,
-	 * not waiting net_rx_action() end.
-	 */
-	if (sd_has_rps_ipi_waiting(sd)) {
-		local_irq_disable();
-		net_rps_action_and_irq_enable(sd);
-	}
-
 	napi->weight = READ_ONCE(dev_rx_weight);
 	while (again) {
 		struct sk_buff *skb;
@@ -5977,7 +5878,7 @@ static int process_backlog(struct napi_struct *napi, int quota)
 			 * We can use a plain write instead of clear_bit(),
 			 * and we dont need an smp_mb() memory barrier.
 			 */
-			napi->state = 0;
+			napi->state = BIT(NAPI_STATE_THREADED);
 			again = false;
 		} else {
 			skb_queue_splice_tail_init(&sd->input_pkt_queue,
@@ -6634,6 +6535,8 @@ static void skb_defer_free_flush(struct softnet_data *sd)
 	if (!READ_ONCE(sd->defer_list))
 		return;
 
+	smp_store_release(&sd->defer_ipi_scheduled, 0);
+
 	spin_lock(&sd->defer_lock);
 	skb = sd->defer_list;
 	sd->defer_list = NULL;
@@ -6647,39 +6550,42 @@ static void skb_defer_free_flush(struct softnet_data *sd)
 	}
 }
 
+static void napi_threaded_poll_loop(struct napi_struct *napi)
+{
+	struct softnet_data *sd;
+
+	for (;;) {
+		bool repoll = false;
+		void *have;
+
+		local_bh_disable();
+		sd = this_cpu_ptr(&softnet_data);
+		sd->in_napi_threaded_poll = true;
+
+		have = netpoll_poll_lock(napi);
+		__napi_poll(napi, &repoll);
+		netpoll_poll_unlock(have);
+
+		sd->in_napi_threaded_poll = false;
+		barrier();
+
+		skb_defer_free_flush(sd);
+		local_bh_enable();
+
+		if (!repoll)
+			break;
+
+		cond_resched();
+	}
+}
+
 static int napi_threaded_poll(void *data)
 {
 	struct napi_struct *napi = data;
-	struct softnet_data *sd;
-	void *have;
 
 	while (!napi_thread_wait(napi)) {
-		for (;;) {
-			bool repoll = false;
 
-			local_bh_disable();
-			sd = this_cpu_ptr(&softnet_data);
-			sd->in_napi_threaded_poll = true;
-
-			have = netpoll_poll_lock(napi);
-			__napi_poll(napi, &repoll);
-			netpoll_poll_unlock(have);
-
-			sd->in_napi_threaded_poll = false;
-			barrier();
-
-			if (sd_has_rps_ipi_waiting(sd)) {
-				local_irq_disable();
-				net_rps_action_and_irq_enable(sd);
-			}
-			skb_defer_free_flush(sd);
-			local_bh_enable();
-
-			if (!repoll)
-				break;
-
-			cond_resched();
-		}
+		napi_threaded_poll_loop(napi);
 	}
 	return 0;
 }
@@ -6714,8 +6620,6 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
 				 */
 				if (!list_empty(&sd->poll_list))
 					goto start;
-				if (!sd_has_rps_ipi_waiting(sd))
-					goto end;
 			}
 			break;
 		}
@@ -6744,8 +6648,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
 	else
 		sd->in_net_rx_action = false;
 
-	net_rps_action_and_irq_enable(sd);
-end:;
+	local_irq_enable();
 }
 
 struct netdev_adjacent {
@@ -11157,7 +11060,7 @@ static int dev_cpu_dead(unsigned int oldcpu)
 	struct sk_buff **list_skb;
 	struct sk_buff *skb;
 	unsigned int cpu;
-	struct softnet_data *sd, *oldsd, *remsd = NULL;
+	struct softnet_data *sd, *oldsd;
 
 	local_irq_disable();
 	cpu = smp_processor_id();
@@ -11189,22 +11092,13 @@ static int dev_cpu_dead(unsigned int oldcpu)
 							    poll_list);
 
 		list_del_init(&napi->poll_list);
-		if (napi->poll == process_backlog)
-			napi->state = 0;
-		else
+		if (!WARN_ON(napi->poll == process_backlog))
 			____napi_schedule(sd, napi);
 	}
 
 	raise_softirq_irqoff(NET_TX_SOFTIRQ);
 	local_irq_enable();
 
-#ifdef CONFIG_RPS
-	remsd = oldsd->rps_ipi_list;
-	oldsd->rps_ipi_list = NULL;
-#endif
-	/* send out pending IPI's on offline CPU */
-	net_rps_send_ipi(remsd);
-
 	/* Process offline CPU's input_pkt_queue */
 	while ((skb = __skb_dequeue(&oldsd->process_queue))) {
 		netif_rx(skb);
@@ -11457,6 +11351,43 @@ static struct pernet_operations __net_initdata default_device_ops = {
  *
  */
 
+static DEFINE_PER_CPU(struct task_struct *, backlog_napi);
+
+static int backlog_napi_should_run(unsigned int cpu)
+{
+	struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu);
+	struct napi_struct *napi = &sd->backlog;
+
+	if (READ_ONCE(sd->defer_list))
+		return 1;
+
+	return test_bit(NAPI_STATE_SCHED, &napi->state);
+}
+
+static void run_backlog_napi(unsigned int cpu)
+{
+	struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu);
+
+	napi_threaded_poll_loop(&sd->backlog);
+}
+
+static void backlog_napi_setup(unsigned int cpu)
+{
+	struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu);
+	struct napi_struct *napi = &sd->backlog;
+
+	napi->thread = this_cpu_read(backlog_napi);
+	set_bit(NAPI_STATE_THREADED, &napi->state);
+}
+
+static struct smp_hotplug_thread backlog_threads = {
+	.store                  = &backlog_napi,
+	.thread_should_run      = backlog_napi_should_run,
+	.thread_fn              = run_backlog_napi,
+	.thread_comm            = "backlog_napi/%u",
+	.setup			= backlog_napi_setup,
+};
+
 /*
  *       This is called single threaded during boot, so no need
  *       to take the rtnl semaphore.
@@ -11497,17 +11428,14 @@ static int __init net_dev_init(void)
 #endif
 		INIT_LIST_HEAD(&sd->poll_list);
 		sd->output_queue_tailp = &sd->output_queue;
-#ifdef CONFIG_RPS
-		INIT_CSD(&sd->csd, rps_trigger_softirq, sd);
-		sd->cpu = i;
-#endif
-		INIT_CSD(&sd->defer_csd, trigger_rx_softirq, sd);
 		spin_lock_init(&sd->defer_lock);
 
 		init_gro_hash(&sd->backlog);
 		sd->backlog.poll = process_backlog;
 		sd->backlog.weight = weight_p;
+		INIT_LIST_HEAD(&sd->backlog.poll_list);
 	}
+	smpboot_register_percpu_thread(&backlog_threads);
 
 	dev_boot_phase = 0;
 
diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c
index 09f7ed1a04e8a..086283cc8d47b 100644
--- a/net/core/net-procfs.c
+++ b/net/core/net-procfs.c
@@ -180,7 +180,7 @@ static int softnet_seq_show(struct seq_file *seq, void *v)
 		   sd->processed, sd->dropped, sd->time_squeeze, 0,
 		   0, 0, 0, 0, /* was fastroute */
 		   0,	/* was cpu_collision */
-		   sd->received_rps, flow_limit_count,
+		   0 /* was received_rps */, flow_limit_count,
 		   input_qlen + process_qlen, (int)seq->index,
 		   input_qlen, process_qlen);
 	return 0;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 33fdf04d4334d..265a8aa6b3228 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -6802,7 +6802,7 @@ nodefer:	__kfree_skb(skb);
 	 * if we are unlucky enough (this seems very unlikely).
 	 */
 	if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1))
-		smp_call_function_single_async(cpu, &sd->defer_csd);
+		__napi_schedule(&sd->backlog);
 }
 
 static void skb_splice_csum_page(struct sk_buff *skb, struct page *page,
-- 
2.40.1

[RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Sebastian Andrzej Siewior]

Once ksoftirqd become active, all softirqs which were raised, would not
be processed immediately but delayed to ksoftirqd. On PREEMPT_RT this
means softirqs, which were raised in a threaded interrupt (at elevated
process priority), would not be served after the interrupt handler
completed its work but will wait until ksoftirqd (normal priority)
becomes running on the CPU. On a busy system with plenty of RT tasks
this could be delayed for quite some time and leads to problems in
general.

This is an undesired situation and it has been attempted to avoid the
situation in which ksoftirqd becomes scheduled. This changed since
commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
and now a threaded interrupt handler will handle soft interrupts at its
end even if ksoftirqd is pending. That means that they will be processed
in the context in which they were raised.

Unfortunately also all other soft interrupts which were raised (or
enqueued) earlier and are not yet handled. This happens if a thread with
higher priority is raised and has to catch up. This isn't a new problem
and the new high priority thread will PI-boost the current sofitrq owner
or start from scratch if ksoftirqd wasn't running yet.

Since pending ksoftirqd no longer blocks other interrupt threads from
handling soft interrupts I belive the warning can be disabled. The
pending softirq work has to be solved differently.

Remove the warning and update the comment.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/interrupt.h |  4 ++--
 kernel/smp.c              |  4 +---
 kernel/softirq.c          | 12 +++++-------
 3 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index a92bce40b04b3..5143ae0ea9356 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -590,9 +590,9 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 
 #ifdef CONFIG_PREEMPT_RT
-extern void do_softirq_post_smp_call_flush(unsigned int was_pending);
+extern void do_softirq_post_smp_call_flush(void);
 #else
-static inline void do_softirq_post_smp_call_flush(unsigned int unused)
+static inline void do_softirq_post_smp_call_flush(void)
 {
 	do_softirq();
 }
diff --git a/kernel/smp.c b/kernel/smp.c
index 385179dae360e..cd7db5ffe95ab 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -554,7 +554,6 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline)
  */
 void flush_smp_call_function_queue(void)
 {
-	unsigned int was_pending;
 	unsigned long flags;
 
 	if (llist_empty(this_cpu_ptr(&call_single_queue)))
@@ -562,10 +561,9 @@ void flush_smp_call_function_queue(void)
 
 	local_irq_save(flags);
 	/* Get the already pending soft interrupts for RT enabled kernels */
-	was_pending = local_softirq_pending();
 	__flush_smp_call_function_queue(true);
 	if (local_softirq_pending())
-		do_softirq_post_smp_call_flush(was_pending);
+		do_softirq_post_smp_call_flush();
 
 	local_irq_restore(flags);
 }
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 807b34ccd7973..aa299cb3ff47b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -281,15 +281,13 @@ static inline void invoke_softirq(void)
 
 /*
  * flush_smp_call_function_queue() can raise a soft interrupt in a function
- * call. On RT kernels this is undesired and the only known functionality
- * in the block layer which does this is disabled on RT. If soft interrupts
- * get raised which haven't been raised before the flush, warn so it can be
- * investigated.
+ * call. On RT kernels this is undesired because the work is no longer processed
+ * in the context where it originated. It is not especially harmfull but best to
+ * be avoided.
  */
-void do_softirq_post_smp_call_flush(unsigned int was_pending)
+void do_softirq_post_smp_call_flush(void)
 {
-	if (WARN_ON_ONCE(was_pending != local_softirq_pending()))
-		invoke_softirq();
+	invoke_softirq();
 }
 
 #else /* CONFIG_PREEMPT_RT */
-- 
2.40.1

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Jakub Kicinski]

On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote:
> The RPS code and "deferred skb free" both send IPI/ function call
> to a remote CPU in which a softirq is raised. This leads to a warning on
> PREEMPT_RT because raising softiqrs from function call led to undesired
> behaviour in the past. I had duct tape in RT for the "deferred skb free"
> and Wander Lairson Costa reported the RPS case.

Could you find a less invasive solution?
backlog is used by veth == most containerized environments.
This change has a very high risk of regression for a lot of people.

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Jesper Dangaard Brouer]

On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
> This is an undesired situation and it has been attempted to avoid the
> situation in which ksoftirqd becomes scheduled. This changed since
> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
> and now a threaded interrupt handler will handle soft interrupts at its
> end even if ksoftirqd is pending. That means that they will be processed
> in the context in which they were raised.

$ git describe --contains d15121be74856
v6.5-rc1~232^2~4

That revert basically removes the "overload" protection that was added
to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
ksoftirqd do its job") in UDP overload situations when UDP socket
receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
doesn't process packets (because softirq steals CPU/sched time from UDP
pid).  Warning Cloudflare (Cc) as this might affect their production
use-cases, and I recommend getting involved to evaluate the effect of
these changes.

I do realize/acknowledge that the reverted patch caused other latency
issues, given it was a "big-hammer" approach affecting other softirq
processing (as can be seen by e.g. the watchdog fixes patches).
Thus, the revert makes sense, but how to regain the "overload"
protection such that RX networking cannot starve processes reading from
the socket? (is this what Sebastian's patchset does?)

--Jesper

Thread link for people Cc'ed: 
https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Yan Zhai]

On Tue, Aug 15, 2023 at 7:08 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote:
>
>
>
> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
> > This is an undesired situation and it has been attempted to avoid the
> > situation in which ksoftirqd becomes scheduled. This changed since
> > commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
> > and now a threaded interrupt handler will handle soft interrupts at its
> > end even if ksoftirqd is pending. That means that they will be processed
> > in the context in which they were raised.
>
> $ git describe --contains d15121be74856
> v6.5-rc1~232^2~4
>
> That revert basically removes the "overload" protection that was added
> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
> ksoftirqd do its job") in UDP overload situations when UDP socket
> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
> doesn't process packets (because softirq steals CPU/sched time from UDP
> pid).  Warning Cloudflare (Cc) as this might affect their production
> use-cases, and I recommend getting involved to evaluate the effect of
> these changes.
>
> I do realize/acknowledge that the reverted patch caused other latency
> issues, given it was a "big-hammer" approach affecting other softirq
> processing (as can be seen by e.g. the watchdog fixes patches).
> Thus, the revert makes sense, but how to regain the "overload"
> protection such that RX networking cannot starve processes reading from
> the socket? (is this what Sebastian's patchset does?)
>
Thanks for notifying us. We will need to evaluate if this is going to
change the picture under serious floods.

Yan

> --Jesper
>
> Thread link for people Cc'ed:
> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Jesper Dangaard Brouer]

On 15/08/2023 14.08, Jesper Dangaard Brouer wrote:
> 
> 
> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
>> This is an undesired situation and it has been attempted to avoid the
>> situation in which ksoftirqd becomes scheduled. This changed since
>> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
>> and now a threaded interrupt handler will handle soft interrupts at its
>> end even if ksoftirqd is pending. That means that they will be processed
>> in the context in which they were raised.
> 
> $ git describe --contains d15121be74856
> v6.5-rc1~232^2~4
> 
> That revert basically removes the "overload" protection that was added
> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
> ksoftirqd do its job") in UDP overload situations when UDP socket
> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
> doesn't process packets (because softirq steals CPU/sched time from UDP
> pid).  Warning Cloudflare (Cc) as this might affect their production
> use-cases, and I recommend getting involved to evaluate the effect of
> these changes.
> 

I did some testing on net-next (with commit d15121be74856 ("Revert 
"softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink.

And I observe the old overload issue occur again, where userspace 
process (udp_sink) process very few packets when running on *same* CPU 
as the NAPI-RX/IRQ processing.  The perf report "comm" clearly shows 
that NAPI runs in the context of the "udp_sink" process, stealing its 
sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details 
below).
What happens are that NAPI takes 64 packets and queue them to the 
udp_sink process *socket*, the udp_sink process *wakeup* process 1 
packet from socket queue and on exit (__local_bh_enable_ip) runs softirq 
that starts NAPI (to again process 64 packets... repeat).


> I do realize/acknowledge that the reverted patch caused other latency
> issues, given it was a "big-hammer" approach affecting other softirq
> processing (as can be seen by e.g. the watchdog fixes patches).
> Thus, the revert makes sense, but how to regain the "overload"
> protection such that RX networking cannot starve processes reading from
> the socket? (is this what Sebastian's patchset does?)
> 

I'm no expert in sched / softirq area of the kernel, but I'm willing to 
help out testing different solution that can regain the "overload" 
protection e.g. avoid packet processing "falls-of-an-edge" (and thus 
opens the kernel to be DDoS'ed easily).
Is this what Sebastian's patchset does?


> 
> Thread link for people Cc'ed: 
> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r

--Jesper
(some testlab results below)

[udp_sink] 
https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c


When udp_sink runs on same CPU and NAPI/softirq
  - UdpInDatagrams: 2,948 packets/sec

$ nstat -n && sleep 1 && nstat
#kernel
IpInReceives                    2831056            0.0
IpInDelivers                    2831053            0.0
UdpInDatagrams                  2948               0.0
UdpInErrors                     2828118            0.0
UdpRcvbufErrors                 2828118            0.0
IpExtInOctets                   130206496          0.0
IpExtInNoECTPkts                2830576            0.0

When udp_sink runs on another CPU than NAPI-RX.
  - UdpInDatagrams: 1,722,307 pps

$ nstat -n && sleep 1 && nstat
#kernel
IpInReceives                    2318560            0.0
IpInDelivers                    2318562            0.0
UdpInDatagrams                  1722307            0.0
UdpInErrors                     596280             0.0
UdpRcvbufErrors                 596280             0.0
IpExtInOctets                   106634256          0.0
IpExtInNoECTPkts                2318136            0.0

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Yan Zhai]

On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote:
>
>
>
> On 15/08/2023 14.08, Jesper Dangaard Brouer wrote:
> >
> >
> > On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
> >> This is an undesired situation and it has been attempted to avoid the
> >> situation in which ksoftirqd becomes scheduled. This changed since
> >> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
> >> and now a threaded interrupt handler will handle soft interrupts at its
> >> end even if ksoftirqd is pending. That means that they will be processed
> >> in the context in which they were raised.
> >
> > $ git describe --contains d15121be74856
> > v6.5-rc1~232^2~4
> >
> > That revert basically removes the "overload" protection that was added
> > to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
> > in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
> > ksoftirqd do its job") in UDP overload situations when UDP socket
> > receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
> > doesn't process packets (because softirq steals CPU/sched time from UDP
> > pid).  Warning Cloudflare (Cc) as this might affect their production
> > use-cases, and I recommend getting involved to evaluate the effect of
> > these changes.
> >
>
> I did some testing on net-next (with commit d15121be74856 ("Revert
> "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink.
>
> And I observe the old overload issue occur again, where userspace
> process (udp_sink) process very few packets when running on *same* CPU
> as the NAPI-RX/IRQ processing.  The perf report "comm" clearly shows
> that NAPI runs in the context of the "udp_sink" process, stealing its
> sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details
> below).
> What happens are that NAPI takes 64 packets and queue them to the
> udp_sink process *socket*, the udp_sink process *wakeup* process 1
> packet from socket queue and on exit (__local_bh_enable_ip) runs softirq
> that starts NAPI (to again process 64 packets... repeat).
>
I think there are two scenarios to consider:
1. Actual DoS scenario. In this case, we would drop DoS packets
through XDP, which might actually relieve the stress. According to
Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too
much time. This is also something I would like to validate again since
I cannot tell if those tests were performed before or after the
reverted commit.
2. Legit elephant flows (so it should not be just dropped). This one
is closer to what you tested above, and it is a much harder issue
since packets are legit and should not be dropped early at XDP. Let
the scheduler move affected processes away seems to be the non-optimal
but straight answer for now. However, I suspect this would impose an
overload issue for those programmed with RFS or ARFS, since flows
would "follow" the processes. They probably have to force threaded
NAPI for tuning.

[1] https://blog.cloudflare.com/how-to-drop-10-million-packets/

>
> > I do realize/acknowledge that the reverted patch caused other latency
> > issues, given it was a "big-hammer" approach affecting other softirq
> > processing (as can be seen by e.g. the watchdog fixes patches).
> > Thus, the revert makes sense, but how to regain the "overload"
> > protection such that RX networking cannot starve processes reading from
> > the socket? (is this what Sebastian's patchset does?)
> >
>
> I'm no expert in sched / softirq area of the kernel, but I'm willing to
> help out testing different solution that can regain the "overload"
> protection e.g. avoid packet processing "falls-of-an-edge" (and thus
> opens the kernel to be DDoS'ed easily).
> Is this what Sebastian's patchset does?
>
>
> >
> > Thread link for people Cc'ed:
> > https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r
>
> --Jesper
> (some testlab results below)
>
> [udp_sink]
> https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c
>
>
> When udp_sink runs on same CPU and NAPI/softirq
>   - UdpInDatagrams: 2,948 packets/sec
>
> $ nstat -n && sleep 1 && nstat
> #kernel
> IpInReceives                    2831056            0.0
> IpInDelivers                    2831053            0.0
> UdpInDatagrams                  2948               0.0
> UdpInErrors                     2828118            0.0
> UdpRcvbufErrors                 2828118            0.0
> IpExtInOctets                   130206496          0.0
> IpExtInNoECTPkts                2830576            0.0
>
> When udp_sink runs on another CPU than NAPI-RX.
>   - UdpInDatagrams: 1,722,307 pps
>
> $ nstat -n && sleep 1 && nstat
> #kernel
> IpInReceives                    2318560            0.0
> IpInDelivers                    2318562            0.0
> UdpInDatagrams                  1722307            0.0
> UdpInErrors                     596280             0.0
> UdpRcvbufErrors                 596280             0.0
> IpExtInOctets                   106634256          0.0
> IpExtInNoECTPkts                2318136            0.0
>
>


-- 

Yan

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Sebastian Andrzej Siewior]

On 2023-08-16 16:48:56 [+0200], Jesper Dangaard Brouer wrote:
> I'm no expert in sched / softirq area of the kernel, but I'm willing to help
> out testing different solution that can regain the "overload" protection
> e.g. avoid packet processing "falls-of-an-edge" (and thus opens the kernel
> to be DDoS'ed easily).
> Is this what Sebastian's patchset does?

I was going to respond but didn't know what so far.
Can you figure out if you are using backlog or not. If you do, could try
my patch. If not could you try to enable napi-thread and see?
Either way I will re-read it again.

Sebastian

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Jesper Dangaard Brouer]

On 16/08/2023 17.15, Yan Zhai wrote:
> On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote:
>>
>> On 15/08/2023 14.08, Jesper Dangaard Brouer wrote:
>>>
>>>
>>> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
>>>> This is an undesired situation and it has been attempted to avoid the
>>>> situation in which ksoftirqd becomes scheduled. This changed since
>>>> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
>>>> and now a threaded interrupt handler will handle soft interrupts at its
>>>> end even if ksoftirqd is pending. That means that they will be processed
>>>> in the context in which they were raised.
>>>
>>> $ git describe --contains d15121be74856
>>> v6.5-rc1~232^2~4
>>>
>>> That revert basically removes the "overload" protection that was added
>>> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
>>> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
>>> ksoftirqd do its job") in UDP overload situations when UDP socket
>>> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
>>> doesn't process packets (because softirq steals CPU/sched time from UDP
>>> pid).  Warning Cloudflare (Cc) as this might affect their production
>>> use-cases, and I recommend getting involved to evaluate the effect of
>>> these changes.
>>>
>>
>> I did some testing on net-next (with commit d15121be74856 ("Revert
>> "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink.
>>
>> And I observe the old overload issue occur again, where userspace
>> process (udp_sink) process very few packets when running on *same* CPU
>> as the NAPI-RX/IRQ processing.  The perf report "comm" clearly shows
>> that NAPI runs in the context of the "udp_sink" process, stealing its
>> sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details
>> below).
>> What happens are that NAPI takes 64 packets and queue them to the
>> udp_sink process *socket*, the udp_sink process *wakeup* process 1
>> packet from socket queue and on exit (__local_bh_enable_ip) runs softirq
>> that starts NAPI (to again process 64 packets... repeat).
>>
> I think there are two scenarios to consider:
 >
> 1. Actual DoS scenario. In this case, we would drop DoS packets
> through XDP, which might actually relieve the stress. According to
> Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too
> much time. This is also something I would like to validate again since

Yes, using XDP to drop packet will/should relieve the stress, as it
basically can discard some of the 64 packets processed by NAPI vs the 1
packet received by userspace (that re-trigger NAPI), giving a better 
balance.

> I cannot tell if those tests were performed before or after the
> reverted commit.

Marek's tests will likely contain the patch 4cd13c21b207 ("softirq: Let
ksoftirqd do its job") as blog is from 2018 and patch from 2016, but
shouldn't matter much.


> 2. Legit elephant flows (so it should not be just dropped). This one
> is closer to what you tested above, and it is a much harder issue
> since packets are legit and should not be dropped early at XDP. Let
> the scheduler move affected processes away seems to be the non-optimal
> but straight answer for now. However, I suspect this would impose an
> overload issue for those programmed with RFS or ARFS, since flows
> would "follow" the processes. They probably have to force threaded
> NAPI for tuning.
>

True, this is the case I don't know how to solve.

For UDP packets it is NOT optimal to let the process "follow"/run on the 
NAPI-RX CPU. For TCP traffic it is faster to run on same CPU, which 
could be related to GRO effect, or simply that tcp_recvmsg gets a stream 
of data (before it invokes __local_bh_enable_ip causing do_softirq).

I have also tested with netperf UDP packets[2] in a scenario that 
doesn't cause "overload" and CPU have idle cycles.  When UDP-netserver 
is running on same CPU as NAPI then I see approx 38% (82020/216362) 
UdpRcvbufErrors [3] (and separate CPUs 2.8%).  Sure, I could increase 
buffer size, but the point is NAPI can enqueue 64 packet and UDP 
receiver dequeue 1 packet.

This reminded me that kernel have a recvmmsg (extra "m") syscall for 
multiple packets.  I tested this (as udop_sink have support), but no 
luck. This is because internally in the kernel (do_recvmmsg) is just a 
loop over ___sys_recvmsg/__skb_recv_udp, which have a BH-spinlock per 
packet that invokes __local_bh_enable_ip/do_softirq.  I guess, we/netdev 
could fix recvmmsg() to bulk-dequeue from socket queue (BH-socket unlock 
is triggering __local_bh_enable_ip/do_softirq) and then have a solution 
for UDP(?).


[2] netperf -H 198.18.1.1 -D1 -l 1200 -t UDP_STREAM -T 0,0 -- -m 1472 -N -n

[3]
$ nstat -n && sleep 1 && nstat
#kernel
IpInReceives                    216362             0.0
IpInDelivers                    216354             0.0
UdpInDatagrams                  134356             0.0
UdpInErrors                     82020              0.0
UdpRcvbufErrors                 82020              0.0
IpExtInOctets                   324600000          0.0
IpExtInNoECTPkts                216400             0.0


> [1] https://blog.cloudflare.com/how-to-drop-10-million-packets/
> 
>>
>>> I do realize/acknowledge that the reverted patch caused other latency
>>> issues, given it was a "big-hammer" approach affecting other softirq
>>> processing (as can be seen by e.g. the watchdog fixes patches).
>>> Thus, the revert makes sense, but how to regain the "overload"
>>> protection such that RX networking cannot starve processes reading from
>>> the socket? (is this what Sebastian's patchset does?)
>>>
>>
>> I'm no expert in sched / softirq area of the kernel, but I'm willing to
>> help out testing different solution that can regain the "overload"
>> protection e.g. avoid packet processing "falls-of-an-edge" (and thus
>> opens the kernel to be DDoS'ed easily).
>> Is this what Sebastian's patchset does?
>>
>>
>>>
>>> Thread link for people Cc'ed:
>>> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r
>>
>> --Jesper
>> (some testlab results below)
>>
>> [udp_sink]
>> https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c
>>
>>
>> When udp_sink runs on same CPU and NAPI/softirq
>>    - UdpInDatagrams: 2,948 packets/sec
>>
>> $ nstat -n && sleep 1 && nstat
>> #kernel
>> IpInReceives                    2831056            0.0
>> IpInDelivers                    2831053            0.0
>> UdpInDatagrams                  2948               0.0
>> UdpInErrors                     2828118            0.0
>> UdpRcvbufErrors                 2828118            0.0
>> IpExtInOctets                   130206496          0.0
>> IpExtInNoECTPkts                2830576            0.0
>>
>> When udp_sink runs on another CPU than NAPI-RX.
>>    - UdpInDatagrams: 1,722,307 pps
>>
>> $ nstat -n && sleep 1 && nstat
>> #kernel
>> IpInReceives                    2318560            0.0
>> IpInDelivers                    2318562            0.0
>> UdpInDatagrams                  1722307            0.0
>> UdpInErrors                     596280             0.0
>> UdpRcvbufErrors                 596280             0.0
>> IpExtInOctets                   106634256          0.0
>> IpExtInNoECTPkts                2318136            0.0
>>
>>
> 
> 

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-08-14 11:24:21 [-0700], Jakub Kicinski wrote:
> On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote:
> > The RPS code and "deferred skb free" both send IPI/ function call
> > to a remote CPU in which a softirq is raised. This leads to a warning on
> > PREEMPT_RT because raising softiqrs from function call led to undesired
> > behaviour in the past. I had duct tape in RT for the "deferred skb free"
> > and Wander Lairson Costa reported the RPS case.
> 
> Could you find a less invasive solution?
> backlog is used by veth == most containerized environments.
> This change has a very high risk of regression for a lot of people.

Looking at the cloudflare ppl here in the thread, I doubt they use
backlog but have proper NAPI so they might not need this.

There is no threaded NAPI for backlog and RPS. This was suggested as the
mitigation for the highload/ DoS case. Can this become a problem or
- backlog is used only by old drivers so they can move to proper NAPI if
  it becomes a problem.
- RPS spreads the load across multiple CPUs so it unlikely to become a
  problem.

Making this either optional in general or mandatory for threaded
interrupts or PREEMPT_RT will probably not make the maintenance of this
code any simpler.

I've been looking at veth. In the xdp case it has its own NAPI instance.
In the non-xdp it uses backlog. This should be called from
ndo_start_xmit and user's write() so BH is off and interrupts are
enabled at this point and it should be kind of rate-limited. Couldn't we
bypass backlog in this case and deliver the packet directly to the
stack?

Sebastian

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Jakub Kicinski]

On Thu, 17 Aug 2023 15:16:12 +0200 Sebastian Andrzej Siewior wrote:
> I've been looking at veth. In the xdp case it has its own NAPI instance.
> In the non-xdp it uses backlog. This should be called from
> ndo_start_xmit and user's write() so BH is off and interrupts are
> enabled at this point and it should be kind of rate-limited. Couldn't we
> bypass backlog in this case and deliver the packet directly to the
> stack?

The backlog in veth eats measurable percentage points of RPS of real
workloads, and I think number of people looked at getting rid of it.
So worthy goal for sure, but may not be a trivial fix.

To my knowledge the two main problems are:
 - we don't want to charge the sending application the processing for
   both "sides" of the connection and all the switching costs.
 - we may get an AA deadlock if the packet ends up looping in any way.

Or at least that's what I remember the problem being at 8am in the
morning :) Adding Daniel and Martin to CC, Paolo would also know this
better than me but I think he's AFK for the rest of the week.

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-08-17 08:30:25 [-0700], Jakub Kicinski wrote:
> On Thu, 17 Aug 2023 15:16:12 +0200 Sebastian Andrzej Siewior wrote:
> > I've been looking at veth. In the xdp case it has its own NAPI instance.
> > In the non-xdp it uses backlog. This should be called from
> > ndo_start_xmit and user's write() so BH is off and interrupts are
> > enabled at this point and it should be kind of rate-limited. Couldn't we
> > bypass backlog in this case and deliver the packet directly to the
> > stack?
> 
> The backlog in veth eats measurable percentage points of RPS of real
> workloads, and I think number of people looked at getting rid of it.
> So worthy goal for sure, but may not be a trivial fix.

We could separate RPS from backlog but then we still process RPS after
backlog so not sure if this gains anything. Letting veth always use its
NAPI in this case would probably do that. Not sure if it helps…

> To my knowledge the two main problems are:
>  - we don't want to charge the sending application the processing for
>    both "sides" of the connection and all the switching costs.

The packet is injected by the user and softirq is served once BH gets
back to 0. So it is served within the task's context and might be
accounted on softirq/ system (might as I think it needs to be observed
by the timer interrupt for the accounting).

>  - we may get an AA deadlock if the packet ends up looping in any way.

Right, forgot about that one.

> Or at least that's what I remember the problem being at 8am in the
> morning :) Adding Daniel and Martin to CC, Paolo would also know this
> better than me but I think he's AFK for the rest of the week.

Sebastian

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Yan Zhai]

On Thu, Aug 17, 2023 at 8:16 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2023-08-14 11:24:21 [-0700], Jakub Kicinski wrote:
> > On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote:
> > > The RPS code and "deferred skb free" both send IPI/ function call
> > > to a remote CPU in which a softirq is raised. This leads to a warning on
> > > PREEMPT_RT because raising softiqrs from function call led to undesired
> > > behaviour in the past. I had duct tape in RT for the "deferred skb free"
> > > and Wander Lairson Costa reported the RPS case.
> >
> > Could you find a less invasive solution?
> > backlog is used by veth == most containerized environments.
> > This change has a very high risk of regression for a lot of people.
>
> Looking at the cloudflare ppl here in the thread, I doubt they use
> backlog but have proper NAPI so they might not need this.
>
Cloudflare does have backlog usage. On some veths we have to turn GRO
off to cope with multi-layer encapsulation, and there is also no XDP
attached on these interfaces, thus the backlog is used. There are also
other usage of backlog, tuntap, loopback and bpf-redirect ingress.
Frankly speaking, making a NAPI instance "threaded" itself is not a
concern. We have threaded NAPI running on some veth for quite a while,
and it performs pretty well. The concern, if any, would be the
maturity of new code. I am happy to help derisk with some lab tests
and dogfooding if generic agreement is reached to proceed with this
idea.

Yan

> There is no threaded NAPI for backlog and RPS. This was suggested as the
> mitigation for the highload/ DoS case. Can this become a problem or
> - backlog is used only by old drivers so they can move to proper NAPI if
>   it becomes a problem.
> - RPS spreads the load across multiple CPUs so it unlikely to become a
>   problem.
>
> Making this either optional in general or mandatory for threaded
> interrupts or PREEMPT_RT will probably not make the maintenance of this
> code any simpler.
>
> I've been looking at veth. In the xdp case it has its own NAPI instance.
> In the non-xdp it uses backlog. This should be called from
> ndo_start_xmit and user's write() so BH is off and interrupts are
> enabled at this point and it should be kind of rate-limited. Couldn't we
> bypass backlog in this case and deliver the packet directly to the
> stack?
>
> Sebastian



-- 

Yan

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-08-18 09:43:08 [-0500], Yan Zhai wrote:
> > Looking at the cloudflare ppl here in the thread, I doubt they use
> > backlog but have proper NAPI so they might not need this.
> >
> Cloudflare does have backlog usage. On some veths we have to turn GRO

Oh. Okay.

> off to cope with multi-layer encapsulation, and there is also no XDP
> attached on these interfaces, thus the backlog is used. There are also
> other usage of backlog, tuntap, loopback and bpf-redirect ingress.
> Frankly speaking, making a NAPI instance "threaded" itself is not a
> concern. We have threaded NAPI running on some veth for quite a while,
> and it performs pretty well. The concern, if any, would be the
> maturity of new code. I am happy to help derisk with some lab tests
> and dogfooding if generic agreement is reached to proceed with this
> idea.

If you have threaded NAPI for veth then you wouldn't be affected by this
code. However, if you _are_ affected by this and you use veth it would
be helpful to figure out if you have problems as of net-next and if this
helps or makes it worse.

As of now Jakub isn't eager to have it and my testing/ convincing is
quite limited. If nobody else yells that something like that would be
helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this
series.

> Yan

Sebastian

Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush().

[Yan Zhai]

On Wed, Aug 16, 2023 at 4:02 PM Jesper Dangaard Brouer
<jbrouer@redhat.com> wrote:
>
>
>
> On 16/08/2023 17.15, Yan Zhai wrote:
> > On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote:
> >>
> >> On 15/08/2023 14.08, Jesper Dangaard Brouer wrote:
> >>>
> >>>
> >>> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote:
> >>>> This is an undesired situation and it has been attempted to avoid the
> >>>> situation in which ksoftirqd becomes scheduled. This changed since
> >>>> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"")
> >>>> and now a threaded interrupt handler will handle soft interrupts at its
> >>>> end even if ksoftirqd is pending. That means that they will be processed
> >>>> in the context in which they were raised.
> >>>
> >>> $ git describe --contains d15121be74856
> >>> v6.5-rc1~232^2~4
> >>>
> >>> That revert basically removes the "overload" protection that was added
> >>> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare).  As described
> >>> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let
> >>> ksoftirqd do its job") in UDP overload situations when UDP socket
> >>> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost
> >>> doesn't process packets (because softirq steals CPU/sched time from UDP
> >>> pid).  Warning Cloudflare (Cc) as this might affect their production
> >>> use-cases, and I recommend getting involved to evaluate the effect of
> >>> these changes.
> >>>
> >>
> >> I did some testing on net-next (with commit d15121be74856 ("Revert
> >> "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink.
> >>
> >> And I observe the old overload issue occur again, where userspace
> >> process (udp_sink) process very few packets when running on *same* CPU
> >> as the NAPI-RX/IRQ processing.  The perf report "comm" clearly shows
> >> that NAPI runs in the context of the "udp_sink" process, stealing its
> >> sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details
> >> below).
> >> What happens are that NAPI takes 64 packets and queue them to the
> >> udp_sink process *socket*, the udp_sink process *wakeup* process 1
> >> packet from socket queue and on exit (__local_bh_enable_ip) runs softirq
> >> that starts NAPI (to again process 64 packets... repeat).
> >>
> > I think there are two scenarios to consider:
>  >
> > 1. Actual DoS scenario. In this case, we would drop DoS packets
> > through XDP, which might actually relieve the stress. According to
> > Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too
> > much time. This is also something I would like to validate again since
>
> Yes, using XDP to drop packet will/should relieve the stress, as it
> basically can discard some of the 64 packets processed by NAPI vs the 1
> packet received by userspace (that re-trigger NAPI), giving a better
> balance.
>
> > I cannot tell if those tests were performed before or after the
> > reverted commit.
>
> Marek's tests will likely contain the patch 4cd13c21b207 ("softirq: Let
> ksoftirqd do its job") as blog is from 2018 and patch from 2016, but
> shouldn't matter much.
>
>
> > 2. Legit elephant flows (so it should not be just dropped). This one
> > is closer to what you tested above, and it is a much harder issue
> > since packets are legit and should not be dropped early at XDP. Let
> > the scheduler move affected processes away seems to be the non-optimal
> > but straight answer for now. However, I suspect this would impose an
> > overload issue for those programmed with RFS or ARFS, since flows
> > would "follow" the processes. They probably have to force threaded
> > NAPI for tuning.
> >
>
> True, this is the case I don't know how to solve.
>
> For UDP packets it is NOT optimal to let the process "follow"/run on the
> NAPI-RX CPU. For TCP traffic it is faster to run on same CPU, which
> could be related to GRO effect, or simply that tcp_recvmsg gets a stream
> of data (before it invokes __local_bh_enable_ip causing do_softirq).
>
To maximize single flow throughput, it is not optimal to run RX on the
same CPU with the receiver, regardless of TCP or UDP. The difference
is that TCP does not have tput issue until 10+ Gbps thanks to GRO. In
some internal benchmarking effort, I found that pinning iperf server
on the same RX would yield ~13-14 Gbps TCP while running on different
cores would have 25G NIC saturated (for both same or different NUMA
case). Despite single flow throughput upper bound getting hit when
running on the same core, CPU cycles to process each packet is
actually reduced. So it is likely more friendly to the production
environment we are dealing with where there are a lot more smaller
flows. It is something I planned to test more (in the past we had
major services pinned on dedicated cores, but recently we start to
unpin to improve tail latency of other services). But with the
protection gone, it adds quite some uncertainty to the picture.

> I have also tested with netperf UDP packets[2] in a scenario that
> doesn't cause "overload" and CPU have idle cycles.  When UDP-netserver
> is running on same CPU as NAPI then I see approx 38% (82020/216362)
> UdpRcvbufErrors [3] (and separate CPUs 2.8%).  Sure, I could increase
> buffer size, but the point is NAPI can enqueue 64 packet and UDP
> receiver dequeue 1 packet.
>
> This reminded me that kernel have a recvmmsg (extra "m") syscall for
> multiple packets.  I tested this (as udop_sink have support), but no
> luck. This is because internally in the kernel (do_recvmmsg) is just a
> loop over ___sys_recvmsg/__skb_recv_udp, which have a BH-spinlock per
> packet that invokes __local_bh_enable_ip/do_softirq.  I guess, we/netdev
> could fix recvmmsg() to bulk-dequeue from socket queue (BH-socket unlock
> is triggering __local_bh_enable_ip/do_softirq) and then have a solution
> for UDP(?).
>
recvmmsg does help getting more packets in a batch, but it has an
issue of buffer allocation upfront: when there are millions of
connections, preallocate too many buffers can mount up memory pressure
a lot.
On an alternative view, enable UDP GRO seems a direct help in this
context, to bring UDP on par with TCP, and reduce the RX overhead. We
already have quite some UDP GRO/GSO use cases for virtual machines and
QUIC handling, time to persuade engineers to add more maybe.

Yan

>
> [2] netperf -H 198.18.1.1 -D1 -l 1200 -t UDP_STREAM -T 0,0 -- -m 1472 -N -n
>
> [3]
> $ nstat -n && sleep 1 && nstat
> #kernel
> IpInReceives                    216362             0.0
> IpInDelivers                    216354             0.0
> UdpInDatagrams                  134356             0.0
> UdpInErrors                     82020              0.0
> UdpRcvbufErrors                 82020              0.0
> IpExtInOctets                   324600000          0.0
> IpExtInNoECTPkts                216400             0.0
>
>
> > [1] https://blog.cloudflare.com/how-to-drop-10-million-packets/
> >
> >>
> >>> I do realize/acknowledge that the reverted patch caused other latency
> >>> issues, given it was a "big-hammer" approach affecting other softirq
> >>> processing (as can be seen by e.g. the watchdog fixes patches).
> >>> Thus, the revert makes sense, but how to regain the "overload"
> >>> protection such that RX networking cannot starve processes reading from
> >>> the socket? (is this what Sebastian's patchset does?)
> >>>
> >>
> >> I'm no expert in sched / softirq area of the kernel, but I'm willing to
> >> help out testing different solution that can regain the "overload"
> >> protection e.g. avoid packet processing "falls-of-an-edge" (and thus
> >> opens the kernel to be DDoS'ed easily).
> >> Is this what Sebastian's patchset does?
> >>
> >>
> >>>
> >>> Thread link for people Cc'ed:
> >>> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r
> >>
> >> --Jesper
> >> (some testlab results below)
> >>
> >> [udp_sink]
> >> https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c
> >>
> >>
> >> When udp_sink runs on same CPU and NAPI/softirq
> >>    - UdpInDatagrams: 2,948 packets/sec
> >>
> >> $ nstat -n && sleep 1 && nstat
> >> #kernel
> >> IpInReceives                    2831056            0.0
> >> IpInDelivers                    2831053            0.0
> >> UdpInDatagrams                  2948               0.0
> >> UdpInErrors                     2828118            0.0
> >> UdpRcvbufErrors                 2828118            0.0
> >> IpExtInOctets                   130206496          0.0
> >> IpExtInNoECTPkts                2830576            0.0
> >>
> >> When udp_sink runs on another CPU than NAPI-RX.
> >>    - UdpInDatagrams: 1,722,307 pps
> >>
> >> $ nstat -n && sleep 1 && nstat
> >> #kernel
> >> IpInReceives                    2318560            0.0
> >> IpInDelivers                    2318562            0.0
> >> UdpInDatagrams                  1722307            0.0
> >> UdpInErrors                     596280             0.0
> >> UdpRcvbufErrors                 596280             0.0
> >> IpExtInOctets                   106634256          0.0
> >> IpExtInNoECTPkts                2318136            0.0
> >>
> >>
> >
> >
>


-- 

Yan

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Jakub Kicinski]

On Fri, 18 Aug 2023 16:57:34 +0200 Sebastian Andrzej Siewior wrote:
> As of now Jakub isn't eager to have it and my testing/ convincing is
> quite limited. If nobody else yells that something like that would be
> helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this
> series.

As tempting as code removal would be, we can still try to explore the
option of letting backlog processing run in threads - as an opt-in on
normal kernels and force it on RT?

But it would be good to wait ~2 weeks before moving forward, if you
don't mind, various core folks keep taking vacations..

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Eric Dumazet]

On Fri, Aug 18, 2023 at 6:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 18 Aug 2023 16:57:34 +0200 Sebastian Andrzej Siewior wrote:
> > As of now Jakub isn't eager to have it and my testing/ convincing is
> > quite limited. If nobody else yells that something like that would be
> > helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this
> > series.
>
> As tempting as code removal would be, we can still try to explore the
> option of letting backlog processing run in threads - as an opt-in on
> normal kernels and force it on RT?

+1

Patch 1/2 as presented is really scary, we would need to test it
extensively on various platforms.

>
> But it would be good to wait ~2 weeks before moving forward, if you
> don't mind, various core folks keep taking vacations..

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Yan Zhai]

On Fri, Aug 18, 2023 at 9:57 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2023-08-18 09:43:08 [-0500], Yan Zhai wrote:
> > > Looking at the cloudflare ppl here in the thread, I doubt they use
> > > backlog but have proper NAPI so they might not need this.
> > >
> > Cloudflare does have backlog usage. On some veths we have to turn GRO
>
> Oh. Okay.
>
> > off to cope with multi-layer encapsulation, and there is also no XDP
> > attached on these interfaces, thus the backlog is used. There are also
> > other usage of backlog, tuntap, loopback and bpf-redirect ingress.
> > Frankly speaking, making a NAPI instance "threaded" itself is not a
> > concern. We have threaded NAPI running on some veth for quite a while,
> > and it performs pretty well. The concern, if any, would be the
> > maturity of new code. I am happy to help derisk with some lab tests
> > and dogfooding if generic agreement is reached to proceed with this
> > idea.
>
> If you have threaded NAPI for veth then you wouldn't be affected by this
> code. However, if you _are_ affected by this and you use veth it would
> be helpful to figure out if you have problems as of net-next and if this
> helps or makes it worse.
>
yes we are still impacted on non-NAPI veths and other scenarios. But
net-next sounds good, still plenty of time to evaluate if it has any
negative impact.

Yan

> As of now Jakub isn't eager to have it and my testing/ convincing is
> quite limited. If nobody else yells that something like that would be
> helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this
> series.
>
> > Yan
>
> Sebastian

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[kernel test robot]

Hello,

kernel test robot noticed a -17.5% regression of phoronix-test-suite.netperf.localhost.UDPRequestResponse.10Seconds.transaction_rate_per_second on:


commit: e9dc861c8b3c291826d6cc1a8674ec0b148cd82d ("[RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.")
url: https://github.com/intel-lab-lkp/linux/commits/Sebastian-Andrzej-Siewior/net-Use-SMP-threads-for-backlog-NAPI/20230814-173835
base: https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git 83b5f0253b1ef352f4333c4fb2d24eff23045f6b
patch link: https://lore.kernel.org/all/20230814093528.117342-2-bigeasy@linutronix.de/
patch subject: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

testcase: phoronix-test-suite
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:

	test: netperf-1.0.4
	cpufreq_governor: performance


In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.udp.ops_per_sec -40.3% regression                                          |
| test machine     | 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory |
| test parameters  | class=os                                                                                        |
|                  | cpufreq_governor=performance                                                                    |
|                  | disk=1HDD                                                                                       |
|                  | fs=ext4                                                                                         |
|                  | nr_threads=1                                                                                    |
|                  | test=udp                                                                                        |
|                  | testtime=60s                                                                                    |
+------------------+-------------------------------------------------------------------------------------------------+
| testcase: change | qperf: qperf.udp.send_bw -24.2% regression                                                      |
| test machine     | 8 threads 1 sockets Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (Haswell) with 8G memory           |
| test parameters  | cluster=cs-localhost                                                                            |
|                  | cpufreq_governor=performance                                                                    |
|                  | runtime=600s                                                                                    |
+------------------+-------------------------------------------------------------------------------------------------+
| testcase: change | uperf: uperf.Runstat_master_Mbps -16.5% regression                                              |
| test machine     | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory       |
| test parameters  | cluster=cs-localhost                                                                            |
|                  | cpufreq_governor=performance                                                                    |
|                  | datasize=64                                                                                     |
|                  | nr_threads=1                                                                                    |
|                  | protocol=UDP                                                                                    |
|                  | runtime=30s                                                                                     |
|                  | winsize=32k                                                                                     |
+------------------+-------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.dccp.ops_per_sec -16.0% regression                                         |
| test machine     | 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory |
| test parameters  | class=os                                                                                        |
|                  | cpufreq_governor=performance                                                                    |
|                  | disk=1HDD                                                                                       |
|                  | fs=ext4                                                                                         |
|                  | nr_threads=1                                                                                    |
|                  | test=dccp                                                                                       |
|                  | testtime=60s                                                                                    |
+------------------+-------------------------------------------------------------------------------------------------+


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202308211627.8b869c6e-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20230821/202308211627.8b869c6e-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-8.3/debian-x86_64-phoronix/lkp-csl-2sp7/netperf-1.0.4/phoronix-test-suite

commit: 
  83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode")
  e9dc861c8b ("net: Use SMP threads for backlog NAPI.")

83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  12507317           -10.7%   11173853        cpuidle..usage
     23.00 ± 14%     +42.0%      32.67 ±  7%  perf-c2c.DRAM.local
      3.91 ±101%      -2.3        1.64 ±149%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
    116234           -17.5%      95851        phoronix-test-suite.netperf.localhost.UDPRequestResponse.10Seconds.transaction_rate_per_second
    237948           +67.3%     398178        vmstat.system.cs
      0.22            -0.1        0.10        mpstat.cpu.all.soft%
      0.53            +0.1        0.63        mpstat.cpu.all.sys%
     20688            +7.4%      22214        proc-vmstat.nr_kernel_stack
     53103            +1.4%      53861        proc-vmstat.nr_slab_unreclaimable
   1991060 ±  8%     +55.7%    3100058        turbostat.C1
      0.14 ±  6%      +0.2        0.31 ±  4%  turbostat.C1%
      0.29 ±  4%      +0.1        0.42 ±  3%  turbostat.C1E%
   5063449 ±  4%     -46.3%    2720011 ±  2%  turbostat.POLL
      0.26 ±  4%      -0.1        0.17 ±  2%  turbostat.POLL%
    114.60            +1.6%     116.47        turbostat.PkgWatt
     48.25 ±  4%      -4.8%      45.93 ±  2%  perf-stat.i.MPKI
 7.498e+08 ±  2%      +8.2%  8.113e+08        perf-stat.i.branch-instructions
  85554181            -4.4%   81810261        perf-stat.i.cache-references
    255667 ±  2%     +66.2%     425026        perf-stat.i.context-switches
      3.23 ±  3%      -4.4%       3.09        perf-stat.i.cpi
    106.09          +172.8%     289.40 ± 22%  perf-stat.i.cpu-migrations
      3590            -7.0%       3340 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.47 ±  5%      -0.1        0.42 ±  2%  perf-stat.i.dTLB-load-miss-rate%
   1930027 ±  2%     -11.0%    1717629 ±  3%  perf-stat.i.dTLB-load-misses
 9.746e+08 ±  2%      +7.7%   1.05e+09        perf-stat.i.dTLB-loads
      0.08 ±  7%      -0.0        0.07 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    172760 ±  4%     -11.6%     152755        perf-stat.i.dTLB-store-misses
 5.334e+08            +7.8%  5.749e+08        perf-stat.i.dTLB-stores
     81.38            -7.8       73.60        perf-stat.i.iTLB-load-miss-rate%
   4442067 ±  2%     -34.2%    2922023        perf-stat.i.iTLB-load-misses
    869196 ±  6%     +32.5%    1151689 ±  2%  perf-stat.i.iTLB-loads
 3.764e+09 ±  2%      +6.7%  4.014e+09        perf-stat.i.instructions
      1237 ±  5%     +26.3%       1563 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.49 ±  2%      +8.8%       0.53        perf-stat.i.ipc
     23.38 ±  2%      +7.9%      25.24        perf-stat.i.metric.M/sec
    120711 ±  5%     +29.5%     156269 ± 14%  perf-stat.i.node-load-misses
     22.77 ±  3%     -10.4%      20.40 ±  2%  perf-stat.overall.MPKI
      3.25 ±  2%      -0.2        3.04        perf-stat.overall.branch-miss-rate%
      2.86 ±  2%      +0.2        3.07 ±  3%  perf-stat.overall.cache-miss-rate%
      1.91 ±  2%      -7.9%       1.75        perf-stat.overall.cpi
      2928 ±  2%      -4.2%       2804 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.20 ±  4%      -0.0        0.16 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  4%      -0.0        0.03        perf-stat.overall.dTLB-store-miss-rate%
     83.63           -11.9       71.73        perf-stat.overall.iTLB-load-miss-rate%
    847.76 ±  2%     +62.0%       1373        perf-stat.overall.instructions-per-iTLB-miss
      0.52            +8.6%       0.57        perf-stat.overall.ipc
 7.356e+08 ±  2%      +8.2%  7.957e+08        perf-stat.ps.branch-instructions
  84014375            -4.4%   80324771        perf-stat.ps.cache-references
    250660 ±  2%     +66.3%     416753        perf-stat.ps.context-switches
 7.033e+09            -1.8%  6.906e+09        perf-stat.ps.cpu-cycles
    104.17          +172.6%     283.93 ± 22%  perf-stat.ps.cpu-migrations
   1896175 ±  2%     -11.0%    1687140 ±  3%  perf-stat.ps.dTLB-load-misses
 9.562e+08 ±  2%      +7.7%   1.03e+09        perf-stat.ps.dTLB-loads
    169740 ±  4%     -11.6%     150033        perf-stat.ps.dTLB-store-misses
 5.232e+08            +7.8%  5.639e+08        perf-stat.ps.dTLB-stores
   4356522 ±  2%     -34.2%    2866282        perf-stat.ps.iTLB-load-misses
    852611 ±  6%     +32.5%    1129564 ±  2%  perf-stat.ps.iTLB-loads
 3.693e+09 ±  2%      +6.6%  3.937e+09        perf-stat.ps.instructions
    118546 ±  5%     +29.4%     153343 ± 14%  perf-stat.ps.node-load-misses
 2.069e+11            +5.0%  2.173e+11        perf-stat.total.instructions


***************************************************************************************************
lkp-csl-d02: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/lkp-csl-d02/udp/stress-ng/60s

commit: 
  83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode")
  e9dc861c8b ("net: Use SMP threads for backlog NAPI.")

83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  12790365 ±  2%     +32.0%   16887592 ±  3%  cpuidle..usage
      4.74            -8.6%       4.33        iostat.cpu.system
     17.33 ± 19%     -45.2%       9.50 ± 28%  perf-c2c.DRAM.local
      1.15 ±  2%      -1.1        0.07        mpstat.cpu.all.soft%
      3.06 ±  2%      +0.6        3.71        mpstat.cpu.all.sys%
      0.47 ±  9%      +0.1        0.55        mpstat.cpu.all.usr%
   3451432           +12.0%    3864232        vmstat.memory.cache
    412424 ±  3%    +198.2%    1230008        vmstat.system.cs
     93962 ±  6%     +21.0%     113707 ±  4%  vmstat.system.in
     19785 ±  5%   +1026.7%     222913 ±  5%  turbostat.C1
      0.06 ±  9%      +0.1        0.12 ±  6%  turbostat.C1%
   6268454 ±  6%     +19.8%    7509215 ±  3%  turbostat.IRQ
   9630603 ±  3%     +40.8%   13558028 ±  4%  turbostat.POLL
      0.25 ±  4%      +0.2        0.46        turbostat.POLL%
      3347 ±  4%  +6.5e+05%   21761039        stress-ng.time.involuntary_context_switches
    121.33           -21.4%      95.33        stress-ng.time.percent_of_cpu_this_job_got
     71.34           -22.6%      55.23        stress-ng.time.system_time
  12757669           +40.4%   17909736 ±  2%  stress-ng.time.voluntary_context_switches
  36420509           -40.3%   21760688        stress-ng.udp.ops
    607006           -40.3%     362677        stress-ng.udp.ops_per_sec
    471964           +81.1%     854856        meminfo.Active
    453420           +85.0%     838688        meminfo.Active(anon)
   3355258           +12.2%    3764944        meminfo.Cached
   1009405           +41.2%    1425669        meminfo.Committed_AS
     57926           +19.5%      69211 ±  2%  meminfo.Mapped
   4156253            +9.9%    4569409        meminfo.Memused
    502429           +82.5%     916869        meminfo.Shmem
    113354           +85.0%     209672        proc-vmstat.nr_active_anon
    842826           +12.2%     945267        proc-vmstat.nr_file_pages
     87951            +8.9%      95743        proc-vmstat.nr_inactive_anon
      8798            +6.2%       9348        proc-vmstat.nr_kernel_stack
     14481           +19.5%      17312 ±  2%  proc-vmstat.nr_mapped
    125608           +82.5%     229221        proc-vmstat.nr_shmem
    113354           +85.0%     209672        proc-vmstat.nr_zone_active_anon
     87951            +8.9%      95743        proc-vmstat.nr_zone_inactive_anon
   1222387           -22.1%     952448        proc-vmstat.numa_hit
   1220389           -22.0%     952356        proc-vmstat.numa_local
    172618           +61.8%     279233 ±  3%  proc-vmstat.pgactivate
   2049577           -32.9%    1375385        proc-vmstat.pgalloc_normal
    228482           +10.4%     252349        proc-vmstat.pgfault
   1847715           -44.3%    1029148        proc-vmstat.pgfree
      0.00 ± 27%     -68.8%       0.00 ± 44%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 41%     -74.5%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00           -58.3%       0.00 ± 44%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      0.00 ±223%    +900.0%       0.00 ± 33%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.00 ± 48%    +130.4%       0.01 ± 13%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.71 ±  2%     -51.3%       0.35 ±  4%  perf-sched.total_wait_and_delay.average.ms
    648341          +176.4%    1792232        perf-sched.total_wait_and_delay.count.ms
      0.71 ±  2%     -51.5%       0.35 ±  4%  perf-sched.total_wait_time.average.ms
    273.60 ±  7%     -34.3%     179.63 ± 28%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     49.49           -43.6%      27.89 ±  3%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.00          +200.0%       0.01        perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
    303.03 ±  4%     -99.8%       0.64 ±  4%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    474.17 ±  3%     +18.8%     563.18 ±  5%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1.50 ± 74%    +322.2%       6.33 ± 23%  perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      2.67 ± 63%    +731.2%      22.17 ± 14%  perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      1.00 ± 81%    +666.7%       7.67 ± 32%  perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
    337.67           +62.6%     549.17 ±  2%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    734.00 ±  3%  +77846.9%     572130        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1491 ± 32%     -33.0%       1000        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.08 ± 44%     -94.7%       0.00 ± 17%  perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
      0.07 ± 28%     -87.0%       0.01 ± 15%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
    273.60 ±  7%     -34.3%     179.63 ± 28%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00 ±223%  +15780.0%       0.13 ± 26%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.09 ± 42%   +2947.0%       2.74 ± 19%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.08 ± 81%  +72717.2%      56.43 ± 69%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.07 ± 30%     -92.0%       0.01 ± 44%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
     49.49           -43.6%      27.89 ±  3%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.00          +150.0%       0.01        perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      0.32 ±  3%     -34.2%       0.21 ± 14%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    303.02 ±  4%     -99.8%       0.63 ±  4%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00 ±223%   +1300.0%       0.00 ± 20%  perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
    474.16 ±  3%     +18.8%     563.18 ±  5%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.26 ± 25%     -97.0%       0.01 ± 80%  perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
      0.44 ± 55%     -91.1%       0.04 ± 76%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1491 ± 32%     -33.0%    1000.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00 ±223%  +73360.0%       0.61 ±  6%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
      0.28 ± 35%   +6429.9%      18.23 ± 14%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.24 ±113%  +4.2e+05%       1000        perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      3.43 ±180%  +19504.1%     672.98 ± 68%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
 1.548e+09 ±  2%     +18.9%   1.84e+09        perf-stat.i.branch-instructions
  22847886 ±  4%      +6.2%   24254832        perf-stat.i.branch-misses
      2.69 ±  6%      +0.4        3.06 ±  3%  perf-stat.i.cache-miss-rate%
  77034877 ±  3%      -5.8%   72570665        perf-stat.i.cache-references
    431302 ±  3%    +199.0%    1289755        perf-stat.i.context-switches
 8.536e+09            +4.6%   8.93e+09        perf-stat.i.cpu-cycles
     99.38          +247.5%     345.39 ±  4%  perf-stat.i.cpu-migrations
     14408 ± 11%    +133.1%      33584 ± 11%  perf-stat.i.cycles-between-cache-misses
 2.218e+09           +11.8%  2.481e+09        perf-stat.i.dTLB-loads
 1.366e+09 ±  2%      +8.2%  1.477e+09        perf-stat.i.dTLB-stores
     91.20            -1.8       89.40        perf-stat.i.iTLB-load-miss-rate%
  11898638           -33.9%    7865700        perf-stat.i.iTLB-load-misses
    998370 ±  7%     -15.0%     848296 ±  7%  perf-stat.i.iTLB-loads
 8.146e+09 ±  2%     +12.6%  9.171e+09        perf-stat.i.instructions
    848.55 ±  4%     +55.5%       1319        perf-stat.i.instructions-per-iTLB-miss
      0.95            +8.1%       1.02        perf-stat.i.ipc
      0.31 ± 44%     -84.8%       0.05 ± 82%  perf-stat.i.major-faults
      0.24            +4.6%       0.25        perf-stat.i.metric.GHz
    144.65 ±  2%     +12.7%     163.06        perf-stat.i.metric.M/sec
      2104 ±  2%     +19.6%       2516        perf-stat.i.minor-faults
    221343 ±  3%     -18.6%     180275 ±  2%  perf-stat.i.node-stores
      2104 ±  2%     +19.6%       2516        perf-stat.i.page-faults
      9.46 ±  2%     -16.3%       7.91 ±  2%  perf-stat.overall.MPKI
      1.48 ±  3%      -0.2        1.32        perf-stat.overall.branch-miss-rate%
      2.26 ±  4%      +0.2        2.50 ±  3%  perf-stat.overall.cache-miss-rate%
      1.05            -7.1%       0.97        perf-stat.overall.cpi
     92.26            -2.0       90.27        perf-stat.overall.iTLB-load-miss-rate%
    684.71           +70.3%       1166        perf-stat.overall.instructions-per-iTLB-miss
      0.95            +7.6%       1.03        perf-stat.overall.ipc
 1.524e+09 ±  2%     +18.9%  1.811e+09        perf-stat.ps.branch-instructions
  22493006 ±  4%      +6.1%   23874609        perf-stat.ps.branch-misses
  75835384 ±  3%      -5.8%   71429667        perf-stat.ps.cache-references
    424565 ±  3%    +199.0%    1269435        perf-stat.ps.context-switches
 8.403e+09            +4.6%   8.79e+09        perf-stat.ps.cpu-cycles
     97.83          +247.5%     339.96 ±  4%  perf-stat.ps.cpu-migrations
 2.184e+09           +11.8%  2.442e+09        perf-stat.ps.dTLB-loads
 1.345e+09            +8.1%  1.454e+09        perf-stat.ps.dTLB-stores
  11713146           -33.9%    7742064        perf-stat.ps.iTLB-load-misses
    982805 ±  7%     -15.0%     834974 ±  7%  perf-stat.ps.iTLB-loads
 8.019e+09 ±  2%     +12.6%  9.027e+09        perf-stat.ps.instructions
      0.31 ± 44%     -84.8%       0.05 ± 82%  perf-stat.ps.major-faults
      2071 ±  2%     +19.6%       2477        perf-stat.ps.minor-faults
    217934 ±  3%     -18.5%     177518 ±  2%  perf-stat.ps.node-stores
      2072 ±  2%     +19.6%       2477        perf-stat.ps.page-faults
 5.177e+11           +11.4%  5.769e+11        perf-stat.total.instructions
     32.52 ±  6%     -20.3       12.19 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
     32.28 ±  6%     -20.2       12.05 ±  2%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
     30.73 ±  6%     -19.6       11.12 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     30.02 ±  6%     -19.2       10.82 ±  2%  perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     33.44 ±  6%     -17.6       15.84 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto
     33.10 ±  6%     -17.5       15.60 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
     34.40 ±  6%     -16.8       17.62 ±  2%  perf-profile.calltrace.cycles-pp.sendto
     14.33 ±  6%     -14.3        0.00        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
     14.25 ±  6%     -14.2        0.00        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb
     18.23 ±  6%     -14.2        4.02 ±  3%  perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     14.11 ±  6%     -14.1        0.00        perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
     17.78 ±  6%     -13.9        3.85 ±  4%  perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     16.84 ±  6%     -13.4        3.39 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     16.36 ±  6%     -13.2        3.19 ±  4%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
     13.08 ±  7%     -13.1        0.00        perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
     12.78 ±  7%     -12.8        0.00        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
     12.74 ±  7%     -12.7        0.00        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq
     11.96 ±  6%     -12.0        0.00        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq
     11.06 ±  6%     -11.1        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      8.47 ±  6%      -3.4        5.02        perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      5.81 ±  7%      -2.4        3.37 ±  2%  perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     10.99 ±  6%      -2.4        8.57 ±  5%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     10.64 ±  6%      -2.2        8.41 ±  4%  perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      4.35 ± 10%      -2.0        2.37 ±  6%  perf-profile.calltrace.cycles-pp.ioctl
      9.63 ±  6%      -1.8        7.87 ±  4%  perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      3.48 ± 10%      -1.7        1.82 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ioctl
      3.37 ±  9%      -1.6        1.75 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
      9.38 ±  7%      -1.6        7.78 ±  4%  perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      3.19 ± 10%      -1.6        1.63 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
      1.83 ± 12%      -1.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.sk_ioctl.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl
      2.23 ± 12%      -1.4        0.78 ±  9%  perf-profile.calltrace.cycles-pp.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64
      2.84 ± 11%      -1.4        1.42 ±  6%  perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
      2.10 ± 12%      -1.4        0.71 ± 10%  perf-profile.calltrace.cycles-pp.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl
      2.45 ± 12%      -1.4        1.08 ± 26%  perf-profile.calltrace.cycles-pp.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.02 ±  8%      -1.3        1.76 ±  4%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      1.58 ± 13%      -1.0        0.62 ± 10%  perf-profile.calltrace.cycles-pp.udp_rmem_release.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg
      2.24 ±  5%      -1.0        1.29 ±  5%  perf-profile.calltrace.cycles-pp.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      2.89 ±  7%      -0.9        1.96 ±  6%  perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
      2.70 ±  7%      -0.9        1.80 ±  6%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg
      1.97 ±  4%      -0.9        1.08 ±  4%  perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      2.35 ±  7%      -0.9        1.47 ±  3%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      7.34 ±  7%      -0.8        6.49 ±  4%  perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
      2.58 ±  7%      -0.8        1.74 ±  7%  perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyout._copy_to_iter.udp_recvmsg.inet_recvmsg
      1.87 ±  7%      -0.8        1.07 ±  6%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      1.67 ±  5%      -0.8        0.92 ±  3%  perf-profile.calltrace.cycles-pp.__mkroute_output.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.78 ±  7%      -0.8        1.03 ±  5%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      3.44 ± 10%      -0.7        2.76 ±  5%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.76 ±  8%      -0.6        1.10 ±  2%  perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg
      3.19 ± 10%      -0.6        2.54 ±  5%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      1.43 ±  6%      -0.6        0.80 ±  7%  perf-profile.calltrace.cycles-pp.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg
      1.07 ± 11%      -0.6        0.46 ± 71%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.02 ± 11%      -0.6        0.42 ± 72%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.07 ±  7%      -0.6        0.49 ± 45%  perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb
      1.54 ±  7%      -0.6        0.98 ±  9%  perf-profile.calltrace.cycles-pp.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
      1.51 ±  7%      -0.5        0.97 ±  9%  perf-profile.calltrace.cycles-pp.skb_release_data.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.sock_recvmsg
      1.02 ± 11%      -0.4        0.61 ±  6%  perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      1.10 ±  8%      -0.4        0.70 ±  9%  perf-profile.calltrace.cycles-pp.__slab_free.skb_release_data.__consume_stateless_skb.udp_recvmsg.inet_recvmsg
      0.98 ±  8%      -0.4        0.60 ± 12%  perf-profile.calltrace.cycles-pp.ipv4_pktinfo_prepare.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
      0.85 ±  9%      -0.2        0.61 ±  8%  perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.74 ± 13%      +0.2        0.92 ±  9%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.73 ± 15%      +0.2        0.98 ± 12%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.48 ± 47%      +0.3        0.75 ± 14%  perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter
      0.96 ± 12%      +0.3        1.28 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
      1.06 ± 12%      +0.3        1.39 ±  8%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
      0.99 ± 12%      +0.3        1.33 ±  8%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
      0.18 ±141%      +0.4        0.56 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
      0.19 ±142%      +0.5        0.65 ± 14%  perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state
      1.43 ±  7%      +0.5        1.90 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.52 ±  8%      +0.5        2.00 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.10 ±223%      +0.5        0.64 ± 15%  perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle
      1.00 ±  9%      +0.5        1.54 ±  5%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      1.76 ± 11%      +0.6        2.34 ± 13%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      0.00            +0.6        0.59 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00            +0.6        0.60 ±  5%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn
      1.14 ±  9%      +0.6        1.75 ±  5%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      2.36 ± 10%      +0.7        3.02 ±  9%  perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.7        0.69 ±  7%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
      2.58 ± 10%      +0.7        3.32 ±  9%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +0.8        0.79 ±  7%  perf-profile.calltrace.cycles-pp.__switch_to_asm
      0.00            +0.8        0.81 ±  7%  perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
      0.00            +0.8        0.84 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog
      0.00            +0.9        0.86 ±  7%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
      0.00            +0.9        0.88 ±  7%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal
      0.00            +0.9        0.92 ±  6%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      0.00            +0.9        0.94 ±  6%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
      0.00            +0.9        0.94 ±  4%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
      0.00            +1.1        1.06 ±  7%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      0.65 ± 10%      +1.1        1.72 ±  2%  perf-profile.calltrace.cycles-pp.clock_nanosleep
      0.47 ± 45%      +1.2        1.63 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.48 ± 45%      +1.2        1.64 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.00            +1.2        1.19 ±  6%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.do_nanosleep
      0.00            +1.2        1.20 ±  6%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.do_nanosleep.hrtimer_nanosleep
      0.37 ± 71%      +1.3        1.62 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      2.29 ±  7%      +1.3        3.57 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      0.00            +1.3        1.29 ±  5%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.27 ±100%      +1.3        1.59 ±  2%  perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.27 ±100%      +1.3        1.59 ±  2%  perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.32 ±  5%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx
      2.37 ±  8%      +1.4        3.72 ±  5%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      0.00            +1.4        1.43 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep
      0.00            +1.4        1.44 ±  3%  perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.09 ±223%      +1.5        1.57 ±  3%  perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64
      1.19 ±  8%      +1.5        2.68 ±  3%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
      1.06 ±  7%      +1.5        2.60 ±  3%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb
      2.82 ±  6%      +1.6        4.39 ±  5%  perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      4.30 ±  7%      +1.9        6.15 ±  4%  perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg
      0.00            +2.0        1.95 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit
      0.00            +2.2        2.17 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit
      0.00            +2.2        2.19 ±  3%  perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.00            +2.2        2.20 ±  3%  perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
     32.32 ±  2%      +2.3       34.61 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +2.5        2.45 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +2.6        2.55 ±  3%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      3.23 ± 10%      +2.6        5.82 ±  7%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +2.7        2.72 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.0        3.00 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      0.00            +3.0        3.04 ±  4%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +3.2        3.16 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      0.00            +3.2        3.20 ±  4%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     28.90 ±  6%      +3.4       32.30 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     40.00 ±  2%      +4.9       44.94        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     35.98 ±  5%      +5.3       41.24 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     36.05 ±  5%      +5.3       41.34 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     36.06 ±  5%      +5.3       41.36 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
      0.00            +8.6        8.61 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop
      0.00            +9.1        9.13 ±  4%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn
      0.00            +9.6        9.60 ±  4%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread
      0.00            +9.6        9.63 ±  4%  perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +9.8        9.84 ±  4%  perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00           +14.1       14.09 ±  4%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00           +14.2       14.16 ±  3%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      0.00           +14.2       14.16 ±  3%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      0.00           +14.2       14.16 ±  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     32.55 ±  6%     -20.4       12.20 ±  2%  perf-profile.children.cycles-pp.__x64_sys_sendto
     32.31 ±  6%     -20.3       12.06 ±  2%  perf-profile.children.cycles-pp.__sys_sendto
     54.70           -20.2       34.52        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     54.00           -20.0       34.04        perf-profile.children.cycles-pp.do_syscall_64
     30.75 ±  6%     -19.6       11.13 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
     30.07 ±  6%     -19.2       10.83 ±  2%  perf-profile.children.cycles-pp.udp_sendmsg
     34.63 ±  6%     -16.9       17.77 ±  2%  perf-profile.children.cycles-pp.sendto
     15.74 ±  5%     -14.6        1.10 ± 10%  perf-profile.children.cycles-pp.__do_softirq
     14.47 ±  6%     -14.3        0.17 ± 29%  perf-profile.children.cycles-pp.__local_bh_enable_ip
     14.26 ±  6%     -14.3        0.01 ±223%  perf-profile.children.cycles-pp.do_softirq
     18.24 ±  6%     -14.2        4.03 ±  3%  perf-profile.children.cycles-pp.udp_send_skb
     17.80 ±  6%     -13.9        3.86 ±  4%  perf-profile.children.cycles-pp.ip_send_skb
     16.84 ±  6%     -13.5        3.39 ±  4%  perf-profile.children.cycles-pp.ip_finish_output2
     16.38 ±  6%     -13.2        3.20 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
     13.10 ±  7%     -13.1        0.00        perf-profile.children.cycles-pp.net_rx_action
      8.49 ±  6%      -3.5        5.03        perf-profile.children.cycles-pp.ip_make_skb
     12.78 ±  7%      -3.1        9.64 ±  4%  perf-profile.children.cycles-pp.__napi_poll
     12.74 ±  7%      -3.1        9.61 ±  4%  perf-profile.children.cycles-pp.process_backlog
     11.98 ±  6%      -2.8        9.14 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     11.06 ±  6%      -2.5        8.61 ±  4%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      5.82 ±  6%      -2.4        3.37 ±  2%  perf-profile.children.cycles-pp.__ip_append_data
     11.00 ±  6%      -2.4        8.58 ±  5%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     10.66 ±  6%      -2.2        8.42 ±  5%  perf-profile.children.cycles-pp.__udp4_lib_rcv
      4.58 ± 10%      -2.1        2.53 ±  6%  perf-profile.children.cycles-pp.ioctl
      9.64 ±  6%      -1.8        7.88 ±  4%  perf-profile.children.cycles-pp.udp_unicast_rcv_skb
      9.42 ±  7%      -1.6        7.80 ±  4%  perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
      3.19 ± 10%      -1.6        1.63 ±  4%  perf-profile.children.cycles-pp.__x64_sys_ioctl
      2.26 ± 12%      -1.4        0.81 ±  9%  perf-profile.children.cycles-pp.sock_do_ioctl
      2.84 ± 11%      -1.4        1.42 ±  6%  perf-profile.children.cycles-pp.do_vfs_ioctl
      2.13 ± 12%      -1.4        0.72 ± 10%  perf-profile.children.cycles-pp.inet_ioctl
      2.49 ± 11%      -1.4        1.10 ± 25%  perf-profile.children.cycles-pp.sock_ioctl
      1.85 ± 12%      -1.3        0.52 ±  9%  perf-profile.children.cycles-pp.sk_ioctl
      3.03 ±  8%      -1.3        1.76 ±  5%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.53 ± 11%      -1.2        0.35 ± 13%  perf-profile.children.cycles-pp.udp_ioctl
      1.48 ± 11%      -1.2        0.31 ± 14%  perf-profile.children.cycles-pp.first_packet_length
      3.86 ±  8%      -1.1        2.73 ±  5%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      4.14 ±  8%      -1.1        3.03 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      3.11 ±  6%      -1.0        2.08 ±  5%  perf-profile.children.cycles-pp.rep_movs_alternative
      1.58 ± 13%      -1.0        0.62 ± 10%  perf-profile.children.cycles-pp.udp_rmem_release
      2.25 ±  6%      -1.0        1.30 ±  5%  perf-profile.children.cycles-pp.ip_generic_getfrag
      2.90 ±  7%      -0.9        1.96 ±  6%  perf-profile.children.cycles-pp._copy_to_iter
      2.78 ±  7%      -0.9        1.85 ±  7%  perf-profile.children.cycles-pp.copyout
      2.38 ±  7%      -0.9        1.50 ±  3%  perf-profile.children.cycles-pp.__ip_make_skb
      1.98 ±  4%      -0.9        1.09 ±  4%  perf-profile.children.cycles-pp.ip_route_output_flow
      2.05 ±  5%      -0.9        1.18 ±  6%  perf-profile.children.cycles-pp.__check_object_size
      1.62 ±  6%      -0.9        0.75 ± 14%  perf-profile.children.cycles-pp.__irq_exit_rcu
      7.35 ±  7%      -0.9        6.49 ±  4%  perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
      2.10 ±  7%      -0.8        1.28 ±  9%  perf-profile.children.cycles-pp.__slab_free
      1.88 ±  7%      -0.8        1.07 ±  6%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      1.56 ±  6%      -0.8        0.77 ±  8%  perf-profile.children.cycles-pp.rcu_do_batch
      1.48 ±  6%      -0.8        0.72 ±  8%  perf-profile.children.cycles-pp.dst_destroy
      1.59 ±  7%      -0.8        0.84 ±  8%  perf-profile.children.cycles-pp.rcu_core
      1.79 ±  8%      -0.8        1.04 ±  5%  perf-profile.children.cycles-pp.__alloc_skb
      1.67 ±  5%      -0.8        0.92 ±  3%  perf-profile.children.cycles-pp.__mkroute_output
      1.76 ±  8%      -0.7        1.11 ±  3%  perf-profile.children.cycles-pp.__ip_select_ident
      1.28 ±  6%      -0.6        0.69 ±  7%  perf-profile.children.cycles-pp.check_heap_object
      1.54 ±  7%      -0.6        0.98 ±  9%  perf-profile.children.cycles-pp.__consume_stateless_skb
      1.51 ±  7%      -0.5        0.97 ±  9%  perf-profile.children.cycles-pp.skb_release_data
      0.62 ±  4%      -0.5        0.14 ± 11%  perf-profile.children.cycles-pp.irqtime_account_irq
      1.40 ±  8%      -0.4        0.97 ± 11%  perf-profile.children.cycles-pp.__entry_text_start
      1.02 ± 11%      -0.4        0.61 ±  6%  perf-profile.children.cycles-pp.skb_set_owner_w
      1.23 ± 11%      -0.4        0.82 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.99 ±  8%      -0.4        0.60 ± 11%  perf-profile.children.cycles-pp.ipv4_pktinfo_prepare
      0.84 ±  7%      -0.4        0.47 ±  7%  perf-profile.children.cycles-pp.rt_dst_alloc
      0.72 ± 10%      -0.4        0.35 ± 15%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.82 ± 14%      -0.4        0.47 ± 11%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.76 ±  7%      -0.3        0.43 ±  7%  perf-profile.children.cycles-pp.dst_alloc
      0.74 ± 10%      -0.3        0.45 ±  6%  perf-profile.children.cycles-pp._copy_from_iter
      0.34 ± 20%      -0.3        0.05 ± 51%  perf-profile.children.cycles-pp.__first_packet_length
      0.56 ± 10%      -0.3        0.28 ± 11%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.77 ±  8%      -0.3        0.49 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.65 ± 14%      -0.3        0.39 ±  9%  perf-profile.children.cycles-pp.stress_udp_client
      0.67 ±  3%      -0.3        0.40 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.49 ± 11%      -0.2        0.25 ± 18%  perf-profile.children.cycles-pp.udp4_lib_lookup2
      0.63 ±  9%      -0.2        0.39 ±  6%  perf-profile.children.cycles-pp.copyin
      0.86 ±  9%      -0.2        0.62 ±  9%  perf-profile.children.cycles-pp.move_addr_to_user
      0.54 ±  5%      -0.2        0.30 ±  7%  perf-profile.children.cycles-pp.rt_set_nexthop
      0.51 ±  9%      -0.2        0.28 ±  7%  perf-profile.children.cycles-pp.ip_rcv
      0.53 ±  8%      -0.2        0.30 ± 23%  perf-profile.children.cycles-pp.aa_sk_perm
      0.52 ± 18%      -0.2        0.30 ± 14%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.40 ±  7%      -0.2        0.18 ± 10%  perf-profile.children.cycles-pp.ipv4_dst_destroy
      0.51 ± 12%      -0.2        0.30 ±  8%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.44 ± 13%      -0.2        0.23 ± 14%  perf-profile.children.cycles-pp.dst_release
      0.42 ±  9%      -0.2        0.22 ± 12%  perf-profile.children.cycles-pp.___slab_alloc
      0.45 ± 20%      -0.2        0.26 ± 16%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.46 ± 10%      -0.2        0.26 ± 13%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.33 ±  6%      -0.2        0.15 ± 28%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.55 ±  8%      -0.2        0.37 ± 10%  perf-profile.children.cycles-pp.__call_rcu_common
      0.38 ±  8%      -0.2        0.20 ± 10%  perf-profile.children.cycles-pp.sock_wfree
      0.48 ± 12%      -0.2        0.31 ± 13%  perf-profile.children.cycles-pp.__fget_light
      0.40 ± 12%      -0.2        0.23 ± 11%  perf-profile.children.cycles-pp._copy_from_user
      0.35 ± 13%      -0.2        0.19 ±  9%  perf-profile.children.cycles-pp.kmem_cache_free
      0.43 ±  7%      -0.2        0.27 ± 10%  perf-profile.children.cycles-pp.__check_heap_object
      0.36 ± 16%      -0.2        0.20 ± 22%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.26 ± 20%      -0.2        0.12 ± 54%  perf-profile.children.cycles-pp.compute_score
      0.36 ± 17%      -0.1        0.22 ± 13%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.32 ± 17%      -0.1        0.18 ± 14%  perf-profile.children.cycles-pp.ip_output
      0.40 ± 10%      -0.1        0.27 ±  9%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.38 ± 14%      -0.1        0.26 ± 13%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.26 ± 11%      -0.1        0.14 ± 12%  perf-profile.children.cycles-pp.__ip_local_out
      0.16 ±  9%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.allocate_slab
      0.28 ± 18%      -0.1        0.17 ± 14%  perf-profile.children.cycles-pp.netif_skb_features
      0.23 ± 18%      -0.1        0.12 ± 23%  perf-profile.children.cycles-pp.rcuref_put_slowpath
      0.29 ±  9%      -0.1        0.18 ± 17%  perf-profile.children.cycles-pp.siphash_3u32
      0.24 ± 10%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.ip_rcv_core
      0.32 ± 12%      -0.1        0.22 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.21 ± 17%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.__put_user_4
      0.18 ± 25%      -0.1        0.09 ±  9%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.22 ± 10%      -0.1        0.13 ± 18%  perf-profile.children.cycles-pp.__cond_resched
      0.28 ±  9%      -0.1        0.19 ± 12%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.19 ± 18%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.18 ± 11%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.23 ± 13%      -0.1        0.16 ± 14%  perf-profile.children.cycles-pp._copy_to_user
      0.18 ± 12%      -0.1        0.10 ± 16%  perf-profile.children.cycles-pp.ip_send_check
      0.20 ±  8%      -0.1        0.13 ± 24%  perf-profile.children.cycles-pp.__get_user_4
      0.10 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.21 ± 15%      -0.1        0.14 ± 19%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.18 ± 15%      -0.1        0.11 ± 26%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.13 ± 20%      -0.1        0.06 ± 46%  perf-profile.children.cycles-pp.xfrm_lookup_route
      0.11 ± 32%      -0.1        0.05 ± 46%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.15 ± 13%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.ipv4_mtu
      0.14 ± 15%      -0.1        0.08 ± 17%  perf-profile.children.cycles-pp.ip_setup_cork
      0.14 ± 27%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.08 ± 33%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.udp4_hwcsum
      0.08 ± 23%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__build_skb_around
      0.20 ±  8%      -0.1        0.14 ± 17%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.10 ± 26%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp.xfrm_lookup_with_ifid
      0.14 ± 15%      -0.1        0.08 ± 25%  perf-profile.children.cycles-pp.rcu_all_qs
      0.12 ± 18%      -0.1        0.07 ± 22%  perf-profile.children.cycles-pp.import_single_range
      0.16 ± 16%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.udp4_csum_init
      0.08 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.nf_hook_slow
      0.14 ±  9%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.__ip_finish_output
      0.14 ± 18%      -0.1        0.08 ± 16%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.09 ± 17%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.kmalloc_slab
      0.17 ± 17%      +0.0        0.22 ±  6%  perf-profile.children.cycles-pp.local_clock_noinstr
      0.00            +0.1        0.05 ± 13%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.48 ±  4%      +0.1        0.54 ±  6%  perf-profile.children.cycles-pp.ktime_get
      0.06 ± 51%      +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.18 ± 11%      +0.1        0.24 ±  8%  perf-profile.children.cycles-pp.reweight_entity
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.perf_exclude_event
      0.02 ±141%      +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.rb_next
      0.09 ± 17%      +0.1        0.16 ± 25%  perf-profile.children.cycles-pp.__list_add_valid
      0.06 ± 48%      +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.update_blocked_averages
      0.07 ± 19%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.rb_insert_color
      0.04 ± 73%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.check_spread
      0.00            +0.1        0.08 ± 29%  perf-profile.children.cycles-pp.clear_buddies
      0.13 ± 17%      +0.1        0.22 ± 12%  perf-profile.children.cycles-pp.place_entity
      0.00            +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.intel_idle_irq
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.09 ±  5%  perf-profile.children.cycles-pp._find_next_and_bit
      0.02 ±141%      +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.rb_erase
      0.24 ± 15%      +0.1        0.36 ± 11%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.22 ± 12%      +0.1        0.34 ± 12%  perf-profile.children.cycles-pp.update_cfs_group
      0.05 ± 47%      +0.1        0.17 ± 22%  perf-profile.children.cycles-pp.update_min_vruntime
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.backlog_napi_should_run
      0.00            +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.idle_cpu
      0.34 ±  9%      +0.1        0.48 ±  6%  perf-profile.children.cycles-pp.llist_reverse_order
      0.00            +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.resched_curr
      0.12 ±  6%      +0.1        0.26 ± 12%  perf-profile.children.cycles-pp.__calc_delta
      0.17 ± 14%      +0.1        0.32 ±  5%  perf-profile.children.cycles-pp.switch_fpu_return
      0.61 ± 10%      +0.2        0.76 ±  6%  perf-profile.children.cycles-pp.select_task_rq
      0.06 ± 46%      +0.2        0.21 ±  9%  perf-profile.children.cycles-pp.pick_next_entity
      0.09 ± 22%      +0.2        0.26 ±  8%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.10 ± 15%      +0.2        0.28 ± 11%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.00            +0.2        0.18 ± 12%  perf-profile.children.cycles-pp.cpu_util
      0.17 ±  6%      +0.2        0.35 ±  8%  perf-profile.children.cycles-pp.update_rq_clock_task
      1.31 ±  6%      +0.2        1.50 ±  4%  perf-profile.children.cycles-pp.menu_select
      0.00            +0.2        0.21 ± 22%  perf-profile.children.cycles-pp.cpuacct_charge
      0.21 ± 14%      +0.3        0.46 ±  8%  perf-profile.children.cycles-pp.available_idle_cpu
      0.16 ± 19%      +0.3        0.50 ±  8%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.00            +0.4        0.35 ± 10%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.35 ±  8%      +0.4        0.72 ± 16%  perf-profile.children.cycles-pp.nohz_run_idle_balance
      0.06 ± 55%      +0.4        0.43 ± 15%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.19 ± 15%      +0.4        0.57 ±  6%  perf-profile.children.cycles-pp.os_xsave
      1.53 ±  4%      +0.4        1.92        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.62 ± 17%      +0.4        1.05 ± 15%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.20 ± 21%      +0.4        0.64 ± 13%  perf-profile.children.cycles-pp.finish_task_switch
      0.06 ± 47%      +0.4        0.51 ± 13%  perf-profile.children.cycles-pp.check_preempt_curr
      0.00            +0.5        0.47 ± 10%  perf-profile.children.cycles-pp.wake_affine
      0.72 ± 18%      +0.5        1.21 ± 15%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.42 ± 10%      +0.5        0.93 ±  6%  perf-profile.children.cycles-pp.update_rq_clock
      0.30 ± 11%      +0.5        0.85 ±  6%  perf-profile.children.cycles-pp.___perf_sw_event
      0.40 ± 11%      +0.6        1.01 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.36 ± 14%      +0.6        0.97 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.97 ± 18%      +0.6        1.62 ± 14%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      1.58 ±  6%      +0.7        2.25 ±  5%  perf-profile.children.cycles-pp.schedule_idle
      0.42 ±  5%      +0.7        1.16 ±  4%  perf-profile.children.cycles-pp.prepare_task_switch
      0.39 ±  8%      +0.8        1.22 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.09 ± 19%      +0.9        0.99 ±  6%  perf-profile.children.cycles-pp.put_prev_entity
      0.62 ± 15%      +0.9        1.56 ±  7%  perf-profile.children.cycles-pp.__switch_to
      0.08 ± 80%      +0.9        1.02 ±  5%  perf-profile.children.cycles-pp.update_sg_lb_stats
      1.06 ±  6%      +1.0        2.04 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      2.31 ±  9%      +1.0        3.31 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.12 ± 44%      +1.0        1.12 ±  5%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.29 ± 16%      +1.0        1.30 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.13 ± 41%      +1.0        1.14 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      2.70 ± 11%      +1.0        3.71 ±  6%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.57 ±  9%      +1.0        1.62 ±  3%  perf-profile.children.cycles-pp.__switch_to_asm
      0.66 ± 11%      +1.1        1.74 ±  2%  perf-profile.children.cycles-pp.clock_nanosleep
      0.48 ± 12%      +1.1        1.59 ±  2%  perf-profile.children.cycles-pp.hrtimer_nanosleep
      0.50 ± 12%      +1.1        1.60 ±  2%  perf-profile.children.cycles-pp.common_nsleep
      1.14 ±  9%      +1.1        2.25 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      0.52 ± 11%      +1.1        1.62 ±  2%  perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep
      0.46 ± 13%      +1.1        1.57 ±  3%  perf-profile.children.cycles-pp.do_nanosleep
      0.17 ± 35%      +1.1        1.31 ±  7%  perf-profile.children.cycles-pp.load_balance
      1.29 ±  9%      +1.3        2.59 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.10 ± 23%      +1.3        1.43 ±  4%  perf-profile.children.cycles-pp.newidle_balance
      3.08 ±  8%      +1.3        4.40 ±  4%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      1.35 ±  7%      +1.5        2.82        perf-profile.children.cycles-pp.enqueue_task_fair
      1.20 ±  8%      +1.5        2.68 ±  3%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.09 ±  8%      +1.5        2.61 ±  3%  perf-profile.children.cycles-pp.loopback_xmit
      1.38 ±  7%      +1.5        2.92        perf-profile.children.cycles-pp.activate_task
      2.92 ±  7%      +1.6        4.54 ±  5%  perf-profile.children.cycles-pp.schedule_timeout
      1.02 ± 10%      +1.8        2.81        perf-profile.children.cycles-pp.update_load_avg
      0.38 ±  9%      +1.8        2.20 ±  4%  perf-profile.children.cycles-pp.__netif_rx
      0.37 ± 11%      +1.8        2.19 ±  3%  perf-profile.children.cycles-pp.netif_rx_internal
      0.34 ±  9%      +1.8        2.17 ±  3%  perf-profile.children.cycles-pp.enqueue_to_backlog
      4.30 ±  7%      +1.9        6.16 ±  4%  perf-profile.children.cycles-pp.__skb_wait_for_more_packets
      1.49 ±  8%      +2.0        3.53 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
     32.57 ±  2%      +2.3       34.84 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     32.55 ±  2%      +2.3       34.82 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     34.42 ±  2%      +2.5       36.94 ±  2%  perf-profile.children.cycles-pp.cpuidle_idle_call
      3.54 ± 10%      +2.6        6.09 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      3.59 ±  8%      +2.6        6.24 ±  6%  perf-profile.children.cycles-pp.poll_idle
      0.00            +2.7        2.73 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.67 ±  7%      +2.9        3.58        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.32 ±  9%      +3.0        3.32 ±  2%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.63 ±  7%      +3.7        4.37        perf-profile.children.cycles-pp.pick_next_task_fair
     39.95 ±  2%      +4.9       44.84        perf-profile.children.cycles-pp.do_idle
     40.00 ±  2%      +4.9       44.94        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     40.00 ±  2%      +4.9       44.94        perf-profile.children.cycles-pp.cpu_startup_entry
     36.06 ±  5%      +5.3       41.36 ±  5%  perf-profile.children.cycles-pp.start_secondary
      2.72 ±  8%      +8.3       10.98        perf-profile.children.cycles-pp.schedule
      4.13 ±  7%      +8.6       12.74        perf-profile.children.cycles-pp.__schedule
      0.00            +9.8        9.85 ±  4%  perf-profile.children.cycles-pp.napi_threaded_poll_loop
      0.13 ± 23%     +14.0       14.16 ±  3%  perf-profile.children.cycles-pp.kthread
      0.13 ± 21%     +14.0       14.16 ±  3%  perf-profile.children.cycles-pp.ret_from_fork
      0.13 ± 21%     +14.0       14.16 ±  3%  perf-profile.children.cycles-pp.ret_from_fork_asm
      0.07 ± 52%     +14.0       14.10 ±  4%  perf-profile.children.cycles-pp.smpboot_thread_fn
      2.94 ±  6%      -1.0        1.98 ±  5%  perf-profile.self.cycles-pp.rep_movs_alternative
      1.48 ± 12%      -0.9        0.61 ± 10%  perf-profile.self.cycles-pp.udp_rmem_release
      2.10 ±  7%      -0.8        1.27 ±  9%  perf-profile.self.cycles-pp.__slab_free
      1.22 ±  7%      -0.6        0.57 ±  5%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      1.30 ± 10%      -0.6        0.72 ± 11%  perf-profile.self.cycles-pp.sock_def_readable
      1.17 ±  8%      -0.6        0.58 ±  6%  perf-profile.self.cycles-pp.udp_sendmsg
      1.46 ±  9%      -0.5        0.92 ±  2%  perf-profile.self.cycles-pp.__ip_select_ident
      1.06 ±  7%      -0.5        0.57 ±  5%  perf-profile.self.cycles-pp.check_heap_object
      1.21 ± 11%      -0.4        0.81 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      1.00 ± 12%      -0.4        0.61 ±  6%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.41 ± 19%      -0.3        0.09 ± 30%  perf-profile.self.cycles-pp.first_packet_length
      0.34 ± 20%      -0.3        0.05 ± 49%  perf-profile.self.cycles-pp.__first_packet_length
      0.73 ±  7%      -0.3        0.46 ±  2%  perf-profile.self.cycles-pp.__sys_sendto
      0.70 ± 20%      -0.3        0.42 ± 20%  perf-profile.self.cycles-pp.udp_recvmsg
      0.74 ±  9%      -0.3        0.48 ±  6%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.61 ± 17%      -0.2        0.37 ±  7%  perf-profile.self.cycles-pp.stress_udp_client
      0.40 ±  8%      -0.2        0.16 ± 18%  perf-profile.self.cycles-pp.ip_finish_output2
      0.35 ±  9%      -0.2        0.13 ± 14%  perf-profile.self.cycles-pp.udp_send_skb
      0.46 ±  9%      -0.2        0.25 ± 25%  perf-profile.self.cycles-pp.aa_sk_perm
      0.70 ±  3%      -0.2        0.49 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.51 ±  6%      -0.2        0.30 ±  8%  perf-profile.self.cycles-pp.__ip_append_data
      0.32 ±  5%      -0.2        0.12 ± 19%  perf-profile.self.cycles-pp.sock_sendmsg
      0.72 ± 15%      -0.2        0.54 ± 14%  perf-profile.self.cycles-pp.__entry_text_start
      0.51 ± 13%      -0.2        0.33 ± 12%  perf-profile.self.cycles-pp.__ip_make_skb
      0.33 ±  6%      -0.2        0.16 ±  9%  perf-profile.self.cycles-pp.process_backlog
      0.37 ±  8%      -0.2        0.20 ± 10%  perf-profile.self.cycles-pp.sock_wfree
      0.47 ± 12%      -0.2        0.31 ± 13%  perf-profile.self.cycles-pp.__fget_light
      0.34 ± 12%      -0.2        0.18 ± 10%  perf-profile.self.cycles-pp.kmem_cache_free
      0.43 ±  7%      -0.2        0.27 ± 10%  perf-profile.self.cycles-pp.__check_heap_object
      0.36 ± 12%      -0.2        0.20 ± 12%  perf-profile.self.cycles-pp._copy_from_user
      0.35 ± 16%      -0.2        0.20 ± 22%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.26 ± 20%      -0.2        0.12 ± 54%  perf-profile.self.cycles-pp.compute_score
      0.39 ±  8%      -0.1        0.25 ± 10%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.31 ± 15%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.31 ± 24%      -0.1        0.17 ± 26%  perf-profile.self.cycles-pp.udp_queue_rcv_one_skb
      0.30 ± 12%      -0.1        0.16 ± 11%  perf-profile.self.cycles-pp.__mkroute_output
      0.34 ± 15%      -0.1        0.20 ±  7%  perf-profile.self.cycles-pp.__alloc_skb
      0.34 ±  5%      -0.1        0.21 ± 13%  perf-profile.self.cycles-pp.__call_rcu_common
      0.22 ± 13%      -0.1        0.09 ± 27%  perf-profile.self.cycles-pp.dst_destroy
      0.30 ± 10%      -0.1        0.18 ± 26%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.22 ± 14%      -0.1        0.10 ± 24%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.38 ± 11%      -0.1        0.26 ± 10%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.27 ± 20%      -0.1        0.16 ± 21%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.32 ± 13%      -0.1        0.22 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.24 ± 11%      -0.1        0.13 ± 13%  perf-profile.self.cycles-pp.ip_rcv_core
      0.22 ± 20%      -0.1        0.12 ± 21%  perf-profile.self.cycles-pp.rcuref_put_slowpath
      0.65 ±  7%      -0.1        0.55 ±  8%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.29 ± 12%      -0.1        0.20 ± 16%  perf-profile.self.cycles-pp.sendto
      0.23 ± 13%      -0.1        0.13 ± 14%  perf-profile.self.cycles-pp.copyout
      0.27 ±  8%      -0.1        0.18 ± 15%  perf-profile.self.cycles-pp.siphash_3u32
      0.23 ± 14%      -0.1        0.14 ± 13%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.22 ± 15%      -0.1        0.13 ± 25%  perf-profile.self.cycles-pp.__check_object_size
      0.23 ± 23%      -0.1        0.14 ± 15%  perf-profile.self.cycles-pp.ip_output
      0.26 ± 14%      -0.1        0.17 ± 14%  perf-profile.self.cycles-pp.copyin
      0.21 ± 18%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.__put_user_4
      0.27 ± 10%      -0.1        0.19 ± 14%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.19 ± 18%      -0.1        0.11 ±  8%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.12 ± 33%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.18 ± 11%      -0.1        0.10 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.28 ± 12%      -0.1        0.20 ± 20%  perf-profile.self.cycles-pp.inet_ioctl
      0.16 ± 18%      -0.1        0.09 ± 17%  perf-profile.self.cycles-pp.__udp4_lib_rcv
      0.11 ± 10%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.rt_set_nexthop
      0.19 ± 13%      -0.1        0.11 ± 20%  perf-profile.self.cycles-pp.dst_release
      0.16 ± 12%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.netif_skb_features
      0.23 ± 12%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.___slab_alloc
      0.23 ± 18%      -0.1        0.15 ±  9%  perf-profile.self.cycles-pp.skb_release_data
      0.22 ± 11%      -0.1        0.14 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.20 ±  8%      -0.1        0.13 ± 25%  perf-profile.self.cycles-pp.__get_user_4
      0.20 ± 19%      -0.1        0.13 ± 13%  perf-profile.self.cycles-pp.ip_route_output_flow
      0.19 ± 15%      -0.1        0.12 ± 11%  perf-profile.self.cycles-pp.udp4_lib_lookup2
      0.17 ± 17%      -0.1        0.10 ± 24%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.10 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.20 ± 16%      -0.1        0.13 ± 17%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.21 ± 12%      -0.1        0.14 ± 18%  perf-profile.self.cycles-pp.__put_user_nocheck_4
      0.17 ±  8%      -0.1        0.10 ± 19%  perf-profile.self.cycles-pp.ip_send_check
      0.16 ± 10%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.15 ± 19%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.ip_make_skb
      0.14 ± 11%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.ipv4_mtu
      0.10 ± 12%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.19 ± 10%      -0.1        0.13 ± 21%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.12 ± 24%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.09 ± 36%      -0.1        0.04 ± 73%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.12 ± 19%      -0.1        0.06 ± 49%  perf-profile.self.cycles-pp.rcu_all_qs
      0.09 ± 24%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.ip_setup_cork
      0.08 ± 17%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__build_skb_around
      0.10 ± 26%      -0.1        0.05 ± 47%  perf-profile.self.cycles-pp.xfrm_lookup_with_ifid
      0.16 ± 16%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.udp4_csum_init
      0.12 ± 18%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.12 ± 16%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.sock_do_ioctl
      0.13 ±  9%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.10 ± 19%      -0.0        0.06 ± 19%  perf-profile.self.cycles-pp.import_single_range
      0.08 ±  8%      -0.0        0.04 ± 73%  perf-profile.self.cycles-pp.ip_generic_getfrag
      0.11 ± 18%      -0.0        0.07 ± 21%  perf-profile.self.cycles-pp.__x64_sys_ioctl
      0.12 ± 14%      -0.0        0.09 ± 18%  perf-profile.self.cycles-pp.skb_consume_udp
      0.06 ± 51%      +0.0        0.11 ± 18%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.00            +0.1        0.06 ± 19%  perf-profile.self.cycles-pp.tick_nohz_idle_enter
      0.18 ± 13%      +0.1        0.24 ±  7%  perf-profile.self.cycles-pp.reweight_entity
      0.08 ± 26%      +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.newidle_balance
      0.02 ±141%      +0.1        0.08 ± 16%  perf-profile.self.cycles-pp.rb_next
      0.08 ± 24%      +0.1        0.15 ± 24%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.13 ± 18%      +0.1        0.20 ± 13%  perf-profile.self.cycles-pp.place_entity
      0.03 ±102%      +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.check_spread
      0.06 ± 48%      +0.1        0.14 ± 17%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.intel_idle_irq
      0.06 ± 50%      +0.1        0.15 ± 10%  perf-profile.self.cycles-pp.select_task_rq
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.activate_task
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp._find_next_and_bit
      0.02 ±142%      +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.02 ±141%      +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.check_preempt_curr
      0.36 ±  3%      +0.1        0.46 ± 12%  perf-profile.self.cycles-pp.do_idle
      0.21 ± 10%      +0.1        0.32 ± 12%  perf-profile.self.cycles-pp.update_cfs_group
      0.17 ± 10%      +0.1        0.28 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.05 ± 47%      +0.1        0.15 ± 25%  perf-profile.self.cycles-pp.update_min_vruntime
      0.00            +0.1        0.11 ± 13%  perf-profile.self.cycles-pp.rb_erase
      0.23 ± 13%      +0.1        0.34 ±  7%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.00            +0.1        0.11 ± 24%  perf-profile.self.cycles-pp.backlog_napi_should_run
      0.00            +0.1        0.12 ± 21%  perf-profile.self.cycles-pp.idle_cpu
      0.34 ±  9%      +0.1        0.48 ±  6%  perf-profile.self.cycles-pp.llist_reverse_order
      0.12 ±  6%      +0.1        0.26 ± 12%  perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.resched_curr
      0.05 ± 45%      +0.1        0.19 ±  9%  perf-profile.self.cycles-pp.pick_next_entity
      0.03 ±101%      +0.1        0.18 ± 12%  perf-profile.self.cycles-pp.put_prev_entity
      0.00            +0.2        0.15 ±  7%  perf-profile.self.cycles-pp.napi_threaded_poll_loop
      0.61 ±  9%      +0.2        0.78 ±  4%  perf-profile.self.cycles-pp.__skb_wait_for_more_packets
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.20 ± 12%      +0.2        0.37 ± 17%  perf-profile.self.cycles-pp.update_rq_clock
      0.09 ± 15%      +0.2        0.26 ±  9%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.68 ±  9%      +0.2        0.86 ± 14%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.2        0.18 ± 12%  perf-profile.self.cycles-pp.cpu_util
      0.09 ± 22%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.12 ± 23%      +0.2        0.30 ±  6%  perf-profile.self.cycles-pp.switch_fpu_return
      0.10 ± 15%      +0.2        0.28 ± 11%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.15 ±  7%      +0.2        0.33 ± 10%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.12 ± 14%      +0.2        0.30 ±  8%  perf-profile.self.cycles-pp.prepare_task_switch
      0.43 ±  8%      +0.2        0.62 ±  8%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.00            +0.2        0.19 ± 13%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.37 ± 21%      +0.2        0.57 ±  6%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.07 ± 17%      +0.2        0.28 ± 17%  perf-profile.self.cycles-pp.set_next_entity
      0.00            +0.2        0.21 ± 22%  perf-profile.self.cycles-pp.cpuacct_charge
      0.16 ± 18%      +0.2        0.39 ± 13%  perf-profile.self.cycles-pp.dequeue_entity
      0.54 ± 10%      +0.2        0.78 ±  5%  perf-profile.self.cycles-pp.schedule_timeout
      0.30 ±  7%      +0.2        0.54 ± 13%  perf-profile.self.cycles-pp.nohz_run_idle_balance
      0.21 ± 14%      +0.3        0.46 ±  8%  perf-profile.self.cycles-pp.available_idle_cpu
      0.11 ± 25%      +0.3        0.38 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.09 ± 14%      +0.3        0.38 ±  7%  perf-profile.self.cycles-pp.schedule
      0.22 ± 18%      +0.3        0.52 ±  9%  perf-profile.self.cycles-pp.enqueue_entity
      0.14 ± 15%      +0.3        0.45 ±  9%  perf-profile.self.cycles-pp.finish_task_switch
      0.16 ± 19%      +0.3        0.50 ±  8%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.16 ± 17%      +0.4        0.52 ± 12%  perf-profile.self.cycles-pp.update_curr
      0.18 ± 16%      +0.4        0.56 ±  6%  perf-profile.self.cycles-pp.os_xsave
      0.00            +0.4        0.39 ±  7%  perf-profile.self.cycles-pp.smpboot_thread_fn
      1.35 ±  6%      +0.4        1.76 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.29 ± 15%      +0.5        0.74 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.27 ± 13%      +0.5        0.76 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
      0.27 ±  8%      +0.6        0.83 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.40 ± 11%      +0.6        1.00 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.36 ± 13%      +0.6        0.96 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.04 ±104%      +0.7        0.70 ±  6%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.60 ± 15%      +0.9        1.53 ±  7%  perf-profile.self.cycles-pp.__switch_to
      0.49 ± 12%      +1.0        1.49 ±  6%  perf-profile.self.cycles-pp.__schedule
      0.56 ± 10%      +1.0        1.61 ±  3%  perf-profile.self.cycles-pp.__switch_to_asm
      2.80 ± 11%      +2.3        5.11 ±  6%  perf-profile.self.cycles-pp.poll_idle



***************************************************************************************************
lkp-hsw-d04: 8 threads 1 sockets Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (Haswell) with 8G memory
=========================================================================================
cluster/compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/testcase:
  cs-localhost/gcc-12/performance/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/600s/lkp-hsw-d04/qperf

commit: 
  83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode")
  e9dc861c8b ("net: Use SMP threads for backlog NAPI.")

83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     38.66 ±  5%      -5.6%      36.47        boot-time.boot
      3575           +14.2%       4083        uptime.idle
    592566           +20.9%     716486        meminfo.Active
    592534           +20.9%     716422        meminfo.Active(anon)
    962633           +13.1%    1088738        meminfo.Committed_AS
    599330           +20.9%     724793        meminfo.Shmem
     70.30           +10.8       81.10        mpstat.cpu.all.idle%
      9.87 ±  3%      -9.9        0.02        mpstat.cpu.all.iowait%
      1.41            +0.2        1.60        mpstat.cpu.all.irq%
      3.21            -3.1        0.06        mpstat.cpu.all.soft%
     13.42            +1.9       15.34        mpstat.cpu.all.sys%
     69.67           +16.3%      81.00        vmstat.cpu.id
      9.33 ±  5%    -100.0%       0.00        vmstat.cpu.wa
      1.00          -100.0%       0.00        vmstat.procs.b
    346923          +101.0%     697484        vmstat.system.cs
     22035 ±  2%      +4.0%      22906 ±  2%  vmstat.system.in
      5757 ±  2%     +11.4%       6412        qperf.sctp.latency
 8.812e+09           -10.7%  7.867e+09        qperf.tcp.bw
      3817           +17.3%       4478        qperf.tcp.latency
      9251        +7.3e+05%   67524707        qperf.time.involuntary_context_switches
     57.00           -16.4%      47.67        qperf.time.percent_of_cpu_this_job_got
    311.63           -18.7%     253.38        qperf.time.system_time
  38933889           -11.7%   34386047        qperf.time.voluntary_context_switches
      2944           +11.7%       3289        qperf.udp.latency
 1.031e+10           -24.2%  7.813e+09        qperf.udp.recv_bw
 1.031e+10           -24.2%  7.813e+09        qperf.udp.send_bw
    148133           +20.9%     179105        proc-vmstat.nr_active_anon
    104164            -2.9%     101127        proc-vmstat.nr_dirty_background_threshold
    208585            -2.9%     202504        proc-vmstat.nr_dirty_threshold
    818642            +3.9%     850179        proc-vmstat.nr_file_pages
   1076298            -2.8%    1045877        proc-vmstat.nr_free_pages
      4474            -3.3%       4325        proc-vmstat.nr_kernel_stack
    149842           +20.9%     181203        proc-vmstat.nr_shmem
      9672            -3.3%       9354        proc-vmstat.nr_slab_unreclaimable
    148133           +20.9%     179105        proc-vmstat.nr_zone_active_anon
  74838089           -12.9%   65171973        proc-vmstat.numa_hit
  74917869           -13.1%   65109537        proc-vmstat.numa_local
    197378           +21.0%     238779        proc-vmstat.pgactivate
 7.196e+08            -9.9%  6.481e+08        proc-vmstat.pgalloc_normal
    688531            +2.2%     703483        proc-vmstat.pgfault
 7.194e+08            -9.9%  6.479e+08        proc-vmstat.pgfree
     35496            +2.6%      36415 ±  2%  proc-vmstat.pgreuse
  11521778 ±  3%     +81.6%   20920988        turbostat.C1
      1.10 ±  7%      +1.4        2.45        turbostat.C1%
    845181 ±  3%     +13.4%     958085        turbostat.C1E
      1.10 ±  3%      +0.1        1.23        turbostat.C1E%
    550023 ±  2%     -64.9%     193246 ±  5%  turbostat.C3
      9.50 ±  3%      -8.7        0.80 ±  4%  turbostat.C3%
     34350 ±  3%    +152.0%      86567 ±  2%  turbostat.C6
      0.33 ±  4%      +0.4        0.77 ±  2%  turbostat.C6%
   3142398           +14.0%    3583310        turbostat.C7s
     63.05            +7.9       70.99        turbostat.C7s%
     11.56 ±  4%     -93.0%       0.81 ±  7%  turbostat.CPU%c3
      0.28 ±  7%     +17.9%       0.33 ±  2%  turbostat.CPU%c6
     36.51           +26.8%      46.27        turbostat.CPU%c7
     21.04            -1.7%      20.69        turbostat.CorWatt
      0.19           +10.5%       0.21        turbostat.IPC
  79969004 ±  2%     -11.9%   70429254 ±  2%  turbostat.POLL
      3.15            -0.4        2.75        turbostat.POLL%
      1015 ± 88%    +526.8%       6363 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.avg
      8122 ± 88%    +526.8%      50910 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.max
      2686 ± 88%    +526.8%      16837 ±  7%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      1.30 ±  6%     +30.2%       1.70 ±  6%  sched_debug.cfs_rq:/.h_nr_running.max
      0.54 ±  6%     +20.6%       0.65 ±  6%  sched_debug.cfs_rq:/.h_nr_running.stddev
    141566 ± 11%     +47.6%     208983 ±  4%  sched_debug.cfs_rq:/.load.avg
    419639 ±  5%     +86.8%     783786 ±  2%  sched_debug.cfs_rq:/.load.max
    172078           +63.6%     281551 ±  3%  sched_debug.cfs_rq:/.load.stddev
    157.27 ±  5%     +31.4%     206.72 ±  6%  sched_debug.cfs_rq:/.load_avg.avg
    568.97 ±  4%     +14.3%     650.15 ±  4%  sched_debug.cfs_rq:/.load_avg.max
     10.27 ±  3%     +54.3%      15.85 ± 18%  sched_debug.cfs_rq:/.load_avg.min
    201.87 ±  3%     +10.4%     222.90 ±  5%  sched_debug.cfs_rq:/.load_avg.stddev
      1015 ± 88%    +526.8%       6363 ±  7%  sched_debug.cfs_rq:/.max_vruntime.avg
      8122 ± 88%    +526.8%      50910 ±  7%  sched_debug.cfs_rq:/.max_vruntime.max
      2686 ± 88%    +526.8%      16837 ±  7%  sched_debug.cfs_rq:/.max_vruntime.stddev
    118822           -11.5%     105134        sched_debug.cfs_rq:/.min_vruntime.avg
    157741 ±  2%     -20.0%     126190 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
     22839 ±  9%     -47.6%      11957 ± 19%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.44 ±  8%     +17.4%       0.51        sched_debug.cfs_rq:/.nr_running.avg
      1.00           +36.4%       1.36 ±  5%  sched_debug.cfs_rq:/.nr_running.max
      0.44 ±  4%     +20.4%       0.53 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
    376.66           +11.8%     421.23 ±  3%  sched_debug.cfs_rq:/.runnable_avg.avg
    907.15 ±  3%     +15.7%       1049 ±  4%  sched_debug.cfs_rq:/.runnable_avg.max
     52787 ± 35%     -45.1%      28955 ± 46%  sched_debug.cfs_rq:/.spread0.max
     22839 ±  9%     -47.6%      11957 ± 19%  sched_debug.cfs_rq:/.spread0.stddev
    840.12 ±  2%      -9.4%     761.42        sched_debug.cfs_rq:/.util_avg.max
    310.09 ±  3%     -13.4%     268.50        sched_debug.cfs_rq:/.util_avg.stddev
    648.39 ± 10%     -13.0%     564.21 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.max
    621193 ±  3%     -10.2%     557626 ±  3%  sched_debug.cpu.avg_idle.avg
      1384 ± 29%     -54.5%     629.33 ± 15%  sched_debug.cpu.clock_task.stddev
      0.00 ± 19%     +31.3%       0.00 ±  2%  sched_debug.cpu.next_balance.stddev
  12671402 ±  2%    +107.0%   26235293 ±  2%  sched_debug.cpu.nr_switches.avg
  17842961 ±  5%     +64.5%   29353353 ±  4%  sched_debug.cpu.nr_switches.max
   9669269 ±  5%    +138.8%   23086217        sched_debug.cpu.nr_switches.min
      0.31 ±  8%     -79.3%       0.06 ± 22%  sched_debug.cpu.nr_uninterruptible.avg
      1.95            -0.2        1.75        perf-stat.i.branch-miss-rate%
  19679514 ±  2%      -8.8%   17950299        perf-stat.i.branch-misses
      2.17 ± 16%      +2.0        4.20 ±  7%  perf-stat.i.cache-miss-rate%
   2501274 ± 19%    +123.6%    5593037 ±  7%  perf-stat.i.cache-misses
  83314250            -9.1%   75711656        perf-stat.i.cache-references
    348365          +101.1%     700470        perf-stat.i.context-switches
      1.64            -7.5%       1.52        perf-stat.i.cpi
 7.816e+09            -9.4%  7.084e+09        perf-stat.i.cpu-cycles
     56.47          +146.7%     139.33        perf-stat.i.cpu-migrations
     25094 ±  4%     -26.5%      18434 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.17 ±  4%      -0.0        0.14        perf-stat.i.dTLB-load-miss-rate%
   2595591 ±  5%     -21.6%    2035642        perf-stat.i.dTLB-load-misses
 1.677e+09            -2.1%  1.641e+09        perf-stat.i.dTLB-loads
      0.06            -0.0        0.05 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    629940            -8.8%     574319 ±  2%  perf-stat.i.dTLB-store-misses
  1.14e+09            +1.7%   1.16e+09        perf-stat.i.dTLB-stores
     41.52            -2.6       38.90        perf-stat.i.iTLB-load-miss-rate%
   1085663           -11.4%     961971        perf-stat.i.iTLB-load-misses
   1910535 ±  3%     +16.1%    2217887        perf-stat.i.iTLB-loads
      5366 ±  2%      +5.4%       5655        perf-stat.i.instructions-per-iTLB-miss
      0.66           +10.2%       0.73        perf-stat.i.ipc
      0.06 ± 56%    +185.7%       0.17 ± 56%  perf-stat.i.major-faults
      0.98            -9.4%       0.89        perf-stat.i.metric.GHz
    396.87           +20.5%     478.42        perf-stat.i.metric.K/sec
      1077            +1.9%       1097        perf-stat.i.minor-faults
   2086591 ± 22%    +137.6%    4958478 ±  7%  perf-stat.i.node-loads
    105179 ± 13%    +101.1%     211467 ±  5%  perf-stat.i.node-stores
      1077            +1.9%       1097        perf-stat.i.page-faults
     15.69            -8.5%      14.37        perf-stat.overall.MPKI
      1.85            -0.2        1.66        perf-stat.overall.branch-miss-rate%
      3.01 ± 20%      +4.4        7.39 ±  7%  perf-stat.overall.cache-miss-rate%
      1.47            -8.7%       1.34        perf-stat.overall.cpi
      3283 ± 24%     -61.2%       1273 ±  8%  perf-stat.overall.cycles-between-cache-misses
      0.15 ±  4%      -0.0        0.12        perf-stat.overall.dTLB-load-miss-rate%
      0.06            -0.0        0.05        perf-stat.overall.dTLB-store-miss-rate%
     36.26 ±  2%      -6.0       30.25        perf-stat.overall.iTLB-load-miss-rate%
      4891 ±  2%     +12.0%       5479        perf-stat.overall.instructions-per-iTLB-miss
      0.68            +9.5%       0.74        perf-stat.overall.ipc
  19646658 ±  2%      -8.8%   17920681        perf-stat.ps.branch-misses
   2497231 ± 19%    +123.6%    5583672 ±  7%  perf-stat.ps.cache-misses
  83175281            -9.1%   75585970        perf-stat.ps.cache-references
    347785          +101.1%     699305        perf-stat.ps.context-switches
 7.803e+09            -9.4%  7.072e+09        perf-stat.ps.cpu-cycles
     56.37          +146.7%     139.10        perf-stat.ps.cpu-migrations
   2591259 ±  5%     -21.6%    2032256        perf-stat.ps.dTLB-load-misses
 1.674e+09            -2.1%  1.639e+09        perf-stat.ps.dTLB-loads
    628893            -8.8%     573359 ±  2%  perf-stat.ps.dTLB-store-misses
 1.138e+09            +1.7%  1.158e+09        perf-stat.ps.dTLB-stores
   1083855           -11.4%     960366        perf-stat.ps.iTLB-load-misses
   1907339 ±  3%     +16.1%    2214176        perf-stat.ps.iTLB-loads
      0.06 ± 56%    +185.8%       0.17 ± 56%  perf-stat.ps.major-faults
      1075            +1.9%       1095        perf-stat.ps.minor-faults
   2083234 ± 22%    +137.6%    4950171 ±  7%  perf-stat.ps.node-loads
    105007 ± 13%    +101.0%     211114 ±  5%  perf-stat.ps.node-stores
      1075            +1.9%       1095        perf-stat.ps.page-faults
      0.00 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter
      0.02 ± 24%     -95.7%       0.00 ±141%  perf-sched.sch_delay.avg.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      0.01 ± 22%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
      0.01 ± 29%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur
      0.00 ± 47%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter
      0.01 ± 14%     -70.0%       0.00 ± 98%  perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_write_iter.vfs_write
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.01 ±  4%     -29.0%       0.01 ± 12%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 59%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.01 ±  6%     -72.7%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
      0.00          +133.3%       0.00 ± 20%  perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 10%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
      0.01 ±  8%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
      0.01 ± 25%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      0.00 ± 25%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id
      0.01 ± 21%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode
      0.01 ± 15%     -25.0%       0.01 ± 10%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.01 ± 24%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ± 30%     -80.9%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.11 ±  3%     -66.4%       0.04 ±  6%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.00 ± 27%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter
      0.02 ±  6%     -96.7%       0.00 ±141%  perf-sched.sch_delay.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      0.02 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
      0.03 ± 32%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur
      0.01 ± 92%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter
      0.02 ±  4%     -78.7%       0.00 ±110%  perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_write_iter.vfs_write
      0.00          -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.04 ± 23%     -41.7%       0.02 ± 36%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.03 ±  3%     +26.3%       0.03 ± 11%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.02 ± 47%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.04 ±  3%   +1509.3%       0.57 ± 85%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.03 ± 13%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
      0.03 ± 32%     -53.4%       0.01 ± 17%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ± 12%     -37.0%       0.01 ± 25%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.04 ±  7%     -45.4%       0.02 ± 38%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.05 ± 26%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
      0.03 ±  3%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      0.01 ± 14%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
      0.03 ± 90%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      0.02 ± 17%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id
      0.05 ± 71%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode
      0.02 ± 10%     -44.3%       0.01 ± 22%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.01 ±  8%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
      0.89 ± 10%     -99.8%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.02 ± 12%    -100.0%       0.00        perf-sched.sch_delay.max.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm
      0.04 ± 13%     -49.2%       0.02 ± 24%  perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      0.35           -49.3%       0.18        perf-sched.total_wait_and_delay.average.ms
    705805           +90.1%    1341585        perf-sched.total_wait_and_delay.count.ms
      0.35           -49.6%       0.18        perf-sched.total_wait_time.average.ms
     17.56 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      5.07          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
     18.73 ± 28%     -73.5%       4.96 ± 70%  perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
    150.40 ±  3%     -98.7%       1.90 ± 14%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    209.60          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
     23.15 ± 10%     +30.3%      30.16 ±  4%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
     41.98 ±  2%     -14.7%      35.81        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    205.40           +82.7%     375.37        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     10.37          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
     14.61 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      1.06          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
    228.86 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      1.06          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
     24.89 ±  2%     +81.1%      45.08 ±  7%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.01           -42.9%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
    154.39           -99.9%       0.20        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    153.91           -98.8%       1.91 ± 13%  perf-sched.wait_and_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
    259.59 ±  3%     +28.0%     332.24 ±  5%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
     14.67 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
     31.67           -93.7%       2.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.33 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
      2135 ±  9%     -31.5%       1463 ±  4%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     58.67 ±  5%     -48.9%      30.00 ±  7%  perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
    390.67           +19.7%     467.67        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
     45.00           -71.1%      13.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     30.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
    293.33 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
     15.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
     15.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
     17.67 ±  2%     -84.9%       2.67 ± 17%  perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
     15.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.msleep.ata_msleep.sata_link_resume
    189.67 ±  2%     -42.7%     108.67 ±  7%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    701375           -36.4%     446163        perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
    381.67        +1.2e+05%     446519        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     31.67           -93.7%       2.00        perf-sched.wait_and_delay.count.syslog_print.do_syslog.kmsg_read.vfs_read
    172.00           -34.3%     113.00 ±  2%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
     17.56 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      5.09          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
     25.21 ± 28%     -70.6%       7.41 ± 81%  perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
    349.75           -98.9%       3.79 ± 14%  perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    352.11          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
     30.49 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
    208.80          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      1.07          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
    338.85          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      1.07          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
    322.33           +52.6%     492.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      1.78 ± 10%     -97.5%       0.04        perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
    349.73           -98.9%       3.81 ± 13%  perf-sched.wait_and_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
     17.56 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      0.00 ± 10%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter
      0.02 ± 19%     -88.5%       0.00 ±141%  perf-sched.wait_time.avg.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      5.06          -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
      0.07 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur
      0.01 ± 35%     -84.6%       0.00 ±141%  perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter
     18.73 ± 28%     -73.5%       4.96 ± 70%  perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00 ± 12%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
    150.39 ±  3%     -98.7%       1.88 ± 14%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00 ± 81%    +233.3%       0.00 ± 14%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
     22.48 ±141%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.01 ±  8%     -34.4%       0.01        perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
    209.59          -100.0%       0.00        perf-sched.wait_time.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
     23.15 ± 10%     +30.3%      30.16 ±  4%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
     41.98 ±  2%     -14.7%      35.81        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    205.39           +82.8%     375.36        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     10.37          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
     14.60 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      1.06          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
    228.86 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      0.03 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id
      0.06 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode
      0.01 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_exec_internal
      1.06          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
     24.88 ±  2%     +81.1%      45.08 ±  7%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.05 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm
    154.38           -99.9%       0.20        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    153.90           -98.8%       1.89 ± 13%  perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
    259.48 ±  3%     +28.0%     332.21 ±  5%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
     17.56 ±  9%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      0.01 ± 35%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter
      0.02 ±  3%     -90.9%       0.00 ±141%  perf-sched.wait_time.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      5.07          -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id
      0.09 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur
      0.03 ± 20%     -95.9%       0.00 ±141%  perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter
     25.21 ± 28%     -70.6%       7.41 ± 81%  perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.00 ± 12%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
    349.72           -98.9%       3.76 ± 14%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00 ± 81%    +200.0%       0.01        perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
    112.33 ±141%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.05 ± 15%   +5840.9%       3.15 ±133%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
    352.10          -100.0%       0.00        perf-sched.wait_time.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
     30.48 ±  5%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register
    208.79          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce
      1.07          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset
    338.84          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu
      0.07 ±  5%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id
      0.11 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode
      0.01 ±  9%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_exec_internal
      1.07          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume
    322.32           +52.6%     491.99        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.93 ±  8%     -95.2%       0.04        perf-sched.wait_time.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.07 ± 14%    -100.0%       0.00        perf-sched.wait_time.max.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm
    349.71           -98.9%       3.79 ± 13%  perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
      8.91 ±  2%      -8.9        0.00        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
      8.76 ±  2%      -8.8        0.00        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb
      8.61 ±  2%      -8.6        0.00        perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
      7.90 ±  2%      -7.9        0.00        perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
      7.52 ±  2%      -7.5        0.00        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
      7.47 ±  2%      -7.5        0.00        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq
     10.66 ±  2%      -7.3        3.34 ±  4%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
      6.89            -6.9        0.00        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq
      6.09            -6.1        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      9.74 ±  7%      -4.8        4.95 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      9.66 ±  8%      -4.8        4.89 ±  3%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      9.19 ±  8%      -4.7        4.47 ±  3%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.33 ±  9%      -4.7        4.64 ±  3%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      9.14 ±  8%      -4.7        4.48 ±  3%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      8.94 ±  8%      -4.7        4.28        perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      8.73 ±  8%      -4.5        4.18 ±  3%  perf-profile.calltrace.cycles-pp.sock_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.26 ±  8%      -4.5        3.81 ±  4%  perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_write_iter.vfs_write.ksys_write.do_syscall_64
      6.18 ±  9%      -4.0        2.17 ±  9%  perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_write_iter.vfs_write.ksys_write
      6.04 ±  9%      -3.9        2.09 ±  9%  perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_write_iter.vfs_write
      6.04 ±  7%      -3.9        2.16 ±  2%  perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      5.62 ±  9%      -3.8        1.77 ±  9%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_write_iter
      5.92 ±  7%      -3.8        2.11 ±  2%  perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     11.16 ±  7%      -3.7        7.48 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto
      5.47 ±  6%      -3.6        1.84 ±  2%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     11.02 ±  7%      -3.6        7.40 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
     10.70 ±  8%      -3.6        7.12 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.81 ±  9%      -3.6        7.23 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     12.09 ±  9%      -3.4        8.64 ±  2%  perf-profile.calltrace.cycles-pp.write
     12.36 ±  7%      -3.4        8.94 ±  2%  perf-profile.calltrace.cycles-pp.sendto
      6.72 ±  3%      -1.9        4.78 ±  4%  perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg
      8.68 ±  6%      -1.8        6.86        perf-profile.calltrace.cycles-pp.recvfrom
      5.27 ±  5%      -1.8        3.48 ±  6%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      5.63 ±  4%      -1.8        3.85 ±  5%  perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      5.12 ±  4%      -1.7        3.38 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      9.18 ±  7%      -1.7        7.45        perf-profile.calltrace.cycles-pp.read
      7.07 ±  6%      -1.6        5.52        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom
      6.05            -1.5        4.54 ±  2%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      6.93 ±  6%      -1.5        5.44        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
      7.50 ±  8%      -1.4        6.05        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      7.37 ±  8%      -1.4        5.96        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      5.78 ±  2%      -1.4        4.38        perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      5.79 ±  5%      -1.4        4.40        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
      5.84 ±  6%      -1.4        4.46        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
      6.26 ±  9%      -1.4        4.91        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      6.06 ±  9%      -1.3        4.74 ±  2%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      5.18 ±  2%      -1.3        3.89 ±  2%  perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      5.23 ±  6%      -1.3        3.98        perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.08 ±  6%      -1.3        3.83        perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      5.07 ±  6%      -1.2        3.82        perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      5.58 ±  9%      -1.2        4.36        perf-profile.calltrace.cycles-pp.sock_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.01 ±  2%      -1.2        3.80 ±  2%  perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      5.38 ±  9%      -1.2        4.18 ±  2%  perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64
      5.14 ±  9%      -1.2        3.97 ±  2%  perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read
      5.11 ±  9%      -1.2        3.95 ±  2%  perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter.vfs_read
      4.14 ±  5%      -1.1        3.02        perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
      4.50            -1.1        3.44 ±  2%  perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
      4.20 ± 10%      -1.1        3.13 ±  2%  perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter
      4.06            -0.9        3.11        perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv
      3.63 ±  3%      -0.9        2.77 ± 12%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.50 ±  3%      -0.8        1.70 ±  6%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      3.65            -0.8        2.86        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb
      3.37 ±  4%      -0.8        2.58 ± 11%  perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      3.20            -0.7        2.51 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
      2.03 ±  7%      -0.6        1.41 ±  6%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
      2.09 ±  3%      -0.6        1.47 ±  8%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      0.95 ± 20%      -0.6        0.35 ± 70%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      2.40 ±  3%      -0.6        1.80 ± 10%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.78 ±  2%      -0.6        2.18 ±  3%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
      2.82 ±  3%      -0.6        2.24 ±  8%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      1.91 ±  7%      -0.6        1.33 ±  7%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
      2.65 ±  3%      -0.6        2.09 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      2.21 ±  3%      -0.6        1.65 ±  9%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.83 ±  7%      -0.5        1.29 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
      1.93 ± 11%      -0.5        1.44 ±  3%  perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      2.31 ±  6%      -0.4        1.86 ± 13%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      1.48 ± 13%      -0.4        1.05 ±  4%  perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.42 ±  4%      -0.3        1.09 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      1.67 ±  7%      -0.3        1.37 ±  2%  perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_write_iter.vfs_write.ksys_write
      0.88 ±  7%      -0.2        0.64 ±  9%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      1.38 ±  8%      -0.2        1.15        perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_write_iter.vfs_write
      0.84 ±  7%      -0.2        0.63 ±  8%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      1.22 ±  4%      -0.2        1.05 ±  7%  perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.88 ± 11%      -0.1        0.76 ±  3%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_write_iter
      0.36 ± 70%      +0.2        0.59        perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.17 ±141%      +0.4        0.56 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.7        0.66 ±  6%  perf-profile.calltrace.cycles-pp.__switch_to_asm
      0.00            +0.8        0.82 ± 14%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
      0.00            +0.9        0.90 ±  4%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
     20.95            +1.0       21.98        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.26 ±  4%      +1.1        2.36 ±  4%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.17 ±  6%      +1.2        2.36 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      0.00            +1.2        1.25        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +1.4        1.36        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +1.4        1.40 ±  4%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
      0.00            +1.4        1.44 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx
     54.93            +1.6       56.51        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.99 ± 29%      +1.9        2.93 ±  6%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
      0.00            +2.0        1.99 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit
      0.76 ± 36%      +2.1        2.87 ±  5%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb
      0.00            +2.2        2.22 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit
      0.00            +2.2        2.23        perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +2.3        2.26 ±  6%  perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.00            +2.3        2.27 ±  5%  perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      0.00            +2.3        2.34        perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +2.5        2.50        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.35            +2.7       23.05        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +3.0        3.04 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +3.2        3.20 ±  6%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      2.71 ± 16%      +3.3        5.98 ± 41%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      2.92 ± 16%      +3.4        6.29 ± 41%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      3.55 ± 15%      +3.6        7.16 ± 42%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      3.56 ± 15%      +3.6        7.17 ± 42%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
     42.83            +3.8       46.63        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +4.6        4.55 ±  2%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop
      0.00            +5.1        5.14        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn
      0.00            +5.6        5.61        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread
      0.00            +5.6        5.64        perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +5.7        5.70        perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +9.6        9.56 ±  2%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     36.67           -10.2       26.50        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     36.16           -10.0       26.13        perf-profile.children.cycles-pp.do_syscall_64
     17.24 ±  2%      -9.1        8.10 ±  2%  perf-profile.children.cycles-pp.udp_sendmsg
      8.96 ±  2%      -8.9        0.06 ±  7%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      8.77 ±  2%      -8.8        0.00        perf-profile.children.cycles-pp.do_softirq
      8.72 ±  2%      -8.6        0.10 ±  8%  perf-profile.children.cycles-pp.__do_softirq
      7.91 ±  2%      -7.9        0.00        perf-profile.children.cycles-pp.net_rx_action
     12.23 ±  2%      -7.9        4.33 ±  5%  perf-profile.children.cycles-pp.udp_send_skb
     11.97            -7.8        4.21 ±  5%  perf-profile.children.cycles-pp.ip_send_skb
     11.09 ±  2%      -7.5        3.61 ±  5%  perf-profile.children.cycles-pp.ip_finish_output2
     10.67 ±  2%      -7.3        3.34 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      9.74 ±  7%      -4.8        4.96 ±  2%  perf-profile.children.cycles-pp.__x64_sys_sendto
      9.67 ±  7%      -4.8        4.89 ±  3%  perf-profile.children.cycles-pp.__sys_sendto
      9.19 ±  8%      -4.7        4.48 ±  3%  perf-profile.children.cycles-pp.sock_sendmsg
      9.34 ±  9%      -4.7        4.66 ±  3%  perf-profile.children.cycles-pp.ksys_write
      9.16 ±  8%      -4.7        4.50 ±  3%  perf-profile.children.cycles-pp.vfs_write
      8.73 ±  8%      -4.5        4.18 ±  3%  perf-profile.children.cycles-pp.sock_write_iter
     12.46 ±  7%      -3.4        9.03 ±  2%  perf-profile.children.cycles-pp.sendto
     12.16 ±  9%      -3.4        8.73 ±  2%  perf-profile.children.cycles-pp.write
     10.63            -2.4        8.18        perf-profile.children.cycles-pp.sock_recvmsg
     10.22            -2.4        7.81        perf-profile.children.cycles-pp.inet_recvmsg
     10.18            -2.4        7.77        perf-profile.children.cycles-pp.udp_recvmsg
      8.36 ±  2%      -2.2        6.17 ±  2%  perf-profile.children.cycles-pp.__skb_recv_udp
      6.73 ±  3%      -2.0        4.77 ±  4%  perf-profile.children.cycles-pp.__skb_wait_for_more_packets
      7.52 ±  2%      -1.9        5.64        perf-profile.children.cycles-pp.__napi_poll
      7.48 ±  2%      -1.9        5.62        perf-profile.children.cycles-pp.process_backlog
      8.76 ±  6%      -1.8        6.93        perf-profile.children.cycles-pp.recvfrom
      9.29 ±  7%      -1.8        7.52        perf-profile.children.cycles-pp.read
      6.91            -1.8        5.14        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      5.66 ±  4%      -1.8        3.91 ±  5%  perf-profile.children.cycles-pp.schedule_timeout
      6.09            -1.5        4.55 ±  2%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      6.05            -1.5        4.54 ±  2%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      5.79 ±  2%      -1.4        4.39 ±  2%  perf-profile.children.cycles-pp.__udp4_lib_rcv
      5.80 ±  5%      -1.4        4.41        perf-profile.children.cycles-pp.__sys_recvfrom
      5.85 ±  6%      -1.4        4.47        perf-profile.children.cycles-pp.__x64_sys_recvfrom
      6.26 ±  9%      -1.4        4.91        perf-profile.children.cycles-pp.ksys_read
      1.54 ± 16%      -1.3        0.22 ± 87%  perf-profile.children.cycles-pp._printk
      1.54 ± 16%      -1.3        0.22 ± 87%  perf-profile.children.cycles-pp.vprintk_emit
      1.54 ± 16%      -1.3        0.22 ± 87%  perf-profile.children.cycles-pp.console_unlock
      1.54 ± 16%      -1.3        0.22 ± 87%  perf-profile.children.cycles-pp.console_flush_all
      6.06 ±  9%      -1.3        4.74 ±  2%  perf-profile.children.cycles-pp.vfs_read
      5.19 ±  2%      -1.3        3.89 ±  2%  perf-profile.children.cycles-pp.udp_unicast_rcv_skb
      1.44 ± 16%      -1.2        0.21 ± 85%  perf-profile.children.cycles-pp.serial8250_console_write
      5.58 ±  9%      -1.2        4.36        perf-profile.children.cycles-pp.sock_read_iter
      1.41 ± 16%      -1.2        0.20 ± 86%  perf-profile.children.cycles-pp.wait_for_lsr
      5.03 ±  2%      -1.2        3.83 ±  2%  perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
      4.51            -1.1        3.44        perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
      4.06            -0.9        3.12 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      3.66            -0.8        2.86        perf-profile.children.cycles-pp.__wake_up_common_lock
      3.62 ±  3%      -0.8        2.82        perf-profile.children.cycles-pp.ip_make_skb
      0.83 ± 17%      -0.7        0.12 ± 92%  perf-profile.children.cycles-pp.io_serial_in
      3.21            -0.7        2.51 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      3.89 ±  5%      -0.7        3.21 ±  6%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      2.87 ±  4%      -0.7        2.21 ±  2%  perf-profile.children.cycles-pp.__ip_append_data
      3.02 ±  5%      -0.6        2.38 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      3.61 ±  6%      -0.6        3.00 ±  5%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      2.79 ±  2%      -0.6        2.19 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      2.66 ±  4%      -0.6        2.06 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      0.57 ± 17%      -0.5        0.08 ± 83%  perf-profile.children.cycles-pp.delay_tsc
      2.49 ±  5%      -0.4        2.10 ±  8%  perf-profile.children.cycles-pp.menu_select
      1.76 ±  2%      -0.4        1.41 ±  5%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      0.81 ±  5%      -0.2        0.57 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      1.01 ±  5%      -0.2        0.78 ± 11%  perf-profile.children.cycles-pp.__smp_call_single_queue
      1.10 ±  3%      -0.2        0.87 ±  9%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      1.76 ±  4%      -0.2        1.56 ±  4%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.78 ± 17%      -0.2        0.59 ± 13%  perf-profile.children.cycles-pp.ip_generic_getfrag
      0.86 ±  5%      -0.2        0.67        perf-profile.children.cycles-pp.select_task_rq
      1.04 ±  5%      -0.2        0.86 ±  8%  perf-profile.children.cycles-pp.__check_object_size
      0.40 ±  5%      -0.2        0.23 ±  7%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.48 ± 20%      -0.2        0.31 ± 13%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      0.65 ±  7%      -0.2        0.48 ± 10%  perf-profile.children.cycles-pp.llist_add_batch
      0.56 ±  7%      -0.2        0.40 ±  5%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.63 ± 13%      -0.2        0.47 ±  9%  perf-profile.children.cycles-pp.ip_route_output_flow
      0.96            -0.2        0.81 ±  7%  perf-profile.children.cycles-pp.__alloc_skb
      0.41 ± 19%      -0.1        0.27 ± 20%  perf-profile.children.cycles-pp.fib_table_lookup
      0.33 ± 13%      -0.1        0.20 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      1.03 ±  3%      -0.1        0.90        perf-profile.children.cycles-pp.native_sched_clock
      1.22 ±  4%      -0.1        1.09 ±  7%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.40 ±  5%      -0.1        0.28 ± 19%  perf-profile.children.cycles-pp.sock_wfree
      0.55 ±  5%      -0.1        0.43 ± 11%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.46 ±  2%      -0.1        0.35 ± 14%  perf-profile.children.cycles-pp.update_cfs_group
      0.61 ±  4%      -0.1        0.51 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.49 ±  6%      -0.1        0.40 ±  8%  perf-profile.children.cycles-pp.__ip_make_skb
      0.31 ±  5%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.61 ±  3%      -0.1        0.52        perf-profile.children.cycles-pp.skb_release_data
      0.32 ±  9%      -0.1        0.23 ±  2%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.19 ± 19%      -0.1        0.10 ± 31%  perf-profile.children.cycles-pp.netif_skb_features
      0.71 ±  2%      -0.1        0.62        perf-profile.children.cycles-pp.sched_clock_cpu
      0.67 ±  4%      -0.1        0.58 ±  2%  perf-profile.children.cycles-pp.local_clock_noinstr
      0.62 ±  3%      -0.1        0.53        perf-profile.children.cycles-pp.__consume_stateless_skb
      0.31 ±  9%      -0.1        0.23 ± 14%  perf-profile.children.cycles-pp.ip_output
      0.36 ± 15%      -0.1        0.28 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      0.28 ± 15%      -0.1        0.20 ± 24%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.35            -0.1        0.27 ±  3%  perf-profile.children.cycles-pp.__check_heap_object
      0.29 ± 15%      -0.1        0.22 ± 18%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.36 ±  4%      -0.1        0.29 ±  7%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.17 ± 14%      -0.1        0.10 ± 16%  perf-profile.children.cycles-pp.__ip_finish_output
      0.41            -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.ip_rcv
      0.16            -0.1        0.10 ± 21%  perf-profile.children.cycles-pp.nohz_run_idle_balance
      0.46 ±  7%      -0.1        0.39 ±  3%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.17 ± 19%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.__xfrm_policy_check2
      0.62            -0.1        0.56        perf-profile.children.cycles-pp.sched_clock
      0.34 ±  5%      -0.1        0.28        perf-profile.children.cycles-pp.kmalloc_reserve
      0.17 ±  7%      -0.1        0.12 ± 21%  perf-profile.children.cycles-pp.ip_rcv_core
      0.15 ± 26%      -0.1        0.09        perf-profile.children.cycles-pp.pick_next_task_idle
      0.23 ±  7%      -0.1        0.18 ± 25%  perf-profile.children.cycles-pp.menu_reflect
      0.56 ±  6%      -0.1        0.51 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.18 ± 11%      -0.0        0.13 ± 20%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.18 ± 14%      -0.0        0.13 ± 14%  perf-profile.children.cycles-pp.nf_hook_slow
      0.12 ± 33%      -0.0        0.07        perf-profile.children.cycles-pp.__update_idle_core
      0.13 ± 15%      -0.0        0.09 ± 10%  perf-profile.children.cycles-pp.move_addr_to_user
      0.18 ± 15%      -0.0        0.14 ± 11%  perf-profile.children.cycles-pp.ct_idle_exit
      0.12 ± 23%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.apparmor_ip_postroute
      0.08 ± 10%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp._copy_from_iter
      0.18 ± 12%      -0.0        0.14 ± 20%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.38 ±  5%      -0.0        0.34 ±  4%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.14 ± 18%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.35 ±  2%      -0.0        0.31        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.14 ±  6%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.ip_setup_cork
      0.13 ± 16%      -0.0        0.09 ± 13%  perf-profile.children.cycles-pp.udp4_csum_init
      0.09 ±  5%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.09 ±  9%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.ipv4_mtu
      0.09 ±  5%      -0.0        0.07 ± 18%  perf-profile.children.cycles-pp.__build_skb_around
      0.10 ±  4%      -0.0        0.08 ± 16%  perf-profile.children.cycles-pp.receiver_wake_function
      0.12 ±  3%      -0.0        0.10 ±  9%  perf-profile.children.cycles-pp.call_cpuidle
      0.10 ± 12%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.__mkroute_output
      0.24 ±  6%      +0.0        0.28 ±  5%  perf-profile.children.cycles-pp.rep_movs_alternative
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.children.cycles-pp.rebalance_domains
      0.22 ±  9%      +0.0        0.26 ±  4%  perf-profile.children.cycles-pp.__calc_delta
      0.18 ±  2%      +0.0        0.22 ± 11%  perf-profile.children.cycles-pp.copyout
      0.08 ± 17%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__list_add_valid
      0.14 ±  8%      +0.1        0.20 ± 12%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.07 ±  7%      +0.1        0.13 ± 21%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.14 ± 12%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.cpuacct_charge
      0.05 ± 70%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.rb_erase
      0.11            +0.1        0.20 ± 13%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.load_balance
      0.14 ± 19%      +0.1        0.23 ± 12%  perf-profile.children.cycles-pp.switch_fpu_return
      0.00            +0.1        0.09 ±  5%  perf-profile.children.cycles-pp.run_backlog_napi
      0.09 ± 14%      +0.1        0.18 ± 16%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.08 ± 14%      +0.1        0.19 ± 10%  perf-profile.children.cycles-pp.mm_cid_get
      0.13 ±  6%      +0.1        0.25 ± 18%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.16 ± 13%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.pick_next_entity
      0.24 ±  9%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.available_idle_cpu
      0.11 ± 19%      +0.2        0.26 ±  5%  perf-profile.children.cycles-pp.update_min_vruntime
      0.30 ±  8%      +0.2        0.47 ± 10%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.73            +0.2        0.96 ± 13%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.2        0.23 ± 12%  perf-profile.children.cycles-pp.backlog_napi_should_run
      0.43 ±  8%      +0.2        0.66 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.46 ±  3%      +0.3        0.74 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.87            +0.3        1.16 ± 10%  perf-profile.children.cycles-pp.prepare_task_switch
      0.09 ± 32%      +0.3        0.40 ±  6%  perf-profile.children.cycles-pp.wake_affine
      0.58 ±  7%      +0.3        0.92 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      1.56 ±  7%      +0.4        1.93 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.4        0.40 ± 15%  perf-profile.children.cycles-pp.check_preempt_wakeup
      2.00 ±  8%      +0.4        2.43        perf-profile.children.cycles-pp.enqueue_task_fair
      0.83 ±  3%      +0.4        1.25 ±  2%  perf-profile.children.cycles-pp.__switch_to
      2.07 ±  8%      +0.4        2.51        perf-profile.children.cycles-pp.activate_task
      0.06 ±  7%      +0.4        0.51 ± 12%  perf-profile.children.cycles-pp.check_preempt_curr
      0.87 ±  3%      +0.5        1.38 ±  6%  perf-profile.children.cycles-pp.__switch_to_asm
      0.80 ±  8%      +0.5        1.31 ±  5%  perf-profile.children.cycles-pp.update_curr
      0.21 ±  7%      +0.7        0.89 ±  3%  perf-profile.children.cycles-pp.put_prev_entity
      0.59 ±  6%      +0.7        1.31 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      2.19 ±  9%      +0.9        3.09 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.45 ±  4%      +1.0        2.41 ±  4%  perf-profile.children.cycles-pp.update_load_avg
     21.07            +1.0       22.11        perf-profile.children.cycles-pp.poll_idle
      2.67 ±  3%      +1.4        4.10 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
     54.93            +1.6       56.51        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     54.93            +1.6       56.51        perf-profile.children.cycles-pp.cpu_startup_entry
     54.80            +1.6       56.41        perf-profile.children.cycles-pp.do_idle
      1.15 ±  6%      +1.8        2.94 ±  6%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.08 ±  6%      +1.8        2.89 ±  5%  perf-profile.children.cycles-pp.loopback_xmit
      1.25 ±  6%      +1.9        3.15 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.29 ± 20%      +2.0        2.27 ±  5%  perf-profile.children.cycles-pp.__netif_rx
      0.26 ± 15%      +2.0        2.23 ±  6%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.28 ± 17%      +2.0        2.26 ±  6%  perf-profile.children.cycles-pp.netif_rx_internal
      4.48            +2.1        6.61        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.00            +2.5        2.51        perf-profile.children.cycles-pp.exit_to_user_mode_loop
     20.35            +2.7       23.05        perf-profile.children.cycles-pp.intel_idle
      0.20 ± 16%      +2.7        2.90        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      7.64 ±  4%      +3.0       10.64 ±  3%  perf-profile.children.cycles-pp.__schedule
     46.39            +3.2       49.59        perf-profile.children.cycles-pp.cpuidle_idle_call
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.children.cycles-pp.x86_64_start_kernel
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.children.cycles-pp.x86_64_start_reservations
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.children.cycles-pp.start_kernel
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.children.cycles-pp.arch_call_rest_init
      3.56 ± 15%      +3.6        7.18 ± 42%  perf-profile.children.cycles-pp.rest_init
      5.29 ±  5%      +3.8        9.05 ±  4%  perf-profile.children.cycles-pp.schedule
     42.90            +3.8       46.70        perf-profile.children.cycles-pp.cpuidle_enter_state
     42.93            +3.8       46.74        perf-profile.children.cycles-pp.cpuidle_enter
      0.00            +5.7        5.70        perf-profile.children.cycles-pp.napi_threaded_poll_loop
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.children.cycles-pp.ret_from_fork_asm
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.children.cycles-pp.ret_from_fork
      1.37 ± 27%      +8.2        9.58 ±  2%  perf-profile.children.cycles-pp.kthread
      0.00            +9.6        9.57 ±  2%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.83 ± 17%      -0.7        0.12 ± 92%  perf-profile.self.cycles-pp.io_serial_in
      4.20            -0.5        3.66 ±  2%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.57 ± 17%      -0.5        0.08 ± 83%  perf-profile.self.cycles-pp.delay_tsc
      0.56 ±  5%      -0.2        0.33 ± 11%  perf-profile.self.cycles-pp.udp_sendmsg
      1.75 ±  3%      -0.2        1.56 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.04 ±  8%      -0.2        0.87 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.65 ±  6%      -0.2        0.48 ±  9%  perf-profile.self.cycles-pp.llist_add_batch
      0.26 ± 25%      -0.2        0.11 ± 19%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.53 ± 10%      -0.1        0.38 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.35 ± 20%      -0.1        0.20 ± 20%  perf-profile.self.cycles-pp.fib_table_lookup
      0.56 ±  9%      -0.1        0.42 ± 21%  perf-profile.self.cycles-pp.do_idle
      1.00 ±  4%      -0.1        0.86        perf-profile.self.cycles-pp.native_sched_clock
      0.37 ± 18%      -0.1        0.24 ± 17%  perf-profile.self.cycles-pp.ip_finish_output2
      0.23 ± 20%      -0.1        0.09 ±  9%  perf-profile.self.cycles-pp.udp_send_skb
      0.39 ± 28%      -0.1        0.26 ±  6%  perf-profile.self.cycles-pp.sock_def_readable
      0.19 ±  5%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.39 ±  6%      -0.1        0.26 ± 16%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      0.32 ± 13%      -0.1        0.20 ±  6%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.44 ±  2%      -0.1        0.32 ± 15%  perf-profile.self.cycles-pp.update_cfs_group
      0.24 ±  9%      -0.1        0.12 ± 18%  perf-profile.self.cycles-pp.newidle_balance
      0.39 ±  7%      -0.1        0.28 ± 19%  perf-profile.self.cycles-pp.sock_wfree
      0.14 ± 10%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.udp_unicast_rcv_skb
      0.31 ±  7%      -0.1        0.22 ± 15%  perf-profile.self.cycles-pp.__ip_append_data
      0.25 ± 12%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.27 ±  4%      -0.1        0.19 ±  4%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.14 ± 14%      -0.1        0.06 ± 72%  perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.17 ± 16%      -0.1        0.09 ±  9%  perf-profile.self.cycles-pp.__check_object_size
      0.35            -0.1        0.27 ±  3%  perf-profile.self.cycles-pp.__check_heap_object
      0.24 ± 12%      -0.1        0.16 ± 11%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.40 ±  5%      -0.1        0.34 ±  3%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.21 ±  6%      -0.1        0.14 ±  5%  perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.45 ±  7%      -0.1        0.39 ±  4%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.12 ± 15%      -0.1        0.06 ± 19%  perf-profile.self.cycles-pp.skb_release_data
      0.15 ±  3%      -0.1        0.09 ± 22%  perf-profile.self.cycles-pp.nohz_run_idle_balance
      0.19 ± 10%      -0.1        0.14 ±  9%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.17 ± 10%      -0.1        0.11 ± 22%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.16 ± 17%      -0.1        0.11 ±  8%  perf-profile.self.cycles-pp.__xfrm_policy_check2
      0.32 ±  9%      -0.1        0.27 ± 12%  perf-profile.self.cycles-pp.do_syscall_64
      0.18 ± 17%      -0.1        0.13 ± 12%  perf-profile.self.cycles-pp.prepare_to_wait_exclusive
      0.18 ± 12%      -0.1        0.13 ± 25%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.17 ± 10%      -0.1        0.12 ± 21%  perf-profile.self.cycles-pp.ip_rcv_core
      0.21 ± 12%      -0.0        0.17 ±  7%  perf-profile.self.cycles-pp.vfs_read
      0.09 ±  9%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.12 ± 21%      -0.0        0.07        perf-profile.self.cycles-pp.apparmor_ip_postroute
      0.32 ±  3%      -0.0        0.28 ± 14%  perf-profile.self.cycles-pp.call_function_single_prep_ipi
      0.69 ±  4%      -0.0        0.65 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.08 ± 22%      -0.0        0.04 ± 70%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.20 ±  6%      -0.0        0.16 ± 15%  perf-profile.self.cycles-pp.sock_read_iter
      0.11 ±  4%      -0.0        0.07 ± 25%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.12 ± 13%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.13 ± 13%      -0.0        0.09 ± 10%  perf-profile.self.cycles-pp.read
      0.13 ± 16%      -0.0        0.09 ± 13%  perf-profile.self.cycles-pp.udp4_csum_init
      0.09 ±  5%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.10 ±  8%      -0.0        0.07 ± 11%  perf-profile.self.cycles-pp.ip_setup_cork
      0.31            -0.0        0.28        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.08 ± 14%      -0.0        0.06 ± 16%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.12 ±  3%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.call_cpuidle
      0.10 ± 17%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.sock_recvmsg
      0.08 ± 11%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.ipv4_mtu
      0.05 ±  8%      +0.0        0.07 ± 17%  perf-profile.self.cycles-pp.select_task_rq
      0.10 ±  4%      +0.0        0.12 ±  3%  perf-profile.self.cycles-pp.put_prev_entity
      0.09 ±  5%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__mkroute_output
      0.03 ± 70%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.21 ±  7%      +0.1        0.26 ±  3%  perf-profile.self.cycles-pp.rep_movs_alternative
      0.05 ±  8%      +0.1        0.11 ± 15%  perf-profile.self.cycles-pp.check_preempt_curr
      0.07 ± 23%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.06 ±  7%      +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.napi_threaded_poll_loop
      0.14 ±  8%      +0.1        0.20 ± 12%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.14 ± 12%      +0.1        0.21 ± 21%  perf-profile.self.cycles-pp.cpuacct_charge
      0.14 ±  6%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.prepare_task_switch
      0.04 ± 71%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.run_backlog_napi
      0.35 ±  7%      +0.1        0.44 ± 12%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.13 ± 18%      +0.1        0.22 ± 14%  perf-profile.self.cycles-pp.switch_fpu_return
      0.10 ±  4%      +0.1        0.20 ± 13%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.16 ±  3%      +0.1        0.26 ± 10%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.21 ±  4%      +0.1        0.31 ± 12%  perf-profile.self.cycles-pp.finish_task_switch
      0.08 ± 14%      +0.1        0.19 ± 10%  perf-profile.self.cycles-pp.mm_cid_get
      0.07 ± 18%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.13 ±  6%      +0.1        0.25 ± 19%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.09 ± 28%      +0.1        0.22 ±  3%  perf-profile.self.cycles-pp.update_min_vruntime
      0.15 ±  8%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.pick_next_entity
      0.00            +0.2        0.16 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.22 ±  5%      +0.2        0.38 ±  5%  perf-profile.self.cycles-pp.available_idle_cpu
      0.30 ±  9%      +0.2        0.46 ± 10%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.14 ±  8%      +0.2        0.31 ±  6%  perf-profile.self.cycles-pp.set_next_entity
      0.15 ± 13%      +0.2        0.33 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.67 ±  3%      +0.2        0.84 ± 12%  perf-profile.self.cycles-pp.___perf_sw_event
      0.14 ± 30%      +0.2        0.33 ±  7%  perf-profile.self.cycles-pp.schedule
      0.00            +0.2        0.22 ± 15%  perf-profile.self.cycles-pp.backlog_napi_should_run
      0.00            +0.2        0.24 ± 19%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.46 ±  4%      +0.3        0.72 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.25 ± 21%      +0.3        0.52 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.3        0.27 ± 15%  perf-profile.self.cycles-pp.smpboot_thread_fn
      0.57 ±  6%      +0.3        0.90 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.41 ±  9%      +0.3        0.74 ± 10%  perf-profile.self.cycles-pp.update_load_avg
      0.84 ± 12%      +0.4        1.24 ±  2%  perf-profile.self.cycles-pp.__schedule
      0.82 ±  3%      +0.4        1.25 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.87 ±  4%      +0.5        1.36 ±  5%  perf-profile.self.cycles-pp.__switch_to_asm
     20.66            +1.1       21.71        perf-profile.self.cycles-pp.poll_idle
     20.35            +2.7       23.05        perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-icl-2sp9: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/datasize/kconfig/nr_threads/protocol/rootfs/runtime/tbox_group/testcase/winsize:
  cs-localhost/gcc-12/performance/64/x86_64-rhel-8.3/1/UDP/debian-11.1-x86_64-20220510.cgz/30s/lkp-icl-2sp9/uperf/32k

commit: 
  83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode")
  e9dc861c8b ("net: Use SMP threads for backlog NAPI.")

83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   9008437           -11.7%    7958274        cpuidle..usage
    148752 ±  8%     +83.9%     273556 ± 20%  numa-numastat.node0.numa_hit
    377578           +60.9%     607490 ±  4%  vmstat.system.cs
    256190           -14.5%     219130 ±  3%  vmstat.system.in
      0.29            -0.3        0.03 ±  5%  mpstat.cpu.all.soft%
      0.95 ±  3%      +0.2        1.13 ±  6%  mpstat.cpu.all.sys%
      0.66 ±  5%      +0.1        0.79 ±  4%  mpstat.cpu.all.usr%
   8973056           -11.6%    7933641        turbostat.C1
      0.36 ±  4%     +16.6%       0.42 ±  4%  turbostat.IPC
   9276844           -11.5%    8213432        turbostat.IRQ
     14235 ±  5%     -18.1%      11654 ±  2%  turbostat.POLL
    502369           +39.1%     699008 ±  2%  meminfo.Active
    502289           +39.1%     698928 ±  2%  meminfo.Active(anon)
   1143243           +10.8%    1266813        meminfo.Committed_AS
    453349 ±  2%     -15.5%     382956        meminfo.Inactive
    453153 ±  2%     -15.5%     382760        meminfo.Inactive(anon)
    610710           +20.8%     737539 ±  2%  meminfo.Shmem
    154.79           -16.5%     129.28        uperf.Netstat_lo_in_Mbps
    154.79           -16.5%     129.28        uperf.Netstat_lo_out_Mbps
    107.68           -16.5%      89.93        uperf.Runstat_127.0.0.1_Mbps
    107.68           -16.5%      89.93        uperf.Runstat_master_Mbps
    372.17 ±  3%  +7.6e+05%    2838502        uperf.time.involuntary_context_switches
   3399305           -16.5%    2839422        uperf.time.voluntary_context_switches
      1297 ± 49%  +26455.4%     344423 ± 46%  numa-meminfo.node0.Active(anon)
     12288 ± 56%     +83.3%      22524 ± 23%  numa-meminfo.node0.AnonHugePages
      5413 ± 14%   +6557.4%     360421 ± 45%  numa-meminfo.node0.Shmem
    275637 ± 14%     -43.8%     154876 ± 19%  numa-meminfo.node1.Inactive
    275539 ± 14%     -43.9%     154681 ± 19%  numa-meminfo.node1.Inactive(anon)
      6538 ±  5%     +16.5%       7619 ±  5%  numa-meminfo.node1.KernelStack
    107211 ± 71%   +1246.5%    1443603 ± 87%  numa-meminfo.node1.Unevictable
    323.83 ± 49%  +26490.4%      86108 ± 46%  numa-vmstat.node0.nr_active_anon
      1353 ± 14%   +6559.8%      90106 ± 45%  numa-vmstat.node0.nr_shmem
    323.83 ± 49%  +26490.4%      86108 ± 46%  numa-vmstat.node0.nr_zone_active_anon
    148760 ±  8%     +83.9%     273535 ± 20%  numa-vmstat.node0.numa_hit
     68899 ± 14%     -43.9%      38681 ± 19%  numa-vmstat.node1.nr_inactive_anon
      6541 ±  5%     +16.5%       7619 ±  5%  numa-vmstat.node1.nr_kernel_stack
     26802 ± 71%   +1246.5%     360900 ± 87%  numa-vmstat.node1.nr_unevictable
     68899 ± 14%     -43.9%      38681 ± 19%  numa-vmstat.node1.nr_zone_inactive_anon
     26802 ± 71%   +1246.5%     360900 ± 87%  numa-vmstat.node1.nr_zone_unevictable
    125572           +39.2%     174744 ±  2%  proc-vmstat.nr_active_anon
    829950            +3.8%     861684        proc-vmstat.nr_file_pages
    113307 ±  2%     -15.5%      95696        proc-vmstat.nr_inactive_anon
     14124            +7.8%      15225        proc-vmstat.nr_kernel_stack
    152687           +20.8%     184396 ±  2%  proc-vmstat.nr_shmem
    125572           +39.2%     174744 ±  2%  proc-vmstat.nr_zone_active_anon
    113307 ±  2%     -15.5%      95696        proc-vmstat.nr_zone_inactive_anon
     13015 ± 54%    +142.4%      31549 ± 14%  proc-vmstat.numa_hint_faults
    540311            +7.6%     581575        proc-vmstat.numa_hit
    474106            +8.7%     515374        proc-vmstat.numa_local
    213704           -62.0%      81170 ±  7%  proc-vmstat.pgactivate
    570252            +7.0%     609921        proc-vmstat.pgalloc_normal
    293333 ±  4%     +11.2%     326321        proc-vmstat.pgfault
      0.00 ± 22%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
      0.00           -72.2%       0.00 ± 44%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00 ± 19%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.01           -40.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
      0.00 ± 24%     -40.7%       0.00 ± 17%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00 ± 74%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
      0.00 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.01 ±  5%  +11583.5%       1.54 ± 73%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      1.54 ± 84%     -99.3%       0.01 ± 22%  perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
      0.00           -33.3%       0.00        perf-sched.total_sch_delay.average.ms
      0.90 ±  2%     -36.4%       0.58 ±  2%  perf-sched.total_wait_and_delay.average.ms
    790959           +82.6%    1444641        perf-sched.total_wait_and_delay.count.ms
      4022 ±  6%     -13.4%       3481 ± 12%  perf-sched.total_wait_and_delay.max.ms
      0.90 ±  2%     -36.4%       0.57 ±  2%  perf-sched.total_wait_time.average.ms
      4022 ±  6%     -13.4%       3481 ± 12%  perf-sched.total_wait_time.max.ms
    224.93 ±  2%     -49.9%     112.72 ± 14%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     60.49           +96.1%     118.62 ±  8%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.01           -64.8%       0.00 ± 11%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
    425.27 ±  8%     -99.8%       0.94 ±  5%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    406.83           -56.0%     179.00 ±  9%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    784366           -38.8%     480361        perf-sched.wait_and_delay.count.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
    767.33 ±  5%  +62580.4%     480967        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.56 ± 82%     -93.3%       0.10 ±127%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
      0.01 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
    224.93 ±  2%     -49.9%     112.71 ± 14%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.04 ±180%  +17278.2%       6.11 ± 42%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.01 ± 24%  +54058.8%       3.07 ± 98%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
     60.49           +96.1%     118.62 ±  8%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.01           -47.2%       0.00 ± 11%  perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg
    425.27 ±  8%     -99.8%       0.94 ±  5%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.01 ± 40%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto
      1497 ± 33%     -33.2%       1000        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.10 ±193%  +12202.9%      12.10 ± 31%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.01 ± 39%  +60486.7%       4.54 ± 78%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.03 ± 41%  +26760.0%       6.94 ± 97%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      5.33 ±  2%     -18.9%       4.32 ±  2%  perf-stat.i.MPKI
 1.469e+09 ±  4%     +17.6%  1.727e+09        perf-stat.i.branch-instructions
      1.42            -0.2        1.21 ±  4%  perf-stat.i.branch-miss-rate%
      4.40 ±  4%      +2.8        7.16 ±  9%  perf-stat.i.cache-miss-rate%
   1666058           +57.2%    2618874 ± 10%  perf-stat.i.cache-misses
  34535681 ±  2%      -6.6%   32270735 ±  3%  perf-stat.i.cache-references
    415718 ±  2%     +61.8%     672627 ±  4%  perf-stat.i.context-switches
      1.22           -13.4%       1.06 ±  2%  perf-stat.i.cpi
    133.38 ±  2%     +25.9%     167.93 ±  2%  perf-stat.i.cpu-migrations
     18722 ±  7%     -20.6%      14861 ±  3%  perf-stat.i.cycles-between-cache-misses
      0.02 ± 15%      -0.0        0.01 ±  8%  perf-stat.i.dTLB-load-miss-rate%
 1.839e+09 ±  2%     +21.7%  2.237e+09        perf-stat.i.dTLB-loads
 9.513e+08 ±  2%     +25.3%  1.192e+09 ±  2%  perf-stat.i.dTLB-stores
 7.334e+09 ±  3%     +16.6%  8.551e+09        perf-stat.i.instructions
      0.90           +16.3%       1.04 ±  3%  perf-stat.i.ipc
     66.54 ±  3%     +21.0%      80.55        perf-stat.i.metric.M/sec
      5641 ±  7%     +15.0%       6485 ±  3%  perf-stat.i.minor-faults
    153120 ± 10%    +245.2%     528535 ± 19%  perf-stat.i.node-load-misses
    113594 ±  9%     -38.1%      70359 ±  6%  perf-stat.i.node-loads
     44891 ± 19%    +140.5%     107942 ± 20%  perf-stat.i.node-store-misses
    255691 ± 10%     +85.2%     473432 ±  7%  perf-stat.i.node-stores
      5641 ±  7%     +15.0%       6485 ±  3%  perf-stat.i.page-faults
      4.71 ±  3%     -19.9%       3.77 ±  2%  perf-stat.overall.MPKI
      1.52            -0.3        1.25 ±  2%  perf-stat.overall.branch-miss-rate%
      4.82 ±  3%      +3.3        8.11 ±  9%  perf-stat.overall.cache-miss-rate%
      1.08           -14.2%       0.93 ±  2%  perf-stat.overall.cpi
      4777 ±  2%     -35.7%       3070 ± 10%  perf-stat.overall.cycles-between-cache-misses
      0.02 ± 11%      -0.0        0.02 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      -0.0        0.00 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
      0.92           +16.6%       1.08 ±  2%  perf-stat.overall.ipc
     57.35 ±  7%     +30.6       87.91 ±  2%  perf-stat.overall.node-load-miss-rate%
 1.425e+09 ±  4%     +17.7%  1.678e+09        perf-stat.ps.branch-instructions
   1615384           +57.5%    2544001 ± 10%  perf-stat.ps.cache-misses
  33513906 ±  2%      -6.5%   31341570 ±  3%  perf-stat.ps.cache-references
    403480 ±  2%     +61.9%     653109 ±  4%  perf-stat.ps.context-switches
    129.49 ±  2%     +25.9%     163.08 ±  2%  perf-stat.ps.cpu-migrations
 1.784e+09 ±  2%     +21.8%  2.173e+09        perf-stat.ps.dTLB-loads
  9.23e+08 ±  2%     +25.4%  1.157e+09 ±  2%  perf-stat.ps.dTLB-stores
 7.116e+09 ±  3%     +16.7%  8.306e+09        perf-stat.ps.instructions
      5470 ±  7%     +15.1%       6298 ±  3%  perf-stat.ps.minor-faults
    148524 ± 10%    +245.7%     513437 ± 19%  perf-stat.ps.node-load-misses
    110108 ±  9%     -37.9%      68336 ±  6%  perf-stat.ps.node-loads
     43541 ± 19%    +140.8%     104863 ± 20%  perf-stat.ps.node-store-misses
    247950 ± 10%     +85.5%     459944 ±  7%  perf-stat.ps.node-stores
      5470 ±  7%     +15.1%       6298 ±  3%  perf-stat.ps.page-faults
 2.415e+11 ±  2%     +20.1%    2.9e+11 ±  4%  perf-stat.total.instructions
     19.24 ±  2%      -8.6       10.69 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto
     19.12 ±  2%      -8.5       10.60 ±  2%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto
      8.48 ±  2%      -8.5        0.00        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
     18.01 ±  2%      -8.4        9.57 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.35 ±  2%      -8.4        0.00        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb
     17.69 ±  2%      -8.3        9.35 ±  2%  perf-profile.calltrace.cycles-pp.udpv6_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      8.24 ±  2%      -8.2        0.00        perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
     17.20 ±  2%      -8.2        9.00 ±  2%  perf-profile.calltrace.cycles-pp.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      7.63 ±  3%      -7.6        0.00        perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
     11.38 ±  2%      -7.4        3.97 ±  2%  perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto
      7.32 ±  3%      -7.3        0.00        perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
     11.10 ±  2%      -7.3        3.84 ±  2%  perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg
      7.24 ±  3%      -7.2        0.00        perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq
     10.61 ±  2%      -7.1        3.47 ±  2%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg
     10.33 ±  2%      -7.0        3.35 ±  2%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
      6.96 ±  3%      -7.0        0.00        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq
      6.44 ±  3%      -6.4        0.00        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     19.76 ±  2%      -5.3       14.49 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_sendto
     19.55 ±  2%      -5.3       14.28 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto
     56.13 ±  3%      -4.9       51.27 ±  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
     20.38 ±  2%      -4.4       15.93 ±  2%  perf-profile.calltrace.cycles-pp.__libc_sendto
     58.25            -4.0       54.23        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      4.04 ± 73%      -3.6        0.42 ±223%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      4.02 ± 73%      -3.6        0.41 ±223%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
     51.35            -3.3       48.06        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     50.36            -3.2       47.12        perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     15.60 ±  4%      -2.0       13.63        perf-profile.calltrace.cycles-pp.__libc_recvfrom
     13.92 ±  5%      -1.7       12.26        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_recvfrom
     13.74 ±  5%      -1.6       12.10        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom
     13.35 ±  4%      -1.6       11.74        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom
     13.27 ±  5%      -1.6       11.68        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom
     12.03 ±  5%      -1.4       10.66 ±  2%  perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.90 ±  5%      -1.4       10.55 ±  2%  perf-profile.calltrace.cycles-pp.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     11.84 ±  5%      -1.3       10.51 ±  2%  perf-profile.calltrace.cycles-pp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.52 ±  6%      -0.6        0.93 ±  9%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      1.31 ±  6%      -0.5        0.79 ± 10%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      4.01 ±  4%      -0.3        3.66 ±  3%  perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto
      2.72 ±  5%      -0.2        2.48 ±  4%  perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg
      1.70 ±  5%      -0.2        1.52 ±  5%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.udpv6_sendmsg
      1.06 ±  9%      -0.2        0.90 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom
      1.54 ±  5%      -0.2        1.37 ±  6%  perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.00 ±  8%      -0.2        0.84 ±  3%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg
      0.94 ±  5%      -0.2        0.78 ±  7%  perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto
      0.83 ± 11%      -0.1        0.71 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg
      0.70 ±  3%      -0.1        0.59 ±  3%  perf-profile.calltrace.cycles-pp.txn_execute_once
      1.14 ±  8%      -0.1        1.04 ±  4%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg
      0.91 ±  3%      -0.1        0.82 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      0.64 ±  7%      +0.1        0.74 ±  4%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
      0.73 ±  8%      +0.1        0.87 ±  6%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single
      1.30 ±  6%      +0.2        1.46 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
      1.35 ±  6%      +0.2        1.53 ±  5%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single
      1.00 ±  8%      +0.2        1.22 ±  7%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.49 ± 45%      +0.2        0.71 ± 10%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.put_prev_entity.pick_next_task_fair.__schedule.schedule
      0.00            +0.6        0.58 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00            +0.6        0.59 ±  7%  perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.put_prev_entity.pick_next_task_fair.__schedule
      0.00            +0.7        0.68 ±  8%  perf-profile.calltrace.cycles-pp.available_idle_cpu.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.update_curr.put_prev_entity.pick_next_task_fair.__schedule.schedule
      0.00            +0.7        0.72 ±  9%  perf-profile.calltrace.cycles-pp.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function
      0.00            +0.7        0.74 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog
      0.00            +0.8        0.79 ±  3%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal
      0.84 ± 42%      +0.8        1.66 ± 13%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      0.00            +0.8        0.81 ±  7%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread
      0.90 ± 40%      +0.8        1.72 ± 13%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
      0.00            +1.1        1.11 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx
      0.00            +1.4        1.42 ±  4%  perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
      1.33 ±  4%      +1.7        3.04 ±  2%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
      1.24 ±  4%      +1.7        2.96 ±  2%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb
      0.00            +1.8        1.76 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit
      0.00            +1.9        1.86 ±  4%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +2.0        1.95 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit
      0.00            +2.0        1.98 ±  3%  perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.00            +2.0        2.00 ±  3%  perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      0.00            +2.5        2.45 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +2.6        2.62 ±  6%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +2.8        2.78 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +2.8        2.84 ±  3%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +3.0        3.00 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.2        3.23 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto
      0.00            +3.4        3.37 ±  3%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto
      0.00            +6.0        6.00 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop
      0.00            +6.5        6.50 ±  4%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn
      0.00            +6.8        6.76 ±  4%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread
      0.00            +6.8        6.82 ±  4%  perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +7.0        7.02 ±  4%  perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00           +10.1       10.12 ±  5%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00           +10.4       10.36 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      0.00           +10.4       10.36 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      0.00           +10.4       10.36 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     19.27 ±  2%      -8.6       10.71 ±  2%  perf-profile.children.cycles-pp.__x64_sys_sendto
     19.15 ±  2%      -8.5       10.62 ±  2%  perf-profile.children.cycles-pp.__sys_sendto
     18.04 ±  2%      -8.4        9.59 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
      8.54 ±  2%      -8.4        0.12 ± 17%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      8.37 ±  2%      -8.4        0.00        perf-profile.children.cycles-pp.do_softirq
     17.72 ±  2%      -8.3        9.38 ±  2%  perf-profile.children.cycles-pp.udpv6_sendmsg
     17.24 ±  2%      -8.2        9.02 ±  2%  perf-profile.children.cycles-pp.udp_sendmsg
      8.99 ±  2%      -8.2        0.78 ± 13%  perf-profile.children.cycles-pp.__do_softirq
      7.64 ±  3%      -7.6        0.00        perf-profile.children.cycles-pp.net_rx_action
     11.40 ±  2%      -7.4        3.98 ±  2%  perf-profile.children.cycles-pp.udp_send_skb
     11.12 ±  2%      -7.3        3.86 ±  2%  perf-profile.children.cycles-pp.ip_send_skb
     10.62 ±  2%      -7.2        3.47 ±  3%  perf-profile.children.cycles-pp.ip_finish_output2
     10.36 ±  2%      -7.0        3.37 ±  2%  perf-profile.children.cycles-pp.__dev_queue_xmit
     34.63 ±  2%      -6.9       27.78        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     34.26 ±  2%      -6.8       27.44        perf-profile.children.cycles-pp.do_syscall_64
     20.56 ±  2%      -4.5       16.10 ±  2%  perf-profile.children.cycles-pp.__libc_sendto
     58.25            -4.0       54.23        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     58.25            -4.0       54.23        perf-profile.children.cycles-pp.cpu_startup_entry
     58.16            -4.0       54.16        perf-profile.children.cycles-pp.do_idle
     54.41            -3.4       50.99        perf-profile.children.cycles-pp.cpuidle_idle_call
     51.50            -3.2       48.30        perf-profile.children.cycles-pp.cpuidle_enter_state
     51.76            -3.2       48.57        perf-profile.children.cycles-pp.cpuidle_enter
     50.47            -3.2       47.32 ±  2%  perf-profile.children.cycles-pp.acpi_idle_enter
     50.40            -3.1       47.27 ±  2%  perf-profile.children.cycles-pp.acpi_safe_halt
     32.12 ±  3%      -2.5       29.60 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
     15.77 ±  4%      -2.0       13.77        perf-profile.children.cycles-pp.__libc_recvfrom
     13.39 ±  4%      -1.6       11.77        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     13.30 ±  5%      -1.6       11.70        perf-profile.children.cycles-pp.__sys_recvfrom
     12.04 ±  5%      -1.4       10.66 ±  2%  perf-profile.children.cycles-pp.sock_recvmsg
     11.91 ±  5%      -1.4       10.55 ±  2%  perf-profile.children.cycles-pp.inet6_recvmsg
     11.86 ±  4%      -1.3       10.52 ±  2%  perf-profile.children.cycles-pp.udpv6_recvmsg
      0.71 ± 11%      -0.7        0.05 ± 47%  perf-profile.children.cycles-pp.select_idle_sibling
      5.66 ±  3%      -0.5        5.18 ±  5%  perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
      0.81 ±  3%      -0.4        0.40 ±  5%  perf-profile.children.cycles-pp.irqtime_account_irq
      2.52 ±  6%      -0.4        2.11 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      4.02 ±  4%      -0.4        3.67 ±  3%  perf-profile.children.cycles-pp.ip_make_skb
      2.74 ±  5%      -0.2        2.49 ±  4%  perf-profile.children.cycles-pp.__ip_append_data
      1.72 ±  6%      -0.2        1.54 ±  5%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.08 ±  9%      -0.2        0.90 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      1.03 ±  9%      -0.2        0.86 ±  3%  perf-profile.children.cycles-pp.copyout
      0.94 ±  5%      -0.2        0.79 ±  8%  perf-profile.children.cycles-pp.ip_route_output_flow
      1.56 ±  5%      -0.1        1.42 ±  5%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.84 ± 11%      -0.1        0.72 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.63 ±  9%      -0.1        0.52 ±  5%  perf-profile.children.cycles-pp.move_addr_to_user
      0.54 ±  4%      -0.1        0.44 ±  5%  perf-profile.children.cycles-pp.__entry_text_start
      0.58 ±  7%      -0.1        0.49 ±  6%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.72 ±  3%      -0.1        0.63 ±  3%  perf-profile.children.cycles-pp.txn_execute_once
      0.24 ± 12%      -0.1        0.18 ± 16%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.29 ±  9%      -0.1        0.22 ±  6%  perf-profile.children.cycles-pp.stats_update
      0.19 ± 12%      -0.1        0.14 ± 13%  perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      0.09 ± 26%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.29 ±  3%      -0.0        0.24 ±  5%  perf-profile.children.cycles-pp._copy_from_user
      0.12 ± 13%      -0.0        0.08 ± 13%  perf-profile.children.cycles-pp.dst_release
      0.16 ±  7%      -0.0        0.12 ± 13%  perf-profile.children.cycles-pp.ip_rcv_core
      0.05 ± 46%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.check_spread
      0.03 ±100%      +0.0        0.08 ± 14%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.09 ± 14%      +0.0        0.14 ± 16%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.04 ± 71%      +0.0        0.08 ± 16%  perf-profile.children.cycles-pp.trigger_load_balance
      0.10 ± 20%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.rb_erase
      0.08 ± 20%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.__list_add_valid
      0.10 ± 23%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.update_min_vruntime
      0.04 ± 73%      +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.clear_buddies
      0.12 ± 15%      +0.1        0.18 ±  9%  perf-profile.children.cycles-pp.__calc_delta
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.10 ± 22%      +0.1        0.19 ± 10%  perf-profile.children.cycles-pp.switch_fpu_return
      0.09 ± 11%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.pick_next_entity
      0.02 ±141%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
      0.18 ±  8%      +0.1        0.30 ± 11%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.13 ±  7%      +0.1        0.26 ± 11%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.27 ±  9%      +0.2        0.43 ± 10%  perf-profile.children.cycles-pp.finish_task_switch
      0.26 ± 10%      +0.2        0.42 ±  7%  perf-profile.children.cycles-pp.os_xsave
      0.32 ±  8%      +0.2        0.55 ±  7%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.65 ±  7%      +0.2        0.89 ±  9%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.2        0.24 ±  7%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.73 ±  6%      +0.3        1.00 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.85 ± 11%      +0.3        1.16 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      0.33 ±  5%      +0.3        0.68 ±  7%  perf-profile.children.cycles-pp.update_blocked_averages
      0.84 ± 13%      +0.4        1.19 ±  7%  perf-profile.children.cycles-pp.check_preempt_curr
      0.52            +0.4        0.89 ±  3%  perf-profile.children.cycles-pp.__switch_to_asm
      0.30 ± 23%      +0.4        0.71 ±  8%  perf-profile.children.cycles-pp.available_idle_cpu
      0.73 ±  8%      +0.4        1.18 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.75 ±  8%      +0.5        1.21 ±  5%  perf-profile.children.cycles-pp.prepare_task_switch
      0.84 ±  4%      +0.5        1.36 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.07 ± 23%      +0.6        0.66 ±  6%  perf-profile.children.cycles-pp.cpuacct_charge
      0.42 ± 11%      +0.7        1.08 ±  4%  perf-profile.children.cycles-pp.set_next_entity
      0.35 ±  8%      +0.7        1.02 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.06 ± 18%      +0.7        0.72 ±  9%  perf-profile.children.cycles-pp.wake_affine
      1.25 ± 11%      +0.7        1.93 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.59 ±  7%      +0.8        1.36 ±  3%  perf-profile.children.cycles-pp.update_curr
      1.49 ± 10%      +0.8        2.28 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.54 ± 11%      +0.8        2.37 ±  3%  perf-profile.children.cycles-pp.activate_task
      0.89 ± 40%      +0.9        1.76 ± 12%  perf-profile.children.cycles-pp.newidle_balance
      2.42 ± 11%      +1.2        3.64 ±  4%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.35 ±  7%      +1.4        2.75 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      0.10 ± 24%      +1.5        1.55 ±  4%  perf-profile.children.cycles-pp.put_prev_entity
      3.50 ±  4%      +1.5        4.99 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      1.34 ±  4%      +1.7        3.05 ±  2%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.27 ±  4%      +1.7        2.99 ±  2%  perf-profile.children.cycles-pp.loopback_xmit
      0.23 ±  5%      +1.7        1.97 ±  3%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.26 ±  6%      +1.7        2.00 ±  3%  perf-profile.children.cycles-pp.__netif_rx
      0.25 ±  8%      +1.7        1.99 ±  3%  perf-profile.children.cycles-pp.netif_rx_internal
      0.00            +3.0        3.03 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.21 ± 14%      +3.2        3.38 ±  2%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.38 ±  5%      +3.2        3.62 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      1.58 ± 22%      +3.5        5.04 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
      6.38 ±  6%      +4.7       11.09 ±  4%  perf-profile.children.cycles-pp.__schedule
      4.20 ± 11%      +5.2        9.45 ±  5%  perf-profile.children.cycles-pp.schedule
      0.00            +7.0        7.04 ±  4%  perf-profile.children.cycles-pp.napi_threaded_poll_loop
      0.24 ±  7%     +10.1       10.37 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
      0.24 ±  7%     +10.1       10.37 ±  5%  perf-profile.children.cycles-pp.ret_from_fork_asm
      0.23 ±  7%     +10.1       10.36 ±  5%  perf-profile.children.cycles-pp.kthread
      0.00           +10.2       10.16 ±  5%  perf-profile.children.cycles-pp.smpboot_thread_fn
     34.07 ±  2%      -2.5       31.56 ±  2%  perf-profile.self.cycles-pp.acpi_safe_halt
      0.69 ±  7%      -0.2        0.50 ±  8%  perf-profile.self.cycles-pp.udp_sendmsg
      0.92 ±  8%      -0.2        0.75 ±  4%  perf-profile.self.cycles-pp.__skb_recv_udp
      1.02 ±  9%      -0.2        0.86 ±  3%  perf-profile.self.cycles-pp.copyout
      0.23 ± 13%      -0.2        0.07 ± 17%  perf-profile.self.cycles-pp.ip_finish_output2
      0.22 ±  7%      -0.1        0.08 ± 16%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.23 ± 13%      -0.1        0.09 ± 14%  perf-profile.self.cycles-pp.udp_send_skb
      0.83 ± 11%      -0.1        0.70 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.47 ±  9%      -0.1        0.35 ±  8%  perf-profile.self.cycles-pp.udpv6_sendmsg
      0.20 ± 15%      -0.1        0.10 ± 13%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.57 ±  7%      -0.1        0.48 ±  6%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.68 ±  3%      -0.1        0.60 ±  4%  perf-profile.self.cycles-pp.txn_execute_once
      0.28 ± 10%      -0.1        0.20 ± 12%  perf-profile.self.cycles-pp.schedule_timeout
      0.28 ± 10%      -0.1        0.21 ±  4%  perf-profile.self.cycles-pp.stats_update
      0.18 ± 10%      -0.1        0.11 ± 12%  perf-profile.self.cycles-pp.sock_sendmsg
      0.16 ±  9%      -0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.60 ±  6%      -0.1        0.53 ±  6%  perf-profile.self.cycles-pp.menu_select
      0.29 ±  8%      -0.1        0.23 ± 13%  perf-profile.self.cycles-pp.udpv6_recvmsg
      0.30 ±  4%      -0.1        0.24 ±  5%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.42 ±  8%      -0.1        0.36 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.18 ± 13%      -0.1        0.13 ± 11%  perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      0.28 ±  3%      -0.0        0.23 ±  4%  perf-profile.self.cycles-pp._copy_from_user
      0.20 ±  5%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.__entry_text_start
      0.12 ± 11%      -0.0        0.09 ± 21%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.08 ± 15%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.08 ± 21%      +0.0        0.12 ± 12%  perf-profile.self.cycles-pp.rb_erase
      0.28 ±  6%      +0.0        0.32 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.02 ± 99%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.03 ±100%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.check_spread
      0.05 ± 50%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.09 ± 22%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.update_min_vruntime
      0.02 ±141%      +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.trigger_load_balance
      0.06 ± 49%      +0.1        0.12 ±  5%  perf-profile.self.cycles-pp.__list_add_valid
      0.01 ±223%      +0.1        0.07 ± 21%  perf-profile.self.cycles-pp.find_busiest_group
      0.00            +0.1        0.06 ± 28%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.28 ± 14%      +0.1        0.34 ±  5%  perf-profile.self.cycles-pp.update_curr
      0.10 ± 18%      +0.1        0.17 ± 18%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.11 ± 16%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.__calc_delta
      0.08 ± 19%      +0.1        0.16 ± 18%  perf-profile.self.cycles-pp.load_balance
      0.01 ±223%      +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.activate_task
      0.04 ± 72%      +0.1        0.11 ± 26%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.11 ± 14%      +0.1        0.18 ± 15%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.03 ±100%      +0.1        0.11 ± 13%  perf-profile.self.cycles-pp.update_blocked_averages
      0.08 ± 11%      +0.1        0.17 ± 16%  perf-profile.self.cycles-pp.pick_next_entity
      0.10 ± 24%      +0.1        0.18 ± 13%  perf-profile.self.cycles-pp.switch_fpu_return
      0.22 ± 18%      +0.1        0.31 ±  9%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.08 ± 19%      +0.1        0.17 ± 12%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.09 ±  5%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.18 ±  7%      +0.1        0.28 ± 11%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.04 ± 71%      +0.1        0.16 ± 20%  perf-profile.self.cycles-pp.put_prev_entity
      0.11 ± 15%      +0.1        0.25 ±  7%  perf-profile.self.cycles-pp.set_next_entity
      0.23 ± 10%      +0.1        0.38 ± 13%  perf-profile.self.cycles-pp.finish_task_switch
      0.00            +0.2        0.15 ± 14%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.30 ± 14%      +0.2        0.45 ± 10%  perf-profile.self.cycles-pp.enqueue_entity
      0.26 ± 11%      +0.2        0.42 ±  6%  perf-profile.self.cycles-pp.os_xsave
      0.18 ± 18%      +0.2        0.34 ±  8%  perf-profile.self.cycles-pp.schedule
      0.00            +0.2        0.16 ± 18%  perf-profile.self.cycles-pp.napi_threaded_poll_loop
      0.58 ±  6%      +0.2        0.78 ± 10%  perf-profile.self.cycles-pp.___perf_sw_event
      0.16 ± 12%      +0.2        0.36 ±  8%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.32 ±  6%      +0.2        0.54 ±  6%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.11 ± 20%      +0.2        0.33 ± 11%  perf-profile.self.cycles-pp.prepare_task_switch
      0.68 ±  5%      +0.3        0.94 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.52            +0.4        0.88 ±  3%  perf-profile.self.cycles-pp.__switch_to_asm
      0.00            +0.4        0.36 ± 11%  perf-profile.self.cycles-pp.smpboot_thread_fn
      0.32 ± 12%      +0.4        0.68 ±  6%  perf-profile.self.cycles-pp.update_load_avg
      0.69 ±  5%      +0.4        1.07 ±  6%  perf-profile.self.cycles-pp.__schedule
      0.28 ± 24%      +0.4        0.70 ±  9%  perf-profile.self.cycles-pp.available_idle_cpu
      0.70 ±  8%      +0.4        1.13 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.82 ±  5%      +0.5        1.30 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.07 ± 23%      +0.6        0.66 ±  6%  perf-profile.self.cycles-pp.cpuacct_charge
      0.31 ±  7%      +0.6        0.92 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_se



***************************************************************************************************
lkp-csl-d02: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/lkp-csl-d02/dccp/stress-ng/60s

commit: 
  83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode")
  e9dc861c8b ("net: Use SMP threads for backlog NAPI.")

83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   5871892 ±  2%     +18.5%    6957517 ±  2%  cpuidle..usage
      3.59            -7.8%       3.31 ±  2%  iostat.cpu.system
    643.03 ±  5%     -23.5%     492.14 ± 17%  sched_debug.cfs_rq:/.min_vruntime.min
    111973 ±  4%    +143.4%     272574 ±  4%  vmstat.system.cs
      0.33 ±  2%      -0.3        0.04 ±  7%  mpstat.cpu.all.soft%
      0.32            +0.0        0.36 ±  2%  mpstat.cpu.all.usr%
     45636 ±  3%    +517.7%     281909        turbostat.C1
      0.14 ±  5%      +0.2        0.31 ±  4%  turbostat.C1%
      4.78 ±  2%      -0.3        4.49 ±  2%  turbostat.C1E%
   2544029 ±  6%     +35.4%    3445596 ±  4%  turbostat.POLL
      0.10 ±  5%      +0.1        0.22 ±  4%  turbostat.POLL%
    156618 ±  5%    +149.8%     391188        meminfo.Active
    140656 ±  6%    +166.8%     375277        meminfo.Active(anon)
     66148 ± 12%     -17.1%      54847 ± 11%  meminfo.AnonHugePages
    661727           +39.5%     922917        meminfo.Committed_AS
     43445           +24.5%      54073 ±  8%  meminfo.Mapped
    154214 ±  5%    +168.1%     413503        meminfo.Shmem
     77137 ±  2%     -16.0%      64808 ±  4%  stress-ng.dccp.messages_per_sec
     73462 ±  2%     -16.0%      61719 ±  4%  stress-ng.dccp.ops
      1224 ±  2%     -16.0%       1028 ±  4%  stress-ng.dccp.ops_per_sec
    832.67 ±  3%    +5e+05%    4159259 ±  4%  stress-ng.time.involuntary_context_switches
     95.00           -14.9%      80.83 ±  5%  stress-ng.time.percent_of_cpu_this_job_got
     58.88           -15.8%      49.55 ±  5%  stress-ng.time.system_time
   2727584 ±  5%     +41.9%    3870138 ±  4%  stress-ng.time.voluntary_context_switches
     35165 ±  6%    +166.8%      93819        proc-vmstat.nr_active_anon
    754437            +8.6%     819232        proc-vmstat.nr_file_pages
     79513            +7.6%      85594        proc-vmstat.nr_inactive_anon
      8763            +6.8%       9356        proc-vmstat.nr_kernel_stack
     10863           +24.5%      13522 ±  8%  proc-vmstat.nr_mapped
     38558 ±  5%    +168.1%     103378        proc-vmstat.nr_shmem
     23483            +3.3%      24247        proc-vmstat.nr_slab_unreclaimable
     35165 ±  6%    +166.8%      93819        proc-vmstat.nr_zone_active_anon
     79513            +7.6%      85594        proc-vmstat.nr_zone_inactive_anon
    443490           +15.7%     513117        proc-vmstat.numa_hit
    445922 ±  2%     +15.0%     512943        proc-vmstat.numa_local
     52965 ±  5%    +168.2%     142076        proc-vmstat.pgactivate
   1505891            -6.2%    1412250 ±  2%  proc-vmstat.pgalloc_normal
    212490            +8.2%     229851        proc-vmstat.pgfault
   1425607           -13.0%    1239802 ±  3%  proc-vmstat.pgfree
     17.80 ±  7%     -13.7%      15.37 ±  7%  perf-stat.i.MPKI
  11245386            +6.4%   11967086 ±  2%  perf-stat.i.branch-misses
      1.46 ±  6%      +0.7        2.16 ±  7%  perf-stat.i.cache-miss-rate%
    828103 ±  4%     +56.9%    1299520 ±  2%  perf-stat.i.cache-misses
 1.027e+08 ±  2%     -14.3%   88075829 ±  4%  perf-stat.i.cache-references
    117427 ±  4%    +143.8%     286319 ±  4%  perf-stat.i.context-switches
     59.68 ±  2%   +1659.6%       1050 ±  8%  perf-stat.i.cpu-migrations
     39134 ± 22%     -55.6%      17366 ±  5%  perf-stat.i.cycles-between-cache-misses
    106602 ± 16%     +48.6%     158428 ± 17%  perf-stat.i.dTLB-load-misses
     84.77            -4.0       80.77        perf-stat.i.iTLB-load-miss-rate%
   3709152 ±  2%     -14.1%    3186224 ±  5%  perf-stat.i.iTLB-load-misses
    584638 ±  4%     +12.1%     655260 ±  2%  perf-stat.i.iTLB-loads
      1964           +13.0%       2221 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      1864           +15.4%       2152        perf-stat.i.minor-faults
     88295 ±  6%     +70.7%     150709 ±  3%  perf-stat.i.node-loads
    137528 ±  7%     +30.7%     179725 ±  5%  perf-stat.i.node-stores
      1864           +15.4%       2152        perf-stat.i.page-faults
     16.23           -15.6%      13.71        perf-stat.overall.MPKI
      0.81 ±  6%      +0.7        1.48 ±  4%  perf-stat.overall.cache-miss-rate%
      6907 ±  5%     -35.3%       4467 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.01 ± 18%      +0.0        0.01 ± 22%  perf-stat.overall.dTLB-load-miss-rate%
     86.38            -3.5       82.92        perf-stat.overall.iTLB-load-miss-rate%
      1705           +18.3%       2017        perf-stat.overall.instructions-per-iTLB-miss
  11070217            +6.4%   11778477 ±  2%  perf-stat.ps.branch-misses
    815402 ±  4%     +56.9%    1279495 ±  2%  perf-stat.ps.cache-misses
 1.011e+08 ±  2%     -14.3%   86684499 ±  4%  perf-stat.ps.cache-references
    115587 ±  4%    +143.8%     281791 ±  4%  perf-stat.ps.context-switches
     58.75 ±  2%   +1659.1%       1033 ±  8%  perf-stat.ps.cpu-migrations
    104986 ± 16%     +48.6%     156010 ± 17%  perf-stat.ps.dTLB-load-misses
   3651135 ±  2%     -14.1%    3135989 ±  5%  perf-stat.ps.iTLB-load-misses
    575492 ±  4%     +12.1%     644966 ±  2%  perf-stat.ps.iTLB-loads
      1835           +15.4%       2119        perf-stat.ps.minor-faults
     86937 ±  6%     +70.7%     148402 ±  3%  perf-stat.ps.node-loads
    135391 ±  7%     +30.7%     176929 ±  5%  perf-stat.ps.node-stores
      1835           +15.4%       2119        perf-stat.ps.page-faults
      0.00 ± 13%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg
      0.00 ± 31%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00           -50.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.do_accept
      0.00          +100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
      0.00           -25.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00 ±223%  +19800.0%       0.10 ±109%  perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg
      0.01 ± 43%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg
      0.00 ± 95%    +450.0%       0.02 ± 26%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.00 ± 11%   +6389.5%       0.21 ±101%  perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.05 ±129%   +2212.4%       1.18 ± 85%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00          +100.0%       0.00        perf-sched.total_sch_delay.average.ms
      2.39 ±  5%     -69.4%       0.73        perf-sched.total_wait_and_delay.average.ms
    194690 ±  5%    +286.0%     751492        perf-sched.total_wait_and_delay.count.ms
      2.39 ±  5%     -69.5%       0.73        perf-sched.total_wait_time.average.ms
    783.24 ±  4%     -19.2%     633.25 ± 14%  perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    147.40 ±  5%     -61.4%      56.84 ±  2%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.00 ± 22%    +170.0%       0.01        perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
    454.52 ±  9%     -99.7%       1.31 ±  2%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    456.60 ±  4%     +16.0%     529.45 ±  2%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.33 ±141%   +2200.0%       7.67 ± 23%  perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio
      2.00 ± 50%   +1025.0%      22.50 ± 20%  perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write
      0.17 ±223%   +6000.0%      10.17 ± 18%  perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
      1.00 ±223%  +2.2e+07%     216471        perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
    113.67 ±  5%    +143.0%     276.17        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      9952 ±  2%     -32.4%       6730        perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.do_accept
    104457 ±  9%    +106.7%     215888        perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
    486.17 ±  9%  +46231.5%     225248        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      4.62 ±223%   +4350.7%     205.65 ±173%  perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
     19.40 ±223%   +4222.6%     838.77 ± 43%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      6.33 ± 47%     -73.0%       1.71 ± 54%  perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
      0.00 ± 81%    +969.2%       0.02 ± 73%  perf-sched.wait_time.avg.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg
    783.24 ±  4%     -19.2%     633.24 ± 14%  perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    147.40 ±  5%     -61.4%      56.84 ±  2%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.00 ± 22%    +105.0%       0.01 ±  5%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
    454.52 ±  9%     -99.7%       1.31 ±  2%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    456.59 ±  4%     +16.0%     529.45 ±  2%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.01 ± 70%    +168.6%       0.02 ±  6%  perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.dccp_feat_entry_new.dccp_feat_signal_nn_change
      0.00 ± 88%    +1e+05%       2.43 ± 85%  perf-sched.wait_time.max.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg
      0.00 ±101%    +443.5%       0.02 ±  8%  perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.task_work_run.exit_to_user_mode_loop
      4.62 ±223%   +4350.7%     205.64 ±173%  perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop
    166.68 ±223%    +400.9%     834.86 ± 44%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
     23.59 ±177%   +3455.0%     838.76 ± 43%  perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      6.32 ± 47%     -73.4%       1.68 ± 55%  perf-sched.wait_time.max.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
     42.69 ±  7%      -7.7       34.98 ±  3%  perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     42.69 ±  7%      -7.7       34.98 ±  3%  perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     42.70 ±  7%      -7.7       35.00 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
     42.70 ±  7%      -7.7       35.00 ±  3%  perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
     42.71 ±  7%      -7.7       35.01 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
     42.71 ±  7%      -7.7       35.01 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.connect
     42.72 ±  7%      -7.7       35.02 ±  3%  perf-profile.calltrace.cycles-pp.connect
     42.12 ±  7%      -7.2       34.92 ±  3%  perf-profile.calltrace.cycles-pp.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
     41.08 ±  7%      -6.9       34.22 ±  3%  perf-profile.calltrace.cycles-pp.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
     28.16 ±  8%      -4.7       23.48 ±  3%  perf-profile.calltrace.cycles-pp.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect
      5.44 ± 19%      -3.8        1.64 ±  2%  perf-profile.calltrace.cycles-pp.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      5.39 ± 19%      -3.8        1.62 ±  2%  perf-profile.calltrace.cycles-pp.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto
      8.41 ± 15%      -3.7        4.69 ± 12%  perf-profile.calltrace.cycles-pp.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
      8.55 ± 14%      -3.6        4.90 ± 13%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.79 ± 14%      -3.4        5.36 ± 13%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      8.83 ± 15%      -3.4        5.41 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      3.54 ± 17%      -2.7        0.86 ±  5%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit
      3.50 ± 17%      -2.7        0.83 ±  6%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet
      3.74 ± 18%      -2.7        1.08 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg
      8.95 ± 14%      -2.6        6.32 ± 12%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      9.04 ± 15%      -2.6        6.43 ± 12%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
      3.91 ± 18%      -2.5        1.38 ±  4%  perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg
      9.29 ± 14%      -2.1        7.15 ± 12%  perf-profile.calltrace.cycles-pp.__send
      5.09 ±  7%      -1.6        3.51 ±  6%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      5.10 ±  6%      -1.6        3.53 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      4.93 ±  6%      -1.5        3.38 ±  7%  perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.19 ±  6%      -1.5        3.66 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      5.22 ±  6%      -1.5        3.69 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv
      5.56 ±  6%      -1.5        4.04 ±  6%  perf-profile.calltrace.cycles-pp.recv
      4.81 ±  6%      -1.5        3.30 ±  7%  perf-profile.calltrace.cycles-pp.dccp_recvmsg.sock_common_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      4.82 ±  6%      -1.5        3.32 ±  7%  perf-profile.calltrace.cycles-pp.sock_common_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      6.63 ±  8%      -1.2        5.41 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect
      3.82 ±  8%      -0.7        3.14 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect
      2.98 ± 10%      -0.4        2.56 ±  6%  perf-profile.calltrace.cycles-pp.inet_ehashfn.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect
      1.25 ±  9%      -0.2        1.10 ±  6%  perf-profile.calltrace.cycles-pp.__cond_resched.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect
      0.65 ± 13%      +0.2        0.86 ±  8%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
      0.69 ± 13%      +0.2        0.92 ±  8%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg
      0.54 ± 46%      +0.3        0.82 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      1.03 ± 26%      +0.4        1.39 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established
      0.91 ± 15%      +0.4        1.31 ±  8%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg.sock_recvmsg
      0.80 ± 26%      +0.4        1.21 ±  7%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv
      0.83 ± 25%      +0.4        1.24 ±  7%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids
      0.58 ± 51%      +0.4        1.00 ± 10%  perf-profile.calltrace.cycles-pp.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
      0.61 ± 51%      +0.4        1.06 ±  9%  perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      0.45 ± 74%      +0.4        0.90 ± 11%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit
      0.65 ±  9%      +0.5        1.11 ± 23%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      0.00            +0.6        0.57 ±  9%  perf-profile.calltrace.cycles-pp.dccp_rcv_state_process.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.22 ±143%      +0.6        0.80 ± 13%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit
      0.10 ±223%      +0.6        0.68 ± 12%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree
      0.00            +0.6        0.58 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +0.6        0.62 ±  9%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.68 ± 12%      +0.6        1.31 ± 11%  perf-profile.calltrace.cycles-pp.clock_nanosleep
      0.00            +0.7        0.65 ±  9%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.18 ±141%      +0.7        0.90 ± 24%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto
      0.00            +0.7        0.74 ±  9%  perf-profile.calltrace.cycles-pp.__dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.00            +0.7        0.74 ± 10%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.40 ± 71%      +0.7        1.14 ± 12%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.38 ± 71%      +0.8        1.13 ± 12%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.09 ±223%      +0.8        0.86 ± 24%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg
      0.26 ±100%      +0.8        1.03 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.27 ±100%      +0.8        1.04 ± 12%  perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.00            +0.8        0.79 ± 11%  perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep
      0.00            +0.8        0.80 ± 12%  perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.00            +0.8        0.80 ±  8%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.28 ±100%      +0.8        1.10 ± 12%  perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep
      0.09 ±223%      +0.9        0.98 ± 12%  perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64
      1.66 ± 23%      +0.9        2.59 ±  7%  perf-profile.calltrace.cycles-pp.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.00            +0.9        0.93 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +1.0        0.97 ±  6%  perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.64 ± 50%      +1.1        1.72 ±  5%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
      0.65 ± 51%      +1.1        1.76 ±  5%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb
      2.36 ± 20%      +1.8        4.14 ±  6%  perf-profile.calltrace.cycles-pp.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      2.05 ± 21%      +1.9        3.92 ±  5%  perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      1.63 ± 14%      +1.9        3.56 ±  4%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      3.16 ± 19%      +2.0        5.14 ±  6%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     40.13 ±  6%      +3.6       43.69 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.00            +5.2        5.16 ±  6%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop
      0.00            +5.5        5.46 ±  5%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn
      0.00            +5.6        5.60 ±  5%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread
      0.00            +5.6        5.62 ±  5%  perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork
      0.00            +5.7        5.66 ±  5%  perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +6.8        6.80 ±  5%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +6.9        6.87 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      0.00            +6.9        6.87 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      0.00            +6.9        6.87 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     58.36 ±  4%     -11.2       47.14 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
     58.49 ±  4%     -11.2       47.32 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     42.69 ±  7%      -7.7       34.98 ±  3%  perf-profile.children.cycles-pp.__inet_stream_connect
     42.69 ±  7%      -7.7       34.98 ±  3%  perf-profile.children.cycles-pp.inet_stream_connect
     42.70 ±  7%      -7.7       35.00 ±  3%  perf-profile.children.cycles-pp.__x64_sys_connect
     42.70 ±  7%      -7.7       35.00 ±  3%  perf-profile.children.cycles-pp.__sys_connect
     42.72 ±  7%      -7.7       35.02 ±  3%  perf-profile.children.cycles-pp.connect
     42.12 ±  7%      -7.2       34.92 ±  3%  perf-profile.children.cycles-pp.dccp_v4_connect
     41.64 ±  7%      -6.9       34.70 ±  3%  perf-profile.children.cycles-pp.__inet_hash_connect
      6.16 ± 10%      -5.1        1.04 ± 12%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      5.63 ± 15%      -5.1        0.56 ± 18%  perf-profile.children.cycles-pp.__do_softirq
      5.04 ± 13%      -5.0        0.04 ± 75%  perf-profile.children.cycles-pp.do_softirq
     28.38 ±  8%      -4.7       23.67 ±  3%  perf-profile.children.cycles-pp.__inet_check_established
      5.46 ± 19%      -3.8        1.66 ±  3%  perf-profile.children.cycles-pp.dccp_write_xmit
      5.39 ± 19%      -3.8        1.63 ±  3%  perf-profile.children.cycles-pp.dccp_xmit_packet
      8.42 ± 15%      -3.7        4.71 ± 12%  perf-profile.children.cycles-pp.dccp_sendmsg
      8.55 ± 14%      -3.6        4.90 ± 13%  perf-profile.children.cycles-pp.sock_sendmsg
      6.19 ± 13%      -3.5        2.69 ±  4%  perf-profile.children.cycles-pp.__ip_queue_xmit
      6.80 ± 12%      -3.4        3.36 ±  5%  perf-profile.children.cycles-pp.dccp_transmit_skb
      5.73 ± 12%      -3.4        2.30 ±  3%  perf-profile.children.cycles-pp.ip_finish_output2
      8.80 ± 14%      -3.4        5.37 ± 13%  perf-profile.children.cycles-pp.__sys_sendto
      8.84 ± 14%      -3.4        5.41 ± 13%  perf-profile.children.cycles-pp.__x64_sys_sendto
      5.64 ± 12%      -3.4        2.22 ±  3%  perf-profile.children.cycles-pp.__dev_queue_xmit
      2.92 ±  5%      -2.7        0.26 ± 15%  perf-profile.children.cycles-pp.__release_sock
      3.25 ±  5%      -2.7        0.60 ± 12%  perf-profile.children.cycles-pp.release_sock
      9.37 ± 14%      -2.1        7.31 ± 12%  perf-profile.children.cycles-pp.__send
      5.09 ±  6%      -1.6        3.51 ±  6%  perf-profile.children.cycles-pp.__sys_recvfrom
      5.10 ±  6%      -1.6        3.53 ±  6%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
      4.93 ±  6%      -1.5        3.38 ±  7%  perf-profile.children.cycles-pp.sock_recvmsg
      5.61 ±  6%      -1.5        4.07 ±  6%  perf-profile.children.cycles-pp.recv
      4.81 ±  6%      -1.5        3.30 ±  7%  perf-profile.children.cycles-pp.dccp_recvmsg
      4.82 ±  6%      -1.5        3.32 ±  7%  perf-profile.children.cycles-pp.sock_common_recvmsg
      7.76 ±  7%      -1.5        6.28 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      5.18 ±  8%      -1.1        4.12 ±  6%  perf-profile.children.cycles-pp.dccp_v4_do_rcv
      3.36 ± 10%      -0.6        2.77 ±  7%  perf-profile.children.cycles-pp.dccp_rcv_established
      4.47 ±  7%      -0.6        3.90 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.96 ±  9%      -0.4        0.60 ±  6%  perf-profile.children.cycles-pp.dccp_rcv_state_process
      0.44 ± 15%      -0.3        0.13 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.50 ± 13%      -0.3        0.20 ± 14%  perf-profile.children.cycles-pp.dccp_rcv_request_sent_state_process
      0.41 ± 18%      -0.2        0.16 ± 12%  perf-profile.children.cycles-pp.dccp_connect
      1.14 ±  7%      -0.2        0.91 ±  9%  perf-profile.children.cycles-pp.dccp_v4_rcv
      0.60 ± 13%      -0.2        0.44 ± 10%  perf-profile.children.cycles-pp.lock_sock_nested
      0.27 ± 12%      -0.2        0.11 ± 28%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.62 ±  8%      -0.2        0.46 ±  9%  perf-profile.children.cycles-pp.__inet_lookup_skb
      0.58 ± 10%      -0.2        0.42 ± 11%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.46 ± 11%      -0.2        0.31 ± 12%  perf-profile.children.cycles-pp.__slab_free
      0.68 ±  7%      -0.1        0.54 ± 10%  perf-profile.children.cycles-pp.__kfree_skb
      0.56 ± 10%      -0.1        0.42 ± 14%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.56 ± 10%      -0.1        0.43 ± 14%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.26 ± 25%      -0.1        0.14 ± 23%  perf-profile.children.cycles-pp.dccp_ackvec_clear_state
      0.16 ± 17%      -0.1        0.04 ± 45%  perf-profile.children.cycles-pp.dccp_finish_passive_close
      0.15 ± 22%      -0.1        0.05 ± 73%  perf-profile.children.cycles-pp.dccp_close
      0.16 ± 20%      -0.1        0.05 ± 74%  perf-profile.children.cycles-pp.inet_release
      0.18 ± 23%      -0.1        0.08 ± 22%  perf-profile.children.cycles-pp.dccp_check_seqno
      0.16 ± 22%      -0.1        0.06 ± 52%  perf-profile.children.cycles-pp.__sock_release
      0.16 ± 22%      -0.1        0.06 ± 52%  perf-profile.children.cycles-pp.sock_close
      0.22 ± 11%      -0.1        0.14 ± 21%  perf-profile.children.cycles-pp.dccp_enqueue_skb
      0.21 ± 15%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.dccp_v4_send_check
      0.25 ± 13%      -0.1        0.20 ±  9%  perf-profile.children.cycles-pp.dccp_feat_activate_values
      0.18 ± 15%      -0.0        0.14 ±  7%  perf-profile.children.cycles-pp.ccid_new
      0.17 ± 16%      -0.0        0.13 ± 15%  perf-profile.children.cycles-pp.__x64_sys_socket
      0.17 ± 16%      -0.0        0.13 ± 15%  perf-profile.children.cycles-pp.__sys_socket
      0.16 ± 14%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.ccid2_hc_tx_init
      0.08 ± 18%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.local_clock_noinstr
      0.08 ± 12%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.04 ± 71%      +0.0        0.08 ± 19%  perf-profile.children.cycles-pp.place_entity
      0.02 ±142%      +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.10 ± 17%      +0.1        0.15 ± 24%  perf-profile.children.cycles-pp.__fget_light
      0.02 ±141%      +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.04 ± 71%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.idle_cpu
      0.12 ± 14%      +0.1        0.18 ± 26%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.02 ±144%      +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.07 ± 53%      +0.1        0.13 ± 44%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.cpuacct_charge
      0.01 ±223%      +0.1        0.08 ± 25%  perf-profile.children.cycles-pp.pick_next_entity
      0.07 ± 51%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.05 ± 45%      +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.switch_fpu_return
      0.02 ±142%      +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.13 ± 30%      +0.1        0.21 ± 11%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.00            +0.1        0.08 ± 32%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.08 ± 21%      +0.1        0.18 ± 15%  perf-profile.children.cycles-pp.os_xsave
      0.00            +0.1        0.11 ± 22%  perf-profile.children.cycles-pp.cpu_util
      0.17 ± 21%      +0.1        0.28 ± 12%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.11 ± 19%      +0.1        0.22 ± 14%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.10 ± 37%      +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.available_idle_cpu
      0.09 ± 36%      +0.1        0.22 ± 15%  perf-profile.children.cycles-pp.wake_affine
      0.35 ± 10%      +0.1        0.48 ± 16%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.00            +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.check_preempt_curr
      0.17 ± 28%      +0.1        0.32 ± 18%  perf-profile.children.cycles-pp.update_rq_clock
      0.12 ± 18%      +0.2        0.28 ± 16%  perf-profile.children.cycles-pp.___perf_sw_event
      0.18 ± 26%      +0.2        0.34 ± 13%  perf-profile.children.cycles-pp.select_task_rq
      0.28 ± 20%      +0.2        0.44 ± 16%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.16 ±  9%      +0.2        0.35 ± 12%  perf-profile.children.cycles-pp.prepare_task_switch
      0.12 ± 24%      +0.2        0.32 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.66 ± 11%      +0.2        0.86 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.17 ± 38%      +0.2        0.37 ± 14%  perf-profile.children.cycles-pp.accept
      0.14 ± 39%      +0.2        0.35 ± 14%  perf-profile.children.cycles-pp.__x64_sys_accept
      0.14 ± 39%      +0.2        0.35 ± 14%  perf-profile.children.cycles-pp.__sys_accept4
      0.14 ± 43%      +0.2        0.35 ± 15%  perf-profile.children.cycles-pp.do_accept
      0.10 ± 64%      +0.2        0.31 ± 14%  perf-profile.children.cycles-pp.inet_csk_accept
      0.10 ± 61%      +0.2        0.32 ± 14%  perf-profile.children.cycles-pp.inet_accept
      0.17 ±  9%      +0.2        0.39 ± 10%  perf-profile.children.cycles-pp.__switch_to
      0.14 ± 17%      +0.2        0.38 ± 11%  perf-profile.children.cycles-pp.update_curr
      0.22 ± 22%      +0.2        0.46 ±  7%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.2        0.25 ±  5%  perf-profile.children.cycles-pp.put_prev_entity
      0.40 ± 12%      +0.3        0.66 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      0.11 ± 30%      +0.3        0.38 ± 10%  perf-profile.children.cycles-pp.set_next_entity
      0.18 ± 34%      +0.3        0.44 ± 23%  perf-profile.children.cycles-pp.kfree_skb_reason
      0.88 ±  8%      +0.3        1.16 ±  4%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.96 ±  9%      +0.3        1.26 ±  4%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.46 ± 13%      +0.3        0.76 ±  5%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.36 ±  5%      +0.3        0.70 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.66 ±  9%      +0.4        1.02 ± 18%  perf-profile.children.cycles-pp.__alloc_skb
      0.01 ±223%      +0.4        0.37 ± 15%  perf-profile.children.cycles-pp.intel_idle_irq
      0.78 ± 21%      +0.4        1.16 ± 10%  perf-profile.children.cycles-pp.__wake_up_common
      0.78 ± 22%      +0.4        1.17 ±  8%  perf-profile.children.cycles-pp.dccp_write_space
      0.94 ± 21%      +0.4        1.34 ±  7%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.49 ± 10%      +0.4        0.90 ± 24%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      0.91 ± 15%      +0.4        1.35 ±  8%  perf-profile.children.cycles-pp.wait_woken
      0.44 ±  7%      +0.5        0.90 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.65 ±  9%      +0.5        1.12 ± 23%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      0.31 ±  8%      +0.5        0.78 ±  4%  perf-profile.children.cycles-pp.update_load_avg
      0.45 ±  7%      +0.5        0.92 ±  4%  perf-profile.children.cycles-pp.activate_task
      0.75 ± 11%      +0.5        1.22 ±  6%  perf-profile.children.cycles-pp.schedule_timeout
      0.97 ± 19%      +0.5        1.44 ±  7%  perf-profile.children.cycles-pp.sock_wfree
      0.12 ± 12%      +0.5        0.64 ±  8%  perf-profile.children.cycles-pp.__netif_rx
      0.11 ± 16%      +0.5        0.64 ±  7%  perf-profile.children.cycles-pp.netif_rx_internal
      0.10 ± 22%      +0.5        0.63 ±  8%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.44 ± 13%      +0.5        0.99 ± 12%  perf-profile.children.cycles-pp.do_nanosleep
      0.13 ± 30%      +0.5        0.68 ±  7%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.48 ± 12%      +0.6        1.03 ± 12%  perf-profile.children.cycles-pp.hrtimer_nanosleep
      0.48 ± 12%      +0.6        1.05 ± 12%  perf-profile.children.cycles-pp.common_nsleep
      0.19 ± 34%      +0.6        0.77 ±  8%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.52 ± 10%      +0.6        1.10 ± 12%  perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep
      0.27 ± 24%      +0.6        0.85 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.49 ±  7%      +0.6        1.09 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.20 ± 34%      +0.6        0.80 ±  7%  perf-profile.children.cycles-pp.find_busiest_group
      0.69 ± 12%      +0.6        1.33 ± 11%  perf-profile.children.cycles-pp.clock_nanosleep
      0.37 ± 18%      +0.7        1.02 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.28 ± 34%      +0.7        0.94 ±  7%  perf-profile.children.cycles-pp.load_balance
      0.44 ± 19%      +0.7        1.12 ±  5%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      1.08 ± 18%      +0.8        1.91 ±  3%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.05 ± 17%      +0.8        1.88 ±  3%  perf-profile.children.cycles-pp.loopback_xmit
      0.03 ±101%      +0.8        0.87 ±  6%  perf-profile.children.cycles-pp.newidle_balance
      0.81 ± 18%      +1.0        1.77 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      4.47 ± 13%      +1.0        5.46 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      4.16 ± 12%      +1.0        5.16 ±  6%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      4.13 ± 12%      +1.0        5.14 ±  6%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      2.88 ± 16%      +1.3        4.15 ±  6%  perf-profile.children.cycles-pp.__sk_receive_skb
      0.23 ± 15%      +1.5        1.74 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.69 ± 10%      +2.0        3.69 ±  4%  perf-profile.children.cycles-pp.poll_idle
      1.00 ±  5%      +2.6        3.58 ±  2%  perf-profile.children.cycles-pp.schedule
      1.49 ±  8%      +2.6        4.12        perf-profile.children.cycles-pp.__schedule
     40.12 ±  6%      +3.5       43.66 ±  4%  perf-profile.children.cycles-pp.do_idle
     40.13 ±  6%      +3.6       43.69 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     40.13 ±  6%      +3.6       43.69 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.00            +5.7        5.67 ±  5%  perf-profile.children.cycles-pp.napi_threaded_poll_loop
      0.08 ± 58%      +6.8        6.87 ±  5%  perf-profile.children.cycles-pp.ret_from_fork_asm
      0.08 ± 58%      +6.8        6.87 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
      0.07 ± 58%      +6.8        6.87 ±  5%  perf-profile.children.cycles-pp.kthread
      0.00            +6.8        6.80 ±  5%  perf-profile.children.cycles-pp.smpboot_thread_fn
     21.29 ±  8%      -3.6       17.73 ±  2%  perf-profile.self.cycles-pp.__inet_check_established
      7.13 ±  7%      -1.1        5.99 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      4.16 ±  7%      -0.7        3.48 ±  3%  perf-profile.self.cycles-pp.__inet_hash_connect
      4.42 ±  7%      -0.6        3.85 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.44 ± 15%      -0.3        0.12 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.55 ± 10%      -0.2        0.38 ± 13%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.28 ± 17%      -0.2        0.12 ± 26%  perf-profile.self.cycles-pp.__sk_receive_skb
      0.46 ± 11%      -0.1        0.31 ± 12%  perf-profile.self.cycles-pp.__slab_free
      0.18 ± 23%      -0.1        0.08 ± 22%  perf-profile.self.cycles-pp.dccp_check_seqno
      0.21 ± 12%      -0.1        0.11 ± 19%  perf-profile.self.cycles-pp.dccp_transmit_skb
      0.22 ± 11%      -0.1        0.14 ± 20%  perf-profile.self.cycles-pp.dccp_enqueue_skb
      0.09 ± 31%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.dccp_ackvec_parsed_add
      0.18 ± 15%      -0.1        0.12 ± 21%  perf-profile.self.cycles-pp.ccid2_hc_tx_packet_recv
      0.07 ±  9%      +0.0        0.12 ± 20%  perf-profile.self.cycles-pp.do_syscall_64
      0.04 ± 71%      +0.0        0.08 ± 20%  perf-profile.self.cycles-pp.place_entity
      0.12 ± 10%      +0.0        0.17 ± 16%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.newidle_balance
      0.06 ± 52%      +0.1        0.12 ± 19%  perf-profile.self.cycles-pp.update_rq_clock
      0.09 ± 22%      +0.1        0.15 ± 24%  perf-profile.self.cycles-pp.__fget_light
      0.05 ± 45%      +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.switch_fpu_return
      0.04 ±101%      +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.02 ±146%      +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.02 ±141%      +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.prepare_task_switch
      0.02 ±142%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.02 ±144%      +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.12 ± 22%      +0.1        0.19 ± 12%  perf-profile.self.cycles-pp.dccp_sendmsg
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.cpuacct_charge
      0.07 ± 51%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.00            +0.1        0.07 ± 21%  perf-profile.self.cycles-pp.pick_next_entity
      0.10 ± 48%      +0.1        0.18 ± 23%  perf-profile.self.cycles-pp.__entry_text_start
      0.27 ± 14%      +0.1        0.35 ± 10%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.12 ± 31%      +0.1        0.20 ± 11%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.03 ±100%      +0.1        0.11 ± 18%  perf-profile.self.cycles-pp.schedule
      0.00            +0.1        0.08 ± 32%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.01 ±223%      +0.1        0.10 ± 14%  perf-profile.self.cycles-pp.idle_cpu
      0.08 ± 23%      +0.1        0.17 ± 14%  perf-profile.self.cycles-pp.os_xsave
      0.01 ±223%      +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.set_next_entity
      0.04 ±101%      +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.09 ± 32%      +0.1        0.19 ± 19%  perf-profile.self.cycles-pp.enqueue_entity
      0.08 ± 28%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.04 ± 45%      +0.1        0.15 ± 15%  perf-profile.self.cycles-pp.update_curr
      0.12 ± 31%      +0.1        0.23 ± 19%  perf-profile.self.cycles-pp.__send
      0.00            +0.1        0.11 ± 22%  perf-profile.self.cycles-pp.cpu_util
      0.12 ± 22%      +0.1        0.22 ± 11%  perf-profile.self.cycles-pp.dccp_write_space
      0.13 ± 19%      +0.1        0.24 ± 23%  perf-profile.self.cycles-pp.__alloc_skb
      0.11 ± 20%      +0.1        0.22 ± 14%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.03 ±100%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.smpboot_thread_fn
      0.10 ± 37%      +0.1        0.21 ± 13%  perf-profile.self.cycles-pp.available_idle_cpu
      0.21 ± 22%      +0.1        0.34 ± 20%  perf-profile.self.cycles-pp.wait_woken
      0.10 ± 13%      +0.1        0.24 ± 18%  perf-profile.self.cycles-pp.___perf_sw_event
      0.10 ± 10%      +0.2        0.26 ± 10%  perf-profile.self.cycles-pp.update_load_avg
      0.12 ± 27%      +0.2        0.32 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.17 ± 11%      +0.2        0.38 ± 10%  perf-profile.self.cycles-pp.__switch_to
      0.18 ± 20%      +0.2        0.40 ± 11%  perf-profile.self.cycles-pp.__schedule
      0.22 ± 22%      +0.2        0.46 ±  7%  perf-profile.self.cycles-pp.__switch_to_asm
      0.01 ±223%      +0.3        0.34 ± 15%  perf-profile.self.cycles-pp.intel_idle_irq
      0.08 ± 48%      +0.4        0.45 ± 12%  perf-profile.self.cycles-pp.update_sg_lb_stats
      1.64 ± 11%      +2.0        3.64 ±  4%  perf-profile.self.cycles-pp.poll_idle





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-08-18 09:21:11 [-0700], Jakub Kicinski wrote:
> As tempting as code removal would be, we can still try to explore the
> option of letting backlog processing run in threads - as an opt-in on
> normal kernels and force it on RT?
> 
> But it would be good to wait ~2 weeks before moving forward, if you
> don't mind, various core folks keep taking vacations..

No problem.  Let me repost it then in two weeks as optional and not
mandatory.

Sebastian

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Paolo Abeni]

On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote:
> @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
>  		 * We can use non atomic operation since we own the queue lock
>  		 */
>  		if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state))
> -			napi_schedule_rps(sd);
> +			__napi_schedule_irqoff(&sd->backlog);
>  		goto enqueue;
>  	}
>  	reason = SKB_DROP_REASON_CPU_BACKLOG;

I *think* that the above could be quite dangerous when cpu ==
smp_processor_id() - that is, with plain veth usage.

Currently, each packet runs into the rx path just after
enqueue_to_backlog()/tx completes.

With this patch there will be a burst effect, where the backlog thread
will run after a few (several) packets will be enqueued, when the
process scheduler will decide - note that the current CPU is already
hosting a running process, the tx thread.

The above can cause packet drops (due to limited buffering) or very
high latency (due to long burst), even in non overload situation, quite
hard to debug.

I think the above needs to be an opt-in, but I guess that even RT
deployments doing some packet forwarding will not be happy with this
on.

Cheers,

Paolo

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote:
> On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote:
> > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
> >  		 * We can use non atomic operation since we own the queue lock
> >  		 */
> >  		if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state))
> > -			napi_schedule_rps(sd);
> > +			__napi_schedule_irqoff(&sd->backlog);
> >  		goto enqueue;
> >  	}
> >  	reason = SKB_DROP_REASON_CPU_BACKLOG;
> 
> I *think* that the above could be quite dangerous when cpu ==
> smp_processor_id() - that is, with plain veth usage.
> 
> Currently, each packet runs into the rx path just after
> enqueue_to_backlog()/tx completes.
> 
> With this patch there will be a burst effect, where the backlog thread
> will run after a few (several) packets will be enqueued, when the
> process scheduler will decide - note that the current CPU is already
> hosting a running process, the tx thread.
> 
> The above can cause packet drops (due to limited buffering) or very
> high latency (due to long burst), even in non overload situation, quite
> hard to debug.
> 
> I think the above needs to be an opt-in, but I guess that even RT
> deployments doing some packet forwarding will not be happy with this
> on.

I've been looking at this again and have been thinking what you said
here. I think part of the problem is that we lack a policy/ mechanism
when a DoS is happening and what to do.

Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its
job"") when a lot of network packets are processed then processing is
moved to ksoftirqd and continues based on how the scheduler schedules
the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system and
it can do something else in between. Any interrupt will not continue the
outstanding softirq backlog but wait for ksoftirqd. So it basically
avoids the networking overload. It throttles the throughput if needed.

This isn't the case after that commit. Now, the CPU can be stuck with
processing networking packets if the packets come in fast enough. Even
if ksoftirqd is woken up, the next interrupt (say the timer) will
continue with at least one round.
By using NAPI-threads it is possible to give the control back to the
scheduler which can throttle the NAPI processing in favour of other
threads that ask for CPU. As you pointed out, waking the thread does not
guarantee that it will immediately do the NAPI work. It can be delayed
based on current load on the system.

This could be influenced by assigning the NAPI-thread a SCHED_FIFO
priority. Based on the priority it could be ensured that the thread
starts right away or "later" if something else is more important.
However, this opens the DoS window again: The scheduler will put the
NAPI thread on CPU as long as it asks for it with no throttling.

If we could somehow define a DoS condition once we are overwhelmed with
packets, then we could act on it and throttle it. This in turn would
allow a SCHED_FIFO priority without the fear of a lockup if the system
is flooded with packets.

> Cheers,
> 
> Paolo

Sebastian

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Ferenc Fejes]

Hi!

On Wed, 2023-09-20 at 17:57 +0200, Sebastian Andrzej Siewior wrote:
> On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote:
> > On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote:
> > > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct
> > > sk_buff *skb, int cpu,
> > >  		 * We can use non atomic operation since we own
> > > the queue lock
> > >  		 */
> > >  		if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd-
> > > >backlog.state))
> > > -			napi_schedule_rps(sd);
> > > +			__napi_schedule_irqoff(&sd->backlog);
> > >  		goto enqueue;
> > >  	}
> > >  	reason = SKB_DROP_REASON_CPU_BACKLOG;
> > 
> > I *think* that the above could be quite dangerous when cpu ==
> > smp_processor_id() - that is, with plain veth usage.
> > 
> > Currently, each packet runs into the rx path just after
> > enqueue_to_backlog()/tx completes.
> > 
> > With this patch there will be a burst effect, where the backlog
> > thread
> > will run after a few (several) packets will be enqueued, when the
> > process scheduler will decide - note that the current CPU is
> > already
> > hosting a running process, the tx thread.
> > 
> > The above can cause packet drops (due to limited buffering) or very
> > high latency (due to long burst), even in non overload situation,
> > quite
> > hard to debug.
> > 
> > I think the above needs to be an opt-in, but I guess that even RT
> > deployments doing some packet forwarding will not be happy with
> > this
> > on.
> 
> I've been looking at this again and have been thinking what you said
> here. I think part of the problem is that we lack a policy/ mechanism
> when a DoS is happening and what to do.
> 
> Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its
> job"") when a lot of network packets are processed then processing is
> moved to ksoftirqd and continues based on how the scheduler schedules
> the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system
> and
> it can do something else in between. Any interrupt will not continue
> the
> outstanding softirq backlog but wait for ksoftirqd. So it basically
> avoids the networking overload. It throttles the throughput if
> needed.
> 
> This isn't the case after that commit. Now, the CPU can be stuck with
> processing networking packets if the packets come in fast enough.
> Even
> if ksoftirqd is woken up, the next interrupt (say the timer) will
> continue with at least one round.
> By using NAPI-threads it is possible to give the control back to the
> scheduler which can throttle the NAPI processing in favour of other
> threads that ask for CPU. As you pointed out, waking the thread does
> not
> guarantee that it will immediately do the NAPI work. It can be
> delayed
> based on current load on the system.
> 
> This could be influenced by assigning the NAPI-thread a SCHED_FIFO
> priority. Based on the priority it could be ensured that the thread
> starts right away or "later" if something else is more important.
> However, this opens the DoS window again: The scheduler will put the
> NAPI thread on CPU as long as it asks for it with no throttling.
> 
> If we could somehow define a DoS condition once we are overwhelmed
> with
> packets, then we could act on it and throttle it. This in turn would
> allow a SCHED_FIFO priority without the fear of a lockup if the
> system
> is flooded with packets.

Can this be avoided if we reuse gro_flush_timeout as the maximum time
the NAPI thread can be scheduled?

> 
> > Cheers,
> > 
> > Paolo
> 
> Sebastian

Ferenc

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Sebastian Andrzej Siewior]

On 2023-09-21 12:41:33 [+0200], Ferenc Fejes wrote:
> Hi!
Hi,

> > If we could somehow define a DoS condition once we are overwhelmed
> > with
> > packets, then we could act on it and throttle it. This in turn would
> > allow a SCHED_FIFO priority without the fear of a lockup if the
> > system
> > is flooded with packets.
> 
> Can this be avoided if we reuse gro_flush_timeout as the maximum time
> the NAPI thread can be scheduled?

First your run time needs to be accounted somehow. I observed that some
cards/ systems tend pull often a few packets on each interrupt and
others pull more packets at a time.
So probably packets in a time frame would make sense. Maybe even plus
packet size assuming larger packets require more processing time.

If you run at SCHED_OTHER you don't care, you can keep it running. With
SCHED_FIFO you would need to decide:
- how much is too much
- what to do once you reach too much

Once you reach too much you could:
- change the scheduling policy to SCHED_OTHER and keep going until it is
  no longer "too much in a given period" so you can flip it back.

- stop processing for a period of time and risk packet loss which is
  defined as better than to continue.

- pulling packets and dropping them instead of injecting into the stack.
  Using xdp/ebpf might be easy since there is an API for that. One could
  even peek at packets to decide if some can be kept.
  This would rely on the fact that the system can do this quick enough
  under a DoS condition.

> 
> Ferenc

Sebastian

Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.

[Paolo Abeni]

On Wed, 2023-09-20 at 17:57 +0200, Sebastian Andrzej Siewior wrote:
> On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote:
> > On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote:
> > > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
> > >  		 * We can use non atomic operation since we own the queue lock
> > >  		 */
> > >  		if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state))
> > > -			napi_schedule_rps(sd);
> > > +			__napi_schedule_irqoff(&sd->backlog);
> > >  		goto enqueue;
> > >  	}
> > >  	reason = SKB_DROP_REASON_CPU_BACKLOG;
> > 
> > I *think* that the above could be quite dangerous when cpu ==
> > smp_processor_id() - that is, with plain veth usage.
> > 
> > Currently, each packet runs into the rx path just after
> > enqueue_to_backlog()/tx completes.
> > 
> > With this patch there will be a burst effect, where the backlog thread
> > will run after a few (several) packets will be enqueued, when the
> > process scheduler will decide - note that the current CPU is already
> > hosting a running process, the tx thread.
> > 
> > The above can cause packet drops (due to limited buffering) or very
> > high latency (due to long burst), even in non overload situation, quite
> > hard to debug.
> > 
> > I think the above needs to be an opt-in, but I guess that even RT
> > deployments doing some packet forwarding will not be happy with this
> > on.
> 
> I've been looking at this again and have been thinking what you said
> here. I think part of the problem is that we lack a policy/ mechanism
> when a DoS is happening and what to do.
> 
> Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its
> job"") when a lot of network packets are processed then processing is
> moved to ksoftirqd and continues based on how the scheduler schedules
> the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system and
> it can do something else in between. Any interrupt will not continue the
> outstanding softirq backlog but wait for ksoftirqd. So it basically
> avoids the networking overload. It throttles the throughput if needed.
> 
> This isn't the case after that commit. Now, the CPU can be stuck with
> processing networking packets if the packets come in fast enough. Even
> if ksoftirqd is woken up, the next interrupt (say the timer) will
> continue with at least one round.
> By using NAPI-threads it is possible to give the control back to the
> scheduler which can throttle the NAPI processing in favour of other
> threads that ask for CPU. As you pointed out, waking the thread does not
> guarantee that it will immediately do the NAPI work. It can be delayed
> based on current load on the system.
> 
> This could be influenced by assigning the NAPI-thread a SCHED_FIFO
> priority. Based on the priority it could be ensured that the thread
> starts right away or "later" if something else is more important.
> However, this opens the DoS window again: The scheduler will put the
> NAPI thread on CPU as long as it asks for it with no throttling.
> 
> If we could somehow define a DoS condition once we are overwhelmed with
> packets, then we could act on it and throttle it. This in turn would
> allow a SCHED_FIFO priority without the fear of a lockup if the system
> is flooded with packets.

I declare ENOCOFFEE before starting, be warned! 

I fear this is becoming a bit too theoretical, but we can infer a DoS
condition if the napi thread enqueues somewhere (socket buffer, qdisc,
tx ring, ???) a packet and the queue utilization is "high" (say > 75%
of max).

I have no idea how to throttle a FIFO thread retaining its priority.

More importantly, this kind of configuration is not really viable for a
generic !PREEMPT_RT build, while the concern I have with napi threaded
backlog/serving the backlog with ksoftirqd applies there.

Cheers,

Paolo