From: Lin Ming <ming.m.lin@intel.com>
To: Mike Galbraith <efault@gmx.de>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@elte.hu>
Cc: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>,
lkml <linux-kernel@vger.kernel.org>
Subject: netperf ~50% regression with 2.6.33-rc1, bisect to 1b9508f
Date: Mon, 25 Jan 2010 18:03:46 +0800 [thread overview]
Message-ID: <1264413826.3642.88.camel@minggr.sh.intel.com> (raw)
Hi,
The netperf lookback regression comes back again.
UDP stream test has ~50% regression with 2.6.33-rc1 compared to 2.6.32.
Testing machine: Nehalem, 2 sockets, 4 cores, hyper thread, 4G mem
Server and client are bind to different physical cpu.
taskset -c 15 ./netserver
taskset -c 0 ./netperf -t UDP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5
-- -P 12384,12888 -s 32768 -S 32768 -m 1024
Bisect to below commit,
commit 1b9508f6831e10d53256825de8904caa22d1ca2c
Author: Mike Galbraith <efault@gmx.de>
Date: Wed Nov 4 17:53:50 2009 +0100
sched: Rate-limit newidle
Rate limit newidle to migration_cost. It's a win for all
stages of sysbench oltp tests.
Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Interesting, this commit was originally fixed the similar UDP stream
test regression on a tulsa machine in 2.6.32-rc1, see below threads
http://marc.info/?t=125722014400001&r=1&w=2
But now it introduces a regression on a Nehalem machine.
This regression seems caused by a lot of rescheduling IPI.
Perf top data as below, note "default_send_IPI_mask_sequence_phys"
samples pcnt function DSO
_______ _____ ___________________________________ _________________
1407.00 6.8% _spin_lock_bh [kernel.kallsyms]
1330.00 6.4% copy_user_generic_string [kernel.kallsyms]
1017.00 4.9% _spin_lock_irq [kernel.kallsyms]
968.00 4.7% default_send_IPI_mask_sequence_phys [kernel.kallsyms]
891.00 4.3% acpi_os_read_port [kernel.kallsyms]
818.00 4.0% sock_alloc_send_pskb [kernel.kallsyms]
776.00 3.8% _spin_lock_irqsave [kernel.kallsyms]
757.00 3.7% __udp4_lib_lookup [kernel.kallsyms]
/proc/interrupts shows a lot of "Rescheduling interrupts" that are send
from CPU 0(client) to CPU 15(server).
With above commit, the idle balance was rate limited, so CPU 15(server,
waiting data from client) is idle at most time.
CPU0(client) executes as below,
try_to_wake_up
check_preempt_curr_idle
resched_task
smp_send_reschedule
This causes a lot of rescheduling IPI.
This commit can't be reverted due to conflict, so I just add below code
to disable "Rate-limit newidle" and the performance was recovered.
diff --git a/kernel/sched.c b/kernel/sched.c
index 18cceee..588fdef 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4421,9 +4421,6 @@ static void idle_balance(int this_cpu, struct rq *this_rq)
this_rq->idle_stamp = this_rq->clock;
- if (this_rq->avg_idle < sysctl_sched_migration_cost)
- return;
-
for_each_domain(this_cpu, sd) {
unsigned long interval;
Lin Ming
next reply other threads:[~2010-01-25 10:19 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-01-25 10:03 Lin Ming [this message]
2010-01-25 11:35 ` netperf ~50% regression with 2.6.33-rc1, bisect to 1b9508f Mike Galbraith
2010-01-25 11:45 ` Mike Galbraith
2010-01-26 9:03 ` Lin Ming
2010-01-25 14:03 ` Peter Zijlstra
2010-01-25 14:19 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1264413826.3642.88.camel@minggr.sh.intel.com \
--to=ming.m.lin@intel.com \
--cc=efault@gmx.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=peterz@infradead.org \
--cc=yanmin_zhang@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.