* [RFC] optimize the locking in the rebalance_domains()
@ 2012-07-22 16:23 Vlad Zolotarov
2012-07-22 16:33 ` Vlad Zolotarov
2012-07-23 2:25 ` Namhyung Kim
0 siblings, 2 replies; 4+ messages in thread
From: Vlad Zolotarov @ 2012-07-22 16:23 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel, Shai Fultheim (Shai@ScaleMP.com)
Ingo, we've noticed that rebalance_domains() will try to take a lock
every time it's called (every jiffy) if SD_SERIALIZE is set (which is a
default configuration). This is done regardless the fact that maybe
there hasn't passed enough time since the last rebalancing in which case
there is no need to take a lock the first place.
The above creates a heavy false sharing problem on the "balancing"
spin-lock on large SMP systems: try_lock() is implemented with an
(atomic) xchng instruction which invalidates the cache line "balancing"
belongs to and therefore creates an intensive cross-NUMA-nodes traffic.
The below patch will minimize the above phenomena to the time slots it's
really needed, namely when the "interval" has really passed.
Pls., comment.
thanks,
vlad
---
kernel/sched/fair.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c099cc6..6777d38 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
interval = msecs_to_jiffies(interval);
interval = clamp(interval, 1UL, max_load_balance_interval);
+ if (!time_after_eq(jiffies, sd->last_balance + interval))
+ goto out;
+
need_serialize = sd->flags & SD_SERIALIZE;
if (need_serialize) {
@@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
goto out;
}
- if (time_after_eq(jiffies, sd->last_balance + interval)) {
- if (load_balance(cpu, rq, sd, idle, &balance)) {
- /*
- * We've pulled tasks over so either we're no
- * longer idle.
- */
- idle = CPU_NOT_IDLE;
- }
- sd->last_balance = jiffies;
+ if (load_balance(cpu, rq, sd, idle, &balance)) {
+ /*
+ * We've pulled tasks over so either we're no
+ * longer idle.
+ */
+ idle = CPU_NOT_IDLE;
}
+ sd->last_balance = jiffies;
+
if (need_serialize)
spin_unlock(&balancing);
out:
--
1.7.9.5
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [RFC] optimize the locking in the rebalance_domains()
2012-07-22 16:23 [RFC] optimize the locking in the rebalance_domains() Vlad Zolotarov
@ 2012-07-22 16:33 ` Vlad Zolotarov
2012-07-23 2:25 ` Namhyung Kim
1 sibling, 0 replies; 4+ messages in thread
From: Vlad Zolotarov @ 2012-07-22 16:33 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel, Shai Fultheim (Shai@ScaleMP.com)
On Sun, 2012-07-22 at 19:23 +0300, Vlad Zolotarov wrote:
> Ingo, we've noticed that rebalance_domains() will try to take a lock
> every time it's called (every jiffy)
This is of course when NOHZ is off... ;)
> if SD_SERIALIZE is set (which is a
> default configuration). This is done regardless the fact that maybe
> there hasn't passed enough time since the last rebalancing in which case
> there is no need to take a lock the first place.
>
> The above creates a heavy false sharing problem on the "balancing"
> spin-lock on large SMP systems: try_lock() is implemented with an
> (atomic) xchng instruction which invalidates the cache line "balancing"
> belongs to and therefore creates an intensive cross-NUMA-nodes traffic.
>
> The below patch will minimize the above phenomena to the time slots it's
> really needed, namely when the "interval" has really passed.
>
> Pls., comment.
> thanks,
> vlad
>
> ---
> kernel/sched/fair.c | 20 +++++++++++---------
> 1 file changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index c099cc6..6777d38 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> interval = msecs_to_jiffies(interval);
> interval = clamp(interval, 1UL, max_load_balance_interval);
>
> + if (!time_after_eq(jiffies, sd->last_balance + interval))
> + goto out;
> +
> need_serialize = sd->flags & SD_SERIALIZE;
>
> if (need_serialize) {
> @@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> goto out;
> }
>
> - if (time_after_eq(jiffies, sd->last_balance + interval)) {
> - if (load_balance(cpu, rq, sd, idle, &balance)) {
> - /*
> - * We've pulled tasks over so either we're no
> - * longer idle.
> - */
> - idle = CPU_NOT_IDLE;
> - }
> - sd->last_balance = jiffies;
> + if (load_balance(cpu, rq, sd, idle, &balance)) {
> + /*
> + * We've pulled tasks over so either we're no
> + * longer idle.
> + */
> + idle = CPU_NOT_IDLE;
> }
> + sd->last_balance = jiffies;
> +
> if (need_serialize)
> spin_unlock(&balancing);
> out:
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC] optimize the locking in the rebalance_domains()
2012-07-22 16:23 [RFC] optimize the locking in the rebalance_domains() Vlad Zolotarov
2012-07-22 16:33 ` Vlad Zolotarov
@ 2012-07-23 2:25 ` Namhyung Kim
2012-07-23 7:50 ` Vlad Zolotarov
1 sibling, 1 reply; 4+ messages in thread
From: Namhyung Kim @ 2012-07-23 2:25 UTC (permalink / raw)
To: Vlad Zolotarov
Cc: Ingo Molnar, linux-kernel, Shai Fultheim (Shai@ScaleMP.com)
Hi, Vlad
On Sun, 22 Jul 2012 19:23:55 +0300, Vlad Zolotarov wrote:
> Ingo, we've noticed that rebalance_domains() will try to take a lock
> every time it's called (every jiffy) if SD_SERIALIZE is set (which is a
> default configuration). This is done regardless the fact that maybe
> there hasn't passed enough time since the last rebalancing in which case
> there is no need to take a lock the first place.
>
> The above creates a heavy false sharing problem on the "balancing"
> spin-lock on large SMP systems: try_lock() is implemented with an
> (atomic) xchng instruction which invalidates the cache line "balancing"
> belongs to and therefore creates an intensive cross-NUMA-nodes traffic.
>
> The below patch will minimize the above phenomena to the time slots it's
> really needed, namely when the "interval" has really passed.
>
> Pls., comment.
>
> thanks,
> vlad
>
> ---
> kernel/sched/fair.c | 20 +++++++++++---------
> 1 file changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index c099cc6..6777d38 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> interval = msecs_to_jiffies(interval);
> interval = clamp(interval, 1UL, max_load_balance_interval);
>
> + if (!time_after_eq(jiffies, sd->last_balance + interval))
> + goto out;
> +
First line looks like white-space-damaged.
Anyway, wouldn't it be better using time_before() here?
Thanks,
Namhyung
> need_serialize = sd->flags & SD_SERIALIZE;
>
> if (need_serialize) {
> @@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> goto out;
> }
>
> - if (time_after_eq(jiffies, sd->last_balance + interval)) {
> - if (load_balance(cpu, rq, sd, idle, &balance)) {
> - /*
> - * We've pulled tasks over so either we're no
> - * longer idle.
> - */
> - idle = CPU_NOT_IDLE;
> - }
> - sd->last_balance = jiffies;
> + if (load_balance(cpu, rq, sd, idle, &balance)) {
> + /*
> + * We've pulled tasks over so either we're no
> + * longer idle.
> + */
> + idle = CPU_NOT_IDLE;
> }
> + sd->last_balance = jiffies;
> +
> if (need_serialize)
> spin_unlock(&balancing);
> out:
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC] optimize the locking in the rebalance_domains()
2012-07-23 2:25 ` Namhyung Kim
@ 2012-07-23 7:50 ` Vlad Zolotarov
0 siblings, 0 replies; 4+ messages in thread
From: Vlad Zolotarov @ 2012-07-23 7:50 UTC (permalink / raw)
To: Namhyung Kim; +Cc: Ingo Molnar, linux-kernel, Shai Fultheim (Shai@ScaleMP.com)
On Mon, 2012-07-23 at 11:25 +0900, Namhyung Kim wrote:
> Hi, Vlad
>
> On Sun, 22 Jul 2012 19:23:55 +0300, Vlad Zolotarov wrote:
> > Ingo, we've noticed that rebalance_domains() will try to take a lock
> > every time it's called (every jiffy) if SD_SERIALIZE is set (which is a
> > default configuration). This is done regardless the fact that maybe
> > there hasn't passed enough time since the last rebalancing in which case
> > there is no need to take a lock the first place.
> >
> > The above creates a heavy false sharing problem on the "balancing"
> > spin-lock on large SMP systems: try_lock() is implemented with an
> > (atomic) xchng instruction which invalidates the cache line "balancing"
> > belongs to and therefore creates an intensive cross-NUMA-nodes traffic.
> >
> > The below patch will minimize the above phenomena to the time slots it's
> > really needed, namely when the "interval" has really passed.
> >
> > Pls., comment.
> >
> > thanks,
> > vlad
> >
> > ---
> > kernel/sched/fair.c | 20 +++++++++++---------
> > 1 file changed, 11 insertions(+), 9 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index c099cc6..6777d38 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> > interval = msecs_to_jiffies(interval);
> > interval = clamp(interval, 1UL, max_load_balance_interval);
> >
> > + if (!time_after_eq(jiffies, sd->last_balance + interval))
> > + goto out;
> > +
>
> First line looks like white-space-damaged.
Looks like it. ;) Thanks for catching. I'll surely fix it if we get to
posting the patch for applying.
> Anyway, wouldn't it be better using time_before() here?
Sure. I'll fix it as well. However I'd like to hear what u and other
people on the mailing list think about the idea in general.
Thanks,
vlad
>
> Thanks,
> Namhyung
>
>
> > need_serialize = sd->flags & SD_SERIALIZE;
> >
> > if (need_serialize) {
> > @@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
> > goto out;
> > }
> >
> > - if (time_after_eq(jiffies, sd->last_balance + interval)) {
> > - if (load_balance(cpu, rq, sd, idle, &balance)) {
> > - /*
> > - * We've pulled tasks over so either we're no
> > - * longer idle.
> > - */
> > - idle = CPU_NOT_IDLE;
> > - }
> > - sd->last_balance = jiffies;
> > + if (load_balance(cpu, rq, sd, idle, &balance)) {
> > + /*
> > + * We've pulled tasks over so either we're no
> > + * longer idle.
> > + */
> > + idle = CPU_NOT_IDLE;
> > }
> > + sd->last_balance = jiffies;
> > +
> > if (need_serialize)
> > spin_unlock(&balancing);
> > out:
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-07-23 7:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-07-22 16:23 [RFC] optimize the locking in the rebalance_domains() Vlad Zolotarov
2012-07-22 16:33 ` Vlad Zolotarov
2012-07-23 2:25 ` Namhyung Kim
2012-07-23 7:50 ` Vlad Zolotarov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).