[PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[Dimitri Sivanich]

The isolcpus option is broken in 2.6.10-rc2-bk2.  The domains are no longer
being properly initialized (which results in a panic at bootup).

The following patch fixes this.

Signed-off-by: Dimitri Sivanich <sivanich@sgi.com>

Index: linux/arch/ia64/kernel/domain.c
===================================================================
--- linux.orig/arch/ia64/kernel/domain.c	2004-12-03 15:43:47.000000000 -0600
+++ linux/arch/ia64/kernel/domain.c	2004-12-06 12:28:42.788584850 -0600
@@ -220,6 +220,23 @@ void __devinit arch_init_sched_domains(v
 						&cpu_to_phys_group);
 	}
 
+	/* Initialize isolated CPU (physical) domains and groups */
+	for_each_cpu_mask(i, cpu_isolated_map) {
+		struct sched_domain *sd = NULL;
+		int group;
+
+		sd = &per_cpu(phys_domains, i);
+		group = cpu_to_phys_group(i);
+		*sd = SD_CPU_INIT;
+		cpu_set(i, sd->span);
+		sd->flags = 0;
+		sd->balance_interval = INT_MAX;
+		sd->groups = &sched_group_phys[group];
+		init_sched_build_groups(sched_group_phys, sd->span,
+						&cpu_to_phys_group);
+		sd->groups->cpu_power = SCHED_LOAD_SCALE;
+	}
+
 #ifdef CONFIG_NUMA
 	init_sched_build_groups(sched_group_allnodes, cpu_default_map,
 				&cpu_to_allnodes_group);
Index: linux/kernel/sched.c
===================================================================
--- linux.orig/kernel/sched.c	2004-12-03 15:44:08.000000000 -0600
+++ linux/kernel/sched.c	2004-12-06 12:28:35.359689609 -0600
@@ -4323,6 +4323,23 @@ static void __devinit arch_init_sched_do
 						&cpu_to_phys_group);
 	}
 
+        /* Initialize isolated CPU (physical) domains and groups */
+	for_each_cpu_mask(i, cpu_isolated_map) {
+		struct sched_domain *sd = NULL;
+		int group;
+
+		sd = &per_cpu(phys_domains, i);
+		group = cpu_to_phys_group(i);
+		*sd = SD_CPU_INIT;
+		cpu_set(i, sd->span);
+		sd->flags = 0;
+		sd->balance_interval = INT_MAX;
+		sd->groups = &sched_group_phys[group];
+		init_sched_build_groups(sched_group_phys, sd->span,
+						&cpu_to_phys_group);
+		sd->groups->cpu_power = SCHED_LOAD_SCALE;
+	}
+
 #ifdef CONFIG_NUMA
 	/* Set up node groups */
 	init_sched_build_groups(sched_group_nodes, cpu_default_map,

Re: [PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[Ingo Molnar]

* Dimitri Sivanich <sivanich@sgi.com> wrote:

> The isolcpus option is broken in 2.6.10-rc2-bk2.  The domains are no longer
> being properly initialized (which results in a panic at bootup).
> 
> The following patch fixes this.
> 
> Signed-off-by: Dimitri Sivanich <sivanich@sgi.com>

Acked-by: Ingo Molnar <mingo@elte.hu>

only a minor nit:

> +	/* Initialize isolated CPU (physical) domains and groups */
> +	for_each_cpu_mask(i, cpu_isolated_map) {
> +		struct sched_domain *sd = NULL;

there's no need to initialize 'sd' to NULL here.

> +		int group;
> +
> +		sd = &per_cpu(phys_domains, i);
> +		group = cpu_to_phys_group(i);

> +	for_each_cpu_mask(i, cpu_isolated_map) {
> +		struct sched_domain *sd = NULL;

ditto.

> +		int group;
> +
> +		sd = &per_cpu(phys_domains, i);
> +		group = cpu_to_phys_group(i);
> +		*sd = SD_CPU_INIT;

	Ingo

Re: [PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[Nick Piggin]

Dimitri Sivanich wrote:

>The isolcpus option is broken in 2.6.10-rc2-bk2.  The domains are no longer
>being properly initialized (which results in a panic at bootup).
>
>The following patch fixes this.
>
>Signed-off-by: Dimitri Sivanich <sivanich@sgi.com>
>

Sorry for not replying earlier, I've been away for a few days.

I don't think this is quite needed, because isolated CPUs should
have their domain set to sched_domain_dummy, I think?

The trick would be to just initialise sched_domain_dummy properly;
it looks like that isn't being done for some reason.

Give me a couple of hours and I'll try testing something that
should solve it.

Re: [PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]



Ingo Molnar wrote:

>* Dimitri Sivanich <sivanich@sgi.com> wrote:
>
>
>>The isolcpus option is broken in 2.6.10-rc2-bk2.  The domains are no longer
>>being properly initialized (which results in a panic at bootup).
>>
>>The following patch fixes this.
>>
>>Signed-off-by: Dimitri Sivanich <sivanich@sgi.com>
>>
>
>Acked-by: Ingo Molnar <mingo@elte.hu>
>
>only a minor nit:
>
>
>>+	/* Initialize isolated CPU (physical) domains and groups */
>>+	for_each_cpu_mask(i, cpu_isolated_map) {
>>+		struct sched_domain *sd = NULL;
>>
>
>there's no need to initialize 'sd' to NULL here.
>
>
>>+		int group;
>>+
>>+		sd = &per_cpu(phys_domains, i);
>>+		group = cpu_to_phys_group(i);
>>
>
>>+	for_each_cpu_mask(i, cpu_isolated_map) {
>>+		struct sched_domain *sd = NULL;
>>
>
>ditto.
>
>
>>+		int group;
>>+
>>+		sd = &per_cpu(phys_domains, i);
>>+		group = cpu_to_phys_group(i);
>>+		*sd = SD_CPU_INIT;
>>
>
>	Ingo
>
>
>

Actually, this is wrong. And I was wrong to say that the fix was
to initialize the dummy domain (because its ->flags are 0, there
is no need for it to have anything else set up).

The isolated domains don't need to be set up because they correctly
point to the dummy domain, which is already set up properly (zeroed).
The oops is just a problem with sched_domain_debug.

Andrew and/or Linus, please make sure Dimitri's patch gets reverted
and the attached one applied before 2.6.10.


[-- Attachment #2: sched-debug-oops.patch --]
[-- Type: text/plain, Size: 1431 bytes --]



Fix an oops in sched_domain_debug when using the isolcpus= option.

Also move a debug check for validating groups into the "for-each-group"
loop, where it should be.

Signed-off-by: Nick Piggin <nickpiggin@yahoo.com.au>


---

 linux-2.6-npiggin/kernel/sched.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff -puN kernel/sched.c~sched-debug-oops kernel/sched.c
--- linux-2.6/kernel/sched.c~sched-debug-oops	2004-12-08 19:51:48.000000000 +1100
+++ linux-2.6-npiggin/kernel/sched.c	2004-12-08 19:53:23.000000000 +1100
@@ -4446,6 +4446,7 @@ static void sched_domain_debug(void)
 				if (sd->parent)
 					printk(" ERROR !SD_LOAD_BALANCE domain has parent");
 				printk("\n");
+				break;
 			}
 
 			printk("span %s\n", str);
@@ -4454,8 +4455,6 @@ static void sched_domain_debug(void)
 				printk(KERN_DEBUG "ERROR domain->span does not contain CPU%d\n", i);
 			if (!cpu_isset(i, group->cpumask))
 				printk(KERN_DEBUG "ERROR domain->groups does not contain CPU%d\n", i);
-			if (!group->cpu_power)
-				printk(KERN_DEBUG "ERROR domain->cpu_power not set\n");
 
 			printk(KERN_DEBUG);
 			for (j = 0; j < level + 2; j++)
@@ -4466,6 +4465,9 @@ static void sched_domain_debug(void)
 					printk(" ERROR: NULL");
 					break;
 				}
+				
+				if (!group->cpu_power)
+					printk(KERN_DEBUG "ERROR group->cpu_power not set\n");
 
 				if (!cpus_weight(group->cpumask))
 					printk(" ERROR empty group:");

_

Re: [PATCH] isolcpus option broken in 2.6.10-rc2-bk2

[Ingo Molnar]

* Nick Piggin <piggin@cyberone.com.au> wrote:

> Actually, this is wrong. And I was wrong to say that the fix was to
> initialize the dummy domain (because its ->flags are 0, there is no
> need for it to have anything else set up).
> 
> The isolated domains don't need to be set up because they correctly
> point to the dummy domain, which is already set up properly (zeroed).
> The oops is just a problem with sched_domain_debug.

(ah ... indeed, good catch. Dimitri's patch ended up working around the
'break' bug in sched_domain_debug() :-| At least this case shows that it
was right to keep SCHED_DOMAIN_DEBUG defined.)

> Andrew and/or Linus, please make sure Dimitri's patch gets reverted
> and the attached one applied before 2.6.10.

Acked-by: Ingo Molnar <mingo@elte.hu>

	Ingo