HT apparently not detected properly on 2.4.23

HT apparently not detected properly on 2.4.23

[Ethan Weinstein]

Hi,

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 
xeons@2.4ghz and hypertherading enabled shows 4 cpu's in 
/proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4

Also, I believe this has been reported before, but the system only seems 
to interrupt on CPU0 with either kernel, unless I apply a patch I found 
here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do.  This 
problem _seems_ to be isolated to the Supermicro, as HT does seem 
properly detected on several other SMP systems I have at work (compaq 
ML370/ML380) with 2.4.23

-Ethan

Re: HT apparently not detected properly on 2.4.23

[Andre Tomt]

On Wed, 2003-12-03 at 19:46, Ethan Weinstein wrote:
> Hi,
> 
> With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 
> xeons@2.4ghz and hypertherading enabled shows 4 cpu's in 
> /proc/cpuinfo|proc/interrupts, with:
> CONFIG_ACPI=y
> CONFIG_ACPI_HT_ONLY=y
> The same config with 2.4.23 only shows 2 cpus, even with:
> CONFIG_NR_CPUS=4

This may be the known problem about CONFIG_NR_CPU's not working properly
in all cases. Try up'ing it to 32.

Re: HT apparently not detected properly on 2.4.23

[Mike Fedyk]

On Wed, Dec 03, 2003 at 08:40:51PM +0100, Andre Tomt wrote:
> On Wed, 2003-12-03 at 19:46, Ethan Weinstein wrote:
> > Hi,
> > 
> > With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 
> > xeons@2.4ghz and hypertherading enabled shows 4 cpu's in 
> > /proc/cpuinfo|proc/interrupts, with:
> > CONFIG_ACPI=y
> > CONFIG_ACPI_HT_ONLY=y
> > The same config with 2.4.23 only shows 2 cpus, even with:
> > CONFIG_NR_CPUS=4
> 
> This may be the known problem about CONFIG_NR_CPU's not working properly
> in all cases. Try up'ing it to 32.

Depending on the logical addressing of your processors, CONFIG_NR_CPUS=8 may
work in this case too.

Some processors/motherboards logically address the CPUs other that 0-3 if
there are 4 processors.

Currently CONFIG_NR_CPUS needs to be big enough to hold the largest logical
processor number.

Re: HT apparently not detected properly on 2.4.23

[Ethan Weinstein]

Mike Fedyk wrote:

> Depending on the logical addressing of your processors, CONFIG_NR_CPUS=8 may
> work in this case too.
> 
> Some processors/motherboards logically address the CPUs other that 0-3 if
> there are 4 processors.
> 
> Currently CONFIG_NR_CPUS needs to be big enough to hold the largest logical
> processor number.

Thanks.  This rings a bell, I seem to remember what I thought to be
"impossible numbering" on the CPUs such as "CPU7" or somesuch in a
dmesg.. Will have to look through logs.  I'll raise CONFIG_NR_CPUS and
see what I get.  Now, the question is, why do we only interrupt on CPU0
on this pasrticular box|chipset with a vanilla kernel? ACPI problem?  My
SMP G4 ppc has an option in menuconfig: "distribute interrupts on all
CPUs by default", the x86 has no such option.

Ethan

Re: HT apparently not detected properly on 2.4.23

[William Lee Irwin III]

On Wed, Dec 03, 2003 at 01:46:38PM -0500, Ethan Weinstein wrote:
> With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 
> xeons@2.4ghz and hypertherading enabled shows 4 cpu's in 
> /proc/cpuinfo|proc/interrupts, with:
> CONFIG_ACPI=y
> CONFIG_ACPI_HT_ONLY=y
> The same config with 2.4.23 only shows 2 cpus, even with:
> CONFIG_NR_CPUS=4
> Also, I believe this has been reported before, but the system only seems 
> to interrupt on CPU0 with either kernel, unless I apply a patch I found 
> here:
> http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
> or run the `irqbalance' program.. which I don't care to do.  This 
> problem _seems_ to be isolated to the Supermicro, as HT does seem 
> properly detected on several other SMP systems I have at work (compaq 
> ML370/ML380) with 2.4.23

You probably have sparse physical APIC ID's.


-- wli

Re: HT apparently not detected properly on 2.4.23

[Ethan Weinstein]

William Lee Irwin III wrote:

> On Wed, Dec 03, 2003 at 01:46:38PM -0500, Ethan Weinstein wrote:
> 
>>With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 
>>xeons@2.4ghz and hypertherading enabled shows 4 cpu's in 
>>/proc/cpuinfo|proc/interrupts, with:
>>CONFIG_ACPI=y
>>CONFIG_ACPI_HT_ONLY=y
>>The same config with 2.4.23 only shows 2 cpus, even with:
>>CONFIG_NR_CPUS=4
>>Also, I believe this has been reported before, but the system only seems 
>>to interrupt on CPU0 with either kernel, unless I apply a patch I found 
>>here:
>>http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
>>or run the `irqbalance' program.. which I don't care to do.  This 
>>problem _seems_ to be isolated to the Supermicro, as HT does seem 
>>properly detected on several other SMP systems I have at work (compaq 
>>ML370/ML380) with 2.4.23
> 
> You probably have sparse physical APIC ID's.
> 
Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like 
it was the numbering scheme:

Dec  3 18:32:28 spicymeatball kernel: CPU 0 (0x0000) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #0 Pentium 4(tm) 
XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0001] id[0x6] 
enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 1 (0x0600) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #6 Pentium 4(tm) 
XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0002] id[0x1] 
enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 2 (0x0100) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #1 Pentium 4(tm) 
XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0003] id[0x7] 
enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 3 (0x0700) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #7 Pentium 4(tm) 
XEON(tm) APIC version 16

But we're still only interrupting on CPU0 with this kernel.


-Ethan

Re: HT apparently not detected properly on 2.4.23

[William Lee Irwin III]

William Lee Irwin III wrote:
>>You probably have sparse physical APIC ID's.

On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like 
> it was the numbering scheme:

It's easy to fix this; just terminate the loop when the count of cpus
kicked hits NR_CPUS or the bit would overflow the map instead of the bit
hitting NR_CPUS. Compare smpboot.c in 2.4 vs. 2.6. No need for the extra
CONFIG_NR_CPUS bloat.


On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> But we're still only interrupting on CPU0 with this kernel.

That's P-IV retardation. Not really fixable, but irqbalance daemons
etc. can help mitigate the severity of the erratum. I'm not sure if
2.4 has the interfaces for the userspace solutions to work or the
kernel solutions either.


-- wli

Re: HT apparently not detected properly on 2.4.23

[Ethan Weinstein]

William Lee Irwin III wrote:
> 
> On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> 
>>But we're still only interrupting on CPU0 with this kernel.
> 
> 
> That's P-IV retardation. Not really fixable, but irqbalance daemons
> etc. can help mitigate the severity of the erratum. I'm not sure if
> 2.4 has the interfaces for the userspace solutions to work or the
> kernel solutions either.
> 

I somehow suspected that, and appreciate the clarification.  I'll see if 
I can't get that patch I found to apply to 2.4.23, it seems to solve 
this a bit more elegantly than the irqbalance program.

            CPU0       CPU1       CPU2       CPU3
   0:      67962      69066      68010      68954    IO-APIC-edge  timer
   1:          1          0          0          1    IO-APIC-edge  keyboard
   2:          0          0          0          0          XT-PIC  cascade
   3:      10027      10944       9904      11063    IO-APIC-edge  serial
   8:          2          0          0          0    IO-APIC-edge  rtc
  16:          9          7          7         18   IO-APIC-level  sym53c8xx
  22:       6842       7336       7166       7429   IO-APIC-level  eth0
  48:       6429       6370       6202       6268   IO-APIC-level  aic79xx
  49:       1939       2018       2067       1949   IO-APIC-level  aic79xx
  54:        340        345        340        352   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:     273824     273846     273844     273845
ERR:          0
MIS:          0

-Ethan

Re: HT apparently not detected properly on 2.4.23

[William Lee Irwin III]

On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like 
> it was the numbering scheme:

Something like this might do the trick. NR_CPUS is already checked
indirectly via max_cpus.


-- wli


===== arch/i386/kernel/smpboot.c 1.17 vs edited =====
--- 1.17/arch/i386/kernel/smpboot.c	Mon Nov  3 05:48:33 2003
+++ edited/arch/i386/kernel/smpboot.c	Wed Dec  3 16:45:27 2003
@@ -1106,7 +1106,7 @@
 	 */
 	Dprintk("CPU present map: %lx\n", phys_cpu_present_map);
 
-	for (bit = 0; bit < NR_CPUS; bit++) {
+	for (bit = 0; bit < BITS_PER_LONG; bit++) {
 		apicid = cpu_present_to_apicid(bit);
 		
 		/* don't try to boot BAD_APICID */

Re: HT apparently not detected properly on 2.4.23

[Marcelo Tosatti]

On Wed, 3 Dec 2003, William Lee Irwin III wrote:

> William Lee Irwin III wrote:
> >>You probably have sparse physical APIC ID's.
> 
> On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> > Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like 
> > it was the numbering scheme:
> 
> It's easy to fix this; just terminate the loop when the count of cpus
> kicked hits NR_CPUS or the bit would overflow the map instead of the bit
> hitting NR_CPUS. Compare smpboot.c in 2.4 vs. 2.6. No need for the extra
> CONFIG_NR_CPUS bloat.
> 
> 
> On Wed, Dec 03, 2003 at 06:41:36PM -0500, Ethan Weinstein wrote:
> > But we're still only interrupting on CPU0 with this kernel.
> 
> That's P-IV retardation. Not really fixable, but irqbalance daemons
> etc. can help mitigate the severity of the erratum. I'm not sure if
> 2.4 has the interfaces for the userspace solutions to work or the
> kernel solutions either.

I think I will just remove CONFIG_NR_CPUS instead... it seems to bring 
more trouble than advantages. 

Re: HT apparently not detected properly on 2.4.23

[William Lee Irwin III]

On Fri, Dec 05, 2003 at 03:12:14PM -0200, Marcelo Tosatti wrote:
> I think I will just remove CONFIG_NR_CPUS instead... it seems to bring 
> more trouble than advantages. 

I've gotten a success report from this patch (forwarded separately),
but that's okay.

To handle sparse physical APIC ID's within 2.4 limitations without
this patch, NR_CPUS == BITS_PER_LONG whenever CONFIG_SMP=y is an
effective requirement.


-- wli

Re[2]: HT apparently not detected properly on 2.4.23

[Russell "Elik" Rademacher]

Hmmmm.....

    SO someone finally realized that HT is not being detected properly dispite all the boot time parameters in the lilo/grub with APIC turned on?  It just seems to be a major problem since 2.4.21 version and I was hoping it was resolved. :)

    This patch you mentioned, can you send a copy to me as well?  Just want to get my clients off my back. :)


-- 
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support

Re[2]: HT apparently not detected properly on 2.4.23

[Russell "Elik" Rademacher]

Hello Ethan,

   Seems that patch of yours finally did the job of reporting it right this time around.  I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot.  And they all finally reports the number of processors correctly as it should be.

   Thanks Ethan.  I going to put it on more servers that got those and see how they all go as well.  As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters.  Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.

   Let see if the Dell servers also report the same as well when I get though with them tonight.

Friday, December 5, 2003, 11:24:37 AM, you wrote:

Ethan Weinstein> Russell "Elik" Rademacher wrote:
>> Hmmmm.....
>> 
>>     SO someone finally realized that HT is not being detected properly
>> dispite all the boot time parameters in the lilo/grub with APIC turned on? 
>> It just seems to be a major problem since 2.4.21 version and I was hoping it
>> was resolved. :)
>> 
>>     This patch you mentioned, can you send a copy to me as well?  Just
>> want to get my clients off my back. :)
>> 
>> 

Ethan Weinstein> heheh. Welcome to linux-kernel.  Try running
Ethan Weinstein> an arch other than x86 if 
Ethan Weinstein> you _really_ like having your problems ignored =)


Ethan Weinstein> ===== arch/i386/kernel/smpboot.c 1.17 vs edited =====
Ethan Weinstein> --- 1.17/arch/i386/kernel/smpboot.c    Mon Nov  3 05:48:33 2003
Ethan Weinstein> +++ edited/arch/i386/kernel/smpboot.c    Wed Dec  3 16:45:27 2003
Ethan Weinstein> @@ -1106,7 +1106,7 @@
Ethan Weinstein>        */
Ethan Weinstein>       Dprintk("CPU present map: %lx\n", phys_cpu_present_map);

Ethan Weinstein> -    for (bit = 0; bit < NR_CPUS; bit++) {
Ethan Weinstein> +    for (bit = 0; bit < BITS_PER_LONG; bit++) {
Ethan Weinstein>           apicid = cpu_present_to_apicid(bit);
Ethan Weinstein>           /* don't try to boot BAD_APICID */





-- 
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support

Re: HT apparently not detected properly on 2.4.23

[Ethan Weinstein]

Russell "Elik" Rademacher wrote:
> Hello Ethan,
> 
>    Seems that patch of yours finally did the job of reporting it right this time around.  I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot.  And they all finally reports the number of processors correctly as it should be.
> 
>    Thanks Ethan.  I going to put it on more servers that got those and see how they all go as well.  As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters.  Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.
> 
>    Let see if the Dell servers also report the same as well when I get though with them tonight.
> 

Glad to help, but don't credit me with that patch, it was courtesy of 
William Irwin.


-E

Re[2]: HT apparently not detected properly on 2.4.23

[Russell "Elik" Rademacher]

Hello Ethan,

   So noted.  Thanks William for this.  Now I have to resume patching the rest of 284 servers from various clients this weekend and have it all brought up to speed.

Friday, December 5, 2003, 4:28:37 PM, you wrote:

Ethan Weinstein> Russell "Elik" Rademacher wrote:
>> Hello Ethan,
>> 
>>    Seems that patch of yours finally did the job of reporting it right
>> this time around.  I had to gank a few servers of different versions and
>> giving the webhosting customers some little inconvience of downtime of 3 to
>> 10 minutes for the linux recompile and reboot.  And they all finally reports
>> the number of processors correctly as it should be.
>> 
>>    Thanks Ethan.  I going to put it on more servers that got those and
>> see how they all go as well.  As for the servers, they are all usually
>> ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek
>> network ports and with 3Ware or SCSI Adaptec adapters.  Got about 80 of them
>> in various types to go though, but 5 of them I that tested it on, it all
>> reports correctly.
>> 
>>    Let see if the Dell servers also report the same as well when I get though with them tonight.
>> 

Ethan Weinstein> Glad to help, but don't credit me with that
Ethan Weinstein> patch, it was courtesy of 
Ethan Weinstein> William Irwin.


Ethan Weinstein> -E




-- 
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support

Re: HT apparently not detected properly on 2.4.23

[William Lee Irwin III]

On Fri, Dec 05, 2003 at 04:31:29PM -0700, Russell Elik Rademacher wrote:
>    So noted.  Thanks William for this.  Now I have to resume patching
> the rest of 284 servers from various clients this weekend and have it
> all brought up to speed.

Good to hear it's resolved the issue.

Thanks for testing.


-- wli