Re: local apic timer ints not working with vmware: nolocala

Re: local apic timer ints not working with vmware: nolocala

[Petr Vandrovec]

On 27 May 03 at 20:53, Brian J. Murrell wrote:
> Using a distribution (Mandrake's Cooker) patched kernel, I am unable to
> successfully boot at 2.4.20ish kernel on VMware 2.0.4 build 1142.  The
> problem seems to be with using local apic timer interrupts.  The last few
> lines of the boot sequence before the kernel hangs are (copied by hand, so
> please excuse typos):
> 
> Using local APIC timer interrupts.
> calibrating APIC timer ...
> ..... CPU clock speed is 1658.7651 MHz.
> ..... host bus clock speed is 0.0000 MHz.
> cpu: 0, clocks: 0, slice: 0
> 
> I do know that I could simply build a kernel with the CONFIG_X86_UP_APIC
> to work around this problem, but that means a "special" kernel just for
> VMware and also means not being able to use vendor supplied kernels, or
> vendor supplied boot media (i.e. CD-ROMs).

Just get VMware 3 or 4 - they (properly) emulate APIC timer and you'll
get information that host bus is running at 66.xxxx MHz. With VMware 2
you have to boot with noapic.
                                                Petr Vandrovec
                                                

Re: local apic timer ints not working with vmware: nolocalapic

[brian]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On Wed, May 28, 2003 at 11:56:29AM +0200, Petr Vandrovec wrote:
> 
> Just get VMware 3 or 4

Unfortunately, I cannot currently afford to buy (outright -- there is
no upgrade path from 2 to 4) a newer version of VMware.  Really, other
than this issue of the local APIC timer, I don't really need or care
to have a newer version of VMware anyway.

> - they (properly) emulate APIC timer and you'll
> get information that host bus is running at 66.xxxx MHz. With VMware 2
> you have to boot with noapic.

If only this worked.  I tried noapic, but it still tries to use the
local APIC timer interrupts.  noapic seems to only disable IO-APIC.
That is why I was "proposing" a new kernel command line arg,
"nolocalapic".

b.

-- 
Brian J. Murrell

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Wed, 28 May 2003 brian@interlinx.bc.ca wrote:

> > - they (properly) emulate APIC timer and you'll
> > get information that host bus is running at 66.xxxx MHz. With VMware 2
> > you have to boot with noapic.
> 
> If only this worked.  I tried noapic, but it still tries to use the
> local APIC timer interrupts.  noapic seems to only disable IO-APIC.
> That is why I was "proposing" a new kernel command line arg,
> "nolocalapic".

I submitted a patch for nolapic before...

Index: linux-2.5.45-ac1/arch/i386/kernel/apic.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.45-ac1/arch/i386/kernel/apic.c,v
retrieving revision 1.1.1.1
diff -u -r1.1.1.1 apic.c
--- linux-2.5.45-ac1/arch/i386/kernel/apic.c	5 Nov 2002 04:47:02 -0000	1.1.1.1
+++ linux-2.5.45-ac1/arch/i386/kernel/apic.c	6 Nov 2002 00:19:38 -0000
@@ -609,11 +609,27 @@
 
 #endif	/* CONFIG_PM */
 
+int enable_local_apic_flag __initdata = 0; /* 0=probe, 1=force, 2=disable e.g. DMI */
+
+static int __init nolapic_setup(char *str)
+{
+	enable_local_apic_flag = 2;
+	return 1;
+}
+
+static int __init lapic_setup(char *str)
+{
+	enable_local_apic_flag = 1;
+	return 1;
+}
+
+__setup("nolapic", nolapic_setup);
+__setup("lapic", lapic_setup);
+
 /*
  * Detect and enable local APICs on non-SMP boards.
  * Original code written by Keir Fraser.
  */
-int dont_enable_local_apic __initdata = 0;
 
 static int __init detect_init_APIC (void)
 {
@@ -621,11 +637,14 @@
 	extern void get_cpu_vendor(struct cpuinfo_x86*);
 
 	/* Disabled by DMI scan or kernel option? */
-	if (dont_enable_local_apic)
+	if (enable_local_apic_flag == 2)
 		return -1;
 
 	/* Workaround for us being called before identify_cpu(). */
 	get_cpu_vendor(&boot_cpu_data);
+	
+	if (enable_local_apic_flag == 1)
+		goto force_apic;
 
 	switch (boot_cpu_data.x86_vendor) {
 	case X86_VENDOR_AMD:
@@ -642,6 +661,7 @@
 		goto no_apic;
 	}
 
+force_apic:
 	if (!cpu_has_apic) {
 		/*
 		 * Some BIOSes disable the local APIC in the
Index: linux-2.5.45-ac1/arch/i386/kernel/dmi_scan.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.45-ac1/arch/i386/kernel/dmi_scan.c,v
retrieving revision 1.1.1.1
diff -u -r1.1.1.1 dmi_scan.c
--- linux-2.5.45-ac1/arch/i386/kernel/dmi_scan.c	5 Nov 2002 04:47:02 -0000	1.1.1.1
+++ linux-2.5.45-ac1/arch/i386/kernel/dmi_scan.c	6 Nov 2002 00:22:02 -0000
@@ -314,9 +314,9 @@
 static int __init local_apic_kills_bios(struct dmi_blacklist *d)
 {
 #ifdef CONFIG_X86_LOCAL_APIC
-	extern int dont_enable_local_apic;
-	if (!dont_enable_local_apic) {
-		dont_enable_local_apic = 1;
+	extern int enable_local_apic_flag;
+	if (!enable_local_apic_flag) {
+		enable_local_apic_flag = 2;
 		printk(KERN_WARNING "%s with broken BIOS detected. "
 		       "Refusing to enable the local APIC.\n",
 		       d->ident);
@@ -333,9 +333,9 @@
 static int __init apm_kills_local_apic(struct dmi_blacklist *d)
 {
 #ifdef CONFIG_X86_LOCAL_APIC
-	extern int dont_enable_local_apic;
-	if (apm_info.bios.version && !dont_enable_local_apic) {
-		dont_enable_local_apic = 1;
+	extern int enable_local_apic_flag;
+	if (apm_info.bios.version && !enable_local_apic_flag) {
+		enable_local_apic_flag = 2;
 		printk(KERN_WARNING "%s with broken BIOS detected. "
 		       "Refusing to enable the local APIC.\n",
 		       d->ident);
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Brian J. Murrell]

On Wed, 28 May 2003 13:42:13 -0400, Zwane Mwaikambo wrote:
> 
> I submitted a patch for nolapic before...

Did you get any response as to whether it was going to be accepted into
the kernel or not?

The unfortunate thing is that even this sort of fix will not help my
situation.  The reason being (which I only discovered by accident when I
set "dont_enable_local_apic = 1" rather than "dont_use_local_apic_timer"
and it didn't correct the booting problem) is that it seems that even if
the local apic is set disabled by setting dont_enable_local_apic = 1 in
arch/i386/kernel/apic.c, setup_APIC_clocks() is still called.

So the jist is that using the local apic timer feature is not dependent on
using the local apic, as per the dont_enable_local_apic and
dont_use_local_apic_timer flags in arch/i386/kernel/apic.c.  Maybe this is
wrong, I dunno unfortunately.

I don't know anything about this APIC stuff so I don't know if that is
correct or not, but it is what happens.

Thanx for the input though, much appreciated,

b.

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Fri, 30 May 2003, Brian J. Murrell wrote:

> On Wed, 28 May 2003 13:42:13 -0400, Zwane Mwaikambo wrote:
> > 
> > I submitted a patch for nolapic before...
> 
> Did you get any response as to whether it was going to be accepted into
> the kernel or not?

Considering the dalay, i'll resend and give it another go, but generally 
it means it's not going anywhere.

> The unfortunate thing is that even this sort of fix will not help my
> situation.  The reason being (which I only discovered by accident when I
> set "dont_enable_local_apic = 1" rather than "dont_use_local_apic_timer"
> and it didn't correct the booting problem) is that it seems that even if
> the local apic is set disabled by setting dont_enable_local_apic = 1 in
> arch/i386/kernel/apic.c, setup_APIC_clocks() is still called.

How did you determine that? printks? Was this with my patch applied? I 
originally did this patch for the exact same problem (buggy local APIC 
implimentation).

(much snipped)
Linux version 2.5.70-mm1 (zwane@montezuma.mastecende.com) (gcc version 
Kernel command line: nolapic nmi_watchdog=2 ro root=/dev/hda1 profile=2 
debug console=tty0 cons0
kernel profiling enabled
Initializing CPU#0
CPU0: Intel Celeron (Mendocino) stepping 05
per-CPU timeslice cutoff: 365.65 usecs.
task migration cache decay timeout: 1 msecs.
SMP motherboard not detected.
Local APIC not detected. Using dummy APIC emulation.
Starting migration thread for cpu 0

> So the jist is that using the local apic timer feature is not dependent on
> using the local apic, as per the dont_enable_local_apic and
> dont_use_local_apic_timer flags in arch/i386/kernel/apic.c.  Maybe this is
> wrong, I dunno unfortunately.
> 
> I don't know anything about this APIC stuff so I don't know if that is
> correct or not, but it is what happens.
> 
> Thanx for the input though, much appreciated,

You're welcome,
	Zwane
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Brian J. Murrell]

On Fri, 30 May 2003 19:23:33 -0400, Zwane Mwaikambo wrote:
> 
> Considering the dalay, i'll resend and give it another go, but generally 
> it means it's not going anywhere.

That sucks.

> 
>> The unfortunate thing is that even this sort of fix will not help my
>> situation.  The reason being (which I only discovered by accident when I
>> set "dont_enable_local_apic = 1" rather than "dont_use_local_apic_timer"
>> and it didn't correct the booting problem) is that it seems that even if
>> the local apic is set disabled by setting dont_enable_local_apic = 1 in
>> arch/i386/kernel/apic.c, setup_APIC_clocks() is still called.
> 
> How did you determine that?

Well, originally it was a mistake in setting the wrong flag to 1
(dont_enable_local_apic rather than dont_use_local_apic_timer) and finding
that it did not cure the problem, when I went back to my change to figure
out why, I noticed that I had set the wrong flag.

> Was this with my patch applied?

I did not try your patch since I did pretty much the same thing with my
"mistake" described above.

> I 
> originally did this patch for the exact same problem (buggy local APIC 
> implimentation).

I know nothing about the APIC stuff, but it seems strange that even though
it's disabled (dont_enable_local_apic = 1) setup_APIC_clocks() is still
called.  Maybe the latter is not dependent on the former, so there should
not be a dependence on dont_enable_local_apic == 0 for setup_APIC_clocks()
to still be used.

> Linux version 2.5.70-mm1 (zwane@montezuma.mastecende.com) (gcc version 
> Kernel command line: nolapic nmi_watchdog=2 ro root=/dev/hda1 profile=2 
> debug console=tty0 cons0
> kernel profiling enabled
> Initializing CPU#0
> CPU0: Intel Celeron (Mendocino) stepping 05
> per-CPU timeslice cutoff: 365.65 usecs.
> task migration cache decay timeout: 1 msecs.
> SMP motherboard not detected.
> Local APIC not detected. Using dummy APIC emulation.
> Starting migration thread for cpu 0

Yes, it's further along in the boot where my system runs into trouble,
right here:

Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1658.7651 MHz.
..... host bus clock speed is 0.0000 MHz.
cpu: 0, clocks: 0, slice: 0

I will take another stab at all of this tomorrow to double-verify what I
am saying here regarding the use of local APIC timer interrupts even if
the local apic usage flag is set to disable (dont_enable_local_apic = 1).

b.

Re: local apic timer ints not working with vmware: nolocalapic

[Brian J. Murrell]

On Fri, 30 May 2003 23:38:16 -0400, Brian J. Murrell wrote:
> 
> I will take another stab at all of this tomorrow to double-verify what I
> am saying here regarding the use of local APIC timer interrupts even if
> the local apic usage flag is set to disable (dont_enable_local_apic = 1).

Just to confirm now, I have modified Zwane's patch add another kernel arg,
[no]locapictimer, which deals with dont_use_local_apic_timer in the same
way his patch deals with the dont_enable_local_apic flag, and indeed, a
kernel booted with "nolapic" does hang in the APIC timer calibration
however a kernel booted with "nolocapictimer" does not.

Is it really valid to go and try to calibrate the APIC timer if it was
disabled by the user, or even DMI?

b.

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Sat, 31 May 2003, Brian J. Murrell wrote:

> Just to confirm now, I have modified Zwane's patch add another kernel arg,
> [no]locapictimer, which deals with dont_use_local_apic_timer in the same
> way his patch deals with the dont_enable_local_apic flag, and indeed, a
> kernel booted with "nolapic" does hang in the APIC timer calibration
> however a kernel booted with "nolocapictimer" does not.
> 
> Is it really valid to go and try to calibrate the APIC timer if it was
> disabled by the user, or even DMI?

bah, the early boot hacks are just ugly, we do detect_init_APIC early in 
traps code and then just blindly go frobbing the APIC anyway.

The following patch will honour the dont_enable_local_apic flag in the 
necessary places. If it works for you i'll forward it.

Index: linux-2.5/arch/i386/kernel/apic.c
===================================================================
RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
retrieving revision 1.50
diff -u -p -B -r1.50 apic.c
--- linux-2.5/arch/i386/kernel/apic.c	30 May 2003 20:14:41 -0000	1.50
+++ linux-2.5/arch/i386/kernel/apic.c	31 May 2003 05:53:34 -0000
@@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
 	return 0;
 
 no_apic:
+	dont_enable_local_apic = 1;
 	printk("No local APIC present or hardware disabled\n");
 	return -1;
 }
@@ -1127,6 +1128,9 @@ asmlinkage void smp_error_interrupt(void
  */
 int __init APIC_init_uniprocessor (void)
 {
+	if (dont_enable_local_apic)
+		return -1;
+
 	if (!smp_found_config && !cpu_has_apic)
 		return -1;
 
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Sat, 31 May 2003, Brian J. Murrell wrote:

> Is it really valid to go and try to calibrate the APIC timer if it was
> disabled by the user, or even DMI?

Nope, could you add me to CC in future messages? It's easier for me to 
spot messages which need replying.

Thanks,
	Zwane
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Brian J. Murrell]

On Sat, 31 May 2003 02:50:35 -0400, Zwane Mwaikambo wrote: 
> 
> The following patch will honour the dont_enable_local_apic flag in the 
> necessary places. If it works for you i'll forward it.

Yesserrie, it works, assuming dont_enable_local_apic gets set to "1"
somewhere, somehow.  In the case of broken hardware (or a broken emulation
of it, like VMware 2.0.4) however, a command line arg (such as the nolapic
patch you posted previously) is still needed to disable the local apic in
the case where there is no DMI information that can be used to add the
broken system to the blacklist.

A combination of your previous nolapic patch and this one you just posted
would do the trick indeed.

Thanx much!

b.

Re: local apic timer ints not working with vmware: nolocalapic

[Maciej W. Rozycki]

On Sat, 31 May 2003, Zwane Mwaikambo wrote:

> > Is it really valid to go and try to calibrate the APIC timer if it was
> > disabled by the user, or even DMI?
> 
> bah, the early boot hacks are just ugly, we do detect_init_APIC early in 
> traps code and then just blindly go frobbing the APIC anyway.

 How about clearing cpu_has_apic and smp_found_config instead?  As I
understand the problem, the local APIC is useless so the kernel shouldn't
pretend it's present.  And the MP-table is useless without a local APIC. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Tue, 3 Jun 2003, Maciej W. Rozycki wrote:

> On Sat, 31 May 2003, Zwane Mwaikambo wrote:
>  How about clearing cpu_has_apic and smp_found_config instead?  As I
> understand the problem, the local APIC is useless so the kernel shouldn't
> pretend it's present.  And the MP-table is useless without a local APIC. 

I agree, but there are already the appropriate checks for wether there is 
an APIC enabled that should suffice.

	Zwane
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Maciej W. Rozycki]

On Wed, 4 Jun 2003, Zwane Mwaikambo wrote:

> > On Sat, 31 May 2003, Zwane Mwaikambo wrote:
> >  How about clearing cpu_has_apic and smp_found_config instead?  As I
> > understand the problem, the local APIC is useless so the kernel shouldn't
> > pretend it's present.  And the MP-table is useless without a local APIC. 
> 
> I agree, but there are already the appropriate checks for wether there is 
> an APIC enabled that should suffice.

 You may have a valid SMP table and discrete local APICs (i82489DX) which
are not reported in CPU capability bits.  The "nolocalapic" option should
handle them, too.  Otherwise it would be a surprising inconsistency. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Thu, 5 Jun 2003, Maciej W. Rozycki wrote:

>  You may have a valid SMP table and discrete local APICs (i82489DX) which
> are not reported in CPU capability bits.  The "nolocalapic" option should
> handle them, too.  Otherwise it would be a surprising inconsistency. 

Good point, out of interest, have you come across broken system like that? 
Regardless i'll update the patch.

Thanks,
	Zwane
-- 
function.linuxpower.ca

Re: local apic timer ints not working with vmware: nolocalapic

[Maciej W. Rozycki]

On Thu, 5 Jun 2003, Zwane Mwaikambo wrote:

> >  You may have a valid SMP table and discrete local APICs (i82489DX) which
> > are not reported in CPU capability bits.  The "nolocalapic" option should
> > handle them, too.  Otherwise it would be a surprising inconsistency. 
> 
> Good point, out of interest, have you come across broken system like that? 

 So far I've met three users of i82489DX-based systems, two of whom helped
me making them work with 2.4.  So at least at the beginning of 2.4.x they
used to work; hopefully nothing has got broken since then (I try to
monitor changes, but without real hardware to do testing something might
have slipped in unnoticed).  I don't know if these people have kept
upgrading their kernels nor whether they still use the systems.  There
were no bug reports, either, which basically may mean anything. 

 Why do you consider the systems broken?

> Regardless i'll update the patch.

 Great!

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: local apic timer ints not working with vmware: nolocalapic

[Zwane Mwaikambo]

On Mon, 9 Jun 2003, Maciej W. Rozycki wrote:

>  Why do you consider the systems broken?

Not necessarily broken, just no reporting of APIC capability. Not that i 
should expect better from Intel (c.f. HT bit, SEP on PPro etc)

> > Regardless i'll update the patch.
> 
>  Great!

How about we only clear smp_found_config when forced.

Index: linux-2.5/arch/i386/kernel/apic.c
===================================================================
RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
retrieving revision 1.42
diff -u -p -B -r1.42 apic.c
--- linux-2.5/arch/i386/kernel/apic.c	26 May 2003 23:59:58 -0000	1.42
+++ linux-2.5/arch/i386/kernel/apic.c	10 Jun 2003 00:14:37 -0000
@@ -602,6 +602,15 @@ static void apic_pm_activate(void) { }
  */
 int dont_enable_local_apic __initdata = 0;
 
+static int __init nolapic_setup(char *str)
+{
+	dont_enable_local_apic = 1;
+	smp_found_config = 0;
+	return 1;
+}
+
+__setup("nolapic", nolapic_setup);
+
 static int __init detect_init_APIC (void)
 {
 	u32 h, l, features;
@@ -609,7 +618,7 @@ static int __init detect_init_APIC (void
 
 	/* Disabled by DMI scan or kernel option? */
 	if (dont_enable_local_apic)
-		return -1;
+		goto no_apic;
 
 	/* Workaround for us being called before identify_cpu(). */
 	get_cpu_vendor(&boot_cpu_data);
@@ -665,6 +674,7 @@ static int __init detect_init_APIC (void
 	return 0;
 
 no_apic:
+	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
 	printk("No local APIC present or hardware disabled\n");
 	return -1;
 }
Index: linux-2.5/Documentation/kernel-parameters.txt
===================================================================
RCS file: /home/cvs/linux-2.5/Documentation/kernel-parameters.txt,v
retrieving revision 1.24
diff -u -p -B -r1.24 kernel-parameters.txt
--- linux-2.5/Documentation/kernel-parameters.txt	6 Jun 2003 15:55:40 -0000	1.24
+++ linux-2.5/Documentation/kernel-parameters.txt	10 Jun 2003 00:14:38 -0000
@@ -625,6 +625,9 @@ running once the system is up.
 
 	nointroute	[IA-64]
 
+	nolapic		[IA-32, APIC]
+			Disable Local APIC.
+
 	nomce		[IA-32] Machine Check Exception
 
 	noresume	[SWSUSP] Disables resume and restore original swap space.

Re: local apic timer ints not working with vmware: nolocalapic

[Maciej W. Rozycki]

On Mon, 9 Jun 2003, Zwane Mwaikambo wrote:

> >  Why do you consider the systems broken?
> 
> Not necessarily broken, just no reporting of APIC capability. Not that i 
> should expect better from Intel (c.f. HT bit, SEP on PPro etc)

 Hmm, how could e.g. an i486 report it?  Remember, the i82489DX is a
discrete APIC implementation, i.e. it's a chip external to the CPU.  The
i82489DX is actually a complete APIC implementation, including both a
local and an I/O unit in a single chip.  And it's also superior to
integrated APIC implementations -- it can address up to 255 units in the
physical destination mode and up to 32 ones in the logical one.

 These APICs were used for i486 systems as well as for Pentium ones meant
to support more than two CPUs (although the Pentium integrated local APICs
can probably support more than two such units in a system, there was no
suitable I/O unit available; the i82093AA chip was introduced later and
the i82489DX is hardware-incompatible to later implementations, e.g. 
using a five-wire inter-APIC bus).

> How about we only clear smp_found_config when forced.

 Looks OK.  Probably a message could be added to report handling of the
local APIC got disabled. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +