[BUG] x86 kenel won't boot under Virtual PC

[BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

I recently discovered that x86 kernels won't boot under Virtual PC.  In this 
case paravirt was not built into the kernel.  The kernel just "hangs" on 
attempted boot with no error messages.  Git-bisect pinpointed the following 
commit as the problem:

commit 32c464f5d9701db45bc1673288594e664065388e
Author: Jan Beulich <jbeulich@novell.com>
Date:   Wed Oct 17 18:04:41 2007 +0200

    x86: multi-byte single instruction NOPs

    Add support for and use the multi-byte NOPs recently documented to be
    available on all PentiumPro and later processors.

    This patch only applies cleanly on top of the "x86: misc.
    constifications" patch sent earlier.

    [ tglx: arch/x86 adaptation ]

    Signed-off-by: Jan Beulich <jbeulich@novell.com>
    Signed-off-by: Andi Kleen <ak@suse.de>
    Signed-off-by: Ingo Molnar <mingo@elte.hu>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

     arch/x86/kernel/alternative.c  |   23 ++++++++++++++++++++++-
     include/asm-x86/processor_32.h |   22 ++++++++++++++++++++++
     include/asm-x86/processor_64.h |   22 ++++++++++++++++++++++
     3 files changed, 66 insertions(+), 1 deletion(-)

:040000 040000 9226efb160ea180e8c419134b82e0a6938868c1a 
12daabe81d9cda0d574815c0957a24c1028c8fb0 M      arch
:040000 040000 7ab3918741abcb203bd25289d2c6e789bed64fc1 
eb662fea6ed7904074c87dd004b9f6d23964c844 M      include

Any suggestion on where to go from here would be appreciated.

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Sun, 31 Aug 2008, David Sanders wrote:
>
> I recently discovered that x86 kernels won't boot under Virtual PC.

What CPU does Virtual PC emulate? As far as Wikipedia is concerned (not 
that I'd take it on complete faith) it emulates a 32-bit Intel Pentium II.

And that commit makes the kernel use the "P6 nops" for such hardware. 
Maybe Virtual PC doesn't support the newer intel nop things?

Intel docs say that it should be available on any intel CPU that has 
CPUID.01H.EAX[11:8] = 0110B or 1111B. That's the "family ID", and Pentium 
II should have a family ID of 6 (ie that 0110B case).

So it sounds like a Virtual PC bug, but I dunno. And maybe we should just 
use the legcay nops for anything that isn't modern (ie P4+ or Core)?

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[Arjan van de Ven]

On Sun, 31 Aug 2008 11:47:04 -0700 (PDT)
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> 
> On Sun, 31 Aug 2008, David Sanders wrote:
> >
> > I recently discovered that x86 kernels won't boot under Virtual PC.
> 
> What CPU does Virtual PC emulate? As far as Wikipedia is concerned
> (not that I'd take it on complete faith) it emulates a 32-bit Intel
> Pentium II.
> 
> And that commit makes the kernel use the "P6 nops" for such hardware. 
> Maybe Virtual PC doesn't support the newer intel nop things?
> 
> Intel docs say that it should be available on any intel CPU that has 
> CPUID.01H.EAX[11:8] = 0110B or 1111B. That's the "family ID", and
> Pentium II should have a family ID of 6 (ie that 0110B case).
> 
> So it sounds like a Virtual PC bug, but I dunno. And maybe we should
> just use the legcay nops for anything that isn't modern (ie P4+ or
> Core)?

it's probably even a security bug in that I don't see what would be
stopping a ring 3 user process from executing these instructions...

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Sun, 31 Aug 2008, Arjan van de Ven wrote:
>
> it's probably even a security bug in that I don't see what would be
> stopping a ring 3 user process from executing these instructions...

Well, it could be that Virtual PC raises a #UD exception in the virtual 
machine. In user space, that would just cause the kernel to kill the poor 
innocent victim. But when the kernel gets a #UD exception on what it 
expects to be a nop, it just won't work.

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Sunday 31 August 2008 14:47, Linus Torvalds wrote:
> On Sun, 31 Aug 2008, David Sanders wrote:
> > I recently discovered that x86 kernels won't boot under Virtual PC.
>
> What CPU does Virtual PC emulate? As far as Wikipedia is concerned (not
> that I'd take it on complete faith) it emulates a 32-bit Intel Pentium II.
>
> And that commit makes the kernel use the "P6 nops" for such hardware.
> Maybe Virtual PC doesn't support the newer intel nop things?
>
> Intel docs say that it should be available on any intel CPU that has
> CPUID.01H.EAX[11:8] = 0110B or 1111B. That's the "family ID", and Pentium
> II should have a family ID of 6 (ie that 0110B case).
>
> So it sounds like a Virtual PC bug, but I dunno. And maybe we should just
> use the legcay nops for anything that isn't modern (ie P4+ or Core)?
>
> 		Linus

Virtual PC does not emulate a processor like it does with the motherboard, 
video, NIC ,etc.  What you see in the virtual machine is the actual 
processor, except that Virtual PC looks at all of your instructions and 
modifies ring 0 code and the like.  It may be that Virtual PC was not 
designed with an awareness of these nops that the commit added.

I would suggest an configuration option to select legacy-nops or newer-nops 
and a kernel boot-time parameter so it can be disabled to allow installation 
of a distribution for example.

I would be happy to submit such a patch if you agree (or I'll try to anyway).

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Sunday 31 August 2008 16:03, David Sanders wrote:
> On Sunday 31 August 2008 14:47, Linus Torvalds wrote:
> > On Sun, 31 Aug 2008, David Sanders wrote:
> > > I recently discovered that x86 kernels won't boot under Virtual PC.
> >
> > What CPU does Virtual PC emulate? As far as Wikipedia is concerned (not
> > that I'd take it on complete faith) it emulates a 32-bit Intel Pentium
> > II.
> >
> > And that commit makes the kernel use the "P6 nops" for such hardware.
> > Maybe Virtual PC doesn't support the newer intel nop things?
> >
> > Intel docs say that it should be available on any intel CPU that has
> > CPUID.01H.EAX[11:8] = 0110B or 1111B. That's the "family ID", and Pentium
> > II should have a family ID of 6 (ie that 0110B case).
> >
> > So it sounds like a Virtual PC bug, but I dunno. And maybe we should just
> > use the legcay nops for anything that isn't modern (ie P4+ or Core)?
> >
> > 		Linus
>
> Virtual PC does not emulate a processor like it does with the motherboard,
> video, NIC ,etc.  What you see in the virtual machine is the actual
> processor, except that Virtual PC looks at all of your instructions and
> modifies ring 0 code and the like.  It may be that Virtual PC was not
> designed with an awareness of these nops that the commit added.
>
> I would suggest an configuration option to select legacy-nops or newer-nops
> and a kernel boot-time parameter so it can be disabled to allow
> installation of a distribution for example.
>
> I would be happy to submit such a patch if you agree (or I'll try to
> anyway).

OK, below is my patch.  The problem is I just built with it and the boot 
problem still exists.  Either the patch is flawed or git-bisect gave me the 
wrong commit.  Any help would be appreciated.
David

Signed-off-by: David Sanders <linux@sandersweb.net>

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index ed92864..a38f362 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -445,6 +445,13 @@ config PARAVIRT_DEBUG
          Enable to debug paravirt_ops internals.  Specifically, BUG if
 	 a paravirt_op is missing when it is called.
 
+config INTEL_NOPS
+	bool "Use legacy x86 NOPS"
+	depends on X86_32
+	help
+	This option tells kernel to use legacy x86 nops
+
+
 config MEMTEST
 	bool "Memtest"
 	help
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 2763cb3..10713e4 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -58,6 +58,16 @@ static int __init setup_noreplace_paravirt(char *str)
 __setup("noreplace-paravirt", setup_noreplace_paravirt);
 #endif
 
+static int intel_legacy_nops = 0;
+
+static int __init setup_intel_legacy_nops(char *str)
+{
+	intel_legacy_nops = 1;
+	return 1;
+}
+
+__setup("intel-legacy-nops", setup_intel_legacy_nops);
+
 #define DPRINTK(fmt, args...) if (debug_alternative) \
 	printk(KERN_DEBUG fmt, args)
 
@@ -167,12 +177,17 @@ const unsigned char *const *find_nop_table(void)
 	const unsigned char *const *noptable = intel_nops;
 	int i;
 
-	for (i = 0; noptypes[i].cpuid >= 0; i++) {
-		if (boot_cpu_has(noptypes[i].cpuid)) {
-			noptable = noptypes[i].noptable;
-			break;
+	if (!intel_legacy_nops) {
+#ifndef CONFIG_INTEL_NOPS
+		for (i = 0; noptypes[i].cpuid >= 0; i++) {
+			if (boot_cpu_has(noptypes[i].cpuid)) {
+				noptable = noptypes[i].noptable;
+				break;
+			}
 		}
+#endif /* CONFIG_INTEL_NOPS */
 	}
+
 	return noptable;
 }

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 1 Sep 2008, David Sanders wrote:
> 
> OK, below is my patch.  The problem is I just built with it and the boot 
> problem still exists.  Either the patch is flawed or git-bisect gave me the 
> wrong commit.  Any help would be appreciated.

Try just reverting the commit that you bisected to, and double-check. If 
your bisect was flawed, there's no point even staring at the nop code ;)

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Monday 01 September 2008 18:22, Linus Torvalds wrote:
> On Mon, 1 Sep 2008, David Sanders wrote:
> > OK, below is my patch.  The problem is I just built with it and the boot
> > problem still exists.  Either the patch is flawed or git-bisect gave me
> > the wrong commit.  Any help would be appreciated.
>
> Try just reverting the commit that you bisected to, and double-check. If
> your bisect was flawed, there's no point even staring at the nop code ;)
>
> 		Linus

Thanks Linus.
I reverted the commit and the problem persits, so nops are definitely not the 
problem.  I started an new bisection but am now in a state that won't compile 
due to errors.  What do I do in that case?
David

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Tue, 2 Sep 2008, David Sanders wrote:
>
> I reverted the commit and the problem persits, so nops are definitely not the 
> problem.  I started an new bisection but am now in a state that won't compile 
> due to errors.  What do I do in that case?

You can try "git bisect skip", but in general the better choice is to just 
do

	git bisect visualize

to open up gitk with the current set of possible targets, and then just 
pick a likely point by hand, and do

	git reset --hard <sha1-of-the-thing-you-picked>

and try that one instead. There's some talk about this in "man git-bisect"
but maybe it's not very good.

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Tuesday 02 September 2008 14:12, Linus Torvalds wrote:
> On Tue, 2 Sep 2008, David Sanders wrote:
> > I reverted the commit and the problem persits, so nops are definitely not
> > the problem.  I started an new bisection but am now in a state that won't
> > compile due to errors.  What do I do in that case?
>
> You can try "git bisect skip", but in general the better choice is to just
> do
>
> 	git bisect visualize
>
> to open up gitk with the current set of possible targets, and then just
> pick a likely point by hand, and do
>
> 	git reset --hard <sha1-of-the-thing-you-picked>
>
> and try that one instead. There's some talk about this in "man git-bisect"
> but maybe it's not very good.
>
> 			Linus

OK thanks, I had to build a new version of git because the one installed with 
my distribution was too old to have git bisect skip.  I am trying to use the 
gitk method you mentioned but it is going slow and on my screen gitk's fonts 
are very hard to read.
David

Re: [BUG] x86 kenel won't boot under Virtual PC

[Peter Zijlstra]

On Tue, 2008-09-02 at 14:44 -0400, David Sanders wrote:
> On Tuesday 02 September 2008 14:12, Linus Torvalds wrote:
> > On Tue, 2 Sep 2008, David Sanders wrote:
> > > I reverted the commit and the problem persits, so nops are definitely not
> > > the problem.  I started an new bisection but am now in a state that won't
> > > compile due to errors.  What do I do in that case?
> >
> > You can try "git bisect skip", but in general the better choice is to just
> > do
> >
> > 	git bisect visualize
> >
> > to open up gitk with the current set of possible targets, and then just
> > pick a likely point by hand, and do
> >
> > 	git reset --hard <sha1-of-the-thing-you-picked>
> >
> > and try that one instead. There's some talk about this in "man git-bisect"
> > but maybe it's not very good.
> >
> > 			Linus
> 
> OK thanks, I had to build a new version of git because the one installed with 
> my distribution was too old to have git bisect skip.  I am trying to use the 
> gitk method you mentioned but it is going slow and on my screen gitk's fonts 
> are very hard to read.

Yeah - gitk's default fonts are a nightmare, I get a semi readable
interface by adding this to ~/.gitk

set mainfont {{dejavu sans mono} 10}
set textfont {{dejavu sans mono} 9}
set uifont {{dejavu sans mono} 10 bold}

But it seems the tcl/tk font rendering is pretty horrible all-round. I
generally prefer to use qgit for this reason.

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Wednesday 03 September 2008 07:09, Peter Zijlstra wrote:
> Yeah - gitk's default fonts are a nightmare, I get a semi readable
> interface by adding this to ~/.gitk
>
> set mainfont {{dejavu sans mono} 10}
> set textfont {{dejavu sans mono} 9}
> set uifont {{dejavu sans mono} 10 bold}
>
> But it seems the tcl/tk font rendering is pretty horrible all-round. I
> generally prefer to use qgit for this reason.
Thanks that is a big improvement.

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Sun, 31 Aug 2008, David Sanders wrote:
> I recently discovered that x86 kernels won't boot under Virtual PC.On Sun, 
31 Aug 2008, David Sanders wrote:

On Sunday 31 August 2008 15:39, Linus Torvalds wrote:
> Well, it could be that Virtual PC raises a #UD exception in the virtual
> machine. In user space, that would just cause the kernel to kill the poor
> innocent victim. But when the kernel gets a #UD exception on what it
> expects to be a nop, it just won't work.

Sorry about the confusion. I was just learning git and applied a couple 
patches wrong that lead me to believe I had not found the answer, when in 
fact I had.  The problem is the multibyte NOP instructions introduced into 
the kernel in 2007.  For 2.6.24-rc1 reverting the patch (32c464f5) corrects 
the problem.  But then of course we cannot use these instructions as we would 
like.  I propose the following patch which adds a boot-time parameter to not 
use these multibyte NOPs in systems that are unprepared to deal with them.  
The patch disables a optimization used in one place by commenting out some 
lines in nop.h.  Please comment.

    x86: Fix to multibyte NOPs breaking Virtual PC (win)

    To be applied to linux-2.6.27-rc5.

    Signed-off-by: David Sanders <linux@sandersweb.net>

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 2763cb3..6a63e78 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -58,6 +58,15 @@ static int __init setup_noreplace_paravirt(char *str)
 __setup("noreplace-paravirt", setup_noreplace_paravirt);
 #endif
 
+static int intel_legacy_nops = 0;
+
+static int __init setup_intel_legacy_nops(char *str)
+{
+	intel_legacy_nops = 1;
+	return 1;
+}
+__setup("intel-legacy-nops", setup_intel_legacy_nops);
+
 #define DPRINTK(fmt, args...) if (debug_alternative) \
 	printk(KERN_DEBUG fmt, args)
 
@@ -165,13 +174,14 @@ static const struct nop {
 const unsigned char *const *find_nop_table(void)
 {
 	const unsigned char *const *noptable = intel_nops;
-	int i;
+	int i = 0;
 
-	for (i = 0; noptypes[i].cpuid >= 0; i++) {
+	while (noptypes[i].cpuid >= 0 && !intel_legacy_nops) {
 		if (boot_cpu_has(noptypes[i].cpuid)) {
 			noptable = noptypes[i].noptable;
 			break;
 		}
+		i++;
 	}
 	return noptable;
 }
diff --git a/include/asm-x86/nops.h b/include/asm-x86/nops.h
index ad0bedd..a6f0571 100644
--- a/include/asm-x86/nops.h
+++ b/include/asm-x86/nops.h
@@ -84,7 +84,7 @@
 #define ASM_NOP6 K7_NOP6
 #define ASM_NOP7 K7_NOP7
 #define ASM_NOP8 K7_NOP8
-#elif defined(CONFIG_X86_P6_NOP)
+/* #elif defined(CONFIG_X86_P6_NOP)
 #define ASM_NOP1 P6_NOP1
 #define ASM_NOP2 P6_NOP2
 #define ASM_NOP3 P6_NOP3
@@ -93,6 +93,7 @@
 #define ASM_NOP6 P6_NOP6
 #define ASM_NOP7 P6_NOP7
 #define ASM_NOP8 P6_NOP8
+*/
 #elif defined(CONFIG_X86_64)
 #define ASM_NOP1 K8_NOP1
 #define ASM_NOP2 K8_NOP2

Re: [BUG] x86 kenel won't boot under Virtual PC

[Jan Beulich]

>--- a/include/asm-x86/nops.h
>+++ b/include/asm-x86/nops.h
>@@ -84,7 +84,7 @@
> #define ASM_NOP6 K7_NOP6
> #define ASM_NOP7 K7_NOP7
> #define ASM_NOP8 K7_NOP8
>-#elif defined(CONFIG_X86_P6_NOP)
>+/* #elif defined(CONFIG_X86_P6_NOP)
> #define ASM_NOP1 P6_NOP1
> #define ASM_NOP2 P6_NOP2
> #define ASM_NOP3 P6_NOP3
>@@ -93,6 +93,7 @@
> #define ASM_NOP6 P6_NOP6
> #define ASM_NOP7 P6_NOP7
> #define ASM_NOP8 P6_NOP8
>+*/
> #elif defined(CONFIG_X86_64)
> #define ASM_NOP1 K8_NOP1
> #define ASM_NOP2 K8_NOP2

I disagree here: If I configure a 686+ kernel, I expect these NOPs to be
that way (and to work). If you want to run on something that's not
compliant, you just shouldn't configure your kernel that way.

Jan

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Fri, 5 Sep 2008, David Sanders wrote:
>
> The patch disables a optimization used in one place by commenting out some 
> lines in nop.h.  Please comment.

Hmm.. I'm not a huge fan of the ASM_NOP mess, but you also disable it for 
64-bit x86 too.

On 32-bit, at least the generic nops are fairly reasonable, but the 
default nops for 64-bit really look pretty sad, and the P6 nops really do 
look better.

So I would suggest perhaps moving the static P6 nop selection into the 
CONFIG_X86_64 thing.

The alternative is to just get rid of that static nop selection, and just 
have two cases: 32-bit and 64-bit, and just pick obviously safe cases for 
them. 

So I think that particular part would be better off with changing the 
Kconfig language instead. Ie something like this..

(I removed the 32-bit CPU's from the choices, except for MPENTIUM4 that 
really should be merged with MPSC - the difference between MPENTIUM4 and 
MPSC seems to be just a totally bogus 32-bit vs 64-bit thing. Yes, yes, 
there are PENTIUM4's without 64-bit, but there are also Prescott chips 
that run 32-bit kernels, so the thing is a bit confused.

		Linus

---
 arch/x86/Kconfig.cpu |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu
index 2c518fb..b225219 100644
--- a/arch/x86/Kconfig.cpu
+++ b/arch/x86/Kconfig.cpu
@@ -382,14 +382,17 @@ config X86_OOSTORE
 # P6_NOPs are a relatively minor optimization that require a family >=
 # 6 processor, except that it is broken on certain VIA chips.
 # Furthermore, AMD chips prefer a totally different sequence of NOPs
-# (which work on all CPUs).  As a result, disallow these if we're
-# compiling X86_GENERIC but not X86_64 (these NOPs do work on all
-# x86-64 capable chips); the list of processors in the right-hand clause
-# are the cores that benefit from this optimization.
+# (which work on all CPUs).  In addition, it looks like Virtual PC
+# does not understand them.
+#
+# As a result, disallow these if we're not compiling for X86_64 (these
+# NOPs do work on all x86-64 capable chips); the list of processors in
+# the right-hand clause are the cores that benefit from this optimization.
 #
 config X86_P6_NOP
 	def_bool y
-	depends on (X86_64 || !X86_GENERIC) && (M686 || MPENTIUMII || MPENTIUMIII || MPENTIUMM || MCORE2 || MPENTIUM4 || MPSC)
+	depends on X86_64
+	depends on (MCORE2 || MPENTIUM4 || MPSC)
 
 config X86_TSC
 	def_bool y

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Fri, 5 Sep 2008, Jan Beulich wrote:
> 
> I disagree here: If I configure a 686+ kernel, I expect these NOPs to be
> that way (and to work). If you want to run on something that's not
> compliant, you just shouldn't configure your kernel that way.

Well, if you actually do a

	git grep 'ASM_NOP[0-9]'

you'll find that just the _definitions_ of those things are the bulk of it 
BY FAR, and that there doesn't seem to be a single user that cares even 
remotely about performance.

So I actually think that the whole thing is a waste of time. We should 
probably 

 - pick a single set of NOP's per 32-bit/64-bit (since the good nops in 
   32-bit aren't 64-bit instructions at all, so we do want different nops 
   depending on _that_)

   The whole static choice by microarchitecture is pure garbage.

 - Probably also just declare that those default nops are single 
   instructions, just so that we never even have to think about it from a 
   dynamic replacement angle.

   Look at the uses again, and realize that it really is just pure garbage 
   to have this kind of complex and subtle stuff going on.

 - Move the optimized nop definitions (K7_NOPx etc) to the only place that 
   cares - asm/x86/kernel/alternative.c. When we do things _dynamically_, 
   it can actually make sense to pick a nop more precisely, but for this 
   whole static thing it's just a pain.

IOW, if it actually _worked_ reasonably, I wouldn't care. But clearly it 
doesn't. And once it's not working reasonably, it should be fixed.

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

On Fri, Sep 05, 2008 at 09:30:14AM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 5 Sep 2008, David Sanders wrote:
> >
> > The patch disables a optimization used in one place by commenting out some 
> > lines in nop.h.  Please comment.
> 
> Hmm.. I'm not a huge fan of the ASM_NOP mess, but you also disable it for 
> 64-bit x86 too.

Heh -- I originally only added it because you're asked for it ...

Yes going back to some simple generic nops and avoid all these problems
would be a great thing.

-Andi

Re: [BUG] x86 kenel won't boot under Virtual PC

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

>  - Move the optimized nop definitions (K7_NOPx etc) to the only place that 
>    cares - asm/x86/kernel/alternative.c. When we do things _dynamically_, 
>    it can actually make sense to pick a nop more precisely, but for this 
>    whole static thing it's just a pain.
> 
> IOW, if it actually _worked_ reasonably, I wouldn't care. But clearly 
> it doesn't. And once it's not working reasonably, it should be fixed.

yes - we had 3-4 tries already and while it looked worthwile initially 
it's clearly not showing signs of getting more robust and whatever 
benefits there might be slightly better NOPs is dwarved by all these 
robustness problems. Peter?

	Ingo

Re: [BUG] x86 kenel won't boot under Virtual PC

[Jeremy Fitzhardinge]

Linus Torvalds wrote:
> On Fri, 5 Sep 2008, Jan Beulich wrote:
>   
>> I disagree here: If I configure a 686+ kernel, I expect these NOPs to be
>> that way (and to work). If you want to run on something that's not
>> compliant, you just shouldn't configure your kernel that way.
>>     
>
> Well, if you actually do a
>
> 	git grep 'ASM_NOP[0-9]'
>
> you'll find that just the _definitions_ of those things are the bulk of it 
> BY FAR, and that there doesn't seem to be a single user that cares even 
> remotely about performance.
>   

Well, the paravirt_ops patching uses multibyte nops to pad out the
unused space in a patch site, and they're generally on hot paths
(otherwise we wouldn't bother with patching).  But even then I don't
think the particular nop chosen matters all that much, and even if it
did using the dumb redundant prefix long nops seems to be as good as or
better than the p6 nops.  The "call mcount" patching ftrace wants to do
would also be pretty common.

> So I actually think that the whole thing is a waste of time. We should 
> probably 
>
>  - pick a single set of NOP's per 32-bit/64-bit (since the good nops in 
>    32-bit aren't 64-bit instructions at all, so we do want different nops 
>    depending on _that_)
>
>    The whole static choice by microarchitecture is pure garbage.
>
>  - Probably also just declare that those default nops are single 
>    instructions, just so that we never even have to think about it from a 
>    dynamic replacement angle.
>   

Yes, that would be a good idea.

>  - Move the optimized nop definitions (K7_NOPx etc) to the only place that 
>    cares - asm/x86/kernel/alternative.c. When we do things _dynamically_, 
>    it can actually make sense to pick a nop more precisely, but for this 
>    whole static thing it's just a pain.
>   

Yep.  We could run the p6 nops with an exception handler to see if the
cpu actually supports them or not.  And if not, just fall back to
something simple and good enough.

    J

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Ingo Molnar wrote:
> * Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
>>  - Move the optimized nop definitions (K7_NOPx etc) to the only place that 
>>    cares - asm/x86/kernel/alternative.c. When we do things _dynamically_, 
>>    it can actually make sense to pick a nop more precisely, but for this 
>>    whole static thing it's just a pain.
>>
>> IOW, if it actually _worked_ reasonably, I wouldn't care. But clearly 
>> it doesn't. And once it's not working reasonably, it should be fixed.
> 
> yes - we had 3-4 tries already and while it looked worthwile initially 
> it's clearly not showing signs of getting more robust and whatever 
> benefits there might be slightly better NOPs is dwarved by all these 
> robustness problems. Peter?

There is already a fix for this based on a first-principles test in tip.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Jan Beulich wrote:
> 
> I disagree here: If I configure a 686+ kernel, I expect these NOPs to be
> that way (and to work). If you want to run on something that's not
> compliant, you just shouldn't configure your kernel that way.
> 

Okay, we do not generate P6 NOPs statically if CONFIG_X86_GENERIC_CPU is 
enabled -- unless, of course gcc/binutils generates them, which they now 
do for certain CPU types.  If you are selecting a specific CPU *and* you 
don't select GENERIC, you should get valid code for the selected CPU only.

As far as using them dynamically, there is code in -tip that determines 
dynamically and explicitly if the long NOPs are available, and only uses 
them if they are.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Friday 05 September 2008 19:31, H. Peter Anvin wrote:
> David Sanders wrote:
> > I recently discovered that x86 kernels won't boot under Virtual PC.  In
> > this case paravirt was not built into the kernel.  The kernel just
> > "hangs" on attempted boot with no error messages.  Git-bisect pinpointed
> > the following commit as the problem:
>
> Hi David,
>
> Could you please confirm if the tip kernel solves the problem?
>
> Either that or apply this patch series to mainline; I'd like to promote
> this to urgent and send it to Linus for inclusion in 2.6.27, but I need
> verification as quickly as possible.
>
> 	-hpa
Peter, sorry about delay in replying I was out of town for weekend.  I will 
apply your patch and git back to you on results.  A quick look a the patches 
indicates that nop.h and the users of nop.h are not addressed so I may have 
to combine your patches with Linus's Kconfig patch to get a working kernel.
David

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Friday 05 September 2008 19:31, H. Peter Anvin wrote:
> David Sanders wrote:
> > I recently discovered that x86 kernels won't boot under Virtual PC.  In
>
> Could you please confirm if the tip kernel solves the problem?
>

Yes the patch you supplied (and now in Linus's tree) fixes (part) of the problem.
In order to get a working kernel I also need a patch for the nops.h issue.  
I have tested and confirmed that the patch posted by Linus works and finally 
solves the problem with virtual PC.

Linus please apply your patch (which I quote below).
---
 arch/x86/Kconfig.cpu |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu
index 2c518fb..b225219 100644
--- a/arch/x86/Kconfig.cpu
+++ b/arch/x86/Kconfig.cpu
@@ -382,14 +382,17 @@ config X86_OOSTORE
 # P6_NOPs are a relatively minor optimization that require a family >=
 # 6 processor, except that it is broken on certain VIA chips.
 # Furthermore, AMD chips prefer a totally different sequence of NOPs
-# (which work on all CPUs).  As a result, disallow these if we're
-# compiling X86_GENERIC but not X86_64 (these NOPs do work on all
-# x86-64 capable chips); the list of processors in the right-hand clause
-# are the cores that benefit from this optimization.
+# (which work on all CPUs).  In addition, it looks like Virtual PC
+# does not understand them.
+#
+# As a result, disallow these if we're not compiling for X86_64 (these
+# NOPs do work on all x86-64 capable chips); the list of processors in
+# the right-hand clause are the cores that benefit from this optimization.
 #
 config X86_P6_NOP
 	def_bool y
-	depends on (X86_64 || !X86_GENERIC) && (M686 || MPENTIUMII || MPENTIUMIII || MPENTIUMM || MCORE2 || MPENTIUM4 || MPSC)
+	depends on X86_64
+	depends on (MCORE2 || MPENTIUM4 || MPSC)
 
 config X86_TSC
 	def_bool y

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

David Sanders wrote:
> 
> Yes the patch you supplied (and now in Linus's tree) fixes (part) of the problem.
> In order to get a working kernel I also need a patch for the nops.h issue.  
> I have tested and confirmed that the patch posted by Linus works and finally 
> solves the problem with virtual PC.
> 
> Linus please apply your patch (which I quote below).

No, please don't.  Instead, David, please enable CONFIG_X86_GENERIC.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Sunday 07 September 2008 21:48, H. Peter Anvin wrote:
> David Sanders wrote:
> > Yes the patch you supplied (and now in Linus's tree) fixes (part) of the
> > problem. In order to get a working kernel I also need a patch for the
> > nops.h issue. I have tested and confirmed that the patch posted by Linus
> > works and finally solves the problem with virtual PC.
> >
> > Linus please apply your patch (which I quote below).
>
> No, please don't.  Instead, David, please enable CONFIG_X86_GENERIC.
>


I checked the distribution I'm using (debian) and the kernel shipped with it 
does not have  CONFIG_X86_GENERIC set.  This means _when_ they ship a 2.6.27 
kernel it won't work with Virtual PC so I won't be able to even install it.
However, with Linus's patch I am guaranteed to work for all kernels built for 
X86_32 and even for X86_64 because in that case the virtual environment will 
support the multibyte NOPs.  I need Linus's fix in order to support the 
Linux-using virtual pc community.  I don't have the resources to lobby each 
individual distribution about their kernel config.  I want it to just work.  
        David

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

David Sanders wrote:
> I checked the distribution I'm using (debian) and the kernel shipped with it 
> does not have  CONFIG_X86_GENERIC set.  This means _when_ they ship a 2.6.27 
> kernel it won't work with Virtual PC so I won't be able to even install it.

Then please file a bug report with Debian.

> However, with Linus's patch I am guaranteed to work for all kernels built for 
> X86_32 and even for X86_64 because in that case the virtual environment will 
> support the multibyte NOPs.  I need Linus's fix in order to support the 
> Linux-using virtual pc community.  I don't have the resources to lobby each 
> individual distribution about their kernel config.  I want it to just work.  

Under that logic we shouldn't even have CPU configurables, since you 
want it to "just work" whatever crap you're running on.  That is EXACTLY 
what CONFIG_X86_GENERIC means, and the fact that any particular 
distribution is broken with respect to not enabling it is a bug in that 
distribution, and not grounds for breaking the upstream kernel.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

> Under that logic we shouldn't even have CPU configurables, since you 

Just for optimization.

> want it to "just work" whatever crap you're running on.  That is EXACTLY 
> what CONFIG_X86_GENERIC means, and the fact that any particular 

At least when I introduced X86_GENERIC it was just an optimization,
not a requirement.

That is the kernel pretty much always did "just work"
(with only a very few exceptions like PAE vs non PAE) on all 
CPUs. The CPU configs also just specified optimizations, not correctness.
The code for all CPUs used to be always there.
X86_GENERIC was mostly just to do things like always use 
the largest cache alignment.

I think someone changed that recently, but imho that wasn't
an improvement. As you can see it just causes endless support
problems.

> distribution is broken with respect to not enabling it is a bug in that 
> distribution, and not grounds for breaking the upstream kernel.

Well it's more like that you guys changed the semantics 
without warnings the distributions. I'm not sure you can blame
Debian for that.

-Andi
-- 
ak@linux.intel.com

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Monday 08 September 2008 00:04, H. Peter Anvin wrote:
> David Sanders wrote:
> > I checked the distribution I'm using (debian) and the kernel shipped with
> > it does not have  CONFIG_X86_GENERIC set.  This means _when_ they ship a
> > 2.6.27 kernel it won't work with Virtual PC so I won't be able to even
> > install it.
>
> Then please file a bug report with Debian.
Well, OK I could do that.  But I can't bug every distribution in existence for 
the same thing.
>
> > However, with Linus's patch I am guaranteed to work for all kernels built
> > for X86_32 and even for X86_64 because in that case the virtual
> > environment will support the multibyte NOPs.  I need Linus's fix in order
> > to support the Linux-using virtual pc community.  I don't have the
> > resources to lobby each individual distribution about their kernel
> > config.  I want it to just work.
>
> Under that logic we shouldn't even have CPU configurables, since you
> want it to "just work" whatever crap you're running on.  That is EXACTLY
> what CONFIG_X86_GENERIC means, and the fact that any particular
> distribution is broken with respect to not enabling it is a bug in that
> distribution, and not grounds for breaking the upstream kernel.
>
> 	-hpa
I don't see that Linus's patch breaks the upstream kernel.  Just the opposite, 
you go and determine in alternative.c that the processor doesn't support NOPL 
and then go ahead and use it anyway in nops.h.  That makes no sense to me.

Could we use the result of find_nop_table() instead of nops.h?

David

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Sun, 7 Sep 2008, H. Peter Anvin wrote:
> 
> Under that logic we shouldn't even have CPU configurables, since you want it
> to "just work" whatever crap you're running on.  That is EXACTLY what
> CONFIG_X86_GENERIC means

I dunno.. Event he help-text doesn't actually agree with that:

  config X86_GENERIC
        bool "Generic x86 support"
        depends on X86_32
        help
          Instead of just including optimizations for the selected
          x86 variant (e.g. PII, Crusoe or Athlon), include some more
          generic optimizations as well. This will make the kernel
          perform better on x86 CPUs other than that selected.
          
          This is really intended for distributors who need more
          generic optimizations.

Also, quite frankly, while the CPU processor type message says

          The kernel will not necessarily run on earlier architectures than
          the one you have chosen, e.g. a Pentium optimized kernel will run on
          a PPro, but not necessarily on a i486.

I thought you agreed that CPU virtualization can be a problem? That was 
the whole excuse for why the dynamic code was changed. Why would it not be 
true for the static code?

The fact is, if you want to run on a Core2 or other modern CPU, then 
"Virtual PC" is apparently buggy in this respect. You worked around it for 
the dynamic choice - but that's totally _pointless_ if you then don't want 
to work around it for the static one.

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Sun, 7 Sep 2008, H. Peter Anvin wrote:
> > 
> > Under that logic we shouldn't even have CPU configurables, since you want it
> > to "just work" whatever crap you're running on.  That is EXACTLY what
> > CONFIG_X86_GENERIC means
> 
> I dunno.. Event he help-text doesn't actually agree with that:
> 
>   config X86_GENERIC
>         bool "Generic x86 support"
>         depends on X86_32
>         help
>           Instead of just including optimizations for the selected
>           x86 variant (e.g. PII, Crusoe or Athlon), include some more
>           generic optimizations as well. This will make the kernel
>           perform better on x86 CPUs other than that selected.
>           
>           This is really intended for distributors who need more
>           generic optimizations.
> 
> Also, quite frankly, while the CPU processor type message says
> 
>           The kernel will not necessarily run on earlier architectures than
>           the one you have chosen, e.g. a Pentium optimized kernel will run on
>           a PPro, but not necessarily on a i486.
> 
> I thought you agreed that CPU virtualization can be a problem? That 
> was the whole excuse for why the dynamic code was changed. Why would 
> it not be true for the static code?
> 
> The fact is, if you want to run on a Core2 or other modern CPU, then 
> "Virtual PC" is apparently buggy in this respect. You worked around it 
> for the dynamic choice - but that's totally _pointless_ if you then 
> don't want to work around it for the static one.

yes. X86_P6_NOPS is a totally insignificant optimization and if it makes 
_any_ CPU not boot (be that virtual or real), then it's frankly not 
worth it.

David, exactly how does the kernel fail to boot with latest -git? 
(v2.6.27-rc5-313-g64f996f or later) Does detect_nopl() run? It really 
should, and it should detect the non-working instructions.

	Ingo

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Ingo Molnar wrote:
> 
> yes. X86_P6_NOPS is a totally insignificant optimization and if it makes 
> _any_ CPU not boot (be that virtual or real), then it's frankly not 
> worth it.
> 
> David, exactly how does the kernel fail to boot with latest -git? 
> (v2.6.27-rc5-313-g64f996f or later) Does detect_nopl() run? It really 
> should, and it should detect the non-working instructions.
> 

Okay, a few things here...

1. First, I wrote up a patch yesterday to update the CONFIG_X86_GENERIC 
description and to make it "default y".  It is currently on 
x86/defconfig, but I think it should be promoted to mainline immediately.

2. X86_P6_NOPS is not the only source of static NOPLs.  If gcc is set to 
optimize for specific architectures, then gcc/binutils will generate 
static NOPLs.  The only way we can prevent that is by not using specific 
-march options, as far as I can tell.

3. I'm not positive that CONFIG_X86_GENERIC currently avoid all cases of 
(2), but it obviously should.  I will verify that today and add a 
followup patch to the Makefiles if necessary.

Given all of this, I really think that putting this on 
CONFIG_X86_GENERIC, *AND* making CONFIG_X86_GENERIC the default is the 
right choice.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Monday 08 September 2008 11:23, Ingo Molnar wrote:
> * Linus Torvalds <torvalds@linux-foundation.org> wrote:
> > On Sun, 7 Sep 2008, H. Peter Anvin wrote:
> > > Under that logic we shouldn't even have CPU configurables, since you
> > > want it to "just work" whatever crap you're running on.  That is
> > > EXACTLY what CONFIG_X86_GENERIC means
> >
> > I dunno.. Event he help-text doesn't actually agree with that:
> >
> >   config X86_GENERIC
> >         bool "Generic x86 support"
> >         depends on X86_32
> >         help
> >           Instead of just including optimizations for the selected
> >           x86 variant (e.g. PII, Crusoe or Athlon), include some more
> >           generic optimizations as well. This will make the kernel
> >           perform better on x86 CPUs other than that selected.
> >
> >           This is really intended for distributors who need more
> >           generic optimizations.
> >
> > Also, quite frankly, while the CPU processor type message says
> >
> >           The kernel will not necessarily run on earlier architectures
> > than the one you have chosen, e.g. a Pentium optimized kernel will run on
> > a PPro, but not necessarily on a i486.
> >
> > I thought you agreed that CPU virtualization can be a problem? That
> > was the whole excuse for why the dynamic code was changed. Why would
> > it not be true for the static code?
> >
> > The fact is, if you want to run on a Core2 or other modern CPU, then
> > "Virtual PC" is apparently buggy in this respect. You worked around it
> > for the dynamic choice - but that's totally _pointless_ if you then
> > don't want to work around it for the static one.
>
> yes. X86_P6_NOPS is a totally insignificant optimization and if it makes
> _any_ CPU not boot (be that virtual or real), then it's frankly not
> worth it.
>
> David, exactly how does the kernel fail to boot with latest -git?
> (v2.6.27-rc5-313-g64f996f or later) Does detect_nopl() run? It really
> should, and it should detect the non-working instructions.
>

Ingo,
With CONFIG_X86_GENERIC=y, the latest v2.6.27 in Linus's tree boots fine.  But 
if you don't select that option (and some distributions don't) it won't boot 
at all.  It just hangs (blank screen) with no error messages and nothing in 
dmesg.  I assume it is hitting one of the ASM_NOP? instructions.
David

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Ingo Molnar wrote:
>>
>> I thought you agreed that CPU virtualization can be a problem? That 
>> was the whole excuse for why the dynamic code was changed. Why would 
>> it not be true for the static code?
>>
>> The fact is, if you want to run on a Core2 or other modern CPU, then 
>> "Virtual PC" is apparently buggy in this respect. You worked around it 
>> for the dynamic choice - but that's totally _pointless_ if you then 
>> don't want to work around it for the static one.
> 
> yes. X86_P6_NOPS is a totally insignificant optimization and if it makes 
> _any_ CPU not boot (be that virtual or real), then it's frankly not 
> worth it.
> 
> David, exactly how does the kernel fail to boot with latest -git? 
> (v2.6.27-rc5-313-g64f996f or later) Does detect_nopl() run? It really 
> should, and it should detect the non-working instructions.
> 

It almost certainly never gets as far as detect_nopl().

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Andi Kleen wrote:
> On Mon, Sep 08, 2008 at 08:09:48AM -0700, Linus Torvalds wrote:
>> On Sun, 7 Sep 2008, H. Peter Anvin wrote:
>>> Under that logic we shouldn't even have CPU configurables, since you want it
>>> to "just work" whatever crap you're running on.  That is EXACTLY what
>>> CONFIG_X86_GENERIC means
>> I dunno.. Event he help-text doesn't actually agree with that:
> 
> The help text matches in how I wrote this option originally.
> The original use case was the 128 byte P4 cache lines.
> 

The help text is indeed out of date.  I did a patch yesterday to, among 
other things, update it; I also want to verify that we are disabling all 
options that can cause gcc or binutils to generate nopl's; I plan to 
push it today.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

On Mon, Sep 08, 2008 at 08:09:48AM -0700, Linus Torvalds wrote:
> On Sun, 7 Sep 2008, H. Peter Anvin wrote:
> > 
> > Under that logic we shouldn't even have CPU configurables, since you want it
> > to "just work" whatever crap you're running on.  That is EXACTLY what
> > CONFIG_X86_GENERIC means
> 
> I dunno.. Event he help-text doesn't actually agree with that:

The help text matches in how I wrote this option originally.
The original use case was the 128 byte P4 cache lines.

-Andi

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

H. Peter Anvin wrote:
> 
> The help text is indeed out of date.  I did a patch yesterday to, among 
> other things, update it; I also want to verify that we are disabling all 
> options that can cause gcc or binutils to generate nopl's; I plan to 
> push it today.
> 

OK, turns out that is already happending, by virtue of:

# add at the end to overwrite eventual tuning options from earlier
# cpu entries
cflags-$(CONFIG_X86_GENERIC)    += $(call tune,generic,$(call tune,i686))

-mtune=generic on 32-bit CPUs disable NOPL generation, and any 
gcc/binutils combo too old to have -mtune=generic won't generate them at 
all.

I think I verified this some time in the past, but I just had to refresh 
my memory.  This was a major reason to put this functionality on 
CONFIG_X86_GENERIC.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

> The help text is indeed out of date.  I did a patch yesterday to, among 
> other things, update it; I also want to verify that we are disabling all 
> options that can cause gcc or binutils to generate nopl's; I plan to 
> push it today.

You (or whoever did those changes) likely broke a lot of distribution setups 
subtly then. Hopefully the changes were worth that.

-Andi
-- 
ak@linux.intel.com

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Andi Kleen wrote:
>> The help text is indeed out of date.  I did a patch yesterday to, among 
>> other things, update it; I also want to verify that we are disabling all 
>> options that can cause gcc or binutils to generate nopl's; I plan to 
>> push it today.
> 
> You (or whoever did those changes) likely broke a lot of distribution setups 
> subtly then. Hopefully the changes were worth that.

Likely broke a lot of distribution setups how?

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Andi Kleen wrote:
> On Mon, Sep 08, 2008 at 08:50:46AM -0700, H. Peter Anvin wrote:
>> Andi Kleen wrote:
>>>> The help text is indeed out of date.  I did a patch yesterday to, among 
>>>> other things, update it; I also want to verify that we are disabling all 
>>>> options that can cause gcc or binutils to generate nopl's; I plan to 
>>>> push it today.
>>> You (or whoever did those changes) likely broke a lot of distribution 
>>> setups subtly then. Hopefully the changes were worth that.
>> Likely broke a lot of distribution setups how?
> 
> Previously they could set CPU x and it would still run on other CPUs
> even if that option was not set. If that's not the case anymore then
> they will have some unhappy users once they update their kernels.
> 

That was broken long before by the gcc and binutils authors.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

On Mon, Sep 08, 2008 at 08:50:46AM -0700, H. Peter Anvin wrote:
> Andi Kleen wrote:
> >>The help text is indeed out of date.  I did a patch yesterday to, among 
> >>other things, update it; I also want to verify that we are disabling all 
> >>options that can cause gcc or binutils to generate nopl's; I plan to 
> >>push it today.
> >
> >You (or whoever did those changes) likely broke a lot of distribution 
> >setups subtly then. Hopefully the changes were worth that.
> 
> Likely broke a lot of distribution setups how?

Previously they could set CPU x and it would still run on other CPUs
even if that option was not set. If that's not the case anymore then
they will have some unhappy users once they update their kernels.

-Andi
-- 
ak@linux.intel.com

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:
> 
> The help text is indeed out of date.  I did a patch yesterday to, among other
> things, update it; I also want to verify that we are disabling all options
> that can cause gcc or binutils to generate nopl's; I plan to push it today.

Peter.

The help text may be out of date because of changes to NOPL usage, but you 
should ask yourself whether the change is actually a _good_ change.

IOW, I really don't see why you are pushing changing the help-text, 
instead of just making the kernel work better.

The fact that some broken gcc/binutils versions may screw us over _anyway_ 
may well mean that we should just push back on _that_ change instead.

Quite frankly, from a user perspective, even a very _technical_ one, 
please tell me what the advantage of not being fairly generic by default 
is. Really.

Yes, there are some _big_ ISA issues where it is worth doing real static 
code selection (as opposed to just instruction selection and scheduling 
etc that still _works_ for everybody, but optimizes for certain 
archtiectures).

So things like cmpxchg/xadd (for atomics) and cmov (for compiler-generated 
code), and bswap (for networking) can really make a big difference, and 
are not really realistic to do dynamically. 

But NOPL? That's simply not _worth_ it being painful over.

And the fact is, the current help text describes

 (a) the historical meaning (optimize for a specific architecture, but 
     don't make extreme choices that are bad for others)
 (b) what people would generally _want_.

and I really don't think that changing the help text is the right solution 
here. It may be "technically correct", but it is simply not user-friendly 
or smart.

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> and I really don't think that changing the help text is the right solution 
> here. It may be "technically correct", but it is simply not user-friendly 
> or smart.
> 

The issue at hand is that at least with the current toolchain, we need 
to pass -march=generic in order for these instructions to be generated. 
  We have an option for this, CONFIG_X86_GENERIC, which distributions 
really *should* be using anyway.

And yes, it should be the default.  The patch I have makes it
"default y" as well as change the help text.

Would it make you happier if this option was forced enabled unless 
CONFIG_EMBEDDED was on?

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

H. Peter Anvin wrote:
> 
> The issue at hand is that at least with the current toolchain, we need 
> to pass -march=generic in order for these instructions to be generated. 
>  We have an option for this, CONFIG_X86_GENERIC, which distributions 
> really *should* be using anyway.
> 

That should have been "-mtune=generic".  Sorry.

> And yes, it should be the default.  The patch I have makes it
> "default y" as well as change the help text.
> 
> Would it make you happier if this option was forced enabled unless 
> CONFIG_EMBEDDED was on?

I guess the other option is to create a new option for selecting the 
dangerous -mtune= variants, and possibly lock *that* option to 
CONFIG_EMBEDDED.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:
> 
> And yes, it should be the default.  The patch I have makes it
> "default y" as well as change the help text.

It sounds like it shouldn't be a default at all, it should just _always_ 
be on, if there really are gcc's that care that much. Most of our 
optimizations have historically really been about _optimizing_, not about 
"it won't work", even if we have had exceptions (but as mentioned, I think 
those exceptions have been way more imporant than NOPL).

> Would it make you happier if this option was forced enabled unless
> CONFIG_EMBEDDED was on?

Yes, putting it behind EMBEDDED will certainly fix the issue. Anybody who 
actually enables EMBEDDED and does all his choices by hand should no 
longer expect to not have to know _exactly_ what he is doing.

So if it's behind EMBEDDED, and defaults to "on", then I have no problem 
with changing the help text to say "If you do this, we'll statically do 
things that really _require_ you to have a CPU that looks _exactly_ like 
the CPU you claimed".

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[David Sanders]

On Monday 08 September 2008 12:15, H. Peter Anvin wrote:
> H. Peter Anvin wrote:
> > The issue at hand is that at least with the current toolchain, we need
> > to pass -march=generic in order for these instructions to be generated.
> >  We have an option for this, CONFIG_X86_GENERIC, which distributions
> > really *should* be using anyway.
>
> That should have been "-mtune=generic".  Sorry.
>
> > And yes, it should be the default.  The patch I have makes it
> > "default y" as well as change the help text.
> >
> > Would it make you happier if this option was forced enabled unless
> > CONFIG_EMBEDDED was on?
>
> I guess the other option is to create a new option for selecting the
> dangerous -mtune= variants, and possibly lock *that* option to
> CONFIG_EMBEDDED.
>
> 	-hpa

The option that works best for users is for the kernel to not use NOPLs on 
32-bit cpu's.  That is what Linus's patch accomplishes.  I am not having any 
issues with gcc or gas generating NOPLs.  The kernel should work out of the 
box for PentiumPro or later cpu's running in 32-bit mode.  I think you are 
trying to over-engineer the issue.  Just apply the patch and move on.
David

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> So if it's behind EMBEDDED, and defaults to "on", then I have no problem 
> with changing the help text to say "If you do this, we'll statically do 
> things that really _require_ you to have a CPU that looks _exactly_ like 
> the CPU you claimed".
> 

Sounds like a plan.  I'll have a patch shortly.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[david]

On Mon, 8 Sep 2008, Linus Torvalds wrote:

> On Mon, 8 Sep 2008, H. Peter Anvin wrote:
>>
>> And yes, it should be the default.  The patch I have makes it
>> "default y" as well as change the help text.
>
> It sounds like it shouldn't be a default at all, it should just _always_
> be on, if there really are gcc's that care that much. Most of our
> optimizations have historically really been about _optimizing_, not about
> "it won't work", even if we have had exceptions (but as mentioned, I think
> those exceptions have been way more imporant than NOPL).
>
>> Would it make you happier if this option was forced enabled unless
>> CONFIG_EMBEDDED was on?
>
> Yes, putting it behind EMBEDDED will certainly fix the issue. Anybody who
> actually enables EMBEDDED and does all his choices by hand should no
> longer expect to not have to know _exactly_ what he is doing.
>
> So if it's behind EMBEDDED, and defaults to "on", then I have no problem
> with changing the help text to say "If you do this, we'll statically do
> things that really _require_ you to have a CPU that looks _exactly_ like
> the CPU you claimed".

I always understood the CPU selection to be "this CPU and ones compatible 
with it will work, others won't" unless generic was enabled. the fact that 
only a few CPU's wouldn't work and the rest was optimization is true, but 
the details of what chips would and wouldn't work were never that clear. 
The difference between a kernel compiled for generic and once compiled for 
a specific CPU can be very significant. (I ran into 30% differences back 
in the 2.4 days between generic and Athlon) This is why all the distros 
don't enable the generic cpu option on their kernels nowdays. I'd hate to 
see all the distros enabling embedded just to get this performance boost

David Lang

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

david@lang.hm wrote:
> I'd hate to see all the distros enabling 
> embedded just to get this performance boost

It's about producing code that works generically, not a performance boost.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, david@lang.hm wrote:
> 
> I always understood the CPU selection to be "this CPU and ones compatible with
> it will work, others won't" unless generic was enabled.

No.

Read the help text.. 

Yes, we care about features that MATTER. But if compiles start using 
features that don't really matter, and make a specific kernel _too_ 
specific, then we need to reign in the madness.

IOW, it's a balance. On one hand, yes, the uarch makes sense. On the 
other, it's just stupid to have to worry about details that don't 
realistically make any difference at all - except whether the machine 
works or not.

And yes, we could just put this up as a Virtual PC bug. It clearly is. But 
in the end, it _still_ all boils down to a balance between "do we actually 
win anything by using NOPL statically" vs "do we lose anything by being 
too damn inconvenient".

		Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

On Mon, Sep 08, 2008 at 09:20:06AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 8 Sep 2008, H. Peter Anvin wrote:
> > 
> > And yes, it should be the default.  The patch I have makes it
> > "default y" as well as change the help text.
> 
> It sounds like it shouldn't be a default at all, it should just _always_ 
> be on, if there really are gcc's that care that much. Most of our 

Originally it was an option because the 128 byte cache line it selects
caused bloat in several important data structures. That was back
then when cache line padded NR_CPUs arrays were still pretty common.
I don't know if it's still a problem, but before making it default y
it would be good to check the dynamic+static memory cost at least.

-Andi

Re: [BUG] x86 kenel won't boot under Virtual PC

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> Linus Torvalds wrote:
>>
>> So if it's behind EMBEDDED, and defaults to "on", then I have no 
>> problem with changing the help text to say "If you do this, we'll 
>> statically do things that really _require_ you to have a CPU that 
>> looks _exactly_ like the CPU you claimed".
>
> Sounds like a plan.  I'll have a patch shortly.

yep. I thought about changing the name and adding a default y but 
keeping the name and also putting it behind EMBEDDED would be perfect.

	Ingo

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, Andi Kleen wrote:
>
> Originally it was an option because the 128 byte cache line it selects
> caused bloat in several important data structures. That was back
> then when cache line padded NR_CPUs arrays were still pretty common.
> I don't know if it's still a problem, but before making it default y
> it would be good to check the dynamic+static memory cost at least.

Btw, I do think that the whole NOPL issue is separate from all the other 
issues. There can be _other_ cases where it really is worth doing some 
"generic" optimizations or being more "specific", and my argument really 
is that NOPL is _not_ one of those cases.

So I'm still not sure that X86_GENERIC is necessarily the answer. The 
answer may be:

 - never use NOPL statically unless we _know_ it works (eg x86-64)

 - never allow such a stupid decision by gcc as to use NOPL on x86-32.

..and then leave X86_GENERIC alone wrt everything else.

Peter - does gcc actually use NOPL in _32-bit_ code too? It really seems 
to be a stupid decision to make a binary not run on other CPU's over 
something as trivial as that. That's something I'd expect out of an Intel 
compiler just to mess with AMD, not out of gcc.

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> Peter - does gcc actually use NOPL in _32-bit_ code too? It really seems 
> to be a stupid decision to make a binary not run on other CPU's over 
> something as trivial as that. That's something I'd expect out of an Intel 
> compiler just to mess with AMD, not out of gcc.
> 

Yes, it does:

   /* We need to decide which NOP sequence to use for 32bit and
      64bit. When -mtune= is used:

      1. For PROCESSOR_I386, PROCESSOR_I486, PROCESSOR_PENTIUM and
      PROCESSOR_GENERIC32, f32_patt will be used.
      2. For PROCESSOR_PENTIUMPRO, PROCESSOR_PENTIUM4, PROCESSOR_NOCONA,
      PROCESSOR_CORE, PROCESSOR_CORE2, and PROCESSOR_GENERIC64,
      alt_long_patt will be used.
      3. For PROCESSOR_ATHLON, PROCESSOR_K6, PROCESSOR_K8 and
      PROCESSOR_AMDFAM10, alt_short_patt will be used.

      When -mtune= isn't used, alt_long_patt will be used if
      cpu_arch_isa_flags has Cpu686. Otherwise, f32_patt will
      be used.

"alt_long_patt" uses NOPL and its variants.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

H. Peter Anvin wrote:
> 
> Yes, it does:
> 
>   /* We need to decide which NOP sequence to use for 32bit and
>      64bit. When -mtune= is used:
> 
>      1. For PROCESSOR_I386, PROCESSOR_I486, PROCESSOR_PENTIUM and
>      PROCESSOR_GENERIC32, f32_patt will be used.
>      2. For PROCESSOR_PENTIUMPRO, PROCESSOR_PENTIUM4, PROCESSOR_NOCONA,
>      PROCESSOR_CORE, PROCESSOR_CORE2, and PROCESSOR_GENERIC64,
>      alt_long_patt will be used.
>      3. For PROCESSOR_ATHLON, PROCESSOR_K6, PROCESSOR_K8 and
>      PROCESSOR_AMDFAM10, alt_short_patt will be used.
> 
>      When -mtune= isn't used, alt_long_patt will be used if
>      cpu_arch_isa_flags has Cpu686. Otherwise, f32_patt will
>      be used.
> 
> "alt_long_patt" uses NOPL and its variants.
> 

I should point out that the code above is from binutils (gas), not from 
gcc.  It is entirely possible that doing something like
"-mtune=core2 -Wa,-mtune=generic" might actually work, but I have not 
dug through the gcc code proper to figure out if that (a) would work, 
and (b) will remain the case...

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[Andi Kleen]

> Btw, I do think that the whole NOPL issue is separate from all the other 
> issues. There can be _other_ cases where it really is worth doing some 
> "generic" optimizations or being more "specific", and my argument really 
> is that NOPL is _not_ one of those cases.
> 
> So I'm still not sure that X86_GENERIC is necessarily the answer. The 

FWIW I personally think Linux should always use very conservative nops. 
Unconditionally. This issue already cost far too much developer time
and it's unlikely to be worth it.

> Peter - does gcc actually use NOPL in _32-bit_ code too? It really seems 

gcc just uses .p2align, so it comes down to binutils

Yes there seems to be a pretty scary ISA selection code in there.

Line 565 of

http://www.google.com/codesearch?hl=en&q=i386_align_code+show:pxieOi43ptA:OXGjJbna8V8:zfd_fdPYgEI&sa=N&cd=1&ct=rc&cs_p=http://gentoo.osuosl.org/distfiles/binutils-2.17.50.0.13.tar.bz2&cs_f=binutils-2.17.50.0.13/gas/config/tc-i386.c

I haven't decoded it completely, but I suspect it does the wrong thing.

-Andi

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:
> Linus Torvalds wrote:
> > 
> > Peter - does gcc actually use NOPL in _32-bit_ code too? It really seems to
> > be a stupid decision to make a binary not run on other CPU's over something
> > as trivial as that. That's something I'd expect out of an Intel compiler
> > just to mess with AMD, not out of gcc.
> > 
> 
> Yes, it does:
> 
>   /* We need to decide which NOP sequence to use for 32bit and
>      64bit. When -mtune= is used:
> 
>      1. For PROCESSOR_I386, PROCESSOR_I486, PROCESSOR_PENTIUM and
>      PROCESSOR_GENERIC32, f32_patt will be used.
>      2. For PROCESSOR_PENTIUMPRO, PROCESSOR_PENTIUM4, PROCESSOR_NOCONA,
>      PROCESSOR_CORE, PROCESSOR_CORE2, and PROCESSOR_GENERIC64,
>      alt_long_patt will be used.
>      3. For PROCESSOR_ATHLON, PROCESSOR_K6, PROCESSOR_K8 and
>      PROCESSOR_AMDFAM10, alt_short_patt will be used.
> 
>      When -mtune= isn't used, alt_long_patt will be used if
>      cpu_arch_isa_flags has Cpu686. Otherwise, f32_patt will
>      be used.
> 
> "alt_long_patt" uses NOPL and its variants.

That is A TOTAL PIECE OF SH*T, and against gcc's own documentation.

"-mtune=x" is very much defined to be a performance _tuning_ option, not 
an "architectural" option. 

Quite frankly, this is a gcc bug. Plain and simple. 

Look at the gcc man-pages. It very much says:

       -mtune=cpu_type
           Set only the instruction scheduling parameters for machine type
           cpu_type.  The instruction set is not changed.

(in various different formats - it says the same things with different 
words for different architectures. For example, for 386/x86-64 it 
_specifically_ says:

       -mtune=cpu-type
           Tune to cpu-type everything applicable about the generated code,
           except for the ABI and the set of available instructions.  The
           choices for cpu-type are:

note the "except for the ABI and the set of available instructions".

Qutie frankly, I think this is a *much* bigger bug than any possible 
VirtualPC idiocy. It's expressly against the whole idea of "-mtune".

Of course, we do set "-march=i686" too, so maybe the gcc people will claim 
that that means "Intel CPU's only". But quite frankly, if they really 
claim that, then they are lying pond-scum:

           i686
               Same as "generic", but when used as "march" option, PentiumPro
               instruction set will be used, so the code will run on all i686
               family chips.

which any sane x86 person would claim includes all the various clone 
manufacturers too. Saying that NOPL is so important that "i686" should 
suddenly be Intel-only is dishonest and totally stupid.

IOW, somebody who has a gcc bugzilla login should just create a bug-report 
on this. There is no question but that this is a bug, plain and simple.

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:
> 
> I should point out that the code above is from binutils (gas), not from gcc.

Doesn't matter. If gcc passes "-mtune=i686" down to gas and thus the end 
result doesn't match gcc's own documentation (and whole point of -mtune), 
then it's still a gcc bug, even if it may not be a "cc1" bug. The fact 
that "cc1" and "gas" are two different phases is pretty much irrelevant. 
gcc calls them both.

But sure, if there are separate bugzillas, post it in both. 

			Linus

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> That is A TOTAL PIECE OF SH*T, and against gcc's own documentation.
> 
> "-mtune=x" is very much defined to be a performance _tuning_ option, not 
> an "architectural" option. 
> 
> Quite frankly, this is a gcc bug. Plain and simple. 
> 

No argument there.

	-hpa

Re: [BUG] x86 kenel won't boot under Virtual PC

[david]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:

> david@lang.hm wrote:
>> I'd hate to see all the distros enabling embedded just to get this 
>> performance boost
>
> It's about producing code that works generically, not a performance boost.

I was arguing against the proposal to enable generic unless someone went 
under embedded to disable it.

if that's not what was being proposed, my apologies for misunderstanding.

David Lang

Re: [BUG] x86 kenel won't boot under Virtual PC

[H. Peter Anvin]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

Linus Torvalds wrote:
>>>
>> Yes, it does:
>>
>>   /* We need to decide which NOP sequence to use for 32bit and
>>      64bit. When -mtune= is used:
>>
>>      1. For PROCESSOR_I386, PROCESSOR_I486, PROCESSOR_PENTIUM and
>>      PROCESSOR_GENERIC32, f32_patt will be used.
>>      2. For PROCESSOR_PENTIUMPRO, PROCESSOR_PENTIUM4, PROCESSOR_NOCONA,
>>      PROCESSOR_CORE, PROCESSOR_CORE2, and PROCESSOR_GENERIC64,
>>      alt_long_patt will be used.
>>      3. For PROCESSOR_ATHLON, PROCESSOR_K6, PROCESSOR_K8 and
>>      PROCESSOR_AMDFAM10, alt_short_patt will be used.
>>
>>      When -mtune= isn't used, alt_long_patt will be used if
>>      cpu_arch_isa_flags has Cpu686. Otherwise, f32_patt will
>>      be used.
>>
>> "alt_long_patt" uses NOPL and its variants.
> 
> That is A TOTAL PIECE OF SH*T, and against gcc's own documentation.
> 
> "-mtune=x" is very much defined to be a performance _tuning_ option, not 
> an "architectural" option. 
> 
> Quite frankly, this is a gcc bug. Plain and simple. 
> 

OK, digging some more into this garbage...

Apparently the situation isn't quite as dire as it first seems.
At least gcc 4.3.0 doesn't actually pass -mtune= to gas; it just drops 
the option on the floor.  This means this bug isn't manifest when 
calling from gcc (as opposed to invoking as directly.)

However, I would still like to push the following patch to be on the 
safe side, ok?

	-hpa


[-- Attachment #2: 0002-x86-prevent-binutils-from-being-smart-and-generat.patch --]
[-- Type: text/x-patch, Size: 1346 bytes --]

>From 0fbbf1008e5177677dbdb6292d94ec40df965d82 Mon Sep 17 00:00:00 2001
From: H. Peter Anvin <hpa@zytor.com>
Date: Mon, 8 Sep 2008 12:01:48 -0700
Subject: [PATCH] x86: prevent binutils from being "smart" and generating NOPLs for us

binutils, contrary to documented behaviour, will generate long NOPs (a
P6-or-higher instruction which is broken on at least some VIA chips,
Virtual PC/Virtual Server, and some versions of Qemu) depending on the
-mtune= option, which is not supposed to change architectural
behaviour.

Pass an explicit override to the assembler, in case ends up passing
the -mtune= parameter to gas (gcc 4.3.0 does not appear to.)

Signed-off-by: H. Peter Anvin <hpa@zytor.com>
---
 arch/x86/Makefile_32.cpu |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/arch/x86/Makefile_32.cpu b/arch/x86/Makefile_32.cpu
index e372b58..b72b4f7 100644
--- a/arch/x86/Makefile_32.cpu
+++ b/arch/x86/Makefile_32.cpu
@@ -45,3 +45,8 @@ cflags-$(CONFIG_MGEODEGX1)	+= -march=pentium-mmx
 # cpu entries
 cflags-$(CONFIG_X86_GENERIC) 	+= $(call tune,generic,$(call tune,i686))
 
+# Bug fix for binutils: this option is required in order to keep
+# binutils from generating NOPL instructions against our will.
+ifneq ($(CONFIG_X86_P6_NOP),y)
+cflags-y			+= $(call cc-option,-Wa$(comma)-mtune=generic32,)
+endif
-- 
1.5.5.1

Re: [BUG] x86 kenel won't boot under Virtual PC

[Linus Torvalds]

On Mon, 8 Sep 2008, H. Peter Anvin wrote:
>
> OK, digging some more into this garbage...
> 
> Apparently the situation isn't quite as dire as it first seems.
> At least gcc 4.3.0 doesn't actually pass -mtune= to gas; it just drops the
> option on the floor.  This means this bug isn't manifest when calling from gcc
> (as opposed to invoking as directly.)

Ok, goodie. And we don't pass the -mtune=xyz option when we use gas 
directly, so we should be all ok.

> However, I would still like to push the following patch to be on the safe
> side, ok?

I have no problem with it, with the fix to actually assemble things. But I 
also don't think it's a huge deal (ie not 2.6.27 stuff - we don't know 
if some odd version of as does something odd with -mtune=generic, or 
whether this can interact oddly with some version of gcc perhaps passing 
the _correct_ -mtune to the assembler?).

		Linus