[PATCH] Pentium Pro - sysenter - doublefault

[PATCH] Pentium Pro - sysenter - doublefault

[Jim Houston]

Hi Everyone,

I upgraded my Pentium Pro system to Redhat 9, installed a
linux-2.6.0-test3 kernel, and it fails with a double-fault when
init starts.

The code which decides if it is o.k. to use sysenter is broken for
some Pentium Pro cpus ,in particular, this bit of code from
arch/i386/kernel/cpu/intel.c:

	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
		clear_bit(X86_FEATURE_SEP, c->x86_capability);

On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP.
This results in a double-fault when init starts.  The double-fault
happens on the sysexit.  The new double-fault handler caught this
nicely, and I was able to debug this with kgdb.

The logic above is exactly what Intel says to do in "IA-32 Intel®
Architecture Software Developer's Manual, Volume 2: Instruction Set
Reference" on page 3-767.  It also says that sysenter was added to the
Pentium II.

I checked the Pentium Pro and Pentium II Specifications Update manuals
hoping to find the details to justify the "mask < 3" portion of the test
above. They both describe sysenter related errata but none which was
fixed in mask 3.

The attached patch avoids using sysenter on all Pentium Pro systems.

Jim Houston - Concurrent Computer Corp.


diff -urN linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c
linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c
--- linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c	2003-08-20
10:30:14.000000000 -0400
+++ linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c	2003-08-21
14:39:35.000000000 -0400
@@ -246,7 +246,7 @@
 	}
 
 	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
-	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
+	if ( c->x86 == 6 && c->x86_model < 3)
 		clear_bit(X86_FEATURE_SEP, c->x86_capability);
 	
 	/* Names for the Pentium II/Celeron processors 

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Mikael Pettersson]

Jim Houston writes:
 > Hi Everyone,
 > 
 > I upgraded my Pentium Pro system to Redhat 9, installed a
 > linux-2.6.0-test3 kernel, and it fails with a double-fault when
 > init starts.
 > 
 > The code which decides if it is o.k. to use sysenter is broken for
 > some Pentium Pro cpus ,in particular, this bit of code from
 > arch/i386/kernel/cpu/intel.c:
 > 
 > 	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
 > 	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
 > 		clear_bit(X86_FEATURE_SEP, c->x86_capability);
 > 
 > On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP.
 > This results in a double-fault when init starts.  The double-fault
 > happens on the sysexit.  The new double-fault handler caught this
 > nicely, and I was able to debug this with kgdb.
 > 
 > The logic above is exactly what Intel says to do in "IA-32 Intel®
 > Architecture Software Developer's Manual, Volume 2: Instruction Set
 > Reference" on page 3-767.  It also says that sysenter was added to the
 > Pentium II.

I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID),
and it says (section 3.4) that SEP is unsupported when the signature
as a whole is less that 0x633. This means all PPros, and PII Model 3s
with steppings less than 3.

The problem is that the kernel check you quoted above is buggy: the
c->x86_model < 3 && c->x86_mask < 3 part fails for late-stepping PPros
since c->x86_mask >= 3 for them. The test should be rewritten as:

        if (c->x86 == 6 && (c->x86_model < 3 ||
                            (c->x86_model == 3 && c->x86_mask < 3)))
                clear_bit(X86_FEATURE_SEP, c->x86_capability);

/Mikael

[PATCH2] Pentium Pro - sysenter - doublefault

[Jim Houston]

On Thu, 2003-08-21 at 17:32, Mikael Pettersson wrote:
>  > The logic above is exactly what Intel says to do in "IA-32 Intel®
>  > Architecture Software Developer's Manual, Volume 2: Instruction Set
>  > Reference" on page 3-767.  It also says that sysenter was added to the
>  > Pentium II.
> 
> I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID),
> and it says (section 3.4) that SEP is unsupported when the signature
> as a whole is less that 0x633. This means all PPros, and PII Model 3s
> with steppings less than 3.
> 

Hi Dave, Everyone,

This make sense. Here is Mikael's suggested code as a patch.

Dave, I picked your name from the maintainers list.  Please 
feed this patch up the chain.


Jim Houston - Concurrent Computer Corp. 


diff -urN linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c
linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c
--- linux-2.6.0-test3.orig/arch/i386/kernel/cpu/intel.c	2003-08-20
10:30:14.000000000 -0400
+++ linux-2.6.0-test3.new/arch/i386/kernel/cpu/intel.c	2003-08-21
21:34:40.000000000 -0400
@@ -246,7 +246,8 @@
 	}
 
 	/* SEP CPUID bug: Pentium Pro reports SEP but doesn't have it */
-	if ( c->x86 == 6 && c->x86_model < 3 && c->x86_mask < 3 )
+	if ( c->x86 == 6 && ((c->x86_model < 3) ||
+				(c->x86_model == 3 && c->x86_mask < 3)))
 		clear_bit(X86_FEATURE_SEP, c->x86_capability);
 	
 	/* Names for the Pentium II/Celeron processors 

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Jim Houston wrote:
> On my cpu model=1 and mask=9, it doesn't clear 86_FEATURE_SEP.
> This results in a double-fault when init starts.  The double-fault
> happens on the sysexit.  The new double-fault handler caught this
> nicely, and I was able to debug this with kgdb.

Does anyone know what the syenter & sysexit instructions do on these
early PPro CPUs?

The Intel documentation is vague, saying only to avoid using them.
I'd like to know what happens if userspace does "sysenter" on one of
these systems.  Does it issue Invalid Opcode, General Protection
fault, or something else?

Jim you can answer this as you have such a Ppro.  Could you please run
this very simple userspace program for me, and report the result?

	int main() { __asm__ ("sysenter"); return 0; }

I expect it to die with SIGILL on Pentium and earlier chips, and
SIGSEGV on "good" PPro and later chips running kernels which don't
enable the sysenter instruction.

But what does it do on your early Intel PPro, the one which is the
subject of this thread?

Thanks,
-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jakob Oestergaard]

On Mon, Aug 25, 2003 at 05:05:14AM +0100, Jamie Lokier wrote:
...
> Jim you can answer this as you have such a Ppro.  Could you please run
> this very simple userspace program for me, and report the result?
> 
> 	int main() { __asm__ ("sysenter"); return 0; }

I tested on two boxes:

Stepping 1 ppro:  SIGSEGV
Stepping 7 ppro:  SIGSEGV

If you need additional info, please just ask.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Jakob Oestergaard wrote:
> > Jim you can answer this as you have such a Ppro.  Could you please run
> > this very simple userspace program for me, and report the result?
> > 
> > 	int main() { __asm__ ("sysenter"); return 0; }
> 
> I tested on two boxes:
> 
> Stepping 1 ppro:  SIGSEGV
> Stepping 7 ppro:  SIGSEGV

Thank you!

So that means the sysenter instruction _does_ exist on the PPro and
early Pentium II, but it isn't usable.

It's safe so long as it is disabled, but as you found, when you enable
and use it, it doesn't work as expected.  (I wonder what the actual
behaviour of sysenter/sysexit are on these CPUs.)

-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Mikael Pettersson wrote:
>  > The logic above is exactly what Intel says to do in "IA-32 Intel®
>  > Architecture Software Developer's Manual, Volume 2: Instruction Set
>  > Reference" on page 3-767.  It also says that sysenter was added to the
>  > Pentium II.
> 
> I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID),
> and it says (section 3.4) that SEP is unsupported when the signature
> as a whole is less that 0x633. This means all PPros, and PII Model 3s
> with steppings less than 3.

So (double-checking) the pseudo-code in "IA-32: Intel Architecture
Software Development Manual, Volume 2: Instruction Set Reference" is buggy?

Oh my!  Perhaps there are other bugs in that behemoth of a manual, too :/

-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Mikael Pettersson wrote:
> I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID),
> and it says (section 3.4) that SEP is unsupported when the signature
> as a whole is less that 0x633. This means all PPros, and PII Model 3s
> with steppings less than 3.

"SEP is unsupported".  It's interesting that Pentium Pro erratum #82
is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment
selector' to SS and CS registers", implying that SYSENTER does
_something_ useful on PPros.

-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Jamie Lokier wrote:
> So that means the sysenter instruction _does_ exist on the PPro and
> early Pentium II, but it isn't usable.

If anyone has information on what the SYSENTER and SYSEXIT
instructions actually do on Intel Pentium Pro or stepping<3 Pentium II
processors, I am very interested.

I'm intrigued to know if the buggy behaviour of these instructions is
really unsafe, or simply hard to use so Intel changed the behaviour.
(An example of hard to use would be SYSENTER not disabling
interrupts).  If they are safe but hard to use, perhaps the ingenuity
of kernel hackers can work around the hardness >:)

Does anyone have a contact at Intel for this question?

Thanks,
-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jakob Oestergaard]

On Mon, Aug 25, 2003 at 07:29:05AM +0100, Jamie Lokier wrote:
> Jamie Lokier wrote:
> > So that means the sysenter instruction _does_ exist on the PPro and
> > early Pentium II, but it isn't usable.
> 
> If anyone has information on what the SYSENTER and SYSEXIT
> instructions actually do on Intel Pentium Pro or stepping<3 Pentium II
> processors, I am very interested.

I dug up a little more from the archeological site (machine room -
sigh)...

-------------------------------------------
model name      : Pentium II (Deschutes)
stepping        : 2
$ ./syse 
Segmentation fault
-------------------------------------------
model name      : Celeron (Mendocino)
stepping        : 0
cpu MHz         : 334.097
$ ./syse
Segmentation fault
-------------------------------------------
CPU: Pentium II/Pentium II Xeon/Celeron (299.94-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x651  Stepping = 1
$ ./syse
Bus error (core dumped)
-------------------------------------------

The last one is a FreeBSD box.


-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jim Houston]

On Mon, 2003-08-25 at 02:29, Jamie Lokier wrote:
> Jamie Lokier wrote:
> > So that means the sysenter instruction _does_ exist on the PPro and
> > early Pentium II, but it isn't usable.
> 
> If anyone has information on what the SYSENTER and SYSEXIT
> instructions actually do on Intel Pentium Pro or stepping<3 Pentium II
> processors, I am very interested.
> 
> I'm intrigued to know if the buggy behaviour of these instructions is
> really unsafe, or simply hard to use so Intel changed the behaviour.
> (An example of hard to use would be SYSENTER not disabling
> interrupts).  If they are safe but hard to use, perhaps the ingenuity
> of kernel hackers can work around the hardness >:)

Hi Jamie,

I tried your test on my machine.  It fails with a segmentation
fault.  I noticed that the Pentium II specifications update manual
starts with rev C0 stepping (ignoring mask rev < 3).
I'm inclined to forgive Intel for not publishing the scary errata that
goes with the first few mask revs, particularly for an old product.

When I was chasing the original problem, I added tracing code 
(compiling the kernel with finstrument-functions) so that when I
got into kgdb after the double-fault I could see that it had just
completed a umask system call.  I'm assuming that it failed on
the sysexit.

I keep the old Pentium Pro around because it has an NMI interrupt
button.

I'm happy that Linus has merged the fix to disable correctly
disable sysenter for these machines.

Jim Houston - Concurrent Computer Corp

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Richard Curnow]

* Jamie Lokier <jamie@shareable.org> [2003-08-26]:
> Could you please run this very simple userspace program for me, and
> report the result?
> 
> 	int main() { __asm__ ("sysenter"); return 0; }
> 
> I expect it to die with SIGILL on Pentium and earlier chips, and
> SIGSEGV on "good" PPro and later chips running kernels which don't
> enable the sysenter instruction.

OK, since I get something different to the other reports I saw:

 1:20PM-malvern-0-534-% ./sysenter
 1:20PM-malvern-STKFLT-535-% echo $?
144

/proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 5
model name      : Pentium II (Deschutes)
stepping        : 0
cpu MHz         : 333.495
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr
bogomips        : 657.40

uname -a:
Linux malvern 2.6.0-test2-mm5 #2 Fri Aug 8 12:06:50 BST 2003 i686 unknown

HTH

-- 
Richard \\\ SuperH Core+Debug Architect /// .. At home ..
  P.    /// richard.curnow@superh.com  ///  rc@rc0.org.uk
Curnow  \\\ http://www.superh.com/    ///  www.rc0.org.uk

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Richard Curnow wrote:
> OK, since I get something different to the other reports I saw:
> 
>  1:20PM-malvern-0-534-% ./sysenter
>  1:20PM-malvern-STKFLT-535-% echo $?
> 144

Hi Richard,

That's because you ran it on a 2.5/2.6 kernel, right?  The test code
is meant for 2.4 kernels and earlier :)

Here is a more universal test:

	int main () {
		asm ("movl %%esp,%%ebp;sysenter" : : "a" (1), "b" (0));
		return 0;
	}

I expect it to do the first of these which is applicable:

	- raise SIGILL on Pentium and earlier Intel CPUs
	- raise SIGILL on non-Intel CPUs which don't have the SEP capability
	- raise SIGSEGV on Pentium Pro CPUs
	- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping < 3
	- raise SIGSEGV on 2.4 kernels
	- exit with status 0 on 2.6 kernels

Enjoy,
-- Jamie

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Richard Curnow]

* Jamie Lokier <jamie@shareable.org> [2003-08-27]:
> Here is a more universal test:
> 
[snip]

> 	- exit with status 0 on 2.6 kernels

Yes, confirmed, that's what it did on 2.6.0-test2-mm5 here.  Sorry, I
can't try 2.4 right now.

-- 
Richard \\\ SuperH Core+Debug Architect /// .. At home ..
  P.    /// richard.curnow@superh.com  ///  rc@rc0.org.uk
Curnow  \\\ http://www.superh.com/    ///  www.rc0.org.uk

RE: [PATCH] Pentium Pro - sysenter - doublefault

[dl-ipaddr]

did you notice the announcement from www.apache.org and www.debian.org?

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jamie Lokier
Sent: Wednesday, August 27, 2003 10:01 PM
To: Jim Houston; linux-kernel@vger.kernel.org; jim.houston@ccur.com
Subject: Re: [PATCH] Pentium Pro - sysenter - doublefault


Richard Curnow wrote:
> OK, since I get something different to the other reports I saw:
> 
>  1:20PM-malvern-0-534-% ./sysenter  1:20PM-malvern-STKFLT-535-% echo 
> $? 144

Hi Richard,

That's because you ran it on a 2.5/2.6 kernel, right?  The test code is
meant for 2.4 kernels and earlier :)

Here is a more universal test:

	int main () {
		asm ("movl %%esp,%%ebp;sysenter" : : "a" (1), "b" (0));
		return 0;
	}

I expect it to do the first of these which is applicable:

	- raise SIGILL on Pentium and earlier Intel CPUs
	- raise SIGILL on non-Intel CPUs which don't have the SEP
capability
	- raise SIGSEGV on Pentium Pro CPUs
	- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping
< 3
	- raise SIGSEGV on 2.4 kernels
	- exit with status 0 on 2.6 kernels

Enjoy,
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in the body of a message to majordomo@vger.kernel.org More majordomo
info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Stan Bubrouski]

Jamie Lokier wrote:

<SNIP>

> I expect it to do the first of these which is applicable:
> 
> 	- raise SIGILL on Pentium and earlier Intel CPUs
> 	- raise SIGILL on non-Intel CPUs which don't have the SEP capability
> 	- raise SIGSEGV on Pentium Pro CPUs
> 	- raise SIGSEGV on Pentium II CPUs with model == 3 and stepping < 3
> 	- raise SIGSEGV on 2.4 kernels
> 	- exit with status 0 on 2.6 kernels
> 
> Enjoy,
> -- Jamie

As expected I get a SIGILL on P166 with 2.4.22

-sb
-----------------------------------------------
The price of freedom? Ask your Senator how much
the RIAA gave him for his Lexus.

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Pasi Savolainen]

* Jamie Lokier <jamie@shareable.org>:
> Richard Curnow wrote:
>> OK, since I get something different to the other reports I saw:
>> 
>>  1:20PM-malvern-0-534-% ./sysenter
>>  1:20PM-malvern-STKFLT-535-% echo $?
>> 144
> 
> Hi Richard,
> 
> That's because you ran it on a 2.5/2.6 kernel, right?  The test code
> is meant for 2.4 kernels and earlier :)

If this is of any help..

- -
pvsavola@a11a:~/code$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 1
model name      : Pentium Pro
stepping        : 6
cpu MHz         : 199.312
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov
bogomips        : 397.31

pvsavola@a11a:~/code$ ./sysent ; echo $?
Segmentation fault
139
pvsavola@a11a:~/code$ uname -a
Linux a11a 2.4.19-ck3-rmap #1 Mon Aug 26 21:38:49 EEST 2002 i686 GNU/Linux
- -

-- 
   Psi -- <http://www.iki.fi/pasi.savolainen>

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Pavel Machek]

Hi!

> > I double-checked AP-485 (24161823.pdf, the "real" reference to CPUID),
> > and it says (section 3.4) that SEP is unsupported when the signature
> > as a whole is less that 0x633. This means all PPros, and PII Model 3s
> > with steppings less than 3.
> 
> "SEP is unsupported".  It's interesting that Pentium Pro erratum #82
> is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment
> selector' to SS and CS registers", implying that SYSENTER does
> _something_ useful on PPros.

Well, with CS==0 machine is not going to survive too long.
If it only happens sometimes you might catch the double fault
and fixup, but....
				Pavel
-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Re: [PATCH] Pentium Pro - sysenter - doublefault

[Jamie Lokier]

Pavel Machek wrote:
> > "SEP is unsupported".  It's interesting that Pentium Pro erratum #82
> > is "SYSENTER/SYSEXIT instructions can implicitly load 'null segment
> > selector' to SS and CS registers", implying that SYSENTER does
> > _something_ useful on PPros.
> 
> Well, with CS==0 machine is not going to survive too long.
> If it only happens sometimes you might catch the double fault
> and fixup, but....

The erratum only applies when you load CS==0 _deliberately_, by setting
the MSR to that.

I'm wondering what happens when you don't do silly things - what is
the undocumented behaviour of SYSENTER/SYSEXIT on those chips?

I vaguely recall reading details about the behaviour change made by
Intel, around the time it was done, but I can't see to find it
anywhere.

-- Jamie