(no subject)

[John Bradford]

richard.brunner@amd.com
In-Reply-To: <20030930073814.GA26649@mail.jlokier.co.uk>
References:  <20030930073814.GA26649@mail.jlokier.co.uk>
Subject: Re: [PATCH] Mutilated form of Andi Kleen's AMD prefetch errata patch

Quote from Jamie Lokier <jamie@shareable.org>:

>    2. Nevertheless, when a kernel _not_ optimised for those AMD processors
>       is run on one, the "alternative" mechanism now correctly replaces
>       prefetch instructions with nops.

> It's a reasonable argument that if a kernel version, say 2.6.0, promises
> to hide the errata from _userspace_ programs, then the generic or
> not-AMD-optimised kernels should do that too.  Otherwise userspace may
> as well provide the workaround itself, by catching SIGSEGV - because it
> is perfectly normal and common to run a generic kernel on an AMD.

This problem, and others like it, would be solved for ever with
Adrian's bitmap-of-supported-cpus patch, though.  With that, there is
no 'generic' kernel, either it supports Athlons, or it doesn't.  100%
elimination of Athlon specific code for other CPUs.  Everybody is
happy.

Of course a kernel compiled strictly for 386s may seem to boot on an
Athlon but not work properly.  So what?  Just don't run the 'wrong'
kernel.

John.

Re: your mail

[Dave Jones]

On Tue, Sep 30, 2003 at 09:17:16AM +0100, John Bradford wrote:
 
 > Of course a kernel compiled strictly for 386s may seem to boot on an
 > Athlon but not work properly.  So what?  Just don't run the 'wrong'
 > kernel.

Wrong answer. How do you intend to install Linux when a distro boot
kernel is compiled for lowest-common-denominator (386), and is the
'wrong' kernel for an Athlon ?

We hashed this argument out a week or so ago, it seems the message
didn't get across. YOU CAN NOT DISABLE ERRATA WORKAROUNDS IN A KERNEL
THAT MAY POSSIBLY BOOT ON HARDWARE THAT WORKAROUND IS FOR.

clearer ?

		Dave

-- 
 Dave Jones     http://www.codemonkey.org.uk

Re: your mail

[Jamie Lokier]

Dave Jones wrote:
> On Tue, Sep 30, 2003 at 09:17:16AM +0100, John Bradford wrote:
>  > Of course a kernel compiled strictly for 386s may seem to boot on an
>  > Athlon but not work properly.  So what?  Just don't run the 'wrong'
>  > kernel.
> 
> Wrong answer. How do you intend to install Linux when a distro boot
> kernel is compiled for lowest-common-denominator (386), and is the
> 'wrong' kernel for an Athlon ?

I'm not sure what the fuss is; a strict 386 kernel runs just fine
without any problems on an Athlon.  But anyway...

Dave, you are conflating "kernel compiled strictly for 386s" with
"compiled for lowest-common-denominator".

They are totally different configurations.  Isn't that why we have
"generic" now?

The latter is for distro boots.  The former is for that
386-as-a-firewall with 1MB of RAM, where it _really_ has to trim
everything it can, and no errata thank you.

I've not heard of anyone actually wanting a strict 386 kernel lately,
but strict 486 is not so unusual.

Just as some people want a P4 optimised kernel, and some people want a
K7 optimised kernel, so some people want a 386 or 486 or Pentium
optimised kernel.  Lowest-common-denominator means it runs on
everything, and isn't really anything to do with 386 any more - that's
not really the lowest-common-denominator, by virtue of the obvious
fact that pure 386 code isn't reliable on all other CPUs.

> We hashed this argument out a week or so ago, it seems the message
> didn't get across. YOU CAN NOT DISABLE ERRATA WORKAROUNDS IN A KERNEL
> THAT MAY POSSIBLY BOOT ON HARDWARE THAT WORKAROUND IS FOR.

I agree.  It shouln't be possible to boot on the wrong hardware: it
should refuse.

There is precedent: X86_GOOD_APIC && X86_LOCAL_APIC: when booted on a
non-MMX P5, it refuses to boot, because it does not contain the errata
workaround.

Unfortunately the kernel has opposite precedents too.

By selecting a PII kernel, it is possible to configure out the code
for X86_PPRO_FENCE and X86_F00F_BUG, yet as far as I can tell, those
_can_ possibly boot on kernels where the errata are needed, and nary a
printk is emitted for it.  Nasty bugs they are, too.

More generally than the CPU, you can also configure out BLK_DEV_RZ1000
which is another crucial workaround that needs to go in any
lowest-common-denominator kernel.  Basically, if you're building a
distro boot kernel, you must turn on all known workarounds.  That's
certainly lowest-common-denominator, but it's a far cry from the
configuration that a 386-as-firewall user wants.

> clearer?

If the kernel had a consistent policy so far, it would be more clear,
but it doesn't.

-- Jamie

Re: your mail

[John Bradford]

Quote from Dave Jones <davej@redhat.com>:
> On Tue, Sep 30, 2003 at 09:17:16AM +0100, John Bradford wrote:
>  
>  > Of course a kernel compiled strictly for 386s may seem to boot on an
>  > Athlon but not work properly.  So what?  Just don't run the 'wrong'
>  > kernel.
> 
> Wrong answer. How do you intend to install Linux when a distro boot
> kernel is compiled for lowest-common-denominator (386), and is the
> 'wrong' kernel for an Athlon ?

I don't.  I *never* suggested doing that.  I clearly said a kernel
compiled *strictly* for 386s.  I.E. Without support for other
processors.

> We hashed this argument out a week or so ago, it seems the message
> didn't get across. YOU CAN NOT DISABLE ERRATA WORKAROUNDS IN A KERNEL
> THAT MAY POSSIBLY BOOT ON HARDWARE THAT WORKAROUND IS FOR.

It seems the message didn't get across to you.

Have you actually looked at Adrian's patch?

*Forget* that 386=lowest-common-denominator.  This
'386=lowest-common-denominator' theme is out of date, and we should be
moving away from it - oh, hang on, that's exactly what Adrian's patch
allows us to do.

A distribution installation kernel needs to boot all supported
hardware - of course it does.  So what?  Just select support for all
the processors in the configurator.  No, don't just select 386,
because 386 doesn't mean 386 and above anymore with Adrian's patch, it
means support 386 and don't bloat the kernel with workarounds for
other processors.  Select *all* processors.  Now you have a nice,
(bloated), kernel that boots on the same hardware that you old '386'
one did.  Fine for installation on diverse hardware.  Rubbish for
performance.

Unless, of course, you object to the possibility that somebody might
go out of their way to compile a 386 specific kernel from source
themselves, then run it on an Athlon.  By chance it will probably
appear to work OK, but won't have the workaround enabled.  So what?
Only somebody who knows exactly what they were doing is likely to do
that - how could it happen by accident?  If you really must, put a
warning in to say, 'This kernel doesn't support your processor', but
doing that just adds more bloat.  OK, so the bloat will be freed after
boot, but it's still bloat on the boot device, which matters in some
embedded systems.

> clearer ?

It's clear that you didn't read my original post, yes.

John.

Re: your mail

[Dave Jones]

On Tue, Sep 30, 2003 at 03:06:27PM +0100, Jamie Lokier wrote:
 > Dave Jones wrote:
 > > On Tue, Sep 30, 2003 at 09:17:16AM +0100, John Bradford wrote:
 > >  > Of course a kernel compiled strictly for 386s may seem to boot on an
 > >  > Athlon but not work properly.  So what?  Just don't run the 'wrong'
 > >  > kernel.
 > > Wrong answer. How do you intend to install Linux when a distro boot
 > > kernel is compiled for lowest-common-denominator (386), and is the
 > > 'wrong' kernel for an Athlon ?
 > I'm not sure what the fuss is; a strict 386 kernel runs just fine
 > without any problems on an Athlon.  But anyway...

Unless it got configured away as proposed in your earlier patch.

 > Dave, you are conflating "kernel compiled strictly for 386s" with
 > "compiled for lowest-common-denominator".
 > 
 > They are totally different configurations.  Isn't that why we have
 > "generic" now?

CONFIG_GENERIC could be extended to offer other options yes,
but right now what it does doesn't really match the name IMO.
Right now its closer to a CONFIG_MAX_CACHELINE_SIZE

 > The latter is for distro boots.  The former is for that
 > 386-as-a-firewall with 1MB of RAM, where it _really_ has to trim
 > everything it can, and no errata thank you.

Again, 'trimming' away a few hundred bytes of errata workarounds
is ridiculous when we have bigger fish to fry where we can save
KBs of .text size, and MB's of runtime memory.

 > I've not heard of anyone actually wanting a strict 386 kernel lately,
 > but strict 486 is not so unusual.

ISTR that current gcc's emit 486 instructions anyway, so its possible
that with a modern toolchain, you can't *build* a 386 kernel.
I'm not sure if that got fixed or not, I don't track gcc lists any more.

 > Just as some people want a P4 optimised kernel, and some people want a
 > K7 optimised kernel, so some people want a 386 or 486 or Pentium
 > optimised kernel.  Lowest-common-denominator means it runs on
 > everything, and isn't really anything to do with 386 any more - that's
 > not really the lowest-common-denominator, by virtue of the obvious
 > fact that pure 386 code isn't reliable on all other CPUs.

Elaborate? "pure 386 code" (whatever that means in your definition)
should run perfectly reliable on every CPU we care about.

 > > We hashed this argument out a week or so ago, it seems the message
 > > didn't get across. YOU CAN NOT DISABLE ERRATA WORKAROUNDS IN A KERNEL
 > > THAT MAY POSSIBLY BOOT ON HARDWARE THAT WORKAROUND IS FOR.
 > I agree.  It shouln't be possible to boot on the wrong hardware: it
 > should refuse.

So first you argue for compiling out a few hundred bytes of errata
workaround, now you want to instead compile in checks & printk's
(which probably add up to not far off the same amount of space).

 > By selecting a PII kernel, it is possible to configure out the code
 > for X86_PPRO_FENCE and X86_F00F_BUG, yet as far as I can tell, those
 > _can_ possibly boot on kernels where the errata are needed, and nary a
 > printk is emitted for it.  Nasty bugs they are, too.

Indeed. That's arguably a bug that occured when someone split the
original CONFIG_M686 into _M686 & MPENTIUMII.

 > More generally than the CPU, you can also configure out BLK_DEV_RZ1000
 > which is another crucial workaround that needs to go in any
 > lowest-common-denominator kernel.

I wouldn't look at the history of drivers/ide/ as a shining example of
good design 8-)

 > Basically, if you're building a
 > distro boot kernel, you must turn on all known workarounds.  That's
 > certainly lowest-common-denominator, but it's a far cry from the
 > configuration that a 386-as-firewall user wants.

Ok, I see what you're getting at, but Adrian's patch turned arch/i386/Kconfig
and arch/i386/Makefile into guacamole.  After spending so much time
getting that crap into something maintainable, it seemed a huge step
backwards to litter it with dozens of ifdefs and duplication.
There has to be a cleaner way of pleasing everyone.

 > > clearer?
 > If the kernel had a consistent policy so far, it would be more clear,
 > but it doesn't.

Agreed, there are some questionable parts.

		Dave

-- 
 Dave Jones     http://www.codemonkey.org.uk

Re: your mail

[Jamie Lokier]

John Bradford wrote:
> Unless, of course, you object to the possibility that somebody might
> go out of their way to compile a 386 specific kernel from source
> themselves, then run it on an Athlon.  By chance it will probably
> appear to work OK, but won't have the workaround enabled.  So what?

Actually the 386 kernel will work just fine on the AMD...  The
workaround is only needed, in the kernel, to protect against the
kernel's own use of non-386 features...

Userspace is a different matter, but userspace has a lot of
model-specific things to worry about beyond this one instruction on
AMD.  In practice: bswap, cmov, cmpxchg, mmx, sse, sse2, so knowing
whether to use prefetch or not is just one more variable for userspace
- and one which any portable app or library will have to know about in
any case.

(Aside: It is quite an anomaly that those cumbersome floating point
instructions are emulated on the older CPUs, yet all the other
instructions aren't emulated.  Emulation is very slow, and forcing
userspace to just use different code instead is good, but that's just
as valid for floating point as it is for MMX, cmpxchg etc.)

> Only somebody who knows exactly what they were doing is likely to do
> that - how could it happen by accident?  If you really must, put a
> warning in to say, 'This kernel doesn't support your processor', but
> doing that just adds more bloat.  OK, so the bloat will be freed after
> boot, but it's still bloat on the boot device, which matters in some
> embedded systems.

To be fair, the kernel really ought to just say that and halt.  That
is a fine compromise.  It won't make embedded systems folks completely
happy, because if you've only got 2MB of NVRAM for your whole kernel
_and_ filesystem including user data (think PDA or cellphone), then a
hundred bytes here or there is actually worth trimming.

But then, those sort of embedded folks should just figure out
compressed software-suspend, and then they can ditch __init data from
the NVRAM image completely.  It's much better to lose all of __init
than just a few bytes here or there, isn't it?

-- Jamie

Re: your mail

[Dave Jones]

On Tue, Sep 30, 2003 at 03:58:54PM +0100, Jamie Lokier wrote:

 > (Aside: It is quite an anomaly that those cumbersome floating point
 > instructions are emulated on the older CPUs, yet all the other
 > instructions aren't emulated.  Emulation is very slow, and forcing
 > userspace to just use different code instead is good, but that's just
 > as valid for floating point as it is for MMX, cmpxchg etc.)

There was a patch around a while back that did 486 emulation on 386
kernels. I think it even made into the Mandrake kernel.

 > To be fair, the kernel really ought to just say that and halt.  That
 > is a fine compromise.  It won't make embedded systems folks completely
 > happy, because if you've only got 2MB of NVRAM for your whole kernel
 > _and_ filesystem including user data (think PDA or cellphone), then a
 > hundred bytes here or there is actually worth trimming.

With such tight constraints, why not just use 2.4 (or even 2.2) which
has much lower memory usage and diskspace requirements ?

		Dave

-- 
 Dave Jones     http://www.codemonkey.org.uk

Re: your mail

[Jamie Lokier]

Dave Jones wrote:
>  > I'm not sure what the fuss is; a strict 386 kernel runs just fine
>  > without any problems on an Athlon.  But anyway...
> 
> Unless it got configured away as proposed in your earlier patch.

No, I don't understand.  What about my patch, or indeed anything else,
stops a "strict 386" kernel from running on an Athlon?

>  > The latter is for distro boots.  The former is for that
>  > 386-as-a-firewall with 1MB of RAM, where it _really_ has to trim
>  > everything it can, and no errata thank you.
> 
> Again, 'trimming' away a few hundred bytes of errata workarounds
> is ridiculous when we have bigger fish to fry where we can save
> KBs of .text size, and MB's of runtime memory.

Well I think both are worthwhile.  Low hanging fruit and all that -
this is an example of a small saving that's very clear and easy.

>  > I've not heard of anyone actually wanting a strict 386 kernel lately,
>  > but strict 486 is not so unusual.
> 
> ISTR that current gcc's emit 486 instructions anyway, so its possible
> that with a modern toolchain, you can't *build* a 386 kernel.
> I'm not sure if that got fixed or not, I don't track gcc lists any more.

Afaict GCC has fine targetting for the 386, better than it did years
ago.  It didn't used to use the "leave" instruction, have an option to
optimise for size, or options for selecting exactly which
architectural instruction set it would use.

Anyway, that there is very little difference between 386 and 486 from
an application point of view anyway.  You may be thinking of the
recent C++ ABI debacle, I think it was, which accidentially turned out
to require some instruction emulation in the Debian kernel.  I think
they've fixed it in GCC now.

>  > Just as some people want a P4 optimised kernel, and some people want a
>  > K7 optimised kernel, so some people want a 386 or 486 or Pentium
>  > optimised kernel.  Lowest-common-denominator means it runs on
>  > everything, and isn't really anything to do with 386 any more - that's
>  > not really the lowest-common-denominator, by virtue of the obvious
>  > fact that pure 386 code isn't reliable on all other CPUs.
> 
> Elaborate? "pure 386 code" (whatever that means in your definition)
> should run perfectly reliable on every CPU we care about.

If that were true, why are we talking about needing workarounds for
non-386 chips to work correctly?

The canonical example is the F00F sequence: reliable on a 386, crashes
a Pentium.  That's a fine example of pure 386 code not being reliable
on a higher CPU.  And that's why it isn't safe to run Linux 1.0 on
your Pentium web server.

> So first you argue for compiling out a few hundred bytes of errata
> workaround, now you want to instead compile in checks & printk's
> (which probably add up to not far off the same amount of space).

Oh, I have nothing against __init space :)

>  > By selecting a PII kernel, it is possible to configure out the code
>  > for X86_PPRO_FENCE and X86_F00F_BUG, yet as far as I can tell, those
>  > _can_ possibly boot on kernels where the errata are needed, and nary a
>  > printk is emitted for it.  Nasty bugs they are, too.
> 
> Indeed. That's arguably a bug that occured when someone split the
> original CONFIG_M686 into _M686 & MPENTIUMII.

It's a bit more complicated.  It dates from before we had the
"alternative" macro, and it was still cool to optimise spin_unlock()
into the most minimal instruction sequence at compile time.

It's only since then that we've been generalising to "M586 should run
on all later models correctly".  Arguably, tidying up in the process.

Now we could use "alternative" to put the locked store or non-locked
store there and it would not look out of place.

If we're honest, Linux seems to have evolved through the 2.5 series
from "optimise the primitives as tight as reasonable for a target
architecture" to "a few nops here and there won't hurt".  Perhaps
Transmeta's malign influence, as nops cost virtually nothing on those :)

Or perhaps it's because CPU models have branched and don't make a
straight line any more.  So we have to do more run-time checking to
keep it sane.

>  > More generally than the CPU, you can also configure out BLK_DEV_RZ1000
>  > which is another crucial workaround that needs to go in any
>  > lowest-common-denominator kernel.
> 
> I wouldn't look at the history of drivers/ide/ as a shining example of
> good design 8-)

No, but as an example of needing to enable all the workarounds for a
distro boot kernel, it's a glorious gem.  Even now people aren't quite
sure if multi-sector mode or DMA should be enabled by default :)

>  > Basically, if you're building a
>  > distro boot kernel, you must turn on all known workarounds.  That's
>  > certainly lowest-common-denominator, but it's a far cry from the
>  > configuration that a 386-as-firewall user wants.
> 
> Ok, I see what you're getting at, but Adrian's patch turned arch/i386/Kconfig
> and arch/i386/Makefile into guacamole.  After spending so much time
> getting that crap into something maintainable, it seemed a huge step
> backwards to litter it with dozens of ifdefs and duplication.
> There has to be a cleaner way of pleasing everyone.

Perhaps it's in a name.  It doesn't help that there's an assumed
linear progression of CPUs to support, up to the point where they
branch off all over the place in feature space.  In the linear part,
CONFIG_M586, CONFIG_M686 etc. seem to mean "support this CPU or
later", whatever later means (and it's not stated exactly).  After the
explosion of different feature directions, they stop meaning that and
just become optimisation knobs, as all the different essential features
are supported at run time.

Personally I think Adrian's patch's heart is in the right place,
simply because the menu options make more sense than the present
rather confusion decision, if you intend to (or might ever, take your
pick) run a kernel compiled for one CPU on another.  I am never sure,
for example, if it's safe to take the hard disk from my K6 and drop it
into a P5MMX box and boot from it.  The kernel config just doesn't
make that clear.

With Adrian's it does, even if the code behind it is a little like
guacamole.  Perhaps the code could be cleaner; I don't see that
individual CPU model support is much different than what we already
have, except for the option to fix features at compile time rather than
run time.

And that gives me an idea.... ;)

-- Jamie

Re: your mail

[Adrian Bunk]

On Tue, Sep 30, 2003 at 03:50:08PM +0100, Dave Jones wrote:
>...
>  > Basically, if you're building a
>  > distro boot kernel, you must turn on all known workarounds.  That's
>  > certainly lowest-common-denominator, but it's a far cry from the
>  > configuration that a 386-as-firewall user wants.
> 
> Ok, I see what you're getting at, but Adrian's patch turned arch/i386/Kconfig
> and arch/i386/Makefile into guacamole.  After spending so much time
> getting that crap into something maintainable, it seemed a huge step
> backwards to litter it with dozens of ifdefs and duplication.
> There has to be a cleaner way of pleasing everyone.
>...

Referring to the latest patch I sent:

arch/i386/Kconfig:
The only problems seem to be some CPU_ONLY_* derived symbols I haven't 
yet found a better solution for.

arch/i386/Makefile:
There are two ifdefs to deal with Pentium 4 and K7/K8 selected at the 
same time:
ifdef CONFIG_CPU_PENTIUM4
  cpuflags-$(CONFIG_CPU_K{7,8})    := ...
else
  cpuflags-$(CONFIG_CPU_K{7,8})    := ...
endif

That's perhaps not optimal but IMHO not that bad.

The dozens of ifdefs were in other areas where I tried to add some 
additional space optimizations. It was a mistake to put them into the 
same patch and in the latest patches I sent they were already separated 
and they are _not_ required for the CPU selection scheme.

> 		Dave

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed