RE: lilo vs other OS bootloaders was: FreeBSD makes progress

RE: lilo vs other OS bootloaders was: FreeBSD makes progress

[Grover, Andrew]

> From: Helge Hafting [mailto:helgehaf@idb.hist.no]
> > I'm not advocating anything similar for Linux, I'm just 
> saying it's an
> > interesting thought experiment - what if the SMP-ness of a 
> machine was
> > abstracted from the kernel proper? How much of the kernel 
> really cares, or
> > really *should* care about SMP/UP?
> You would also get rid of performance.  The agnostic kernel would be
> slower than simply running the SMP kernel on UP.
> 
> Here's why:
> You can easily make an "agnostic kernel & modules" by changing the
> spinlocks to function calls.  Then you'll provide a null stub 
> call site
> for running UP, and the real spinlock code for running SMP.
> Unfortunately, this gives the overhead of a function call, 
> both for SMP
> and for UP.  This overhead is usually _bigger_ than the overhead of a
> inlined spinlock.

Obviously moving the spinlock behind a function call would be slower.
However, I'm not sure whether this would really hurt overall kernel
performance, for two reasons: First, I would think that the requirement to
use the lock instruction would overshadow any function call overhead.
Second, I would guess that minimization of the time the kernel spins on held
locks is much more important than whether acquiring an unheld lock takes 4
instructions or 8.

Anyways, if I ever go back for my masters degree I think modularizing SMP/UP
(and looking at the performance impact) would be an interesting thesis
project ;-)

Regards -- Andy

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Alan Cox]

> However, I'm not sure whether this would really hurt overall kernel
> performance, for two reasons: First, I would think that the requirement to
> use the lock instruction would overshadow any function call overhead.

AMD lock overhead for an L1 cached exclusive line is basically on the 
synchronization point, it doesn't go to memory. I believe later Pentium I*
do the same. 

Alan

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Tim Hockin]

"Grover, Andrew" wrote:
 
> Anyways, if I ever go back for my masters degree I think modularizing SMP/UP
> (and looking at the performance impact) would be an interesting thesis
> project ;-)

They give master's degrees for stuff like that?  sheeeeeeeit..

-- 
Tim Hockin
Systems Software Engineer
Sun Microsystems, Cobalt Server Appliances
thockin@sun.com

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Pavel Machek]

Hi!

> Here's Linux:
> 
> Drivers (SMP agnostic)
> Kernel (SMP/UP specific)
> 
> Here's Windows:
> 
> Drivers (SMP agnostic)
> Kernel (SMP agnostic)
> HAL (SMP/UP specific, contains locking primitive funcs etc.)
> 
> So they use the same kernel and just switch out the HAL.

-> overhead.

If you don't care about overhead, just run SMP kernels even on UP.
								Pavel
-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[H. Peter Anvin]

Followup to:  <m1iteth8j8.fsf@frodo.biederman.org>
By author:    ebiederm@xmission.com (Eric W. Biederman)
In newsgroup: linux.dev.kernel
>
> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> 
> > > So of course I realize this wouldn't happen any time soon, but has any
> > > discussion taken place regarding enhancing the bootloader (grub? Steal
> > > FreeBSD's?) to load modular drivers very early, and possibly abstracting
> > > SMP/UP from the kernel proper? Wouldn't this be a better solution than
> > > initrd?
> > 
> > All the discussion we have has been based on seriously enhancing and
> > expanding the use of the initrd/ramfs layer. Remember we can begin running
> > from ramfs without interrupts, pci bus scans or the like. The things it cant
> > do are - pick a kernel by processor type, pick SMP/non SMP.
> > 
> > As it happens both of those are things that are deeply buried in the whole
> > compile choices and how we generate the code itself - so they do need to
> > be boot loader driven (or user driven)
> > 
> > So the path for ACPI could indeed go
> > 
> > load kernel
> > load initial ramfs
> > Discover we have ACPI
> > load acpi core
> > load acpi irq router
> > load acpi timers
> > [init hardware]
> > load ide disk
> > load ext3
> > mount /
> 
> Sounds about right.  
> 
> If we really need to do weird things like pick a kernel by processor
> type, or pick SMP/non SMP.  You can even do those from an initramfs
> with a linux  booting linux kernel patch.
> 

I discussed something like this before using "Genesis"; the idea
behind it was to make sure we have enough modules to access the
filesystem before the kernel starts (by using pre-initialization code
that can read filesystem using the firmware interface.)  Unfortunately
life got busy on me, but I still think that it is probably the right
way to approach this.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > So of course I realize this wouldn't happen any time soon, but has any
> > discussion taken place regarding enhancing the bootloader (grub? Steal
> > FreeBSD's?) to load modular drivers very early, and possibly abstracting
> > SMP/UP from the kernel proper? Wouldn't this be a better solution than
> > initrd?
> 
> All the discussion we have has been based on seriously enhancing and
> expanding the use of the initrd/ramfs layer. Remember we can begin running
> from ramfs without interrupts, pci bus scans or the like. The things it cant
> do are - pick a kernel by processor type, pick SMP/non SMP.
> 
> As it happens both of those are things that are deeply buried in the whole
> compile choices and how we generate the code itself - so they do need to
> be boot loader driven (or user driven)
> 
> So the path for ACPI could indeed go
> 
> load kernel
> load initial ramfs
> Discover we have ACPI
> load acpi core
> load acpi irq router
> load acpi timers
> [init hardware]
> load ide disk
> load ext3
> mount /

Sounds about right.  

If we really need to do weird things like pick a kernel by processor
type, or pick SMP/non SMP.  You can even do those from an initramfs
with a linux  booting linux kernel patch.

Eric

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Horst von Brand]

"Grover, Andrew" <andrew.grover@intel.com> said:

[...]

> Here's Linux:
> 
> Drivers (SMP agnostic)

The SMP/UP difference is pervasive in Linux, I don't think there is a
single SMP-agnostic driver (as far as _binaries_ go). Source is mostly
SMP-agnostic.

> Kernel (SMP/UP specific)
> 
> Here's Windows:
> 
> Drivers (SMP agnostic)
> Kernel (SMP agnostic)
> HAL (SMP/UP specific, contains locking primitive funcs etc.)
> 
> So they use the same kernel and just switch out the HAL.

And each SMP/UP aware operation _has_ to be a separate function (no inlines
allowed this way). Doable with not too much overhead, AFAICS (at least in
principle); but maintaining (and synchronizing) a clean split will soon be
a nightmare. Or end up in two separate kernels anyway ;-)

[...]

> For one thing, it would get rid of the hundreds of "#ifdef CONFIG_SMP"s in
> the kernel. ;-)

At least the #ifdef's isolate just the places that must be different in
both cases. It might be worthwhile to abstract them out from the source
(not necesarily from the binary kernel).
-- 
Dr. Horst H. von Brand                Usuario #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Helge Hafting]

"Grover, Andrew" wrote:
> 
> > >  ANdreas Dilger wrote:
> > > > Win2K even abstracts all SMP/UP code into a module (the
> > HAL) and loads this
> > > > at boot, thus using the same kernel for both.
> >
> > >     the only possibility of this shows how ugly is SMP in win2k...
> >
> > Not necessarily. More likely the difference between SMP and
> > UP is marketing-only and both have the overhead of SMP
> > locking, etc..
> 
> No, they don't do this by running an SMP kernel on UP, they do it by
> abstracting functions that care about SMP into another module.
> 
> Here's Linux:
> 
> Drivers (SMP agnostic)
> Kernel (SMP/UP specific)
> 
Linux modules are not agnostic, they too are SMP specific.
Because a module may use spinlocks.

Modules should really be compiled with the kernel.  They are an
intersting way to save some memory, they are not a way of
distributing drivers or anything like that.

> I'm not advocating anything similar for Linux, I'm just saying it's an
> interesting thought experiment - what if the SMP-ness of a machine was
> abstracted from the kernel proper? How much of the kernel really cares, or
> really *should* care about SMP/UP?
> 
> For one thing, it would get rid of the hundreds of "#ifdef CONFIG_SMP"s in
> the kernel. ;-)

You would also get rid of performance.  The agnostic kernel would be
slower than simply running the SMP kernel on UP.

Here's why:
You can easily make an "agnostic kernel & modules" by changing the
spinlocks to function calls.  Then you'll provide a null stub call site
for running UP, and the real spinlock code for running SMP.

Unfortunately, this gives the overhead of a function call, both for SMP
and for UP.  This overhead is usually _bigger_ than the overhead of a
inlined spinlock.

So if you aim for a simple distribution - go for SMP.  The loss on UP is
small. I have haerd of cases when the SMP setup code fails on UP
when smp hardware is missing.  You could of course work to eliminate
that.

Helge Hafting

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[David S. Miller]

   From: "Grover, Andrew" <andrew.grover@intel.com>
   Date: Tue, 4 Sep 2001 14:52:17 -0700 
   
   I'm not advocating anything similar for Linux, I'm just saying it's an
   interesting thought experiment - what if the SMP-ness of a machine was
   abstracted from the kernel proper? How much of the kernel really cares, or
   really *should* care about SMP/UP?

Every spinlock :-) You'd have to either accept their overhead, or have
some way to nop out the instructions on uniprocessor boots.  There
would still be the space overhead after such code patching.

I remember the Digital UNIX folks did something interesting in this
area.  There should be a paper online somewhere.

Later,
David S. Miller
davem@redhat.com

RE: lilo vs other OS bootloaders was: FreeBSD makes progress

[Grover, Andrew]

> >  ANdreas Dilger wrote:
> > > Win2K even abstracts all SMP/UP code into a module (the 
> HAL) and loads this
> > > at boot, thus using the same kernel for both.
> 
> >     the only possibility of this shows how ugly is SMP in win2k...
> 
> Not necessarily. More likely the difference between SMP and
> UP is marketing-only and both have the overhead of SMP
> locking, etc..

No, they don't do this by running an SMP kernel on UP, they do it by
abstracting functions that care about SMP into another module.

Here's Linux:

Drivers (SMP agnostic)
Kernel (SMP/UP specific)

Here's Windows:

Drivers (SMP agnostic)
Kernel (SMP agnostic)
HAL (SMP/UP specific, contains locking primitive funcs etc.)

So they use the same kernel and just switch out the HAL.

I'm not advocating anything similar for Linux, I'm just saying it's an
interesting thought experiment - what if the SMP-ness of a machine was
abstracted from the kernel proper? How much of the kernel really cares, or
really *should* care about SMP/UP?

For one thing, it would get rid of the hundreds of "#ifdef CONFIG_SMP"s in
the kernel. ;-)

Regards -- Andy

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Jamie Lokier]

Alan Cox wrote:
> All the discussion we have has been based on seriously enhancing and
> expanding the use of the initrd/ramfs layer. Remember we can begin running
> from ramfs without interrupts, pci bus scans or the like. The things it cant
> do are - pick a kernel by processor type, pick SMP/non SMP.

The kernel could be chosen by processor type, if you added a "reboot
into a new kernel" function.

It would be rather large for one initramfs, as _all_ of the modules have
combinations of SMP/non-SMP x i386/486/586/686/athlon/686-PAE versions,
not just the core kernel.

It may still be a useful function for CDROM or network boots though.
I.e. initramfs selects an optimised kernel and set of modules to run,
and replaces the current generic kernel with the optimised one.

-- Jamie

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Samium Gromoff]

 ANdreas Dilger wrote: 
> Win2K even abstracts all SMP/UP code into a module (the HAL) and loads this
> at boot, thus using the same kernel for both.
    the only possibility of this shows how ugly is SMP in win2k...
   this is a situation where they are or geniuses or idiots.
   MS never proved to be geniuses so they still are idiots...

cheers,
 Sam

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Rik van Riel]

On Sat, 1 Sep 2001, Samium Gromoff wrote:
>  ANdreas Dilger wrote:
> > Win2K even abstracts all SMP/UP code into a module (the HAL) and loads this
> > at boot, thus using the same kernel for both.

>     the only possibility of this shows how ugly is SMP in win2k...

Not necessarily. More likely the difference between SMP and
UP is marketing-only and both have the overhead of SMP
locking, etc..

cheers,

Rik
-- 
IA64: a worthy successor to i860.

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Alan Cox]

> Did someone any benchmarking?

Yes

> I expect the loss of performance per application a none issue.
> What do you think: >0.5%?

It depends on the actual application

> Are you considering interrupt latency in the first place?

IRQ latency isnt the issue. Older x86 processors have dreadful locking 
performance. On an Athlon or later pIII there seems to be very little
difference, but other stuff you can measure it.

Locks cost. On older CPU's atomic operations go back to main memory and on
newer processors they still cause pipeline stalls. Calling back to non
inlined locks has a small hit too.

> Then obviously BeOS is also engineered from idiots...
> Oh, and QNX/RTP has separate kernels for UP/SMP. And they
> don't need UP/SMP versions of "modules".

I doubt they were idiots. I suspect they had different engineering
constraints like "base system must fit on one floppy disk".

Alan

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Peter Wächtler]

Samium Gromoff wrote:
> 
>  ANdreas Dilger wrote:
> > Win2K even abstracts all SMP/UP code into a module (the HAL) and loads this
> > at boot, thus using the same kernel for both.
>     the only possibility of this shows how ugly is SMP in win2k...
>    this is a situation where they are or geniuses or idiots.
>    MS never proved to be geniuses so they still are idiots...
> 

Did someone any benchmarking?
I expect the loss of performance per application a none issue.
What do you think: >0.5%?
Are you considering interrupt latency in the first place?

Then obviously BeOS is also engineered from idiots...
Oh, and QNX/RTP has separate kernels for UP/SMP. And they
don't need UP/SMP versions of "modules".

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Alan Cox]

> So of course I realize this wouldn't happen any time soon, but has any
> discussion taken place regarding enhancing the bootloader (grub? Steal
> FreeBSD's?) to load modular drivers very early, and possibly abstracting
> SMP/UP from the kernel proper? Wouldn't this be a better solution than
> initrd?

All the discussion we have has been based on seriously enhancing and
expanding the use of the initrd/ramfs layer. Remember we can begin running
from ramfs without interrupts, pci bus scans or the like. The things it cant
do are - pick a kernel by processor type, pick SMP/non SMP.

As it happens both of those are things that are deeply buried in the whole
compile choices and how we generate the code itself - so they do need to
be boot loader driven (or user driven)

So the path for ACPI could indeed go

load kernel
load initial ramfs
Discover we have ACPI
load acpi core
load acpi irq router
load acpi timers
[init hardware]
load ide disk
load ext3
mount /

Alan

Re: lilo vs other OS bootloaders was: FreeBSD makes progress

[Andreas Dilger]

On Aug 31, 2001  14:49 -0700, Grover, Andrew wrote:
> Just for discussion's sake, I would like to point out that other OSs do have
> loaders that can load boot drivers, and they can use this to increase the
> modularity of their kernel. FreeBSD's and Win2k's bootloaders are examples.
> Win2K even abstracts all SMP/UP code into a module (the HAL) and loads this
> at boot, thus using the same kernel for both.

Just FYI, this is just around the corner.  Al Viro has made it mandatory
(I believe) to have a very simple initramfs, for doing things like mounting
the root filesystem and setting up other services which are now done in
the kernel at boot time.  This initramfs (very similar to initrd) is at
the end of the kernel image, so it can't get lost and doesn't require
sending a separate file (i.e. for network booting, etc).

> possibly abstracting SMP/UP from the kernel proper?

Will never happen, as there would probably be overhead for both UP and SMP
to do this.  If you want something like this (for ease of admin or so),
you can generally run the SMP kernel on UP systems and take the performance
hit, but not everyone will do that.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

lilo vs other OS bootloaders was: FreeBSD makes progress

[Grover, Andrew]

OK, sounds like enhancing LILO to load modules beyond the kernel is
unworkable.

Just for discussion's sake, I would like to point out that other OSs do have
loaders that can load boot drivers, and they can use this to increase the
modularity of their kernel. FreeBSD's and Win2k's bootloaders are examples.
Win2K even abstracts all SMP/UP code into a module (the HAL) and loads this
at boot, thus using the same kernel for both.

So of course I realize this wouldn't happen any time soon, but has any
discussion taken place regarding enhancing the bootloader (grub? Steal
FreeBSD's?) to load modular drivers very early, and possibly abstracting
SMP/UP from the kernel proper? Wouldn't this be a better solution than
initrd?

Just curious -- Andy


> From: Russell Coker [mailto:russell@coker.com.au]
> > It would be really cool if lilo or grub could pull this 
> trick, too. Without
> > it, we're left with compiling ACPI into the kernel, or 
> doing some initrd
> > trick, both of which have disadvantages.
> 
> LILO is architecturally incapable of doing such things 
> without a huge degree 
> of hacking.  LILO has no support for .b files that set state. 
>  Having a .b 
> file that loads one of two other files depending on what it 
> detects would be 
> possible (but a gross hack).
> 
> Then if LILO determined that ACPI hardware was present how 
> would it tell the 
> kernel to load a module?
> 
> Having a statically linked program on an initrd which will 
> run "modprobe 
> acpi" if it detects appropriate hardware is very easy to do 
> for an install or 
> rescue disk.
> 
> For a running system the scripts that make the initrd image 
> can check for the 
> presence of ACPI to determine whether the ACPI module belongs 
> in the initrd.