[ANNOUNCE] hotplug-ng 001 release

[ANNOUNCE] hotplug-ng 001 release

[Greg KH]

I'd like to announce, yet-another-hotplug based userspace project:
linux-ng.  This collection of code replaces the existing linux-hotplug
package with very tiny, compiled executable programs, instead of the
existing bash scripts.

It currently provides the following:
	- a /sbin/hotplug multiplexer.  Works identical to the existing
	  bash /sbin/hotplug.
	- autoload programs for usb, scsi, and pci modules.  These
	  programs determine what module needs to be loaded when the
	  kernel emits a hotplug event for these types of devices.  This
	  works just like the existing linux-hotplug scripts, with a few
	  exceptions.

But why redo this all in .c code?  What's wrong with shell scripts?
Nothing is wrong with shell scripts, unless you don't want to have an
interpreter in your initramfs/initrd and you want to provide
/sbin/hotplug and autoload module functionality.  Or if you have a huge
box that spawns a zillion hotplug events all at once, and you need to be
able to handle all of that with the minimum amount of processing time
and memory.

So, how small are these programs?  Take a look:
   text    data     bss     dec     hex filename
   4669      32     124    4825    12d9 hotplug
   5077       8     348    5433    1539 module_pci
   4925       8     412    5345    14e1 module_scsi
   5349       8     348    5705    1649 module_usb

Those are all static binaries, linked with klibc (which is included in
the hotplug-ng package, just like udev.)

This compares to the following bash scripts:
-rwxr-xr-x  1 root root  4412 Feb 10 15:28 /sbin/hotplug
-rw-r--r--  1 root root   702 Sep 24 08:04 /etc/hotplug/blacklist
-rw-r--r--  1 root root  5293 Sep 24 08:04 /etc/hotplug/hotplug.functions
-rwxr-xr-x  1 root root  3739 Sep 24 08:04 /etc/hotplug/pci.agent
-rwxr-xr-x  1 root root  1459 Sep 24 08:04 /etc/hotplug/scsi.agent
-rwxr-xr-x  1 root root 13466 Sep 24 08:04 /etc/hotplug/usb.agent
-rw-r--r--  1 root root 39306 Sep 24 08:04 /etc/hotplug/usb.distmap
-rw-r--r--  1 root root  4364 Sep 24 08:04 /etc/hotplug/usb.handmap
-rw-r--r--  1 root root   189 Sep 24 08:04 /etc/hotplug/usb.usermap

All of which are loaded into memory for each hotplug event (for specific
hotplug events, only that bus type of file is loaded.)

But what about speed?  With a completely unscientific measurement on my
old, slow laptop, it takes about 2 seconds from the time I plug a usb
device into the machine, for the proper module to be loaded.  With the
hotplug-ng program, it takes less than a second.

And for those of you who might remember the old dietHotplug program that
also did the same thing in a tiny amount of space, this project
obsoletes that one.  dietHotplug had to be rebuilt for every kernel that
was used on the system, hotplug-ng uses the ability for modprobe to
determine the module that needs to be loaded based on the module
aliases[1].

The code can be found at:
	kernel.org/pub/linux/utils/kernel/hotplug/hotplug-ng-001.tar.gz
for those who wish to poke around in it.

I still have a few more programs to write to get it up to the same
functionality as the existing hotplug scripts (firmware, ieee1392, etc.)
but those will be done soon.  I'd like to get people's comments on the
idea, and welcome suggestions and even patches :)

hotplug-ng development is done in a BitKeeper repository located at:
	bk://linuxusb.bkbits.net/hotplug-ng

If anyone ever wants a tarball of the current bk tree, just email me.

thanks,

greg k-h

[1] modprobe as it currently works stops loading modules when it finds
an alias that matches.  This does not work for drivers that claim to
support "all devices" and then later on fail on devices that they really
don't support.  For that, all matching drivers need to be loaded for the
system to work properly.  The linux-hotplug scripts handle this
correctly, so if you rely on this functionality, please stick with that
package for now.  I'll be modifying modprobe to add this feature in the
near future.

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Thu, Feb 10, 2005 at 04:40:33PM -0800, Greg KH wrote:
> I'd like to announce, yet-another-hotplug based userspace project:
> linux-ng.

Bah.  The name is hotplug-ng.  Sorry about that...

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Kasper Sandberg]

hey greg

i remember for some months back, you posted something similar.. is this
a version thats ready for use? if it is! im gonna use it! :D

On Thu, 2005-02-10 at 16:52 -0800, Greg KH wrote:
> On Thu, Feb 10, 2005 at 04:40:33PM -0800, Greg KH wrote:
> > I'd like to announce, yet-another-hotplug based userspace project:
> > linux-ng.
> 
> Bah.  The name is hotplug-ng.  Sorry about that...
> 
> greg k-h
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, 2005-02-11 at 02:30 +0100, Kasper Sandberg wrote:
> hey greg
> 
> i remember for some months back, you posted something similar.. is this
> a version thats ready for use? if it is! im gonna use it! :D

Yes, this is that version, cleaned up and given a proper build system,
and even tested on my machines here :)

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Kasper Sandberg]

On Thu, 2005-02-10 at 22:41 -0800, Greg KH wrote:
> On Fri, 2005-02-11 at 02:30 +0100, Kasper Sandberg wrote:
> > hey greg
> > 
> > i remember for some months back, you posted something similar.. is this
> > a version thats ready for use? if it is! im gonna use it! :D
> 
> Yes, this is that version, cleaned up and given a proper build system,
> and even tested on my machines here :)
ah cool. and in that case, you probably also have ebuilds for it, if you
do, please post them somewhere :)
> 
> thanks,
> 
> greg k-h
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 12:47:07PM +0100, Kasper Sandberg wrote:
> On Thu, 2005-02-10 at 22:41 -0800, Greg KH wrote:
> > On Fri, 2005-02-11 at 02:30 +0100, Kasper Sandberg wrote:
> > > hey greg
> > > 
> > > i remember for some months back, you posted something similar.. is this
> > > a version thats ready for use? if it is! im gonna use it! :D
> > 
> > Yes, this is that version, cleaned up and given a proper build system,
> > and even tested on my machines here :)
> ah cool. and in that case, you probably also have ebuilds for it, if you
> do, please post them somewhere :)

I don't have an ebuild for it yet, but it's on my list to get done.  And
when I do so, it will just show up in the normal gentoo tree.  The main
"issue" with this is I need to create a virtual for the hotplug service
so it doesn't conflict with the existing hotplug package.  Not a big
deal, just not as simple as adding a single ebuild to the tree.

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Kasper Sandberg]

On Fri, 2005-02-11 at 09:06 -0800, Greg KH wrote:
> On Fri, Feb 11, 2005 at 12:47:07PM +0100, Kasper Sandberg wrote:
> > On Thu, 2005-02-10 at 22:41 -0800, Greg KH wrote:
> > > On Fri, 2005-02-11 at 02:30 +0100, Kasper Sandberg wrote:
> > > > hey greg
> > > > 
> > > > i remember for some months back, you posted something similar.. is this
> > > > a version thats ready for use? if it is! im gonna use it! :D
> > > 
> > > Yes, this is that version, cleaned up and given a proper build system,
> > > and even tested on my machines here :)
> > ah cool. and in that case, you probably also have ebuilds for it, if you
> > do, please post them somewhere :)
> 
> I don't have an ebuild for it yet, but it's on my list to get done.  And
> when I do so, it will just show up in the normal gentoo tree.  The main
> "issue" with this is I need to create a virtual for the hotplug service
> so it doesn't conflict with the existing hotplug package.  Not a big
> deal, just not as simple as adding a single ebuild to the tree.
just make it provide virtual/hotplug, then do a fgrep -r
"hotplug" /usr/portage/ and then replace with virtual/hotplug ;)
> 
> thanks,
> 
> greg k-h
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [ANNOUNCE] hotplug-ng 001 release

[Patrick McFarland]

Greg KH wrote:
> I'd like to announce, yet-another-hotplug based userspace project:
> linux-ng.  This collection of code replaces the existing linux-hotplug
> package with very tiny, compiled executable programs, instead of the
> existing bash scripts.
> 
> It currently provides the following:
> 	- a /sbin/hotplug multiplexer.  Works identical to the existing
> 	  bash /sbin/hotplug.
> 	- autoload programs for usb, scsi, and pci modules.  These
> 	  programs determine what module needs to be loaded when the
> 	  kernel emits a hotplug event for these types of devices.  This
> 	  works just like the existing linux-hotplug scripts, with a few
> 	  exceptions.
> 
> But why redo this all in .c code?  What's wrong with shell scripts?
> Nothing is wrong with shell scripts, unless you don't want to have an
> interpreter in your initramfs/initrd and you want to provide
> /sbin/hotplug and autoload module functionality.  Or if you have a huge
> box that spawns a zillion hotplug events all at once, and you need to be
> able to handle all of that with the minimum amount of processing time
> and memory.

Wow, thats pretty awesome. Just the other day I said, "Why is hotplug written 
in sh? Isn't that horribly inefficient way of handling something that needs to 
be done quickly using the least amount of resources possible?" It seems you 
were reading my mind.

Please, continue this project and encourage distros to switch to it (when it 
exceeds hotplug in functionality and stability). Ubuntu currently is trying to 
reduce boot time, and I bet something like this would factor in (even a few 
seconds helps).

-- 
Patrick "Diablo-D3" McFarland || pmcfarland@downeast.net
"Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd
all be running around in darkened rooms, munching magic pills and listening to
repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Thu, Feb 10, 2005 at 08:07:23PM -0500, Patrick McFarland wrote:
> 
> Please, continue this project and encourage distros to switch to it (when 
> it exceeds hotplug in functionality and stability). Ubuntu currently is 
> trying to reduce boot time, and I bet something like this would factor in 
> (even a few seconds helps).

Thanks for the kind words.

All distros are trying to reduce boot time.  I don't think that the
module autoload time has been fingered as taking any serious ammount of
boot time due to it happening in the background of everything else.  But
yes, this should help it out (as long as we aren't trading I/O time for
CPU time, like it can happen if you don't have a populated modules.alias
file...)

And yes, I've been in contact with the Ubuntu developer already about
using this code, instead of dietHotplug.

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Vojtech Pavlik]

On Thu, Feb 10, 2005 at 05:16:09PM -0800, Greg KH wrote:

> > Please, continue this project and encourage distros to switch to it (when 
> > it exceeds hotplug in functionality and stability). Ubuntu currently is 
> > trying to reduce boot time, and I bet something like this would factor in 
> > (even a few seconds helps).
> 
> Thanks for the kind words.
> 
> All distros are trying to reduce boot time.  I don't think that the
> module autoload time has been fingered as taking any serious ammount of
> boot time due to it happening in the background of everything else.

Hotplug scripts were identified as one of the major culprits of slow boot
when we did the analysis for SuSE 9.2. They took 40+ seconds from the
total boot time.

-- 
Vojtech Pavlik
SuSE Labs, SuSE CR

Re: [ANNOUNCE] hotplug-ng 001 release

[Lee Revell]

On Thu, 2005-02-10 at 17:16 -0800, Greg KH wrote:
> All distros are trying to reduce boot time.

They certainly aren't all trying very hard.  Debian and Fedora (last
time I checked) do not even run the init scripts in parallel.

Lee

Re: [ANNOUNCE] hotplug-ng 001 release

[Paolo Ciarrocchi]

On Sun, 13 Feb 2005 23:06:51 -0500, Lee Revell <rlrevell@joe-job.com> wrote:
> On Thu, 2005-02-10 at 17:16 -0800, Greg KH wrote:
> > All distros are trying to reduce boot time.
> 
> They certainly aren't all trying very hard.  Debian and Fedora (last
> time I checked) do not even run the init scripts in parallel.

Is there any distro that is running the init scripts in parallel ?
-- 
Paolo <paolo dot ciarrocchi at gmail dot com>
msn: paolo407@hotmail.com
hello: ciarrop

Re: [ANNOUNCE] hotplug-ng 001 release

[Prakash Punnoor]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

Paolo Ciarrocchi schrieb:
> On Sun, 13 Feb 2005 23:06:51 -0500, Lee Revell <rlrevell@joe-job.com> wrote:
>
>>On Thu, 2005-02-10 at 17:16 -0800, Greg KH wrote:
>>
>>>All distros are trying to reduce boot time.
>>
>>They certainly aren't all trying very hard.  Debian and Fedora (last
>>time I checked) do not even run the init scripts in parallel.
>
>
> Is there any distro that is running the init scripts in parallel ?

Gentoo.

--
Prakash Punnoor

formerly known as Prakash K. Cheemplavam

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] hotplug-ng 001 release

[Lee Revell]

On Mon, 2005-02-14 at 09:51 +0100, Prakash Punnoor wrote:
> Paolo Ciarrocchi schrieb:
> > On Sun, 13 Feb 2005 23:06:51 -0500, Lee Revell <rlrevell@joe-job.com> wrote:
> >
> >>On Thu, 2005-02-10 at 17:16 -0800, Greg KH wrote:
> >>
> >>>All distros are trying to reduce boot time.
> >>
> >>They certainly aren't all trying very hard.  Debian and Fedora (last
> >>time I checked) do not even run the init scripts in parallel.
> >
> >
> > Is there any distro that is running the init scripts in parallel ?
> 
> Gentoo.
> 

Last I heard Gentoo does not even do it by default.

I don't see why so much effort goes into improving boot time on the
kernel side when the most obvious user space problem is ignored.

Lee

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Mon, Feb 14, 2005 at 06:04:00PM -0500, Lee Revell wrote:
> On Mon, 2005-02-14 at 09:51 +0100, Prakash Punnoor wrote:
> > Paolo Ciarrocchi schrieb:
> > > On Sun, 13 Feb 2005 23:06:51 -0500, Lee Revell <rlrevell@joe-job.com> wrote:
> > >
> > >>On Thu, 2005-02-10 at 17:16 -0800, Greg KH wrote:
> > >>
> > >>>All distros are trying to reduce boot time.
> > >>
> > >>They certainly aren't all trying very hard.  Debian and Fedora (last
> > >>time I checked) do not even run the init scripts in parallel.
> > >
> > >
> > > Is there any distro that is running the init scripts in parallel ?
> > 
> > Gentoo.
> > 
> 
> Last I heard Gentoo does not even do it by default.

Gentoo doesn't do much "by default" :)  But it is an option.

> I don't see why so much effort goes into improving boot time on the
> kernel side when the most obvious user space problem is ignored.

What user space problem is that?

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Lee Revell]

On Mon, 2005-02-14 at 15:16 -0800, Greg KH wrote:
> > I don't see why so much effort goes into improving boot time on the
> > kernel side when the most obvious user space problem is ignored.
> 
> What user space problem is that?

That init scripts with no interdependencies are run sequentially rather
than in parallel.

There was an article from IBM a while back with a neat hack that used a
parallel make to fire off groups of init scripts in parallel.  I would
expect more interest in this from the distros.

Lee

Re: [ANNOUNCE] hotplug-ng 001 release

[Stefan Seyfried]

Lee Revell wrote:

> That init scripts with no interdependencies are run sequentially rather
> than in parallel.
> 
> There was an article from IBM a while back with a neat hack that used a
> parallel make to fire off groups of init scripts in parallel.  I would
> expect more interest in this from the distros.

You can boot a SUSE 9.2 with parallel init scripts (default AFAIR),
sequential init scripts and with a Makefile based solution. "Normal"
(not Makefile based) parallel booting is possible much longer on SUSE,
at least since 9.0 IIRC.
And guess what? "Parallel booting" alone, regardless of the mechanism
does not make much of a difference for the boot time.

Stefan

Re: [ANNOUNCE] hotplug-ng 001 release

[kernel]

On Tue, 2005-02-15 at 10:15, Stefan Seyfried wrote:
> You can boot a SUSE 9.2 with parallel init scripts (default AFAIR),
> sequential init scripts and with a Makefile based solution. "Normal"
> (not Makefile based) parallel booting is possible much longer on SUSE,
> at least since 9.0 IIRC.
> And guess what? "Parallel booting" alone, regardless of the mechanism
> does not make much of a difference for the boot time.
> 

My experience has been that hardware detection is what slows boot
process.  I've tested on various distros, Red Hat Linux, Slackware
Linux, SUSE, and Debian.

Starting services never seems to take any time (noticeable time).  But
when it lands on detecting hardware, that's where the time is chewed. 
Typically with hotplug (all using 2.4 kernels) it's about 30 seconds,
which is the same as the rest of the boot process in my testing lab. 
1394, USB, and PCMCIA seem to be the slowest (because when I remove
these devices or turn off detection of these types boot time is *much*
faster).


Two things that I believe should be addressed;

1) Speeding up boot time (even if that means moving some hardware
detection and recognition to after login)

-and-

2) Proper identification of filesystem types.  Would love to have an
agreed upon by majority change that would change the mounting of
filesystems (identifying FS TYPE) to be more accurate.  


regards,

-fd

Re: [ANNOUNCE] hotplug-ng 001 release

[Roland Dreier]

    Lee> I don't see why so much effort goes into improving boot time
    Lee> on the kernel side when the most obvious user space problem
    Lee> is ignored.

How much of a win is it to run init scripts in parallel?  I seem to
recall seeing tests that show that it doesn't make much difference and
may even slow things down by causing more disk seeks as various things
start up at the same time and cause reads of different files to get
interleaved.

On the other hand, hotplug is an area that real profiling of real
systems booting has identified as something that can be improved, and
Greg's hotplug-ng seems to be a step towards a measurable improvement.

 - R.

[OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Lee Revell]

On Mon, 2005-02-14 at 15:21 -0800, Roland Dreier wrote:
>     Lee> I don't see why so much effort goes into improving boot time
>     Lee> on the kernel side when the most obvious user space problem
>     Lee> is ignored.
> 
> How much of a win is it to run init scripts in parallel?  I seem to
> recall seeing tests that show that it doesn't make much difference and
> may even slow things down by causing more disk seeks as various things
> start up at the same time and cause reads of different files to get
> interleaved.
> 

This is why Windows XP reserves sapce at the beginning of the disk for
the files read during the boot process and caches copies of them there.

But, I was referring more to things like GDM not being started until all
the other init scripts are done.  Why not start it first, and let the
network initialize while the user is logging in?

> On the other hand, hotplug is an area that real profiling of real
> systems booting has identified as something that can be improved, and
> Greg's hotplug-ng seems to be a step towards a measurable improvement.

Agreed.

Lee

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Tim Bird]

Lee Revell wrote:
> But, I was referring more to things like GDM not being started until all
> the other init scripts are done.  Why not start it first, and let the
> network initialize while the user is logging in?

There are a number of techniques used by CE vendors to get fast bootup
time.  Some CE products boot Linux in under 1 second.  Sony's
best Linux boot time in the lab (from power on to user space)
was 148 milliseconds, on an ARM chip (running at 200 MHZ I believe).

Every product I know of that boots in under 1 second does it
by completely eliminating RC scripts, and using a custom init
program.

For anyone interested, CELF has some resources on this topic at:
http://tree.celinuxforum.org/CelfPubWiki/BootupTimeResources

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Lee Revell]

On Mon, 2005-02-14 at 16:16 -0800, Tim Bird wrote:
> Lee Revell wrote:
> > But, I was referring more to things like GDM not being started until all
> > the other init scripts are done.  Why not start it first, and let the
> > network initialize while the user is logging in?
> 
> There are a number of techniques used by CE vendors to get fast bootup
> time.  Some CE products boot Linux in under 1 second.  Sony's
> best Linux boot time in the lab (from power on to user space)
> was 148 milliseconds, on an ARM chip (running at 200 MHZ I believe).

The reason I marked by response OT is that the time from power on to
userspace does not seem to be a big problem.  It's the amount of time
from user space to presenting a login prompt that's way too long.  My
distro (Debian) runs all the init scripts one at a time, and GDM is the
last thing that gets run.  There is just no reason for this.  We should
start X and initialize the display and get the login prompt up there
ASAP, and let the system acquire the DHCP lease and start sendmail and
apache and get the date from the NTP server *in the background while I
am logging in*.  It's not rocket science.

Lee

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Kyle Moffett]

On Feb 14, 2005, at 20:17, Lee Revell wrote:
> On Mon, 2005-02-14 at 16:16 -0800, Tim Bird wrote:
>> Lee Revell wrote:
>>> But, I was referring more to things like GDM not being started until 
>>> all
>>> the other init scripts are done.  Why not start it first, and let the
>>> network initialize while the user is logging in?
>>
>> There are a number of techniques used by CE vendors to get fast bootup
>> time.  Some CE products boot Linux in under 1 second.  Sony's
>> best Linux boot time in the lab (from power on to user space)
>> was 148 milliseconds, on an ARM chip (running at 200 MHZ I believe).
>
> The reason I marked by response OT is that the time from power on to
> userspace does not seem to be a big problem.  It's the amount of time
> from user space to presenting a login prompt that's way too long.  My
> distro (Debian) runs all the init scripts one at a time, and GDM is the
> last thing that gets run.  There is just no reason for this.  We should
> start X and initialize the display and get the login prompt up there
> ASAP, and let the system acquire the DHCP lease and start sendmail and
> apache and get the date from the NTP server *in the background while I
> am logging in*.  It's not rocket science.

Such a system needs a drastically different bootup process than 
currently
exists, including the ability to specify init-script dependencies.  
(Like
for example user login via GDM (and with our setup, GDM working at all)
requires that AFS is mounted and NIS is working, which both require the
network to be available, which requires...  You can see where this is
going.  I think eventually we need a better /sbin/init, one that can use
a traditional legacy /etc/inittab file in addition to a newfangled
simultaneous boot process with lots of ways to start various kinds of
services.  Unfortunately such a system will need a _LOT_ of work and
testing to make sure it doesn't break existing setups.  Oh well, I can
dream, can't I? :-D

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Gábor Lénárt]

On Mon, Feb 14, 2005 at 08:45:39PM -0500, Kyle Moffett wrote:
> >last thing that gets run.  There is just no reason for this.  We should
> >start X and initialize the display and get the login prompt up there
> >ASAP, and let the system acquire the DHCP lease and start sendmail and
> >apache and get the date from the NTP server *in the background while I
> >am logging in*.  It's not rocket science.
> 
> Such a system needs a drastically different bootup process than 
> currently
> exists, including the ability to specify init-script dependencies.  
> (Like

Ok, so see Gentoo. Exactly fits your needs, it seems ;-) Dependencies are
supported, even paralell execution of init scripts are supported by default
design (you need to change only one setting to do this, IMHO, in
/etc/conf.d/rc). So it is already solved if you talking about the paralell
execution with dependency info in init scripts ...

Also the smf ability of Solaris10 seems to be interesting, but I don't want
to talk about it, since I have no time to see it in action, I've only read
about it, so ...

- Gábor

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Paolo Ciarrocchi]

On Tue, 15 Feb 2005 08:32:22 +0100, Gábor Lénárt <lgb@lgb.hu> wrote:
> On Mon, Feb 14, 2005 at 08:45:39PM -0500, Kyle Moffett wrote:
> > >last thing that gets run.  There is just no reason for this.  We should
> > >start X and initialize the display and get the login prompt up there
> > >ASAP, and let the system acquire the DHCP lease and start sendmail and
> > >apache and get the date from the NTP server *in the background while I
> > >am logging in*.  It's not rocket science.
> >
> > Such a system needs a drastically different bootup process than
> > currently
> > exists, including the ability to specify init-script dependencies.
> > (Like
> 
> Ok, so see Gentoo. Exactly fits your needs, it seems ;-) Dependencies are
> supported, even paralell execution of init scripts are supported by default
> design (you need to change only one setting to do this, IMHO, in
> /etc/conf.d/rc). So it is already solved if you talking about the paralell
> execution with dependency info in init scripts ...

So... why is Gentoo the only distro the uses parallel execution of
init scripts ?

-- 
Paolo <paolo dot ciarrocchi at gmail dot com>
msn: paolo407@hotmail.com
hello: ciarrop

Re: [OT] speeding boot process

[Bernd Petrovitsch]

On Tue, 2005-02-15 at 09:34 +0100, Paolo Ciarrocchi wrote:
[...]
> So... why is Gentoo the only distro the uses parallel execution of
> init scripts ?

Because no other distro bothered to implement it.

Apart from that we as quite far off-topic for LKML since this has
nothing to do with kernel.
The only reason getting this on-topic is to try to get the best SWSUSP
support in the kernel so one simply does not need this.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Helge Hafting]

Kyle Moffett wrote:

> On Feb 14, 2005, at 20:17, Lee Revell wrote:
>
>> On Mon, 2005-02-14 at 16:16 -0800, Tim Bird wrote:
>>
>>> Lee Revell wrote:
>>>
>>>> But, I was referring more to things like GDM not being started 
>>>> until all
>>>> the other init scripts are done.  Why not start it first, and let the
>>>> network initialize while the user is logging in?
>>>
>>>
>>> There are a number of techniques used by CE vendors to get fast bootup
>>> time.  Some CE products boot Linux in under 1 second.  Sony's
>>> best Linux boot time in the lab (from power on to user space)
>>> was 148 milliseconds, on an ARM chip (running at 200 MHZ I believe).
>>
>>
>> The reason I marked by response OT is that the time from power on to
>> userspace does not seem to be a big problem.  It's the amount of time
>> from user space to presenting a login prompt that's way too long.  My
>> distro (Debian) runs all the init scripts one at a time, and GDM is the
>> last thing that gets run.  There is just no reason for this.  We should
>> start X and initialize the display and get the login prompt up there
>> ASAP, and let the system acquire the DHCP lease and start sendmail and
>> apache and get the date from the NTP server *in the background while I
>> am logging in*.  It's not rocket science.
>
>
> Such a system needs a drastically different bootup process than currently
> exists, including the ability to specify init-script dependencies.  (Like

The init-script dependencies are specifies already - at least on debian.
Look at the most used runlevel, ls /etc/rc2.d:

S10sysklogd@
S11klogd
S14ppp
S18portmap
S19slapd
S20alsa
S20cupsys
S20dhcp3-server
S20exim
...
The numbers specify ordering constraint, low numbers must start before 
high numbers.
But plenty of scripts have no interdependencies, I have 18 scripts 
numbered S20.
Today they start sequentially anyway, in alphabetical order.  Several of 
them simply
wait for hardware, which could be done in parallel.  So parallel 
execution would be a win.
One could always have a user-settable limit on how much to run in 
parallel, to help
out those low-memory machines. 

> for example user login via GDM (and with our setup, GDM working at all)
> requires that AFS is mounted and NIS is working, which both require the
> network to be available, which requires...  You can see where this is

Yes.  I don't think the X system will start before other things in a 
standard
installation, precisely because it *might* need the network and a bunch 
of other
things to validate the user.  So most of the gain will come from the 
parallel starting
of earlier scripts.
I think one can safely say that the xserver it won't need web, email
or the printer subsystems to get up though, and that could also save 
some time.

> going.  I think eventually we need a better /sbin/init, one that can use
> a traditional legacy /etc/inittab file in addition to a newfangled
> simultaneous boot process with lots of ways to start various kinds of
> services.  Unfortunately such a system will need a _LOT_ of work and
> testing to make sure it doesn't break existing setups.  Oh well, I can
> dream, can't I? :-D

It isn't all that hard.  An init with one change - scripts with the same 
number
runs in parallel instead of sequentially. (And possibly some 
configurable limit
on parallel scripts).  Then the rest of the work lies in an efficient 
and correct numbering of those
scripts.  The paranoid can renumber them so they run in the same
order as they used to.  Most people can use the existing numbering, 
perhaps they
discover an ordering bug or two which is trivially fixed using "mv" to 
renumber scripts.
And the more daring can renumber their scripts for maximum parallelism.

Even with today's system the goal of starting X early is achievable.  If you
know that X doesn't need the network on _your_ machine - renumber it to
start really early.  The modification is easy enough to do. :-)

If it still starts slow, swithc to a light session manager (like xdm) and
a lightweight window manager that doesn't bring in kde or gnome.

Helge Hafting

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Bernd Petrovitsch]

On Tue, 2005-02-15 at 09:55 +0100, Helge Hafting wrote:
[...]
> The init-script dependencies are specifies already - at least on debian.

These are not dependencies but "only" the sequence of startup (and it is
not only Debian but also Fedora/RedHat, SuSE, Mandrake and probably all
except Gentoo).
Yuo get a much stricter ordering and sorting (and thus much simpler to
implement in a shell script).

> Look at the most used runlevel, ls /etc/rc2.d:
> 
> S10sysklogd@
> S11klogd
> S14ppp
> S18portmap
> S19slapd
> S20alsa
> S20cupsys
> S20dhcp3-server
> S20exim
> ...
> The numbers specify ordering constraint, low numbers must start before 
> high numbers.
> But plenty of scripts have no interdependencies, I have 18 scripts 
> numbered S20.

This would be a win (especially if the numbers are tweked to tune this)
with a relatively small effort.
However for real dependencies and parallelism you want the info similar
to creat a Makefile from it (i.e. the explicit dependency from service X
to service Y). As a consequence you can get rid of the numbers (since
they are not needed any more).

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Helge Hafting]

Bernd Petrovitsch wrote:

>On Tue, 2005-02-15 at 09:55 +0100, Helge Hafting wrote:
>[...]
>  
>
>>The init-script dependencies are specifies already - at least on debian.
>>    
>>
>
>These are not dependencies but "only" the sequence of startup (and it is
>not only Debian but also Fedora/RedHat, SuSE, Mandrake and probably all
>except Gentoo).
>  
>
Yes, it is a sequence.  It it derived from real dependencies though,
where nondependent stuff have the same number.

>Yuo get a much stricter ordering and sorting (and thus much simpler to
>implement in a shell script).
>  
>
Correct.

>This would be a win (especially if the numbers are tweked to tune this)
>with a relatively small effort.
>However for real dependencies and parallelism you want the info similar
>to creat a Makefile from it (i.e. the explicit dependency from service X
>to service Y). As a consequence you can get rid of the numbers (since
>they are not needed any more).
>  
>
Now that is a really good idea.  Init could simply run "make -j init2" to
enter runlevel 2.  A suitable makefile would list all dependencies, and
of course the targets needed for "init2", "init3" and so on.

It might not be that much work either.  Parallel make exists already, 
and the
first attempt at a makefile could simply implement the current sequence that
is known to work. Then the tweaking comes. :-)

Helge Hafting

 

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Paulo Marques]

Helge Hafting wrote:
> Now that is a really good idea.  Init could simply run "make -j init2" to
> enter runlevel 2.  A suitable makefile would list all dependencies, and
> of course the targets needed for "init2", "init3" and so on.
> 
> It might not be that much work either.  Parallel make exists already, 
> and the
> first attempt at a makefile could simply implement the current sequence 
> that
> is known to work. Then the tweaking comes. :-)

Someone already mentioned this work before on this thread. I just 
googled for it and found the link:

http://www-106.ibm.com/developerworks/linux/library/l-boot.html?ca=dgr-lnxw04BootFaster

-- 
Paulo Marques - www.grupopie.com

All that is necessary for the triumph of evil is that good men do nothing.
Edmund Burke (1729 - 1797)

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Bernd Petrovitsch]

On Tue, 2005-02-15 at 14:20 +0100, Helge Hafting wrote:
> Bernd Petrovitsch wrote:
> >On Tue, 2005-02-15 at 09:55 +0100, Helge Hafting wrote:
[...]
> >These are not dependencies but "only" the sequence of startup (and it is
> >not only Debian but also Fedora/RedHat, SuSE, Mandrake and probably all
> >except Gentoo).
> >  
> Yes, it is a sequence.  It it derived from real dependencies though,
> where nondependent stuff have the same number.

Yes, of course. Sorry if that wasn't clear.

> >However for real dependencies and parallelism you want the info similar
> >to creat a Makefile from it (i.e. the explicit dependency from service X
> >to service Y). As a consequence you can get rid of the numbers (since
> >they are not needed any more).
> >  
> Now that is a really good idea.  Init could simply run "make -j init2" to

Actually I just used it to illustrate the idea.
As with Gentoo, I'm not a guru there so other people must comment if
Gentoo actually starts services in parallel.

> enter runlevel 2.  A suitable makefile would list all dependencies, and
> of course the targets needed for "init2", "init3" and so on.
> 
> It might not be that much work either.  Parallel make exists already, 
> and the
> first attempt at a makefile could simply implement the current sequence that
> is known to work. Then the tweaking comes. :-)

You just have to cope with the potentially parallel output reliably
especially in error cases (which is IMHO basically the most work).

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Miquel van Smoorenburg]

In article <4211F706.4030104@aitel.hist.no>,
Helge Hafting  <helge.hafting@aitel.hist.no> wrote:
>Bernd Petrovitsch wrote:
>>This would be a win (especially if the numbers are tweked to tune this)
>>with a relatively small effort.
>>However for real dependencies and parallelism you want the info similar
>>to creat a Makefile from it (i.e. the explicit dependency from service X
>>to service Y). As a consequence you can get rid of the numbers (since
>>they are not needed any more).
>>
>Now that is a really good idea.  Init could simply run "make -j init2" to
>enter runlevel 2.  A suitable makefile would list all dependencies, and
>of course the targets needed for "init2", "init3" and so on.

It's not too hard to script it using 'tsort', either.

The hard part is getting all the dependencies of the scripts right.
And once you've done that, to _keep_ them right.

Now how do you implement that on a Debian system that is package-wise
somewhere between potato and sarge ... (yes, I've encountered those).

Solveable, not trivial.

Mike.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Jim Crilly]

Lee Revell said the following:
> The reason I marked by response OT is that the time from power on to
> userspace does not seem to be a big problem.  It's the amount of time
> from user space to presenting a login prompt that's way too long.  My
> distro (Debian) runs all the init scripts one at a time, and GDM is the
> last thing that gets run.  There is just no reason for this.  We should
> start X and initialize the display and get the login prompt up there
> ASAP, and let the system acquire the DHCP lease and start sendmail and
> apache and get the date from the NTP server *in the background while I
> am logging in*.  It's not rocket science.

That is one of the "features" that I hate most about Windows. All you really do 
is move the delays around. For instance, my Windows machine at work is joined 
to a domain and has the Novell NetWare client installed for NDS auth (don't 
ask) so when the GINA comes up, I hit Ctrl+Alt+Del, enter my password and wait 
staring at an hourglass while the GINA waits for network connectivity to come 
up so that it can authenticate me. And even if authentication was local and I 
got logged in right away, chances are good that I have mapped drives or 
printers on the network that I will have to wait for during the login process 
anyway.

I agree boot up is too slow and that some things should be started in the 
background, but not things that are required for the main purpose of the box to 
work properly, what should be started sync and what should be async is a hard 
decision IMO. Right now I use swsusp2 to work around this, it takes less time 
to resume my session + 500M of file cache than it does to boot manually and 
start all of my apps back up, but obviously that's not a real solution.

> 
> Lee
> 

Jim.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Nigel Cunningham]

Ah Jim.

On Tue, 2005-02-15 at 14:38, Jim Crilly wrote:
> I agree boot up is too slow and that some things should be started in the 
> background, but not things that are required for the main purpose of the box to 
> work properly, what should be started sync and what should be async is a hard 
> decision IMO. Right now I use swsusp2 to work around this, it takes less time 
> to resume my session + 500M of file cache than it does to boot manually and 

You warmed my heart until...

> start all of my apps back up, but obviously that's not a real solution.

Why not? :> I guess you mean to the problem of slow booting in the first
place - I would agree with you there, but is there are reason why we
should have booting being the norm instead of normally suspending and
resuming, and only rebooting for new kernels/hardware/etc.

Regards,

Nigel

-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Jim Crilly]

Nigel Cunningham said the following:
> You warmed my heart until...

Good to know someone reads my email =)

> Why not? :> I guess you mean to the problem of slow booting in the first
> place - I would agree with you there, but is there are reason why we
> should have booting being the norm instead of normally suspending and
> resuming, and only rebooting for new kernels/hardware/etc.

Don't get me wrong, I would go nuts without swsusp2 on my notebook and I don't 
see why that shouldn't be a valid avenue to pursue; even for servers it doesn't 
seem like a terribly bad idea. But for me it only works on 1 out of my 4 
machines. The 3 non-working machines have their root and swap on SCSI devices 
and to top it off 2 of them are non-x86 architectures.

Another issue would be dual-booting, which a lot of people still do for some 
strange reason. At least I had noticed that Windows tends to have problems when 
filesystems it had mounted before the hibernation are altered while it's not 
running. I'm not sure if similar issues would apply to Linux, hell I'm not even 
sure if it still applies to Windows because that was so long ago that I had 
noticed.

> 
> Regards,
> 
> Nigel
> 

Jim.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Nigel Cunningham]

Hi.

On Tue, 2005-02-15 at 17:15, Jim Crilly wrote:
> Nigel Cunningham said the following:
> > You warmed my heart until...
> 
> Good to know someone reads my email =)
> 
> > Why not? :> I guess you mean to the problem of slow booting in the first
> > place - I would agree with you there, but is there are reason why we
> > should have booting being the norm instead of normally suspending and
> > resuming, and only rebooting for new kernels/hardware/etc.
> 
> Don't get me wrong, I would go nuts without swsusp2 on my notebook and I don't 
> see why that shouldn't be a valid avenue to pursue; even for servers it doesn't 
> seem like a terribly bad idea. But for me it only works on 1 out of my 4 
> machines. The 3 non-working machines have their root and swap on SCSI devices 
> and to top it off 2 of them are non-x86 architectures.

Okay. So it's a lack of hardware support then. I need to bug people to
get SCSI PM support working, and to lend me non-x86 and x86-64 some more
:>

> Another issue would be dual-booting, which a lot of people still do for some 
> strange reason. At least I had noticed that Windows tends to have problems when 
> filesystems it had mounted before the hibernation are altered while it's not 
> running. I'm not sure if similar issues would apply to Linux, hell I'm not even 
> sure if it still applies to Windows because that was so long ago that I had 
> noticed.

Suspend certainly doesn't like filesystems being mounted under it - it
writes the image without remounting ro or unmounting. I think I saw a
patch Tim had that remounted ro, but you still have to be careful as the
saved memory contains a picture of the state of superblocks and so on.

Regards,

Nigel

-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Lee Revell]

On Tue, 2005-02-15 at 01:15 -0500, Jim Crilly wrote:
> Another issue would be dual-booting, which a lot of people still do for some 
> strange reason.

Um, to reverse engineer Windows drivers?

Lee

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[jlnance]

On Mon, Feb 14, 2005 at 08:17:25PM -0500, Lee Revell wrote:

> from user space to presenting a login prompt that's way too long.  My
> distro (Debian) runs all the init scripts one at a time, and GDM is the
> last thing that gets run.  There is just no reason for this.  We should
> start X and initialize the display and get the login prompt up there
> ASAP, and let the system acquire the DHCP lease and start sendmail and
> apache and get the date from the NTP server *in the background while I
> am logging in*.  It's not rocket science.

This is debatable.  Windows does something like this.  It really annoys
me that I will get a windows desktop very quickly after logging in
but that I can't do anything with it until some mystrey initialization
takes place.  I would hate to be able to log into my linux machine but
not be able to check email for the first 15 seconds.

Jim

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Chris Larson]

* jlnance@unity.ncsu.edu (jlnance@unity.ncsu.edu) wrote:
> On Mon, Feb 14, 2005 at 08:17:25PM -0500, Lee Revell wrote:
> 
> > from user space to presenting a login prompt that's way too long.  My
> > distro (Debian) runs all the init scripts one at a time, and GDM is the
> > last thing that gets run.  There is just no reason for this.  We should
> > start X and initialize the display and get the login prompt up there
> > ASAP, and let the system acquire the DHCP lease and start sendmail and
> > apache and get the date from the NTP server *in the background while I
> > am logging in*.  It's not rocket science.
> 
> This is debatable.  Windows does something like this.  It really annoys
> me that I will get a windows desktop very quickly after logging in
> but that I can't do anything with it until some mystrey initialization
> takes place.  I would hate to be able to log into my linux machine but
> not be able to check email for the first 15 seconds.

Wouldn't it be sufficient to have an applet in your UI (or dialog,
depending on your preference), which communicates with init and displays
the final initialization steps?  Don't check your email until it says it has
started the services for email.
--
Chris Larson - kergoth at handhelds dot org
Linux Software Systems Engineer - clarson at ti dot com
Core Developer/Architect - BitBake, OpenEmbedded, OpenZaurus

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Jim Crilly]

> 
> Wouldn't it be sufficient to have an applet in your UI (or dialog,
> depending on your preference), which communicates with init and displays
> the final initialization steps?  Don't check your email until it says it has
> started the services for email.

So now instead of watching the boot messages or bootsplash you want to watch 
an icon on the task bar? In both cases you're just sitting and waiting on 
things to start, so why is waiting in one place better than another?

> --
> Chris Larson - kergoth at handhelds dot org
> Linux Software Systems Engineer - clarson at ti dot com
> Core Developer/Architect - BitBake, OpenEmbedded, OpenZaurus

Jim.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Helge Hafting]

On Thu, Feb 17, 2005 at 01:37:09PM -0500, jlnance@unity.ncsu.edu wrote:
> On Mon, Feb 14, 2005 at 08:17:25PM -0500, Lee Revell wrote:
> 
> This is debatable.  Windows does something like this.  It really annoys
> me that I will get a windows desktop very quickly after logging in
> but that I can't do anything with it until some mystrey initialization
> takes place.  I would hate to be able to log into my linux machine but
> not be able to check email for the first 15 seconds.

Well, this will depend on your email server (pop? imap? other?)
being up.  Is it local on your machine, or external?  Either way,
being able to launch an email client (or some "new mail" notification
app) shouldn't be a problem.  It will simply not notice new mail until
the mail service comes up - but it won't fail.  It'll be as if the
mail arrived a little later.

Helge Hafting

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Jim Crilly]

Helge Hafting wrote:
> 
> 
> Well, this will depend on your email server (pop? imap? other?)
> being up.  Is it local on your machine, or external?  Either way,
> being able to launch an email client (or some "new mail" notification
> app) shouldn't be a problem.  It will simply not notice new mail until
> the mail service comes up - but it won't fail.  It'll be as if the
> mail arrived a little later.

Sure it can fail, when you start it up it'll most likely fail to login to 
the mail server, whether it's local or not, if certain services aren't 
already started. If you're using local direct access via maildir or mbox, 
that'll work fine but most people access remote server for their mail.

> 
> Helge Hafting

Jim.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Helge Hafting]

On Sat, Feb 19, 2005 at 12:56:25AM -0500, Jim Crilly wrote:
> Helge Hafting wrote:
> >
> >
> >Well, this will depend on your email server (pop? imap? other?)
> >being up.  Is it local on your machine, or external?  Either way,
> >being able to launch an email client (or some "new mail" notification
> >app) shouldn't be a problem.  It will simply not notice new mail until
> >the mail service comes up - but it won't fail.  It'll be as if the
> >mail arrived a little later.
> 
> Sure it can fail, when you start it up it'll most likely fail to login to 
> the mail server, whether it's local or not, if certain services aren't 
> already started. If you're using local direct access via maildir or mbox, 
> that'll work fine but most people access remote server for their mail.
> 
No problem with the remote server, it does not depend on the local boot process.
The mail program connects directly to the remote server, all you need is the
network and it comes up so fast, it will come up way before X in a parallel boot.

Helge Hafting

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Jim Crilly]

Helge Hafting wrote:
> No problem with the remote server, it does not depend on the local boot process.
> The mail program connects directly to the remote server, all you need is the
> network and it comes up so fast, it will come up way before X in a parallel boot.

Depends on the implementation and what's required for network connectivity 
to the remote server. When I said that I was talking about the method that 
Lee Revell talked about, where he said "We should start X and initialize the 
display and get the login prompt up there ASAP, and let the system acquire 
the DHCP lease and start sendmail and apache and get the date from the NTP 
server *in the background while I am logging in*.  It's not rocket science". 
Obviously, if the network is started synchronously it won't matter.

> Helge Hafting

Jim.

Re: [OT] speeding boot process (was Re: [ANNOUNCE] hotplug-ng 001 release)

[Paolo Ciarrocchi]

On Mon, 14 Feb 2005 18:45:20 -0500, Lee Revell <rlrevell@joe-job.com> wrote:
> On Mon, 2005-02-14 at 15:21 -0800, Roland Dreier wrote:
> >     Lee> I don't see why so much effort goes into improving boot time
> >     Lee> on the kernel side when the most obvious user space problem
> >     Lee> is ignored.
> >
> > How much of a win is it to run init scripts in parallel?  I seem to
> > recall seeing tests that show that it doesn't make much difference and
> > may even slow things down by causing more disk seeks as various things
> > start up at the same time and cause reads of different files to get
> > interleaved.
> >
> 
> This is why Windows XP reserves sapce at the beginning of the disk for
> the files read during the boot process and caches copies of them there.
> 
> But, I was referring more to things like GDM not being started until all
> the other init scripts are done.  Why not start it first, and let the
> network initialize while the user is logging in?
> 
> > On the other hand, hotplug is an area that real profiling of real
> > systems booting has identified as something that can be improved, and
> > Greg's hotplug-ng seems to be a step towards a measurable improvement.

Did anyone measure the improvement ?

-- 
Paolo <paolo dot ciarrocchi at gmail dot com>
msn: paolo407@hotmail.com
hello: ciarrop

Re: [ANNOUNCE] hotplug-ng 001 release

[Diego Calleja]

El Mon, 14 Feb 2005 18:04:00 -0500,
Lee Revell <rlrevell@joe-job.com> escribió:

> Last I heard Gentoo does not even do it by default.
> 
> I don't see why so much effort goes into improving boot time on the
> kernel side when the most obvious user space problem is ignored.


There's stuff that it could be done in the kernel to help improving those numbers,
IMHO.

xp logs all the io done the first two minutes after booting. The next time it boots
it tries to read all those files at once so the programs will find stuff in memory
instead of having to do lots of small seeks. Some people in the linux field have
got a list of the files used at startup and they've thrown it at a "readhead" script,
which seems to help but IMHO it's somewhat "hacky" compared with the 
xp's trick. xp also does that when you start a program, it saves a log of all the
io done and it preloads it efficiently at startup - it improves "cold-cache" loading
times a _lot_. I haven't seen any alternative for that in the linux world, and
being able to keep track of al the io done by a given process would fix that (some
people has put used printk's for that, but i think it can be done better)

Also, it analyzes all those io "logs" and defragments (in background every 3 days,
and with low load without the user noticing it) the disk according to the _use_ of the
systems. Linux kernel can keep a file unfragmented, but currently there's no way
linux can do decisions like "this system starts openoffice, so I'm going to move the
binaries to another place of the disk where they'll load faster" or "when X program
uses /lib/libfoo.so it also uses /lib/libbar.so, so I'm going to put those two together
in the disk because that will avoid seeks". Kernel only can keep a single file
unfragmented, but it doesn't know about how several files must be (un)fragmented
between them. Being able to defragment things seems to be the one fix that (even
mac os x does it)

Userspace is where the problem is, but it's not going to be fixed. Ever. If
something, it's going to be worse - it's how software works. And even if you make
openoffice "fast", you still could _improve_ things with the tricks described
above. Disks are too slow, and things like demand-loading executables generate
too many small seeks, and programs can't control demand-loading so I don't
think userspace is the only with work to do.

Re: [ANNOUNCE] hotplug-ng 001 release

[Lee Revell]

On Tue, 2005-02-15 at 00:43 +0100, Diego Calleja wrote:
> There's stuff that it could be done in the kernel to help improving those numbers,
> IMHO.
> 
> xp logs all the io done the first two minutes after booting. The next time it boots
> it tries to read all those files at once so the programs will find stuff in memory
> instead of having to do lots of small seeks.

Thanks for the detailed explanation.  ISTR hearing that some of the
distros are working on similar solutions.  Now this would be a big win,
as anyone who has seen how much faster XP boots than Win2K can tell you.
And would certainly require kernel support.

Of course resuming from suspend will always be faster than booting but
for the forseeable future we will have to reboot from time to time.  And
XP's boot time currently is way, way better than ours.  FWIW, OSX still
takes forever to boot so we are not the only ones with this problem.

I wonder if XP's solution is patented.

Lee

Re: [ANNOUNCE] hotplug-ng 001 release

[Diego Calleja]

El Tue, 15 Feb 2005 14:51:06 -0500,
Lee Revell <rlrevell@joe-job.com> escribió:


> Of course resuming from suspend will always be faster than booting but
> for the forseeable future we will have to reboot from time to time.  And
> XP's boot time currently is way, way better than ours.  FWIW, OSX still
> takes forever to boot so we are not the only ones with this problem.


What I wonder is if we can have a "process-to-disk" thing and use it to improve
other things. Some OSs implement this (DFbsd, for one), but I think we could
use it to do some cool things, ie: instead of closing gnome and restarting all
the apps when you login again, you could do something like "when you're
closing gnome, dump all the process' images to disk and restart all the process
when you login again". This way your desktop would be *always* in the
same state you left it (including things like the text buffer in your terminal). You
could use it to speed up some things ej: instead of loading openoffice, load a saved
image of a void document. Of course there're lots of problems, like what happens
if you change a file which was being used by a suspended process, disconnection
between app <-> xserver (x folks are working on thinks like that because of wireless
connections i think) , what happens if you update a library that a image is supposed to
use, can users "restart" images or just only root, etc but i think it'd be interesting to
discuss if it's feasible

(in the X world there's already some "signal" sent to programs, but if we were able
to do it by "sending a process' image to disk" it'd be much easier and cleaner IMHO)

Re: [ANNOUNCE] hotplug-ng 001 release

[Chris Friesen]

Diego Calleja wrote:
> Of course there're lots of problems, like what happens
> if you change a file which was being used by a suspended process, 

That one is easy.  Store a checksum of the file in use when you go to 
sleep  If on wakeup the checksum is different, pop up a window that says 
"the file *foo* has been modified by another application, do you want to 
reload?".

 > what happens if you update a library that a image is supposed to
> use

Same as updating it on a running system.  Don't do that unless you 
really know what you're doing.

Chris

Re: [ANNOUNCE] hotplug-ng 001 release

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On Tue, 15 Feb 2005 14:51:06 EST, Lee Revell said:

> I wonder if XP's solution is patented.

If it is, IBM's OS/360 and OS/VS1 and MVS had prior art way back in the 70's.
There were *plenty* of products that would look at the system call usage and
spit out an ordered load list for SYS1.LINKLIB and SYS1.LPALIB - so the idea of
machine-optimizing the list of things to cache for a fast startup is *not*
new.

I'd not be surprised to find out that somebody did something like that on the 7094 ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Optimizing disk-I/O [was Re: [ANNOUNCE] hotplug-ng 001 release]

[Linas Vepstas]

On Tue, Feb 15, 2005 at 12:43:29AM +0100, Diego Calleja was heard to remark:
> 
> Also, it analyzes all those io "logs" and defragments 

I dislike hearing/reading about what XP does, since its probably patented,
and I don't want that shadow hanging over Linux.

I assume that the following simple idea, obvious to any practictioner 
versed in the state of the art, is not patented or patentable:

> linux can do decisions like "this system starts openoffice, so I'm going to move the
> binaries to another place of the disk where they'll load faster" or "when X program
> uses /lib/libfoo.so it also uses /lib/libbar.so, so I'm going to put those two together
> in the disk because that will avoid seeks". 

Now I like this idea. It need not have anything to do with startup,
or with any particular program or distro whatsoever.  Rather, one 
would have a daemon keeping track of disk i/o patterns, and constantly 
trying to figure out if there is a rearrangement of the sectors on disk
that would minimize i/o seeks based on past uasge. 

The optimization routine could be some simulated annealing or 
genetic algorithm or whatever whiz-bang technique someone is into.
Just keep it running in the background, low priority, constantly...
This would give you the best "time weighted" disk access performance,
although it would potentially hurt boot times, since most users spend
most of thier time doing disk access other than booting ... 

--linas

Re: Optimizing disk-I/O [was Re: [ANNOUNCE] hotplug-ng 001 release]

[Adam Goode]

Mac OS X has a similar thing, with a pretty simple description of how
they do it:

http://developer.apple.com/technotes/tn/tn1150.html#HotFile



Adam



On Tue, 2005-02-15 at 13:56 -0600, Linas Vepstas wrote:
> On Tue, Feb 15, 2005 at 12:43:29AM +0100, Diego Calleja was heard to remark:
> > 
> > Also, it analyzes all those io "logs" and defragments 
> 
> I dislike hearing/reading about what XP does, since its probably patented,
> and I don't want that shadow hanging over Linux.
> 
> I assume that the following simple idea, obvious to any practictioner 
> versed in the state of the art, is not patented or patentable:
> 
> > linux can do decisions like "this system starts openoffice, so I'm going to move the
> > binaries to another place of the disk where they'll load faster" or "when X program
> > uses /lib/libfoo.so it also uses /lib/libbar.so, so I'm going to put those two together
> > in the disk because that will avoid seeks". 
> 
> Now I like this idea. It need not have anything to do with startup,
> or with any particular program or distro whatsoever.  Rather, one 
> would have a daemon keeping track of disk i/o patterns, and constantly 
> trying to figure out if there is a rearrangement of the sectors on disk
> that would minimize i/o seeks based on past uasge. 
> 
> The optimization routine could be some simulated annealing or 
> genetic algorithm or whatever whiz-bang technique someone is into.
> Just keep it running in the background, low priority, constantly...
> This would give you the best "time weighted" disk access performance,
> although it would potentially hurt boot times, since most users spend
> most of thier time doing disk access other than booting ... 
> 
> --linas
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Optimizing disk-I/O [was Re: [ANNOUNCE] hotplug-ng 001 release]

[Diego Calleja]

El Tue, 15 Feb 2005 15:46:56 -0500,
Adam Goode <adam@evdebs.org> escribió:

> Mac OS X has a similar thing, with a pretty simple description of how
> they do it:
> 
> http://developer.apple.com/technotes/tn/tn1150.html#HotFile

Also all those things are part of darwin i think (or IOW, we can look at how they
did it)

Re: Optimizing disk-I/O [was Re: [ANNOUNCE] hotplug-ng 001 release]

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

On Tue, 15 Feb 2005 13:56:14 CST, Linas Vepstas said:

> Now I like this idea. It need not have anything to do with startup,
> or with any particular program or distro whatsoever.  Rather, one 
> would have a daemon keeping track of disk i/o patterns, and constantly 
> trying to figure out if there is a rearrangement of the sectors on disk
> that would minimize i/o seeks based on past uasge. 

More prior art - a company called FDR made a disk compactor product for IBM's
OS series, and I seem to remember a SHARE (IBM mainframe user group) tape that
had a program to read the I/O trace table and generate an optimal "what goes
where" command stream for FDR's software.  Again a late 70s to early 80s thing.

(Probably not enough to be "prior art" by itself, but certainly goes towards
the "obviousness to a practitioner in the field" criteria - if *I* knew about it
as a junior sysadmin at a college in middle-of-nowhere upstate NY knew about
it in 1983.. ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [ANNOUNCE] hotplug-ng 001 release

[Bill Nottingham]

Greg KH (gregkh@suse.de) said: 
> I'd like to announce, yet-another-hotplug based userspace project:
> linux-ng.  This collection of code replaces the existing linux-hotplug
> package with very tiny, compiled executable programs, instead of the
> existing bash scripts.
> 
> It currently provides the following:
> 	- a /sbin/hotplug multiplexer.  Works identical to the existing
> 	  bash /sbin/hotplug.

How does this interact with current usage of udevsend as the hotplug
multiplexer?

Bill

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Thu, 2005-02-10 at 22:18 -0500, Bill Nottingham wrote:
> Greg KH (gregkh@suse.de) said: 
> > I'd like to announce, yet-another-hotplug based userspace project:
> > linux-ng.  This collection of code replaces the existing linux-hotplug
> > package with very tiny, compiled executable programs, instead of the
> > existing bash scripts.
> > 
> > It currently provides the following:
> > 	- a /sbin/hotplug multiplexer.  Works identical to the existing
> > 	  bash /sbin/hotplug.
> 
> How does this interact with current usage of udevsend as the hotplug
> multiplexer?

First off, not everyone wants to use udev (I know, horrible thought...)
This provides those people a solution to a "I want a tiny /sbin/hotplug"
problem.

Also, udevstart working as /sbin/hotplug is great for keeping things in
order, which is important during normal operation.  But during the boot
sequence, the odds of getting out-of-order events, or even remove
events, is somewhat limited.  So, this /sbin/hotplug replacement can
work in an initrd/initramfs image when udevstart would be overkill.

And finally, even if you do use udevstart to manager /sbin/hotplug
events, you still need a module autoloader program.  This package
provides executables for that problem, if you don't want to (or you
can't) use the existing linux-hotplug scripts.  udev will never do the
module loading logic, so there's no duplication in this case.

Hope this helps.  I do realize the whole hotplug process is getting a
bit complicated.  I hope to write up some good documentation on what all
is involved to help clear up some of the confusion that the combination
of udev, udevsend, udevd, hal, /etc/hotplug.d/, /etc/dev.d/,
and /sbin/hotplug have caused lately.

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Christian Bornträger]

On Friday 11 February 2005 07:46, Greg KH wrote:
> And finally, even if you do use udevstart to manager /sbin/hotplug
> events, you still need a module autoloader program.  This package
> provides executables for that problem, if you don't want to (or you
> can't) use the existing linux-hotplug scripts.  udev will never do the
> module loading logic, so there's no duplication in this case.

Greg,
the pci module autoloader is a real agent, which means it depends on having a 
hotplug event. Are you planning to support a scan for already present 
devices, like the pci.rc file in current hotplug solutions?

cheers

Christian

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 05:19:22PM +0100, Christian Borntr?ger wrote:
> On Friday 11 February 2005 07:46, Greg KH wrote:
> > And finally, even if you do use udevstart to manager /sbin/hotplug
> > events, you still need a module autoloader program.  This package
> > provides executables for that problem, if you don't want to (or you
> > can't) use the existing linux-hotplug scripts.  udev will never do the
> > module loading logic, so there's no duplication in this case.
> 
> Greg,
> the pci module autoloader is a real agent, which means it depends on having a 
> hotplug event. Are you planning to support a scan for already present 
> devices, like the pci.rc file in current hotplug solutions?

It's not only pci, but all types of busses need this kind of "coldplug"
functionality.  And yes, I have plans to provide that functionality in
this package too.

In fact, if anyone looking to contribute some well defined and easy to
test code... :)

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Erik Andersen]

On Fri Feb 11, 2005 at 09:01:44AM -0800, Greg KH wrote:
> It's not only pci, but all types of busses need this kind of "coldplug"
> functionality.  And yes, I have plans to provide that functionality in
> this package too.
> 
> In fact, if anyone looking to contribute some well defined and easy to
> test code... :)

The pcimodules patch to pciutils does this sortof coldplug device
scanning for pci devices:
http://www.linuxfromscratch.org/patches/downloads/pciutils/pciutils-2.1.11-pcimodules-1.patch

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 12:01:54PM -0700, Erik Andersen wrote:
> On Fri Feb 11, 2005 at 09:01:44AM -0800, Greg KH wrote:
> > It's not only pci, but all types of busses need this kind of "coldplug"
> > functionality.  And yes, I have plans to provide that functionality in
> > this package too.
> > 
> > In fact, if anyone looking to contribute some well defined and easy to
> > test code... :)
> 
> The pcimodules patch to pciutils does this sortof coldplug device
> scanning for pci devices:
> http://www.linuxfromscratch.org/patches/downloads/pciutils/pciutils-2.1.11-pcimodules-1.patch

Yes, but that uses the modules.pcimap files, which we want to get rid of
someday.  It also uses the /proc/pci interface instead of sysfs, so it
probably doesn't handle machines with pci domains very well...

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Erik van Konijnenburg]

On Fri, Feb 11, 2005 at 11:23:23AM -0800, Greg KH wrote:
> On Fri, Feb 11, 2005 at 12:01:54PM -0700, Erik Andersen wrote:
> > On Fri Feb 11, 2005 at 09:01:44AM -0800, Greg KH wrote:
> > > It's not only pci, but all types of busses need this kind of "coldplug"
> > > functionality.  And yes, I have plans to provide that functionality in
> > > this package too.
> > > 
> > > In fact, if anyone looking to contribute some well defined and easy to
> > > test code... :)
> > 
> > The pcimodules patch to pciutils does this sortof coldplug device
> > scanning for pci devices:
> > http://www.linuxfromscratch.org/patches/downloads/pciutils/pciutils-2.1.11-pcimodules-1.patch
> 
> Yes, but that uses the modules.pcimap files, which we want to get rid of
> someday.  It also uses the /proc/pci interface instead of sysfs, so it
> probably doesn't handle machines with pci domains very well...

Could you give pointers to the "get rid of modules.pcimap" discussion?

Thanks.
Erik

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 10:37:31PM +0100, Erik van Konijnenburg wrote:
> On Fri, Feb 11, 2005 at 11:23:23AM -0800, Greg KH wrote:
> > On Fri, Feb 11, 2005 at 12:01:54PM -0700, Erik Andersen wrote:
> > > On Fri Feb 11, 2005 at 09:01:44AM -0800, Greg KH wrote:
> > > > It's not only pci, but all types of busses need this kind of "coldplug"
> > > > functionality.  And yes, I have plans to provide that functionality in
> > > > this package too.
> > > > 
> > > > In fact, if anyone looking to contribute some well defined and easy to
> > > > test code... :)
> > > 
> > > The pcimodules patch to pciutils does this sortof coldplug device
> > > scanning for pci devices:
> > > http://www.linuxfromscratch.org/patches/downloads/pciutils/pciutils-2.1.11-pcimodules-1.patch
> > 
> > Yes, but that uses the modules.pcimap files, which we want to get rid of
> > someday.  It also uses the /proc/pci interface instead of sysfs, so it
> > probably doesn't handle machines with pci domains very well...
> 
> Could you give pointers to the "get rid of modules.pcimap" discussion?

Search linux-kernel for it.  It's come up many times between me and
Rusty Russell.  He's basically waiting for the hotplug scripts (or
whatever) to be fixed up enough to not need them anymore.

Do you have some other need for them?

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Erik van Konijnenburg]

On Fri, Feb 11, 2005 at 01:49:57PM -0800, Greg KH wrote:
> On Fri, Feb 11, 2005 at 10:37:31PM +0100, Erik van Konijnenburg wrote:
> > On Fri, Feb 11, 2005 at 11:23:23AM -0800, Greg KH wrote:
> > > On Fri, Feb 11, 2005 at 12:01:54PM -0700, Erik Andersen wrote:
> > > > On Fri Feb 11, 2005 at 09:01:44AM -0800, Greg KH wrote:
> > > > > It's not only pci, but all types of busses need this kind of "coldplug"
> > > > > functionality.  And yes, I have plans to provide that functionality in
> > > > > this package too.
> > > > > 
> > > > > In fact, if anyone looking to contribute some well defined and easy to
> > > > > test code... :)
> > > > 
> > > > The pcimodules patch to pciutils does this sortof coldplug device
> > > > scanning for pci devices:
> > > > http://www.linuxfromscratch.org/patches/downloads/pciutils/pciutils-2.1.11-pcimodules-1.patch
> > > 
> > > Yes, but that uses the modules.pcimap files, which we want to get rid of
> > > someday.  It also uses the /proc/pci interface instead of sysfs, so it
> > > probably doesn't handle machines with pci domains very well...
> > 
> > Could you give pointers to the "get rid of modules.pcimap" discussion?
> 
> Search linux-kernel for it.  It's come up many times between me and
> Rusty Russell.  He's basically waiting for the hotplug scripts (or
> whatever) to be fixed up enough to not need them anymore.
> 
> Do you have some other need for them?

Yep, I'm working on a proof of concept to show how modules.pcimap
can be used to build a more reliable mkinitrd.  The interesting part about
modules.pcimap is that it can not only describe what's in the running
kernel, but also what's in the kernel about to be installed.

For an old version of the idea, see

	http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=284294

Regards,
Erik

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 11:06:57PM +0100, Erik van Konijnenburg wrote:
> For an old version of the idea, see
> 
> 	http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=284294

Very nice stuff.  Ok, that's a good reason not to get rid of these
files, although they can be generated on the fly from the modules
themselves (like depmod does it.)

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Ingo Oeser]

Hi,

Greg KH write:
> Very nice stuff.  Ok, that's a good reason not to get rid of these
> files, although they can be generated on the fly from the modules
> themselves (like depmod does it.)

Time to resurrect modinfo? ;-)
Didn't we plan to get rid of that, too?

If we like to use information from modules, there should be a scriptable 
tool to extract this kind of information, otherwise it will be a bitch to 
maintain those tools.


Regards

Ingo Oeser

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Sat, Feb 12, 2005 at 01:48:49AM +0100, Ingo Oeser wrote:
> Hi,
> 
> Greg KH write:
> > Very nice stuff.  Ok, that's a good reason not to get rid of these
> > files, although they can be generated on the fly from the modules
> > themselves (like depmod does it.)
> 
> Time to resurrect modinfo? ;-)
> Didn't we plan to get rid of that, too?

Not that I know of, modinfo works great for me here :)

> If we like to use information from modules, there should be a scriptable 
> tool to extract this kind of information, otherwise it will be a bitch to 
> maintain those tools.

modinfo works well for me in this manner, it doesn't for you?

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Olivier Galibert]

On Thu, Feb 10, 2005 at 04:40:33PM -0800, Greg KH wrote:
> 	- autoload programs for usb, scsi, and pci modules.  These
> 	  programs determine what module needs to be loaded when the
> 	  kernel emits a hotplug event for these types of devices.  This
> 	  works just like the existing linux-hotplug scripts, with a few
> 	  exceptions.

firmware?

  OG.

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 10:52:37AM +0100, Olivier Galibert wrote:
> On Thu, Feb 10, 2005 at 04:40:33PM -0800, Greg KH wrote:
> > 	- autoload programs for usb, scsi, and pci modules.  These
> > 	  programs determine what module needs to be loaded when the
> > 	  kernel emits a hotplug event for these types of devices.  This
> > 	  works just like the existing linux-hotplug scripts, with a few
> > 	  exceptions.
> 
> firmware?

On the todo list :)

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

Greg KH wrote:
> I'd like to announce, yet-another-hotplug based userspace project:
> linux-ng.  This collection of code replaces the existing linux-hotplug
> package with very tiny, compiled executable programs, instead of the
> existing bash scripts.
>

cpio is running to setup a test partition.

But one question: This is yet another package with its
own private copy of klibc. Whats the reason behind this
non-modular approach?


Regards

Harri

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Fri, Feb 11, 2005 at 09:06:40PM +0100, Harald Dunkel wrote:
> Greg KH wrote:
> >I'd like to announce, yet-another-hotplug based userspace project:
> >linux-ng.  This collection of code replaces the existing linux-hotplug
> >package with very tiny, compiled executable programs, instead of the
> >existing bash scripts.
> >
> 
> cpio is running to setup a test partition.
> 
> But one question: This is yet another package with its
> own private copy of klibc. Whats the reason behind this
> non-modular approach?

Because we don't have an easy way yet to build against a copy of klibc
on a system?  For right now, it's the simplest way to ensure that it
works for everyone, once klibc moves into the kernel tree I can remove
it from udev and hotplug-ng.

Is it causing problems for you?

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Greg KH wrote:
>
> Because we don't have an easy way yet to build against a copy of klibc
> on a system?  For right now, it's the simplest way to ensure that it
> works for everyone, once klibc moves into the kernel tree I can remove
> it from udev and hotplug-ng.
>

If it is not possible to use klibc together with a non-Linux
system (e.g. FreeBSD or Mach), then I would suggest to make
klibc an optional kernel patch and drop it from udev and
hotplug.

> Is it causing problems for you?
>

Some months ago I had contributed a patch to add an install
target to the klibc Makefiles. I just wonder why it has been
ignored.


Regards

Harri

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Sat, Feb 12, 2005 at 09:30:54AM +0100, Harald Dunkel wrote:
> Greg KH wrote:
> >
> >Because we don't have an easy way yet to build against a copy of klibc
> >on a system?  For right now, it's the simplest way to ensure that it
> >works for everyone, once klibc moves into the kernel tree I can remove
> >it from udev and hotplug-ng.
> >
> 
> If it is not possible to use klibc together with a non-Linux
> system (e.g. FreeBSD or Mach), then I would suggest to make
> klibc an optional kernel patch and drop it from udev and
> hotplug.

But it is not possible to use udev or hotplug-ng on a non-Linux system,
right?

As far as "optional kernel patch"?  What do you mean?  People are
working on adding klibc to the main kernel tree, nothing optional about
that.

> >Is it causing problems for you?
> >
> 
> Some months ago I had contributed a patch to add an install
> target to the klibc Makefiles. I just wonder why it has been
> ignored.

Don't know, I'm not in charge of klibc development.  Why not ask on the
klibc mailing list?

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Greg KH wrote:
> On Sat, Feb 12, 2005 at 09:30:54AM +0100, Harald Dunkel wrote:
>
>>
>>If it is not possible to use klibc together with a non-Linux
>>system (e.g. FreeBSD or Mach), then I would suggest to make
>>klibc an optional kernel patch and drop it from udev and
>>hotplug.
>
>
> But it is not possible to use udev or hotplug-ng on a non-Linux system,
> right?
>

Thats not the point. The point is to remove the copy of the
klibc sources from packages like udev and hotplug-ng and
to use the existing klibc sources or binaries on the target
system instead. Just to keep it modular.

> As far as "optional kernel patch"?  What do you mean?  People are
> working on adding klibc to the main kernel tree, nothing optional about
> that.
>

I do not know the internals of klibc that much. Is it possible
to use klibc on non-Linux systems, e.g. on Mach or FreeBSD?
Maybe by adding some #ifdefs to klibc's kernel interface?

If yes, then making klibc an integrated part of the Linux
kernel source tree and dropping the independent development
tree would be a restriction to the use of klibc.


AFAIK the plan for initramfs is to move as much functionality
as possible from kernel to user space. Why not do the same
thing for the sources? Everything that is supposed to be run
in user space could be removed from the kernel source tree
and managed seperately, either in a set of userspace modules
like klibc, hotplug, udev, initramfs, etc., or in a monolithic
"userspace-tools" source tree.

Surely klibc belongs to the user space.


Regards

Harri

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [ANNOUNCE] hotplug-ng 001 release

[Greg KH]

On Tue, Feb 15, 2005 at 06:39:37AM +0100, Harald Dunkel wrote:
> Greg KH wrote:
> >On Sat, Feb 12, 2005 at 09:30:54AM +0100, Harald Dunkel wrote:
> >
> >>
> >>If it is not possible to use klibc together with a non-Linux
> >>system (e.g. FreeBSD or Mach), then I would suggest to make
> >>klibc an optional kernel patch and drop it from udev and
> >>hotplug.
> >
> >
> >But it is not possible to use udev or hotplug-ng on a non-Linux system,
> >right?
> >
> 
> Thats not the point. The point is to remove the copy of the
> klibc sources from packages like udev and hotplug-ng and
> to use the existing klibc sources or binaries on the target
> system instead. Just to keep it modular.

Again, my point is I'll take klibc out of the udev and hotplug-ng trees
when it is actually on people's systems because it is in the kernel
source tree.  Until that happens I'll continue to keep it in the udev
and hotplug-ng trees, ok?

> >As far as "optional kernel patch"?  What do you mean?  People are
> >working on adding klibc to the main kernel tree, nothing optional about
> >that.
> >
> 
> I do not know the internals of klibc that much. Is it possible
> to use klibc on non-Linux systems, e.g. on Mach or FreeBSD?
> Maybe by adding some #ifdefs to klibc's kernel interface?

I don't know, and I really don't care :)

> If yes, then making klibc an integrated part of the Linux
> kernel source tree and dropping the independent development
> tree would be a restriction to the use of klibc.

Huh?  You are free to take klibc and do whatever you want to with it
based on the license it has.  No one is restricting you from doing that,
right?

> AFAIK the plan for initramfs is to move as much functionality
> as possible from kernel to user space.

Yes.

> Why not do the same
> thing for the sources? Everything that is supposed to be run
> in user space could be removed from the kernel source tree
> and managed seperately, either in a set of userspace modules
> like klibc, hotplug, udev, initramfs, etc., or in a monolithic
> "userspace-tools" source tree.

Because we still want to actually be able to boot a kernel, right?  :)

Let's just get the early boot stuff building and working properly, and
then we'll worry about ripping the stuff out of the kernel tree then.

> Surely klibc belongs to the user space.

Hm, like the other things in the kernel source tree that are also only
in userspace?  :)

thanks,

greg k-h

Re: [ANNOUNCE] hotplug-ng 001 release

[Michal Rokos]

Hello,

I was unable to compile hotplug-ng against klibc until this patch went
in:

--- /dev/null   2005-02-14 09:23:10.000000000 +0100
+++ hotplug-ng/klibc/include/features.h 2005-02-11 16:18:35.000000000
+0100
@@ -0,0 +1,5 @@
+#ifndef        _FEATURES_H
+#define        _FEATURES_H     1
+
+#endif /* features.h  */
+

My system is debian (sid);
gcc version 3.3.5 (Debian 1:3.3.5-8)
libc6-2.3.2.ds1-20

The same goes for udev + klibc.

Or is there other trick?

Michal

PS: Error message was:
Precompiling hotplug.c:                                               [ERROR]
gcc -pipe -DLOG -Os -fomit-frame-pointer -D_GNU_SOURCE -Wall -nostdinc 
-mregpar
m=3 -DREGPARM=3 -march=i386 -Os -g -falign-functions=0 -falign-jumps=0 
-falign-
loops=0 -D__KLIBC__ -fno-builtin-printf 
-I/home/michal/WORK/devel/bk/hotplug-ng
/klibc_fixups 
-include /home/michal/WORK/devel/bk/hotplug-ng/klibc_fixups/klibc
_fixups.h -I/home/michal/WORK/devel/bk/hotplug-ng/klibc/include 
-I/home/michal/
WORK/devel/bk/hotplug-ng/klibc/include/arch/i386 
-I/home/michal/WORK/devel/bk/h
otplug-ng/klibc/include/bits32 -I/usr/lib/gcc-lib/i486-linux/3.3.5/include 
-I/l
ib/modules/2.6.11-rc4-mr/build/include 
-I/home/michal/WORK/devel/bk/hotplug-ng/
libsysfs/sysfs -I/home/michal/WORK/devel/bk/hotplug-ng/libsysfs -c -o 
hotplug.o
 hotplug.c
In file included 
from /lib/modules/2.6.11-rc4-mr/build/include/linux/posix_type
s.h:47,
                 from /home/michal/WORK/devel/bk/hotplug-ng/klibc/include/sys/t
ypes.h:15,
                 from /home/michal/WORK/devel/bk/hotplug-ng/klibc/include/unist
d.h:11,
                 from /home/michal/WORK/devel/bk/hotplug-ng/klibc_fixups/klibc_
fixups.h:7,
                 from <command line>:8:
/usr/lib/gcc-lib/i486-linux/3.3.5/include/asm/posix_types.h:13:22: features.h:
No such file or directory
make: *** [hotplug.o] Error 1

Re: [ANNOUNCE] hotplug-ng 001 release

[Michael Tokarev]

> On Fri, Feb 11, 2005 at 11:46:27PM +0300, Roman Kagan wrote:
> > On Fri, Feb 11, 2005 at 11:36:27AM -0800, Greg KH wrote:
> > > Do one thing, and do it well.
> > > 
> > > udev is for naming devices, not loading modules.
> > 
> > Well currently udev is doing a lot more: serializing, waiting for sysfs
> > dir to appear, etc.  Not that I disagree with your first statement,
> > though.
> > 
> > > Why, each type of autoload program needs to know how to handle the
> > > different bus types.  So again, a single program doing a single thing
> > > well.
> > 
> > udev already pokes enough in sysfs and has quite a lot of
> > subsystem-specific knowledge, so adding module name generation is not
> > too much of extra functionality.
> 
> If you look at the hotplug scripts, they really don't need to touch
> sysfs at all.  (Note that the scsi one does, but I think we can fix the
> scsi hotplug event to solve this issue...)
> 
> So module autoload has really nothing to do with sysfs.

I tend to disagree, with two points.

1.  Well yes, module autoloading may be done without sysfs just fine
 (but see below).  Yet, module autoloading isn't the only "usage" of
 hotplug subsystem, right?  Ie, in 99.9% of times when a hotplug
 event is emitted, there will be other scripts executed to handle
 the event, and that scripts will depend on sysfs information.

2. And even for module autoloading, $DEVPATH/driver symlink is very-very
 handy, as it's the only (and good) way to check whenever we really want
 to mess with all the module stuff for this device at all: if the symlink
 is present, just do nothing.  Modprobe - if we're moving this functionality
 into modprobe which seems to be a good idea - while loading matching modules
 one by one, can check $DEVPATH/driver to see whenever it should stop walking
 the list.

/mjt