Wait for console to become available, v3.2

Wait for console to become available, v3.2

[David VomLehn]

Parallelization to improve boot times has been successful enough that race
conditions now exist between the init_post() open of /dev/console and
initialization of the console device. When this occurs, opening /dev/console
fails and any applications inherited from init have no standard in/out/error
devices. This is expected behavior if no console device is available, but
quite unfortunate in the case where the console is just a bit slow waking up.

Some buses, such as USB, offer no guarantees about how long it takes to
discover devices, so there is no reliable way to distinguish between a missing
console and a slow one.  The pragmatic approach taken in this patch is to
wait for a while to see if a console shows up, and just go on if it doesn't.
The default delay is 1000 msec (1 second). This value is conjured out of
thing air; any suggestions for a value that more closely approximates the
effective delays from the olden days before USB consoles starting failing are
more than welcome.

There is one new command line parameters:
consoledelay=msec	If msec is an integer, interpret it the maximum number
			of milliseconds to wait for a preferred console device
			to be registered. If it is the string "forever", wait
			indefinitely for the console device. In either case
			continue as soon as the console device is registered.

It could be argued that providing both a command line parameter and a Kconfig
option for the console wait time is over-engineering. There doesn't really
seem to be a reasonable way to determine a default, though, which drives the
Kconfig option. The selection of USB devices connected to the system can
only be known at runtime, which drives the need for a command line parameter.

History
v3.2	Use only a single new command line parameter by adding the possible
	value "forever".  Add message specifying the parameter to modify to
	increase the delay for console initialization
v3.1	Correct the abbreviation for milliseconds in
	Documentation/kernel-parameters.txt to be "ms", not "mS". Thanks to
	Peter Anvin for this.
v3	Increase the default delay to 1 second and add kernel command line
	parameters to override the default delay. Thanks to David Brownell for
	his helpful suggestions.
v2	Wait for the preferred console rather than any console. Make the
	delay interval a tunable.
v1	Initial version

Signed-off-by: David VomLehn <dvomlehn@cisco.com>
---
 Documentation/kernel-parameters.txt |    7 +++++++
 arch/mips/configs/powertv_defconfig |    2 +-
 init/Kconfig                        |   12 ++++++++++++
 kernel/printk.c                     |   35 +++++++++++++++++++++++++++++++++++
 4 files changed, 55 insertions(+), 1 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 6172e43..80a631f 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -507,6 +507,13 @@ and is between 256 and 4096 characters. It is defined in the file
 			console=brl,ttyS0
 		For now, only VisioBraille is supported.
 
+	consoledelay=ms	[KNL] If ms is an integer, wait up to ms milliseconds
+			for the preferred console to be registered, then
+			continue. Useful for systems where a console may not
+			be plugged in, such as for USB serial devices. To
+			wait indefinitely, specify the string "forever" for
+			the value of ms, i.e. consoledelay=forever.
+
 	coredump_filter=
 			[KNL] Change the default value for
 			/proc/<pid>/coredump_filter.
diff --git a/arch/mips/configs/powertv_defconfig b/arch/mips/configs/powertv_defconfig
index 7311e63..b4a046d 100644
--- a/arch/mips/configs/powertv_defconfig
+++ b/arch/mips/configs/powertv_defconfig
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.30-rc2
-# Thu Apr 16 11:29:44 2009
+# Fri Apr 17 12:38:42 2009
 #
 CONFIG_MIPS=y
 
diff --git a/init/Kconfig b/init/Kconfig
index 7be4d38..a04eba1 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -835,6 +835,18 @@ config PRINTK
 	  very difficult to diagnose system problems, saying N here is
 	  strongly discouraged.
 
+config PRINTK_CONSOLE_WAIT
+	int "Default number of milliseconds to wait for console device"
+	default 1000
+	help
+	  Some systems use console devices, such as USB serial devices, which
+	  may not be present or which may take an unspecified amount of time
+	  to be initialized. This setting is the default for the maximum number
+	  of milliseconds the system will wait for a console to be registered.
+	  This value can be overridden by the command line parameters
+	  consolewait, to wait forever, or consoledelay=msec, to give a
+	  different value for the wait.
+
 config BUG
 	bool "BUG() support" if EMBEDDED
 	default y
diff --git a/kernel/printk.c b/kernel/printk.c
index 5052b54..1edc5a3 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -105,6 +105,11 @@ static unsigned log_start;	/* Index into log_buf: next char to be read by syslog
 static unsigned con_start;	/* Index into log_buf: next char to be sent to consoles */
 static unsigned log_end;	/* Index into log_buf: most-recently-written-char + 1 */
 
+/* Definitions controlling the wait for a console device to be initialized */
+#define	CONSOLE_DELAY_FOREVER	(-1)
+static long console_delay = CONFIG_PRINTK_CONSOLE_WAIT; /* In milliseconds */
+static DECLARE_WAIT_QUEUE_HEAD(console_wq);
+
 /*
  *	Array of consoles built from command line options (console=)
  */
@@ -195,6 +200,21 @@ out:
 
 __setup("log_buf_len=", log_buf_len_setup);
 
+/*
+ * Set the delay, in milliseconds, to wait for the console.
+ * @str:	Pointer to the start of the delay value
+ * Returns one.
+ */
+static int __init consoledelay_setup(char *str)
+{
+	if (strcmp(str, "forever") == 0)
+		console_delay = CONSOLE_DELAY_FOREVER;
+	else
+		console_delay = simple_strtoul(str, NULL, 10);
+	return 1;
+}
+__setup("consoledelay=", consoledelay_setup);
+
 #ifdef CONFIG_BOOT_PRINTK_DELAY
 
 static unsigned int boot_delay; /* msecs delay after each printk during bootup */
@@ -1081,6 +1101,20 @@ struct tty_driver *console_device(int *index)
 	struct console *c;
 	struct tty_driver *driver = NULL;
 
+	/* The console device may not be initialized yet. We can either wait
+	 * forever if consolewait is specified on the command line, or wait
+	 * for some number of milliseconds if console_delay=msec is used */
+	if (console_delay == CONSOLE_DELAY_FOREVER)
+		wait_event(console_wq, preferred_console >= 0);
+
+	else if (wait_event_timeout(console_wq, preferred_console >= 0,
+		msecs_to_jiffies(console_delay)) == 0) {
+		pr_warning("No preferred console after waiting %ld msec; "
+			"continuing anyway\n", console_delay);
+		pr_info("Increase boot parameter console_delay to wait "
+			"longer\n");
+	}
+
 	acquire_console_sem();
 	for (c = console_drivers; c != NULL; c = c->next) {
 		if (!c->device)
@@ -1230,6 +1264,7 @@ void register_console(struct console *console)
 		spin_unlock_irqrestore(&logbuf_lock, flags);
 	}
 	release_console_sem();
+	wake_up_all(&console_wq);
 }
 EXPORT_SYMBOL(register_console);
 

Re: Wait for console to become available, v3.2

[Ingo Molnar]

* David VomLehn <dvomlehn@cisco.com> wrote:

> Parallelization to improve boot times has been successful enough 
> that race conditions now exist between the init_post() open of 
> /dev/console and initialization of the console device. When this 
> occurs, opening /dev/console fails and any applications inherited 
> from init have no standard in/out/error devices. This is expected 
> behavior if no console device is available, but quite unfortunate 
> in the case where the console is just a bit slow waking up.
> 
> Some buses, such as USB, offer no guarantees about how long it 
> takes to discover devices, so there is no reliable way to 
> distinguish between a missing console and a slow one.  The 
> pragmatic approach taken in this patch is to wait for a while to 
> see if a console shows up, and just go on if it doesn't. The 
> default delay is 1000 msec (1 second). This value is conjured out 
> of thing air; any suggestions for a value that more closely 
> approximates the effective delays from the olden days before USB 
> consoles starting failing are more than welcome.

hm, this really seems like a bad hack and a workaround to me and as 
such it is not really an acceptable solution.

The proper approach would be to use one of the async_synchronize*() 
facilities in kernel/async.c to properly order the opening of the 
console with device init.

Certain subsystems like storage (SCSI, libata, mount code and 
modules) has already been extended to this scheme.

So i think the right approach, if you want to speed up bootup, would 
be to extend the same concepts to console discovery, init and open 
methods.

	Ingo

Re: Wait for console to become available, v3.2

[David Brownell]

On Monday 20 April 2009, Ingo Molnar wrote:
> The proper approach would be to use one of the async_synchronize*() 
> facilities in kernel/async.c to properly order the opening of the 
> console with device init.

Stepping back a moment from "how" to make sure "what" is agreed on.

I think I see three scenarios here:

 - Classic PC or server, where there's a meaningful console;

 - Deeply embedded systems, where there isn't;

 - Development stages of "deeply embedded", where there *may* be one.

If that's correct, then async_synchronize() isn't a full answer...

I think a fair number of cases can be papered over with a serial
console with no hardware flow control, which isn't hooked up to
anything.  Maybe the board needs a test/development jig to get one;
without it, the "console" bits spill out into the aether.  Linux
can dump bits to ttyS0, which will act like /dev/null.

But the problem case here seems to be one where such un-hooked-up
serial ports are not realistic options.  Which is why the regression
in USB console functionality has been troublesome.


Is that correct?

- Dave

Re: Wait for console to become available, v3.2

[Ingo Molnar]

* David Brownell <david-b@pacbell.net> wrote:

> On Monday 20 April 2009, Ingo Molnar wrote:
>
> > The proper approach would be to use one of the 
> > async_synchronize*() facilities in kernel/async.c to properly 
> > order the opening of the console with device init.
> 
> Stepping back a moment from "how" to make sure "what" is agreed 
> on.
> 
> I think I see three scenarios here:
> 
>  - Classic PC or server, where there's a meaningful console;
> 
>  - Deeply embedded systems, where there isn't;
> 
>  - Development stages of "deeply embedded", where there *may* be 
>    one.
> 
> If that's correct, then async_synchronize() isn't a full answer...
> 
> I think a fair number of cases can be papered over with a serial 
> console with no hardware flow control, which isn't hooked up to 
> anything.  Maybe the board needs a test/development jig to get 
> one; without it, the "console" bits spill out into the aether.  
> Linux can dump bits to ttyS0, which will act like /dev/null.
> 
> But the problem case here seems to be one where such un-hooked-up 
> serial ports are not realistic options.  Which is why the 
> regression in USB console functionality has been troublesome.
> 
> Is that correct?

Not sure i understand the complication.

The practical issue addressed by this particular patch i've replied 
to is that if /dev/console is sys_open()-ed by an app "too early", 
an error code is returned and subsequently user-space (and all its 
child tasks) remain console-less.

Whether the console has flow control or not is immaterial. 
Initialization sequence ordering details should be transparent as 
far as sys_open() of a console is concerned.

If there is _no console_, then obviously there's nothing to wait for 
and the async wait will succeed immediately. The console driver 
still returns an error on sys_open() but that is expected behavior.

The solution proposed in the patch i'm replying to is a boot flag 
that causes the kernel to wait N milliseconds. This is incorrect and 
wrong on several levels. If the console can be initialized via a 
proper handshake that should be exposed precisely - we should not 
wait neither longer nor shorter than needed. _If_ there's no proper 
handshake and a magic delay is needed (because there's some 
hardware-internal delay needed at init time) before the console can 
be used then that should be in the console driver itself and the 
async-wait will wait for that automatically.

But never should we have delays like this in generic code. 
Regardless of embedded versus server versus desktop issues.

	Ingo

Re: Wait for console to become available, v3.2

[Arjan van de Ven]

On Tue, 21 Apr 2009 08:43:46 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> The proper approach would be to use one of the async_synchronize*() 
> facilities in kernel/async.c to properly order the opening of the 
> console with device init.

Unfortunately this is not the answer.

First of all, USB does not use the async_* infrastructure, so it does
not do anything.
But more importantly... USB *CANNOT* do this fundamental operation.
USB does not have the capability to know when you have seen all devices
that are connected. Devices just show up a random amount of time after
you turn the power on for the bus.... there's no "and now we've seen
all" operation.

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Wait for console to become available, v3.2

[Ingo Molnar]

* Arjan van de Ven <arjan@infradead.org> wrote:

> On Tue, 21 Apr 2009 08:43:46 +0200
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > The proper approach would be to use one of the async_synchronize*() 
> > facilities in kernel/async.c to properly order the opening of the 
> > console with device init.
> 
> Unfortunately this is not the answer.
> 
> First of all, USB does not use the async_* infrastructure, so it does
> not do anything.

[ Of course it does not. Two kernel cycles ago nothing had any async 
  infrastructure ;-) We barely have any use of it today. But cases 
  like this are the way to extend its use. ]

> But more importantly... USB *CANNOT* do this fundamental 
> operation. USB does not have the capability to know when you have 
> seen all devices that are connected. Devices just show up a random 
> amount of time after you turn the power on for the bus.... there's 
> no "and now we've seen all" operation.

Yes - and this is fundamentally true of any hotplug bus design.

Nevertheless the wish has been expressed to wait for such a device 
to become available, in this very thread we are discussing.

What i'm saying is: instead of "wait 2000 msecs, maybe it works out" 
hack, there should be a proper sleep+event based approach to the 
same. With perhaps a _timeout_ for the "no console arrived" negative 
case as well. (which timeout can be set to zero in the "I _know_ 
there's no console around".)

Do you see the fundamental design difference between the two 
solutions?

	Ingo

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Ingo Molnar wrote:
> * Arjan van de Ven <arjan@infradead.org> wrote:
> > But more importantly... USB *CANNOT* do this fundamental 
> > operation. USB does not have the capability to know when you have 
> > seen all devices that are connected. Devices just show up a random 
> > amount of time after you turn the power on for the bus.... there's 
> > no "and now we've seen all" operation.
> 
> Yes - and this is fundamentally true of any hotplug bus design.

It's not fundamental, for devices you know are plugged in at boot.
All it takes is for the bus to support a synchronous "enumerate all"
procedure.  That _could_ involve a timeout, but there are better ways.
But not for USB.

> What i'm saying is: instead of "wait 2000 msecs, maybe it works out" 
> hack, there should be a proper sleep+event based approach to the 
> same. With perhaps a _timeout_ for the "no console arrived" negative 
> case as well. (which timeout can be set to zero in the "I _know_ 
> there's no console around".)

Isn't the proposed code doing exactly that?  It sleeps waiting for a
console, with a timeout of X msecs.  If a console arrives before that,
doesn't it wake immediately?

> Do you see the fundamental design difference between the two 
> solutions?

The only difference I see is: If you asked for a USB console, the
timeout should begin relative to when the USB host driver kicks the
bus, not the first /dev/console open.

Conceptually the timeout is a USB characteristic, not a generic
console one.

For other (non-USB) devices which might take a while to initialise, it
should work as you describe.  E.g. an ethernet TCP console should
register ("i'm trying to give you a console") followed by ("got it")
or ("failed, carry on without me").

-- Jamie

Re: Wait for console to become available, v3.2

[Ingo Molnar]

* Jamie Lokier <jamie@shareable.org> wrote:

> Ingo Molnar wrote:
> > * Arjan van de Ven <arjan@infradead.org> wrote:
> > > But more importantly... USB *CANNOT* do this fundamental 
> > > operation. USB does not have the capability to know when you have 
> > > seen all devices that are connected. Devices just show up a random 
> > > amount of time after you turn the power on for the bus.... there's 
> > > no "and now we've seen all" operation.
> > 
> > Yes - and this is fundamentally true of any hotplug bus design.
> 
> It's not fundamental, for devices you know are plugged in at boot. 
> All it takes is for the bus to support a synchronous "enumerate 
> all" procedure.  That _could_ involve a timeout, but there are 
> better ways. But not for USB.
> 
> > What i'm saying is: instead of "wait 2000 msecs, maybe it works 
> > out" hack, there should be a proper sleep+event based approach 
> > to the same. With perhaps a _timeout_ for the "no console 
> > arrived" negative case as well. (which timeout can be set to 
> > zero in the "I _know_ there's no console around".)
> 
> Isn't the proposed code doing exactly that?  It sleeps waiting for 
> a console, with a timeout of X msecs.  If a console arrives before 
> that, doesn't it wake immediately?

Yes - except that the timeout is aspecific. I.e. we'll time out even 
if the delay happens not due to a device arriving too late (or not 
arriving at all), but due to some earlier init step taking an 
unexpectedly long time.

> > Do you see the fundamental design difference between the two 
> > solutions?
> 
> The only difference I see is: If you asked for a USB console, the 
> timeout should begin relative to when the USB host driver kicks 
> the bus, not the first /dev/console open.
> 
> Conceptually the timeout is a USB characteristic, not a generic 
> console one.
> 
> For other (non-USB) devices which might take a while to 
> initialise, it should work as you describe.  E.g. an ethernet TCP 
> console should register ("i'm trying to give you a console") 
> followed by ("got it") or ("failed, carry on without me").

I guess we could live with this one too but still it feels wrong to 
me. It would be much better to push the decision to time out into 
individual console drivers.

Also, if we end up doing this - the patch, as proposed right now, 
appears to add a default 1000 msecs sleep to every __tty_open() 
call:

+config PRINTK_CONSOLE_WAIT
+       int "Default number of milliseconds to wait for console device"
+       default 1000

Does this only delay init during a console-less bootup - or are 
there other later apps that might trigger the delay?

	Ingo

Re: Wait for console to become available, v3.2

[Alan Stern]

On Tue, 21 Apr 2009, Jamie Lokier wrote:

> Ingo Molnar wrote:
> > * Arjan van de Ven <arjan@infradead.org> wrote:
> > > But more importantly... USB *CANNOT* do this fundamental 
> > > operation. USB does not have the capability to know when you have 
> > > seen all devices that are connected. Devices just show up a random 
> > > amount of time after you turn the power on for the bus.... there's 
> > > no "and now we've seen all" operation.
> > 
> > Yes - and this is fundamentally true of any hotplug bus design.
> 
> It's not fundamental, for devices you know are plugged in at boot.
> All it takes is for the bus to support a synchronous "enumerate all"
> procedure.  That _could_ involve a timeout, but there are better ways.
> But not for USB.

Is that last sentence necessarily true?  I suppose it is true that a
USB device isn't obligated to make its presence on the bus known
immediately, but nevertheless, most of them do.  In theory we could add
code to the USB subsystem to detect when the hub driver has become idle
and therefore all devices that were initially plugged in have been
probed.

That probably would work for solving the console problem.  Storage 
devices present additional problems, however:

When usb-storage discovers a device, it generally delays a while before 
scanning it.  (The default delay is 5 seconds, but it is adjustable by 
a module parameter.)  And then when the scanning function is called, 
the SCSI subsystem turns it into an asynchronous request -- that's why 
the kernel log usually says "usb-storage: device scan complete" before 
the scan has in fact begun.

So I don't know about determining when all USB mass-storage devices 
have been detected, but USB serial devices should be practical.

If somebody would like to suggest a programming interface (a waitqueue 
perhaps?) by which the USB hub driver could send a notification when it 
becomes idle, I could implement it.

Alan Stern

Re: Wait for console to become available, v3.2

[Alan Cox]

> +config PRINTK_CONSOLE_WAIT
> +       int "Default number of milliseconds to wait for console device"
> +       default 1000
> 
> Does this only delay init during a console-less bootup - or are 
> there other later apps that might trigger the delay?

The console proper needs to be event based not timeout hacks. Can we
please fix this stuff properly ? To start with there is no reason that the
USB console can't implement a "maybe we have hardware, maybe I buffer 64K
until it shows up" behaviour and bind to hardware as and when USB serial
devices get inserted. We do much of this for the VT drivers so we save
messages before PCI and fb come up.

The timeout wait proposed is also wrong I agree. Take a timestamp at the
point we are ready to mount the root fs, any delays on root mount, console
appearance etc should then be tested against this timestamp not as delays
versus some undefined later event.

Alan

Re: Wait for console to become available, v3.2

[David VomLehn]

On Tue, Apr 21, 2009 at 04:26:27PM +0200, Ingo Molnar wrote:
> 
> * Jamie Lokier <jamie@shareable.org> wrote:
> 
> > Ingo Molnar wrote:
...
> > > What i'm saying is: instead of "wait 2000 msecs, maybe it works 
> > > out" hack, there should be a proper sleep+event based approach 
> > > to the same. With perhaps a _timeout_ for the "no console 
> > > arrived" negative case as well. (which timeout can be set to 
> > > zero in the "I _know_ there's no console around".)
> > 
> > Isn't the proposed code doing exactly that?  It sleeps waiting for 
> > a console, with a timeout of X msecs.  If a console arrives before 
> > that, doesn't it wake immediately?
> 
> Yes - except that the timeout is aspecific. I.e. we'll time out even 
> if the delay happens not due to a device arriving too late (or not 
> arriving at all), but due to some earlier init step taking an 
> unexpectedly long time.

I agree, but this is also a pretty good description of why things actually
worked before--it just so happened that work intervening between starting
USB enumeration and the opening of /dev/console was long enough that
USB serial devices were initialized. So, no, it's not good. But it's a
pretty close equivalent to previous behavior.

That being said, I'm also thinking that the timeout interval should start when
bus initialization is started. The point between the start of USB
initialization and when a particular device is initialized should be, as
a practical matter and for a particular hardware configuration, more or
less constant, and so more predictable for users.

Basing the wait interval from the start of bus initialization has the
additional advantage that we can also implement similar timers, such as one
for network devices, off the same base time. Since the ip= kernel parameter
is similarly broken, I've been looking at what it would take to get it
working again, too. Or at least, with deference to the constraints of USB and
acknowledging the importance of luck, working as well as it used to.

> Also, if we end up doing this - the patch, as proposed right now, 
> appears to add a default 1000 msecs sleep to every __tty_open() 
> call:
> 
> +config PRINTK_CONSOLE_WAIT
> +       int "Default number of milliseconds to wait for console device"
> +       default 1000
> 
> Does this only delay init during a console-less bootup - or are 
> there other later apps that might trigger the delay?

Sleeping and polling for an event, rather than using a suitable notification
primitive, causes me existential anguish. Seriously, I *hate* code like that
and systems that make you do such brutish things. Linux is not one of those
throwbacks, so as soon as a console shows up, we continue. Thus, we only wait
the whole interval if the console device is very slow to initialize or if
there is no console device, at all.
--
David VomLehn

Re: Wait for console to become available, v3.2

[David VomLehn]

> If somebody would like to suggest a programming interface (a waitqueue 
> perhaps?) by which the USB hub driver could send a notification when it 
> becomes idle, I could implement it.

I actually started the USB console stuff with exactly this approach, but
switched to the approach that's out there. A minor drawback, which is
probably obvious, is that you actually wait for some interval without
getting anything to do before you think things are idle.  But a bigger
drawback is that you lose the ability to chose appropriate intervals for
different classes of devices.

So far, there appear to be three possible USB boot devices: consoles, network
devices, and boot devices. A system may not have all of these and so may not
need to wait as long as a system with all of them. One of my goals is to
preserve as much of the reduction in boot time as possible, even on systems
that use USB devices that may or may not be plugged in.

> Alan Stern

David VomLehn

Re: Wait for console to become available, v3.2

[David Woodhouse]

On Tue, 2009-04-21 at 00:13 -0700, David Brownell wrote:
> 
> Stepping back a moment from "how" to make sure "what" is agreed on.
> 
> I think I see three scenarios here:
> 
>  - Classic PC or server, where there's a meaningful console;
> 
>  - Deeply embedded systems, where there isn't;
> 
>  - Development stages of "deeply embedded", where there *may* be one.
> 
> If that's correct, then async_synchronize() isn't a full answer...
> 
> I think a fair number of cases can be papered over with a serial
> console with no hardware flow control, which isn't hooked up to
> anything.  Maybe the board needs a test/development jig to get one;
> without it, the "console" bits spill out into the aether.  Linux
> can dump bits to ttyS0, which will act like /dev/null.
> 
> But the problem case here seems to be one where such un-hooked-up
> serial ports are not realistic options.  Which is why the regression
> in USB console functionality has been troublesome.

We can provide un-hooked-up /dev/console though. Rather than just
failing to open it, why can't we make __tty_open() give you a dummy tty
driver which is basically equivalent to /dev/null? And then 'replace' it
with the real console driver if/when that later gets registered? The
latter will be a high-caffeine job, but surely not impossible?

The kernel output is going to be spewed when a console registers with
CON_PRINTBUFFER anyway, and if we printk a warning about userspace
console output being lost, that ought to be good enough to notify the
user that something may have been lost. For bonus points, we could even
make that 'dummy' tty driver buffer a limited amount of userspace
output, maybe.

If userspace cares, let _it_ wait, by using an ioctl to see what tty
device it's _really_ attached to.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: Wait for console to become available, v3.2

[David VomLehn]

On Tue, Apr 21, 2009 at 06:11:11PM +0100, David Woodhouse wrote:
...
> The kernel output is going to be spewed when a console registers with
> CON_PRINTBUFFER anyway, and if we printk a warning about userspace
> console output being lost, that ought to be good enough to notify the
> user that something may have been lost. For bonus points, we could even
> make that 'dummy' tty driver buffer a limited amount of userspace
> output, maybe.

What in the world are users going to do when they see a message about
output being lost? There is no way to recover the data and no way to
prevent it in the future. I don't think this is a good approach.

Remember, even though the previous functionality of USB consoles was, in
theory, less reliable than it appeared, it actually has been working well
for years. Thus, tossing output will count as a regression for most of the
world.

> If userspace cares, let _it_ wait, by using an ioctl to see what tty
> device it's _really_ attached to.

I have a notion, perhaps misguided, that userspace *always* cares about
seeing all the output they have generated.

I have no objection to a userspace solution, in the long run, but such an
approach represents a significantly different approach to constructing boot
filesystems and we need a transitional period. In the interim, let's get
things working as they were.

> David Woodhouse                            Open Source Technology Centre

David VomLehn

Re: Wait for console to become available, v3.2

[Linus Torvalds]

On Tue, 21 Apr 2009, David Woodhouse wrote:
> 
> We can provide un-hooked-up /dev/console though. Rather than just
> failing to open it, why can't we make __tty_open() give you a dummy tty
> driver which is basically equivalent to /dev/null? And then 'replace' it
> with the real console driver if/when that later gets registered? The
> latter will be a high-caffeine job, but surely not impossible?

This sounds like a good option. 

Right now we look through all the console_drivers at open time in that

	struct tty_driver *console_device(int *index)

thing, but it should be possible to just do something like

	static struct tty_driver console_driver;

and then just make "console_device()" always return _that_ tty driver 
instead. Then, we could make that console_device tty driver just at 
run-time spread the IO to the proper device(s), rather than fix the one 
particular device at "open()" time.

[ Actually, looking closer we should not use that particular name: we 
  already have something called a "console_driver" which is really the 
  "current VT" driver.

  So we should call it 'system_console_device' or something, but you 
  get the idea. ]

Anybody want to try it? Just make it ignore any IO if there are no 
registered consoles. The patch shouldn't even be all that big, I suspect.

		Linus

Re: Wait for console to become available, v3.2

[Linus Torvalds]

On Tue, 21 Apr 2009, David VomLehn wrote:
> 
> What in the world are users going to do when they see a message about
> output being lost? There is no way to recover the data and no way to
> prevent it in the future. I don't think this is a good approach.

Sure there is. The console messages are saved too, so doing 'dmesg' will 
get you all the data that was generated before the console went on-line.

We _already_ lose data in that sense (although we could replay it for the 
first console connected - maybe we even do, I'm too lazy to check).

			Linus

Re: Wait for console to become available, v3.2

[David Woodhouse]

On Tue, 2009-04-21 at 10:29 -0700, David VomLehn wrote:
> On Tue, Apr 21, 2009 at 06:11:11PM +0100, David Woodhouse wrote:
> ...
> > The kernel output is going to be spewed when a console registers with
> > CON_PRINTBUFFER anyway, and if we printk a warning about userspace
> > console output being lost, that ought to be good enough to notify the
> > user that something may have been lost. For bonus points, we could even
> > make that 'dummy' tty driver buffer a limited amount of userspace
> > output, maybe.
> 
> What in the world are users going to do when they see a message about
> output being lost? There is no way to recover the data and no way to
> prevent it in the future. I don't think this is a good approach.
> 
> Remember, even though the previous functionality of USB consoles was, in
> theory, less reliable than it appeared, it actually has been working well
> for years. Thus, tossing output will count as a regression for most of the
> world.

So implement a small amount of buffering, as I suggested, and it should
be sufficient to cover the cases that were only working before by blind
luck anyway.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: Wait for console to become available, v3.2

[David VomLehn]

 Tue, Apr 21, 2009 at 10:37:41AM -0700, Linus Torvalds wrote:
> 
> 
> On Tue, 21 Apr 2009, David VomLehn wrote:
> > 
> > What in the world are users going to do when they see a message about
> > output being lost? There is no way to recover the data and no way to
> > prevent it in the future. I don't think this is a good approach.
> 
> Sure there is. The console messages are saved too, so doing 'dmesg' will 
> get you all the data that was generated before the console went on-line.
> 
> We _already_ lose data in that sense (although we could replay it for the 
> first console connected - maybe we even do, I'm too lazy to check).

There are two different things coming out on the console, kernel log buffer
messages and userspace output to /dev/console. Kernel log buffer output
is replayed for each console as it connects. There might be quite a bit of
it, but it is pretty much the same from boot to boot, so the guy working on
the kernel, me, can tune the size of log_buf pretty easily. This is not
the problem.

The problem is the userspace output to /dev/console. I work in the embedded
space and my box does not simply throw up a few windows and wait placidly for
the user to do something. Instead, it immediately starts running a whole bunch
of software with a whole bunch of startup messages. In short order, I am
running over 300 threads. I have no control over how much output comes out,
I have no idea how much kernel memory to allocate for this, and I'm quote sure
it's enough that I wouldn't want to waste that much space even if I did. No
matter what, this looks like we'd need a new tunable for the size.

And then, what happens if I *never* get a console, because one isn't plugged
in? This happens a lot in the embedded world. How long am I supposed to keep
around this big chunk of buffered data? Sounds like I need *another* tunable.
Yuck.

> 			Linus

Re: Wait for console to become available, v3.2

[Alan Stern]

On Tue, 21 Apr 2009, David VomLehn wrote:

> > If somebody would like to suggest a programming interface (a waitqueue 
> > perhaps?) by which the USB hub driver could send a notification when it 
> > becomes idle, I could implement it.
> 
> I actually started the USB console stuff with exactly this approach, but
> switched to the approach that's out there. A minor drawback, which is
> probably obvious, is that you actually wait for some interval without
> getting anything to do before you think things are idle.

Sorry, I don't understand.  You've got a waitqueue, and you wake up to
try again every time a console device is registered.  So if the console
device does exist, you'll find it as quickly as possible.  There's a
problem only when no console device is ever registered.  In that case
you want to stop waiting when all the devices present at boot time have
been probed (as opposed to waiting for the user to plug in a USB serial
device, for example).

So then the problem becomes: How do you tell when all the USB devices 
present at boot time have been probed?  That's exactly what this new 
interface is supposed to tell you.

>  But a bigger
> drawback is that you lose the ability to chose appropriate intervals for
> different classes of devices.

Why do you need to choose any intervals at all?  Just keep waiting
until you know there's no point waiting any longer.  If you stop 
waiting before then, you run the risk of missing a device that could 
have worked.

> So far, there appear to be three possible USB boot devices: consoles, network
> devices, and boot devices.

Is that last supposed to be "disk drives containing root filesystems"?

> A system may not have all of these and so may not
> need to wait as long as a system with all of them. One of my goals is to
> preserve as much of the reduction in boot time as possible, even on systems
> that use USB devices that may or may not be plugged in.

Let's take console devices as an example.  If one is present then you
do want to wait until it is detected, instead of booting as quickly as
possible, right?  Which implies that if one isn't present, you want to 
wait until you _know_ that it's not present.  Which means waiting all 
all the devices present during boot have been probed.

This approach should work fine with USB consoles and network devices;
it's not so easy to implement for USB disk drives.  So let's not worry 
about them.

Now the only drawback I can see would crop up if some device takes a
very long time to probe (perhaps because of errors and retries).  If
there's no other console device, it could slow down the boot procedure
quite a lot.  But there doesn't appear to be any logical alternative,
given your goals as stated above.

Alan Stern

Re: Wait for console to become available, v3.2

[Alan Cox]

> > failing to open it, why can't we make __tty_open() give you a dummy tty
> > driver which is basically equivalent to /dev/null? And then 'replace' it
> > with the real console driver if/when that later gets registered? The
> > latter will be a high-caffeine job, but surely not impossible?
> 
> This sounds like a good option. 

Don't confuse "/dev/console" and "the console device" in printk terms.
The two are not the same thing and the latter is subject to stuff like
redirection.

> thing, but it should be possible to just do something like
> 
> 	static struct tty_driver console_driver;

No - a console device can fail to open so you need to try the next one in
that situation. Some platforms use this feature.

> Anybody want to try it? Just make it ignore any IO if there are no 
> registered consoles. The patch shouldn't even be all that big, I suspect.

It will work, but on its own it won't actually fix the problem people
have, which is wanting to wait for a real console to appear. Extending it
to sleep (if not O_NDELAY) and loop back through the list when the list
changes would however do the trick, that or a tty device that buffers
then spews into the real device when it appears.

Definitely the right path, and we almost have every tty device in
existance containing a struct tty_port, at which point it gets even
cleaner.

The printk() console needs to buffer up data and replay it but we already
pretty much do what is needed even for vt consoles because the fb isn't
ready before the first printk.

Re: Wait for console to become available, v3.2

[David VomLehn]

On Tue, Apr 21, 2009 at 03:09:52PM -0400, Alan Stern wrote:
> On Tue, 21 Apr 2009, David VomLehn wrote:
> 
> > > If somebody would like to suggest a programming interface (a waitqueue 
> > > perhaps?) by which the USB hub driver could send a notification when it 
> > > becomes idle, I could implement it.
> > 
> > I actually started the USB console stuff with exactly this approach, but
> > switched to the approach that's out there. A minor drawback, which is
> > probably obvious, is that you actually wait for some interval without
> > getting anything to do before you think things are idle.
> 
> Sorry, I don't understand.  You've got a waitqueue, and you wake up to
> try again every time a console device is registered.  So if the console
> device does exist, you'll find it as quickly as possible.  There's a
> problem only when no console device is ever registered.  In that case
> you want to stop waiting when all the devices present at boot time have
> been probed (as opposed to waiting for the user to plug in a USB serial
> device, for example).
> 
> So then the problem becomes: How do you tell when all the USB devices 
> present at boot time have been probed?  That's exactly what this new 
> interface is supposed to tell you.

The current patch already uses a waitqueue and wakes up as soon as a console
is registered, so that's the same. I agree that the other piece is determining
when all USB devices at boot time have been probed. For this suggestion to
work, two conditions must be met:
1.	The USB hub driver becoming idle must indicate that every device has
	been probed.
2.	The USB driver must become idle after every USB device has been
	probed.

If both conditions are met, this is a superior to a timeout for determining
that the console will not show up. It would surprise me a bit if the first
condition was true. Still, if it is true that the USB hub driver will be
idle for some interval I, where I is greater than the timeout we would
otherwise use, it is still a better approach.

There is one other possible gotcha to this approach. The code as I wrote it is
bus-agnostic, i.e. it has no idea where the console is located. As far as
I know, the long time to probe possible console devices only arises with
USB. If this is not tree, we need to handle any other cases that can arise.

> >  But a bigger
> > drawback is that you lose the ability to chose appropriate intervals for
> > different classes of devices.
> 
> Why do you need to choose any intervals at all?  Just keep waiting
> until you know there's no point waiting any longer.  If you stop 
> waiting before then, you run the risk of missing a device that could 
> have worked.

The question about appropriate intervals is that I need to wait for some
devices, like consoles and network devices, but not all devices, like printers
and cameras. There is no telling whether any two devices require the same
interval when I am tuning it for my particular system. And if I have a USB
console and no USB Ethernet device, I only want to wait for the console
probe interval.

> > So far, there appear to be three possible USB boot devices: consoles, network
> > devices, and boot devices.
> 
> Is that last supposed to be "disk drives containing root filesystems"?

Yup.

> > A system may not have all of these and so may not
> > need to wait as long as a system with all of them. One of my goals is to
> > preserve as much of the reduction in boot time as possible, even on systems
> > that use USB devices that may or may not be plugged in.
> 
> Let's take console devices as an example.  If one is present then you
> do want to wait until it is detected, instead of booting as quickly as
> possible, right?  Which implies that if one isn't present, you want to 
> wait until you _know_ that it's not present.  Which means waiting all 
> all the devices present during boot have been probed.
> 
> This approach should work fine with USB consoles and network devices;
> it's not so easy to implement for USB disk drives.  So let's not worry 
> about them.

I'm kinda flexible about whether all possible consoles show up; the patch
presently only waits for a console to be identified as the "preferred" console.
This gets the system going as soon as you have some place to write output,
but I won't argue with anyone who says we need to wait for all possible
consoles.

> Now the only drawback I can see would crop up if some device takes a
> very long time to probe (perhaps because of errors and retries).  If
> there's no other console device, it could slow down the boot procedure
> quite a lot.  But there doesn't appear to be any logical alternative,
> given your goals as stated above.

If there is a way to guarantee that we are done with the probing, I'm
all for using that. Will waiting for the USB hub driver to be idle do this?

> Alan Stern

David VomLehn

Re: Wait for console to become available, v3.2

[David VomLehn]

On Tue, Apr 21, 2009 at 08:25:01PM +0100, Alan Cox wrote:
> > Anybody want to try it? Just make it ignore any IO if there are no 
> > registered consoles. The patch shouldn't even be all that big, I suspect.
> 
> It will work, but on its own it won't actually fix the problem people
> have, which is wanting to wait for a real console to appear. Extending it
> to sleep (if not O_NDELAY) and loop back through the list when the list
> changes would however do the trick, that or a tty device that buffers
> then spews into the real device when it appears.
> 
> Definitely the right path, and we almost have every tty device in
> existance containing a struct tty_port, at which point it gets even
> cleaner.
> 
> The printk() console needs to buffer up data and replay it but we already
> pretty much do what is needed even for vt consoles because the fb isn't
> ready before the first printk.

We're already good on printk, the size of log_buf is configurable and log_buf
is dumped to each console as it gets registered. Even then, printk output is
not an especially big issue since you have dmesg to get to anything you
might have missed.

Other the other hand, userspace output is not buffered and has the potential
to be much larger. Any approach that buffers it risks either truncation or
gobbling up memory. There is also the human factors issue that users will
pretty much ignore printk output because they don't particularly understand
it. They understand, and expect to see, the output they generate. I'm pretty
sure they won't like for it to go missing under mysterious circumstances.

David VomLehn

Re: Wait for console to become available, v3.2

[David VomLehn]

By now, pretty much everone is probably up to speed on what the USB folks are
telling us:

	There is no way to avoid using some sort of timeout to decide
	whether the console is ever going to appear.

So:
1.	We want the minimum timeout possible so that headless systems will
	not be delayed any more than is necessary to detect that no console
	is available.
2.	As I understand it, USB devices don't generally come with any
	guarantee what the maximum time to initialize will be.

Therefore, as far as I can see, this means that we can't empirically
determine an appropriate timeout until we know what USB devices are
connected to a particular system. This, in turn, means that any default
timeout we might choose will have to have a command line parameter to
override it.

Naturally, we want to help a user determine as easily as possible just how
long it took to detect whatever serial device they may want to use so that
they can set the timeout so the system will boot as quickly as possible when
being used in a headless mode. This probably means picking a stable point
from which to start measuring the timeout, as well as printing how long it
took to detect the console, if there was one.

An open question is whether we want to wait for all possible consoles or
whether we can proceed with as few as one.

David VomLehn

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Alan Cox wrote:
> To start with there is no reason that the
> USB console can't implement a "maybe we have hardware, maybe I buffer 64K
> until it shows up" behaviour and bind to hardware as and when USB serial
> devices get inserted.

Ah, that doesn't work, when you want to use the USB serial console
_if_ there's a USB serial adapter plugged in, and a different (lower
priority) console if it's not.

-- Jamie

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Linus Torvalds wrote:
> [ Actually, looking closer we should not use that particular name: we 
>   already have something called a "console_driver" which is really the 
>   "current VT" driver.

Speaking of names... The word "console" is used to mean three
different things in the kernel.  Last time I had to understand the
console code (because it wasn't showing on some device) that didn't
help.  But it was a 2.4 kernel 2 years ago so I didn't submit a patch :-)

It would be a sensible though boring cleanup to change all the "VC"
(virtual console) stuff to use the name "VT" (virtual terminal)
consistently.

-- Jamie

Re: Wait for console to become available, v3.2

[Alan Cox]

> It would be a sensible though boring cleanup to change all the "VC"
> (virtual console) stuff to use the name "VT" (virtual terminal)
> consistently.

Then we would have two completely different meanings for virtual terminal
in the kernel.

Alan

Re: Wait for console to become available, v3.2

[Alan Cox]

> Ah, that doesn't work, when you want to use the USB serial console
> _if_ there's a USB serial adapter plugged in, and a different (lower
> priority) console if it's not.

I don't see the problem. If there is a lower priority console available
then the kernel can use that anyway and clearly that would be the right
thing to do.

The case that wouldn't work perhaps is the 'no console present and a USB
or firewire one might appear' - for that to work you'd need the queue to
be smarter - or even the lowest priority console...

Alan

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Alan Cox wrote:
> > It would be a sensible though boring cleanup to change all the "VC"
> > (virtual console) stuff to use the name "VT" (virtual terminal)
> > consistently.
> 
> Then we would have two completely different meanings for virtual terminal
> in the kernel.

As opposed to three meanings for console - causing a bit of confusion
you had to point out to Linus earlier :-)

-- Jamie

Re: Wait for console to become available, v3.2

[Alan Stern]

On Tue, 21 Apr 2009, David VomLehn wrote:

> The current patch already uses a waitqueue and wakes up as soon as a console
> is registered, so that's the same. I agree that the other piece is determining
> when all USB devices at boot time have been probed. For this suggestion to
> work, two conditions must be met:
> 1.	The USB hub driver becoming idle must indicate that every device has
> 	been probed.

That's not exactly right.  When a new hub is registered, the hub driver 
doesn't become idle until after all the devices plugged into that hub 
have been initialized and probed.

So the hub driver becoming idle indicates that every device attached to
any registered USB host controller has been probed, but there may still
be some unregistered controllers.  This won't be an issue if you first
wait for all the new async probing stuff to end; by then all the PCI 
devices (including USB host controllers) should be registered.

> 2.	The USB driver must become idle after every USB device has been
> 	probed.

Under normal conditions it will.  There are some abnormal conditions
which cause the hub driver never to become idle, such as a faulty
device that continually connects and disconnects itself electrically
to/from the USB bus.  (Each connection event will create more work for
the hub driver.)  I have seen reports of this sort of thing.

I guess a better criterion would be for the hub driver to maintain a
count of registered but unscanned hubs.  That wouldn't be subject to
quite the same starvation issue.  Although there is another issue which
occasionally comes up with buggy devices: The hub driver's
device-initialization procedure has so many retries and timeouts that
it can take several minutes before the driver gives up on a bad device.

> If both conditions are met, this is a superior to a timeout for determining
> that the console will not show up. It would surprise me a bit if the first
> condition was true.

Why?  Doesn't it make sense that the hub driver would remain running as 
long as there was still work for it do to?

>  Still, if it is true that the USB hub driver will be
> idle for some interval I, where I is greater than the timeout we would
> otherwise use, it is still a better approach.
> 
> There is one other possible gotcha to this approach. The code as I wrote it is
> bus-agnostic, i.e. it has no idea where the console is located. As far as
> I know, the long time to probe possible console devices only arises with
> USB. If this is not tree, we need to handle any other cases that can arise.

That is a very good objection.  For this sort of approach to work,
ultimately it would have to be implemented for every hotpluggable bus
of interest.

> > >  But a bigger
> > > drawback is that you lose the ability to chose appropriate intervals for
> > > different classes of devices.
> > 
> > Why do you need to choose any intervals at all?  Just keep waiting
> > until you know there's no point waiting any longer.  If you stop 
> > waiting before then, you run the risk of missing a device that could 
> > have worked.
> 
> The question about appropriate intervals is that I need to wait for some
> devices, like consoles and network devices, but not all devices, like printers
> and cameras. There is no telling whether any two devices require the same
> interval when I am tuning it for my particular system. And if I have a USB
> console and no USB Ethernet device, I only want to wait for the console
> probe interval.

I still don't understand.  Let's say you want the console init code to
wait for the preferred console device to be registered, and that device
could potentially be either a USB serial or a USB network device.  And
let's say that your system has a USB serial device but no USB network
devices.

The console delay routine will then wait until the USB serial device is
discovered and registered, at which point its job is done.  It won't
continue to wait until all the USB devices present at boot time have
been probed, since there's no reason to wait after the first console
device is discovered.

So not only would you not need to tune two different delay times, you 
wouldn't even need to tune one!  You wouldn't need to specify any delay 
times at all.

> I'm kinda flexible about whether all possible consoles show up; the patch
> presently only waits for a console to be identified as the "preferred" console.
> This gets the system going as soon as you have some place to write output,
> but I won't argue with anyone who says we need to wait for all possible
> consoles.

Which essentially means waiting until _all_ the devices present at boot 
time (both USB and non-USB) have been probed.  Right?  And it 
explicitly means ruling out waiting for new devices which the user 
might plug in after the system has booted, since a new console device 
could be plugged in at any time.

> > Now the only drawback I can see would crop up if some device takes a
> > very long time to probe (perhaps because of errors and retries).  If
> > there's no other console device, it could slow down the boot procedure
> > quite a lot.  But there doesn't appear to be any logical alternative,
> > given your goals as stated above.
> 
> If there is a way to guarantee that we are done with the probing, I'm
> all for using that. Will waiting for the USB hub driver to be idle do this?

I'm pretty sure something like it can be made to work, subject to the 
delays caused by bad devices.

Alan Stern

Re: Wait for console to become available, v3.2

[David VomLehn]

On Wed, Apr 22, 2009 at 11:40:09AM -0400, Alan Stern wrote:
> On Tue, 21 Apr 2009, David VomLehn wrote:
...
> > There is one other possible gotcha to this approach. The code as I wrote it is
> > bus-agnostic, i.e. it has no idea where the console is located. As far as
> > I know, the long time to probe possible console devices only arises with
> > USB. If this is not tree, we need to handle any other cases that can arise.
> 
> That is a very good objection.  For this sort of approach to work,
> ultimately it would have to be implemented for every hotpluggable bus
> of interest.

Alright, I know you (Alan (Stern)) know about USB. I think someone else
mentioned Firewire. What other hotpluggable buses do we need to worry about
and how can we get them interested?

> The console delay routine will then wait until the USB serial device is
> discovered and registered, at which point its job is done.  It won't
> continue to wait until all the USB devices present at boot time have
> been probed, since there's no reason to wait after the first console
> device is discovered.

For each boot device, we need to determine whether a) we for some subset
instance of a device class, or b) until all possible instances of a device
class have been probed. In case a, we exit the wait if either:
1.	A suitable device has been registered, i.e. any console, a specific
	network device, etc., or
2.	All devices have been probed
If we exit for reason 2, it means that no such device is present,
and we go on to the do the appropriate thing for that device class.

In case b, we exit the wait only when all devices have been probed. Then
we either have a device of that class or not, and we do the appropriate thing.

There is a special case for USB consoles, which might apply to other buses
and classes of devices, as well: if USB_CONSOLE is not configured, you don't
need to wait for notification from USB. This helps guide the design of the
required infrastructure.

> So not only would you not need to tune two different delay times, you 
> wouldn't even need to tune one!  You wouldn't need to specify any delay 
> times at all.

This would make me and, clearly, a lot of other people happy, as well.

> Which essentially means waiting until _all_ the devices present at boot 
> time (both USB and non-USB) have been probed.  Right?  And it 
> explicitly means ruling out waiting for new devices which the user 
> might plug in after the system has booted, since a new console device 
> could be plugged in at any time.

Yes, I think we absolutely exclude devices plugged in after boot. I think such
usage, if it needs to be supported, can be handled with hot plugging. I'm a
little squishy on whether we might to be able to hot plug USB consoles as I
have a request to support this, but I think it's a separate issue from
supporting hot pluggable boot devices and I'm not worrying about it now.

> > If there is a way to guarantee that we are done with the probing, I'm
> > all for using that. Will waiting for the USB hub driver to be idle do this?
> 
> I'm pretty sure something like it can be made to work, subject to the 
> delays caused by bad devices.

There will always be bad devices, but let's see if we can support the good
devices, first. If the bad devices will make a commitment to rehabilitate
themselves, perhaps we can consider their needs later.

I'm going to see if I can come up with some sort of concrete proposal for
waiting for hotpluggable boot devices. I'd appreciate it if you could think
about how to implement the USB-specific part of this. Once we have something
that makes sense, and works, we can go back and pick up the details for
the console, IP autoconfig, and anything else that someone might need done.

> Alan Stern

David VomLehn

Re: Wait for console to become available, v3.2

[Alan Cox]

> Alright, I know you (Alan (Stern)) know about USB. I think someone else
> mentioned Firewire. What other hotpluggable buses do we need to worry about
> and how can we get them interested?

In theory any console can be hot pluggable - PCI-X hotplug video cards,
hotplug serial ports. In fact right now bad things occur if you
hot-unplug consoles.

> 2.	All devices have been probed
> If we exit for reason 2, it means that no such device is present,
> and we go on to the do the appropriate thing for that device class.

USB has no notion of #2 really

> Yes, I think we absolutely exclude devices plugged in after boot. I think such
> usage, if it needs to be supported, can be handled with hot plugging. I'm a
> little squishy on whether we might to be able to hot plug USB consoles as I
> have a request to support this, but I think it's a separate issue from
> supporting hot pluggable boot devices and I'm not worrying about it now.

This still assumes you can tell the difference, but USB in particular is
rather like undergraduate students - they turn up when they feel like it
at some point after they get kicked into life.

I don't think that is a big problem because our bus model is basically
that all devices are discovered asynchronously (increasingly so with
Arjan's speed up patches). A console model therefore has to deal with
console addition and removal being dynamic.

Re: Wait for console to become available, v3.2

[Alan Stern]

On Wed, 22 Apr 2009, Alan Cox wrote:

> > 2.	All devices have been probed
> > If we exit for reason 2, it means that no such device is present,
> > and we go on to the do the appropriate thing for that device class.
> 
> USB has no notion of #2 really

That's true, it doesn't.  But it _does_ rather have a notion of "all 
devices that were present at boot time have been probed".  That should 
be good enough.

Alan Stern

Re: Wait for console to become available, v3.2

[David VomLehn]

On Wed, Apr 22, 2009 at 05:24:18PM -0400, Alan Stern wrote:
> On Wed, 22 Apr 2009, Alan Cox wrote:
> 
> > > 2.	All devices have been probed
> > > If we exit for reason 2, it means that no such device is present,
> > > and we go on to the do the appropriate thing for that device class.
> > 
> > USB has no notion of #2 really
> 
> That's true, it doesn't.  But it _does_ rather have a notion of "all 
> devices that were present at boot time have been probed".  That should 
> be good enough.
> 
> Alan Stern

Okay, I've been slugging away at this and have a proposed interface. The
interface is at the level of devices of particular types because that would
maximize the opportunities for concurrency, but I don't know if interesting
buses actually provide that level of detail early enough for this to be
useful. If not, the fallback is just to do things on a bus granularity.

If bus granularity is the best that can be done, it is still interesting
to distingish between boot device types. If a boot device type is not 
upported by a particular bus, we don't have to wait for probing to complete
on that bus.

I'm still scratching my head, trying to figure out where I can tap into the
'USB notion of "all devices that were present at boot time have been probed"'
that Alan mentions above, so I don't have a specific implementation yet. I'm
open to any tips in this area.

Below is a write-up of the interface that looks like it should be added to
Documentation/driver-model/bus.txt. Comments are much appreciated.

----------------------------- CUT HERE -------------------------------------
Synchronization with hot pluggable boot devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Earlier versions of the Linux kernel used a single-threaded approach to
boot initialization. This took a number of seconds, which mean that
hotpluggable devices used or configured during boot were generally available
before they were used. Modern kernels use a multithreaded approach, which
requires synchronization between code that wants to act upon boot devices
and the probing and initialization of those devices. Support of fine-grained
boot concurrency requires distinguishing between types of boot devices, so
that devices can be used as soon as they are initialized.

There currently two types supported:
-	consoles
-	network devices

There is a distinction between the hardware type and the boot device type.
>From the hardware view, most any serial device can be used as a console,
but console support is generally be configured separately. For example,
consider USB serial devices. These should be considered a console only if
the software is configured to support this usage, which is done by enabling
the CONFIG_USB_SERIAL_CONSOLE option. If this option is disabled, the USB bus
driver should not report that it has found any console devices.

Two functions are available for use by the per-bus support code:

bus_bootdev_found(int bootdev_mask)
	This function must be called each time a boot device device is found.
	It is passed a bit mask created by ORing any of the following flags
	that apply to the device found:
		BOOTDEV_CONSOLE_MASK
		BOOTDEV_NETDEV_MASK
	There is no need to call this function for a given device if it is
	known that it cannot be used as a boot device. If it is not
	possible to determine whether a device is usable as a boot device,
	or what the specific type of boot device it may be, the argument
	BOOTDEV_ANY_MASK can be passed.  This should be used only when
	necessary as it reduces the level of concurrency that can be
	achieved from boot time initialization.

bus_bootdev_initialized(int bootdev_mask)
	This function must be called by the bus support code after it calls the
	initialiation function for a boot device. Every call to
	bus_bootdev_found must be matched by a call to
	bus_bootdev_initialized with the same bootdev_mask.

	Device initialization functions must have called the appropriate
	registration function for that device before bus_bootdev_initialized
	is called. So, for console devices, register_console must be called
	before bus_bootdev_initialized, and for network devices,
	register_netdevice must be called first.

It is possible for bus code to avoid the need to call bus_bootdev_found
for each device if it calls it once at the beginning of device discovery and
once at the end with a mask for all of the possible types of boot devices
that could be on that bus. This approach should be avoid if at all possible
because it reduces the available boot-time concurency, which can lead
to increased boot times.

When it is time to use a given boot device type, the following function
should be used:

bus_wait_for_bootdev(enum bootdev_type type, bool (*done)(void))
	The type is one of:
		BOOTDEV_CONSOLE
		BOOTDEV_NETDEV
	The done function checks to see whether all of the boot devices
	required have been registered. If it determines that this is the
	case, it can return true. This will cause the wait to be terminated
	even though some boot devices of the given type have not yet been
	initialized, which can decrease the time to boot. If this is not
	possible, it can return false. In that case, bus_wait_for_bootdev
	won't return until all devices of the given type on all buses have
	been initialized. Not surprisingly, this limit the opportunities to
	reduce boot time.

Re: Wait for console to become available, v3.2

[Alan Stern]

On Thu, 23 Apr 2009, David VomLehn wrote:

> I'm still scratching my head, trying to figure out where I can tap into the
> 'USB notion of "all devices that were present at boot time have been probed"'
> that Alan mentions above, so I don't have a specific implementation yet. I'm
> open to any tips in this area.

This is a general notion; it applies to all subsystems that do
asynchronous device discovery or registration.  The idea is simple
enough: Keep a count of all devices which are known or suspected to be
present, and decrement the count each time one of them is registered or
determined not to exist.  When the count first reaches 0, all devices
that were present (or reachable) at boot time will have been
registered.

Isn't this basically what you had in mind?  If you're concerned about 
the USB implementation, don't be -- I can write it quite easily.

> Below is a write-up of the interface that looks like it should be added to
> Documentation/driver-model/bus.txt. Comments are much appreciated.
> 
> ----------------------------- CUT HERE -------------------------------------
> Synchronization with hot pluggable boot devices

It's not just hot-pluggable devices.  These ideas are appropriate 
whenever detection or registration is asynchronous.

> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Earlier versions of the Linux kernel used a single-threaded approach to
> boot initialization. This took a number of seconds, which mean that
> hotpluggable devices used or configured during boot were generally available
> before they were used. Modern kernels use a multithreaded approach, which
> requires synchronization between code that wants to act upon boot devices
> and the probing and initialization of those devices. Support of fine-grained
> boot concurrency requires distinguishing between types of boot devices, so
> that devices can be used as soon as they are initialized.
> 
> There currently two types supported:
> -	consoles
> -	network devices

You might as well add block devices too.

> There is a distinction between the hardware type and the boot device type.
> From the hardware view, most any serial device can be used as a console,
> but console support is generally be configured separately. For example,
> consider USB serial devices. These should be considered a console only if
> the software is configured to support this usage, which is done by enabling
> the CONFIG_USB_SERIAL_CONSOLE option. If this option is disabled, the USB bus
> driver should not report that it has found any console devices.
> 
> Two functions are available for use by the per-bus support code:
> 
> bus_bootdev_found(int bootdev_mask)
> 	This function must be called each time a boot device device is found.
> 	It is passed a bit mask created by ORing any of the following flags
> 	that apply to the device found:
> 		BOOTDEV_CONSOLE_MASK
> 		BOOTDEV_NETDEV_MASK
> 	There is no need to call this function for a given device if it is
> 	known that it cannot be used as a boot device. If it is not
> 	possible to determine whether a device is usable as a boot device,
> 	or what the specific type of boot device it may be, the argument
> 	BOOTDEV_ANY_MASK can be passed.  This should be used only when
> 	necessary as it reduces the level of concurrency that can be
> 	achieved from boot time initialization.
> 
> bus_bootdev_initialized(int bootdev_mask)
> 	This function must be called by the bus support code after it calls the
> 	initialiation function for a boot device. Every call to
> 	bus_bootdev_found must be matched by a call to
> 	bus_bootdev_initialized with the same bootdev_mask.

Even if the device can't be initialized for some reason, or if it turns
out not the be the sort of boot device originally thought, this
function still has to be called.  That should be made extremely clear.

> 	Device initialization functions must have called the appropriate
> 	registration function for that device before bus_bootdev_initialized
> 	is called. So, for console devices, register_console must be called
> 	before bus_bootdev_initialized, and for network devices,
> 	register_netdevice must be called first.

It's not clear how either of these two functions is bus-specific.

> It is possible for bus code to avoid the need to call bus_bootdev_found
> for each device if it calls it once at the beginning of device discovery and
> once at the end with a mask for all of the possible types of boot devices
> that could be on that bus. This approach should be avoid if at all possible
> because it reduces the available boot-time concurency, which can lead
> to increased boot times.
> 
> When it is time to use a given boot device type, the following function
> should be used:
> 
> bus_wait_for_bootdev(enum bootdev_type type, bool (*done)(void))
> 	The type is one of:
> 		BOOTDEV_CONSOLE
> 		BOOTDEV_NETDEV
> 	The done function checks to see whether all of the boot devices
> 	required have been registered. If it determines that this is the
> 	case, it can return true. This will cause the wait to be terminated
> 	even though some boot devices of the given type have not yet been
> 	initialized, which can decrease the time to boot. If this is not
> 	possible, it can return false. In that case, bus_wait_for_bootdev
> 	won't return until all devices of the given type on all buses have
> 	been initialized. Not surprisingly, this limit the opportunities to
> 	reduce boot time.

This looks like a good plan and not hard to implement.  It even should 
be possible to fit USB disk drives into the scheme.

Alan Stern

Re: Wait for console to become available, v3.2

[David VomLehn]

On Fri, Apr 24, 2009 at 03:20:42PM -0400, Alan Stern wrote:
> On Thu, 23 Apr 2009, David VomLehn wrote:
> 
> > I'm still scratching my head, trying to figure out where I can tap into the
> > 'USB notion of "all devices that were present at boot time have been probed"'
> > that Alan mentions above, so I don't have a specific implementation yet. I'm
> > open to any tips in this area.
> 
> This is a general notion; it applies to all subsystems that do
> asynchronous device discovery or registration.  The idea is simple
> enough: Keep a count of all devices which are known or suspected to be
> present, and decrement the count each time one of them is registered or
> determined not to exist.  When the count first reaches 0, all devices
> that were present (or reachable) at boot time will have been
> registered.
> 
> Isn't this basically what you had in mind?  If you're concerned about 
> the USB implementation, don't be -- I can write it quite easily.

I will absolutely take you up on your offer to lash this into USB. I'm still
fine-tuning a few details but will send out a patch soon, probably today. It's
a bit hard to test it and I'm sure it will need revision, but it will help a
whole bunch to have a USB implementation to play with.

> It's not just hot-pluggable devices.  These ideas are appropriate 
> whenever detection or registration is asynchronous.

This is very true, and I should modify the nomenclature to reflect this.

> > There currently two types supported:
> > -	consoles
> > -	network devices
> 
> You might as well add block devices too.

Yes, we should. My guess is block devices that hang off of buses besides USB
would be able to use this.

> > bus_bootdev_initialized(int bootdev_mask)
> > 	This function must be called by the bus support code after it calls the
> > 	initialiation function for a boot device. Every call to
> > 	bus_bootdev_found must be matched by a call to
> > 	bus_bootdev_initialized with the same bootdev_mask.
> 
> Even if the device can't be initialized for some reason, or if it turns
> out not the be the sort of boot device originally thought, this
> function still has to be called.  That should be made extremely clear.

True, and I was hoping that was clear enough, but I'll go back and emphasize
this more.

> > 	Device initialization functions must have called the appropriate
> > 	registration function for that device before bus_bootdev_initialized
> > 	is called. So, for console devices, register_console must be called
> > 	before bus_bootdev_initialized, and for network devices,
> > 	register_netdevice must be called first.
> 
> It's not clear how either of these two functions is bus-specific.

They aren't, not directly.  I *think* there isn't a problem here, but it
is certainly possible that the driver wouldn't register the device until some
time after its probe function is called. If that's the case, we would still
have a race condition when we went to use the device, which would defeat the
whole purpose of this change. Determining whether actually have a problem
here is probably too difficult to do beforehand; if an issue exists will find
it during use. Not the ideal approach, but this is engineering and not math.

> This looks like a good plan and not hard to implement.  It even should 
> be possible to fit USB disk drives into the scheme.

That would definitely rock. 

> Alan Stern

David VomLehn

Re: Wait for console to become available, v3.2

[Jamie Lokier]

David VomLehn wrote:
> > This looks like a good plan and not hard to implement.  It even should 
> > be possible to fit USB disk drives into the scheme.
> 
> That would definitely rock. 

How about this, perhaps in the generic device model:

  1. Whenever a device's existence is detected by its parent bus,
     add the device id to a pool of "potential devices worth waiting for".

  2. Whenever a device is registered, remove that device id from the pool.

  3. Whenever a device is itself a bus, or has subdevices or
     attributes to be discovered, it triggers step 1 for all devices
     found by enumeration (or in the case of USB, whatever you have to
     wait for).  Then the bus can declare itself initialised.
 
  4. The top-level enumeration behaves as though there was a root bus,
     onto which the real buses like PCI etc. are attached as in step 3.

  5. Waiting for console / boot device / userspace waiting for other
     mount points all check this pool for device ids of matching type.

In this, the pool serves the same role as Alan Stern's global counter,
the difference being you can wait for particular types of device when
you need to, and this is more explicit about how a hierarchy is handled.

Device ids in this pool are simply "category" values for what the
device is relevant to - and a waitqueue.  If a PCI device is a serial
port, then goes into category "serial port", because it's relevant if
serial console is requested on the boot command line.

When waiting for a newly powered USB bus to settle, you may get
notification of all devices on it, but you might not know enough about
each device until the individual drivers start fetching descriptors.
Then you can either make every device go temporarily into the pool,
much as if it were a little bus itself, until it has detailed
information about what type of device it is.  Or you can wait until
all those devices have fetched descriptors before the USB bus declares
that its enumeration is complete and removes its own id.

-- Jamie

Re: Wait for console to become available, v3.2

[David VomLehn]

On Fri, Apr 24, 2009 at 11:19:51PM +0100, Jamie Lokier wrote:
> David VomLehn wrote:
> > > This looks like a good plan and not hard to implement.  It even should 
> > > be possible to fit USB disk drives into the scheme.
> > 
> > That would definitely rock. 
> 
> How about this, perhaps in the generic device model:
> 
>   1. Whenever a device's existence is detected by its parent bus,
>      add the device id to a pool of "potential devices worth waiting for".
> 
>   2. Whenever a device is registered, remove that device id from the pool.
> 
>   3. Whenever a device is itself a bus, or has subdevices or
>      attributes to be discovered, it triggers step 1 for all devices
>      found by enumeration (or in the case of USB, whatever you have to
>      wait for).  Then the bus can declare itself initialised.
>  
>   4. The top-level enumeration behaves as though there was a root bus,
>      onto which the real buses like PCI etc. are attached as in step 3.
> 
>   5. Waiting for console / boot device / userspace waiting for other
>      mount points all check this pool for device ids of matching type.
> In this, the pool serves the same role as Alan Stern's global counter,
> the difference being you can wait for particular types of device when
> you need to, and this is more explicit about how a hierarchy is handled.
> 
> Device ids in this pool are simply "category" values for what the
> device is relevant to - and a waitqueue.  If a PCI device is a serial
> port, then goes into category "serial port", because it's relevant if
> serial console is requested on the boot command line.
> 
> When waiting for a newly powered USB bus to settle, you may get
> notification of all devices on it, but you might not know enough about
> each device until the individual drivers start fetching descriptors.
> Then you can either make every device go temporarily into the pool,
> much as if it were a little bus itself, until it has detailed
> information about what type of device it is.  Or you can wait until
> all those devices have fetched descriptors before the USB bus declares
> that its enumeration is complete and removes its own id.

I think this is over-engineered. This focused on boot devices, so you really
don't care about things like buses, and I don't perceive a broader use. What
really matters is particular boot device types, wherever they came from.

I'm just about to send out an RFC patch. It may work, but it sure hasn't been
tested (so we can actually be sure it doesn't work) and it is not integrated
with any bus. Following it will be a patch to handle the specific case of a
console, but even this is not intended for use.

I've been thinking about the issue of handling device classes because, as you
clearly understand, distingishing between them can give you finer granularity
during boot initialization. There are really three possible steps:
1. Discover a device exists.
2. Discover the device type
3. Completion of the probe function for the device.

The existing code is great if the interval between 1 and 2, or 2 and 3, is
nearly zero. In the first case, you do nothing at step 1 and at step 2 you
indicate that a boot device of the given type it found. In the second case,
you indicate that you have found a device of unknown type was found (passing
BOOTDEV_ANY_MASK) at step 1, ignore the information at step 2, and report
completion of the probe for a generic device type at step 3 (again passing
BOOTDEV_ANY_MASK).

There is one additional possibility, that there is a significant amount of time
that passes between steps 1, 2, and 3. The existing interfaces already handle
that, but I'm thinking a clearer interface is in order. The key is that, when
you indicate a possible boot device was found, and when you indicate the
completion of probing, you are actually passing a mask of boot device types.

Say that the device is actually a console, my favorite example. In this case,
you'd pass BOOTDEV_ANY_MASK to bootdev_found at step 1, indicating that you
don't really know the device type. This increments the pending count for all
boot device types. At step 2, you find out you have a console, so you pass
BOOTDEV_ANY_MASK & ~BOOTDEV_CONSOLE_MASK to bootdev_probe_done. This decrements
the pending count for all device types except consoles. Then, at step 3, you
call bootdev_probe_done with BOOTDEV_CONSOLE_MASK. Which decrements the pending
count for console devices and wakes up any waiters.

This is useful functionality only if we have cases where the intervals between
each of the three steps is significant. If so, then my approach is to add a
function:
	void bootdev_type_found(int old_mask, enum bootdev_type type);
which is passed the mask that was passed to bootdev_found and the type the
device is now known to be. Implementation-wise, all it really does, then, is
to call bootdev_probe_done:
	bootdev_probe_done(old_mask & ~(1 << type));
This would be a cleaner interface than calling bootdev_probe_done twice..

The key question is, are there cases where there is enough time between steps
1 and 2, and steps 2 and 3, to add this complexity? If not, let's skip it.

> -- Jamie

David VomLehn

Re: Wait for console to become available, v3.2

[Jamie Lokier]

David VomLehn wrote:
> I think this is over-engineered. This focused on boot devices, so you really
> don't care about things like buses, and I don't perceive a broader use. What
> really matters is particular boot device types, wherever they came from.

I'm thinking this broader use:

    - My boot _script_ is waiting for a disk which identifies as
      UUID=392852908752345749857 to appear before it can mount it on
      /data.  If there's no such disk, it proceeds without it.  It's a
      USB disk, behind a USB hub.

    - My boot script is looking to see if I'm holding down 'z' on the
      keyboard, to do something different.  But how does it know if
      there's a USB keyboard plugged in (behind two USB hubs) that
      hasn't finished being detected?

It just seemed to fit comfortably into what's being discussed.

(I do have these a system with these requirements, by the way.  It's
solved at the moment by waiting 5 seconds after booting, and by using
an older kernel which doesn't have boot parallelisation yet...)

There was a thread about BTRFS wanting to match up multiple disks
being scranned with volume ids some months ago, which might have
similar requirements, I'm not sure.

> I've been thinking about the issue of handling device classes because, as you
> clearly understand, distingishing between them can give you finer granularity
> during boot initialization. There are really three possible steps:
> 1. Discover a device exists.
> 2. Discover the device type
> 3. Completion of the probe function for the device.

Yes.

> The existing code is great if the interval between 1 and 2, or 2 and 3, is
> nearly zero. In the first case, you do nothing at step 1 and at step 2 you
> indicate that a boot device of the given type it found. In the second case,
> you indicate that you have found a device of unknown type was found (passing
> BOOTDEV_ANY_MASK) at step 1, ignore the information at step 2, and report
> completion of the probe for a generic device type at step 3 (again passing
> BOOTDEV_ANY_MASK).

Yes.

> There is one additional possibility, that there is a significant
> amount of time that passes between steps 1, 2, and 3. The existing
> interfaces already handle that, but I'm thinking a clearer interface
> is in order. The key is that, when you indicate a possible boot
> device was found, and when you indicate the completion of probing,
> you are actually passing a mask of boot device types.

This too, yes.

> Say that the device is actually a console, my favorite example. In
> this case, you'd pass BOOTDEV_ANY_MASK to bootdev_found at step 1,
> indicating that you don't really know the device type. This
> increments the pending count for all boot device types. At step 2,
> you find out you have a console, so you pass BOOTDEV_ANY_MASK &
> ~BOOTDEV_CONSOLE_MASK to bootdev_probe_done. This decrements the
> pending count for all device types except consoles. Then, at step 3,
> you call bootdev_probe_done with BOOTDEV_CONSOLE_MASK. Which
> decrements the pending count for console devices and wakes up any
> waiters.

Only one problem I see: what happens when there's an attempt to open
/dev/console before you increment the pending count?  It seems to me
you have to wait for all buses to have been detected, which is why I
mentioned buses, as some buses are _themselves_ slow devices to detect.

> The key question is, are there cases where there is enough time between steps
> 1 and 2, and steps 2 and 3, to add this complexity? If not, let's skip it.

The time between enumerating that a USB device exists and what it's
class is (could be a console?), and actually initialising the device
to find out if it's then usable, including loading firmware, can be a
little while.

I don't know if the times are long enough to matter.

Possibly related to all this: it would be really nice if the ATA
rather slow probe time didn't have to delay boot scripts until they
depend on the not-yet-probed disks, as sometimes they might not.

-- Jamie
 

Re: Wait for console to become available, v3.2

[Alan Stern]

On Sat, 25 Apr 2009, Jamie Lokier wrote:

> I'm thinking this broader use:
> 
>     - My boot _script_ is waiting for a disk which identifies as
>       UUID=392852908752345749857 to appear before it can mount it on
>       /data.  If there's no such disk, it proceeds without it.  It's a
>       USB disk, behind a USB hub.
> 
>     - My boot script is looking to see if I'm holding down 'z' on the
>       keyboard, to do something different.  But how does it know if
>       there's a USB keyboard plugged in (behind two USB hubs) that
>       hasn't finished being detected?
> 
> It just seemed to fit comfortably into what's being discussed.

Are you suggesting this new interface be exported to userspace somehow?

Alan Stern

Re: Wait for console to become available, v3.2

[David VomLehn]

On Sat, Apr 25, 2009 at 02:41:52AM +0100, Jamie Lokier wrote:
> David VomLehn wrote:
> > I think this is over-engineered. This focused on boot devices, so you really
> > don't care about things like buses, and I don't perceive a broader use. What
> > really matters is particular boot device types, wherever they came from.
> 
> I'm thinking this broader use:
> 
>     - My boot _script_ is waiting for a disk which identifies as
>       UUID=392852908752345749857 to appear before it can mount it on
>       /data.  If there's no such disk, it proceeds without it.  It's a
>       USB disk, behind a USB hub.
> 
>     - My boot script is looking to see if I'm holding down 'z' on the
>       keyboard, to do something different.  But how does it know if
>       there's a USB keyboard plugged in (behind two USB hubs) that
>       hasn't finished being detected?
> 
> It just seemed to fit comfortably into what's being discussed.
> 
> (I do have these a system with these requirements, by the way.  It's
> solved at the moment by waiting 5 seconds after booting, and by using
> an older kernel which doesn't have boot parallelisation yet...)

If the keyboard is being used for the console, this should fix your problem.
If not, I think you can fix it by using a hotplug script.

> Only one problem I see: what happens when there's an attempt to open
> /dev/console before you increment the pending count?  It seems to me
> you have to wait for all buses to have been detected, which is why I
> mentioned buses, as some buses are _themselves_ slow devices to detect.

A valid point. I just sent an updated patch to address this.

> > The key question is, are there cases where there is enough time between steps
> > 1 and 2, and steps 2 and 3, to add this complexity? If not, let's skip it.
> 
> The time between enumerating that a USB device exists and what it's
> class is (could be a console?), and actually initialising the device
> to find out if it's then usable, including loading firmware, can be a
> little while.
> 
> I don't know if the times are long enough to matter.

Me neither, but I went ahead and added bus_bootdev_type_found. It's a trivial
piece of code and will allow this case to be handled cleanly.

> Possibly related to all this: it would be really nice if the ATA
> rather slow probe time didn't have to delay boot scripts until they
> depend on the not-yet-probed disks, as sometimes they might not.

Related to boot time, yes, but it sounds like it's not related to the
boot device issue. By the time you get to boot scripts, you've initialized
all the boot devices you're going to.

> -- Jamie

David VomLehn

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Alan Stern wrote:
> On Sat, 25 Apr 2009, Jamie Lokier wrote:
> 
> > I'm thinking this broader use:
> > 
> >     - My boot _script_ is waiting for a disk which identifies as
> >       UUID=392852908752345749857 to appear before it can mount it on
> >       /data.  If there's no such disk, it proceeds without it.  It's a
> >       USB disk, behind a USB hub.
> > 
> >     - My boot script is looking to see if I'm holding down 'z' on the
> >       keyboard, to do something different.  But how does it know if
> >       there's a USB keyboard plugged in (behind two USB hubs) that
> >       hasn't finished being detected?
> > 
> > It just seemed to fit comfortably into what's being discussed.
> 
> Are you suggesting this new interface be exported to userspace somehow?

Not directly.  Only in the same way that open("/dev/console") delays
until there's a console, so reading the keyboard can delay until we
know if we had a keyboard plugged in at boot, and looking for a disk
called UUID=392852908752345749857 can wait until we know if there was
one plugged in at boot time.

The latter issue with UUID is done in userspace now by reading all
disks, but I'm under the impression changes are planned in that
department because reading every disk from userspace to locate
specific ones is too slow on big systems.

-- Jamie

Re: Wait for console to become available, v3.2

[Alan Stern]

On Sun, 26 Apr 2009, Jamie Lokier wrote:

> > Are you suggesting this new interface be exported to userspace somehow?
> 
> Not directly.  Only in the same way that open("/dev/console") delays
> until there's a console, so reading the keyboard can delay until we
> know if we had a keyboard plugged in at boot, and looking for a disk
> called UUID=392852908752345749857 can wait until we know if there was
> one plugged in at boot time.
> 
> The latter issue with UUID is done in userspace now by reading all
> disks, but I'm under the impression changes are planned in that
> department because reading every disk from userspace to locate
> specific ones is too slow on big systems.

IIUC, David is proposing that no userspace process should get started
until some (all?) of the console devices and the block device
containing the root filesystem (all block devices present at boot?)
have been registered.  That would remove the need for your delays, at 
least in part.

As for searching for a particular UUID, I believe recent changes to 
sysfs/udev should improve the situation.  There will be a "by_UUID" 
directory somewhere, containing a bunch of symbolic links whose names 
are the UUID values of all the registered drives.  Programs won't have 
to read every disk; they'll only have to search through this directory.

Alan Stern

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Alan Stern wrote:
> On Sun, 26 Apr 2009, Jamie Lokier wrote:
> 
> > > Are you suggesting this new interface be exported to userspace somehow?
> > 
> > Not directly.  Only in the same way that open("/dev/console") delays
> > until there's a console, so reading the keyboard can delay until we
> > know if we had a keyboard plugged in at boot, and looking for a disk
> > called UUID=392852908752345749857 can wait until we know if there was
> > one plugged in at boot time.
> > 
> > The latter issue with UUID is done in userspace now by reading all
> > disks, but I'm under the impression changes are planned in that
> > department because reading every disk from userspace to locate
> > specific ones is too slow on big systems.
> 
> IIUC, David is proposing that no userspace process should get started
> until some (all?) of the console devices and the block device
> containing the root filesystem (all block devices present at boot?)
> have been registered.  That would remove the need for your delays, at 
> least in part.
> 
> As for searching for a particular UUID, I believe recent changes to 
> sysfs/udev should improve the situation.  There will be a "by_UUID" 
> directory somewhere, containing a bunch of symbolic links whose names 
> are the UUID values of all the registered drives.  Programs won't have 
> to read every disk; they'll only have to search through this directory.

That will be great.

_If_ the system doesn't wait for all block devices present at boot to
be enumerated before the boot script, then when the script looks in
that directory for a specific UUID, it would be good to wait until 
"has everything present at boot been enumerated?" says yes.

Otherwise you have hacks like my boot script which waits 5 seconds for
a disk to show up on USB, and then continues if not.  It sounds
awfully like waiting X seconds for a USB console to show up :-)

Since this is all about making boot faster, it would be quite nice not
to wait for all block devices before starting the boot script, or at
least the initramfs module-loading script :-)

That's outside the scope of bootconsole and root device.  I certainly
won't demand it's done, but I'm sure you can see a strong similarity:
the ability to wait in the kernel until a class of devices that were
present at boot have finished enumerating.

-- Jamie

Re: Wait for console to become available, v3.2

[Kay Sievers]

On Sun, Apr 26, 2009 at 23:37, Jamie Lokier <jamie@shareable.org> wrote:
> Alan Stern wrote:
>> As for searching for a particular UUID, I believe recent changes to
>> sysfs/udev should improve the situation.  There will be a "by_UUID"
>> directory somewhere, containing a bunch of symbolic links whose names
>> are the UUID values of all the registered drives.  Programs won't have
>> to read every disk; they'll only have to search through this directory.
>
> That will be great.

It is like that for years now, and it is completely in userspace at
/dev/disk/by-uuid/. I don't know of any plans to add filesystem/raid
signature uuid probing support to the kernel. And I think that will be
almost impossible to get right in the kernel.

Yes, you need to look at all the disks from userspace, if you request
to mount them by filesystem metadata like label/UUID. It works fine
even on boxes with thousands of disks.

> _If_ the system doesn't wait for all block devices present at boot to
> be enumerated before the boot script, then when the script looks in
> that directory for a specific UUID, it would be good to wait until
> "has everything present at boot been enumerated?" says yes.

That's what distros do with initramfs today.

> Otherwise you have hacks like my boot script which waits 5 seconds for
> a disk to show up on USB, and then continues if not.  It sounds
> awfully like waiting X seconds for a USB console to show up :-)
>
> Since this is all about making boot faster, it would be quite nice not
> to wait for all block devices before starting the boot script, or at
> least the initramfs module-loading script :-)

You wait in initramfs until the device show up. Usually dynamic udev
rules created from the root= string of the kernel commandline create a
rule which matches on the UUID, and creates a /dev/root symlink when
the device shows up. As soon as this link shows up,  init mounts it
and goes ahead.

Thanks,
Kay

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Kay Sievers wrote:
> > _If_ the system doesn't wait for all block devices present at boot to
> > be enumerated before the boot script, then when the script looks in
> > that directory for a specific UUID, it would be good to wait until
> > "has everything present at boot been enumerated?" says yes.
> 
> That's what distros do with initramfs today.

I don't see how that's possible.  Haven't we been discussing how the
USB driver does not have any support (yet) for saying when it's found
every device present at boot time, and that it can probably be added?

I have a boot script which does this:

    1. *If* USB disk plugged in, mount it on /mnt/usb, do some stuff.
    2. If not, do something else.

If I'm not mistaken, the only way to do that right now is pick an
arbitrary time, wait, and proceed with 2 if you didn't see the disk.

The desired USB console behaviour is very similar.  Use it if it's
plugged in at boot, use something else otherwise, and no arbitrary
delays.

Did I miss something?
> > Since this is all about making boot faster, it would be quite nice not
> > to wait for all block devices before starting the boot script, or at
> > least the initramfs module-loading script :-)
> 
> You wait in initramfs until the device show up. Usually dynamic udev
> rules created from the root= string of the kernel commandline create a
> rule which matches on the UUID, and creates a /dev/root symlink when
> the device shows up. As soon as this link shows up,  init mounts it
> and goes ahead.

That's fine when you're waiting for a device to appear that you know
will appear.

The USB console excitement, and my mention of USB disks and such, is
how to decide when the device _isn't_ going to appear, without an
arbitrary timeout.

-- Jamie

Re: Wait for console to become available, v3.2

[Kay Sievers]

On Mon, Apr 27, 2009 at 01:12, Jamie Lokier <jamie@shareable.org> wrote:
> Kay Sievers wrote:
>> > _If_ the system doesn't wait for all block devices present at boot to
>> > be enumerated before the boot script, then when the script looks in
>> > that directory for a specific UUID, it would be good to wait until
>> > "has everything present at boot been enumerated?" says yes.
>>
>> That's what distros do with initramfs today.
>
> I don't see how that's possible.  Haven't we been discussing how the
> USB driver does not have any support (yet) for saying when it's found
> every device present at boot time, and that it can probably be added?

It's easy for the rootfs, or any other mandatory filesystem, unlike it
is for the console stuff.

There is no timeout, distro's initramfs waits for the specified root
device until it appears. It's simple, there is nothing else to do for
it. It does not care what bus it is, or who is probing what for how
long. It loads all drivers it finds hardware for, and then spins until
the device shows up and continues.

Kay

Re: Wait for console to become available, v3.2

[Jamie Lokier]

Kay Sievers wrote:
> On Mon, Apr 27, 2009 at 01:12, Jamie Lokier <jamie@shareable.org> wrote:
> > Kay Sievers wrote:
> >> > _If_ the system doesn't wait for all block devices present at boot to
> >> > be enumerated before the boot script, then when the script looks in
> >> > that directory for a specific UUID, it would be good to wait until
> >> > "has everything present at boot been enumerated?" says yes.
> >>
> >> That's what distros do with initramfs today.
> >
> > I don't see how that's possible.  Haven't we been discussing how the
> > USB driver does not have any support (yet) for saying when it's found
> > every device present at boot time, and that it can probably be added?
> 
> It's easy for the rootfs, or any other mandatory filesystem, unlike it
> is for the console stuff.
> 
> There is no timeout, distro's initramfs waits for the specified root
> device until it appears. It's simple, there is nothing else to do for
> it. It does not care what bus it is, or who is probing what for how
> long. It loads all drivers it finds hardware for, and then spins until
> the device shows up and continues.

Ok.  I'm specifically _not_ talking about mandatory filesystems.
Thanks :-)

-- Jamie