Re: Flames over -- Re: Which is simpler?

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

Both of you are missing an important difference between Suspend-to-RAM and 
Suspend-to-Disk.

Suspend-to-RAM is a true suspend operation, in that the hardware's state
is maintained _in the hardware_.  External buses like USB will retain
suspend power, for instance (assuming the motherboard supports it; some
don't).

Suspend-to-Disk, by contrast, is _not_ a true suspend.  It can more 
accurately be described as checkpoint-and-turn-off.  Hardware state is not 
maintained.  (Some systems may support a special ACPI state that does 
maintain suspend power to external buses during shutdown, I forget what 
it's called.  And I down't know whether swsusp uses this state.)

So for example, let's say you have a filesystem mounted on a USB flash or
disk drive.  With Suspend-to-RAM, there's a very good chance that the
connection and filesystem will still be intact when you resume.  With
Suspend-to-Disk, the USB connection will terminate when the computer shuts
down.  When you resume, the device will be gone and your filesystem will
be screwed.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> Both of you are missing an important difference between Suspend-to-RAM and 
> Suspend-to-Disk.
> 
> Suspend-to-RAM is a true suspend operation, in that the hardware's state
> is maintained _in the hardware_.  External buses like USB will retain
> suspend power, for instance (assuming the motherboard supports it; some
> don't).
> 
> Suspend-to-Disk, by contrast, is _not_ a true suspend.  It can more 
> accurately be described as checkpoint-and-turn-off.  Hardware state is not 
> maintained.  (Some systems may support a special ACPI state that does 
> maintain suspend power to external buses during shutdown, I forget what 
> it's called.  And I down't know whether swsusp uses this state.)
> 

I would disagree.  The only difference between the two is WHERE the 
state is maintained - ram vs. disk.  I won't really argue it though, 
because it's just semantics -- call it whatever you want.

> So for example, let's say you have a filesystem mounted on a USB flash or
> disk drive.  With Suspend-to-RAM, there's a very good chance that the
> connection and filesystem will still be intact when you resume.  With
> Suspend-to-Disk, the USB connection will terminate when the computer shuts
> down.  When you resume, the device will be gone and your filesystem will
> be screwed.
> 

This is not true.  The USB bus is shut down either way, and provided 
that you have not unplugged the disk, nothing will be screwed when you 
resume from disk or ram.

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Sun, 12 Feb 2006, Phillip Susi wrote:

> Alan Stern wrote:
> > Both of you are missing an important difference between Suspend-to-RAM and 
> > Suspend-to-Disk.
> > 
> > Suspend-to-RAM is a true suspend operation, in that the hardware's state
> > is maintained _in the hardware_.  External buses like USB will retain
> > suspend power, for instance (assuming the motherboard supports it; some
> > don't).
> > 
> > Suspend-to-Disk, by contrast, is _not_ a true suspend.  It can more 
> > accurately be described as checkpoint-and-turn-off.  Hardware state is not 
> > maintained.  (Some systems may support a special ACPI state that does 
> > maintain suspend power to external buses during shutdown, I forget what 
> > it's called.  And I down't know whether swsusp uses this state.)
> > 
> 
> I would disagree.  The only difference between the two is WHERE the 
> state is maintained - ram vs. disk.  I won't really argue it though, 
> because it's just semantics -- call it whatever you want.

It's not just semantics.  There's a real difference between maintaining
state in the hardware and maintaining it somewhere else.  The biggest
difference is that if the hardware retains suspend power, it is able to
detect disconnections.  When the system resumes, it _knows_ whether a
device was attached the entire time, as opposed to being unplugged and
replugged (or possibly a different device plugged in!) while the system
was asleep.  If the hardware is down completely, there is no way of
telling for certain whether a device attached to some port is the same one
that was there when the system got suspended.

Another difference is the possibility of remote wakeup.  Clearly it can't 
happen when there's no power available.

> > So for example, let's say you have a filesystem mounted on a USB flash or
> > disk drive.  With Suspend-to-RAM, there's a very good chance that the
> > connection and filesystem will still be intact when you resume.  With
> > Suspend-to-Disk, the USB connection will terminate when the computer shuts
> > down.  When you resume, the device will be gone and your filesystem will
> > be screwed.
> > 
> 
> This is not true.  The USB bus is shut down either way, and provided 
> that you have not unplugged the disk, nothing will be screwed when you 
> resume from disk or ram.

Have you actually tried it?  I have.  

In any case, it is undeniably true that if the bus is shut down then all
the USB connections are lost.  When you resume it will be the same as if
you had unplugged all the USB devices and then replugged them.  Not a good
thing to do when they contain mounted filesystems; all the memory mappings
are invalidated.

(Bear in mind that whether a USB bus gets shut down depends on the
motherboard; some supply suspend power and some don't.  It depends on the
USB controller too; some support low-power states other than "completely
off" and others don't.)

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 12, 2006, at 19:51, Phillip Susi wrote:
> Alan Stern wrote:
>> Both of you are missing an important difference between Suspend-to- 
>> RAM and Suspend-to-Disk. Suspend-to-RAM is a true suspend  
>> operation, in that the hardware's state is maintained _in the  
>> hardware_.  External buses like USB will retain suspend power, for  
>> instance (assuming the motherboard supports it; some don't).
>> Suspend-to-Disk, by contrast, is _not_ a true suspend.  It can  
>> more accurately be described as checkpoint-and-turn-off.  Hardware  
>> state is not maintained.  (Some systems may support a special ACPI  
>> state that does maintain suspend power to external buses during  
>> shutdown, I forget what it's called.  And I down't know whether  
>> swsusp uses this state.)
>
> I would disagree.  The only difference between the two is WHERE the  
> state is maintained - ram vs. disk.  I won't really argue it  
> though, because it's just semantics -- call it whatever you want.

 From the simple perspective, yes, that's the only difference.  On  
the other hand, from an efficiency standpoint, they are _completely_  
different, to a degree that the OS needs to treat them as such or  
performance will suck.  Software suspend (to disk, network, file,  
etc) _requires_ a copy; it's completely mandatory because RAM is  
guaranteed to go away.  It freezes everything, checkpoints the kernel  
with a lot of really complex and oft-buggy code, unfreezes things,  
stores data, and shuts off.  Hardware suspend (to RAM only)  
implicitly needs no copy.  The CPU and memory architecture itself  
supports the low-power state, so you don't have to do much of  
anything special about kernel or userspace threads, you just have to  
suspend the device tree once (already exists for per-device power  
management) and then tell the firmware to finish up.  The former is O 
(memory), the latter is O(1).

For a significant majority of people, hardware suspend is  
significantly better.  I never understood why people need graphical  
splash screens during suspend/resume... until I tried software- 
suspend.  With hardware-suspend, I close it, 2 seconds later the  
little white light on the front comes on and it's sleeping, I open  
it, 2 seconds later it's connected to wireless again and ready to  
use.  With software-suspend, it just _doesn't_ work that fast because  
of the inherent way it does things; hard disks on laptops are _slow_.

>> So for example, let's say you have a filesystem mounted on a USB  
>> flash or disk drive.  With Suspend-to-RAM, there's a very good  
>> chance that the connection and filesystem will still be intact  
>> when you resume.  With  Suspend-to-Disk, the USB connection will  
>> terminate when the computer shuts down.  When you resume, the  
>> device will be gone and your filesystem will be screwed.
>
> This is not true.  The USB bus is shut down either way, and  
> provided that you have not unplugged the disk, nothing will be  
> screwed when you resume from disk or ram.

It depends on the hardware.  For the disk case, yes, this is true.   
On the other hand, for hardware-suspend a number of devices like  
keyboards and mice may still be in a low-power suspend mode, allowing  
you to wake the computer by pushing keys or mouse buttons.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.  The first method is far more difficult.
   -- C.A.R. Hoare

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> 
> It's not just semantics.  There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else.  The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections.  When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep.  If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.
> 

During suspend the hardware is usually completely powered off, and in 
either case, there is nothing running on the CPU to monitor device 
insertion/removal.  When the system is resumed the kernel decides if the 
hardware has changed the same way for either system: it probes the 
hardware to see if it is still there.  There isn't anything special that 
monitors device insertion/removal while suspended to ram.

>> This is not true.  The USB bus is shut down either way, and provided 
>> that you have not unplugged the disk, nothing will be screwed when you 
>> resume from disk or ram.
> 
> Have you actually tried it?  I have.  

If it doesn't work then you have found a bug and should file a report. 
The state of the kernel after resuming from either suspend to disk, or 
suspend to ram is the same.  The filesystem is still mounted, and any 
dirty pages in ram will be flushed just like normal.  Whether the disk 
is connected via SCSI, SATA, USB, or whatever does not matter.

> 
> In any case, it is undeniably true that if the bus is shut down then all
> the USB connections are lost.  When you resume it will be the same as if
> you had unplugged all the USB devices and then replugged them.  Not a good
> thing to do when they contain mounted filesystems; all the memory mappings
> are invalidated.
> 

The kernel knows that the same device is still there on the bus, and so 
it picks up right where it left off.  If it thinks the device has been 
unplugged and reconnected, that is a bug.

> (Bear in mind that whether a USB bus gets shut down depends on the
> motherboard; some supply suspend power and some don't.  It depends on the
> USB controller too; some support low-power states other than "completely
> off" and others don't.)
> 
> Alan Stern
> 

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 12, 2006, at 22:52, Phillip Susi wrote:
> Alan Stern wrote:
>> It's not just semantics.  There's a real difference between  
>> maintaining state in the hardware and maintaining it somewhere  
>> else.  The biggest difference is that if the hardware retains  
>> suspend power, it is able to detect disconnections.  When the  
>> system resumes, it _knows_ whether a device was attached the  
>> entire time, as opposed to being unplugged and replugged (or  
>> possibly a different device plugged in!) while the system was  
>> asleep.  If the hardware is down completely, there is no way of  
>> telling for certain whether a device attached to some port is the  
>> same one that was there when the system got suspended.
>
> During suspend the hardware is usually completely powered off,

This is true for software suspend, but not for hardware suspend (see  
the differences now?)  This is why the two are independent and should  
not be mashed together into one "Generic Suspend".  Let me bring up  
the example of my PowerBook again.  It's RAM is fully powered right  
now, running from battery, and it has another couple days of sleep- 
charge left before I have to worry about plugging it in again.  When  
I open it, the firmware automatically powers up the CPU and other  
hardware and returns control to the OS.  I can _also_ trigger it to  
wake by leaving it closed and connecting an external VGA and USB (it  
wakes every time I connect a USB, but my suspend script puts it to  
sleep again if it's closed and has no external VGA).

> and in either case, there is nothing running on the CPU to monitor  
> device insertion/removal.

You don't need the CPU, just a good USB controller and hubs with low- 
power modes and such.  The fact that plugging in a USB keyboard/mouse  
and a VGA monitor is enough to wake the system when properly  
configured should be proof enough.

> When the system is resumed the kernel decides if the hardware has  
> changed the same way for either system: it probes the hardware to  
> see if it is still there.  There isn't anything special that  
> monitors device insertion/removal while suspended to ram.

Sometimes not, but again, it depends on the hardware.

Cheers,
Kyle Moffett

--
I have yet to see any problem, however complicated, which, when you  
looked at it in the right way, did not become still more complicated.
   -- Poul Anderson

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Sun, 12 Feb 2006, Phillip Susi wrote:

> During suspend the hardware is usually completely powered off, and in 
> either case, there is nothing running on the CPU to monitor device 
> insertion/removal.

Like Kyle said, this depends to some extent on the system.  However, 
during Suspend-to-RAM it is definitely true that the RAM at least is 
powered on.  Other components may be powered as well.  Otherwise you 
wouldn't be able to awaken the system by pressing a button/opening the 
case/whatever.

>  When the system is resumed the kernel decides if the 
> hardware has changed the same way for either system: it probes the 
> hardware to see if it is still there.  There isn't anything special that 
> monitors device insertion/removal while suspended to ram.

Sorry, but you're wrong.  First of all, testing if hardware is there is
different from testing whether it has changed -- it could have changed 
while the system was asleep, with the result that hardware is indeed there 
but it's not the _same_ hardware.

Second, with USB at any rate, in addition to checking that hardware is 
still there, the kernel queries the USB controller to see if a disconnect 
occurred while the system was asleep.  (If the controller wasn't powered 
during that time then it will report that every USB device was 
disconnected.)

Third, there is indeed something special that monitors USB device 
insertion/removal while suspended to RAM -- the USB host controller does 
so if it has suspend power.

> >> This is not true.  The USB bus is shut down either way, and provided 
> >> that you have not unplugged the disk, nothing will be screwed when you 
> >> resume from disk or ram.
> > 
> > Have you actually tried it?  I have.  
> 
> If it doesn't work then you have found a bug and should file a report. 

No.  It does work exactly as designed and it's not buggy.  You just don't
understand it.

> The state of the kernel after resuming from either suspend to disk, or 
> suspend to ram is the same.  The filesystem is still mounted, and any 
> dirty pages in ram will be flushed just like normal.  Whether the disk 
> is connected via SCSI, SATA, USB, or whatever does not matter.

Don't be silly.  Dirty pages can't be flushed to disks that are no longer
attached!  And if a USB disk was unplugged while the system was asleep,
the kernel will know that it is no longer attached.  I don't know which
other bus drivers check for this sort of thing.

> The kernel knows that the same device is still there on the bus, and so 
> it picks up right where it left off.  If it thinks the device has been 
> unplugged and reconnected, that is a bug.

It's not a bug if the device _has_ been unplugged and reconnected.  When 
that happens, there's no way for the kernel to tell whether the device 
there now is the same as the device that used to be there.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> This is true for software suspend, but not for hardware suspend (see 
> the differences now?)  This is why the two are independent and should 
> not be

No, that is not necessarily correct.  Sometimes the ACPI bios can leave 
certain devices in a standby mode so they can wake the machine, but it 
does not have to, and often does not.  Thus when suspended to ram, 
typically your usb hard drive and almost allways your ide/sata/scsi 
drive will be completely powered off. 

> mashed together into one "Generic Suspend".  Let me bring up the 
> example of my PowerBook again.  It's RAM is fully powered right now, 
> running from battery, and it has another couple days of sleep-charge 
> left before I have to worry about plugging it in again.  When I open 
> it, the firmware automatically powers up the CPU and other hardware 
> and returns control to the OS.  I can _also_ trigger it to wake by 
> leaving it closed and connecting an external VGA and USB (it wakes 
> every time I connect a USB, but my suspend script puts it to sleep 
> again if it's closed and has no external VGA).
>

Then your motherboard keeps the bus in a lower power state such that it 
is capable of causing a wake event when state changes.  I'm also fairly 
sure that when such a wake event happens, there is no indication to the 
kernel about why it was woken up.  Because of that, and the fact that 
not all systems even support such wake modes, the kernel must reprobe 
all hardware when it wakes up, and hopefully finds the same devices that 
were there when it went to sleep.  It does this for both types of suspend. 

>> and in either case, there is nothing running on the CPU to monitor 
>> device insertion/removal.
>
> You don't need the CPU, just a good USB controller and hubs with 
> low-power modes and such.  The fact that plugging in a USB 
> keyboard/mouse and a VGA monitor is enough to wake the system when 
> properly configured should be proof enough.
>

That is not proof of anything other than the bus controller has the 
capability of generating a wake event.  As I said before, once woken up, 
the kernel must probe all hardware and if there is a new device, it will 
find it.  The hardware only detected that something changed and thus, 
generated a wake even, it did not notice exactly what that change was 
and inform the kernel. 

>> When the system is resumed the kernel decides if the hardware has 
>> changed the same way for either system: it probes the hardware to see 
>> if it is still there.  There isn't anything special that monitors 
>> device insertion/removal while suspended to ram.
>
> Sometimes not, but again, it depends on the hardware.

Again, always not since the hardware doesn't actually tell the kernel 
what happend, it just wakes it up. 

>
> Cheers,
> Kyle Moffett
>
> -- 
> I have yet to see any problem, however complicated, which, when you 
> looked at it in the right way, did not become still more complicated.
>   -- Poul Anderson
>
>

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> Sorry, but you're wrong.  First of all, testing if hardware is there is
> different from testing whether it has changed -- it could have changed 
> while the system was asleep, with the result that hardware is indeed there 
> but it's not the _same_ hardware.
>
>   

If you believe I am wrong, then please offer proof.  Specifically, refer 
to the kernel code that handles verifying that the hardware is still 
there after a resume.  I very well may be wrong, but you can not simply 
surmise this to be so without any proof.  It is currently my belief that 
the kernel probes the hardware the same way after a cold boot, resume 
from s-t-r, and resume from s-t-d.  Specifically it looks to see if a 
device is located in the same bus location with the same serial number, 
etc, and if so, access to it continues exactly where it left off prior 
to the suspend. 

This is why you can suspend to disk, and when you resume, your open 
files are still valid; the disk is found to be still there, so it 
continues to be accessible.  Nothing gets clobbered as a result of the 
disk seeming to be disconnected and reconnected.  If that does happen, 
it is a bug. 

> Second, with USB at any rate, in addition to checking that hardware is 
> still there, the kernel queries the USB controller to see if a disconnect 
> occurred while the system was asleep.  (If the controller wasn't powered 
> during that time then it will report that every USB device was 
> disconnected.)
>   

AFAIK, there is no interface by which the kernel can query that 
information from the controller, maybe you could show me?  If that is 
the case however, then I consider that to be a bug with the USB bus and 
the kernel's handling of it.  The kernel needs to be able to assume that 
nothing was disconnected while it was shutdown, provided that the same 
devices are there now as when it went to sleep.  This is how it behaves 
for say, SCSI disks in desktops/servers, where the controller certainly 
is completely powered off.  It should work the same for USB. 
> Third, there is indeed something special that monitors USB device 
> insertion/removal while suspended to RAM -- the USB host controller does 
> so if it has suspend power.
>
>   

Could you site references to that?  AFAIK, the host controller is only 
capable of generating a wake event when the bus state changes; it does 
not have a means of informing the OS what has changed, aside from the OS 
enumerating the devices on the bus again. 

> No.  It does work exactly as designed and it's not buggy.  You just don't
> understand it.
>
>   

If the mounted filesystem becomes corrupted over a hibernation because 
the kernel thinks the drive was unplugged, then plugged back in, that 
clearly is a bug. It does not do this for disks connected via other bus 
types, and it clearly is undesirable to corrupt data in this way. 

>> The state of the kernel after resuming from either suspend to disk, or 
>> suspend to ram is the same.  The filesystem is still mounted, and any 
>> dirty pages in ram will be flushed just like normal.  Whether the disk 
>> is connected via SCSI, SATA, USB, or whatever does not matter.
>>     
>
> Don't be silly.  Dirty pages can't be flushed to disks that are no longer
> attached!  And if a USB disk was unplugged while the system was asleep,
> the kernel will know that it is no longer attached.  I don't know which
> other bus drivers check for this sort of thing.
>
>   
The disk is still attached of course.  The scenario we are talking about 
does not actually involve disconnecting the drive.  Since the drive is 
not actually disconnected, if the kernel believes it has been, it's a bug. 
>> The kernel knows that the same device is still there on the bus, and so 
>> it picks up right where it left off.  If it thinks the device has been 
>> unplugged and reconnected, that is a bug.
>>     
>
> It's not a bug if the device _has_ been unplugged and reconnected.  When 
> that happens, there's no way for the kernel to tell whether the device 
> there now is the same as the device that used to be there.
>
>   

Again, we're not talking about it actually being unplugged, though since 
the kernel has no way of knowing it, you can unplug a scsi disk while 
hibernated, then plug it back in before you resume, and it will work 
just fine since the same type of hardware with the same serial number et 
al is found in the same place on the bus. 

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Mon, 13 Feb 2006, Phillip Susi wrote:

> If you believe I am wrong, then please offer proof.  Specifically, refer 
> to the kernel code that handles verifying that the hardware is still 
> there after a resume.  I very well may be wrong, but you can not simply 
> surmise this to be so without any proof.  It is currently my belief that 
> the kernel probes the hardware the same way after a cold boot, resume 
> from s-t-r, and resume from s-t-d.  Specifically it looks to see if a 
> device is located in the same bus location with the same serial number, 
> etc, and if so, access to it continues exactly where it left off prior 
> to the suspend. 

Okay.  Take a look at drivers/usb/core/hub.c.  The usb_resume_device()  
routine is called when resuming either from STR or STD.  If
CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
calls finish_device_resume().  Inside finish_device_resume() is a call to
usb_get_status(), which will fail if the device has not been connected and
powered-up throughout the entire suspend.  That failure will cause 
hub_port_resume() to call hub_port_logical_disconnect(), which has the 
effect of doing a logical disconnect on the device.

There are other, redundant code paths that perform this check even when 
CONFIG_USB_SUSPEND isn't set, but they are more difficult to describe.  
For example, look at uhci_check_and_reset_hc() in 
drivers/usb/host/pci-quirks.c.

You'll find that nowhere in the resume pathway does the kernel check
serial numbers or anything else of that nature.  If the power session has
not been interrupted, that's sufficient proof that the device hasn't been
unplugged.

> This is why you can suspend to disk, and when you resume, your open 
> files are still valid; the disk is found to be still there, so it 
> continues to be accessible.  Nothing gets clobbered as a result of the 
> disk seeming to be disconnected and reconnected.  If that does happen, 
> it is a bug. 

It may work that way with SCSI disks or IDE disks, which are not
hotpluggable.  But it does happen with USB disks, and it's not a bug;
it's by design.


> > Second, with USB at any rate, in addition to checking that hardware is 
> > still there, the kernel queries the USB controller to see if a disconnect 
> > occurred while the system was asleep.  (If the controller wasn't powered 
> > during that time then it will report that every USB device was 
> > disconnected.)
> >   
> 
> AFAIK, there is no interface by which the kernel can query that 
> information from the controller, maybe you could show me?

Again, in hub.c look at hub_events().  It's a rather long routine, but at 
some point you can see where it checks (portchange & 
USB_PORT_STAT_C_CONNECTION).  The constant stands for "Port Status Changed 
Connection", meaning there has been a plug/unplug event.  If the test 
succeeds then connect_change is set to 1, causing 
hub_port_connect_change() to be called.  One of the first things that 
routine does is call usb_disconnect() on the port's child device.

>  If that is 
> the case however, then I consider that to be a bug with the USB bus and 
> the kernel's handling of it.  The kernel needs to be able to assume that 
> nothing was disconnected while it was shutdown, provided that the same 
> devices are there now as when it went to sleep.

You've got it exactly backwards.  The kernel doesn't need to make that 
assumption because the hardware will _tell_ it whether anything was 
disconnected.  Rather, the kernel needs to _avoid_ making the assumption 
that the device there now is the same as the device that was there before, 
merely because a serial number (or something equivalent) happens to match.  
Note: many USB mass storage devices don't have serial numbers.

>  This is how it behaves 
> for say, SCSI disks in desktops/servers, where the controller certainly 
> is completely powered off.  It should work the same for USB. 

No it shouldn't.  USB is a different kind of bus from SCSI; it has
different specifications and standards, and it should behave differently.


> > Third, there is indeed something special that monitors USB device 
> > insertion/removal while suspended to RAM -- the USB host controller does 
> > so if it has suspend power.
> >
> >   
> 
> Could you site references to that?  AFAIK, the host controller is only 
> capable of generating a wake event when the bus state changes; it does 
> not have a means of informing the OS what has changed, aside from the OS 
> enumerating the devices on the bus again. 

Take a look, for example, at the UHCI specification (available from 
<http://developer.intel.com/technology/usb/uhci11d.htm>).  Section 2.1.7 
describes the Port Status and Control register, which indicates (for each 
port) whether a connect-change event has taken place, as well as many 
other things.


> If the mounted filesystem becomes corrupted over a hibernation because 
> the kernel thinks the drive was unplugged, then plugged back in, that 
> clearly is a bug. It does not do this for disks connected via other bus 
> types, and it clearly is undesirable to corrupt data in this way. 

I agree that it's annoying and undesirable, but it's not a bug.  Other 
buses would work this way too if they were hotpluggable, like USB.


> > It's not a bug if the device _has_ been unplugged and reconnected.  When 
> > that happens, there's no way for the kernel to tell whether the device 
> > there now is the same as the device that used to be there.
> >
> >   
> 
> Again, we're not talking about it actually being unplugged, though since 
> the kernel has no way of knowing it, you can unplug a scsi disk while 
> hibernated, then plug it back in before you resume, and it will work 
> just fine since the same type of hardware with the same serial number et 
> al is found in the same place on the bus. 

As I said above, SCSI isn't the same as USB.

By the way, usb-storage in 2.4 used to work (still does, in fact) more 
along the lines you're describing.  You could unplug a drive and the 
kernel's disk data structures would be kept intact.  Later on, when you 
replugged the drive it would be re-associated with those data structures, 
using some not-go-great heuristics for trying to find a match, and you 
could pick up where you left off.

Then at some point during the 2.5 development sequence, Linus put his foot
down.  He said that when a device goes away, it's _gone_!  That was the
end of it.  Ever since, unplugging a USB drive (or any other kind of USB
device) causes all its device structures to be released.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> Okay.  Take a look at drivers/usb/core/hub.c.  The usb_resume_device()  
> routine is called when resuming either from STR or STD.  If
> CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
> calls finish_device_resume().  Inside finish_device_resume() is a call to
> usb_get_status(), which will fail if the device has not been connected and
> powered-up throughout the entire suspend.  That failure will cause 
> hub_port_resume() to call hub_port_logical_disconnect(), which has the 
> effect of doing a logical disconnect on the device.
>   

Interesting.  How does usb_get_status() decide if the device has been 
connected or not the entire time?  And do you not agree that if it 
indicates that the device was disconnected during hibernation, when it 
in fact, was not, that is a bug?
> There are other, redundant code paths that perform this check even when 
> CONFIG_USB_SUSPEND isn't set, but they are more difficult to describe.  
> For example, look at uhci_check_and_reset_hc() in 
> drivers/usb/host/pci-quirks.c.
>
> You'll find that nowhere in the resume pathway does the kernel check
> serial numbers or anything else of that nature.  If the power session has
> not been interrupted, that's sufficient proof that the device hasn't been
> unplugged.
>
>   
And what if the entire bus was completely powered off, which some 
systems do?  I know my SCSI bus is completely powered off during 
hibernation, yet the kernel has no problem figuring out that the same 
devices are still connected after resume, so it doesn't generate a 
disconnect event. 
> It may work that way with SCSI disks or IDE disks, which are not
> hotpluggable.  But it does happen with USB disks, and it's not a bug;
> it's by design.
>   
SCSI and IDE very well can be hot pluggable.  I have hot plugged 
external SCSI devices numerous times, and even internal IDE drives a 
time or two.  Clearly if the kernel thinks you disconnected your drive 
and causes data loss, when this is not true, it is a bug.  If it is a 
flaw by design, that is still a class of bug. 
> Again, in hub.c look at hub_events().  It's a rather long routine, but at 
> some point you can see where it checks (portchange & 
> USB_PORT_STAT_C_CONNECTION).  The constant stands for "Port Status Changed 
> Connection", meaning there has been a plug/unplug event.  If the test 
> succeeds then connect_change is set to 1, causing 
> hub_port_connect_change() to be called.  One of the first things that 
> routine does is call usb_disconnect() on the port's child device.
>
>   
Interesting again.  This bit exists for each node on the bus and is 
tracked by the hardware?  If that is the case, and the hardware is 
informing the kernel that all devices were disconnected during 
hibernation when this is not the case, then this clearly is a bug in the 
hardware, and the kernel possibly should work around it knowing that the 
hardware lies. 
> You've got it exactly backwards.  The kernel doesn't need to make that 
> assumption because the hardware will _tell_ it whether anything was 
>   
Even if some hardware does, there is a lot of hardware that does not.  
If hardware that is capable of delivering this information does so in an 
unreliable manner ( i.e. it lies and says everything is disconnected 
when it isn't ) then the kernel should ignore that information. 
> disconnected.  Rather, the kernel needs to _avoid_ making the assumption 
> that the device there now is the same as the device that was there before, 
> merely because a serial number (or something equivalent) happens to match.  
> Note: many USB mass storage devices don't have serial numbers.
>
>   
If the hardware is capable of accurately and reliably informing the 
kernel about this information, then that would be useful, but if it is 
not, then seeing a device that appears to be the same as the one you 
expect to be there is good enough to decide to continue using it rather 
than force data loss.  It works fine like that for SCSI and IDE because 
the user expects the system to work properly after they suspend/resume 
when they don't mess with the hardware.  If you actually disconnect a 
device and replace it with another one that otherwise looks the same, 
but isn't really, then all bets are off. 
>>  This is how it behaves 
>> for say, SCSI disks in desktops/servers, where the controller certainly 
>> is completely powered off.  It should work the same for USB. 
>>     
>
> No it shouldn't.  USB is a different kind of bus from SCSI; it has
> different specifications and standards, and it should behave differently.
>
>   
Not if "differently" means "causes data loss when hibernating". 
> Take a look, for example, at the UHCI specification (available from 
> <http://developer.intel.com/technology/usb/uhci11d.htm>).  Section 2.1.7 
> describes the Port Status and Control register, which indicates (for each 
> port) whether a connect-change event has taken place, as well as many 
> other things.
>
>   
I'll have to read up on that.  If that is the case, then it seems the 
hardware is broken because it incorrectly indicates disconnects that did 
not actually happen. If this is a known problem, then the kernel should 
work around it to avoid data loss. 
>
> I agree that it's annoying and undesirable, but it's not a bug.  Other 
> buses would work this way too if they were hotpluggable, like USB.
>   
Other busses can be hot plugged in this way without causing incorrect 
disconnect events and data loss over hibernation.  The fact that USB 
causes data loss in the face of such a benegin event as hibernating and 
resuming ( with no actual hardware change ) is a bug.  Claiming that 
data loss is an acceptable price to pay for being able to hot plug is 
silly. 
> As I said above, SCSI isn't the same as USB.
>   
Right... which doesn't make this any less of a bug in the USB stack. 
> By the way, usb-storage in 2.4 used to work (still does, in fact) more 
> along the lines you're describing.  You could unplug a drive and the 
> kernel's disk data structures would be kept intact.  Later on, when you 
> replugged the drive it would be re-associated with those data structures, 
> using some not-go-great heuristics for trying to find a match, and you 
> could pick up where you left off.
>
> Then at some point during the 2.5 development sequence, Linus put his foot
> down.  He said that when a device goes away, it's _gone_!  That was the
> end of it.  Ever since, unplugging a USB drive (or any other kind of USB
> device) causes all its device structures to be released.
>   

If you ACTUALLY unplug the drive, that's fine... if you don't though, 
and the kernel thinks you did, that is a bug.  If the kernel can 
reasonably decide that the drive has not actually been unplugged even 
though the busted hardware indicates it has, then it should not generate 
a disconnect. 

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Mon, 13 Feb 2006, Phillip Susi wrote:

> Alan Stern wrote:
> > Okay.  Take a look at drivers/usb/core/hub.c.  The usb_resume_device()  
> > routine is called when resuming either from STR or STD.  If
> > CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
> > calls finish_device_resume().  Inside finish_device_resume() is a call to
> > usb_get_status(), which will fail if the device has not been connected and
> > powered-up throughout the entire suspend.  That failure will cause 
> > hub_port_resume() to call hub_port_logical_disconnect(), which has the 
> > effect of doing a logical disconnect on the device.
> >   
> 
> Interesting.  How does usb_get_status() decide if the device has been 
> connected or not the entire time?

It doesn't decide that -- the device itself does.  If the device was
connected the entire time then it will respond properly.  If it was
disconnected then it will reset itself, losing its address.  Hence it will
not reply to further requests at the old address.  usb_get_status() simply
indicates whether or not a response was received.

>  And do you not agree that if it 
> indicates that the device was disconnected during hibernation, when it 
> in fact, was not, that is a bug?

Note: By "disconnected", I mean that the power session was interrupted.  
So even if the cable remained plugged in, if the bus suspend power wasn't
present then the device was disconnected.  Note also that it is impossible
to tell whether the cable has been unplugged -- the hardware is capable of
detecting only whether or not the power session was interrupted.

Given those caveats, yes, I agree that the routine should not indicate the 
device was disconnected if in fact it wasn't.

> And what if the entire bus was completely powered off, which some 
> systems do?  I know my SCSI bus is completely powered off during 
> hibernation, yet the kernel has no problem figuring out that the same 
> devices are still connected after resume, so it doesn't generate a 
> disconnect event. 

Does the kernel have any problem figuring out when a _different_ device 
of the same type is connected at the old address after resume?

With USB, if the entire bus is powered off then every device on it is
automatically disconnected.  By definition.

> > It may work that way with SCSI disks or IDE disks, which are not
> > hotpluggable.  But it does happen with USB disks, and it's not a bug;
> > it's by design.
> >   
> SCSI and IDE very well can be hot pluggable.  I have hot plugged 
> external SCSI devices numerous times, and even internal IDE drives a 
> time or two.

Have you tried unplugging a SCSI or IDE drive while it was mounted and the 
system was suspended, and then plugging in a different drive in its place?

>  Clearly if the kernel thinks you disconnected your drive 
> and causes data loss, when this is not true, it is a bug.  If it is a 
> flaw by design, that is still a class of bug. 

No.  A bug is unintentional whereas a design flaw is intentional.

You are ignoring the question of how the kernel can tell whether two 
devices are in fact the same.  There is no safe way to do this, other than 
having the hardware verify that the device was connected the whole time.

> > Again, in hub.c look at hub_events().  It's a rather long routine, but at 
> > some point you can see where it checks (portchange & 
> > USB_PORT_STAT_C_CONNECTION).  The constant stands for "Port Status Changed 
> > Connection", meaning there has been a plug/unplug event.  If the test 
> > succeeds then connect_change is set to 1, causing 
> > hub_port_connect_change() to be called.  One of the first things that 
> > routine does is call usb_disconnect() on the port's child device.
> >
> >   
> Interesting again.  This bit exists for each node on the bus and is 
> tracked by the hardware?

Yes.  The registers in the host controller only keep track of devices 
plugged directly into the computer.  Similar registers in external hubs 
keep track of the devices plugged into them.

>  If that is the case, and the hardware is 
> informing the kernel that all devices were disconnected during 
> hibernation when this is not the case, then this clearly is a bug in the 
> hardware, and the kernel possibly should work around it knowing that the 
> hardware lies. 

It's a matter of definition.  By definition, "disconnected" means 
essentially the same thing as "power interrupted".  If you use the wrong 
definition, of course you will think that the hardware lies.

> > You've got it exactly backwards.  The kernel doesn't need to make that 
> > assumption because the hardware will _tell_ it whether anything was 
> >   
> Even if some hardware does, there is a lot of hardware that does not.  
> If hardware that is capable of delivering this information does so in an 
> unreliable manner ( i.e. it lies and says everything is disconnected 
> when it isn't ) then the kernel should ignore that information. 

But it is very reliable.  And all USB hardware does it; it's part of the 
specification.

> If the hardware is capable of accurately and reliably informing the 
> kernel about this information, then that would be useful, but if it is 
> not, then seeing a device that appears to be the same as the one you 
> expect to be there is good enough to decide to continue using it rather 
> than force data loss.  It works fine like that for SCSI and IDE because 
> the user expects the system to work properly after they suspend/resume 
> when they don't mess with the hardware.  If you actually disconnect a 
> device and replace it with another one that otherwise looks the same, 
> but isn't really, then all bets are off.

Indeed.  With USB, many devices look the same.
 
> > No it shouldn't.  USB is a different kind of bus from SCSI; it has
> > different specifications and standards, and it should behave differently.
> >
> >   
> Not if "differently" means "causes data loss when hibernating". 

It works both ways.  What about "causes data loss when a different device 
is plugged in"?

> Other busses can be hot plugged in this way without causing incorrect 
> disconnect events and data loss over hibernation.  The fact that USB 
> causes data loss in the face of such a benegin event as hibernating and 
> resuming ( with no actual hardware change ) is a bug.  Claiming that 
> data loss is an acceptable price to pay for being able to hot plug is 
> silly. 

> If you ACTUALLY unplug the drive, that's fine... if you don't though, 
> and the kernel thinks you did, that is a bug.  If the kernel can 
> reasonably decide that the drive has not actually been unplugged even 
> though the busted hardware indicates it has, then it should not generate 
> a disconnect. 

You are complaining because you don't like the way USB was designed.  
That's fine, but it leaves you advocating a non-standardized position.

Can you suggest a _reliable_ way to tell if the USB device present at a 
port after resuming is the same device as was there before suspending?

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Rafael J. Wysocki]

Hi,

On Monday 13 February 2006 22:24, Alan Stern wrote:
> On Mon, 13 Feb 2006, Phillip Susi wrote:
}-- snip --{
> You are complaining because you don't like the way USB was designed.  
> That's fine, but it leaves you advocating a non-standardized position.
> 
> Can you suggest a _reliable_ way to tell if the USB device present at a 
> port after resuming is the same device as was there before suspending?

It seems to follow from your discussion that if I have a mounted filesystem
on a USB device and I suspend to disk, I can lose data unless the filesystem
has been mounted with "sync".

If this is the case, there should be a big fat warning in the swsusp
documentation, but there's nothing like that in there (at lease I can't find
it easily).

[If this is not the case, I've missed something and sorry for the noise.]

Greetings,
Rafael

Re: Flames over -- Re: Which is simpler?

[J. Bruce Fields]

On Mon, Feb 13, 2006 at 04:24:30PM -0500, Alan Stern wrote:
> Can you suggest a _reliable_ way to tell if the USB device present at a 
> port after resuming is the same device as was there before suspending?

That's not really enough, is it?  What if you suspend, unplug your usb
flash drive, plug it into your camera, take a few pictures, plug it back
into the camera, then resume?

If you don't throw away the old mount, that sounds equivalent to
allowing writes to /dev/hda1 while it has a local filesystem mounted on
it.  There may be limits to what you can do to prevent that in this
case, but it seems best to err on the side of caution.

--b.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> It doesn't decide that -- the device itself does.  If the device was
> connected the entire time then it will respond properly.  If it was
> disconnected then it will reset itself, losing its address.  Hence it will
> not reply to further requests at the old address.  usb_get_status() simply
> indicates whether or not a response was received.
>   

I see.  Then this information is unreliable and should not be trusted ( 
when resuming from a suspend ), as it leads to incorrect behavior. 
> Note: By "disconnected", I mean that the power session was interrupted.  
> So even if the cable remained plugged in, if the bus suspend power wasn't
> present then the device was disconnected.  Note also that it is impossible
> to tell whether the cable has been unplugged -- the hardware is capable of
> detecting only whether or not the power session was interrupted.
>
> Given those caveats, yes, I agree that the routine should not indicate the 
> device was disconnected if in fact it wasn't.
>
>   
Exactly.  Yes, there is no good way to determine _for certain_ that the 
user did no do something stupid, such as replace the drive with another 
one just like it, or change the contents in another machine, but that is 
no reason to assume that the user DID do something like that, and break 
the mount, when they in fact, did nothing of the sort. 
> Does the kernel have any problem figuring out when a _different_ device 
> of the same type is connected at the old address after resume?
>
> With USB, if the entire bus is powered off then every device on it is
> automatically disconnected.  By definition.
>
>   
Then the kernel needs to ignore those disconnect events ( provided that 
the device appears to still actually be there ) because they are false 
and lead to data loss. 
> Have you tried unplugging a SCSI or IDE drive while it was mounted and the 
> system was suspended, and then plugging in a different drive in its place?
>   
No, because that would be a foolish user error. 
> No.  A bug is unintentional whereas a design flaw is intentional.
>   
It doesn't matter if it is broken by design, or broken by 
implementation; it's still broken. 
> You are ignoring the question of how the kernel can tell whether two 
> devices are in fact the same.  There is no safe way to do this, other than 
> having the hardware verify that the device was connected the whole time.
>
>   
If there is no better way to tell for sure that the device that is now 
there is not the same as the one that was there, then the kernel must 
assume the user did not do something stupid and continue to use the 
device as if it was not disconnected ( because odds are, it in fact, was 
not ).  In other words, which is more safe?  ALLWAYS loosing data 
because you can't be absolutely sure that the device is the same, or 
only loosing data if the user does something as foolish as swapping 
drives while suspended, and you can't tell they did?
> It's a matter of definition.  By definition, "disconnected" means 
> essentially the same thing as "power interrupted".  If you use the wrong 
> definition, of course you will think that the hardware lies.
>
>   
And that appears to be exactly what the kernel is doing; interpreting 
the hardware "power interrupted" flag as "disconnected", which leads to 
broken behavior and data loss. 
>
> But it is very reliable.  And all USB hardware does it; it's part of the 
> specification.
>   

Except when the bus is powered down, in which case, using that flag as 
an indicator that the device has in fact changed, is not reliable. 
> It works both ways.  What about "causes data loss when a different device 
> is plugged in"?
>
>   
Again, that's user error, not normal operation.  You shouldn't go 
changing devices around while suspended because it _might_ confuse the 
kernel.  The way things are now, the kernel _will_ get confused when the 
user does nothing wrong. 
> You are complaining because you don't like the way USB was designed.  
> That's fine, but it leaves you advocating a non-standardized position.
>   
No, I am complaining because the kernel interprets the notice that the 
bus gives that the device _may_ have changed as a notice that the device 
in fact, _has_ changed, and causes data loss when nothing is actually 
wrong, and this could easily be avoided. 
> Can you suggest a _reliable_ way to tell if the USB device present at a 
> port after resuming is the same device as was there before suspending?
>
> Alan Stern
>   
Again, if it appears to be the same as best you can tell, there is no 
reason to require absolute certainty that it actually is; you can assume 
that the user did not do foolish things while suspended.  This is a much 
safer assumption than always assuming that they DID because that always 
leads to data loss, whereas assuming they didn't ( which they won't most 
of the time ) doesn't lead to data loss when they didn't. 

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 13, 2006, at 18:47:41, Phillip Susi wrote:
> Alan Stern wrote:
>> It doesn't decide that -- the device itself does.  If the device  
>> was connected the entire time then it will respond properly.  If  
>> it was disconnected then it will reset itself, losing its  
>> address.  Hence it will not reply to further requests at the old  
>> address.  usb_get_status() simply indicates whether or not a  
>> response was received.
>
> I see.  Then this information is unreliable and should not be  
> trusted ( when resuming from a suspend ), as it leads to incorrect  
> behavior.

No, that information is the most reliable that can be obtained.  It  
tells us that we can no longer make any guarantees about the device  
or its state.  The USB spec is quite clear on this point.

>> Note: By "disconnected", I mean that the power session was  
>> interrupted.  So even if the cable remained plugged in, if the bus  
>> suspend power wasn't present then the device was disconnected.   
>> Note also that it is impossible to tell whether the cable has been  
>> unplugged -- the hardware is capable of detecting only whether or  
>> not the power session was interrupted.
>>
>> Given those caveats, yes, I agree that the routine should not  
>> indicate the device was disconnected if in fact it wasn't.
>
> Exactly.  Yes, there is no good way to determine _for certain_ that  
> the user did no do something stupid, such as replace the drive with  
> another one just like it, or change the contents in another  
> machine, but that is no reason to assume that the user DID do  
> something like that, and break the mount, when they in fact, did  
> nothing of the sort.

Except we can't reliably decide that.  Say I plug my USB camera in,  
mount it, and download some pictures.  I then suspend the computer,  
unplug the camera after suspending, take more pictures, plug it back  
in and resume.  That's a fairly reasonable situation and the computer  
considering the camera's state to be unchanged would be a serious bug  
and probably result in data loss.  By contrast, just considering the  
camera to be spontaneously unplugged would cause no more data loss  
than actually spontaneously unplugging the flash drive.

>> Does the kernel have any problem figuring out when a _different_  
>> device of the same type is connected at the old address after resume?
>>
>> With USB, if the entire bus is powered off then every device on it  
>> is automatically disconnected.  By definition.
>
> Then the kernel needs to ignore those disconnect events ( provided  
> that the device appears to still actually be there ) because they  
> are false and lead to data loss.

This is why hardware suspend is a good thing.  When I suspend and  
resume my laptop, there are _no_ USB disconnects.  The controller  
puts all the hubs into low-power mode, but it never disconnects them  
or causes problems.

>> You are complaining because you don't like the way USB was  
>> designed.  That's fine, but it leaves you advocating a non- 
>> standardized position.
>
> No, I am complaining because the kernel interprets the notice that  
> the bus gives that the device _may_ have changed as a notice that  
> the device in fact, _has_ changed,

No, you have this wrong.  The USB spec indicates that the device  
_HAS_ changed and all old state should be thrown away (even the  
address).  There is no way around that issue.  USB was designed to  
support hardware suspend; you can put all the hardware in low-power  
mode and still be able to detect changes.

In fact, I would argue that turning off all the busses completely  
when you want to maintain a connection to a device is broken.  If you  
want to maintain the connection, you should keep the busses powered.   
Otherwise, according to the USB spec, it's the _kernel_ that is  
terminating the connection, and assuming that it exists after  
explicitly terminating it is wrong.

Cheers,
Kyle Moffett

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> 
> No, that information is the most reliable that can be obtained.  It 
> tells us that we can no longer make any guarantees about the device or 
> its state.  The USB spec is quite clear on this point.
> 

That is what it says but the kernel is interpreting it as "this device 
HAS been removed" rather than "this device MAY have been removed".  That 
is wrong and should be fixed.

> 
> Except we can't reliably decide that.  Say I plug my USB camera in, 
> mount it, and download some pictures.  I then suspend the computer, 
> unplug the camera after suspending, take more pictures, plug it back in 
> and resume.  That's a fairly reasonable situation and the computer 
> considering the camera's state to be unchanged would be a serious bug 
> and probably result in data loss.  By contrast, just considering the 
> camera to be spontaneously unplugged would cause no more data loss than 
> actually spontaneously unplugging the flash drive.
> 

But again, that would be user error and thus, the data loss can not be 
avoided and is their fault.  Having data loss result from user error is 
far more acceptable than having data loss ALLWAYS result from a 
perfectly acceptable user action, namely hibernating the machine and 
resuming it some time later without altering anything.  You already 
teach users not to unplug the drive without ejecting it from the desktop 
first, why should you also force them to eject before hibernating?

> 
> This is why hardware suspend is a good thing.  When I suspend and resume 
> my laptop, there are _no_ USB disconnects.  The controller puts all the 
> hubs into low-power mode, but it never disconnects them or causes problems.
> 

That's fantastic for your system, but not all systems will maintain 
standby power to USB, and users expect to be able to suspend to disk and 
not loose data, just like they do with non USB drives.

> 
> No, you have this wrong.  The USB spec indicates that the device _HAS_ 
> changed and all old state should be thrown away (even the address).  
> There is no way around that issue.  USB was designed to support hardware 
> suspend; you can put all the hardware in low-power mode and still be 
> able to detect changes.

That's great, except that feature is not always used so you must be able 
to live without it.  The fact that the hardware flag is set is no 
indication that the hardware HAS changed, you said so yourself; all it 
knows is that the bus/device lost power.  The use case we are talking 
about is one in which power loss happens, but the device is still the 
same, and so access to it should not be interrupted.

> 
> In fact, I would argue that turning off all the busses completely when 
> you want to maintain a connection to a device is broken.  If you want to 
> maintain the connection, you should keep the busses powered.  Otherwise, 
> according to the USB spec, it's the _kernel_ that is terminating the 
> connection, and assuming that it exists after explicitly terminating it 
> is wrong.
> 

Yes, assuming that it exists is wrong.  Probing the hardware and seeing 
that it exists and is _probably_ the same device is entirely different. 
  In that case it is preferable to assume the probable case rather than 
the improbable one because it will lead to less data loss.

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 13, 2006, at 21:09, Phillip Susi wrote:
> Kyle Moffett wrote:
>> No, that information is the most reliable that can be obtained.   
>> It tells us that we can no longer make any guarantees about the  
>> device or its state.  The USB spec is quite clear on this point.
>
> That is what it says but the kernel is interpreting it as "this  
> device HAS been removed" rather than "this device MAY have been  
> removed".  That is wrong and should be fixed.

No, that's _exactly_ what the spec says (well, not verbatim but close  
enough).  When you disconnect, both the master and slave devices are  
perfectly free to assume that the connection is completely broken and  
no state is maintained.  Anything that breaks that assumption is  
against the spec and likely to break in odd scenarios.

> [multiple data-loss arguments]

Which causes worse data-loss, writing out cached pages and filesystem  
metadata to a filesystem that has changed in the mean-time (possibly  
allocating those for metadata, etc) or forcibly unmounting it as  
though the user pulled the cable?  Most filesystems are designed to  
handle the latter (it's the same as a hard-shutdown), whereas _none_  
are designed to handle the former.

A good set of suspend scripts should handle the hardware-suspend with  
no extra work because hardware supporting hardware-suspend basically  
inevitably supports USB low-power-mode, and handle software-suspend  
by either forcibly syncing and unmounting USB filesystems or by  
failing the suspend and asking the user to.  You also could patch the  
kernel to fail a powerdown software suspend if some USB device is  
mounted or otherwise unremovably in-use.

>> In fact, I would argue that turning off all the busses completely  
>> when you want to maintain a connection to a device is broken.  If  
>> you want to maintain the connection, you should keep the busses  
>> powered.  Otherwise, according to the USB spec, it's the _kernel_  
>> that is terminating the connection, and assuming that it exists  
>> after explicitly terminating it is wrong.
>
> Yes, assuming that it exists is wrong.  Probing the hardware and  
> seeing that it exists and is _probably_ the same device is entirely  
> different.  In that case it is preferable to assume the probable  
> case rather than the improbable one because it will lead to less  
> data loss.

Except that would make Linux broken with respect to the USB spec.  It  
is fallacious to assume that a USB device that the kernel has told to  
disconnect will still have the same state when the kernel tries to  
reconnect, even _if_ you could reliably identify it (which you can't  
because there is no serial number of any sort on a lot of devices.

Cheers,
Kyle Moffett

--
I lost interest in "blade servers" when I found they didn't throw  
knives at people who weren't supposed to be in your machine room.
   -- Anthony de Boer

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Mon, 13 Feb 2006, Kyle Moffett wrote:

> Which causes worse data-loss, writing out cached pages and filesystem  
> metadata to a filesystem that has changed in the mean-time (possibly  
> allocating those for metadata, etc) or forcibly unmounting it as  
> though the user pulled the cable?  Most filesystems are designed to  
> handle the latter (it's the same as a hard-shutdown), whereas _none_  
> are designed to handle the former.

That's a good point.  Furthermore, any decent suspend script will flush
all dirty buffers to disk before suspending anything.

> A good set of suspend scripts should handle the hardware-suspend with  
> no extra work because hardware supporting hardware-suspend basically  
> inevitably supports USB low-power-mode,

Unfortunately a lot of hardware doesn't support USB low-power mode.  I 
guess you'd say therefore it doesn't really support hardware-suspend.  
This may be so, but it's small comfort to the owners of those systems.

I have to admit, although technically Phillip's argument is wrong, from a
useability standpoint it is right.  Windows allows users to disconnect and
reconnect USB storage devices while the system is hibernating, with no
apparent ill effects -- although I've never tried to unplug one device and
then plug in a different one on the same port while the computer was
asleep.  I don't know to what extent Windows checks descriptors/serial 
numbers/disk labels/whatever when it wakes up.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 13, 2006, at 23:28, Alan Stern wrote:
> On Mon, 13 Feb 2006, Kyle Moffett wrote:
>> A good set of suspend scripts should handle the hardware-suspend  
>> with no extra work because hardware supporting hardware-suspend  
>> basically inevitably supports USB low-power-mode,
>
> Unfortunately a lot of hardware doesn't support USB low-power  
> mode.  I guess you'd say therefore it doesn't really support  
> hardware-suspend.  This may be so, but it's small comfort to the  
> owners of those systems.
>
> I have to admit, although technically Phillip's argument is wrong,  
> from a usability standpoint it is right.  Windows allows users to  
> disconnect and reconnect USB storage devices while the system is  
> hibernating, with no  apparent ill effects -- although I've never  
> tried to unplug one device and then plug in a different one on the  
> same port while the computer was asleep.  I don't know to what  
> extent Windows checks descriptors/serial  numbers/disk labels/ 
> whatever when it wakes up.

For the software-suspend/no-low-power-mode case, I see a couple of  
practical and spec-conforming options:

1)  The kernel should notice that it has a filesystem mounted from a  
hotpluggable block device and abort the suspend process.  This isn't  
terribly user friendly, but is guaranteed to prevent data loss, and a  
good set of suspend scripts could notice the reason for failure and  
report it to the user (optionally unmounting the filesystems  
automatically and retrying).

2)  The kernel should notice that it has a filesystem mounted from a  
hotpluggable block device and forcibly unmount said filesystem.  This  
is also not user-friendly, and has the disadvantage of not being  
easily userspace-controllable.

3)  The kernel should notice that it has a filesystem mounted from a  
hotpluggable block device and forcibly disconnect the mount.   
Beforehand, uswsusp would have saved information about all mounted  
blockdevs into the suspend file/disk.  When resuming, early userspace  
would reread that information and attempt to relocate the block  
devices from userspace, using any tools available to it at the time  
(including a bunch of fs-probing tools and such).  After it's scanned  
devices and found any that it could reliably get, it would pass that  
information to the kernel being resumed which would use it to  
reattach filesystems to disks.  This is a lot more complicated, but  
more user friendly.  It has the downside of making the kernel do a  
lot of extra unreliable work looking up paths and files again, but it  
might work in a good percentage of the cases.  I doubt the advantages  
of this one over (1) or (2) are worth the added complexity, though.

Cheers,
Kyle Moffett

--
Simple things should be simple and complex things should be possible
   -- Alan Kay

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> No, that's _exactly_ what the spec says (well, not verbatim but close 
> enough).  When you disconnect, both the master and slave devices are 
> perfectly free to assume that the connection is completely broken and no 
> state is maintained.  Anything that breaks that assumption is against 
> the spec and likely to break in odd scenarios.
> 

Perfectly free to != required to.  When you know that the connection is 
"broken" due to suspend and there is no way to differentiate between 
that and the device really being disconnected, you have two choices:

1) Assume the user is a masochistic idiot and replaced the device with 
one that looks just like it, or connected the device somewhere else and 
modified it before replacing it, or

2) Assume the user is sane and did no such thing.

Assumption #2 will be correct most of the time and is a valid use case, 
the first is not. If either assumption is wrong, it will lead to data 
loss.  All else being equal, which case do you optimize for?  The common 
and correct case, or the uncommon error case?

> 
> Which causes worse data-loss, writing out cached pages and filesystem 
> metadata to a filesystem that has changed in the mean-time (possibly 
> allocating those for metadata, etc) or forcibly unmounting it as though 
> the user pulled the cable?  Most filesystems are designed to handle the 
> latter (it's the same as a hard-shutdown), whereas _none_ are designed 
> to handle the former.
> 

So you condemn the common correct use case to always suffer data loss to 
give _slightly_ better protection to the uncommon and incorrect use 
case?  I think most users prefer a system that works right when you use 
it right to one that doesn't break quite as badly when you do something 
stupid.

Also why is this argument more valid for USB than SCSI?  I am just as 
free to unplug a scsi disk and replace it with a different one while 
hibernated, yet I don't suffer data loss when I don't do such 
foolishness.  By your argument, because the kernel can not know for sure 
that I didn't do that, it must disconnect and break all the drives on 
the bus.

> A good set of suspend scripts should handle the hardware-suspend with no 
> extra work because hardware supporting hardware-suspend basically 
> inevitably supports USB low-power-mode, and handle software-suspend by 
> either forcibly syncing and unmounting USB filesystems or by failing the 
> suspend and asking the user to.  You also could patch the kernel to fail 
> a powerdown software suspend if some USB device is mounted or otherwise 
> unremovably in-use.
> 

That would prevent data loss, so it would be better than the current 
method of silently loosing data because you assume the user is out to 
get you.  There are other downsides to it though.  What if someone wants 
to boot from a usb drive and suspend to disk?

> 
> Except that would make Linux broken with respect to the USB spec.  It is 

Please point out where in the USB spec it states that the OS _MUST_ 
assume that the device is no longer the same after a bus shutdown and 
restart, even though it otherwise appears to be.

> fallacious to assume that a USB device that the kernel has told to 
> disconnect will still have the same state when the kernel tries to 
> reconnect, even _if_ you could reliably identify it (which you can't 
> because there is no serial number of any sort on a lot of devices.
> 

It is worse to assume that it HAS changed, when that is guaranteed to 
lead to data loss even though it most likely has not changed.  This is 
no different than assuming that the user knows what they are doing when 
they fdisk a drive.  You don't refuse to fdisk a drive just because you 
can't be 100% certain that they aren't doing something stupid that _may_ 
result in data loss, so you assume they know what they are doing and do 
as they say.

Re: Flames over -- Re: Which is simpler?

[hackmiester / Hunter Fuller]

[-- Attachment #1: Type: text/plain, Size: 3657 bytes --]

On Sunday 12 February 2006 20:19, Alan Stern wrote:
> On Sun, 12 Feb 2006, Phillip Susi wrote:
> > Alan Stern wrote:
> > > Both of you are missing an important difference between Suspend-to-RAM
> > > and Suspend-to-Disk.
> > >
> > > Suspend-to-RAM is a true suspend operation, in that the hardware's
> > > state is maintained _in the hardware_.  External buses like USB will
> > > retain suspend power, for instance (assuming the motherboard supports
> > > it; some don't).
> > >
> > > Suspend-to-Disk, by contrast, is _not_ a true suspend.  It can more
> > > accurately be described as checkpoint-and-turn-off.  Hardware state is
> > > not maintained.  (Some systems may support a special ACPI state that
> > > does maintain suspend power to external buses during shutdown, I forget
> > > what it's called.  And I down't know whether swsusp uses this state.)
> >
> > I would disagree.  The only difference between the two is WHERE the
> > state is maintained - ram vs. disk.  I won't really argue it though,
> > because it's just semantics -- call it whatever you want.
>
> It's not just semantics.  There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else.  The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections.  When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep.  If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.
>
> Another difference is the possibility of remote wakeup.  Clearly it can't
> happen when there's no power available.

Well... sort of. I know that my notebook can be brought up over LAN when it is 
off - suspended to disk or otherwise.
>
> > > So for example, let's say you have a filesystem mounted on a USB flash
> > > or disk drive.  With Suspend-to-RAM, there's a very good chance that
> > > the connection and filesystem will still be intact when you resume. 
> > > With Suspend-to-Disk, the USB connection will terminate when the
> > > computer shuts down.  When you resume, the device will be gone and your
> > > filesystem will be screwed.
> >
> > This is not true.  The USB bus is shut down either way, and provided
> > that you have not unplugged the disk, nothing will be screwed when you
> > resume from disk or ram.
>
> Have you actually tried it?  I have.
>
> In any case, it is undeniably true that if the bus is shut down then all
> the USB connections are lost.  When you resume it will be the same as if
> you had unplugged all the USB devices and then replugged them.  Not a good
> thing to do when they contain mounted filesystems; all the memory mappings
> are invalidated.

It all depends on the machine I guess. Mine keeps even the CD drive running 
when it's off!
>
> (Bear in mind that whether a USB bus gets shut down depends on the
> motherboard; some supply suspend power and some don't.  It depends on the
> USB controller too; some support low-power states other than "completely
> off" and others don't.)
>
> Alan Stern
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
--hackmiester
If you can read this, you don't need glasses.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Tue, 14 Feb 2006, Kyle Moffett wrote:

> For the software-suspend/no-low-power-mode case, I see a couple of  
> practical and spec-conforming options:
> 
> 1)  The kernel should notice that it has a filesystem mounted from a  
> hotpluggable block device and abort the suspend process.  This isn't  
> terribly user friendly, but is guaranteed to prevent data loss, and a  
> good set of suspend scripts could notice the reason for failure and  
> report it to the user (optionally unmounting the filesystems  
> automatically and retrying).
...

There are some difficulties connected with these suggestions.  They're
maybe not impossible, but it would be tricky.

For one thing, how does one go about telling at the USB level whether one
of the devices contains a mounted filesystem?  Or any open file references
at all, for that matter?  No doubt there's a way to do it, but I don't
know what it is.

For another, hardware behavior is very idiosyncratic.  It's sometimes 
possible to tell that a particular system doesn't supply suspend power to 
a USB controller (because, for instance, the controller doesn't support 
PCI PM or doesn't support D3hot), but this is far from reliable -- I've 
seen mistakes both ways.  It's not as simple as "power available during 
STR, not available during STD".

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 14, 2006, at 01:27, Phillip Susi wrote:
> Kyle Moffett wrote:
>> No, that's _exactly_ what the spec says (well, not verbatim but  
>> close enough).  When you disconnect, both the master and slave  
>> devices are perfectly free to assume that the connection is  
>> completely broken and no state is maintained.  Anything that  
>> breaks that assumption is against the spec and likely to break in  
>> odd scenarios.
>
> Perfectly free to != required to.

In this case they _are_ equivalent due to symmetry.  If the other  
device _may_ assume that the connection is broken, then you _must_  
assume that the connection is broken.  Since either device _may_  
assume that, both devices therefore _must_ according to spec.

>> Which causes worse data-loss, writing out cached pages and  
>> filesystem metadata to a filesystem that has changed in the mean- 
>> time (possibly allocating those for metadata, etc) or forcibly  
>> unmounting it as though the user pulled the cable?  Most  
>> filesystems are designed to handle the latter (it's the same as a  
>> hard-shutdown), whereas _none_ are designed to handle the former.
>
> So you condemn the common correct use case to always suffer data  
> loss to give _slightly_ better protection to the uncommon and  
> incorrect use case?

No, as I said before, a good set of USB suspend scripts can solve  
this problem.  All of the ones I am aware of *now* already sync all  
data, which is good enough to prevent data-loss in _every_ case where  
the device is spontaneously unplugged.  On the other hand, this is  
_never_ good enough if the device is accidentally switched underneath  
us while suspended (and that's not so terribly uncommon, I know a lot  
of people who would do that accidentally, myself included).

> I think most users prefer a system that works right when you use it  
> right to one that doesn't break quite as badly when you do  
> something stupid.

I think you just proved my point.  Running the "sync" command a  
couple times then unplugging the USB stick basically never results in  
data loss even if the filesystem is mounted.  Spontaneously switching  
block devices under a mounted filesystem is guaranteed to either  
panic the machine or cause massive data corruption or both.

> Also why is this argument more valid for USB than SCSI?  I am just  
> as free to unplug a scsi disk and replace it with a different one  
> while hibernated, yet I don't suffer data loss when I don't do such  
> foolishness.

SCSI != USB.  Users generally don't expect to hotplug SCSI devices  
while booted and running (with the exception of some _really_  
expensive hotplug-bays where we expect the admin to know what the  
hell they're doing).  On the other hand, users _do_ expect to hotplug  
random USB devices whenever they feel like it.

Cheers,
Kyle Moffett

--
Q: Why do programmers confuse Halloween and Christmas?
A: Because OCT 31 == DEC 25.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> In this case they _are_ equivalent due to symmetry.  If the other 
> device _may_ assume that the connection is broken, then you _must_ 
> assume that the connection is broken.  Since either device _may_ 
> assume that, both devices therefore _must_ according to spec.
>
Only insofar as the kernel can not assume the device is still 
connected.  If the kernel sees that it is immediately reconnected and 
has every reason to believe that the 'disconnect' was only a result of 
momentary power loss, and it can probably continue to access the device 
with no consequences, then it should do so. 
>
> No, as I said before, a good set of USB suspend scripts can solve this 
> problem.  All of the ones I am aware of *now* already sync all data, 
> which is good enough to prevent data-loss in _every_ case where the 
> device is spontaneously unplugged.  On the other hand, this is _never_ 
> good enough if the device is accidentally switched underneath us while 
> suspended (and that's not so terribly uncommon, I know a lot of people 
> who would do that accidentally, myself included).
>

I suppose this is true.  Syncing before suspend will (mostly) keep the 
kernel from shooting the user in the foot. 

>> I think most users prefer a system that works right when you use it 
>> right to one that doesn't break quite as badly when you do something 
>> stupid.
>
> I think you just proved my point.  Running the "sync" command a couple 
> times then unplugging the USB stick basically never results in data 
> loss even if the filesystem is mounted.  Spontaneously switching block 
> devices under a mounted filesystem is guaranteed to either panic the 
> machine or cause massive data corruption or both.
>

But who cares?  There are plenty of really stupid things users can do to 
hose their system, it isn't right to prevent them from doing something 
perfectly reasonable just because it reduces the damage done when they 
do something completely unreasonable. 

>> Also why is this argument more valid for USB than SCSI?  I am just as 
>> free to unplug a scsi disk and replace it with a different one while 
>> hibernated, yet I don't suffer data loss when I don't do such 
>> foolishness.
>
> SCSI != USB.  Users generally don't expect to hotplug SCSI devices 
> while booted and running (with the exception of some _really_ 
> expensive hotplug-bays where we expect the admin to know what the hell 
> they're doing).  On the other hand, users _do_ expect to hotplug 
> random USB devices whenever they feel like it.
>

So because SCSI is more expensive than USB, it is ok to assume it will 
only be used by people who know what they are doing?  And since USB will 
be used by people who don't know what they are doing, we must assume 
they will always do silly things ( swap or modify the drive while 
suspended ), at the expense of those who don't?

Re: Flames over -- Re: Which is simpler?

[Olivier Galibert]

On Tue, Feb 14, 2006 at 11:23:53AM -0500, Kyle Moffett wrote:
> SCSI != USB.  Users generally don't expect to hotplug SCSI devices  
> while booted and running (with the exception of some _really_  
> expensive hotplug-bays where we expect the admin to know what the  
> hell they're doing).  On the other hand, users _do_ expect to hotplug  
> random USB devices whenever they feel like it.

I do expect to be able to move non-mounted disks around while
suspended to disk, whatever their kind is (ide, sata, scsi, whatever).
That's one of the main reasons you want a reliable suspend-to-disk on
servers, another one being riding predicted powerloss (moving boxes
around can be called a powerloss).

  OG.

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:

> Hi,
> 
> On Monday 13 February 2006 22:24, Alan Stern wrote:
> > On Mon, 13 Feb 2006, Phillip Susi wrote:
> }-- snip --{
> > You are complaining because you don't like the way USB was designed.  
> > That's fine, but it leaves you advocating a non-standardized position.
> > 
> > Can you suggest a _reliable_ way to tell if the USB device present at a 
> > port after resuming is the same device as was there before suspending?
> 
> It seems to follow from your discussion that if I have a mounted filesystem
> on a USB device and I suspend to disk, I can lose data unless the filesystem
> has been mounted with "sync".

That's right.  It depends on your hardware, and it could be true even for
suspend-to-RAM.  In fact, even with "-o sync" you can lose data if your
programs have information in buffers they haven't written out to disk.

If you're lucky, your hardware will support low-power modes for USB
controllers while the system is asleep.  Lots of hardware doesn't,
however.  Shutting off the power to a USB controller is equivalent to 
unplugging all the attached devices.

Remember that it's always a bad idea to unplug a disk drive containing a
mounted filesystem.  With USB that's true even when your system is asleep!
The safest thing is to unmount all USB-based filesystems before suspending 
and remount them after resuming.

> If this is the case, there should be a big fat warning in the swsusp
> documentation, but there's nothing like that in there (at lease I can't find
> it easily).
> 
> [If this is not the case, I've missed something and sorry for the noise.]

I'm not aware of any warnings about this in the documentation.  If you 
would like to add something, please go ahead.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1765 bytes --]

On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:

> Hi,
> 
> On Monday 13 February 2006 22:24, Alan Stern wrote:
> > On Mon, 13 Feb 2006, Phillip Susi wrote:
> }-- snip --{
> > You are complaining because you don't like the way USB was designed.  
> > That's fine, but it leaves you advocating a non-standardized position.
> > 
> > Can you suggest a _reliable_ way to tell if the USB device present at a 
> > port after resuming is the same device as was there before suspending?
> 
> It seems to follow from your discussion that if I have a mounted filesystem
> on a USB device and I suspend to disk, I can lose data unless the filesystem
> has been mounted with "sync".

That's right.  It depends on your hardware, and it could be true even for
suspend-to-RAM.  In fact, even with "-o sync" you can lose data if your
programs have information in buffers they haven't written out to disk.

If you're lucky, your hardware will support low-power modes for USB
controllers while the system is asleep.  Lots of hardware doesn't,
however.  Shutting off the power to a USB controller is equivalent to 
unplugging all the attached devices.

Remember that it's always a bad idea to unplug a disk drive containing a
mounted filesystem.  With USB that's true even when your system is asleep!
The safest thing is to unmount all USB-based filesystems before suspending 
and remount them after resuming.

> If this is the case, there should be a big fat warning in the swsusp
> documentation, but there's nothing like that in there (at lease I can't find
> it easily).
> 
> [If this is not the case, I've missed something and sorry for the noise.]

I'm not aware of any warnings about this in the documentation.  If you 
would like to add something, please go ahead.

Alan Stern


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Olivier Galibert wrote:
>
> I do expect to be able to move non-mounted disks around while
> suspended to disk, whatever their kind is (ide, sata, scsi, whatever).
> That's one of the main reasons you want a reliable suspend-to-disk on
> servers, another one being riding predicted powerloss (moving boxes
> around can be called a powerloss).

You also expect to have mounted disks survive the suspend.  Think about 
the root filesystem.  You can't unmount that and it would be really bad 
if that went offline ( kernel panic anyone? ) after a suspend.  And yes, 
the root fs can be on a USB drive. 

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 14, 2006, at 13:39, Phillip Susi wrote:
>>> I think most users prefer a system that works right when you use  
>>> it right to one that doesn't break quite as badly when you do  
>>> something stupid.
>>
>> I think you just proved my point.  Running the "sync" command a  
>> couple times then unplugging the USB stick basically never results  
>> in data loss even if the filesystem is mounted.  Spontaneously  
>> switching block devices under a mounted filesystem is guaranteed  
>> to either panic the machine or cause massive data corruption or both.
>
> But who cares?  There are plenty of really stupid things users can  
> do to hose their system, it isn't right to prevent them from doing  
> something perfectly reasonable just because it reduces the damage  
> done when they do something completely unreasonable.

How is swapping USB devices while suspended unreasonable?  Come to  
think of it, I did it inadvertently about 15 minutes ago with my  
PowerBook (while booted into Mac OS).  I had a USB key that I was  
copying a file to for someone.  After shutting the lid, I unplugged  
mouse, USB key, and power block.  About 30 minutes later I had  
somebody else with a key who wanted to give me a file.  I had the key  
plugged in before the laptop was finished waking up from sleep.  Now,  
I don't know for certain that neither key had a serial number, but  
the two I have here in my hand certainly don't, and I could _easily_  
see somebody swapping USB keys not knowing that they're "not supposed  
to do that" and getting massive data corruption when the filesystem  
reads and writes pages from a completely different block device.

>> SCSI != USB.  Users generally don't expect to hotplug SCSI devices  
>> while booted and running (with the exception of some _really_  
>> expensive hotplug-bays where we expect the admin to know what the  
>> hell they're doing).  On the other hand, users _do_ expect to  
>> hotplug random USB devices whenever they feel like it.
>
> So because SCSI is more expensive than USB, it is ok to assume it  
> will only be used by people who know what they are doing?  And  
> since USB will be used by people who don't know what they are  
> doing, we must assume they will always do silly things ( swap or  
> modify the drive while suspended ), at the expense of those who don't?

Did you read what I wrote?  People don't generally expect to randomly  
plug and unplug SCSI drives whenever they feel like it.  They _do_  
expect to randomly plug and unplug USB drives, mice, keyboards,  
tablets, network adapters, etc, because _everything_ supports such  
random plugging.

Creating an extremely odd and hard to predict failure mode (when you  
reconnect USB devices while suspended on hardware that doesn't  
support proper USB suspend) with a high probability of causing data  
corruption or crashes is wrong.  Especially since you could easily  
teach users that "You need to eject USB things before you sleep the  
computer _or_ just fix the kernel to do it for you.  That's probably  
something we should be doing for all network filesystems anyways.

Cheers,
Kyle Moffett

--
I didn't say it would work as a defense, just that they can spin that  
out for years in court if it came to it.
   -- Rob Landley

Re: Flames over -- Re: Which is simpler?

[Rafael J. Wysocki]

Hi,

On Tuesday 14 February 2006 20:26, Alan Stern wrote:
> On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:
> > On Monday 13 February 2006 22:24, Alan Stern wrote:
> > > On Mon, 13 Feb 2006, Phillip Susi wrote:
> > }-- snip --{
> > > You are complaining because you don't like the way USB was designed.  
> > > That's fine, but it leaves you advocating a non-standardized position.
> > > 
> > > Can you suggest a _reliable_ way to tell if the USB device present at a 
> > > port after resuming is the same device as was there before suspending?
> > 
> > It seems to follow from your discussion that if I have a mounted filesystem
> > on a USB device and I suspend to disk, I can lose data unless the filesystem
> > has been mounted with "sync".
> 
> That's right.  It depends on your hardware, and it could be true even for
> suspend-to-RAM.  In fact, even with "-o sync" you can lose data if your
> programs have information in buffers they haven't written out to disk.
> 
> If you're lucky, your hardware will support low-power modes for USB
> controllers while the system is asleep.  Lots of hardware doesn't,
> however.  Shutting off the power to a USB controller is equivalent to 
> unplugging all the attached devices.
> 
> Remember that it's always a bad idea to unplug a disk drive containing a
> mounted filesystem.  With USB that's true even when your system is asleep!
> The safest thing is to unmount all USB-based filesystems before suspending 
> and remount them after resuming.

Still, this may be impossible if the box is suspending because of its
battery running critical.

I'm afraid this behavior will cause support problems to appear in the long
run.   [BTW, I wonder if it's USB-only, or some other subsystems behave
in a similar way, like ieee1394 or external SATA.  And how about
NFS/CIFS/whatever network filesystems mounted on the suspending box?]

I think one solution to consider could be to add an abstract fs layer
on top of the removable filesystem, like subfs in SuSE, with the ability
to retain the user information accross device disconnects and to update
the fs state from the actual device when it reappears in the system and
to resolve possible conflicts (to a reasonable extent).

> > If this is the case, there should be a big fat warning in the swsusp
> > documentation, but there's nothing like that in there (at lease I can't find
> > it easily).
> > 
> > [If this is not the case, I've missed something and sorry for the noise.]
> 
> I'm not aware of any warnings about this in the documentation.  If you 
> would like to add something, please go ahead.

I'm going to do this.  Can I use your explanation above?

Rafael

Re: Flames over -- Re: Which is simpler?

[Rafael J. Wysocki]

Hi,

On Tuesday 14 February 2006 20:26, Alan Stern wrote:
> On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:
> > On Monday 13 February 2006 22:24, Alan Stern wrote:
> > > On Mon, 13 Feb 2006, Phillip Susi wrote:
> > }-- snip --{
> > > You are complaining because you don't like the way USB was designed.  
> > > That's fine, but it leaves you advocating a non-standardized position.
> > > 
> > > Can you suggest a _reliable_ way to tell if the USB device present at a 
> > > port after resuming is the same device as was there before suspending?
> > 
> > It seems to follow from your discussion that if I have a mounted filesystem
> > on a USB device and I suspend to disk, I can lose data unless the filesystem
> > has been mounted with "sync".
> 
> That's right.  It depends on your hardware, and it could be true even for
> suspend-to-RAM.  In fact, even with "-o sync" you can lose data if your
> programs have information in buffers they haven't written out to disk.
> 
> If you're lucky, your hardware will support low-power modes for USB
> controllers while the system is asleep.  Lots of hardware doesn't,
> however.  Shutting off the power to a USB controller is equivalent to 
> unplugging all the attached devices.
> 
> Remember that it's always a bad idea to unplug a disk drive containing a
> mounted filesystem.  With USB that's true even when your system is asleep!
> The safest thing is to unmount all USB-based filesystems before suspending 
> and remount them after resuming.

Still, this may be impossible if the box is suspending because of its
battery running critical.

I'm afraid this behavior will cause support problems to appear in the long
run.   [BTW, I wonder if it's USB-only, or some other subsystems behave
in a similar way, like ieee1394 or external SATA.  And how about
NFS/CIFS/whatever network filesystems mounted on the suspending box?]

I think one solution to consider could be to add an abstract fs layer
on top of the removable filesystem, like subfs in SuSE, with the ability
to retain the user information accross device disconnects and to update
the fs state from the actual device when it reappears in the system and
to resolve possible conflicts (to a reasonable extent).

> > If this is the case, there should be a big fat warning in the swsusp
> > documentation, but there's nothing like that in there (at lease I can't find
> > it easily).
> > 
> > [If this is not the case, I've missed something and sorry for the noise.]
> 
> I'm not aware of any warnings about this in the documentation.  If you 
> would like to add something, please go ahead.

I'm going to do this.  Can I use your explanation above?

Rafael

Re: Flames over -- Re: Which is simpler?

[Lee Revell]

On Tue, 2006-02-14 at 21:41 +0100, Rafael J. Wysocki wrote:
> [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA.  And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]
> 

NFS is stateless, it does not care about stuff like this.  There's no
concept of an open file.  write() does not return until the data is
committed to stable storage.

Lee

Re: Flames over -- Re: Which is simpler?

[Lee Revell]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On Tue, 2006-02-14 at 21:41 +0100, Rafael J. Wysocki wrote:
> [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA.  And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]
> 

NFS is stateless, it does not care about stuff like this.  There's no
concept of an open file.  write() does not return until the data is
committed to stable storage.

Lee


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> How is swapping USB devices while suspended unreasonable?  Come to 
> think of it, I did it inadvertently about 15 minutes ago with my PowerBook

Because you can not go yanking devices out from under the kernel without 
it's knowledge or consent.  This is no more acceptable than ejecting a 
floppy without first unmounting it; the only difference is that the 
floppy drive doesn't erroneously inform the kernel that you have done 
this simply because you suspend. 

> (while booted into Mac OS).  I had a USB key that I was copying a file 
> to for someone.  After shutting the lid, I unplugged mouse, USB key, 
> and power block.  About 30 minutes later I had somebody else with a 
> key who wanted to give me a file.  I had the key plugged in before the 
> laptop was finished waking up from sleep.  Now, I don't know for 
> certain that neither key had a serial number, but the two I have here 
> in my hand certainly don't, and I could _easily_ see somebody swapping 
> USB keys not knowing that they're "not supposed to do that" and 
> getting massive data corruption when the filesystem reads and writes 
> pages from a completely different block device.
>

Then they shot themselves in the foot.  That is no different than 
switching mounted floppies while suspended, or removing a mounted IDE 
hard drive while suspended, so they get what they deserve. 

> Did you read what I wrote?  People don't generally expect to randomly 
> plug and unplug SCSI drives whenever they feel like it.  They _do_ 
> expect to randomly plug and unplug USB drives, mice, keyboards, 
> tablets, network adapters, etc, because _everything_ supports such 
> random plugging.
>

No, they don't.  Users do not expect ( or should not and are told not to 
by admins ) to be able to yank out their usb memory stick while it is 
mounted.  They are told to always unmount first.  If they fail to do so, 
then they get what they asked for.  It doesn't matter if the disk is 
SCSI or USB; you don't go yanking it out without unmounting it first, or 
you will loose data. 

> Creating an extremely odd and hard to predict failure mode (when you 
> reconnect USB devices while suspended on hardware that doesn't support 
> proper USB suspend) with a high probability of causing data corruption 
> or crashes is wrong.  Especially since you could easily teach users 
> that "You need to eject USB things before you sleep the computer _or_ 
> just fix the kernel to do it for you.  That's probably something we 
> should be doing for all network filesystems anyways.
>

Users are already told to eject/unmount the media before removing it.  
If they fail to do that, it doesn't matter if the system is suspended or 
not; they broke the drive when they yanked it out while mounted.  As for 
fixing the kernel to unmount it for you, that is not always possible; 
take the root on usb case. 

Maybe it does make sense to have hibernation scripts unmount removable 
media for the common silly user who can't remember to unmount disks 
before ejecting/unplugging them, but you should be able to suspend a 
system with its root on a usb disk and not have the kernel panic for no 
good reason when it resumes. 

Re: Flames over -- Re: Which is simpler?

[Kyle Moffett]

On Feb 14, 2006, at 16:13, Phillip Susi wrote:
> Because you can not go yanking devices out from under the kernel  
> without it's knowledge or consent.  This is no more acceptable than  
> ejecting a floppy without first unmounting it; the only difference  
> is that the floppy drive doesn't erroneously inform the kernel that  
> you have done this simply because you suspend.

Yes, but causing it to overwrite data over random blockdevs when the  
unexpected happens is _not_ OK.

>> Did you read what I wrote?  People don't generally expect to  
>> randomly plug and unplug SCSI drives whenever they feel like it.   
>> They _do_ expect to randomly plug and unplug USB drives, mice,  
>> keyboards, tablets, network adapters, etc, because _everything_  
>> supports such random plugging.
>
> No, they don't.  Users do not expect ( or should not and are told  
> not to by admins ) to be able to yank out their usb memory stick  
> while it is mounted.  They are told to always unmount first.  If  
> they fail to do so, then they get what they asked for.  It doesn't  
> matter if the disk is SCSI or USB; you don't go yanking it out  
> without unmounting it first, or you will loose data.

You've obviously never administrated any kind of large scale linux  
lab.  We had so many people just saving files on their USB sticks and  
pulling them that we would daily get people reporting that they  
couldn't _mount_ their USB stick because the last user hadn't  
unmounted it first.  At one point we had to write a Perl script run  
from cron that ran every 10 minutes and verified if USB-stick mounts  
were still good, and if not it forcefully unmounted them.  Now  
obviously this is the kind of behavior that we want to avoid, but end- 
users are bound to do stupid stuff until it comes back and bites them  
at least twice (except on Linux with -o sync mounts it usually  
doesn't).  Think about how most linux distros automatically turn on - 
o sync on USB keys and the like; they do it for precisely this reason.

> Maybe it does make sense to have hibernation scripts unmount  
> removable media for the common silly user who can't remember to  
> unmount disks before ejecting/unplugging them, but you should be  
> able to suspend a system with its root on a usb disk and not have  
> the kernel panic for no good reason when it resumes.

I described a workable method to handle root-on-USB (and I'm not  
sure, but I doubt it works now).  You would have early userspace find  
your USB filesystems again and pass device information to the  
resuming kernel, which would then restore RAM and then use the passed  
information to attach its filesystems again.

One other alternative that just occurs to me now is to have a special  
stackable filesystem that can suspend all IO and unmount the  
filesystem underneath it then have the filesystem be easily  
reattachable later on (the stackable layer would look up paths again  
on remount.  You would have an mlocked program start during suspend  
that would create a new namespace with a basic tmpfs, procfs, and  
sysfs.  It would be able to rescan all removable devices on resume  
and reattach them to the stackable mounts in the primary namespace  
through /proc/<pid>/root.  This seems to me to be the most race-free  
and most flexible solution.  It will even handle network suspend and  
resume without too much extra effort.

Cheers,
Kyle Moffett

--
Simple things should be simple and complex things should be possible
   -- Alan Kay

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Kyle Moffett wrote:
> On Feb 14, 2006, at 16:13, Phillip Susi wrote:
>> Because you can not go yanking devices out from under the kernel 
>> without it's knowledge or consent.  This is no more acceptable than 
>> ejecting a floppy without first unmounting it; the only difference is 
>> that the floppy drive doesn't erroneously inform the kernel that you 
>> have done this simply because you suspend.
> 
> Yes, but causing it to overwrite data over random blockdevs when the 
> unexpected happens is _not_ OK.
> 

Why not?  That is exactly what happens if you suspend and swap floppies. 
  Just don't do that and you're fine.  It is nice if you can handle the 
swapped case more gracefully, but not at the cost of data loss in the 
_normal_ case.

> You've obviously never administrated any kind of large scale linux lab.  
> We had so many people just saving files on their USB sticks and pulling 
> them that we would daily get people reporting that they couldn't _mount_ 
> their USB stick because the last user hadn't unmounted it first.  At one 
> point we had to write a Perl script run from cron that ran every 10 
> minutes and verified if USB-stick mounts were still good, and if not it 

This is a very good argument for _correctly_ detecting device removal. 
I am not arguing against that.  The problem is with _incorrectly_ 
detecting device removal.

> forcefully unmounted them.  Now obviously this is the kind of behavior 
> that we want to avoid, but end-users are bound to do stupid stuff until 
> it comes back and bites them at least twice (except on Linux with -o 
> sync mounts it usually doesn't).  Think about how most linux distros 
> automatically turn on -o sync on USB keys and the like; they do it for 
> precisely this reason.
> 

Some distros do.  Some do not.  I side with those who do not because 
doing so promotes stupid users ( they won't learn until it bites them 
twice ) and because it burns out the flash memory 10 times faster and 
slows down access.

> 
> I described a workable method to handle root-on-USB (and I'm not sure, 
> but I doubt it works now).  You would have early userspace find your USB 
> filesystems again and pass device information to the resuming kernel, 
> which would then restore RAM and then use the passed information to 
> attach its filesystems again.
> 

There is currently no such method of reconnecting a filesystem to a 
device once the device has been broken by an eject, is there?  Also that 
amounts to much the same thing as I have been saying all along: the 
system should check to see if the device that is there now is the same 
as the one that was there before, and if so, resume using it.

> One other alternative that just occurs to me now is to have a special 
> stackable filesystem that can suspend all IO and unmount the filesystem 
> underneath it then have the filesystem be easily reattachable later on 
> (the stackable layer would look up paths again on remount.  You would 
> have an mlocked program start during suspend that would create a new 
> namespace with a basic tmpfs, procfs, and sysfs.  It would be able to 
> rescan all removable devices on resume and reattach them to the 
> stackable mounts in the primary namespace through /proc/<pid>/root.  
> This seems to me to be the most race-free and most flexible solution.  
> It will even handle network suspend and resume without too much extra 
> effort.
> 

That seems rather complex with little benefit.  The only advantage that 
would have over not breaking the mount in the first place is in the 
pedantic case where the user screws with the media during suspend. 
Seems like a high price to pay to cleanly deal with a pedantic error 
condition.

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Tue, 14 Feb 2006, Rafael J. Wysocki wrote:

> > Remember that it's always a bad idea to unplug a disk drive containing a
> > mounted filesystem.  With USB that's true even when your system is asleep!
> > The safest thing is to unmount all USB-based filesystems before suspending 
> > and remount them after resuming.
> 
> Still, this may be impossible if the box is suspending because of its
> battery running critical.

Yes.  When there's nothing you can do... do nothing.  :-)

> I'm afraid this behavior will cause support problems to appear in the long
> run.   [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA.  And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]

Some networked filesystems may have this problem, for instance, if they 
rely on some sort of keep-alive to maintain a connection.  I don't know 
anything about 1394 or SATA.

> I think one solution to consider could be to add an abstract fs layer
> on top of the removable filesystem, like subfs in SuSE, with the ability
> to retain the user information accross device disconnects and to update
> the fs state from the actual device when it reappears in the system and
> to resolve possible conflicts (to a reasonable extent).

Maybe.  To me this seems an unavoidable policy decision that should be 
settled by one of the big Linux panjandrums.  Maybe someone can convince 
Andrew (or even Linus) to offer a ruling...

> > I'm not aware of any warnings about this in the documentation.  If you 
> > would like to add something, please go ahead.
> 
> I'm going to do this.  Can I use your explanation above?

Feel free to use it.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1718 bytes --]

On Tue, 14 Feb 2006, Rafael J. Wysocki wrote:

> > Remember that it's always a bad idea to unplug a disk drive containing a
> > mounted filesystem.  With USB that's true even when your system is asleep!
> > The safest thing is to unmount all USB-based filesystems before suspending 
> > and remount them after resuming.
> 
> Still, this may be impossible if the box is suspending because of its
> battery running critical.

Yes.  When there's nothing you can do... do nothing.  :-)

> I'm afraid this behavior will cause support problems to appear in the long
> run.   [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA.  And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]

Some networked filesystems may have this problem, for instance, if they 
rely on some sort of keep-alive to maintain a connection.  I don't know 
anything about 1394 or SATA.

> I think one solution to consider could be to add an abstract fs layer
> on top of the removable filesystem, like subfs in SuSE, with the ability
> to retain the user information accross device disconnects and to update
> the fs state from the actual device when it reappears in the system and
> to resolve possible conflicts (to a reasonable extent).

Maybe.  To me this seems an unavoidable policy decision that should be 
settled by one of the big Linux panjandrums.  Maybe someone can convince 
Andrew (or even Linus) to offer a ruling...

> > I'm not aware of any warnings about this in the documentation.  If you 
> > would like to add something, please go ahead.
> 
> I'm going to do this.  Can I use your explanation above?

Feel free to use it.

Alan Stern


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

Hi!



> > I would disagree.  The only difference between the two is WHERE the 
> > state is maintained - ram vs. disk.  I won't really argue it though, 
> > because it's just semantics -- call it whatever you want.
> 
> It's not just semantics.  There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else.  The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections.  When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep.  If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.

I have strange system here (intel dual core prototype) that supplies
usb power even while it is "off". I'll need to find a way to fix that.

(Not that it is important to this discussion, just a reminder that strange hw exists)

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

Hi!


> >You are ignoring the question of how the kernel can tell whether two 
> >devices are in fact the same.  There is no safe way to do this, 
> >other than having the hardware verify that the device was connected 
> >the whole time.
> >
> >  
> If there is no better way to tell for sure that the device that is 
> now there is not the same as the one that was there, then the kernel 
> must assume the user did not do something stupid and continue to use 

Must?! Are you Linus or what?

> the device as if it was not disconnected ( because odds are, it in 
> fact, was not ).  In other words, which is more safe?  ALLWAYS 
> loosing data because you can't be absolutely sure that the device is 
> the same, or only loosing data if the user does something as foolish 
> as swapping drives while suspended, and you can't tell they did?

You are missing this. In 1st case, no data is actually lost, because of
sync in suspend code;
while second case is "goodbye, filesystem".
				Pavel
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Pavel Machek wrote:
> 
> Must?! Are you Linus or what?
> 

Non sequitur.

> 
> You are missing this. In 1st case, no data is actually lost, because of
> sync in suspend code;
> while second case is "goodbye, filesystem".

Provided that you sync before suspending, and there are no open files on 
the filesystem, then yes, no data will be lost.  If there are open files 
on the fs, such as a half saved document, or a running binary, or say, 
the whole root fs, then you're going to loose data and even panic the 
kernel, sync or no sync.  From the user perspective, this is unacceptable.


Why should the user give up such functionality just because the 
connection to the drive thy are using is USB?  Every other type of drive 
and interface does not suffer from this problem.

Maybe Linux should take a page from windows' playbook here.  I believe 
windows handles this scenario with a USB drive the same way it does when 
you eject a floppy and reinsert it.  The driver detects that the 
media/drive _may_ have changed and so it fails requests from the 
filesystem with an error code indicating this.  The filesystem then sets 
an override flag so it can send down some reads to verify the media. 
Generally the FS reads the super block and compares it with the in 
memory one to make sure it appears to be the same media, and if so, 
continues normal access without data loss.

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

On So 18-02-06 11:34:17, Phillip Susi wrote:
> Pavel Machek wrote:
> >You are missing this. In 1st case, no data is actually lost, because of
> >sync in suspend code;
> >while second case is "goodbye, filesystem".
> 
> Provided that you sync before suspending, and there are no open files on 
> the filesystem, then yes, no data will be lost.  If there are open files 
> on the fs, such as a half saved document, or a running binary, or say, 
> the whole root fs, then you're going to loose data and even panic the 
> kernel, sync or no sync.  From the user perspective, this is
>unacceptable.

While with your solution, you do not loose one open file, you loose
whole filesystem. Which is unacceptable.

> Why should the user give up such functionality just because the 
> connection to the drive thy are using is USB?  Every other type of drive 
> and interface does not suffer from this problem.

Because it is okay to unplug usb disk on runtime, while it is not okay
to unplug ATA disk on runtime. And because alternatives suck even more.

> Maybe Linux should take a page from windows' playbook here.  I believe 
> windows handles this scenario with a USB drive the same way it does when 
> you eject a floppy and reinsert it.  The driver detects that the 
> media/drive _may_ have changed and so it fails requests from the 
> filesystem with an error code indicating this.  The filesystem then sets 
> an override flag so it can send down some reads to verify the media. 
> Generally the FS reads the super block and compares it with the in 
> memory one to make sure it appears to be the same media, and if so, 
> continues normal access without data loss.

Feel free to implement that.
								Pavel

-- 
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Fri, 17 Feb 2006, Pavel Machek wrote:

> > If there is no better way to tell for sure that the device that is 
> > now there is not the same as the one that was there, then the kernel 
> > must assume the user did not do something stupid and continue to use 
> 
> Must?! Are you Linus or what?

Technical issues aside, the question about recogizing formerly-attached 
devices when waking up from a system sleep is really about 
user-friendliness.  The choice between what Phillip is advocating and what 
the rest of us have been saying is a policy choice, one that the kernel 
can't easily avoid making.

Given that this is a policy decision, it might not be such a bad idea to 
ask Andrew or Linus for their opinion.  In fact, I'll do that right now.


To summarize the background:

When a hotpluggable bus (USB for sure, possibly others as well) loses 
power, the hardware interprets this as a disconnection of all attached 
devices.  When power is restored, it appears to the kernel as though a 
completely new set of devices has now been plugged in.

This is unfriendly for people whose motherboard/USB-controller hardware
doesn't supply minimal power during hardware- or software-suspend.  Any
mounted filesystems on a USB storage device get blown away when the system
resumes, since the kernel thinks the device was unplugged.

In principle the kernel is capable of detecting that a device present on a
port during resume is very similar to the device that had been present
during the suspend.  Through a combination of checks (descriptors, serial
numbers, superblocks, maybe others) it could try to verify this and then
keep the device data structures intact, as though the device had been
connected all along.

The advantage is that people wouldn't lose half-stored data on their
removable drives, they wouldn't have unmount before suspending and remount 
after resuming, and they would be able to suspend with their root
filesystem on a USB drive without causing a panic upon resume.

The disadvantage is that sometimes people do switch removable drives or
removable media while the system is asleep.  If the kernel gets fooled
into thinking the new device is the same as the old one, it would proceed
to destroy the data on the new device by overwriting it with data from the
old one.  (Not to mention that this kind of approach is contrary to the
USB specification.)

On the other hand, it's what Windows does.  We don't want people to say 
that Linux is incapable of implementing a feature that Windows has had for 
a long time.  :-)


So the question is: What should the kernel do?  Assume the device has 
changed (as it does now) or make some checks to see if it might still be 
the same (dangerous though that may be)?

If you really want to waffle, you can recommend that the capability to do
this be present but normally disabled, controlled by a flag somewhere in
/sys/kernel -- thus pushing the decision off to userspace.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2957 bytes --]

On Fri, 17 Feb 2006, Pavel Machek wrote:

> > If there is no better way to tell for sure that the device that is 
> > now there is not the same as the one that was there, then the kernel 
> > must assume the user did not do something stupid and continue to use 
> 
> Must?! Are you Linus or what?

Technical issues aside, the question about recogizing formerly-attached 
devices when waking up from a system sleep is really about 
user-friendliness.  The choice between what Phillip is advocating and what 
the rest of us have been saying is a policy choice, one that the kernel 
can't easily avoid making.

Given that this is a policy decision, it might not be such a bad idea to 
ask Andrew or Linus for their opinion.  In fact, I'll do that right now.


To summarize the background:

When a hotpluggable bus (USB for sure, possibly others as well) loses 
power, the hardware interprets this as a disconnection of all attached 
devices.  When power is restored, it appears to the kernel as though a 
completely new set of devices has now been plugged in.

This is unfriendly for people whose motherboard/USB-controller hardware
doesn't supply minimal power during hardware- or software-suspend.  Any
mounted filesystems on a USB storage device get blown away when the system
resumes, since the kernel thinks the device was unplugged.

In principle the kernel is capable of detecting that a device present on a
port during resume is very similar to the device that had been present
during the suspend.  Through a combination of checks (descriptors, serial
numbers, superblocks, maybe others) it could try to verify this and then
keep the device data structures intact, as though the device had been
connected all along.

The advantage is that people wouldn't lose half-stored data on their
removable drives, they wouldn't have unmount before suspending and remount 
after resuming, and they would be able to suspend with their root
filesystem on a USB drive without causing a panic upon resume.

The disadvantage is that sometimes people do switch removable drives or
removable media while the system is asleep.  If the kernel gets fooled
into thinking the new device is the same as the old one, it would proceed
to destroy the data on the new device by overwriting it with data from the
old one.  (Not to mention that this kind of approach is contrary to the
USB specification.)

On the other hand, it's what Windows does.  We don't want people to say 
that Linux is incapable of implementing a feature that Windows has had for 
a long time.  :-)


So the question is: What should the kernel do?  Assume the device has 
changed (as it does now) or make some checks to see if it might still be 
the same (dangerous though that may be)?

If you really want to waffle, you can recommend that the capability to do
this be present but normally disabled, controlled by a flag somewhere in
/sys/kernel -- thus pushing the decision off to userspace.

Alan Stern


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Alan Stern <stern@rowland.harvard.edu> wrote:
>
> On Fri, 17 Feb 2006, Pavel Machek wrote:
> 
> > > If there is no better way to tell for sure that the device that is 
> > > now there is not the same as the one that was there, then the kernel 
> > > must assume the user did not do something stupid and continue to use 
> > 
> > Must?! Are you Linus or what?
> 
> Technical issues aside, the question about recogizing formerly-attached 
> devices when waking up from a system sleep is really about 
> user-friendliness.  The choice between what Phillip is advocating and what 
> the rest of us have been saying is a policy choice, one that the kernel 
> can't easily avoid making.
> 
> Given that this is a policy decision, it might not be such a bad idea to 
> ask Andrew or Linus for their opinion.  In fact, I'll do that right now.
> 
> 
> To summarize the background:
> 
> When a hotpluggable bus (USB for sure, possibly others as well) loses 
> power, the hardware interprets this as a disconnection of all attached 
> devices.  When power is restored, it appears to the kernel as though a 
> completely new set of devices has now been plugged in.
> 
> This is unfriendly for people whose motherboard/USB-controller hardware
> doesn't supply minimal power during hardware- or software-suspend.  Any
> mounted filesystems on a USB storage device get blown away when the system
> resumes, since the kernel thinks the device was unplugged.

I'm all for being friendly to people.

> In principle the kernel is capable of detecting that a device present on a
> port during resume is very similar to the device that had been present
> during the suspend.  Through a combination of checks (descriptors, serial
> numbers, superblocks, maybe others) it could try to verify this and then
> keep the device data structures intact, as though the device had been
> connected all along.
> 
> The advantage is that people wouldn't lose half-stored data on their
> removable drives, they wouldn't have unmount before suspending and remount 
> after resuming, and they would be able to suspend with their root
> filesystem on a USB drive without causing a panic upon resume.
> 
> The disadvantage is that sometimes people do switch removable drives or
> removable media while the system is asleep.  If the kernel gets fooled
> into thinking the new device is the same as the old one, it would proceed
> to destroy the data on the new device by overwriting it with data from the
> old one.  (Not to mention that this kind of approach is contrary to the
> USB specification.)
> 
> On the other hand, it's what Windows does.  We don't want people to say 
> that Linux is incapable of implementing a feature that Windows has had for 
> a long time.  :-)
> 
> 
> So the question is: What should the kernel do?  Assume the device has 
> changed (as it does now) or make some checks to see if it might still be 
> the same (dangerous though that may be)?
> 
> If you really want to waffle, you can recommend that the capability to do
> this be present but normally disabled, controlled by a flag somewhere in
> /sys/kernel -- thus pushing the decision off to userspace.
> 

Correct me if I'm wrong, but this really only affects storage devices, yes?
That narrows the scope of the problem quite some.

I suspect we could do a very good job of working out whether we're still
talking to the same fs if filesystem drivers were to help in that.

But I suspect we could do an even better job if we did that in userspace.

The logic to determine whether the new device is the same as the old device
can be arbitrarily complex, with increasing levels of success.  Various
heuristics can be applied, some of which will involve knowledge of
filesystem layout, etc.

So would it not be possible to optionally punt the device naming decision
up to the hotplug scripts?  So code up there can go do direct-IO reads of
the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
superblocks, disk labels, partition tables, inode numbering, etc?  Go look
up a database, work out what that filesystem was doing last time we saw it,
etc?

We could of course add things to the filesystems to help this process, but
it'd be good if all the state tracking and magic didn't have to be locked
up in the kernel.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

[-- Attachment #1: Type: text/plain, Size: 4285 bytes --]

Alan Stern <stern@rowland.harvard.edu> wrote:
>
> On Fri, 17 Feb 2006, Pavel Machek wrote:
> 
> > > If there is no better way to tell for sure that the device that is 
> > > now there is not the same as the one that was there, then the kernel 
> > > must assume the user did not do something stupid and continue to use 
> > 
> > Must?! Are you Linus or what?
> 
> Technical issues aside, the question about recogizing formerly-attached 
> devices when waking up from a system sleep is really about 
> user-friendliness.  The choice between what Phillip is advocating and what 
> the rest of us have been saying is a policy choice, one that the kernel 
> can't easily avoid making.
> 
> Given that this is a policy decision, it might not be such a bad idea to 
> ask Andrew or Linus for their opinion.  In fact, I'll do that right now.
> 
> 
> To summarize the background:
> 
> When a hotpluggable bus (USB for sure, possibly others as well) loses 
> power, the hardware interprets this as a disconnection of all attached 
> devices.  When power is restored, it appears to the kernel as though a 
> completely new set of devices has now been plugged in.
> 
> This is unfriendly for people whose motherboard/USB-controller hardware
> doesn't supply minimal power during hardware- or software-suspend.  Any
> mounted filesystems on a USB storage device get blown away when the system
> resumes, since the kernel thinks the device was unplugged.

I'm all for being friendly to people.

> In principle the kernel is capable of detecting that a device present on a
> port during resume is very similar to the device that had been present
> during the suspend.  Through a combination of checks (descriptors, serial
> numbers, superblocks, maybe others) it could try to verify this and then
> keep the device data structures intact, as though the device had been
> connected all along.
> 
> The advantage is that people wouldn't lose half-stored data on their
> removable drives, they wouldn't have unmount before suspending and remount 
> after resuming, and they would be able to suspend with their root
> filesystem on a USB drive without causing a panic upon resume.
> 
> The disadvantage is that sometimes people do switch removable drives or
> removable media while the system is asleep.  If the kernel gets fooled
> into thinking the new device is the same as the old one, it would proceed
> to destroy the data on the new device by overwriting it with data from the
> old one.  (Not to mention that this kind of approach is contrary to the
> USB specification.)
> 
> On the other hand, it's what Windows does.  We don't want people to say 
> that Linux is incapable of implementing a feature that Windows has had for 
> a long time.  :-)
> 
> 
> So the question is: What should the kernel do?  Assume the device has 
> changed (as it does now) or make some checks to see if it might still be 
> the same (dangerous though that may be)?
> 
> If you really want to waffle, you can recommend that the capability to do
> this be present but normally disabled, controlled by a flag somewhere in
> /sys/kernel -- thus pushing the decision off to userspace.
> 

Correct me if I'm wrong, but this really only affects storage devices, yes?
That narrows the scope of the problem quite some.

I suspect we could do a very good job of working out whether we're still
talking to the same fs if filesystem drivers were to help in that.

But I suspect we could do an even better job if we did that in userspace.

The logic to determine whether the new device is the same as the old device
can be arbitrarily complex, with increasing levels of success.  Various
heuristics can be applied, some of which will involve knowledge of
filesystem layout, etc.

So would it not be possible to optionally punt the device naming decision
up to the hotplug scripts?  So code up there can go do direct-IO reads of
the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
superblocks, disk labels, partition tables, inode numbering, etc?  Go look
up a database, work out what that filesystem was doing last time we saw it,
etc?

We could of course add things to the filesystems to help this process, but
it'd be good if all the state tracking and magic didn't have to be locked
up in the kernel.

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Pavel Machek wrote:
>> Provided that you sync before suspending, and there are no open files on 
>> the filesystem, then yes, no data will be lost.  If there are open files 
>> on the fs, such as a half saved document, or a running binary, or say, 
>> the whole root fs, then you're going to loose data and even panic the 
>> kernel, sync or no sync.  From the user perspective, this is
>> unacceptable.
> 
> While with your solution, you do not loose one open file, you loose
> whole filesystem. Which is unacceptable.
> 

Only if the user is foolish enough to modify the media somehow while the 
system is suspended and replace it, which is exactly how non USB disks 
currently behave.

>> Why should the user give up such functionality just because the 
>> connection to the drive thy are using is USB?  Every other type of drive 
>> and interface does not suffer from this problem.
> 
> Because it is okay to unplug usb disk on runtime, while it is not okay
> to unplug ATA disk on runtime. And because alternatives suck even more.
> 

Actually, no, it is not okay to unplug a usb disk at runtime while it is 
mounted.  It never has been and it never will be.  Also we aren't 
talking about runtime, we're talking about while the system is 
suspended, when there is no way for the kernel to know whether or not 
the device was unplugged, since it _allways_ appears to have been 
unplugged.  The alternative in the uncommon case ( where the user 
modifies the media while suspended ) does not suck any worse than it 
currently does on non usb media, and the common case ( where the user 
doesn't ) sucks worse currently with usb than others.

>> Maybe Linux should take a page from windows' playbook here.  I believe 
>> windows handles this scenario with a USB drive the same way it does when 
>> you eject a floppy and reinsert it.  The driver detects that the 
>> media/drive _may_ have changed and so it fails requests from the 
>> filesystem with an error code indicating this.  The filesystem then sets 
>> an override flag so it can send down some reads to verify the media. 
>> Generally the FS reads the super block and compares it with the in 
>> memory one to make sure it appears to be the same media, and if so, 
>> continues normal access without data loss.
> 
> Feel free to implement that.
> 								Pavel
> 

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Andrew Morton wrote:
> Correct me if I'm wrong, but this really only affects storage devices, yes?
> That narrows the scope of the problem quite some.
> 
> I suspect we could do a very good job of working out whether we're still
> talking to the same fs if filesystem drivers were to help in that.
> 

Exactly, which is why windows involves the filesystem driver.

> But I suspect we could do an even better job if we did that in userspace.
> 
> The logic to determine whether the new device is the same as the old device
> can be arbitrarily complex, with increasing levels of success.  Various
> heuristics can be applied, some of which will involve knowledge of
> filesystem layout, etc.
> 
> So would it not be possible to optionally punt the device naming decision
> up to the hotplug scripts?  So code up there can go do direct-IO reads of
> the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
> superblocks, disk labels, partition tables, inode numbering, etc?  Go look
> up a database, work out what that filesystem was doing last time we saw it,
> etc?
> 
> We could of course add things to the filesystems to help this process, but
> it'd be good if all the state tracking and magic didn't have to be locked
> up in the kernel.


Hrm... interesting but sounds like that could be sticky.  For instance, 
what if the user script that does the verifying happens to be ON the 
volume to be verified?

I think this kind of thing is really going to have to be in kernel, like 
a remount.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Andrew Morton wrote:
> Correct me if I'm wrong, but this really only affects storage devices, yes?
> That narrows the scope of the problem quite some.
> 
> I suspect we could do a very good job of working out whether we're still
> talking to the same fs if filesystem drivers were to help in that.
> 

Exactly, which is why windows involves the filesystem driver.

> But I suspect we could do an even better job if we did that in userspace.
> 
> The logic to determine whether the new device is the same as the old device
> can be arbitrarily complex, with increasing levels of success.  Various
> heuristics can be applied, some of which will involve knowledge of
> filesystem layout, etc.
> 
> So would it not be possible to optionally punt the device naming decision
> up to the hotplug scripts?  So code up there can go do direct-IO reads of
> the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
> superblocks, disk labels, partition tables, inode numbering, etc?  Go look
> up a database, work out what that filesystem was doing last time we saw it,
> etc?
> 
> We could of course add things to the filesystems to help this process, but
> it'd be good if all the state tracking and magic didn't have to be locked
> up in the kernel.


Hrm... interesting but sounds like that could be sticky.  For instance, 
what if the user script that does the verifying happens to be ON the 
volume to be verified?

I think this kind of thing is really going to have to be in kernel, like 
a remount.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Phillip Susi <psusi@cfl.rr.com> wrote:
>
> > But I suspect we could do an even better job if we did that in userspace.
>  > 
>  > The logic to determine whether the new device is the same as the old device
>  > can be arbitrarily complex, with increasing levels of success.  Various
>  > heuristics can be applied, some of which will involve knowledge of
>  > filesystem layout, etc.
>  > 
>  > So would it not be possible to optionally punt the device naming decision
>  > up to the hotplug scripts?  So code up there can go do direct-IO reads of
>  > the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
>  > superblocks, disk labels, partition tables, inode numbering, etc?  Go look
>  > up a database, work out what that filesystem was doing last time we saw it,
>  > etc?
>  > 
>  > We could of course add things to the filesystems to help this process, but
>  > it'd be good if all the state tracking and magic didn't have to be locked
>  > up in the kernel.
> 
> 
>  Hrm... interesting but sounds like that could be sticky.  For instance, 
>  what if the user script that does the verifying happens to be ON the 
>  volume to be verified?

Well that would be a bug.  Solutions would be a) don't put the scripts on a
removable/power-downable device or b) use tmpfs.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

Phillip Susi <psusi@cfl.rr.com> wrote:
>
> > But I suspect we could do an even better job if we did that in userspace.
>  > 
>  > The logic to determine whether the new device is the same as the old device
>  > can be arbitrarily complex, with increasing levels of success.  Various
>  > heuristics can be applied, some of which will involve knowledge of
>  > filesystem layout, etc.
>  > 
>  > So would it not be possible to optionally punt the device naming decision
>  > up to the hotplug scripts?  So code up there can go do direct-IO reads of
>  > the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
>  > superblocks, disk labels, partition tables, inode numbering, etc?  Go look
>  > up a database, work out what that filesystem was doing last time we saw it,
>  > etc?
>  > 
>  > We could of course add things to the filesystems to help this process, but
>  > it'd be good if all the state tracking and magic didn't have to be locked
>  > up in the kernel.
> 
> 
>  Hrm... interesting but sounds like that could be sticky.  For instance, 
>  what if the user script that does the verifying happens to be ON the 
>  volume to be verified?

Well that would be a bug.  Solutions would be a) don't put the scripts on a
removable/power-downable device or b) use tmpfs.

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

On Ne 19-02-06 00:52:45, Phillip Susi wrote:
> Pavel Machek wrote:
> >>Provided that you sync before suspending, and there are no open files on 
> >>the filesystem, then yes, no data will be lost.  If there are open files 
> >>on the fs, such as a half saved document, or a running binary, or say, 
> >>the whole root fs, then you're going to loose data and even panic the 
> >>kernel, sync or no sync.  From the user perspective, this is
> >>unacceptable.
> >
> >While with your solution, you do not loose one open file, you loose
> >whole filesystem. Which is unacceptable.
> 
> Only if the user is foolish enough to modify the media somehow while the 
> system is suspended and replace it, which is exactly how non USB disks 
> currently behave.

"Foolish enough"? Multiple users told you that they consider that use
case okay for hotpluggable drives.

> >>connection to the drive thy are using is USB?  Every other type of drive 
> >>and interface does not suffer from this problem.
> >
> >Because it is okay to unplug usb disk on runtime, while it is not okay
> >to unplug ATA disk on runtime. And because alternatives suck even more.
> >
> 
> Actually, no, it is not okay to unplug a usb disk at runtime while it is 
> mounted.  It never has been and it never will be.  Also we aren't 

Ever heard about "journalling"?

> talking about runtime, we're talking about while the system is 
> suspended, when there is no way for the kernel to know whether or
> not 

If it is okay during runtime, it should be okay while suspended. Don't
expect users to know about power on USB buses. You may call any system
that does not support standby power on USB broken if you wish... 

> the device was unplugged, since it _allways_ appears to have been 
> unplugged.  The alternative in the uncommon case ( where the user 
> modifies the media while suspended ) does not suck any worse than it 
> currently does on non usb media, and the common case ( where the user 
> doesn't ) sucks worse currently with usb than others.

"Does not such any worse than non-usb" does not cut it here. USB disks
are too easy to unplug/replug.

Anyway, your mail came without a patch, again. That's useless; if you
implement layer above floppies/usb sticks that can recognize same
disk, maybe we can talk about that.
								Pavel
-- 
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Pavel Machek wrote:
> 
> "Foolish enough"? Multiple users told you that they consider that use
> case okay for hotpluggable drives.
> 

They are confused.  It has never been, is not, and never will be okay to 
disconnect a drive while it is mounted.  It is simply an easier mistake 
to make with USB.  Easy or not, it is still a mistake.

This user error may be rather common, but as long as you sacrifice other 
functionality to prevent this user error from being particularly 
irritating to users, they will not learn and will continue to make this 
mistake.  Let them be bitten by it once or twice, and they will learn 
they need to properly unmount before yanking.

> 
> Ever heard about "journalling"?
> 

Yes, it makes it faster to recover the filesystem to a consistent state 
after a crash.  It does NOT make it okay to disconnect the drive 
willy-nilly, and does not prevent data loss.

> 
> If it is okay during runtime, it should be okay while suspended. Don't
> expect users to know about power on USB buses. You may call any system
> that does not support standby power on USB broken if you wish... 
> 

It is only okay to disconnect the drive at any time, running or 
suspended, after you unmount it.  Fail to do that and you're asking for 
trouble.

> 
> "Does not such any worse than non-usb" does not cut it here. USB disks
> are too easy to unplug/replug.
> 
> Anyway, your mail came without a patch, again. That's useless; if you
> implement layer above floppies/usb sticks that can recognize same
> disk, maybe we can talk about that.


That's exactly what I've been talking about.  You can add some logic 
that can hopefully notice if the user has modified the volume, but those 
tests need not be 100% foolproof in order to assume that the user did 
not actually modify the volume while suspended.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Andrew Morton wrote:
>>
>>  Hrm... interesting but sounds like that could be sticky.  For instance, 
>>  what if the user script that does the verifying happens to be ON the 
>>  volume to be verified?
> 
> Well that would be a bug.  Solutions would be a) don't put the scripts on a
> removable/power-downable device or b) use tmpfs.
> 

I don't see how it could be a bug, just think of the root on usb case. 
Keeping the program locked in ram would sidestep that issue, but tmpfs 
is pagable right?  Swap on usb?

Also, this user space program isn't going to be able to keep fully up to 
date on what the disk looks like.  Imagine a filesystem that keeps a 
generation counter in the super block and increments it every time it 
writes to the disk.  The user space daemon might read that, then the fs 
changes it, you suspend, and when you wake up, the daemon thinks the 
media changed because it wasn't fully up to date.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Andrew Morton wrote:
>>
>>  Hrm... interesting but sounds like that could be sticky.  For instance, 
>>  what if the user script that does the verifying happens to be ON the 
>>  volume to be verified?
> 
> Well that would be a bug.  Solutions would be a) don't put the scripts on a
> removable/power-downable device or b) use tmpfs.
> 

I don't see how it could be a bug, just think of the root on usb case. 
Keeping the program locked in ram would sidestep that issue, but tmpfs 
is pagable right?  Swap on usb?

Also, this user space program isn't going to be able to keep fully up to 
date on what the disk looks like.  Imagine a filesystem that keeps a 
generation counter in the super block and increments it every time it 
writes to the disk.  The user space daemon might read that, then the fs 
changes it, you suspend, and when you wake up, the daemon thinks the 
media changed because it wasn't fully up to date.

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Sun, 19 Feb 2006, Phillip Susi wrote:

> > If it is okay during runtime, it should be okay while suspended. Don't
> > expect users to know about power on USB buses. You may call any system
> > that does not support standby power on USB broken if you wish... 
> > 
> 
> It is only okay to disconnect the drive at any time, running or 
> suspended, after you unmount it.  Fail to do that and you're asking for 
> trouble.

I'm confused.  Weren't you arguing, only a few days ago, that it _should_
be okay to unplug a USB drive while the system is suspended?  After all,
since there's no difference (as far as the kernel can see) between power
loss on the bus and an actual unplug, you can hardly say that one should
be allowed and the other not.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Sun, 19 Feb 2006, Phillip Susi wrote:

> Andrew Morton wrote:
> >>
> >>  Hrm... interesting but sounds like that could be sticky.  For instance, 
> >>  what if the user script that does the verifying happens to be ON the 
> >>  volume to be verified?
> > 
> > Well that would be a bug.  Solutions would be a) don't put the scripts on a
> > removable/power-downable device or b) use tmpfs.
> > 
> 
> I don't see how it could be a bug, just think of the root on usb case. 
> Keeping the program locked in ram would sidestep that issue, but tmpfs 
> is pagable right?  Swap on usb?
> 
> Also, this user space program isn't going to be able to keep fully up to 
> date on what the disk looks like.  Imagine a filesystem that keeps a 
> generation counter in the super block and increments it every time it 
> writes to the disk.  The user space daemon might read that, then the fs 
> changes it, you suspend, and when you wake up, the daemon thinks the 
> media changed because it wasn't fully up to date.

There are other, harder problems associated with doing this is userspace.

Really, the device needs to be considered separately at each level of
driver processing.  At the USB level it may still appear to be the same,
but at the SCSI level it may appear different.  Or at the SCSI level it
may be the same, but at the gendisk level it may be different (different
media, partition table changed).  Or at the gendisk level it may be the
same, but at the filesystem level one or more of the partitions may be
different (superblock changed).

Each level would need its own way of checking for changes and recreating 
the appropriate data structures.  And while making the determinations, we 
would need to temporarily set the device to read-only.  Yes, userspace 
could do _some_ of the work, but it would need a lot of help from the 
kernel.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1858 bytes --]

On Sun, 19 Feb 2006, Phillip Susi wrote:

> Andrew Morton wrote:
> >>
> >>  Hrm... interesting but sounds like that could be sticky.  For instance, 
> >>  what if the user script that does the verifying happens to be ON the 
> >>  volume to be verified?
> > 
> > Well that would be a bug.  Solutions would be a) don't put the scripts on a
> > removable/power-downable device or b) use tmpfs.
> > 
> 
> I don't see how it could be a bug, just think of the root on usb case. 
> Keeping the program locked in ram would sidestep that issue, but tmpfs 
> is pagable right?  Swap on usb?
> 
> Also, this user space program isn't going to be able to keep fully up to 
> date on what the disk looks like.  Imagine a filesystem that keeps a 
> generation counter in the super block and increments it every time it 
> writes to the disk.  The user space daemon might read that, then the fs 
> changes it, you suspend, and when you wake up, the daemon thinks the 
> media changed because it wasn't fully up to date.

There are other, harder problems associated with doing this is userspace.

Really, the device needs to be considered separately at each level of
driver processing.  At the USB level it may still appear to be the same,
but at the SCSI level it may appear different.  Or at the SCSI level it
may be the same, but at the gendisk level it may be different (different
media, partition table changed).  Or at the gendisk level it may be the
same, but at the filesystem level one or more of the partitions may be
different (superblock changed).

Each level would need its own way of checking for changes and recreating 
the appropriate data structures.  And while making the determinations, we 
would need to temporarily set the device to read-only.  Yes, userspace 
could do _some_ of the work, but it would need a lot of help from the 
kernel.

Alan Stern


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Alan Stern wrote:
> 
> I'm confused.  Weren't you arguing, only a few days ago, that it _should_
> be okay to unplug a USB drive while the system is suspended?  After all,

No, I wasn't arguing that it should be okay to unplug a usb drive while 
the system is suspended, I was saying that it is better for the kernel 
to assume you did not when it can't really tell, since you aren't 
supposed to do that anyhow.

> since there's no difference (as far as the kernel can see) between power
> loss on the bus and an actual unplug, you can hardly say that one should
> be allowed and the other not.

   But there _IS_ a difference between power loss and actual unplug, so 
I very well can say one is allowed and the other is not.  The fact that 
the kernel can not tell the difference is simply a limitation that must 
be dealt with.

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

> >since there's no difference (as far as the kernel can see) between power
> >loss on the bus and an actual unplug, you can hardly say that one should
> >be allowed and the other not.
> 
>   But there _IS_ a difference between power loss and actual unplug, so 
> I very well can say one is allowed and the other is not.  The fact that 
> the kernel can not tell the difference is simply a limitation that must 
> be dealt with.

Kernel can not tell the diference, and just because you don't like the
behaviour does not mean we have to work around hardware limitation.

You deal with it. Post a patch.
								Pavel
-- 
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Alan Stern <stern@rowland.harvard.edu> wrote:
>
> On Sun, 19 Feb 2006, Phillip Susi wrote:
> 
> > Andrew Morton wrote:
> > >>
> > >>  Hrm... interesting but sounds like that could be sticky.  For instance, 
> > >>  what if the user script that does the verifying happens to be ON the 
> > >>  volume to be verified?
> > > 
> > > Well that would be a bug.  Solutions would be a) don't put the scripts on a
> > > removable/power-downable device or b) use tmpfs.
> > > 
> > 
> > I don't see how it could be a bug, just think of the root on usb case. 
> > Keeping the program locked in ram would sidestep that issue, but tmpfs 
> > is pagable right?  Swap on usb?
> > 
> > Also, this user space program isn't going to be able to keep fully up to 
> > date on what the disk looks like.  Imagine a filesystem that keeps a 
> > generation counter in the super block and increments it every time it 
> > writes to the disk.  The user space daemon might read that, then the fs 
> > changes it, you suspend, and when you wake up, the daemon thinks the 
> > media changed because it wasn't fully up to date.
> 
> There are other, harder problems associated with doing this is userspace.
> 
> Really, the device needs to be considered separately at each level of
> driver processing.  At the USB level it may still appear to be the same,
> but at the SCSI level it may appear different.  Or at the SCSI level it
> may be the same, but at the gendisk level it may be different (different
> media, partition table changed).  Or at the gendisk level it may be the
> same, but at the filesystem level one or more of the partitions may be
> different (superblock changed).
> 
> Each level would need its own way of checking for changes and recreating 
> the appropriate data structures.  And while making the determinations, we 
> would need to temporarily set the device to read-only.  Yes, userspace 
> could do _some_ of the work, but it would need a lot of help from the 
> kernel.
> 

There are two things here: a) identifying the device and b) putting it into
the correct place in the various data structures.  I suspect these can (and
should) be separated out.

For a), the current kernel behaviour is what we want - make the thing
appear at a new place in the namespace and in the hierarchy.  Then
userspace can do whatever needs to be done to identify the device, and
apply some sort of policy decision to the result.

It's what comes next that we're missing: the ability for userspace to tell
the krnrel what the decice's naming and positioning _should_ be.  The
kernel will then atomically tear down what it currently has and will then
set everything up as desired.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

[-- Attachment #1: Type: text/plain, Size: 2661 bytes --]

Alan Stern <stern@rowland.harvard.edu> wrote:
>
> On Sun, 19 Feb 2006, Phillip Susi wrote:
> 
> > Andrew Morton wrote:
> > >>
> > >>  Hrm... interesting but sounds like that could be sticky.  For instance, 
> > >>  what if the user script that does the verifying happens to be ON the 
> > >>  volume to be verified?
> > > 
> > > Well that would be a bug.  Solutions would be a) don't put the scripts on a
> > > removable/power-downable device or b) use tmpfs.
> > > 
> > 
> > I don't see how it could be a bug, just think of the root on usb case. 
> > Keeping the program locked in ram would sidestep that issue, but tmpfs 
> > is pagable right?  Swap on usb?
> > 
> > Also, this user space program isn't going to be able to keep fully up to 
> > date on what the disk looks like.  Imagine a filesystem that keeps a 
> > generation counter in the super block and increments it every time it 
> > writes to the disk.  The user space daemon might read that, then the fs 
> > changes it, you suspend, and when you wake up, the daemon thinks the 
> > media changed because it wasn't fully up to date.
> 
> There are other, harder problems associated with doing this is userspace.
> 
> Really, the device needs to be considered separately at each level of
> driver processing.  At the USB level it may still appear to be the same,
> but at the SCSI level it may appear different.  Or at the SCSI level it
> may be the same, but at the gendisk level it may be different (different
> media, partition table changed).  Or at the gendisk level it may be the
> same, but at the filesystem level one or more of the partitions may be
> different (superblock changed).
> 
> Each level would need its own way of checking for changes and recreating 
> the appropriate data structures.  And while making the determinations, we 
> would need to temporarily set the device to read-only.  Yes, userspace 
> could do _some_ of the work, but it would need a lot of help from the 
> kernel.
> 

There are two things here: a) identifying the device and b) putting it into
the correct place in the various data structures.  I suspect these can (and
should) be separated out.

For a), the current kernel behaviour is what we want - make the thing
appear at a new place in the namespace and in the hierarchy.  Then
userspace can do whatever needs to be done to identify the device, and
apply some sort of policy decision to the result.

It's what comes next that we're missing: the ability for userspace to tell
the krnrel what the decice's naming and positioning _should_ be.  The
kernel will then atomically tear down what it currently has and will then
set everything up as desired.


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Bernd Eckenfels]

Phillip Susi <psusi@cfl.rr.com> wrote:
> No, I wasn't arguing that it should be okay to unplug a usb drive while 
> the system is suspended, I was saying that it is better for the kernel 
> to assume you did not when it can't really tell, since you aren't 
> supposed to do that anyhow.

I wonder if this problem can be solved with a generic multi-path layer.
Every time a new Device shows up, it is tested for known volumnes and
re-attached. That way the USB layer can still generate new devices, and the
filesystem layer will not lose the connection to the devices. 

(with the addtional benefit that it also works cleaner for real multipath
server environments)

Gruss
Bernd

Re: Flames over -- Re: Which is simpler?

[Oliver Neukum]

Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> For a), the current kernel behaviour is what we want - make the thing
> appear at a new place in the namespace and in the hierarchy.  Then
> userspace can do whatever needs to be done to identify the device, and
> apply some sort of policy decision to the result.

How? If you have a running user space the connection to the open files
is already severed, as any access in that time window must fail.
For the rest we have udev.

	Oliver

Re: Flames over -- Re: Which is simpler?

[Oliver Neukum]

Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> For a), the current kernel behaviour is what we want - make the thing
> appear at a new place in the namespace and in the hierarchy.  Then
> userspace can do whatever needs to be done to identify the device, and
> apply some sort of policy decision to the result.

How? If you have a running user space the connection to the open files
is already severed, as any access in that time window must fail.
For the rest we have udev.

	Oliver

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Oliver Neukum <oliver@neukum.org> wrote:
>
> Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > For a), the current kernel behaviour is what we want - make the thing
> > appear at a new place in the namespace and in the hierarchy.  Then
> > userspace can do whatever needs to be done to identify the device, and
> > apply some sort of policy decision to the result.
> 
> How? If you have a running user space the connection to the open files
> is already severed, as any access in that time window must fail.

That's a separate issue, which we haven't discussed yet.  We have a device
which has gone away and which might come back later on.  Presently we will
return an I/O error if I/O is attempted in that window.  Obviously we'll
need to do something different, such as block reads and block or defer writes.

So an overall picture would be something like:

- When device is not present, reads block and writes are deferred or block

- When device appears (at a new point in the namespace) the hotplug
  scripts will go off and will identify it and will apply some policy based
  upon that identification.  That policy could be one of:

  a) accept device at its new address

  b) accept device at its new address, abandon the old address (all
     those blocked reads and writes suddenly return -EIO).

  c) remove device from its new address, splice it back to where it
     used to be, permit those blocked reads and writes to proceed.

- For devices which are absent-and-blocked, provide some ability for both
  a timeout and manual cancellation, to unblock all those readers and
  writers, make them get -EIO.

The number of potential races is, of course, huge.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Oliver Neukum <oliver@neukum.org> wrote:
>
> Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > For a), the current kernel behaviour is what we want - make the thing
> > appear at a new place in the namespace and in the hierarchy.  Then
> > userspace can do whatever needs to be done to identify the device, and
> > apply some sort of policy decision to the result.
> 
> How? If you have a running user space the connection to the open files
> is already severed, as any access in that time window must fail.

That's a separate issue, which we haven't discussed yet.  We have a device
which has gone away and which might come back later on.  Presently we will
return an I/O error if I/O is attempted in that window.  Obviously we'll
need to do something different, such as block reads and block or defer writes.

So an overall picture would be something like:

- When device is not present, reads block and writes are deferred or block

- When device appears (at a new point in the namespace) the hotplug
  scripts will go off and will identify it and will apply some policy based
  upon that identification.  That policy could be one of:

  a) accept device at its new address

  b) accept device at its new address, abandon the old address (all
     those blocked reads and writes suddenly return -EIO).

  c) remove device from its new address, splice it back to where it
     used to be, permit those blocked reads and writes to proceed.

- For devices which are absent-and-blocked, provide some ability for both
  a timeout and manual cancellation, to unblock all those readers and
  writers, make them get -EIO.

The number of potential races is, of course, huge.

Re: Flames over -- Re: Which is simpler?

[Olivier Galibert]

On Sun, Feb 19, 2006 at 08:43:44PM +0100, Pavel Machek wrote:
> Kernel can not tell the diference, and just because you don't like the
> behaviour does not mean we have to work around hardware limitation.
> 
> You deal with it. Post a patch.

It would take more than one patch, but it could be done:

Suspend time:

  1- Freeze all processes enough that no filesystem activity happens
  anymore.

  2- Do a map of the currently-opened files.  That is, for each opened
  file, find a workable path to it.  If you have the dentry, you have
  it.  Otherwise, on a filesystem that supports inodes, just
  sillyrename them to something in /.suspend.  For the non-inode
  filesystems, require not dropping the dentries for reliable suspend.
  They probably need to keep them in some form anyway for handling
  multiple opens of the same file.

  3- Sync the filesystems to the point that they're in clean state.

  4- Add the path mapping from (2) to the suspend image.

  5- Allow userspace to pick up any information it finds relevant
  about the filesystems/devices and put that info in the image too.


Resume time:

  6- Resume userspace checks the existing devices, find a mapping (or
  a lack of) using the information from (5) and the paths from (4)

  7- Rebuild the userspace from the image.

  8- Re-open/remap the files from the paths, sillyrenamed entries are
  unlinked once reopened.  Paths not found and filesystems not found
  are send to a virtual file on a virtual filesystem that just -EIOs
  or SEGVs mmaps.


Pros:
- You never lose filesystems or files, ever.  Only thing you can lose
  is processes.

- You can touch files in the suspended filesystems (device plugging,
  restart on the wrong kernel) without damage.

- You can remount the filesystems ro after 3, which allows access to
  whatever files the chrome may like.

- You only have to save the anonymous pages and the metadata.

Cons:
- Suspend may be slower since the write patterns are more dispersed
  than just swap.  OTOH, it will be no slower than sync(1).


There are probably things I haven't thought of, but I think it's the
kind of approach you need to be sure to be reliable no matter what.

  OG.

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

On Po 20-02-06 01:56:17, Olivier Galibert wrote:
> On Sun, Feb 19, 2006 at 08:43:44PM +0100, Pavel Machek wrote:
> > Kernel can not tell the diference, and just because you don't like the
> > behaviour does not mean we have to work around hardware limitation.
> > 
> > You deal with it. Post a patch.
> 
> It would take more than one patch, but it could be done:

Actually, if you really want to do this, it would probably make sense
to do at blockdevice level -- with device mapper magic or something.

That way you could prompt user "return that flash driver, I still want
to write to it" after surprise unplug, etc. And suspend is special
case of surprise unplug, then replug.

								Pavel
-- 
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...

Re: Flames over -- Re: Which is simpler?

[Olivier Galibert]

On Mon, Feb 20, 2006 at 02:01:02AM +0100, Pavel Machek wrote:
> Actually, if you really want to do this, it would probably make sense
> to do at blockdevice level -- with device mapper magic or something.
> 
> That way you could prompt user "return that flash driver, I still want
> to write to it" after surprise unplug, etc. And suspend is special
> case of surprise unplug, then replug.

I'm not sure.  Suspend is not a surprise, so you can do things so that
you don't lose anything (what I described is pretty much unmounting
while keeping file references).  Surprise unplug, there is no way you
can make the filesystem clean if it wasn't already.

I also think that USB flash and this kind of things should go back to
clean state as soon as possible even whe mountd, but that's a
different issue.

  OG.

Re: Flames over -- Re: Which is simpler?

[Alan Stern]

On Mon, 20 Feb 2006, Olivier Galibert wrote:

> On Mon, Feb 20, 2006 at 02:01:02AM +0100, Pavel Machek wrote:
> > Actually, if you really want to do this, it would probably make sense
> > to do at blockdevice level -- with device mapper magic or something.
> > 
> > That way you could prompt user "return that flash driver, I still want
> > to write to it" after surprise unplug, etc. And suspend is special
> > case of surprise unplug, then replug.
> 
> I'm not sure.  Suspend is not a surprise, so you can do things so that
> you don't lose anything (what I described is pretty much unmounting
> while keeping file references).  Surprise unplug, there is no way you
> can make the filesystem clean if it wasn't already.

I like Pavel's suggestion.  It's simpler at the device driver level 
(unless you're working on the device-mapper driver!) and it can apply to 
situations other than unplug-during-suspend.

As for making the filesystem clean following a surprise unplug...  You can
if the device is plugged back in later and you recognize it as the same
device.

> I also think that USB flash and this kind of things should go back to
> clean state as soon as possible even whe mountd, but that's a
> different issue.

A new mount option, like "-o clean"?  Something that would flush data
writes immediately (like "sync") and would flush metadata writes whenever
a file is closed and other similar occasions?  People have been asking for 
something like this.  Mounting with "sync" is very bad for flash memory 
devices, and mounting without it messes up progress-bar displays in user 
programs.  Something in between would be appreciated.  Especially for the 
VFAT driver.

Alan Stern

Re: Flames over -- Re: Which is simpler?

[Oliver Neukum]

Am Sonntag, 19. Februar 2006 22:02 schrieb Andrew Morton:
> Oliver Neukum <oliver@neukum.org> wrote:
> >
> > Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > > For a), the current kernel behaviour is what we want - make the thing
> > > appear at a new place in the namespace and in the hierarchy.  Then
> > > userspace can do whatever needs to be done to identify the device, and
> > > apply some sort of policy decision to the result.
> > 
> > How? If you have a running user space the connection to the open files
> > is already severed, as any access in that time window must fail.
> 
> That's a separate issue, which we haven't discussed yet.  We have a device
> which has gone away and which might come back later on.  Presently we will
> return an I/O error if I/O is attempted in that window.  Obviously we'll
> need to do something different, such as block reads and block or defer writes.

But how do you handle memory management?
If you simply block writes, the system will stall random tasks laundering
pages, including those needed to make progress. Even syncing before
suspend won't help you, as a running user space may dirty pages.
And what about the rootfs?

	Oliver

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Oliver Neukum <oliver@neukum.org> wrote:
>
> Am Sonntag, 19. Februar 2006 22:02 schrieb Andrew Morton:
> > Oliver Neukum <oliver@neukum.org> wrote:
> > >
> > > Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > > > For a), the current kernel behaviour is what we want - make the thing
> > > > appear at a new place in the namespace and in the hierarchy.  Then
> > > > userspace can do whatever needs to be done to identify the device, and
> > > > apply some sort of policy decision to the result.
> > > 
> > > How? If you have a running user space the connection to the open files
> > > is already severed, as any access in that time window must fail.
> > 
> > That's a separate issue, which we haven't discussed yet.  We have a device
> > which has gone away and which might come back later on.  Presently we will
> > return an I/O error if I/O is attempted in that window.  Obviously we'll
> > need to do something different, such as block reads and block or defer writes.
> 
> But how do you handle memory management?
> If you simply block writes, the system will stall random tasks laundering
> pages, including those needed to make progress. Even syncing before
> suspend won't help you, as a running user space may dirty pages.

Well of _course_ that will happen.  If we have dirty pages we need to
either keep them in memory or lose them.  If someone wants to run a massive
memory stresstest, unplug the disks in the middle of it and have the
machine serenely sail along as if nothing happened then they're being
unreasonable.

> And what about the rootfs?

If you disconnect that then everything stops until you reconnect it, provided
all the tools needed to handle the reconnect are in the correct place - if
the system providers got that wrong then the machine is obviously toast.

Your questions boil down to "what if the user is crazy or the implementation
is buggy?".   Let's assume neither is true, OK?

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Oliver Neukum <oliver@neukum.org> wrote:
>
> Am Sonntag, 19. Februar 2006 22:02 schrieb Andrew Morton:
> > Oliver Neukum <oliver@neukum.org> wrote:
> > >
> > > Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > > > For a), the current kernel behaviour is what we want - make the thing
> > > > appear at a new place in the namespace and in the hierarchy.  Then
> > > > userspace can do whatever needs to be done to identify the device, and
> > > > apply some sort of policy decision to the result.
> > > 
> > > How? If you have a running user space the connection to the open files
> > > is already severed, as any access in that time window must fail.
> > 
> > That's a separate issue, which we haven't discussed yet.  We have a device
> > which has gone away and which might come back later on.  Presently we will
> > return an I/O error if I/O is attempted in that window.  Obviously we'll
> > need to do something different, such as block reads and block or defer writes.
> 
> But how do you handle memory management?
> If you simply block writes, the system will stall random tasks laundering
> pages, including those needed to make progress. Even syncing before
> suspend won't help you, as a running user space may dirty pages.

Well of _course_ that will happen.  If we have dirty pages we need to
either keep them in memory or lose them.  If someone wants to run a massive
memory stresstest, unplug the disks in the middle of it and have the
machine serenely sail along as if nothing happened then they're being
unreasonable.

> And what about the rootfs?

If you disconnect that then everything stops until you reconnect it, provided
all the tools needed to handle the reconnect are in the correct place - if
the system providers got that wrong then the machine is obviously toast.

Your questions boil down to "what if the user is crazy or the implementation
is buggy?".   Let's assume neither is true, OK?

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

Andrew Morton <akpm@osdl.org> wrote:
>
>  > If you simply block writes, the system will stall random tasks laundering
>  > pages, including those needed to make progress. Even syncing before
>  > suspend won't help you, as a running user space may dirty pages.
> 
>  Well of _course_ that will happen.

Actually, it won't happen.  There's already logic in there to help pdflush,
kswapd and memory-allocating tasks avoid blocking on congested queues. 
It's trivial to extend that to avoidance of hotunplugged queues.

Things like sync(), fsync(), O_SYNC and reads will necessarily block.

We may or may not decide to block on page-dirtyings.  Again, that's trivial
to do in balance_dirty_pages().

Race conditions are pretty much unavoidable - if someone goes and disables
a device when we're partway through and committed to I/O submission then
things will get very sticky.  But we can have a pretty successful solution
to all of this without a ton of effort.

But this is all the easy part.

Re: Flames over -- Re: Which is simpler?

[Andrew Morton]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

Andrew Morton <akpm@osdl.org> wrote:
>
>  > If you simply block writes, the system will stall random tasks laundering
>  > pages, including those needed to make progress. Even syncing before
>  > suspend won't help you, as a running user space may dirty pages.
> 
>  Well of _course_ that will happen.

Actually, it won't happen.  There's already logic in there to help pdflush,
kswapd and memory-allocating tasks avoid blocking on congested queues. 
It's trivial to extend that to avoidance of hotunplugged queues.

Things like sync(), fsync(), O_SYNC and reads will necessarily block.

We may or may not decide to block on page-dirtyings.  Again, that's trivial
to do in balance_dirty_pages().

Race conditions are pretty much unavoidable - if someone goes and disables
a device when we're partway through and committed to I/O submission then
things will get very sticky.  But we can have a pretty successful solution
to all of this without a ton of effort.

But this is all the easy part.

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

David Brownell wrote:
> On Monday 20 February 2006 8:07 am, Phillip Susi wrote:
>> And this is exactly how non USB hardware has behaved for eons, and it 
>> hasn't been a problem. 
> 
> How many billions of years exactly?  :)
> 

<G>

> Of course it sometimes _has_ been a problem.  Repeating your claim
> doesn't make it true.  And the user model of USB was certainly so
> those problems could be _prevented_ rather than continued forever
> into new generations of hardware.
> 

But it hasn't been prevented, just changed into a less destructive, but 
more prevelant problem.  If you want to try to solve the problem then it 
should be solved in such a way that it does not cause other problems ( 
breaking mounts when you suspend ) and the solution should be 
generalized to all disks rather than just USB.

> The fact that MS-DOS did something does not make it a good idea.
> 
> 
> This is LKML.  Pointing out when patches are overdue
> can't realistically be taken as a flame; it's a
> standard way of moving beyond discussion to action.
> (Or helping someone self-educate about issues they
> won't see until they, too, look more deeply ...)
> 

I think you got the thread confused.  The flame was:

 >>> changing all that stuff, he also needs stop being a
 >>> member of the "never submitted a USB patch" club.

> However, responding to a "request for patch" in that
> way certainly does come across as a flame.

Re: Flames over -- Re: Which is simpler?

[David Brownell]

On Monday 20 February 2006 8:07 am, Phillip Susi wrote:
> 
> And this is exactly how non USB hardware has behaved for eons, and it 
> hasn't been a problem. 

How many billions of years exactly?  :)

Of course it sometimes _has_ been a problem.  Repeating your claim
doesn't make it true.  And the user model of USB was certainly so
those problems could be _prevented_ rather than continued forever
into new generations of hardware.

The fact that MS-DOS did something does not make it a good idea.


> >>> But yes, you're right ... if he's serious about
> >>> changing all that stuff, he also needs stop being a
> >>> member of the "never submitted a USB patch" club.
> >>> Ideally, starting with small things.
> >>
> >> You're moving off into left field.
> > 
> > Not hardly.  Unless all you're doing here is flaming?
> > One point of $SUBJECT was that flames were _over_ ...
> 
> Left field is where flames are, which is what the comment was that I was 
> replying to -- a flame.

This is LKML.  Pointing out when patches are overdue
can't realistically be taken as a flame; it's a
standard way of moving beyond discussion to action.
(Or helping someone self-educate about issues they
won't see until they, too, look more deeply ...)

However, responding to a "request for patch" in that
way certainly does come across as a flame.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Olivier Galibert wrote:
> 
> If I understand USB correctly, it's all point-to-point, there is no
> broadcast going on.  For enumeration purposes hubs are not
> transparent, and ports are separated.  But I wouldn't rely on the
> ports numbers on hubs to stay constant w.r.t the physical ports.  I
> don't think it's required.
> 

Hrm... that last part is crucial I think.  If the kernel can tell there 
is a device still physically connected to the same port after the 
suspend, then it could reconnect that device to the existing node, and 
maybe notify the filesystem that the disk may have changed, so it should 
attempt to verify it, the way windows does.

As for the heuristics, don't most filesystems keep a mount count in the 
superblock?  What better way to decide if things are changed than having 
the filesystem check that the superblock looks to be the same and 
specifically that it's mount count has not increased.  I don't think the 
tests need to be that complex to get rather reasonably accurate results, 
and trying to put them in user space is asking for a lot of race 
conditions that would be solved by having the filesystem handle it.

> Since it's an hotplug bus, enumeration before suspend happened as the
> devices were plugged in.  So the order a reboot enumeration will see
> them is unknown in practice.
> 

Ahh, right... I didn't think of that; the bus may have changed since the 
initial enumeration, so a re-enumeration will be different.  I was just 
thinking of the case where it didn't change while suspended.

> It may be fixable by storing some kind of physical address in the
> tree, but losing a filesystem because you replugged the usb drive in
> the wrong physical port between suspend and resume would be annoying.
> And I don't know how stable the "physical" positions are in the first
> place.

Re: Flames over -- Re: Which is simpler?

[Olivier Galibert]

On Mon, Feb 20, 2006 at 01:20:23PM -0500, Phillip Susi wrote:
> I'm not all that familiar with USB, but I'd imagine it is somewhat like 
> I2C/SMBUS: each device has a descriptor block that contains information 
> about itself.  This is going to be unique for any given device.

That one of the problems, there are no reliable serial numbers or
things of the kind.  It all becomes heuristics (model name, capacity,
partitioning, actual contents...) which kinda suck to have in kernel
space.


> The host controller begins by querying the broadcast address, and
> all unconfigured devices respond.

If I understand USB correctly, it's all point-to-point, there is no
broadcast going on.  For enumeration purposes hubs are not
transparent, and ports are separated.  But I wouldn't rely on the
ports numbers on hubs to stay constant w.r.t the physical ports.  I
don't think it's required.


> Because of this, given the same hardware on the bus, the same 
> enumeration will happen every time, and the host can assign whatever 
> address it wants to each device should it choose to do so rather than 
> just assign them in ascending order.

Since it's an hotplug bus, enumeration before suspend happened as the
devices were plugged in.  So the order a reboot enumeration will see
them is unknown in practice.

It may be fixable by storing some kind of physical address in the
tree, but losing a filesystem because you replugged the usb drive in
the wrong physical port between suspend and resume would be annoying.
And I don't know how stable the "physical" positions are in the first
place.

  OG.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

Olivier Galibert wrote:
> USB has this additional problem that devices lose their addresses when
> the power is removed (it's very agressively hotplug).  So you can have
> the devices moving around under your feet between poweroff and poweron
> just because the devices happened to have enumerated in a different
> order at boot time.
> 

Isn't that address abstracted out by the usb layer?  i.e. there is no 
relationship between the device number and the usb bus address, so 
there's no reason the usb layer can't update it's internal data 
structures to point the old device to the new bus address.  Also if the 
USB host controller wants to, can't it assign any address it likes to 
each device on the bus?  It doesn't HAVE to assign them in sequence as 
the devices are enumerated does it?

I'm not all that familiar with USB, but I'd imagine it is somewhat like 
I2C/SMBUS: each device has a descriptor block that contains information 
about itself.  This is going to be unique for any given device.  The 
host controller begins by querying the broadcast address, and all 
unconfigured devices respond.  At some point a bit in their descriptor 
blocks will differ, and there will be a collision, at which point, the 
device trying to transmit a high bit will yield and let the others 
continue.  Eventually only one device will be left having sent its 
entire descriptor block to the host, and the host can then assign a 
unique address to that device.  The host repeats this until all devices 
have been assigned an address.

Because of this, given the same hardware on the bus, the same 
enumeration will happen every time, and the host can assign whatever 
address it wants to each device should it choose to do so rather than 
just assign them in ascending order.  If the host wanted to, it could 
power down the bus, and when it powers back up, it could assign the same 
addresses that they had before to the devices as they are enumerated, 
and new devices would get unused addresses.

Re: Flames over -- Re: Which is simpler?

[Olivier Galibert]

On Mon, Feb 20, 2006 at 11:07:54AM -0500, Phillip Susi wrote:
> Again, the hardware is perfectly free to power off the usb bus during 
> suspend to ram.  Most systems choose not to because they allow wake from 
> usb, but not all do.

USB has this additional problem that devices lose their addresses when
the power is removed (it's very agressively hotplug).  So you can have
the devices moving around under your feet between poweroff and poweron
just because the devices happened to have enumerated in a different
order at boot time.

  OG.

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

David Brownell wrote:
> 
> Exactly:  ignore those disconnects in "some" cases.  Suspend-to-RAM
> will typically never report disconnects without a real unplug.  You
> want to add special casing for hibernate/swsusp.  (A point in favor
> of someone's claim that hibernate/swsusp is structurally harder.)
> 

Typical != always.  It may be more common for suspend to maintain usb 
power, but both suspend and hibernate may or may not maintain usb power, 
so the kernel needs to be prepared to deal with that in both cases.

> 
> Now with /dev/input/mice, the driver stack above USB is able to mask
> such disconnects.  It's not like mice maintain state that matters.
> The "ignore" is in stack layers way above USB, which can know a very
> important thing about mice:  they are stateless.
> 
> But with storage media, there is no such mechanism ... and there's
> significant state involved.  Adding a "reconnect" mechanism, and
> getting it wrong for storage, likely means corrupted file systems.
> And where even if it _is_ the same physical disk, there's no good
> reason to expect it hasn't been modified on some other usb host.
> (Toss hardware in bag, reuse as needed...)
> 

And this is exactly how non USB hardware has behaved for eons, and it 
hasn't been a problem.  If you want to add some verification to make 
reasonably sure that the media has not been modified, that is great, but 
you can't just automatically dump the filesystem and kill running 
processes and loose data just because something bad _may_ have happened.

> No thanks, I prefer data integrity.  And for that matter, re $SUBJECT,
> the much simpler approach of working _with_ the hardware architecture,
> not against it.
> 

Again, the hardware is perfectly free to power off the usb bus during 
suspend to ram.  Most systems choose not to because they allow wake from 
usb, but not all do.

> 
>>> But yes, you're right ... if he's serious about
>>> changing all that stuff, he also needs stop being a
>>> member of the "never submitted a USB patch" club.
>>> Ideally, starting with small things.
>> You're moving off into left field.
> 
> Not hardly.  Unless all you're doing here is flaming?
> One point of $SUBJECT was that flames were _over_ ...
> 

Left field is where flames are, which is what the comment was that I was 
replying to -- a flame.

Re: Flames over -- Re: Which is simpler?

[David Brownell]

On Saturday 18 February 2006 10:06 pm, Phillip Susi wrote:
> David Brownell wrote:
> > 
> > Hardware is CORRECTLY reporting electrical disconnects,
> > but Philip is wanting Linux to ignore those reports.
> 
> No, not ignore, just realize that an electrical disconnection does not 
> necessarily mean that the volume can no longer be accessed.

Exactly:  ignore those disconnects in "some" cases.  Suspend-to-RAM
will typically never report disconnects without a real unplug.  You
want to add special casing for hibernate/swsusp.  (A point in favor
of someone's claim that hibernate/swsusp is structurally harder.)


Now with /dev/input/mice, the driver stack above USB is able to mask
such disconnects.  It's not like mice maintain state that matters.
The "ignore" is in stack layers way above USB, which can know a very
important thing about mice:  they are stateless.

But with storage media, there is no such mechanism ... and there's
significant state involved.  Adding a "reconnect" mechanism, and
getting it wrong for storage, likely means corrupted file systems.
And where even if it _is_ the same physical disk, there's no good
reason to expect it hasn't been modified on some other usb host.
(Toss hardware in bag, reuse as needed...)

No thanks, I prefer data integrity.  And for that matter, re $SUBJECT,
the much simpler approach of working _with_ the hardware architecture,
not against it.


> > But yes, you're right ... if he's serious about
> > changing all that stuff, he also needs stop being a
> > member of the "never submitted a USB patch" club.
> > Ideally, starting with small things.
> 
> You're moving off into left field.

Not hardly.  Unless all you're doing here is flaming?
One point of $SUBJECT was that flames were _over_ ...

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

David Brownell wrote:
> 
> Hardware is CORRECTLY reporting electrical disconnects,
> but Philip is wanting Linux to ignore those reports.
> 
> 

No, not ignore, just realize that an electrical disconnection does not 
necessarily mean that the volume can no longer be accessed.

> 
> No patch possible.  Reading the other messages in that
> thread, Philip is advocating Linux ignore the USB spec.
> (Which is what _he_ appears to have been doing...)
> 

Non sequitur.  The USB spec does not say the kernel must force unmount 
the drive.

> What he has to do is more than submit a patch.  He first
> needs to lobby the USB-IF to change the USB spec, and
> get every peripheral vendor to stop shipping USB devices
> and instead switch over to "Philip-USB".  Then get all
> the billions of USB peripherals to go into the recycle
> bin and be replaced with products conforming to his
> new variant.  It all seems highly unlikely.  ;)
> 
> 
> But yes, you're right ... if he's serious about
> changing all that stuff, he also needs stop being a
> member of the "never submitted a USB patch" club.
> Ideally, starting with small things.
> 


You're moving off into left field.

Re: Flames over -- Re: Which is simpler?

[David Brownell]

On Wednesday 15 February 2006 3:43 pm, Pavel Machek wrote:
> 
> > Are you quite certain about that?  This is not 
> > the case for SCSI disks, but for USB, maybe it 
> > can provide sufficient information to the 
> > kernel about state changes without having to do 
> > a full rescan.  If that is the case, and the 
> > hardware is erroneously reporting that all 

Hardware is CORRECTLY reporting electrical disconnects,
but Philip is wanting Linux to ignore those reports.


> > devices were disconnected and reconnected after 
> > an ACPI suspend to disk, then such hardware is 
> > broken and the kernel should be patched to work 
> > around it. 
> 
> No patch was attached...

No patch possible.  Reading the other messages in that
thread, Philip is advocating Linux ignore the USB spec.
(Which is what _he_ appears to have been doing...)

What he has to do is more than submit a patch.  He first
needs to lobby the USB-IF to change the USB spec, and
get every peripheral vendor to stop shipping USB devices
and instead switch over to "Philip-USB".  Then get all
the billions of USB peripherals to go into the recycle
bin and be replaced with products conforming to his
new variant.  It all seems highly unlikely.  ;)


But yes, you're right ... if he's serious about
changing all that stuff, he also needs stop being a
member of the "never submitted a USB patch" club.
Ideally, starting with small things.

Re: Flames over -- Re: Which is simpler?

[Pavel Machek]

> Are you quite certain about that?  This is not 
> the case for SCSI disks, but for USB, maybe it 
> can provide sufficient information to the 
> kernel about state changes without having to do 
> a full rescan.  If that is the case, and the 
> hardware is erroneously reporting that all 
> devices were disconnected and reconnected after 
> an ACPI suspend to disk, then such hardware is 
> broken and the kernel should be patched to work 
> around it. 

No patch was attached...

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: Flames over -- Re: Which is simpler?

[David Brownell]

On Monday 13 February 2006 10:05 pm, Phillip Susi wrote:
> David Brownell wrote:
> > No, not "AFAIK" ... since when I told you explicitly that was untrue,
> > you then ignored that statement.  And didn't look at the specs that
> > I pointed you towards, which provide the details.  (USB 2.0 spec re
> > hubs; and of course the Linux-USB hub driver ... www.usb.org)
> 
> I ignored nothing.  I fully accepted your explanation as true and 
> pointed out that it changes nothing;

Sorry, I still can't see a way to read your response to me in that way.
When I said "X", you said "AFAIK, X is false".  More than once in the
same post ... e.g. you say "all hardware must be re-probed", I said "all
is wrong" and provided a common counterexample with USB, then you still
said "all/AFAIK".  And then tried to switch to another topic (see below).
I don't have time to waste on such non-dialogue.

> data loss in this perfectly valid  
> use case just because the kernel can not be absolutely certain the user 
> did not do something stupid while suspended is unacceptable.

Odd, data loss wasn't even mentioned in any of the comments of yours
to which I was responding.  I was providing corrections to what you
were writing about suspend-to-RAM cases.

- Dave

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

David Brownell wrote:
> No, not "AFAIK" ... since when I told you explicitly that was untrue,
> you then ignored that statement.  And didn't look at the specs that
> I pointed you towards, which provide the details.  (USB 2.0 spec re
> hubs; and of course the Linux-USB hub driver ... www.usb.org)

I ignored nothing.  I fully accepted your explanation as true and 
pointed out that it changes nothing; data loss in this perfectly valid 
use case just because the kernel can not be absolutely certain the user 
did not do something stupid while suspended is unacceptable.

> 
> The events that a hub receives say pretty exactly what happened.
> You should know that already, since USB behaves that way even
> when the system is _not_ suspended ... 
> 

How it behaves while running and how it behaves while suspended are 
usually two very different things.

> The full mechanism for USB is more like wakeup signaling on USB triggering
> hub wakeup (possibly cascading through a few layers of external hub), at
> some point triggering root hub wakeup, which maps to a PME# signal.  That
> relies on no more than VBUS being powered at a fraction of a milliAmpere,
> and the equivalent of a pair of voltage comparators triggering wakeup when
> USB signaling changes from J to K states for something like 10 msec.
> 
> 
> 
> Did you read about the PME# signal in the PCI PM spec?  www.pci-sig.com
> Maybe you could try that. 
> 

No, I took your word for it without protest as it doesn't change my main 
argument: data loss in the face of normal usage is not acceptable. 
Claiming that it has to be this way because the alternative _might_ 
result in data loss in the worst (mis)use case is an untenable position.


> Also the ACPI spec ... the early chapters give a decent overview of the
> different components of that model.  (ISTR two chapters try that, with
> the second being more to-the-point despite some duplicated graphics.)
> 
> - Dave
> 
>  
> 

Re: Flames over -- Re: Which is simpler?

[David Brownell]

On Monday 13 February 2006 12:08 pm, Phillip Susi wrote:
> David Brownell wrote:
> > What ide drive?  Oh, you're talking about PC-ish systems, not
> > embedded ones that don't _have_ rotating media to power off.
> 
> Yes, that's exactly what the discussion is about; disk drives with 
> mounted filesystems and what happens to them when you suspend/resume.

Odd, the original stuff I noticed talked about Linux PM in general,
and specifically why STR deserves a heck of a lot more attention than
it's been getting so far ... not just systems with rotating media.
(Too much attention on rotating media means that really low power
systems have been getting a bit shortchanged in Linux.)


Be that as it may:
 
> > Your experience is very different from mine; I've observed that
> > most PC hardware keeps USB powered in suspend-to-ram states, so
> > a keyboard or mouse action may wake the system up, just as it can
> > with many PS2 style keyboards and mice.  Same thing for Ethernet,
> > using various types of wakeup event including "magic packet".
> > See /proc/acpi/wakeup, and the related parts of the ACPI specs.
> > (And USB specs, and lots more ... this info is widespread.)
> 
> As I have said before, some systems can keep the USB bus in a low power 
> mode where it can wake the system, but AFAIK, waking the system is all 
> they can do in this state; they can not tell the kernel that device x 
> has been connected and device y has been disconnected, ... 

No, not "AFAIK" ... since when I told you explicitly that was untrue,
you then ignored that statement.  And didn't look at the specs that
I pointed you towards, which provide the details.  (USB 2.0 spec re
hubs; and of course the Linux-USB hub driver ... www.usb.org)

The events that a hub receives say pretty exactly what happened.
You should know that already, since USB behaves that way even
when the system is _not_ suspended ... 

The full mechanism for USB is more like wakeup signaling on USB triggering
hub wakeup (possibly cascading through a few layers of external hub), at
some point triggering root hub wakeup, which maps to a PME# signal.  That
relies on no more than VBUS being powered at a fraction of a milliAmpere,
and the equivalent of a pair of voltage comparators triggering wakeup when
USB signaling changes from J to K states for something like 10 msec.


> > Read about the #PME signal status in the PCI PM capabilities.
> >
> > And the USB remote wakeup reporting done by USB hubs; you can
> > even look at the drivers/usb/core/hub.c code and see how usb
> > wakeup events (of various types) are handled.
> >
> > You don't seem to know what you're talking about here.   
> 
> Which is why I qualified my statements with "AFAIK".  Maybe you could 
> enlighten me.  Does the #PME signal carry enough information to inform 
> the kernel that the reason the system is being woken up is because you 
> unplugged the mouse from the usb hub? 

Did you read about the PME# signal in the PCI PM spec?  www.pci-sig.com
Maybe you could try that. 

Also the ACPI spec ... the early chapters give a decent overview of the
different components of that model.  (ISTR two chapters try that, with
the second being more to-the-point despite some duplicated graphics.)

- Dave

 

Re: Flames over -- Re: Which is simpler?

[Phillip Susi]

David Brownell wrote:
> What ide drive?  Oh, you're talking about PC-ish systems, not
> embedded ones that don't _have_ rotating media to power off.
>   

Yes, that's exactly what the discussion is about; disk drives with 
mounted filesystems and what happens to them when you suspend/resume. 
> Your experience is very different from mine; I've observed that
> most PC hardware keeps USB powered in suspend-to-ram states, so
> a keyboard or mouse action may wake the system up, just as it can
> with many PS2 style keyboards and mice.  Same thing for Ethernet,
> using various types of wakeup event including "magic packet".
> See /proc/acpi/wakeup, and the related parts of the ACPI specs.
> (And USB specs, and lots more ... this info is widespread.)
>   

As I have said before, some systems can keep the USB bus in a low power 
mode where it can wake the system, but AFAIK, waking the system is all 
they can do in this state; they can not tell the kernel that device x 
has been connected and device y has been disconnected, the kernel must 
figure that out by probing the hardware and comparing the list with what 
it knew to be connected prior to suspending.  Even if some hardware can 
provide that information, kernel can not rely on it. 

<snip>

> You should also remember a useful point from the ACPI spec:
> if you're taking the system apart with a screwdriver, then
> you've gone out-of-spec.  So swapping IDE drives is strongly
> in the "user error" category ... unlike swapping USB drives,
> which is equally strongly in the "normal operation" category.
> (So it would be an error if Linux didn't properly detect when
> users unplug all usb devices after putting their laptops into
> STR and before carrying them someplace else...)
>
>   
Yes, it would be an error if the kernel did not notice that you actually 
did unplug a usb disk while it was suspended.  It is just as much an 
error however, if the kernel erroneously decides that you disconnected 
the drive, and thus breaks the mount on it, when you in fact, did no 
such thing. 
>
> Read about the #PME signal status in the PCI PM capabilities.
>
> And the USB remote wakeup reporting done by USB hubs; you can
> even look at the drivers/usb/core/hub.c code and see how usb
> wakeup events (of various types) are handled.
>
> You don't seem to know what you're talking about here.
>
>   

Which is why I qualified my statements with "AFAIK".  Maybe you could 
enlighten me.  Does the #PME signal carry enough information to inform 
the kernel that the reason the system is being woken up is because you 
unplugged the mouse from the usb hub?  I was under the impression that 
this was not the case, instead the kernel at best knows it is being 
woken up because the USB host controller generated a wake event.  
Because of this and the fact that SCSI busses that I have seen do no 
such thing, I assumed the kernel would handle USB in the same way, 
specifically, that it would reenumerate and provided the same hardware 
is still there that it expects to be there, then it would continue 
operation as normal rather than deciding the drive was unplugged. 
> You were wrong then too... both about probing "all hardware" and about not
> being able to distinguish wakeup event sources.  
>   

Are you quite certain about that?  This is not the case for SCSI disks, 
but for USB, maybe it can provide sufficient information to the kernel 
about state changes without having to do a full rescan.  If that is the 
case, and the hardware is erroneously reporting that all devices were 
disconnected and reconnected after an ACPI suspend to disk, then such 
hardware is broken and the kernel should be patched to work around it. 

Re: Flames over -- Re: Which is simpler?

[David Brownell]

I think "simpler" isn't the issue; "swsusp" is a kind of checkpoint
even in its ACPI versions, while standby/str modes are substantially
different beasts.

True standby/suspend modes are classic "power management" things:
part of the system stay in low power modes (not necessarily "off")
to reduce power consumption, while other parts are in active use.
The more deeply they're suspended, the less the system looks like
"idle" and the more it looks like "suspend-to-ram".


Phillip Susi wrote:
> 	Thus when suspended to ram, 
> typically your usb hard drive and almost allways your ide/sata/scsi 
> drive will be completely powered off. 

What ide drive?  Oh, you're talking about PC-ish systems, not
embedded ones that don't _have_ rotating media to power off.

Your experience is very different from mine; I've observed that
most PC hardware keeps USB powered in suspend-to-ram states, so
a keyboard or mouse action may wake the system up, just as it can
with many PS2 style keyboards and mice.  Same thing for Ethernet,
using various types of wakeup event including "magic packet".
See /proc/acpi/wakeup, and the related parts of the ACPI specs.
(And USB specs, and lots more ... this info is widespread.)

At the PCI hardware level, that basically means staying in
what's called a D3hot state:  controller maintains power, and
might be a wakeup event source.  If you read the EHCI spec
you'll notice careful specification of how to maintain USB
connections by providing only minimal VBUS current using the
Vaux supply ... not all boards go that extra distance for
power savings, maintaining USB state (and allowing wakeup)
using a lower power D3cold state (with ACPI S3) not D3hot.

(And there's gobs of "legacy" PCI PM hardware still being
manufactured, using out-of-band signaling rather than the
standardized PCI PM stuff.  For that, see the relevant USB
controller specs ... e.g. OHCI, Intel UHCI, VIA UHCI.)

For non-PCI hardware, that's actually a system option.  Most
systems have many low power modes, including ones where it'd
make sense to support wakeup from USB and ones where it would
not; external transceivers seem to give the most flexibility.

You should also remember a useful point from the ACPI spec:
if you're taking the system apart with a screwdriver, then
you've gone out-of-spec.  So swapping IDE drives is strongly
in the "user error" category ... unlike swapping USB drives,
which is equally strongly in the "normal operation" category.
(So it would be an error if Linux didn't properly detect when
users unplug all usb devices after putting their laptops into
STR and before carrying them someplace else...)


> Then your motherboard keeps the bus in a lower power state such that it 
> is capable of causing a wake event when state changes.  I'm also fairly 
> sure that when such a wake event happens, there is no indication to the 
> kernel about why it was woken up. 

Read about the #PME signal status in the PCI PM capabilities.

And the USB remote wakeup reporting done by USB hubs; you can
even look at the drivers/usb/core/hub.c code and see how usb
wakeup events (of various types) are handled.

You don't seem to know what you're talking about here.


> 	 As I said before, once woken up, 
> the kernel must probe all hardware and if there is a new device, it will 
> find it.  The hardware only detected that something changed and thus, 
> generated a wake even, it did not notice exactly what that change was 
> and inform the kernel. 

You were wrong then too... both about probing "all hardware" and about not
being able to distinguish wakeup event sources.  

- Dave