[2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[bert hubert]

Ok,

This is about stupid users (including me) unplugging USB devices whilst
still mounted, and expecting sane semantics.

This has generally not been the 'Unix' or even 'Linux' way, but people
expect it to work. I also see no clear automated and robust solution from
userspace. "Don't do that then" is a pretty weak answer, especially since we
want to work on the desktop.

The expected behaviour is that on forceably unplugging an USB memory stick,
the created SCSI device should vanish, along with the mounts based on it.

When the user plugs in the device again, people expect to see it get the
first available name, and be available for remount, possible automated.

What actually happens with 2.6.9-rc4 is this:

[plug in the memory stick]
usb 1-2: new high speed USB device using address 9
scsi7 : SCSI emulation for USB Mass Storage devices
  Vendor: M-Sys     Model: DiskOnKey         Rev: 4.20
  Type:   Direct-Access                      ANSI SCSI revision: 02
Attached scsi removable disk sda at scsi7, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi7, channel 0, id 0, lun 0,  type 0

Note that the numbers 7 and 9 increase which each unplug/replug event, even
when no mount was attempted. This may be a bad thing in and of itself.

Now we mount:

# mount /dev/sda1 /keychain
# grep /keychain /proc/mounts 
/dev/sda1 /keychain vfat rw,nodiratime,fmask=0033,dmask=0033 0 0

dmesg reports:
USB Mass Storage device found at 8
SCSI device sda: 499712 512-byte hdwr sectors (256 MB)
sda: Write Protect is off
sda: Mode Sense: 45 00 00 08
sda: assuming drive cache: write through
SCSI device sda: 499712 512-byte hdwr sectors (256 MB)
sda: Write Protect is off
sda: Mode Sense: 45 00 00 08
sda: assuming drive cache: write through
 sda: sda1

Note the duplication.

Now we unplug the memory stick:

usb 1-2: USB disconnect, address 9
# grep /keychain /proc/mounts 
/dev/sda1 /keychain vfat rw,nodiratime,fmask=0033,dmask=0033 0 0

# ls /keychain
# 

No errors reported by ls, dmesg is filled with:
scsi7 (0:0): rejecting I/O to dead device
FAT: Directory bread(block 517) failed
scsi7 (0:0): rejecting I/O to dead device
FAT: Directory bread(block 518) failed
scsi7 (0:0): rejecting I/O to dead device
FAT: Directory bread(block 519) failed
scsi7 (0:0): rejecting I/O to dead device
FAT: Directory bread(block 520) failed

We can unmount /keychain, this reports in dmesg:
SCSI error: host 7 id 0 lun 0 return code = 4000000
	Sense class 0, sense error 0, extended sense 0

# grep /keychain /proc/mounts
#

On reinsert, we can mount again:
usb 1-2: new high speed USB device using address 10
scsi8 : SCSI emulation for USB Mass Storage devices
  Vendor: M-Sys     Model: DiskOnKey         Rev: 4.20
  Type:   Direct-Access                      ANSI SCSI revision: 02
Attached scsi removable disk sda at scsi8, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi8, channel 0, id 0, lun 0,  type 0

Sometimes however, sda appears to be still 'occupied', and higher names are
used.

Now - the perhaps intended behaviour where the user can replug the USB
device when it was disconnected by accident also does not work. When we do
this, things get really out of whack, /dev/sda1 has now become invalid.

Unmounting and unplugging and replugging saves us.

Greg, others, I hope you agree this needs work. I hope we have the
infrastructure to umount based on USB disconnect events, or, alternatively,
will support 'replugging' which at least does part of what people expect.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[Kay Sievers]

On Mon, Oct 11, 2004 at 02:07:01PM +0200, bert hubert wrote:
> Ok,
> 
> This is about stupid users (including me) unplugging USB devices whilst
> still mounted, and expecting sane semantics.
> 
> This has generally not been the 'Unix' or even 'Linux' way, but people
> expect it to work. I also see no clear automated and robust solution from
> userspace. "Don't do that then" is a pretty weak answer, especially since we
> want to work on the desktop.
> 
> The expected behaviour is that on forceably unplugging an USB memory stick,
> the created SCSI device should vanish, along with the mounts based on it.

That is clearly bejond the scope of the kernel or hotplug. This policy
belongs to some other device management software. We are currently working on
HAL as one example, to make that happen.

> When the user plugs in the device again, people expect to see it get the
> first available name, and be available for remount, possible automated.
...
> Sometimes however, sda appears to be still 'occupied', and higher names are
> used.
> 
> Now - the perhaps intended behaviour where the user can replug the USB
> device when it was disconnected by accident also does not work. When we do
> this, things get really out of whack, /dev/sda1 has now become invalid.

Forget about the kernel device names, these are "cookies" to access the
device and have no other meaning. Never rely on that longer as your
device session lasts! You may use a udev rule to create a stable name for
your device based on some unique device property and that will work.
Btw: With HAL we even don't care about the /dev-name, all volumes are
recognized by uuid or fslabel.

> Unmounting and unplugging and replugging saves us.
> 
> Greg, others, I hope you agree this needs work. I hope we have the
> infrastructure to umount based on USB disconnect events, or, alternatively,
> will support 'replugging' which at least does part of what people expect.

Yes, we need to make the unplug of mounted devices more safe, especially
with sync mount, but don't expect the kernel or hotplug to do anything
like that. It's up to some policy software higher in the stack.

Thanks,
Kay

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[David Brownell]

On Monday 11 October 2004 8:37 am, Kay Sievers wrote:
> On Mon, Oct 11, 2004 at 02:07:01PM +0200, bert hubert wrote:
> > 
> > The expected behaviour is that on forceably unplugging an USB
> > memory stick, 
> > the created SCSI device should vanish, along with the mounts based on it.
> 
> That is clearly bejond the scope of the kernel or hotplug. This policy
> belongs to some other device management software

I've got to disagree with the "clearly", if not that entire claim.

"Clearly" would imply there's some other sane default policy
that doesn't amount to "wedge the system".

- Dave

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[bert hubert]

On Mon, Oct 11, 2004 at 05:37:19PM +0200, Kay Sievers wrote:

> > The expected behaviour is that on forceably unplugging an USB memory stick,
> > the created SCSI device should vanish, along with the mounts based on it.
> 
> That is clearly bejond the scope of the kernel or hotplug. This policy
> belongs to some other device management software. We are currently working on
> HAL as one example, to make that happen.

There is no way for userspace to do this. I thank you for your kneejerk 'it
is a userspace problem' report though - somebody has to do it.

Not very productive however. The kernel does not perform minimal services,
and gets confused to boot.

Now I'm all for offloading hotplug &c to userspace but when a device is
gone, people expect the associated devices to vanish as well, not linger
around generating invisible errors in dmesg.

> Yes, we need to make the unplug of mounted devices more safe, especially
> with sync mount, but don't expect the kernel or hotplug to do anything
> like that. It's up to some policy software higher in the stack.

Pray tell how I can do this from 'higher in the stack'. At least something
needs to happen, because now there is the possibility of lingering devices,
and I'm pretty sure we could get un-umountable mounts pointing there.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[James Bruce]

There really is a 90% solution all in userspace, which is at the 
bottom.  The rest of this message is mostly an argument for why the 
"user expecations" aren't really supportable.

bert hubert wrote:

>This is about stupid users (including me) unplugging USB devices whilst
>still mounted, and expecting sane semantics.
>
>This has generally not been the 'Unix' or even 'Linux' way, but people
>expect it to work. I also see no clear automated and robust solution from
>userspace. "Don't do that then" is a pretty weak answer, especially since we
>want to work on the desktop.
>  
>
How do you expect writing to a device followed by a forced dismount to 
work if you aren't using a data journaled file system, and you don't 
tell it needs to clean its caches?  Our lab has been using memory sticks 
for embedded development for 5 years and we've managed to teach people 
"don't do that" pretty well (in fact I just taught another person 
today).  It is of course a mistake everyone makes a few times while 
learning.  However the last time I used Windows for this, you had to 
click on the taskbar to shut down the USB storage device, which is 
basically the same thing as unmounting.  Not doing so often required a 
reboot.

With *nix, most data only gets written at unmount, so the only way this 
can "sanely" work is for mounts you haven't written to.  That case is of 
course not currently handled very well, but writing would be damn near 
impossible to unmount well.  In order to keep the device consistent, the 
only thing you can do is wait for the user to reinsert the device and 
then clear your caches.  However they might have modified the storage in 
the meantime on another device, so you'd need some sort of consistency 
check and a mirror of pretty much everything in order to do that check.  
See how this is gets complicated real quickly?

In the days of DOS, you could just cut the power to the computer to turn 
it off; eventually users were educated not to do that, for much the same 
reason (unwritten data to storage devices).  I think with a well 
designed UI, most of these errors can be eliminated.

>The expected behaviour is that on forceably unplugging an USB memory stick,
>the created SCSI device should vanish, along with the mounts based on it.
>  
>
Along with any data that hasen't yet been written to the drive.  You're 
quite likely to corrupt a fragile FS such as FAT.

>When the user plugs in the device again, people expect to see it get the
>first available name, and be available for remount, possible automated.
>  
>
Automated mounting with special fixed names can already be done, this 
has little to do with forced dismounting.  Use something like udev for 
this part.

># mount /dev/sda1 /keychain
># grep /keychain /proc/mounts 
>/dev/sda1 /keychain vfat rw,nodiratime,fmask=0033,dmask=0033 0 0
>  
>
Read-write VFAT without being mounted sync will pretty much never work 
for forced-dismount if you have written anything.

>Unmounting and unplugging and replugging saves us.
>  
>
You're likely to have corruption too if you did any writing.

>Greg, others, I hope you agree this needs work. I hope we have the
>infrastructure to umount based on USB disconnect events, or, alternatively,
>will support 'replugging' which at least does part of what people expect.
>  
>
That infrastructure would have to include knowing when to clear caches 
before the user ever disconnects the device.  In other words, not 
possible, unless you force it to sync constantly which is not very 
healthy for a flash device (limited number or writes before it dies).  
Replugging is the only possibility that could ever support writes.  The 
only case that could really be *solved* is the "read-only or no-writes" 
condition, which is only 50% of the time for something like flash used 
to transfer files.  The only thing would could reasonably expect with 
writing is to be able to clear the pinned resources somehow.

Well, what are we to do then when new university students have to use 
the use the system for a class?  Simply wrap copies in a script like the 
following:

copy-to-memstick:
    if(!mounted) mount /memstick
    rsync $arg1 /memstick
    umount /memstick

All I ever expect the kernel to eventually support is forced dismount of 
devices that haven't been written to.  I think from there its up to 
userspace to sync whenever it thinks its done copying, or perhaps even 
to speculatively unmount something that hasen't been used in a while.  A 
common data-journaled file system for use on flash could change things, 
but I'm not holding my breath for other devices or OSes to support 
something like that.

If you have an idea how your proposed behavior could be implemented with 
details sorted out, such as writing and where the data in caches goes, 
then please prove me wrong.  That would make our students happier anyway...

 - Jim Bruce

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[Oliver Neukum]

 
> With *nix, most data only gets written at unmount, so the only way this 
> can "sanely" work is for mounts you haven't written to.  That case is of 

This is not a law of nature. You can mount sync as well. That, of course,
sucks in terms of performance and wear. A reasonable compromise
would be to do sync on close.
Supermount did this years ago.

> course not currently handled very well, but writing would be damn near 
> impossible to unmount well.  In order to keep the device consistent, the 
> only thing you can do is wait for the user to reinsert the device and 
> then clear your caches.  However they might have modified the storage in 

You cannot. That's giving mlock() to everybody.

[..]
> Automated mounting with special fixed names can already be done, this 
> has little to do with forced dismounting.  Use something like udev for 
> this part.

Exactly.

[..] 
> All I ever expect the kernel to eventually support is forced dismount of 
> devices that haven't been written to.  I think from there its up to

Devices break. You have to cope with devices going away suddenly.
You are not required to ensure data integrity in all cases, but the system
must not suffer. To allow that you must be able to get rid of the mounts
even if users do not cooperate. 

	Regards
		Oliver

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[bert hubert]

On Tue, Oct 12, 2004 at 12:24:44PM +0200, Oliver Neukum wrote:

> Devices break. You have to cope with devices going away suddenly.
> You are not required to ensure data integrity in all cases, but the system
> must not suffer. To allow that you must be able to get rid of the mounts
> even if users do not cooperate. 

Well, in retrospect, the kernel appears to offer the following semantics,
perhaps unintentionally:

	When a device goes away for any reason, but there are mounts that
	refer to it, the device nominally stays around and an umount will
	always succeed, removing the vestiges of the device with it.

This would in fact allow something in userspace listening to hotplug events
to umount on a disconnect event from USB. Except that I'm not sure if the
semantics above are guaranteed - they may just be an accident.

Things get more complicated if we have logical volumes or raid partitions
which ultimately depend on a device that is removed. In this case, userspace
should be aware of all dependencies in order to know which mountpoints to
umount. This might even include loopback mounts.

The kernel knows the dependencies implicitly and might be in a better
position to know what is invalidated by a disconnect, and which devices
disappear because of dependencies on it.

I'm hoping either Greg or Al will chime in - it appears as if part of the
infrastructure is there, but not quite developed.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: [2.6.9-rc4] USB && mass-storage && disconnect broken semantics

[Linas Vepstas]

On Tue, Oct 12, 2004 at 12:24:44PM +0200, Oliver Neukum was heard to remark:
>  
> > With *nix, most data only gets written at unmount, so the only way this 
> > can "sanely" work is for mounts you haven't written to.  That case is of 
> 
> This is not a law of nature. You can mount sync as well. That, of course,
> sucks in terms of performance and wear. A reasonable compromise
> would be to do sync on close.
> Supermount did this years ago.

As a practical matter, sync-on-file-close should solve most 
of the practical problem of data corruption if the device is 
yanked before being onmounted.  However, when I read 
'man 2 open' there is no O_SYNC_ON_CLOSE.  

Similarly 'man 8 mount' doesn't list any option -o synconclose

It sure would be nice to be able to set up a sync-on-file-close
in the hotplug equiv of /etc/fstab for USB devices ... 

(When I think of sync-on-file-close, I don't mean 'global sync 
when the file is closed', I mean 'sync only that file's data and 
metadata only when the file is closed'.  That way, you don't slow 
down systems doing a lot of i/o on other, unrelated files)

--linas