Device removal handling

Device removal handling

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am trying to understand how device removal is supposed to be handled
and make sure that it works correctly.  So far, what I see is discouraging.

I began by trying to remove the msdos partition table code from dmraid
and leave partition handling up to kpartx.  I realized that this is not
so simple because in order to deactivate the raid array, the partitions
must first be deactivated, which will not be possible if dmraid does not
know about them.  It also seems that dmsetup fails to remove a dm device
that is in use.

I then tried forcing the removal of an underlying physical disk by
writing a 1 to /sys/block/sda/device/remove.  This apparently removed
the disk, but left it in a dangling state since it was apparently still
being referenced by the dm device, which seems to have no notion that
the underlying disk has been removed.  At this point the /sys entry has
gone, so it is no longer even possible to have udev look up the holders
of the device so they can be removed.  This prevents the disk from
reusing the same identifier when it is rescanned.

Shouldn't it be possible to remove an underlying dm-device that is still
in use like you can with the physical disk, and shouldn't such a removal
trigger some kind of notification to the dm-device stacked on top, so
that it can release it?

Also, why do the /sys entries disappear before the device is released?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4aB5UACgkQJ4UciIs+XuLQYACbB3YCsYoEyfUm5aLITa2wajSp
M8AAoJ5e1sqflyJfwk38A8aZabmb+wkW
=44S4
-----END PGP SIGNATURE-----

Re: Device removal handling

[Alasdair G Kergon]

If you want to remove a dm device forcibly, replace it with an 'error' target
that will return I/O errors.  (dmsetup remove -f)

(But devices should be removed from the top down, obviously, and if
it's your main filesystem, you could e.g. set up a ramdisk do do
this from.)

Alasdair

Re: Device removal handling

[Phillip Susi]

On 7/10/2011 6:03 PM, Alasdair G Kergon wrote:
> (But devices should be removed from the top down, obviously, and if
> it's your main filesystem, you could e.g. set up a ramdisk do do
> this from.)

I disagree.  Right now dmraid relies on doing this, but this is not 
possible if it is not aware of the higher level mappings because they 
were created by kpartx.  LVM and multipath will have the same problem if 
kpartx recognizes partitions contained within an LV.

Hardware removal also necessitates bottom up removal, since the initial 
event comes from unplugging the hardware at the bottom of the stack. 
The system needs to propagate notification of that up the stack so that 
the higher layers can react appropriately and release the removed lower 
layer device.

Re: Device removal handling

[Alasdair G Kergon]

On Mon, Jul 11, 2011 at 11:08:05AM -0400, Phillip Susi wrote:
> Hardware removal also necessitates bottom up removal, since the initial  

That's back-to-front!  You stop using a device first, then you remove it.
You really don't want to be removing devices while they are in use if you
can avoid it.

If something goes wrong and a device disappears, then yes, that disappearance
should propagate up the stack to be handled as best it can by each layer.

Alasdair

Re: Device removal handling

[Phillip Susi]

On 7/11/2011 11:23 AM, Alasdair G Kergon wrote:
> That's back-to-front!  You stop using a device first, then you remove it.
> You really don't want to be removing devices while they are in use if you
> can avoid it.

That is a pre plug and play world view.  These days the kernel needs to 
be able to handle surprise removal as well.  Also when you do want to 
remove a device, polluting user space tools with all kinds of hacks to 
try and figure out what the higher layers are and clean them up is very 
error prone and often is not done correctly.  It would be preferable to 
use the same mechanism to request the removal of a device, and let the 
kernel worry about notifying any higher layers to clean up.

> If something goes wrong and a device disappears, then yes, that disappearance
> should propagate up the stack to be handled as best it can by each layer.

The question then is, how should that work?  I can't believe the block 
layer does not already have some kind of mechanism for this.  It isn't 
much different than handling medium ejection.  If it does, then dm just 
needs to use it.

Re: Device removal handling

[Hannes Reinecke]

On 07/11/2011 05:08 PM, Phillip Susi wrote:
> On 7/10/2011 6:03 PM, Alasdair G Kergon wrote:
>> (But devices should be removed from the top down, obviously, and if
>> it's your main filesystem, you could e.g. set up a ramdisk do do
>> this from.)
>
> I disagree. Right now dmraid relies on doing this, but this is not
> possible if it is not aware of the higher level mappings because
> they were created by kpartx. LVM and multipath will have the same
> problem if kpartx recognizes partitions contained within an LV.
>
> Hardware removal also necessitates bottom up removal, since the
> initial event comes from unplugging the hardware at the bottom of
> the stack. The system needs to propagate notification of that up the
> stack so that the higher layers can react appropriately and release
> the removed lower layer device.
>
Yes.

I've been facing this problem for a long time now;
eg if you're having a dm-mirror with one leg local and another one 
on iSCSI. Try shutting down iSCSI here with LVM going bonkers ...

I wouldn't mind replacing a failed device with an error target.
What I would mind is to having to code that logic into each and 
every subsystem which might find itself running underneath 
LVM/device-mapper.

What I would like to see is this logic being pushed into
eg dmsetup, so that one call to dmsetup would be sufficient to 
achieve this. And without heavy parsing of arguments etc.
Something like 'dmraid replace error' would be cool.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: Device removal handling

[Alasdair G Kergon]

On Tue, Jul 12, 2011 at 08:24:44AM +0200, Hannes Reinecke wrote:
> What I would like to see is this logic being pushed into
> eg dmsetup, so that one call to dmsetup would be sufficient to achieve 
> this. And without heavy parsing of arguments etc.
> Something like 'dmraid replace error' would be cool.

We already have dmsetup remove -f which replaced a device by the error
target then attempts to remove it (which will fail if it is still in
use).

You have a need for this without the removal attempt?

Alasdair

Re: Device removal handling

[Phillip Susi]

On 7/12/2011 6:58 AM, Alasdair G Kergon wrote:
> We already have dmsetup remove -f which replaced a device by the error
> target then attempts to remove it (which will fail if it is still in
> use).
>
> You have a need for this without the removal attempt?

The question is, how do you get udev to invoke this when the underlying 
device is removed?  It has no idea what dm device was using the disk. 
Further, if it was a mirror, then you don't want to try to kill the 
mirror if the other leg is still good, you just want the mirror to fail 
and release the removed disk.