Stopped disk causes LVM to go crazy

Stopped disk causes LVM to go crazy

[Alireza Nematollahi]

[-- Attachment #1.1: Type: text/plain, Size: 9374 bytes --]

hi guys, 

I have a system with two SATA disks and I have LVM on the disks. 
I use sdparm (sdparm -C stop /dev/sdb) to spin-down one of them. It works fine and I can hear the disk spinning down and all...But when I run any LVM command, say lvscan or whatever, I get lots of error messages: 


	> lvscan 
end_request: I/O error, dev sdb, sector 20971839 
end_request: I/O error, dev sdb, sector 447 
/dev/dm-0: read failed after 0 of 4096 at 10737352704: Input/outend_request: I/O error, dev sdb, sector 21176639 
put error 
/dev/dm-0: read failed after 0 of 4096 at 0: Input/end_request: I/O error, dev sdb, sector 20971967 
output error 
/dev/dm-1: read failed after 0 of 4096 at 104792end_request: I/O error, dev sdb, sector 42148159 
064: Input/output error 
/dev/dm-1: read failed after 0 of 409end_request: I/O error, dev sdb, sector 21176767 
6 at 0: Input/output error 
/dev/dm-2: read failed after 0 of end_request: I/O error, dev sdb, sector 42352959 
4096 at 10737352704: Input/output error 
/dev/dm-2: read faileend_request: I/O error, dev sdb, sector 42148287 
d after 0 of 4096 at 0: Input/output error 
/dev/dm-3: read faend_request: I/O error, dev sdb, sector 63324479 
iled after 0 of 4096 at 104792064: Input/output error 
/dev/dmend_request: I/O error, dev sdb, sector 42353087 
-3: read failed after 0 of 4096 at 0: Input/output error 
/devend_request: I/O error, dev sdb, sector 63529279 
/dm-4: read failed after 0 of 4096 at 10737352704: Input/output end_request: I/O error, dev sdb, sector 63324607 
error 
/dev/dm-4: read failed after 0 of 4096 at 0: Input/outpend_request: I/O error, dev sdb, sector 287274137 
ut error 
/dev/dm-5: read failed after 0 of 4096 at 104792064:end_request: I/O error, dev sdb, sector 287258265 
Input/output error 
/dev/dm-5: read failed after 0 of 4096 atend_request: I/O error, dev sdb, sector 287258029 
0: Input/output error 
/dev/sdb1: read failed after 0 of 512 end_request: I/O error, dev sdb, sector 281410605 
at 8126464: Input/output error 
/dev/sdb1: read failed after 0end_request: I/O error, dev sdb, sector 281410367 
of 2048 at 0: Input/output error 
/dev/sdb2: read failed afteend_request: I/O error, dev sdb, sector 63 
r 0 of 4096 at 2993881088: Input/output error 
/dev/sdb2: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb4: read failed after 0 of 1024 at 144082075648: Input/output error 
/dev/sdb4: read failed after 0 of 2048 at 0: Input/output error 
ACTIVE '/dev/blahblah/blah' [10.00 GB] inherit 
ACTIVE '/dev/blahblah/blah' [100.00 MB] inherit 
ACTIVE '/dev/blahblah/blah' [10.00 GB] inherit 
ACTIVE '/dev/blahblah/blah' [100.00 MB] inherit 
ACTIVE '/dev/blahblah/blah' [10.00 GB] inherit 
ACTIVE '/dev/blahblah/blah' [100.00 MB] inherit 
File descriptor 3 left open 
end_request: I/O error, dev sdb, sector 20971839 
end_request: I/O error, dev sdb, sector 447 
/dev/dm-0: read failed after 0 of 4096 at 10737352704: Input/outend_request: I/O error, dev sdb, sector 21176639 
put error 
/dev/dm-0: read failed after 0 of 4096 at 0: Input/end_request: I/O error, dev sdb, sector 20971967 
output error 
/dev/dm-1: read failed after 0 of 4096 at 104792end_request: I/O error, dev sdb, sector 42148159 
064: Input/output error 
/dev/dm-1: read failed after 0 of 409end_request: I/O error, dev sdb, sector 21176767 
6 at 0: Input/output error 
/dev/dm-2: read failed after 0 of end_request: I/O error, dev sdb, sector 42352959 
4096 at 10737352704: Input/output error 
/dev/dm-2: read faileend_request: I/O error, dev sdb, sector 42148287 
d after 0 of 4096 at 0: Input/output error 
/dev/dm-3: read faend_request: I/O error, dev sdb, sector 63324479 
iled after 0 of 4096 at 104792064: Input/output error 
/dev/dmend_request: I/O error, dev sdb, sector 42353087 
-3: read failed after 0 of 4096 at 0: Input/output error 
/devend_request: I/O error, dev sdb, sector 63529279 
/dm-4: read failed after 0 of 4096 at 10737352704: Input/output end_request: I/O error, dev sdb, sector 63324607 
error 
/dev/dm-4: read failed after 0 of 4096 at 0: Input/outpend_request: I/O error, dev sdb, sector 287274137 
ut error 
/dev/dm-5: read failed after 0 of 4096 at 104792064:end_request: I/O error, dev sdb, sector 287258265 
Input/output error 
/dev/dm-5: read failed after 0 of 4096 atend_request: I/O error, dev sdb, sector 287258029 
0: Input/output error 
/dev/sdb1: read failed after 0 of 512 end_request: I/O error, dev sdb, sector 281410605 
at 8126464: Input/output error 
/dev/sdb1: read failed after 0end_request: I/O error, dev sdb, sector 281410367 
of 2048 at 0: Input/output error 
/dev/sdb2: read failed afteend_request: I/O error, dev sdb, sector 63 
r 0 of 4096 at 2993881088: Input/output error 
/dev/sdb2: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb4: read failed after 0 of 1024 at 144082075648: Input/output error 
/dev/sdb4: read failed after 0 of 2048 at 0: Input/output erroend_request: I/O error, dev sdb, sector 20971839 
r 
end_request: I/O error, dev sdb, sector 4474zwPa" now active 

/dev/dm-0: read failed after 0 of 4096 at 10737352704: Input/end_request: I/O error, dev sdb, sector 21176639 
output error 
/dev/dm-0: read failed after 0 of 4096 at 0: Inpend_request: I/O error, dev sdb, sector 20971967 
ut/output error 
/dev/dm-1: read failed after 0 of 4096 at 104end_request: I/O error, dev sdb, sector 42148159 
792064: Input/output error 
/dev/dm-1: read failed after 0 of end_request: I/O error, dev sdb, sector 21176767 
4096 at 0: Input/output error 
/dev/dm-2: read failed after 0 end_request: I/O error, dev sdb, sector 42352959 
of 4096 at 10737352704: Input/output error 
/dev/dm-2: read faend_request: I/O error, dev sdb, sector 42148287 
iled after 0 of 4096 at 0: Input/output error 
/dev/dm-3: readend_request: I/O error, dev sdb, sector 63324479 
failed after 0 of 4096 at 104792064: Input/output error 
/devend_request: I/O error, dev sdb, sector 42353087 
/dm-3: read failed after 0 of 4096 at 0: Input/output error 
/end_request: I/O error, dev sdb, sector 63529279 
dev/dm-4: read failed after 0 of 4096 at 10737352704: Input/outpend_request: I/O error, dev sdb, sector 63324607 
ut error 
/dev/dm-4: read failed after 0 of 4096 at 0: Input/oend_request: I/O error, dev sdb, sector 287274137 
utput error 
/dev/dm-5: read failed after 0 of 4096 at 1047920end_request: I/O error, dev sdb, sector 287258265 
64: Input/output error 
/dev/dm-5: read failed after 0 of 4096end_request: I/O error, dev sdb, sector 287258029 
at 0: Input/output error 
/dev/sdb1: read failed after 0 of 5end_request: I/O error, dev sdb, sector 281410605 
12 at 8126464: Input/output error 
/dev/sdb1: read failed afteend_request: I/O error, dev sdb, sector 281410367 
r 0 of 2048 at 0: Input/output error 
/dev/sdb2: read failed aend_request: I/O error, dev sdb, sector 63 
fter 0 of 4096 at 2993881088: Input/output error 
/dev/sdb2: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb4end_request: I/O error, dev sdb, sector 20971839 
: read failed after 0 of 1024 at 144082075648: Input/output erroend_request: I/O error, dev sdb, sector 447 
r 
/dev/sdb4: read failed after 0 of 2048 at 0: Input/output eend_request: I/O error, dev sdb, sector 21176639 
rror 
/dev/dm-0: read failed after 0 of 4096 at 10737352704: Iend_request: I/O error, dev sdb, sector 20971967 
nput/output error 
/dev/dm-0: read failed after 0 of 4096 at 0end_request: I/O error, dev sdb, sector 42148159 
: Input/output error 
/dev/dm-1: read failed after 0 of 4096 aend_request: I/O error, dev sdb, sector 21176767 
t 104792064: Input/output error 
/dev/dm-1: read failed after end_request: I/O error, dev sdb, sector 42352959 
0 of 4096 at 0: Input/output error 
/dev/dm-2: read failed aftend_request: I/O error, dev sdb, sector 42148287 
er 0 of 4096 at 10737352704: Input/output error 
/dev/dm-2: reend_request: I/O error, dev sdb, sector 63324479 
ad failed after 0 of 4096 at 0: Input/output error 
/dev/dm-3:end_request: I/O error, dev sdb, sector 42353087 
read failed after 0 of 4096 at 104792064: Input/output error 
end_request: I/O error, dev sdb, sector 63529279 
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output errorend_request: I/O error, dev sdb, sector 63324607 

/dev/dm-4: read failed after 0 of 4096 at 10737352704: Inputend_request: I/O error, dev sdb, sector 287274137 
/output error 
/dev/dm-4: read failed after 0 of 4096 at 0: Inend_request: I/O error, dev sdb, sector 287258265 
put/output error 
/dev/dm-5: read failed after 0 of 4096 at 10end_request: I/O error, dev sdb, sector 287258029 
4792064: Input/output error 
/dev/dm-5: read failed after 0 ofend_request: I/O error, dev sdb, sector 281410605 
4096 at 0: Input/output error 
/dev/sdb1: read failed after 0end_request: I/O error, dev sdb, sector 281410367 
of 512 at 8126464: Input/output error 
/dev/sdb1: read failedend_request: I/O error, dev sdb, sector 63 
after 0 of 2048 at 0: Input/output error 
/dev/sdb2: read failed after 0 of 4096 at 2993881088: Input/output error 
/dev/sdb2: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb4: read failed after 0 of 1024 at 144082075648: Input/output error 
/dev/sdb4: read failed after 0 of 2048 at 0: Input/output error Even when I physically remove the disk it still happens. Is there any way I can stop LVM from querying both disks or at least to stop displaying the error messages? 

Thanks 

[-- Attachment #1.2: Type: text/html, Size: 10480 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Stopped disk causes LVM to go crazy

[Bryn M. Reeves]

Alireza Nematollahi wrote:
> hi guys,
> 
> I have a system with two SATA disks and I have LVM on the disks. I
> use sdparm (sdparm -C stop /dev/sdb) to spin-down one of them. It
> works fine and I can hear the disk spinning down and all...But when
> I run any LVM command, say lvscan or whatever, I get lots of error
> messages:

What did you expect? :)

If the volume group is active when you remove/spin down the disks LVM 
has no way of knowing that you wanted to do that. To the software, it 
just looks like a drive has failed or gone away while it was in use.

<snip>

> /dev/sdb2: read failed after 0 of 4096 at 2993881088: Input/output
> error /dev/sdb2: read failed after 0 of 4096 at 0: Input/output
> error /dev/sdb4: read failed after 0 of 1024 at 144082075648:
> Input/output error /dev/sdb4: read failed after 0 of 2048 at 0:

> Input/output error Even when I physically remove the disk it still
> happens. Is there any way I can stop LVM from querying both disks
> or at least to stop displaying the error messages?

Tell LVM to stop using the disk first. See the man page for vgchange 
for full details but you want to run:

vgchange -an <volume group>

To deactivate the volume group *before* you start removing devices 
that it is using.

Regards,
Bryn.

Re: Stopped disk causes LVM to go crazy

[Nuno Fernandes]

[-- Attachment #1.1: Type: text/plain, Size: 2185 bytes --]

On Wednesday 11 February 2009 20:04:46 Alireza Nematollahi wrote:
> Hello,
>
> I think I did it in order:
> 1- nothing is mounted
>
> > mount
>
> rootfs on / type rootfs (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw)
> tmpfs on /dev/shm type tmpfs (rw)
> rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
>
> 2- deactivating VG/LVs
>
> > vgchange -an disk2
>
> 3- stop the storage
>
> > sdparm -C stop /dev/sdb
>
> 4- checking for LVs on other disk
>
> > lvscan
>
> Isn't it the right order? I'm new to LVM and that's why I may be
> confused...
>
> I don't like doing "echo 1 > /sys/block/sdb/device/delete" since I can't
> get the device restarted when I do sdparm -C start /dev/sdb...
>
> Thanks
>
>
>
> ----- Original Message -----
> From: "Bryn M. Reeves" <bmr@redhat.com>
> To: "Alireza Nematollahi" <alirezan@sfu.ca>
> Cc: "device-mapper development" <dm-devel@redhat.com>
> Sent: Wednesday, February 11, 2009 10:50:45 AM GMT -08:00 US/Canada Pacific
> Subject: Re: [dm-devel] Stopped disk causes LVM to go crazy
>
> Alireza Nematollahi wrote:
> > hi. Thanks for your reply. Well, I had tried that before, but no luck. I
> > tried it again, and still no luck:
>
> You have to do it *BEFORE* you disable/remove the storage. Doing it
> afterwards will just throw all the errors you are seeing since the LVM
> tools can no longer access the device.
>
> You have to do things in the correct order; first unmount any file
> systems, then de-activate LVs/VGs, then stop or remove the storage
> hardware.
>
> If you do this in the wrong order you'll end up with the problems
> you're seeing.
>
> For now, you can clean up /dev/sdb by deleting it via sysfs:
>
> echo 1 > /sys/block/sdb/device/delete
>
> Which will stop other commands that scan the disks from tripping over
> the dead device. You can also manually delete any remaining device
> mapper devices that reference the removed disk (see similar thread
> yesterday).

Add /dev/sdb to the filter so that it doesn't get scanned...

lvm.conf:
    filter = [ "r/sdb/", "a/.*/" ]

Best regards,
./npf


>
> Regards,
> Bryn.



[-- Attachment #1.2: Type: text/html, Size: 14638 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Stopped disk causes LVM to go crazy

[Alireza Nematollahi]

[-- Attachment #1.1: Type: text/plain, Size: 1908 bytes --]

Hello, 

I think I did it in order: 
1- nothing is mounted 
> mount 
rootfs on / type rootfs (rw) 
proc on /proc type proc (rw) 
sysfs on /sys type sysfs (rw) 
devpts on /dev/pts type devpts (rw) 
tmpfs on /dev/shm type tmpfs (rw) 
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 
nfsd on /proc/fs/nfsd type nfsd (rw) 

2- deactivating VG/LVs 
> vgchange -an disk2 

3- stop the storage 
> sdparm -C stop /dev/sdb 

4- checking for LVs on other disk 
> lvscan 


Isn't it the right order? I'm new to LVM and that's why I may be confused... 

I don't like doing "echo 1 > /sys/block/sdb/device/delete" since I can't get the device restarted when I do sdparm -C start /dev/sdb... 

Thanks 



----- Original Message ----- 
From: "Bryn M. Reeves" <bmr@redhat.com> 
To: "Alireza Nematollahi" <alirezan@sfu.ca> 
Cc: "device-mapper development" <dm-devel@redhat.com> 
Sent: Wednesday, February 11, 2009 10:50:45 AM GMT -08:00 US/Canada Pacific 
Subject: Re: [dm-devel] Stopped disk causes LVM to go crazy 

Alireza Nematollahi wrote: 
> hi. Thanks for your reply. Well, I had tried that before, but no luck. I tried it again, and still no luck: 

You have to do it *BEFORE* you disable/remove the storage. Doing it 
afterwards will just throw all the errors you are seeing since the LVM 
tools can no longer access the device. 

You have to do things in the correct order; first unmount any file 
systems, then de-activate LVs/VGs, then stop or remove the storage 
hardware. 

If you do this in the wrong order you'll end up with the problems 
you're seeing. 

For now, you can clean up /dev/sdb by deleting it via sysfs: 

echo 1 > /sys/block/sdb/device/delete 

Which will stop other commands that scan the disks from tripping over 
the dead device. You can also manually delete any remaining device 
mapper devices that reference the removed disk (see similar thread 
yesterday). 

Regards, 
Bryn. 


[-- Attachment #1.2: Type: text/html, Size: 2271 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Stopped disk causes LVM to go crazy

[Bryn M. Reeves]

Alireza Nematollahi wrote:
> hi. Thanks for your reply. Well, I had tried that before, but no luck. I tried it again, and still no luck: 

You have to do it *BEFORE* you disable/remove the storage. Doing it 
afterwards will just throw all the errors you are seeing since the LVM 
tools can no longer access the device.

You have to do things in the correct order; first unmount any file 
systems, then de-activate LVs/VGs, then stop or remove the storage 
hardware.

If you do this in the wrong order you'll end up with the problems 
you're seeing.

For now, you can clean up /dev/sdb by deleting it via sysfs:

echo 1 > /sys/block/sdb/device/delete

Which will stop other commands that scan the disks from tripping over 
the dead device. You can also manually delete any remaining device 
mapper devices that reference the removed disk (see similar thread 
yesterday).

Regards,
Bryn.

Re: Stopped disk causes LVM to go crazy

[Alireza Nematollahi]

[-- Attachment #1.1: Type: text/plain, Size: 3701 bytes --]

hi. Thanks for your reply. Well, I had tried that before, but no luck. I tried it again, and still no luck: 
> vgchange -an disk2 
0 logical volume(s) in volume group "disk2" now active 
> sdparm -C stop /dev/sdb 
> lvscan 
end_request: I/O error, dev sdb, sector 0 
printk: 72 messages suppressed. 
Buffer I/O error on device sdb, logical block 0 
Buffer I/O error on device sdb, logical block 1 
Buffer I/O error on device sdb, logical block 2 
Buffer I/O error on device sdb, logical block 3 
end_request: I/O error, dev sdb, sector 0 
Buffer I/O error on device sdb, logical block 0 
end_request: I/O error, dev sdb, sector 287277824 
/dev/sdb: read failed after 0 of 4096 at 0: Input/output error 
end_request: I/O error, dev sdb, sector 0 
/dev/sdb: read failed after 0 of 4096 at 147086245888: Input/oend_request: I/O error, dev sdb, sector 287274137 
utput error 
/dev/sdb: read failed after 0 of 4096 at 0: Inputend_request: I/O error, dev sdb, sector 287258265 
/output error 
/dev/sdb1: read failed after 0 of 512 at 812646end_request: I/O error, dev sdb, sector 287258029 
4: Input/output error 
/dev/sdb1: read failed after 0 of 2048 end_request: I/O error, dev sdb, sector 281410605 
at 0: Input/output error 
/dev/sdb2: read failed after 0 of 20end_request: I/O error, dev sdb, sector 281410367 
48 at 2993881088: Input/output error 
/dev/sdb2: read failed aend_request: I/O error, dev sdb, sector 63 
fter 0 of 2048 at 0: Input/output error 
/dev/sdb4: read failed after 0 of 1024 at 144082075648: Input/output error 
/dev/sdb4: read failed after 0 of 2048 at 0: Input/output error 
ACTIVE '/dev/disk1/blah' [10.00 GB] inherit 
ACTIVE '/dev/disk1/blah' [100.00 MB] inherit 
ACTIVE '/dev/disk1/blah' [10.00 GB] inherit 
ACTIVE '/dev/disk1/blah' [100.00 MB] inherit 
ACTIVE '/dev/disk1/blah' [10.00 GB] inherit 
ACTIVE '/dev/disk1/blah' [100.00 MB] inherit 

I'm using older LVM version: 
> lvscan --version 
LVM version: 2.02.25 (2007-04 27) 
Library version: 1.02.19 (2007-04-27) 
Driver version: 4.11.0 

Any ideas? 

Thanks 

----- Original Message ----- 
From: "Bryn M. Reeves" <bmr@redhat.com> 
To: "device-mapper development" <dm-devel@redhat.com> 
Sent: Wednesday, February 11, 2009 2:39:20 AM GMT -08:00 US/Canada Pacific 
Subject: Re: [dm-devel] Stopped disk causes LVM to go crazy 

Alireza Nematollahi wrote: 
> hi guys, 
> 
> I have a system with two SATA disks and I have LVM on the disks. I 
> use sdparm (sdparm -C stop /dev/sdb) to spin-down one of them. It 
> works fine and I can hear the disk spinning down and all...But when 
> I run any LVM command, say lvscan or whatever, I get lots of error 
> messages: 

What did you expect? :) 

If the volume group is active when you remove/spin down the disks LVM 
has no way of knowing that you wanted to do that. To the software, it 
just looks like a drive has failed or gone away while it was in use. 

<snip> 

> /dev/sdb2: read failed after 0 of 4096 at 2993881088: Input/output 
> error /dev/sdb2: read failed after 0 of 4096 at 0: Input/output 
> error /dev/sdb4: read failed after 0 of 1024 at 144082075648: 
> Input/output error /dev/sdb4: read failed after 0 of 2048 at 0: 

> Input/output error Even when I physically remove the disk it still 
> happens. Is there any way I can stop LVM from querying both disks 
> or at least to stop displaying the error messages? 

Tell LVM to stop using the disk first. See the man page for vgchange 
for full details but you want to run: 

vgchange -an <volume group> 

To deactivate the volume group *before* you start removing devices 
that it is using. 

Regards, 
Bryn. 

-- 
dm-devel mailing list 
dm-devel@redhat.com 
https://www.redhat.com/mailman/listinfo/dm-devel 

[-- Attachment #1.2: Type: text/html, Size: 4713 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]