Simulating faulty disk

Simulating faulty disk

[Yathindra]

Hi,

I'm trying to simulate a faulty disk behavior on linux. Basically, I 
want to inject various disk failure patterns
such as medium errors, unresponsive disk etc.

What is the best way to go about it. I heard about scsi_debug driver but 
it can only simulate medium
errors on fixed sector 0x1234. Also, scsi fault injection using 
Systemtap seems to be a user space tool
limited to files.

Any suggestions is greatly appreciated.

Thanks,
Yathi

Re: Simulating faulty disk

[Yathindra]

Hi Bryan,

I saw this too but it does not have features to delay the commands. And 
it works only at the file level.

I thought about scsi_debug driver but it seems to able to inject medium 
errors on sector 0x1234 only.
I'm not sure why that limitation exists though.

Thanks,
Yathi
On 10/19/2011 10:47 PM, Bryan Mesich wrote:
> On Wed, Oct 19, 2011 at 09:18:57PM -0600, Yathindra wrote:
>> Hi,
>>
>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>> want to inject various disk failure patterns
>> such as medium errors, unresponsive disk etc.
>>
>> What is the best way to go about it. I heard about scsi_debug driver but
>> it can only simulate medium
>> errors on fixed sector 0x1234. Also, scsi fault injection using
>> Systemtap seems to be a user space tool
>> limited to files.
>>
>> Any suggestions is greatly appreciated.
> I saw this go by the linux-scsi list a while back thinking it
> might be useful sometime down the road:
>
> http://lwn.net/Articles/265187/
>
> I think the project is hosted on sourceforge.net at the following
> URL:
>
> http://scsifaultinjtst.sourceforge.net/
>
> I had also read through part of a paper that was co-authored by
> the same person (from linux symposium 2008):
>
> http://www.linuxsymposium.org/archives/OLS/Reprints-2008/tanaka-reprint.pdf
>
> Doesn't look like there has been much activity since 2009, but it
> looks like it has the functionality you need.
>
> Bryan
>
>> Thanks,
>> Yathi

Re: Simulating faulty disk

[Yathindra]

Hi,

Can anybody with experience in using scsi_debug driver tell me if it can
be used to inject medium errors on various file/disk blocks.

And also, can it delay commands so we can simulate an unresponsive disk.

Thanks,
Yathi
On 10/20/2011 1:30 AM, Yathindra wrote:
> Hi Bryan,
>
> I saw this too but it does not have features to delay the commands. 
> And it works only at the file level.
>
> I thought about scsi_debug driver but it seems to able to inject 
> medium errors on sector 0x1234 only.
> I'm not sure why that limitation exists though.
>
> Thanks,
> Yathi
> On 10/19/2011 10:47 PM, Bryan Mesich wrote:
>> On Wed, Oct 19, 2011 at 09:18:57PM -0600, Yathindra wrote:
>>> Hi,
>>>
>>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>>> want to inject various disk failure patterns
>>> such as medium errors, unresponsive disk etc.
>>>
>>> What is the best way to go about it. I heard about scsi_debug driver 
>>> but
>>> it can only simulate medium
>>> errors on fixed sector 0x1234. Also, scsi fault injection using
>>> Systemtap seems to be a user space tool
>>> limited to files.
>>>
>>> Any suggestions is greatly appreciated.
>> I saw this go by the linux-scsi list a while back thinking it
>> might be useful sometime down the road:
>>
>> http://lwn.net/Articles/265187/
>>
>> I think the project is hosted on sourceforge.net at the following
>> URL:
>>
>> http://scsifaultinjtst.sourceforge.net/
>>
>> I had also read through part of a paper that was co-authored by
>> the same person (from linux symposium 2008):
>>
>> http://www.linuxsymposium.org/archives/OLS/Reprints-2008/tanaka-reprint.pdf 
>>
>>
>> Doesn't look like there has been much activity since 2009, but it
>> looks like it has the functionality you need.
>>
>> Bryan
>>
>>> Thanks,
>>> Yathi
>

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 04:18 AM, Yathindra wrote:
> I'm trying to simulate a faulty disk behavior on linux. Basically, I
> want to inject various disk failure patterns
> such as medium errors, unresponsive disk etc.

If using a device-mapper device is acceptable for your purpose there are 
the dm-flakey and dm-delay targets that can add delays, drop and corrupt 
writes and introduce intermittent I/O errors.

The targets can be stacked to give devices having any combination of 
these behaviours.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

Thanks for that info. Could you please tell me where I can find more 
details about using it.

Thanks again,
Yathi

On 10/20/2011 3:27 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:18 AM, Yathindra wrote:
>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>> want to inject various disk failure patterns
>> such as medium errors, unresponsive disk etc.
>
> If using a device-mapper device is acceptable for your purpose there 
> are the dm-flakey and dm-delay targets that can add delays, drop and 
> corrupt writes and introduce intermittent I/O errors.
>
> The targets can be stacked to give devices having any combination of 
> these behaviours.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 04:45 PM, Yathindra wrote:
> Hi Bryn,
>
> Thanks for that info. Could you please tell me where I can find more
> details about using it.
>
> Thanks again,
> Yathi

There's a couple of brief files describing the targets in the 
Documentation/device-mapper directory:

http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
http://www.kernel.org/doc/Documentation/device-mapper/delay.txt

They both work like the linear target (except that they introduce the 
specified faults according to their parameters).

To set up a device with both you'd need to use a pair of stacked devices 
so something like:

dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"

This will create a device /dev/mapper/f0 that is drops out for 1s in 
every 10 and has a 500ms delay on reads and writes. SECTORS is the size 
of the underlying device in sectors (use e.g. $(blockdev --getsize)).

You can optionally specify separate delays/device for writes to the 
delay layer and write loss / write corruption for the flakey layer - see 
the doc files and also the comments above flakey_ctr and delay_ctr for 
more details.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Thanks so much Bryn. Appreciate it.

On 10/20/2011 9:56 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:45 PM, Yathindra wrote:
>> Hi Bryn,
>>
>> Thanks for that info. Could you please tell me where I can find more
>> details about using it.
>>
>> Thanks again,
>> Yathi
>
> There's a couple of brief files describing the targets in the 
> Documentation/device-mapper directory:
>
> http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
> http://www.kernel.org/doc/Documentation/device-mapper/delay.txt
>
> They both work like the linear target (except that they introduce the 
> specified faults according to their parameters).
>
> To set up a device with both you'd need to use a pair of stacked 
> devices so something like:
>
> dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
> dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"
>
> This will create a device /dev/mapper/f0 that is drops out for 1s in 
> every 10 and has a 500ms delay on reads and writes. SECTORS is the 
> size of the underlying device in sectors (use e.g. $(blockdev 
> --getsize)).
>
> You can optionally specify separate delays/device for writes to the 
> delay layer and write loss / write corruption for the flakey layer - 
> see the doc files and also the comments above flakey_ctr and delay_ctr 
> for more details.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

Sorry to disturb you again. I tried to create a delay device below but 
it failed.

 >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
device-mapper: reload ioctl failed: Invalid argument
Command failed

In /var/log/messages I see this,
  device-mapper: ioctl: error adding target to table

 >uname -r
2.6.32-24-generic-pae

 >dmsetup --version
Library version:   1.02.39 (2009-10-26)
Driver version:    4.15.0

Am I doing something wrong.

Thanks,
Yathi

On 10/20/2011 9:56 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:45 PM, Yathindra wrote:
>> Hi Bryn,
>>
>> Thanks for that info. Could you please tell me where I can find more
>> details about using it.
>>
>> Thanks again,
>> Yathi
>
> There's a couple of brief files describing the targets in the 
> Documentation/device-mapper directory:
>
> http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
> http://www.kernel.org/doc/Documentation/device-mapper/delay.txt
>
> They both work like the linear target (except that they introduce the 
> specified faults according to their parameters).
>
> To set up a device with both you'd need to use a pair of stacked 
> devices so something like:
>
> dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
> dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"
>
> This will create a device /dev/mapper/f0 that is drops out for 1s in 
> every 10 and has a 500ms delay on reads and writes. SECTORS is the 
> size of the underlying device in sectors (use e.g. $(blockdev 
> --getsize)).
>
> You can optionally specify separate delays/device for writes to the 
> delay layer and write loss / write corruption for the flakey layer - 
> see the doc files and also the comments above flakey_ctr and delay_ctr 
> for more details.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 05:46 PM, Yathindra wrote:
>  >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
> /dev/sdb 0 500"
> device-mapper: reload ioctl failed: Invalid argument
> Command failed
>
> In /var/log/messages I see this,
> device-mapper: ioctl: error adding target to table

I don't see this here:

# dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
# dmsetup info -c d0
Name             Maj Min Stat Open Targ Event  UUID 

d0               253   3 L--w    1    1      0
# dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
flakey /dev/mapper/d0 0 9 1"
# dmsetup info -c f0
Name             Maj Min Stat Open Targ Event  UUID 

f0               253   4 L--w    0    1      0

This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally compiled 
dm-delay since it's not enabled in the Fedora kernels.

Might want to move this thread over to dm-devel if you're having trouble 
with it.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Oh I see. Maybe dm-delay is not enabled in ubuntu either.

Could you please tell me how I could enable it.

Thanks,
Yathi

On 10/20/2011 11:40 AM, Bryn M. Reeves wrote:
> On 10/20/2011 05:46 PM, Yathindra wrote:
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>>
>> In /var/log/messages I see this,
>> device-mapper: ioctl: error adding target to table
>
> I don't see this here:
>
> # dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
> /dev/sdb 0 500"
> # dmsetup info -c d0
> Name             Maj Min Stat Open Targ Event  UUID
> d0               253   3 L--w    1    1      0
> # dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
> flakey /dev/mapper/d0 0 9 1"
> # dmsetup info -c f0
> Name             Maj Min Stat Open Targ Event  UUID
> f0               253   4 L--w    0    1      0
>
> This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally 
> compiled dm-delay since it's not enabled in the Fedora kernels.
>
> Might want to move this thread over to dm-devel if you're having 
> trouble with it.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

I compiled a fresh kernel and enabled flakey and delay modules. Now am 
able to create disks but
it fails to mount.

dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` flakey 
/dev/mapper/d0 0 9 1"
mkfs -t ext3 /dev/mapper/f0
...
...
ext2fs_mkdir: Attempt to read block from filesystem resulted in short 
read while creating root dir

Can we even mount such a device ?

Thanks,
Yathi



On 10/20/2011 11:40 AM, Bryn M. Reeves wrote:
> On 10/20/2011 05:46 PM, Yathindra wrote:
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>>
>> In /var/log/messages I see this,
>> device-mapper: ioctl: error adding target to table
>
> I don't see this here:
>
> # dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
> /dev/sdb 0 500"
> # dmsetup info -c d0
> Name             Maj Min Stat Open Targ Event  UUID
> d0               253   3 L--w    1    1      0
> # dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
> flakey /dev/mapper/d0 0 9 1"
> # dmsetup info -c f0
> Name             Maj Min Stat Open Targ Event  UUID
> f0               253   4 L--w    0    1      0
>
> This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally 
> compiled dm-delay since it's not enabled in the Fedora kernels.
>
> Might want to move this thread over to dm-devel if you're having 
> trouble with it.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/21/2011 03:47 PM, Yathindra wrote:
> dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
> /dev/sdb 0 500"
> dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` flakey
> /dev/mapper/d0 0 9 1"
> mkfs -t ext3 /dev/mapper/f0
> ...
> ...
> ext2fs_mkdir: Attempt to read block from filesystem resulted in short
> read while creating root dir
>
> Can we even mount such a device ?

It's a flakey device :)

With that table you'll have a device that is failing write I/Os for 1s 
in every ten. If that coincides with mkfs trying to write to the device 
it will create a corrupted file system.

Depending on what you are trying to test you may want to put data on the 
device first.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Ok :)  Bryn is there a more elaborate document which shows simple examples
to work with device mapper. For a newbie like me it would be really helpful.

Thanks,
Yathi

On 10/21/2011 8:56 AM, Bryn M. Reeves wrote:
> On 10/21/2011 03:47 PM, Yathindra wrote:
>> dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` flakey
>> /dev/mapper/d0 0 9 1"
>> mkfs -t ext3 /dev/mapper/f0
>> ...
>> ...
>> ext2fs_mkdir: Attempt to read block from filesystem resulted in short
>> read while creating root dir
>>
>> Can we even mount such a device ?
>
> It's a flakey device :)
>
> With that table you'll have a device that is failing write I/Os for 1s 
> in every ten. If that coincides with mkfs trying to write to the 
> device it will create a corrupted file system.
>
> Depending on what you are trying to test you may want to put data on 
> the device first.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

 >mount -t ext3 /dev/sdb /mnt
Copied some data into /mnt
 >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
device-mapper: reload ioctl failed: Invalid argument
Command failed
 >umount /mnt
 > dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
Worked.

Basically, am trying to test how regular userspace commands to 
read/write files
behave when I/O fails. For this, I need to put a file system and then 
carry on testing.

But as seen from above, dmsetup is failing to create a flakey/delay 
device when /dev/sdb
has a filesystem on top of it. Is there any workaround ?

Thanks,
Yathi

On 10/21/2011 9:00 AM, Yathindra wrote:
> Ok :)  Bryn is there a more elaborate document which shows simple 
> examples
> to work with device mapper. For a newbie like me it would be really 
> helpful.
>
> Thanks,
> Yathi
>
> On 10/21/2011 8:56 AM, Bryn M. Reeves wrote:
>> On 10/21/2011 03:47 PM, Yathindra wrote:
>>> dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>>> /dev/sdb 0 500"
>>> dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` flakey
>>> /dev/mapper/d0 0 9 1"
>>> mkfs -t ext3 /dev/mapper/f0
>>> ...
>>> ...
>>> ext2fs_mkdir: Attempt to read block from filesystem resulted in short
>>> read while creating root dir
>>>
>>> Can we even mount such a device ?
>>
>> It's a flakey device :)
>>
>> With that table you'll have a device that is failing write I/Os for 
>> 1s in every ten. If that coincides with mkfs trying to write to the 
>> device it will create a corrupted file system.
>>
>> Depending on what you are trying to test you may want to put data on 
>> the device first.
>>
>> Regards,
>> Bryn.
>

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/21/2011 04:13 PM, Yathindra wrote:
>  >mount -t ext3 /dev/sdb /mnt
> Copied some data into /mnt
>  >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
> /dev/sdb 0 500"
> device-mapper: reload ioctl failed: Invalid argument
> Command failed

You can't create a mapped device on something while it is mounted as 
device-mapper needs to open it exclusively.

> But as seen from above, dmsetup is failing to create a flakey/delay
> device when /dev/sdb
> has a filesystem on top of it. Is there any workaround ?

Set up the file system the way you want it on sdb first. Then unmount it 
and create the device-mapper devices and mount that.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Is there any way we can create a regular scsi device and then 
dynamically change its personality
to delay or flakey ?
Or
Is there a way to dynamically change the parameters of flakey/delay device.

So we can start with no errors and then we can inject some errors.

Thanks,
Yathi

On 10/21/2011 9:21 AM, Bryn M. Reeves wrote:
> On 10/21/2011 04:13 PM, Yathindra wrote:
>> >mount -t ext3 /dev/sdb /mnt
>> Copied some data into /mnt
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>
> You can't create a mapped device on something while it is mounted as 
> device-mapper needs to open it exclusively.
>
>> But as seen from above, dmsetup is failing to create a flakey/delay
>> device when /dev/sdb
>> has a filesystem on top of it. Is there any workaround ?
>
> Set up the file system the way you want it on sdb first. Then unmount 
> it and create the device-mapper devices and mount that.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

All I want to do is simulate a delay or flakey device. Mount it. And see 
how regular read/write commands
such as cat, echo, vim etc behave.

*  I created a delay device d0 and mounted it on /mnt
*  Copied few files into /mnt
*  Invoked dd command on /dev/mapper/d0 and it was slow
*  Invoked cat command on some file in /mnt but it was not slow

What is wrong here?

Steps I followed are shown below:
==========================================================
root@nodea:/users/ydev# dmsetup remove_all

#Created a delay device d0
root@nodea:/users/ydev# dmsetup create d0 --table="0 `blockdev --getsize 
/dev/sdb` delay /dev/sdb 0 500"

#Put a filesystem on top of d0
root@nodea:/users/ydev# mkfs -t ext3 /dev/mapper/d0
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
....
....
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
....

#Mounted the device-mapper device on /mnt.
root@nodea:/users/ydev# mount -t ext3 /dev/mapper/d0 /mnt

#Copied some arbitrary files into /mnt
root@nodea:/users/ydev# cp -r stap_tests /mnt

#Invoked a dd command to check if it is slowing down and it did as seen 
below.

root@nodea:/# dd if=/dev/mapper/d0 of=/dev/null bs=400k count=1
1+0 records in
1+0 records out
409600 bytes (410 kB) copied, 1.01523 s, 403 kB/s

#But it did not slow down cat command as seen below.

root@nodea:/mnt/stap_tests# sysctl vm.drop_caches=3
vm.drop_caches = 3

root@nodea:/mnt/stap_tests# time cat 12M > /dev/null

real    0m0.034s
user    0m0.000s
sys     0m0.004s

==========================================================

Thanks,
Yathi



On 10/21/2011 9:21 AM, Bryn M. Reeves wrote:
> On 10/21/2011 04:13 PM, Yathindra wrote:
>> >mount -t ext3 /dev/sdb /mnt
>> Copied some data into /mnt
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>
> You can't create a mapped device on something while it is mounted as 
> device-mapper needs to open it exclusively.
>
>> But as seen from above, dmsetup is failing to create a flakey/delay
>> device when /dev/sdb
>> has a filesystem on top of it. Is there any workaround ?
>
> Set up the file system the way you want it on sdb first. Then unmount 
> it and create the device-mapper devices and mount that.
>
> Regards,
> Bryn.