error injection

error injection

[Jojy Varghese]

Hi
 I am trying to dynamically add error injection to my virtual
disk(LVM) for testing+ debugging purpose. I saw "faulty" personality
module in the kernel and was wondering if there was any documentation
on its usage. I am not looking to set up a RAID but a simple mapped
device. So the basic use case is that I need to be able to dynamically
add/remove error sectors and also be able to have granular error
configuration like read error, read+write error etc.

thanks in advance
Jojy

Re: error injection

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4287 bytes --]

On Wed, 28 Sep 2011 12:48:37 -0700 Jojy Varghese <jojy.varghese@gmail.com>
wrote:

> Hi
>  I am trying to dynamically add error injection to my virtual
> disk(LVM) for testing+ debugging purpose. I saw "faulty" personality
> module in the kernel and was wondering if there was any documentation
> on its usage. I am not looking to set up a RAID but a simple mapped
> device. So the basic use case is that I need to be able to dynamically
> add/remove error sectors and also be able to have granular error
> configuration like read error, read+write error etc.
> 
> thanks in advance
> Jojy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

The 'faulty' md personality is described briefly in the 'md.4' man page which
is included in the mdadm distribution.
I've included the relevant part below.

Configuring the type of faults is described in mdadm.8 under the '-p
--layout=' section.  So can adjust the settings using mdadm --grow.
so:
  mdadm -B /dev/md0 -l faulty -n1 /dev/sda

will build a 'faulty' device which provides access to /dev/sda, but
introduces faults.  Initially no faults will be introduces.

  mdadm -G /dev/md0 --layout=rt400

will tell md0 to generate a read error every 400 requests, but not to
remember the error - rt == readtransient
   --layout=rp400
will create a persistent error every 400 reads subsequent reads of the same
block will produce the same error.  at most 50 persistent errors can be
recorded.
  mdadm -G /dev/md0 --layout=clear
will stop producing new errors
  mdadm -G /dev/md0 --layout=flush
will forget all persistent errors.


from md.4:

   FAULTY
       The FAULTY md module is provided for testing purposes.  A faulty  array
       has  exactly  one  component device and is normally assembled without a
       superblock, so the md array created provides direct access  to  all  of
       the data in the component device.

       The  FAULTY module may be requested to simulate faults to allow testing
       of other md levels or of filesystems.  Faults can be chosen to  trigger
       on  read requests or write requests, and can be transient (a subsequent
       read/write at the address will probably succeed) or persistent  (subse-
       quent  read/write of the same address will fail).  Further, read faults
       can be "fixable" meaning that they persist until a write request at the
       same address.

       Fault  types  can  be requested with a period.  In this case, the fault
       will recur repeatedly after the given number of requests of  the  rele-
       vant type.  For example if persistent read faults have a period of 100,
       then every 100th read request would generate a fault,  and  the  faulty
       sector  would be recorded so that subsequent reads on that sector would
       also fail.

       There is a limit to the number of faulty sectors that  are  remembered.
       Faults  generated  after  this  limit is exhausted are treated as tran-
       sient.

       The list of faulty sectors can be flushed, and the active list of fail-
       ure modes can be cleared.


from mdadm.8:

              When setting the failure mode for level faulty, the options are:
              write-transient, wt, read-transient, rt,  write-persistent,  wp,
              read-persistent,  rp, write-all, read-fixable, rf, clear, flush,
              none.

              Each failure mode can be followed by a number, which is used  as
              a  period between fault generation.  Without a number, the fault
              is generated once on the first relevant request.  With a number,
              the  fault  will be generated after that many requests, and will
              continue to be generated every time the period elapses.

              Multiple failure modes can be current  simultaneously  by  using
              the --grow option to set subsequent failure modes.

              "clear"  or  "none"  will remove any pending or periodic failure
              modes, and "flush" will clear any persistent faults.



NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: error injection

[Jojy Varghese]

Thanks Neil. I tried setting my sda7 partition to generate write
errors every 40 bytes(writing 1 byte at a time). I did :

1. Create a array with:
mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7

After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get:

/dev/md127:
        Version : 1.2
  Creation Time : Wed Sep 28 17:35:50 2011
     Raid Level : faulty
     Array Size : 969410424 (924.50 GiB 992.68 GB)
   Raid Devices : 1
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Sep 28 17:35:50 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : eng-dev16.lab.local:me0  (local to host eng-dev16.lab.local)
           UUID : 96f4be10:312f9574:f40107aa:d9f278ba
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8        7        0      active sync   /dev/sda7


2. Set write fault level with:

mdadm -G /dev/md/me0 --layout=wp40



  After this when i write > 40 bytes into /dev/md127, i dont get any
I/O errors. I am sure i am doing something wrong here.


Any help is much appreciated.

Thanks
Jojy

On Wed, Sep 28, 2011 at 4:11 PM, NeilBrown <neilb@suse.de> wrote:
> On Wed, 28 Sep 2011 12:48:37 -0700 Jojy Varghese <jojy.varghese@gmail.com>
> wrote:
>
>> Hi
>>  I am trying to dynamically add error injection to my virtual
>> disk(LVM) for testing+ debugging purpose. I saw "faulty" personality
>> module in the kernel and was wondering if there was any documentation
>> on its usage. I am not looking to set up a RAID but a simple mapped
>> device. So the basic use case is that I need to be able to dynamically
>> add/remove error sectors and also be able to have granular error
>> configuration like read error, read+write error etc.
>>
>> thanks in advance
>> Jojy
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> The 'faulty' md personality is described briefly in the 'md.4' man page which
> is included in the mdadm distribution.
> I've included the relevant part below.
>
> Configuring the type of faults is described in mdadm.8 under the '-p
> --layout=' section.  So can adjust the settings using mdadm --grow.
> so:
>  mdadm -B /dev/md0 -l faulty -n1 /dev/sda
>
> will build a 'faulty' device which provides access to /dev/sda, but
> introduces faults.  Initially no faults will be introduces.
>
>  mdadm -G /dev/md0 --layout=rt400
>
> will tell md0 to generate a read error every 400 requests, but not to
> remember the error - rt == readtransient
>   --layout=rp400
> will create a persistent error every 400 reads subsequent reads of the same
> block will produce the same error.  at most 50 persistent errors can be
> recorded.
>  mdadm -G /dev/md0 --layout=clear
> will stop producing new errors
>  mdadm -G /dev/md0 --layout=flush
> will forget all persistent errors.
>
>
> from md.4:
>
>   FAULTY
>       The FAULTY md module is provided for testing purposes.  A faulty  array
>       has  exactly  one  component device and is normally assembled without a
>       superblock, so the md array created provides direct access  to  all  of
>       the data in the component device.
>
>       The  FAULTY module may be requested to simulate faults to allow testing
>       of other md levels or of filesystems.  Faults can be chosen to  trigger
>       on  read requests or write requests, and can be transient (a subsequent
>       read/write at the address will probably succeed) or persistent  (subse-
>       quent  read/write of the same address will fail).  Further, read faults
>       can be "fixable" meaning that they persist until a write request at the
>       same address.
>
>       Fault  types  can  be requested with a period.  In this case, the fault
>       will recur repeatedly after the given number of requests of  the  rele-
>       vant type.  For example if persistent read faults have a period of 100,
>       then every 100th read request would generate a fault,  and  the  faulty
>       sector  would be recorded so that subsequent reads on that sector would
>       also fail.
>
>       There is a limit to the number of faulty sectors that  are  remembered.
>       Faults  generated  after  this  limit is exhausted are treated as tran-
>       sient.
>
>       The list of faulty sectors can be flushed, and the active list of fail-
>       ure modes can be cleared.
>
>
> from mdadm.8:
>
>              When setting the failure mode for level faulty, the options are:
>              write-transient, wt, read-transient, rt,  write-persistent,  wp,
>              read-persistent,  rp, write-all, read-fixable, rf, clear, flush,
>              none.
>
>              Each failure mode can be followed by a number, which is used  as
>              a  period between fault generation.  Without a number, the fault
>              is generated once on the first relevant request.  With a number,
>              the  fault  will be generated after that many requests, and will
>              continue to be generated every time the period elapses.
>
>              Multiple failure modes can be current  simultaneously  by  using
>              the --grow option to set subsequent failure modes.
>
>              "clear"  or  "none"  will remove any pending or periodic failure
>              modes, and "flush" will clear any persistent faults.
>
>
>
> NeilBrown
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error injection

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1893 bytes --]

On Wed, 28 Sep 2011 17:59:49 -0700 Jojy Varghese <jojy.varghese@gmail.com>
wrote:

> Thanks Neil. I tried setting my sda7 partition to generate write
> errors every 40 bytes(writing 1 byte at a time). I did :

md doesn't see byte writes.  It sees sectors or more - usually whole pages or
groups of pages.

> 
> 1. Create a array with:
> mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7

-C will write a superblock to /dev/sda7 which you don't really want.  It
doesn't hurt, but I always used -B (--build) to avoid any metadata.

> 
> After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get:
> 
> /dev/md127:
>         Version : 1.2
>   Creation Time : Wed Sep 28 17:35:50 2011
>      Raid Level : faulty
>      Array Size : 969410424 (924.50 GiB 992.68 GB)
>    Raid Devices : 1
>   Total Devices : 1
>     Persistence : Superblock is persistent
> 
>     Update Time : Wed Sep 28 17:35:50 2011
>           State : clean
>  Active Devices : 1
> Working Devices : 1
>  Failed Devices : 0
>   Spare Devices : 0
> 
>            Name : eng-dev16.lab.local:me0  (local to host eng-dev16.lab.local)
>            UUID : 96f4be10:312f9574:f40107aa:d9f278ba
>          Events : 0
> 
>     Number   Major   Minor   RaidDevice State
>        0       8        7        0      active sync   /dev/sda7
> 
> 
> 2. Set write fault level with:
> 
> mdadm -G /dev/md/me0 --layout=wp40
> 
> 
> 
>   After this when i write > 40 bytes into /dev/md127, i dont get any
> I/O errors. I am sure i am doing something wrong here.

When you write to /dev/md127 it will just go into the page cache and
eventually be flushed to the device in one write.
Use O_DIRECT or O_SYNC and it will be flushed out more quickly, but always
write at least 512  bytes at a time.

NeilBrown



> 
> 
> Any help is much appreciated.
> 
> Thanks
> Jojy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: error injection

[Jojy Varghese]

Thanks Neil. Also, is there any way to find the current fault blocks being set?

On Wed, Sep 28, 2011 at 6:08 PM, NeilBrown <neilb@suse.de> wrote:
> On Wed, 28 Sep 2011 17:59:49 -0700 Jojy Varghese <jojy.varghese@gmail.com>
> wrote:
>
>> Thanks Neil. I tried setting my sda7 partition to generate write
>> errors every 40 bytes(writing 1 byte at a time). I did :
>
> md doesn't see byte writes.  It sees sectors or more - usually whole pages or
> groups of pages.
>
>>
>> 1. Create a array with:
>> mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7
>
> -C will write a superblock to /dev/sda7 which you don't really want.  It
> doesn't hurt, but I always used -B (--build) to avoid any metadata.
>
>>
>> After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get:
>>
>> /dev/md127:
>>         Version : 1.2
>>   Creation Time : Wed Sep 28 17:35:50 2011
>>      Raid Level : faulty
>>      Array Size : 969410424 (924.50 GiB 992.68 GB)
>>    Raid Devices : 1
>>   Total Devices : 1
>>     Persistence : Superblock is persistent
>>
>>     Update Time : Wed Sep 28 17:35:50 2011
>>           State : clean
>>  Active Devices : 1
>> Working Devices : 1
>>  Failed Devices : 0
>>   Spare Devices : 0
>>
>>            Name : eng-dev16.lab.local:me0  (local to host eng-dev16.lab.local)
>>            UUID : 96f4be10:312f9574:f40107aa:d9f278ba
>>          Events : 0
>>
>>     Number   Major   Minor   RaidDevice State
>>        0       8        7        0      active sync   /dev/sda7
>>
>>
>> 2. Set write fault level with:
>>
>> mdadm -G /dev/md/me0 --layout=wp40
>>
>>
>>
>>   After this when i write > 40 bytes into /dev/md127, i dont get any
>> I/O errors. I am sure i am doing something wrong here.
>
> When you write to /dev/md127 it will just go into the page cache and
> eventually be flushed to the device in one write.
> Use O_DIRECT or O_SYNC and it will be flushed out more quickly, but always
> write at least 512  bytes at a time.
>
> NeilBrown
>
>
>
>>
>>
>> Any help is much appreciated.
>>
>> Thanks
>> Jojy
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error injection

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

On Wed, 28 Sep 2011 19:06:17 -0700 Jojy Varghese <jojy.varghese@gmail.com>
wrote:

> Thanks Neil. Also, is there any way to find the current fault blocks being set?
> 

No.  All you can get is what is shown in "/proc/mdstat".

It wouldn't be too hard to add something to /proc/mdstat or /sys/.... to show
that information.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: error injection

[Zdenek Kabelac]

Dne 28.9.2011 21:32, Jojy Varghese napsal(a):
> Hi
>   I am trying to dynamically add error injection to our virtual
> disk(LVM) for testing+ debugging purpose. I saw "faulty" personality
> module in the kernel and was wondering if there was any documentation
> on its usage. I am not looking to set up a RAID but a simple mapped
> device. So the basic use case is that I need to be able to dynamically
> add/remove error sectors and also be able to have granular error
> configuration like read error, read+write error etc.
> 
> thanks in advance
> Jojy


Do you mean linux-kernel/Documentation/device-mapper/dm-flakey.txt

Or just plain  dm-error target ?
There you could easily remap your device to be build for partial error
segments (i.e. replace single linear mapping with a list of error/linear segments)

Zdenek

error injection

[Jojy Varghese]

Hi
  I am trying to dynamically add error injection to our virtual
disk(LVM) for testing+ debugging purpose. I saw "faulty" personality
module in the kernel and was wondering if there was any documentation
on its usage. I am not looking to set up a RAID but a simple mapped
device. So the basic use case is that I need to be able to dynamically
add/remove error sectors and also be able to have granular error
configuration like read error, read+write error etc.

thanks in advance
Jojy