Understanding raid array status: Active vs Clean

Understanding raid array status: Active vs Clean

[George Duffield]

I recently created a raid 5 array under Arch Linux running on a HP
Microserver using pretty much the same topography as I do under Ubuntu
Server.  The creation process went fine and the array is accessible,
however, from the outset it's only ever reported status as Active
rather than Clean.

After creating the array, watch -d cat /proc/mdstat returned:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda1[0] sdc1[2] sde1[5] sdb1[1] sdd1[3]
      11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
[5/5] [UUUUU]
      bitmap: 2/22 pages [8KB], 65536KB chunk

unused devices: <none>

which to me pretty much looks like the array sync completed successfully.

I then updated the config file, assembled the array and formatted it using:
mdadm --detail --scan >> /etc/mdadm.conf
mdadm --assemble --scan
mkfs.ext4 -v -L offsitestorage -b 4096 -E stride=128,stripe-width=512 /dev/md0

mdadm --detail /dev/md0 returns:

/dev/md0:
        Version : 1.2
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
  Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Apr 17 18:55:01 2014
          State : active
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : audioliboffsite:0  (local to host audioliboffsite)
           UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
         Events : 11306

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       5       8       65        4      active sync   /dev/sde1

So, I'm now left wondering why the state of the array isn't "clean"?
Is it normal for arrays to show a state of "active" instead of clean
under Arch - is it simply a matter of Arch is packaged with a more
recent version of mdadm than Ubuntu Server?

Thx

Re: Understanding raid array status: Active vs Clean

[George Duffield]

Anyone able to provide some insight please?

On Mon, May 26, 2014 at 10:08 PM, George Duffield
<forumscollective@gmail.com> wrote:
> I recently created a raid 5 array under Arch Linux running on a HP
> Microserver using pretty much the same topography as I do under Ubuntu
> Server.  The creation process went fine and the array is accessible,
> however, from the outset it's only ever reported status as Active
> rather than Clean.
>
> After creating the array, watch -d cat /proc/mdstat returned:
>
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sda1[0] sdc1[2] sde1[5] sdb1[1] sdd1[3]
>       11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
> [5/5] [UUUUU]
>       bitmap: 2/22 pages [8KB], 65536KB chunk
>
> unused devices: <none>
>
> which to me pretty much looks like the array sync completed successfully.
>
> I then updated the config file, assembled the array and formatted it using:
> mdadm --detail --scan >> /etc/mdadm.conf
> mdadm --assemble --scan
> mkfs.ext4 -v -L offsitestorage -b 4096 -E stride=128,stripe-width=512 /dev/md0
>
> mdadm --detail /dev/md0 returns:
>
> /dev/md0:
>         Version : 1.2
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>    Raid Devices : 5
>   Total Devices : 5
>     Persistence : Superblock is persistent
>
>   Intent Bitmap : Internal
>
>     Update Time : Thu Apr 17 18:55:01 2014
>           State : active
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>          Events : 11306
>
>     Number   Major   Minor   RaidDevice State
>        0       8        1        0      active sync   /dev/sda1
>        1       8       17        1      active sync   /dev/sdb1
>        2       8       33        2      active sync   /dev/sdc1
>        3       8       49        3      active sync   /dev/sdd1
>        5       8       65        4      active sync   /dev/sde1
>
> So, I'm now left wondering why the state of the array isn't "clean"?
> Is it normal for arrays to show a state of "active" instead of clean
> under Arch - is it simply a matter of Arch is packaged with a more
> recent version of mdadm than Ubuntu Server?
>
> Thx

Re: Understanding raid array status: Active vs Clean

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]

On Mon, 26 May 2014 22:08:40 +0200 George Duffield
<forumscollective@gmail.com> wrote:

> I recently created a raid 5 array under Arch Linux running on a HP
> Microserver using pretty much the same topography as I do under Ubuntu
> Server.  The creation process went fine and the array is accessible,
> however, from the outset it's only ever reported status as Active
> rather than Clean.
> 
> After creating the array, watch -d cat /proc/mdstat returned:
> 
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sda1[0] sdc1[2] sde1[5] sdb1[1] sdd1[3]
>       11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
> [5/5] [UUUUU]
>       bitmap: 2/22 pages [8KB], 65536KB chunk
> 
> unused devices: <none>
> 
> which to me pretty much looks like the array sync completed successfully.
> 
> I then updated the config file, assembled the array and formatted it using:
> mdadm --detail --scan >> /etc/mdadm.conf
> mdadm --assemble --scan
> mkfs.ext4 -v -L offsitestorage -b 4096 -E stride=128,stripe-width=512 /dev/md0
> 
> mdadm --detail /dev/md0 returns:
> 
> /dev/md0:
>         Version : 1.2
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>    Raid Devices : 5
>   Total Devices : 5
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Thu Apr 17 18:55:01 2014
>           State : active
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 512K
> 
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>          Events : 11306
> 
>     Number   Major   Minor   RaidDevice State
>        0       8        1        0      active sync   /dev/sda1
>        1       8       17        1      active sync   /dev/sdb1
>        2       8       33        2      active sync   /dev/sdc1
>        3       8       49        3      active sync   /dev/sdd1
>        5       8       65        4      active sync   /dev/sde1
> 
> So, I'm now left wondering why the state of the array isn't "clean"?
> Is it normal for arrays to show a state of "active" instead of clean
> under Arch - is it simply a matter of Arch is packaged with a more
> recent version of mdadm than Ubuntu Server?

I doubt there is a difference between Ubuntu and Arch here.

The array should show "active" in "mdadm --detail" output for 200ms after the
last write, and then switch to 'clean'.
So if you are writing every 100ms, it will always say "active".

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Understanding raid array status: Active vs Clean

[forumscollective]

> I doubt there is a difference between Ubuntu and Arch here.
> 
> The array should show "active" in "mdadm --detail" output for 200ms after the
> last write, and then switch to 'clean'.
> So if you are writing every 100ms, it will always say "active".
> 
> NeilBrown

For some reason this array has never shown Clean status, only Active.

It isn't written to other than when NFS mounted for adding new content.

Any idea what would cause constant writing - I presume from what I see that the initial array sync completed?

Re: Understanding raid array status: Active vs Clean

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Thu, 29 May 2014 07:52:02 +0200 forumscollective@gmail.com wrote:

> > I doubt there is a difference between Ubuntu and Arch here.
> > 
> > The array should show "active" in "mdadm --detail" output for 200ms after the
> > last write, and then switch to 'clean'.
> > So if you are writing every 100ms, it will always say "active".
> > 
> > NeilBrown
> 
> For some reason this array has never shown Clean status, only Active.
> 
> It isn't written to other than when NFS mounted for adding new content.
> 
> Any idea what would cause constant writing - I presume from what I see that the initial array sync completed?--

Hmmm...
Do the numbers in /proc/diskstats change?

  watch -d 'grep md0 /proc/diskstats'

What is in /sys/block/md0/md/safe_mode_delay?
What if you change that to a different number (it is in seconds and can be
fractional)?

What  kernel version (uname -a)?

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Understanding raid array status: Active vs Clean

[George Duffield]

Apologies for the long delay in responding - I had further issues with
Microservers trashing the first drive in the backplane, including one
of the drives for the array in question (in the case of the array it
seems the drive lost power and dropped out the array, albeit it's
fully functional now and passes SMART testing).  As a result I've
built new machines using a mini-itx motherboards and made a clean
install of Arch Linux - finished that last night, so now have the
array migrated to the new machine and powered up, albeit in degraded
mode.  I'd appreciate some advice re rebuilding this array (by adding
back the drive in question).  I've set out below pertinent info
relating to the array and hard drives in the system as well as my
intended recovery strategy.  As can be seen from lsblk, /dev/sdb1 is
the drive that is no longer recognised as being part of the array.  It
has not been written to since the incident occurred.  Is there a quick
& easy to reintegrate it into the array or is my only option to run:
# mdadm /dev/md0 --add /dev/sdb1

and let it take its course?

The machine has a 3.5Ghz i3 CPU and currently has 8GB ram installed, I
can swap out the 4GB chips and replace with 8GB chips if 16GB RAM will
significantly increase the rebuild speed.  I'd also like to speed up
the rebuild as far as possible, so my plan is to set the following
parameters, (but I've no idea what safe numbers would be).

dev.raid.speed_limit_min =
dev.raid.speed_limit_max =

Current values are:
# sysctl dev.raid.speed_limit_min
dev.raid.speed_limit_min = 1000
# sysctl dev.raid.speed_limit_max
dev.raid.speed_limit_max = 200000

Set readahead:
# blockdev --setra 65536 /dev/md0

Set stripe_cache_size to 32 MiB:
# echo 32768 > /sys/block/md0/md/stripe_cache_size

Turn on bitmaps:
# mdadm --grow --bitmap=internal /dev/md0

Rebuild the array by reintegrating /dev/sdb1:
# mdadm /dev/md0 --add /dev/sdb1

Turn off bitmaps after rebuild is completed:
# mdadm --grow --bitmap=none /dev/md0


Thanks for your time and patience.


Current Array and hardware stats:
-------------------------------------------------

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
  Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
   Raid Devices : 5
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Jun  3 17:38:15 2014
          State : active, degraded
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : audioliboffsite:0  (local to host audioliboffsite)
           UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
         Events : 11314

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       65        1      active sync   /dev/sde1
       2       8       81        2      active sync   /dev/sdf1
       3       8       33        3      active sync   /dev/sdc1
       5       8       49        4      active sync   /dev/sdd1

# lsblk -i
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda       8:0    1  7.5G  0 disk
|-sda1    8:1    1  512M  0 part  /boot
`-sda2    8:2    1    7G  0 part  /
sdb       8:16   0  2.7T  0 disk
`-sdb1    8:17   0  2.7T  0 part
sdc       8:32   0  2.7T  0 disk
`-sdc1    8:33   0  2.7T  0 part
  `-md0   9:0    0 10.9T  0 raid5
sdd       8:48   0  2.7T  0 disk
`-sdd1    8:49   0  2.7T  0 part
  `-md0   9:0    0 10.9T  0 raid5
sde       8:64   0  2.7T  0 disk
`-sde1    8:65   0  2.7T  0 part
  `-md0   9:0    0 10.9T  0 raid5
sdf       8:80   0  2.7T  0 disk
`-sdf1    8:81   0  2.7T  0 part
  `-md0   9:0    0 10.9T  0 raid5







I've answered your questions below as best I can:

>> Any idea what would cause constant writing - I presume from what I see that the initial array sync completed?--
>
> Hmmm...
> Do the numbers in /proc/diskstats change?
>
>   watch -d 'grep md0 /proc/diskstats'


Nope, they remain constant


> What is in /sys/block/md0/md/safe_mode_delay?

0.203 is the value at present - I can try changing it afrter
rebuilding the array.


> What if you change that to a different number (it is in seconds and can be
> fractional)?
>
> What  kernel version (uname -a)?

3.14.6-1-ARCH #1 SMP PREEMPT Sun Jun 8 10:08:38 CEST 2014 x86_64 GNU/Linux

Re: Understanding raid array status: Active vs Clean

[George Duffield]

A little more information if it helps deciding on the best recovery
strategy.  As can be seen all drives still in the array have event
count:
Events : 11314

The drive that fell out of the array has an event count of:
Events : 11306

Unless mdadm writes to the drives when a machine is booted or the
array partitioned I know for certain that the array has not been
written to i.e. no files have been added or deleted.

Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
to me the following guidance applies:
If the event count closely matches but not exactly, use "mdadm
--assemble --force /dev/mdX <list of devices>" to force mdadm to
assemble the array anyway using the devices with the closest possible
event count. If the event count of a drive is way off, this probably
means that drive has been out of the array for a long time and
shouldn't be included in the assembly. Re-add it after the assembly so
it's sync:ed up using information from the drives with closest event
counts.

However, in my case the array has been auto assebled by mdadm at boot
time.  How would I best go about adding /dev/sdb1 back into the array?


Superblock information:

# mdadm --examine /dev/sd[bcdef]1

/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
           Name : audioliboffsite:0  (local to host audioliboffsite)
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : e9663464:5b912bb1:a5617fe9:19abfc55

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun  3 17:31:02 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : fb31415f - correct
         Events : 11306

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
           Name : audioliboffsite:0  (local to host audioliboffsite)
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 71052522:8b78da02:3e0cd6da:f3b3eb3e

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun  3 17:38:15 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : e5177c43 - correct
         Events : 11314

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
           Name : audioliboffsite:0  (local to host audioliboffsite)
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 2bd0953f:2319fe92:2dbe7e53:4b16fc80

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun  3 17:38:15 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4d64fbdf - correct
         Events : 11314

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
           Name : audioliboffsite:0  (local to host audioliboffsite)
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 3e1155bb:a4b65803:caf487e4:9bb01396

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun  3 17:38:15 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : df9fab5c - correct
         Events : 11314

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
           Name : audioliboffsite:0  (local to host audioliboffsite)
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 1714ea64:c1610064:b8603f47:eaaffc3c

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun  3 17:38:15 2014
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : f37cc48f - correct
         Events : 11314

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)




Checking event count on all drives making up the array (and the member
that "failed"):

[root@audioliboffsite ~]# mdadm --examine /dev/sdb
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
[root@audioliboffsite ~]# mdadm --examine /dev/sdc
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
[root@audioliboffsite ~]# mdadm --examine /dev/sdd
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
[root@audioliboffsite ~]# mdadm --examine /dev/sde
/dev/sde:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
[root@audioliboffsite ~]# mdadm --examine /dev/sdf
/dev/sdf:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)


On Tue, Jun 17, 2014 at 4:31 PM, George Duffield
<forumscollective@gmail.com> wrote:
> Apologies for the long delay in responding - I had further issues with
> Microservers trashing the first drive in the backplane, including one
> of the drives for the array in question (in the case of the array it
> seems the drive lost power and dropped out the array, albeit it's
> fully functional now and passes SMART testing).  As a result I've
> built new machines using a mini-itx motherboards and made a clean
> install of Arch Linux - finished that last night, so now have the
> array migrated to the new machine and powered up, albeit in degraded
> mode.  I'd appreciate some advice re rebuilding this array (by adding
> back the drive in question).  I've set out below pertinent info
> relating to the array and hard drives in the system as well as my
> intended recovery strategy.  As can be seen from lsblk, /dev/sdb1 is
> the drive that is no longer recognised as being part of the array.  It
> has not been written to since the incident occurred.  Is there a quick
> & easy to reintegrate it into the array or is my only option to run:
> # mdadm /dev/md0 --add /dev/sdb1
>
> and let it take its course?
>
> The machine has a 3.5Ghz i3 CPU and currently has 8GB ram installed, I
> can swap out the 4GB chips and replace with 8GB chips if 16GB RAM will
> significantly increase the rebuild speed.  I'd also like to speed up
> the rebuild as far as possible, so my plan is to set the following
> parameters, (but I've no idea what safe numbers would be).
>
> dev.raid.speed_limit_min =
> dev.raid.speed_limit_max =
>
> Current values are:
> # sysctl dev.raid.speed_limit_min
> dev.raid.speed_limit_min = 1000
> # sysctl dev.raid.speed_limit_max
> dev.raid.speed_limit_max = 200000
>
> Set readahead:
> # blockdev --setra 65536 /dev/md0
>
> Set stripe_cache_size to 32 MiB:
> # echo 32768 > /sys/block/md0/md/stripe_cache_size
>
> Turn on bitmaps:
> # mdadm --grow --bitmap=internal /dev/md0
>
> Rebuild the array by reintegrating /dev/sdb1:
> # mdadm /dev/md0 --add /dev/sdb1
>
> Turn off bitmaps after rebuild is completed:
> # mdadm --grow --bitmap=none /dev/md0
>
>
> Thanks for your time and patience.
>
>
> Current Array and hardware stats:
> -------------------------------------------------
>
> # mdadm --detail /dev/md0
> /dev/md0:
>         Version : 1.2
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>    Raid Devices : 5
>   Total Devices : 4
>     Persistence : Superblock is persistent
>
>   Intent Bitmap : Internal
>
>     Update Time : Tue Jun  3 17:38:15 2014
>           State : active, degraded
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>          Events : 11314
>
>     Number   Major   Minor   RaidDevice State
>        0       0        0        0      removed
>        1       8       65        1      active sync   /dev/sde1
>        2       8       81        2      active sync   /dev/sdf1
>        3       8       33        3      active sync   /dev/sdc1
>        5       8       49        4      active sync   /dev/sdd1
>
> # lsblk -i
> NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
> sda       8:0    1  7.5G  0 disk
> |-sda1    8:1    1  512M  0 part  /boot
> `-sda2    8:2    1    7G  0 part  /
> sdb       8:16   0  2.7T  0 disk
> `-sdb1    8:17   0  2.7T  0 part
> sdc       8:32   0  2.7T  0 disk
> `-sdc1    8:33   0  2.7T  0 part
>   `-md0   9:0    0 10.9T  0 raid5
> sdd       8:48   0  2.7T  0 disk
> `-sdd1    8:49   0  2.7T  0 part
>   `-md0   9:0    0 10.9T  0 raid5
> sde       8:64   0  2.7T  0 disk
> `-sde1    8:65   0  2.7T  0 part
>   `-md0   9:0    0 10.9T  0 raid5
> sdf       8:80   0  2.7T  0 disk
> `-sdf1    8:81   0  2.7T  0 part
>   `-md0   9:0    0 10.9T  0 raid5
>
>
>
>
>
>
>
> I've answered your questions below as best I can:
>
>>> Any idea what would cause constant writing - I presume from what I see that the initial array sync completed?--
>>
>> Hmmm...
>> Do the numbers in /proc/diskstats change?
>>
>>   watch -d 'grep md0 /proc/diskstats'
>
>
> Nope, they remain constant
>
>
>> What is in /sys/block/md0/md/safe_mode_delay?
>
> 0.203 is the value at present - I can try changing it afrter
> rebuilding the array.
>
>
>> What if you change that to a different number (it is in seconds and can be
>> fractional)?
>>
>> What  kernel version (uname -a)?
>
> 3.14.6-1-ARCH #1 SMP PREEMPT Sun Jun 8 10:08:38 CEST 2014 x86_64 GNU/Linux

Re: Understanding raid array status: Active vs Clean

[George Duffield]

Please ignore my reference to the array being partitioned, what I'd
intended to write follows:
Unless mdadm writes to the drives when a machine is booted or the
array MOUNTED I know for certain that the array has not been
written to i.e. no files have been added or deleted from a user
perspective.  The degraded array has been mounted and files read from
the array, but that's it.

Would really appreciate some input here so I can get on with growing
my main array once this "backup" machine is fully functional and I
know the underlying files are intact.

On Wed, Jun 18, 2014 at 3:25 PM, George Duffield
<forumscollective@gmail.com> wrote:
> A little more information if it helps deciding on the best recovery
> strategy.  As can be seen all drives still in the array have event
> count:
> Events : 11314
>
> The drive that fell out of the array has an event count of:
> Events : 11306
>
> Unless mdadm writes to the drives when a machine is booted or the
> array partitioned I know for certain that the array has not been
> written to i.e. no files have been added or deleted.
>
> Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
> to me the following guidance applies:
> If the event count closely matches but not exactly, use "mdadm
> --assemble --force /dev/mdX <list of devices>" to force mdadm to
> assemble the array anyway using the devices with the closest possible
> event count. If the event count of a drive is way off, this probably
> means that drive has been out of the array for a long time and
> shouldn't be included in the assembly. Re-add it after the assembly so
> it's sync:ed up using information from the drives with closest event
> counts.
>
> However, in my case the array has been auto assebled by mdadm at boot
> time.  How would I best go about adding /dev/sdb1 back into the array?
>
>
> Superblock information:
>
> # mdadm --examine /dev/sd[bcdef]1
>
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>    Raid Devices : 5
>
>  Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=0 sectors
>           State : clean
>     Device UUID : e9663464:5b912bb1:a5617fe9:19abfc55
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun  3 17:31:02 2014
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : fb31415f - correct
>          Events : 11306
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 0
>    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdc1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>    Raid Devices : 5
>
>  Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=0 sectors
>           State : clean
>     Device UUID : 71052522:8b78da02:3e0cd6da:f3b3eb3e
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun  3 17:38:15 2014
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : e5177c43 - correct
>          Events : 11314
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 3
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdd1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>    Raid Devices : 5
>
>  Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=0 sectors
>           State : clean
>     Device UUID : 2bd0953f:2319fe92:2dbe7e53:4b16fc80
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun  3 17:38:15 2014
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : 4d64fbdf - correct
>          Events : 11314
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 4
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sde1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>    Raid Devices : 5
>
>  Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=0 sectors
>           State : clean
>     Device UUID : 3e1155bb:a4b65803:caf487e4:9bb01396
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun  3 17:38:15 2014
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : df9fab5c - correct
>          Events : 11314
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 1
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdf1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>    Raid Devices : 5
>
>  Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=0 sectors
>           State : clean
>     Device UUID : 1714ea64:c1610064:b8603f47:eaaffc3c
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun  3 17:38:15 2014
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : f37cc48f - correct
>          Events : 11314
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 2
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
>
>
> Checking event count on all drives making up the array (and the member
> that "failed"):
>
> [root@audioliboffsite ~]# mdadm --examine /dev/sdb
> /dev/sdb:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> [root@audioliboffsite ~]# mdadm --examine /dev/sdc
> /dev/sdc:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> [root@audioliboffsite ~]# mdadm --examine /dev/sdd
> /dev/sdd:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> [root@audioliboffsite ~]# mdadm --examine /dev/sde
> /dev/sde:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> [root@audioliboffsite ~]# mdadm --examine /dev/sdf
> /dev/sdf:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
>
>
> On Tue, Jun 17, 2014 at 4:31 PM, George Duffield
> <forumscollective@gmail.com> wrote:
>> Apologies for the long delay in responding - I had further issues with
>> Microservers trashing the first drive in the backplane, including one
>> of the drives for the array in question (in the case of the array it
>> seems the drive lost power and dropped out the array, albeit it's
>> fully functional now and passes SMART testing).  As a result I've
>> built new machines using a mini-itx motherboards and made a clean
>> install of Arch Linux - finished that last night, so now have the
>> array migrated to the new machine and powered up, albeit in degraded
>> mode.  I'd appreciate some advice re rebuilding this array (by adding
>> back the drive in question).  I've set out below pertinent info
>> relating to the array and hard drives in the system as well as my
>> intended recovery strategy.  As can be seen from lsblk, /dev/sdb1 is
>> the drive that is no longer recognised as being part of the array.  It
>> has not been written to since the incident occurred.  Is there a quick
>> & easy to reintegrate it into the array or is my only option to run:
>> # mdadm /dev/md0 --add /dev/sdb1
>>
>> and let it take its course?
>>
>> The machine has a 3.5Ghz i3 CPU and currently has 8GB ram installed, I
>> can swap out the 4GB chips and replace with 8GB chips if 16GB RAM will
>> significantly increase the rebuild speed.  I'd also like to speed up
>> the rebuild as far as possible, so my plan is to set the following
>> parameters, (but I've no idea what safe numbers would be).
>>
>> dev.raid.speed_limit_min =
>> dev.raid.speed_limit_max =
>>
>> Current values are:
>> # sysctl dev.raid.speed_limit_min
>> dev.raid.speed_limit_min = 1000
>> # sysctl dev.raid.speed_limit_max
>> dev.raid.speed_limit_max = 200000
>>
>> Set readahead:
>> # blockdev --setra 65536 /dev/md0
>>
>> Set stripe_cache_size to 32 MiB:
>> # echo 32768 > /sys/block/md0/md/stripe_cache_size
>>
>> Turn on bitmaps:
>> # mdadm --grow --bitmap=internal /dev/md0
>>
>> Rebuild the array by reintegrating /dev/sdb1:
>> # mdadm /dev/md0 --add /dev/sdb1
>>
>> Turn off bitmaps after rebuild is completed:
>> # mdadm --grow --bitmap=none /dev/md0
>>
>>
>> Thanks for your time and patience.
>>
>>
>> Current Array and hardware stats:
>> -------------------------------------------------
>>
>> # mdadm --detail /dev/md0
>> /dev/md0:
>>         Version : 1.2
>>   Creation Time : Thu Apr 17 01:13:52 2014
>>      Raid Level : raid5
>>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>>     Persistence : Superblock is persistent
>>
>>   Intent Bitmap : Internal
>>
>>     Update Time : Tue Jun  3 17:38:15 2014
>>           State : active, degraded
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>   Spare Devices : 0
>>
>>          Layout : left-symmetric
>>      Chunk Size : 512K
>>
>>            Name : audioliboffsite:0  (local to host audioliboffsite)
>>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>>          Events : 11314
>>
>>     Number   Major   Minor   RaidDevice State
>>        0       0        0        0      removed
>>        1       8       65        1      active sync   /dev/sde1
>>        2       8       81        2      active sync   /dev/sdf1
>>        3       8       33        3      active sync   /dev/sdc1
>>        5       8       49        4      active sync   /dev/sdd1
>>
>> # lsblk -i
>> NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
>> sda       8:0    1  7.5G  0 disk
>> |-sda1    8:1    1  512M  0 part  /boot
>> `-sda2    8:2    1    7G  0 part  /
>> sdb       8:16   0  2.7T  0 disk
>> `-sdb1    8:17   0  2.7T  0 part
>> sdc       8:32   0  2.7T  0 disk
>> `-sdc1    8:33   0  2.7T  0 part
>>   `-md0   9:0    0 10.9T  0 raid5
>> sdd       8:48   0  2.7T  0 disk
>> `-sdd1    8:49   0  2.7T  0 part
>>   `-md0   9:0    0 10.9T  0 raid5
>> sde       8:64   0  2.7T  0 disk
>> `-sde1    8:65   0  2.7T  0 part
>>   `-md0   9:0    0 10.9T  0 raid5
>> sdf       8:80   0  2.7T  0 disk
>> `-sdf1    8:81   0  2.7T  0 part
>>   `-md0   9:0    0 10.9T  0 raid5
>>
>>
>>
>>
>>
>>
>>
>> I've answered your questions below as best I can:
>>
>>>> Any idea what would cause constant writing - I presume from what I see that the initial array sync completed?--
>>>
>>> Hmmm...
>>> Do the numbers in /proc/diskstats change?
>>>
>>>   watch -d 'grep md0 /proc/diskstats'
>>
>>
>> Nope, they remain constant
>>
>>
>>> What is in /sys/block/md0/md/safe_mode_delay?
>>
>> 0.203 is the value at present - I can try changing it afrter
>> rebuilding the array.
>>
>>
>>> What if you change that to a different number (it is in seconds and can be
>>> fractional)?
>>>
>>> What  kernel version (uname -a)?
>>
>> 3.14.6-1-ARCH #1 SMP PREEMPT Sun Jun 8 10:08:38 CEST 2014 x86_64 GNU/Linux

Re: Understanding raid array status: Active vs Clean

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 4862 bytes --]

On Wed Jun 18, 2014 at 03:25:27PM +0200, George Duffield wrote:

> A little more information if it helps deciding on the best recovery
> strategy.  As can be seen all drives still in the array have event
> count:
> Events : 11314
> 
> The drive that fell out of the array has an event count of:
> Events : 11306
> 
> Unless mdadm writes to the drives when a machine is booted or the
> array partitioned I know for certain that the array has not been
> written to i.e. no files have been added or deleted.
> 
> Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
> to me the following guidance applies:
> If the event count closely matches but not exactly, use "mdadm
> --assemble --force /dev/mdX <list of devices>" to force mdadm to
> assemble the array anyway using the devices with the closest possible
> event count. If the event count of a drive is way off, this probably
> means that drive has been out of the array for a long time and
> shouldn't be included in the assembly. Re-add it after the assembly so
> it's sync:ed up using information from the drives with closest event
> counts.
> 
> However, in my case the array has been auto assebled by mdadm at boot
> time.  How would I best go about adding /dev/sdb1 back into the array?
> 
That doesn't matter here - a force assemble would have left out the
drive with the lower event count as well. As there's a bitmap on the
array then either a --re-add or a --add (these should be treated the
same for arrays with persistent superblocks) should just synch any
differences since the disk was failed.

> On Tue, Jun 17, 2014 at 4:31 PM, George Duffield
> <forumscollective@gmail.com> wrote:
> > Apologies for the long delay in responding - I had further issues with
> > Microservers trashing the first drive in the backplane, including one
> > of the drives for the array in question (in the case of the array it
> > seems the drive lost power and dropped out the array, albeit it's
> > fully functional now and passes SMART testing).  As a result I've
> > built new machines using a mini-itx motherboards and made a clean
> > install of Arch Linux - finished that last night, so now have the
> > array migrated to the new machine and powered up, albeit in degraded
> > mode.  I'd appreciate some advice re rebuilding this array (by adding
> > back the drive in question).  I've set out below pertinent info
> > relating to the array and hard drives in the system as well as my
> > intended recovery strategy.  As can be seen from lsblk, /dev/sdb1 is
> > the drive that is no longer recognised as being part of the array.  It
> > has not been written to since the incident occurred.  Is there a quick
> > & easy to reintegrate it into the array or is my only option to run:
> > # mdadm /dev/md0 --add /dev/sdb1
> >
> > and let it take its course?
> >
> > The machine has a 3.5Ghz i3 CPU and currently has 8GB ram installed, I
> > can swap out the 4GB chips and replace with 8GB chips if 16GB RAM will
> > significantly increase the rebuild speed.  I'd also like to speed up
> > the rebuild as far as possible, so my plan is to set the following
> > parameters, (but I've no idea what safe numbers would be).
> >
> > dev.raid.speed_limit_min =
> > dev.raid.speed_limit_max =
> >
> > Current values are:
> > # sysctl dev.raid.speed_limit_min
> > dev.raid.speed_limit_min = 1000
> > # sysctl dev.raid.speed_limit_max
> > dev.raid.speed_limit_max = 200000
> >
You can set these as high as you like, though it can impact other tasks.
I'd suggest bumping the speed_limit_min up gradually and seeing how it
goes (unless you're hitting speed_limit_max already).

> > Set readahead:
> > # blockdev --setra 65536 /dev/md0
> >
> > Set stripe_cache_size to 32 MiB:
> > # echo 32768 > /sys/block/md0/md/stripe_cache_size
> >
> > Turn on bitmaps:
> > # mdadm --grow --bitmap=internal /dev/md0
> >
> > Rebuild the array by reintegrating /dev/sdb1:
> > # mdadm /dev/md0 --add /dev/sdb1
> >
> > Turn off bitmaps after rebuild is completed:
> > # mdadm --grow --bitmap=none /dev/md0
> >
I'm not sure you can modify the bitmaps on degraded arrays anyway, and
adding one before replacing a failed member won't do any good
regardless. The bitmap is only used if the disk used to be an active
member of the array, so will be ignored until the disk is fully synched
anyway. If you were adding a new disk (rather than just re-adding the
existing failed disk) then it might speed things up to drop the bitmap
until the array rebuild is complete (if this is possible).

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Understanding raid array status: Active vs Clean

[George Duffield]

Thx Robin

I've run:
# mdadm --manage /dev/md0 --re-add /dev/sdb1
mdadm: re-added /dev/sdb1

# mdadm --detail /dev/md0 now returns:

/dev/md0:
        Version : 1.2
  Creation Time : Thu Apr 17 01:13:52 2014
     Raid Level : raid5
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
  Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jun 18 19:46:38 2014
          State : active
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : audioliboffsite:0  (local to host audioliboffsite)
           UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
         Events : 11319

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       65        1      active sync   /dev/sde1
       2       8       81        2      active sync   /dev/sdf1
       3       8       33        3      active sync   /dev/sdc1
       5       8       49        4      active sync   /dev/sdd1


# watch cat /proc/mdstat returns:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdd1[5] sdc1[3] sde1[1] sdf1[2]
      11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
[5/5] [UUUUU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>


# watch -d 'grep md0 /proc/diskstats' returns:
   9       0 md0 348 0 2784 0 0 0 0 0 0 0 0

and the output never changes.

So, array seems OK, and I'm back to the question that started this
thread - why would this array's state be Active rather than Clean?




On Wed, Jun 18, 2014 at 5:03 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Wed Jun 18, 2014 at 03:25:27PM +0200, George Duffield wrote:
>
>> A little more information if it helps deciding on the best recovery
>> strategy.  As can be seen all drives still in the array have event
>> count:
>> Events : 11314
>>
>> The drive that fell out of the array has an event count of:
>> Events : 11306
>>
>> Unless mdadm writes to the drives when a machine is booted or the
>> array partitioned I know for certain that the array has not been
>> written to i.e. no files have been added or deleted.
>>
>> Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
>> to me the following guidance applies:
>> If the event count closely matches but not exactly, use "mdadm
>> --assemble --force /dev/mdX <list of devices>" to force mdadm to
>> assemble the array anyway using the devices with the closest possible
>> event count. If the event count of a drive is way off, this probably
>> means that drive has been out of the array for a long time and
>> shouldn't be included in the assembly. Re-add it after the assembly so
>> it's sync:ed up using information from the drives with closest event
>> counts.
>>
>> However, in my case the array has been auto assebled by mdadm at boot
>> time.  How would I best go about adding /dev/sdb1 back into the array?
>>
> That doesn't matter here - a force assemble would have left out the
> drive with the lower event count as well. As there's a bitmap on the
> array then either a --re-add or a --add (these should be treated the
> same for arrays with persistent superblocks) should just synch any
> differences since the disk was failed.

Re: Understanding raid array status: Active vs Clean

[George Duffield]

# cat /sys/block/md0/md/safe_mode_delay returns:

0.203

changing the value to 0.500:
# echo 0.503 > /sys/block/md0/md/safe_mode_delay

makes no difference to the array state.



On Wed, Jun 18, 2014 at 5:57 PM, George Duffield
<forumscollective@gmail.com> wrote:
> Thx Robin
>
> I've run:
> # mdadm --manage /dev/md0 --re-add /dev/sdb1
> mdadm: re-added /dev/sdb1
>
> # mdadm --detail /dev/md0 now returns:
>
> /dev/md0:
>         Version : 1.2
>   Creation Time : Thu Apr 17 01:13:52 2014
>      Raid Level : raid5
>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>    Raid Devices : 5
>   Total Devices : 5
>     Persistence : Superblock is persistent
>
>   Intent Bitmap : Internal
>
>     Update Time : Wed Jun 18 19:46:38 2014
>           State : active
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>            Name : audioliboffsite:0  (local to host audioliboffsite)
>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>          Events : 11319
>
>     Number   Major   Minor   RaidDevice State
>        0       8       17        0      active sync   /dev/sdb1
>        1       8       65        1      active sync   /dev/sde1
>        2       8       81        2      active sync   /dev/sdf1
>        3       8       33        3      active sync   /dev/sdc1
>        5       8       49        4      active sync   /dev/sdd1
>
>
> # watch cat /proc/mdstat returns:
>
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdb1[0] sdd1[5] sdc1[3] sde1[1] sdf1[2]
>       11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
> [5/5] [UUUUU]
>       bitmap: 0/22 pages [0KB], 65536KB chunk
>
> unused devices: <none>
>
>
> # watch -d 'grep md0 /proc/diskstats' returns:
>    9       0 md0 348 0 2784 0 0 0 0 0 0 0 0
>
> and the output never changes.
>
> So, array seems OK, and I'm back to the question that started this
> thread - why would this array's state be Active rather than Clean?
>
>
>
>
> On Wed, Jun 18, 2014 at 5:03 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> On Wed Jun 18, 2014 at 03:25:27PM +0200, George Duffield wrote:
>>
>>> A little more information if it helps deciding on the best recovery
>>> strategy.  As can be seen all drives still in the array have event
>>> count:
>>> Events : 11314
>>>
>>> The drive that fell out of the array has an event count of:
>>> Events : 11306
>>>
>>> Unless mdadm writes to the drives when a machine is booted or the
>>> array partitioned I know for certain that the array has not been
>>> written to i.e. no files have been added or deleted.
>>>
>>> Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
>>> to me the following guidance applies:
>>> If the event count closely matches but not exactly, use "mdadm
>>> --assemble --force /dev/mdX <list of devices>" to force mdadm to
>>> assemble the array anyway using the devices with the closest possible
>>> event count. If the event count of a drive is way off, this probably
>>> means that drive has been out of the array for a long time and
>>> shouldn't be included in the assembly. Re-add it after the assembly so
>>> it's sync:ed up using information from the drives with closest event
>>> counts.
>>>
>>> However, in my case the array has been auto assebled by mdadm at boot
>>> time.  How would I best go about adding /dev/sdb1 back into the array?
>>>
>> That doesn't matter here - a force assemble would have left out the
>> drive with the lower event count as well. As there's a bitmap on the
>> array then either a --re-add or a --add (these should be treated the
>> same for arrays with persistent superblocks) should just synch any
>> differences since the disk was failed.

Re: Understanding raid array status: Active vs Clean

[George Duffield]

Can anyone give me some hints as to why this array would remain Active
rather than report Clean?  Any help/ insights much appreciated.

On Wed, Jun 18, 2014 at 6:04 PM, George Duffield
<forumscollective@gmail.com> wrote:
> # cat /sys/block/md0/md/safe_mode_delay returns:
>
> 0.203
>
> changing the value to 0.500:
> # echo 0.503 > /sys/block/md0/md/safe_mode_delay
>
> makes no difference to the array state.
>
>
>
> On Wed, Jun 18, 2014 at 5:57 PM, George Duffield
> <forumscollective@gmail.com> wrote:
>> Thx Robin
>>
>> I've run:
>> # mdadm --manage /dev/md0 --re-add /dev/sdb1
>> mdadm: re-added /dev/sdb1
>>
>> # mdadm --detail /dev/md0 now returns:
>>
>> /dev/md0:
>>         Version : 1.2
>>   Creation Time : Thu Apr 17 01:13:52 2014
>>      Raid Level : raid5
>>      Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>>   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
>>    Raid Devices : 5
>>   Total Devices : 5
>>     Persistence : Superblock is persistent
>>
>>   Intent Bitmap : Internal
>>
>>     Update Time : Wed Jun 18 19:46:38 2014
>>           State : active
>>  Active Devices : 5
>> Working Devices : 5
>>  Failed Devices : 0
>>   Spare Devices : 0
>>
>>          Layout : left-symmetric
>>      Chunk Size : 512K
>>
>>            Name : audioliboffsite:0  (local to host audioliboffsite)
>>            UUID : aba348c6:8dc7b4a7:4e282ab5:40431aff
>>          Events : 11319
>>
>>     Number   Major   Minor   RaidDevice State
>>        0       8       17        0      active sync   /dev/sdb1
>>        1       8       65        1      active sync   /dev/sde1
>>        2       8       81        2      active sync   /dev/sdf1
>>        3       8       33        3      active sync   /dev/sdc1
>>        5       8       49        4      active sync   /dev/sdd1
>>
>>
>> # watch cat /proc/mdstat returns:
>>
>> Personalities : [raid6] [raid5] [raid4]
>> md0 : active raid5 sdb1[0] sdd1[5] sdc1[3] sde1[1] sdf1[2]
>>       11720536064 blocks super 1.2 level 5, 512k chunk, algorithm 2
>> [5/5] [UUUUU]
>>       bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> unused devices: <none>
>>
>>
>> # watch -d 'grep md0 /proc/diskstats' returns:
>>    9       0 md0 348 0 2784 0 0 0 0 0 0 0 0
>>
>> and the output never changes.
>>
>> So, array seems OK, and I'm back to the question that started this
>> thread - why would this array's state be Active rather than Clean?
>>
>>
>>
>>
>> On Wed, Jun 18, 2014 at 5:03 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>>> On Wed Jun 18, 2014 at 03:25:27PM +0200, George Duffield wrote:
>>>
>>>> A little more information if it helps deciding on the best recovery
>>>> strategy.  As can be seen all drives still in the array have event
>>>> count:
>>>> Events : 11314
>>>>
>>>> The drive that fell out of the array has an event count of:
>>>> Events : 11306
>>>>
>>>> Unless mdadm writes to the drives when a machine is booted or the
>>>> array partitioned I know for certain that the array has not been
>>>> written to i.e. no files have been added or deleted.
>>>>
>>>> Per https://raid.wiki.kernel.org/index.php/RAID_Recovery it would seem
>>>> to me the following guidance applies:
>>>> If the event count closely matches but not exactly, use "mdadm
>>>> --assemble --force /dev/mdX <list of devices>" to force mdadm to
>>>> assemble the array anyway using the devices with the closest possible
>>>> event count. If the event count of a drive is way off, this probably
>>>> means that drive has been out of the array for a long time and
>>>> shouldn't be included in the assembly. Re-add it after the assembly so
>>>> it's sync:ed up using information from the drives with closest event
>>>> counts.
>>>>
>>>> However, in my case the array has been auto assebled by mdadm at boot
>>>> time.  How would I best go about adding /dev/sdb1 back into the array?
>>>>
>>> That doesn't matter here - a force assemble would have left out the
>>> drive with the lower event count as well. As there's a bitmap on the
>>> array then either a --re-add or a --add (these should be treated the
>>> same for arrays with persistent superblocks) should just synch any
>>> differences since the disk was failed.

Re: Understanding raid array status: Active vs Clean

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On Sun, 22 Jun 2014 16:32:31 +0200 George Duffield
<forumscollective@gmail.com> wrote:

> Can anyone give me some hints as to why this array would remain Active
> rather than report Clean?  Any help/ insights much appreciated.

It is a mystery.

> 
> On Wed, Jun 18, 2014 at 6:04 PM, George Duffield
> <forumscollective@gmail.com> wrote:
> > # cat /sys/block/md0/md/safe_mode_delay returns:
> >
> > 0.203
> >
> > changing the value to 0.500:
> > # echo 0.503 > /sys/block/md0/md/safe_mode_delay
> >
> > makes no difference to the array state.

What if you write a smaller number?  e.g. 0.1

What does /sys/block/md0/md/array_state show?
If you "echo clean" into that file, does that "fix" it?
Does it stay fixed if you write to the array?

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Understanding raid array status: Active vs Clean

[George Duffield]

>> > changing the value to 0.500:
>> > # echo 0.503 > /sys/block/md0/md/safe_mode_delay
>> >
>> > makes no difference to the array state.
>
> What if you write a smaller number?  e.g. 0.1

No change to array state.

> What does /sys/block/md0/md/array_state show?

Funnily enough, it shows Clean

Re: Understanding raid array status: Active vs Clean

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

On Sat, 28 Jun 2014 05:01:00 +0200 George Duffield
<forumscollective@gmail.com> wrote:

> >> > changing the value to 0.500:
> >> > # echo 0.503 > /sys/block/md0/md/safe_mode_delay
> >> >
> >> > makes no difference to the array state.
> >
> > What if you write a smaller number?  e.g. 0.1
> 
> No change to array state.
> 
> > What does /sys/block/md0/md/array_state show?
> 
> Funnily enough, it shows Clean


Ahh - I found it.
In get_array_info() in drivers/md/md.c:

	if (mddev->in_sync)
		info.state = (1<<MD_SB_CLEAN);
	if (mddev->bitmap && mddev->bitmap_info.offset)
		info.state = (1<<MD_SB_BITMAP_PRESENT);


that last line should be "|=".

Because you have a bitmap, the 'clean' state is being hidden.

Though if you have a bitmap, the 'clean' state isn't really important because
the bitmap knows which regions are 'clean' and which are not.

But it should be fixed.  I'll send a patch next week.

Thanks for persisting.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]