mdadm raid6 recovery status

mdadm raid6 recovery status

[Paramasivam, Meenakshisundaram]

Hi,

We have 8 disks (8x2TB=16TB) on an enclosure, with mdadm raid 6 serving ~12TB volume 
from a SATA JBOD through 2 e-SATA ports to host machine. We  used up 7 TB. One of e-SATA port 
connecting 4 drives did not respond on controller, possibly due heat issues from adjacent video card 
FX4800, or the controller momentarily went bad since we also had power glitches on multiple machines 
at the same time. This incident made mdadm to report the 4 drives  with "removed" status. After 
relocating the card inside the machine and rebooting, all of the drives physically came up fine. 
Now I was hoping to get 7 TB of data back, and I did:

#mdadm --assemble --scan
mdadm: /dev/md2 assembled from 4 drives - not enough to start the array.

Now I wanted to recover the data, so I did the following steps. I see 
only 2 TB of 7 TB now when I mount. I know that data is still on drives 
and I did not zero-superblock on any of the drives. How do I get data 
back, reliably and quickly? Using mdadm v3.1.2  on FC14. Any pointers/suggestions
 is very much appreciated. Thanks. 

#mdadm -v --assemble --force /dev/md2 /dev/sd{b,c,d,e,f,g,h,i}
mdadm: looking for devices for /dev/md2
mdadm: /dev/sdb is identified as a member of /dev/md2, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md2, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md2, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md2, slot 3.
mdadm: /dev/sdf is identified as a member of /dev/md2, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md2, slot 5.
mdadm: /dev/sdh is identified as a member of /dev/md2, slot 6.
mdadm: /dev/sdi is identified as a member of /dev/md2, slot 7.
mdadm: forcing event count in /dev/sdh(6) from 220810 upto 220815
mdadm: forcing event count in /dev/sdf(4) from 220809 upto 220815
mdadm: forcing event count in /dev/sdi(7) from 220809 upto 220815
mdadm: clearing FAULTY flag for device 4 in /dev/md2 for /dev/sdf
mdadm: clearing FAULTY flag for device 6 in /dev/md2 for /dev/sdh
mdadm: clearing FAULTY flag for device 7 in /dev/md2 for /dev/sdi
mdadm: added /dev/sdc to /dev/md2 as 1
mdadm: added /dev/sdd to /dev/md2 as 2
mdadm: added /dev/sde to /dev/md2 as 3
mdadm: added /dev/sdf to /dev/md2 as 4
mdadm: added /dev/sdg to /dev/md2 as 5
mdadm: added /dev/sdh to /dev/md2 as 6
mdadm: added /dev/sdi to /dev/md2 as 7
mdadm: added /dev/sdb to /dev/md2 as 0
mdadm: /dev/md2 has been started with 7 drives (out of 8).

# mdadm -D /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Fri Dec 16 17:56:14 2011
     Raid Level : raid6
     Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 7
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Mon Mar 26 13:53:25 2012
          State : clean, degraded

 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K
           UUID : 4fcdcafa:fea0c196:4d5dd1d0:da2b21e5
         Events : 0.220827

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       0        0        5      removed
       6       8      112        6      active sync   /dev/sdh
       7       8      128        7      active sync   /dev/sdi

# mdadm /dev/md2 -a /dev/sdg
mdadm: re-added /dev/sdg

This step took 24 hrs to rebuild
#mdadm --stop

#mdadm --assemble --scan
        where /etc/mdadm.conf has a line
            ARRAY /dev/md2 level=raid6 num-devices=8 \
                       devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi
            [yes no partitions, had been thsi way and has always worked]

#mount   /dev/md2   /myarray
mounts fine, however, I now see only 2TB instead of 7 TB for /myarray. I need to get all of the data back and I am stuck here.

Sundar

Re: mdadm raid6 recovery status

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

On Wed, 28 Mar 2012 02:22:33 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:


> #mount   /dev/md2   /myarray
> mounts fine, however, I now see only 2TB instead of 7 TB for /myarray. I need to get all of the data back and I am stuck here.

What exactly do you mean by "see only 2TB"??
The md array is reporting that it is 12TB in size (12000 GB).

What reports 2TB?  /proc/partitions? df? du? ?/

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: mdadm raid6 recovery status

[Paramasivam, Meenakshisundaram]

[root@in-rady-neuro9 ~]# df -kl /myarray
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2             11537161976    162432 10950945196   1% /myarray
Should be 7TB of used space.

[root@in-rady-neuro9 ~]# cat /proc/partitions
major minor  #blocks  name

   8        0  438960128 sda
   8        1     512000 sda1
   8        2   51200000 sda2
   8        3  387247104 sda3
   8       16 1953514584 sdb
   8       32 1953514584 sdc
   8       48 1953514584 sdd
   8       64 1953514584 sde
   8       80 1953514584 sdf
   8       96 1953514584 sdg
   8      112 1953514584 sdh
   8      128 1953514584 sdi
 253        0  346226688 dm-0
 253        1   40992768 dm-1

sd[b-i] are raid devices

[root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Fri Dec 16 17:56:14 2011
     Raid Level : raid6
     Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Tue Mar 27 14:58:01 2012
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 4fcdcafa:fea0c196:4d5dd1d0:da2b21e5
         Events : 0.220864

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       8       96        5      active sync   /dev/sdg
       6       8      112        6      active sync   /dev/sdh
       7       8      128        7      active sync   /dev/sdi
[root@in-rady-neuro9 ~]#


________________________________________
From: NeilBrown [neilb@suse.de]
Sent: Wednesday, March 28, 2012 12:11 AM
To: Paramasivam, Meenakshisundaram
Cc: linux-raid@vger.kernel.org
Subject: Re: mdadm raid6 recovery status

On Wed, 28 Mar 2012 02:22:33 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:


> #mount   /dev/md2   /myarray
> mounts fine, however, I now see only 2TB instead of 7 TB for /myarray. I need to get all of the data back and I am stuck here.

What exactly do you mean by "see only 2TB"??
The md array is reporting that it is 12TB in size (12000 GB).

What reports 2TB?  /proc/partitions? df? du? ?/

NeilBrown

Re: mdadm raid6 recovery status

[John Robinson]

On 28/03/2012 13:49, Paramasivam, Meenakshisundaram wrote:
> [root@in-rady-neuro9 ~]# df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> Should be 7TB of used space.

The device is showing as a 12TB array. Maybe fsck lost some of your data?

> [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> /dev/md2:
>          Version : 0.90
>    Creation Time : Fri Dec 16 17:56:14 2011
>       Raid Level : raid6
>       Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
>    Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)<<<====== Wrong! Should be 7TB of used array space.

No it shouldn't. Your component devices are indeed 2TB, and that is what 
is shown here.

md doesn't know how much data you have on the filesystem on top of the 
array.

Cheers,

John.

Re: mdadm raid6 recovery status

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

> [root@in-rady-neuro9 ~]# df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> Should be 7TB of used space.

This is bad.  Something has happened to your filesystem.
It is almost as though someone ran "mkfs" on the array.
I don't know much about recovery after such an action, but I doubt you
will get much back.

> 
> [root@in-rady-neuro9 ~]# cat /proc/partitions
> major minor  #blocks  name
> 
>    8        0  438960128 sda
>    8        1     512000 sda1
>    8        2   51200000 sda2
>    8        3  387247104 sda3
>    8       16 1953514584 sdb
>    8       32 1953514584 sdc
>    8       48 1953514584 sdd
>    8       64 1953514584 sde
>    8       80 1953514584 sdf
>    8       96 1953514584 sdg
>    8      112 1953514584 sdh
>    8      128 1953514584 sdi
>  253        0  346226688 dm-0
>  253        1   40992768 dm-1

No md2 ???

> 
> sd[b-i] are raid devices
> 
> [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Fri Dec 16 17:56:14 2011
>      Raid Level : raid6
>      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
>   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.

"Used Dev Size" isn't "how much of the array is used by the filesystem" -
mdadm doesn't know anything about filesystems.
It is "How much of each individual device is used by the array", which is
usually a little less than the size of the smallest device.
So 2TB is correct here.


NeilBrown



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: mdadm raid6 recovery status

[Paramasivam, Meenakshisundaram]

Good news: Got ALL of our data back. [Actually it was 4.96TB not 7TB]. 
                     mdadm is a good one.

Bad news: "df" is reporting wrong, while "du" is showing full size.
# df -kl /myarray
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2             11537161976    162432 10950945196   1% /myarray
# du -sk /myarray
5326133556      /myarray
#

I never looked into du or looked in depth of the files & folders and simply got mislead by reported "df" usage; data was there all along. We definitely want "df" for the array's filesystem (ext3) to report right.

Now that we are backing up all of the data (at 400 Mbps) over network, I want to know if "df" reporting can be fixed easily or should I do new array creation and restore data from backup. 

We are ordering a new RAID card, just to be on safer side.

Sundar

________________________________________
From: NeilBrown [neilb@suse.de]
Sent: Wednesday, March 28, 2012 7:27 PM
To: Paramasivam, Meenakshisundaram
Cc: linux-raid@vger.kernel.org
Subject: Re: mdadm raid6 recovery status

On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

> [root@in-rady-neuro9 ~]# df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> Should be 7TB of used space.

This is bad.  Something has happened to your filesystem.
It is almost as though someone ran "mkfs" on the array.
I don't know much about recovery after such an action, but I doubt you
will get much back.

>
> [root@in-rady-neuro9 ~]# cat /proc/partitions
> major minor  #blocks  name
>
>    8        0  438960128 sda
>    8        1     512000 sda1
>    8        2   51200000 sda2
>    8        3  387247104 sda3
>    8       16 1953514584 sdb
>    8       32 1953514584 sdc
>    8       48 1953514584 sdd
>    8       64 1953514584 sde
>    8       80 1953514584 sdf
>    8       96 1953514584 sdg
>    8      112 1953514584 sdh
>    8      128 1953514584 sdi
>  253        0  346226688 dm-0
>  253        1   40992768 dm-1

No md2 ???

>
> sd[b-i] are raid devices
>
> [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Fri Dec 16 17:56:14 2011
>      Raid Level : raid6
>      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
>   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.

"Used Dev Size" isn't "how much of the array is used by the filesystem" -
mdadm doesn't know anything about filesystems.
It is "How much of each individual device is used by the array", which is
usually a little less than the size of the smallest device.
So 2TB is correct here.


NeilBrown

RE: mdadm raid6 recovery status

[Paramasivam, Meenakshisundaram]

Clarification:
>>should I do new array creation 
I meant running newfs on assembled 12 TB array, and restore data from backup, to resolve "df" reporting problem.

________________________________________
From: Paramasivam, Meenakshisundaram
Sent: Thursday, March 29, 2012 1:33 PM
To: NeilBrown
Cc: linux-raid@vger.kernel.org
Subject: RE: mdadm raid6 recovery status

Good news: Got ALL of our data back. [Actually it was 4.96TB not 7TB].
                     mdadm is a good one.

Bad news: "df" is reporting wrong, while "du" is showing full size.
# df -kl /myarray
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2             11537161976    162432 10950945196   1% /myarray
# du -sk /myarray
5326133556      /myarray
#

I never looked into du or looked in depth of the files & folders and simply got mislead by reported "df" usage; data was there all along. We definitely want "df" for the array's filesystem (ext3) to report right.

Now that we are backing up all of the data (at 400 Mbps) over network, I want to know if "df" reporting can be fixed easily or should I do new array creation and restore data from backup.

We are ordering a new RAID card, just to be on safer side.

Sundar

________________________________________
From: NeilBrown [neilb@suse.de]
Sent: Wednesday, March 28, 2012 7:27 PM
To: Paramasivam, Meenakshisundaram
Cc: linux-raid@vger.kernel.org
Subject: Re: mdadm raid6 recovery status

On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

> [root@in-rady-neuro9 ~]# df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> Should be 7TB of used space.

This is bad.  Something has happened to your filesystem.
It is almost as though someone ran "mkfs" on the array.
I don't know much about recovery after such an action, but I doubt you
will get much back.

>
> [root@in-rady-neuro9 ~]# cat /proc/partitions
> major minor  #blocks  name
>
>    8        0  438960128 sda
>    8        1     512000 sda1
>    8        2   51200000 sda2
>    8        3  387247104 sda3
>    8       16 1953514584 sdb
>    8       32 1953514584 sdc
>    8       48 1953514584 sdd
>    8       64 1953514584 sde
>    8       80 1953514584 sdf
>    8       96 1953514584 sdg
>    8      112 1953514584 sdh
>    8      128 1953514584 sdi
>  253        0  346226688 dm-0
>  253        1   40992768 dm-1

No md2 ???

>
> sd[b-i] are raid devices
>
> [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Fri Dec 16 17:56:14 2011
>      Raid Level : raid6
>      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
>   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.

"Used Dev Size" isn't "how much of the array is used by the filesystem" -
mdadm doesn't know anything about filesystems.
It is "How much of each individual device is used by the array", which is
usually a little less than the size of the smallest device.
So 2TB is correct here.


NeilBrown

Re: mdadm raid6 recovery status

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3790 bytes --]

On Thu, 29 Mar 2012 18:47:14 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

> 
> Clarification:
> >>should I do new array creation 
> I meant running newfs on assembled 12 TB array, and restore data from backup, to resolve "df" reporting problem.

I would suggest asking on 
    linux-ext4@vger.kernel.org

be sure to give lots of details - kernel version etc.
It would be worth running 
   fsck -n /dev/md2
first and see if it reports anything strange.
Maybe  just a fsck will fix it.

NeilBrown


> 
> ________________________________________
> From: Paramasivam, Meenakshisundaram
> Sent: Thursday, March 29, 2012 1:33 PM
> To: NeilBrown
> Cc: linux-raid@vger.kernel.org
> Subject: RE: mdadm raid6 recovery status
> 
> Good news: Got ALL of our data back. [Actually it was 4.96TB not 7TB].
>                      mdadm is a good one.
> 
> Bad news: "df" is reporting wrong, while "du" is showing full size.
> # df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> # du -sk /myarray
> 5326133556      /myarray
> #
> 
> I never looked into du or looked in depth of the files & folders and simply got mislead by reported "df" usage; data was there all along. We definitely want "df" for the array's filesystem (ext3) to report right.
> 
> Now that we are backing up all of the data (at 400 Mbps) over network, I want to know if "df" reporting can be fixed easily or should I do new array creation and restore data from backup.
> 
> We are ordering a new RAID card, just to be on safer side.
> 
> Sundar
> 
> ________________________________________
> From: NeilBrown [neilb@suse.de]
> Sent: Wednesday, March 28, 2012 7:27 PM
> To: Paramasivam, Meenakshisundaram
> Cc: linux-raid@vger.kernel.org
> Subject: Re: mdadm raid6 recovery status
> 
> On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
> <mparamas@iupui.edu> wrote:
> 
> > [root@in-rady-neuro9 ~]# df -kl /myarray
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > /dev/md2             11537161976    162432 10950945196   1% /myarray
> > Should be 7TB of used space.
> 
> This is bad.  Something has happened to your filesystem.
> It is almost as though someone ran "mkfs" on the array.
> I don't know much about recovery after such an action, but I doubt you
> will get much back.
> 
> >
> > [root@in-rady-neuro9 ~]# cat /proc/partitions
> > major minor  #blocks  name
> >
> >    8        0  438960128 sda
> >    8        1     512000 sda1
> >    8        2   51200000 sda2
> >    8        3  387247104 sda3
> >    8       16 1953514584 sdb
> >    8       32 1953514584 sdc
> >    8       48 1953514584 sdd
> >    8       64 1953514584 sde
> >    8       80 1953514584 sdf
> >    8       96 1953514584 sdg
> >    8      112 1953514584 sdh
> >    8      128 1953514584 sdi
> >  253        0  346226688 dm-0
> >  253        1   40992768 dm-1
> 
> No md2 ???
> 
> >
> > sd[b-i] are raid devices
> >
> > [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> > /dev/md2:
> >         Version : 0.90
> >   Creation Time : Fri Dec 16 17:56:14 2011
> >      Raid Level : raid6
> >      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
> >   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.
> 
> "Used Dev Size" isn't "how much of the array is used by the filesystem" -
> mdadm doesn't know anything about filesystems.
> It is "How much of each individual device is used by the array", which is
> usually a little less than the size of the smallest device.
> So 2TB is correct here.
> 
> 
> NeilBrown
> 
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: mdadm raid6 recovery status

[Paramasivam, Meenakshisundaram]

Thanks. Is it safe to fsck -n (and then fsck -y) on /dev/md2 when sdg is just added to md2 and is in "spare rebulding" status and recovery is only at 4% completed? BTW, we got all of the data backed up.

Background:
When we assembled md2 we forced rest of the drives and did not include this drive. After 24 hrs, when I tried to assemble all of the drives into md2, I got "md: kicking non-fresh sdg from array!" message through dmesg, and was removed from md2 (through mdadm --detail). I just did:
# mdadm /dev/md2 --add /dev/sdg
mdadm: re-added /dev/sdg

Sundar

________________________________________
From: NeilBrown [neilb@suse.de]
Sent: Thursday, March 29, 2012 3:41 PM
To: Paramasivam, Meenakshisundaram
Cc: linux-raid@vger.kernel.org
Subject: Re: mdadm raid6 recovery status

On Thu, 29 Mar 2012 18:47:14 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

>
> Clarification:
> >>should I do new array creation
> I meant running newfs on assembled 12 TB array, and restore data from backup, to resolve "df" reporting problem.

I would suggest asking on
    linux-ext4@vger.kernel.org

be sure to give lots of details - kernel version etc.
It would be worth running
   fsck -n /dev/md2
first and see if it reports anything strange.
Maybe  just a fsck will fix it.

NeilBrown


>
> ________________________________________
> From: Paramasivam, Meenakshisundaram
> Sent: Thursday, March 29, 2012 1:33 PM
> To: NeilBrown
> Cc: linux-raid@vger.kernel.org
> Subject: RE: mdadm raid6 recovery status
>
> Good news: Got ALL of our data back. [Actually it was 4.96TB not 7TB].
>                      mdadm is a good one.
>
> Bad news: "df" is reporting wrong, while "du" is showing full size.
> # df -kl /myarray
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             11537161976    162432 10950945196   1% /myarray
> # du -sk /myarray
> 5326133556      /myarray
> #
>
> I never looked into du or looked in depth of the files & folders and simply got mislead by reported "df" usage; data was there all along. We definitely want "df" for the array's filesystem (ext3) to report right.
>
> Now that we are backing up all of the data (at 400 Mbps) over network, I want to know if "df" reporting can be fixed easily or should I do new array creation and restore data from backup.
>
> We are ordering a new RAID card, just to be on safer side.
>
> Sundar
>
> ________________________________________
> From: NeilBrown [neilb@suse.de]
> Sent: Wednesday, March 28, 2012 7:27 PM
> To: Paramasivam, Meenakshisundaram
> Cc: linux-raid@vger.kernel.org
> Subject: Re: mdadm raid6 recovery status
>
> On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
> <mparamas@iupui.edu> wrote:
>
> > [root@in-rady-neuro9 ~]# df -kl /myarray
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > /dev/md2             11537161976    162432 10950945196   1% /myarray
> > Should be 7TB of used space.
>
> This is bad.  Something has happened to your filesystem.
> It is almost as though someone ran "mkfs" on the array.
> I don't know much about recovery after such an action, but I doubt you
> will get much back.
>
> >
> > [root@in-rady-neuro9 ~]# cat /proc/partitions
> > major minor  #blocks  name
> >
> >    8        0  438960128 sda
> >    8        1     512000 sda1
> >    8        2   51200000 sda2
> >    8        3  387247104 sda3
> >    8       16 1953514584 sdb
> >    8       32 1953514584 sdc
> >    8       48 1953514584 sdd
> >    8       64 1953514584 sde
> >    8       80 1953514584 sdf
> >    8       96 1953514584 sdg
> >    8      112 1953514584 sdh
> >    8      128 1953514584 sdi
> >  253        0  346226688 dm-0
> >  253        1   40992768 dm-1
>
> No md2 ???
>
> >
> > sd[b-i] are raid devices
> >
> > [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> > /dev/md2:
> >         Version : 0.90
> >   Creation Time : Fri Dec 16 17:56:14 2011
> >      Raid Level : raid6
> >      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
> >   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.
>
> "Used Dev Size" isn't "how much of the array is used by the filesystem" -
> mdadm doesn't know anything about filesystems.
> It is "How much of each individual device is used by the array", which is
> usually a little less than the size of the smallest device.
> So 2TB is correct here.
>
>
> NeilBrown
>
>

Re: mdadm raid6 recovery status

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 5319 bytes --]

On Fri, 30 Mar 2012 21:22:34 +0000 "Paramasivam, Meenakshisundaram"
<mparamas@iupui.edu> wrote:

> 
> Thanks. Is it safe to fsck -n (and then fsck -y) on /dev/md2 when sdg is just added to md2 and is in "spare rebulding" status and recovery is only at 4% completed? BTW, we got all of the data backed up.

It is certainly safe to "fsck -n" while there is a spare rebuilding.  Each
process will slow down the other but that shouldn't be a problem.
I would never say that "fsck -y" is safe without seeing the output for "fsck
-n" however the rebuilding the spare should affect the safety of "fsck -y".

And it is always nice to hear that people have their data safe -  thanks.

NeilBrown


> 
> Background:
> When we assembled md2 we forced rest of the drives and did not include this drive. After 24 hrs, when I tried to assemble all of the drives into md2, I got "md: kicking non-fresh sdg from array!" message through dmesg, and was removed from md2 (through mdadm --detail). I just did:
> # mdadm /dev/md2 --add /dev/sdg
> mdadm: re-added /dev/sdg
> 
> Sundar
> 
> ________________________________________
> From: NeilBrown [neilb@suse.de]
> Sent: Thursday, March 29, 2012 3:41 PM
> To: Paramasivam, Meenakshisundaram
> Cc: linux-raid@vger.kernel.org
> Subject: Re: mdadm raid6 recovery status
> 
> On Thu, 29 Mar 2012 18:47:14 +0000 "Paramasivam, Meenakshisundaram"
> <mparamas@iupui.edu> wrote:
> 
> >
> > Clarification:
> > >>should I do new array creation
> > I meant running newfs on assembled 12 TB array, and restore data from backup, to resolve "df" reporting problem.
> 
> I would suggest asking on
>     linux-ext4@vger.kernel.org
> 
> be sure to give lots of details - kernel version etc.
> It would be worth running
>    fsck -n /dev/md2
> first and see if it reports anything strange.
> Maybe  just a fsck will fix it.
> 
> NeilBrown
> 
> 
> >
> > ________________________________________
> > From: Paramasivam, Meenakshisundaram
> > Sent: Thursday, March 29, 2012 1:33 PM
> > To: NeilBrown
> > Cc: linux-raid@vger.kernel.org
> > Subject: RE: mdadm raid6 recovery status
> >
> > Good news: Got ALL of our data back. [Actually it was 4.96TB not 7TB].
> >                      mdadm is a good one.
> >
> > Bad news: "df" is reporting wrong, while "du" is showing full size.
> > # df -kl /myarray
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > /dev/md2             11537161976    162432 10950945196   1% /myarray
> > # du -sk /myarray
> > 5326133556      /myarray
> > #
> >
> > I never looked into du or looked in depth of the files & folders and simply got mislead by reported "df" usage; data was there all along. We definitely want "df" for the array's filesystem (ext3) to report right.
> >
> > Now that we are backing up all of the data (at 400 Mbps) over network, I want to know if "df" reporting can be fixed easily or should I do new array creation and restore data from backup.
> >
> > We are ordering a new RAID card, just to be on safer side.
> >
> > Sundar
> >
> > ________________________________________
> > From: NeilBrown [neilb@suse.de]
> > Sent: Wednesday, March 28, 2012 7:27 PM
> > To: Paramasivam, Meenakshisundaram
> > Cc: linux-raid@vger.kernel.org
> > Subject: Re: mdadm raid6 recovery status
> >
> > On Wed, 28 Mar 2012 12:49:18 +0000 "Paramasivam, Meenakshisundaram"
> > <mparamas@iupui.edu> wrote:
> >
> > > [root@in-rady-neuro9 ~]# df -kl /myarray
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > /dev/md2             11537161976    162432 10950945196   1% /myarray
> > > Should be 7TB of used space.
> >
> > This is bad.  Something has happened to your filesystem.
> > It is almost as though someone ran "mkfs" on the array.
> > I don't know much about recovery after such an action, but I doubt you
> > will get much back.
> >
> > >
> > > [root@in-rady-neuro9 ~]# cat /proc/partitions
> > > major minor  #blocks  name
> > >
> > >    8        0  438960128 sda
> > >    8        1     512000 sda1
> > >    8        2   51200000 sda2
> > >    8        3  387247104 sda3
> > >    8       16 1953514584 sdb
> > >    8       32 1953514584 sdc
> > >    8       48 1953514584 sdd
> > >    8       64 1953514584 sde
> > >    8       80 1953514584 sdf
> > >    8       96 1953514584 sdg
> > >    8      112 1953514584 sdh
> > >    8      128 1953514584 sdi
> > >  253        0  346226688 dm-0
> > >  253        1   40992768 dm-1
> >
> > No md2 ???
> >
> > >
> > > sd[b-i] are raid devices
> > >
> > > [root@in-rady-neuro9 ~]# mdadm --detail /dev/md2
> > > /dev/md2:
> > >         Version : 0.90
> > >   Creation Time : Fri Dec 16 17:56:14 2011
> > >      Raid Level : raid6
> > >      Array Size : 11721086976 (11178.10 GiB 12002.39 GB)
> > >   Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)  <<<====== Wrong! Should be 7TB of used array space.
> >
> > "Used Dev Size" isn't "how much of the array is used by the filesystem" -
> > mdadm doesn't know anything about filesystems.
> > It is "How much of each individual device is used by the array", which is
> > usually a little less than the size of the smallest device.
> > So 2TB is correct here.
> >
> >
> > NeilBrown
> >
> >
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]