Help Reassembling a raid5 array

Help Reassembling a raid5 array

[Andrew Wilkins]

Hi, please can someone help me with a raid array which as gone wrong
On Friday a backup server was rebooted, but failed to boot, possibly
with an unrelated issue


On Monday someone booted from a usb disk to try and recover it
they were originally able to assemble the array but now they are not,
2 out of the 5 drives are showing out of date.
Device 1 is showing
  Bad Block Log : 512 entries available at offset 72 sectors
  Array State : AAAAA
and is showing 256 less a events
last updated 2 hours before the others

Device 4 is showing
 Array State : A.AAA
and 2 less events
last update 1 hour before the others

The rest all show
 Array State : A.AA.

I've tried manually assembling without device 1, but even with --force
it refused to assemble, force seems to do nothing at all, i am
assuming this maybe because the array state has been updated to say
missing for 2 devices.
The example here
https://raid.wiki.kernel.org/index.php/RAID_Recovery
seems to suggest that in my situation recreating might be my only option

Are there any better options before i go ahead and do this?


If not is the best case to create with device 1 missing then try and
recover once device 4 is back to normal?
Is there anything i can backup other than the array stats that i already have?

Also i am wonder if anyone can shed any light on how it might have got
into this state?
There are a few bad blocks showing on device 1, but am not sure if
that is enough to point toward it being a drive failure on it, and if
the last updates are reliable to point to when failures happened,
nothing went wrong till this afternoon.


Thanks If anyone can Help


Copy of the full array status:



/dev/sda:
   MBR Magic : aa55
Partition[0] :     60592833 sectors at           63 (type 0c)
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
/dev/sde:
   MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
/dev/sdf:
   MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
/dev/sda1:
   MBR Magic : aa55
Partition[0] :    432871117 sectors at   3224498923 (type 07)
Partition[1] :   1953460034 sectors at   3272020941 (type 16)
Partition[3] :    924335794 sectors at     50200576 (type 00)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
           Name : srv-bk1:0
  Creation Time : Thu Jun 20 13:09:09 2013
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 1953259520 (931.39 GiB 1000.07 GB)
     Array Size : 3906516992 (3725.55 GiB 4000.27 GB)
  Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 6b4406c8:0fd049d6:81c29c9a:7969b355

    Update Time : Mon Dec  7 15:32:11 2015
       Checksum : b242234c - correct
         Events : 1285255

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : A.AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x2
     Array UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:abb20cf1
           Name : srv-bk1:0
  Creation Time : Thu Jun 20 13:09:09 2013
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 1953259520 (931.39 GiB 1000.07 GB)
     Array Size : 3906516992 (3725.55 GiB 4000.27 GB)
  Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
Recovery Offset : 152682776 sectors
   Unused Space : before=262056 sectors, after=1024 sectors
          State : clean
    Device UUID : 7c3d5188:39eddb20:96973dee:9c71dc89

    Update Time : Mon Dec  7 12:40:28 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4e0caeec - correct
         Events : 1284999

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
           Name : srv-bk1:0
  Creation Time : Thu Jun 20 13:09:09 2013
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 1953259520 (931.39 GiB 1000.07 GB)
     Array Size : 3906516992 (3725.55 GiB 4000.27 GB)
  Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 2ec58774:2cea69a3:660a0aba:98c57ed7

    Update Time : Mon Dec  7 15:32:11 2015
       Checksum : cd1c6232 - correct
         Events : 1285255

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : A.AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
           Name : srv-bk1:0
  Creation Time : Thu Jun 20 13:09:09 2013
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 1953259520 (931.39 GiB 1000.07 GB)
     Array Size : 3906516992 (3725.55 GiB 4000.27 GB)
  Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : e784451d:b024c581:badf6818:2a8a481e

    Update Time : Mon Dec  7 15:32:11 2015
       Checksum : f95df654 - correct
         Events : 1285255

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : A.AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
           Name : srv-bk1:0
  Creation Time : Thu Jun 20 13:09:09 2013
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 1953259520 (931.39 GiB 1000.07 GB)
     Array Size : 3906516992 (3725.55 GiB 4000.27 GB)
  Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1024 sectors
          State : clean
    Device UUID : 119b942e:5012da9f:42384d61:5f81fd95

    Update Time : Mon Dec  7 14:20:10 2015
       Checksum : e95a3900 - correct
         Events : 1285253

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing)

Re: Help Reassembling a raid5 array

[Phil Turmel]

Hi Andrew,

On 12/07/2015 08:26 PM, Andrew Wilkins wrote:
> Hi, please can someone help me with a raid array which as gone wrong
> On Friday a backup server was rebooted, but failed to boot, possibly
> with an unrelated issue
> 
> 
> On Monday someone booted from a usb disk to try and recover it
> they were originally able to assemble the array but now they are not,
> 2 out of the 5 drives are showing out of date.
> Device 1 is showing
>   Bad Block Log : 512 entries available at offset 72 sectors
>   Array State : AAAAA
> and is showing 256 less a events
> last updated 2 hours before the others
> 
> Device 4 is showing
>  Array State : A.AAA
> and 2 less events
> last update 1 hour before the others
> 
> The rest all show
>  Array State : A.AA.
> 
> I've tried manually assembling without device 1, but even with --force
> it refused to assemble, force seems to do nothing at all, i am
> assuming this maybe because the array state has been updated to say
> missing for 2 devices.

> The example here
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> seems to suggest that in my situation recreating might be my only option

Possibly. Let's try all other options first.

> Are there any better options before i go ahead and do this?

At least one anomaly below.

> /dev/sdb1:

>      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1

> /dev/sdc1:

>      Array UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:abb20cf1

How did this happen ? ^^^^^^^^^^^^^^^^^^^^^^

> /dev/sdd1:

>      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1

> /dev/sde1:

>      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1

> /dev/sdf1:

>      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1

Try forcing re-assembly with sdb1, sdd1, sde1, and sdf1.

If that doesn't work with your current environment, try again with new
kernel and mdadm from booting a rescue environment.  I usually recommend
the LiveCD from sysrescuecd.org.

If that succeeds (in either case), add sdc1 and let it rebuild before
doing anything else.

Phil

Re: Help Reassembling a raid5 array

[Andrew Wilkins]

That was my bad I was gonna blank out all the uuids then figured it
was pointless

On 8 Dec 2015 03:40, "Phil Turmel" <philip@turmel.org> wrote:
>
> Hi Andrew,
>
> On 12/07/2015 08:26 PM, Andrew Wilkins wrote:
> > Hi, please can someone help me with a raid array which as gone wrong
> > On Friday a backup server was rebooted, but failed to boot, possibly
> > with an unrelated issue
> >
> >
> > On Monday someone booted from a usb disk to try and recover it
> > they were originally able to assemble the array but now they are not,
> > 2 out of the 5 drives are showing out of date.
> > Device 1 is showing
> >   Bad Block Log : 512 entries available at offset 72 sectors
> >   Array State : AAAAA
> > and is showing 256 less a events
> > last updated 2 hours before the others
> >
> > Device 4 is showing
> >  Array State : A.AAA
> > and 2 less events
> > last update 1 hour before the others
> >
> > The rest all show
> >  Array State : A.AA.
> >
> > I've tried manually assembling without device 1, but even with --force
> > it refused to assemble, force seems to do nothing at all, i am
> > assuming this maybe because the array state has been updated to say
> > missing for 2 devices.
>
> > The example here
> > https://raid.wiki.kernel.org/index.php/RAID_Recovery
> > seems to suggest that in my situation recreating might be my only option
>
> Possibly. Let's try all other options first.
>
> > Are there any better options before i go ahead and do this?
>
> At least one anomaly below.
>
> > /dev/sdb1:
>
> >      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
>
> > /dev/sdc1:
>
> >      Array UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:abb20cf1
>
> How did this happen ? ^^^^^^^^^^^^^^^^^^^^^^
>
> > /dev/sdd1:
>
> >      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
>
> > /dev/sde1:
>
> >      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
>
> > /dev/sdf1:
>
> >      Array UUID : 7847e506:deaf87e5:7c6cd502:abb20cf1
>
> Try forcing re-assembly with sdb1, sdd1, sde1, and sdf1.
>
> If that doesn't work with your current environment, try again with new
> kernel and mdadm from booting a rescue environment.  I usually recommend
> the LiveCD from sysrescuecd.org.
>
> If that succeeds (in either case), add sdc1 and let it rebuild before
> doing anything else.
>
> Phil

Re: Help Reassembling a raid5 array

[Phil Turmel]

Hi Andrew,

{Added the list back.  Convention on kernel.org is reply-to-all, trim
replies, and either interleave your reply or bottom post.  I've left
some context untrimmed for the list to see.}

On 12/08/2015 01:03 PM, Andrew Wilkins wrote:
> Hi Again
> Thanks for your help
> 
> I had tried exactly what you suggested several times and it did not
> work which is why i thought i was out of options. It had been loaded
> had loaded from an ubuntu 15.10 disk for recovery
> A colleague loaded a purpose build rescue disk from here
> http://www.sysresccd.org/SystemRescueCd_Homepage
> and the same thing worked, unfortunately i don't have any more details
> on what version it was, but i assume the latest

Does this mean you did or did not use system rescue CD for this incident?

> the only difference between our commands was i used /dev/md/0 and my
> colleague used /dev/md0
> but the difference i assume is just to do with where the different
> distros mount arrays, the extra / was something ubuntu was able to put
> in all by it's self when running --scan

Correct. The difference between /dev/md0 and /dev/md/0 is not relevant.

> this is what i was trying in ubuntu and the output
> 
> root@ubuntu:/home/ubuntu# mdadm --assemble --force /dev/md/0 /dev/sdb1
> /dev/sdd1 /dev/sde1 /dev/sdf1 -v
> mdadm: looking for devices for /dev/md/0
> mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot 0.
> mdadm: /dev/sdd1 is identified as a member of /dev/md/0, slot 2.
> mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 3.
> mdadm: /dev/sdf1 is identified as a member of /dev/md/0, slot 4.
> mdadm: no uptodate device for slot 2 of /dev/md/0
> mdadm: added /dev/sdd1 to /dev/md/0 as 2
> mdadm: added /dev/sde1 to /dev/md/0 as 3
> mdadm: added /dev/sdf1 to /dev/md/0 as 4 (possibly out of date)
> mdadm: added /dev/sdb1 to /dev/md/0 as 0
> mdadm: /dev/md/0 assembled from 3 drives - not enough to start the array.
> 
> 
> It's like --force made no difference, does this seem like a bug for it
> to work on this array on one distro but not the other ? or will it be
> down to differences in versions and/or how they are setup

There are differences.  There have been bugs with --force in the past.
That's why I requested you use a recent liveCD.  New kernel and new
mdadm.  Also, try putting --force twice.

Phil