How to fix mistake on raid: mdadm create instead of assemble?

How to fix mistake on raid: mdadm create instead of assemble?

[Santiago DIEZ]

Hi guys,

I'm new to RAID although I've had a server running raid5 for a while.
It was delivered preinstalled like this and I never really wondered
how to monitor and maintain it. This quick introduction just to let
you understand why I'm such an idiot asking such a silly question.

Now what happened?

I have a server with 4 disks and raid5 configured. /dev/md10 is made
of sda10 , sdb10 , sdc10 and sdd10 .

Unfortunately, /dev/sdd died, the server crashed, etc. After restart,
md10 did not rebuilt. I understood sdd was dead and did not try to
force rebuild or even touch the existing system.

First thing I did is ddrescue the remaining partitions sd[abc]10 .
ddrescue did not stumble into any read error so I assume all remaining
partitions are perfectly safe.

Then I examined the partitions with :

================================================================================
# mdadm --examine /dev/loop[012]

/dev/loop0:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 9d37bc89:711887ae:a4d2adc2:26fd5302
  Creation Time : Wed Jan 25 09:08:11 2012
     Raid Level : raid5
  Used Dev Size : 1926247296 (1837.01 GiB 1972.48 GB)
     Array Size : 5778741888 (5511.04 GiB 5917.43 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 10

    Update Time : Mon Sep  5 23:29:23 2016
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 9d0ce26d - correct
         Events : 81589

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       10        0      active sync

   0     0       8       10        0      active sync
   1     1       8       26        1      active sync
   2     2       8       42        2      active sync
   3     3       0        0        3      faulty removed
/dev/loop1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 9d37bc89:711887ae:a4d2adc2:26fd5302
  Creation Time : Wed Jan 25 09:08:11 2012
     Raid Level : raid5
  Used Dev Size : 1926247296 (1837.01 GiB 1972.48 GB)
     Array Size : 5778741888 (5511.04 GiB 5917.43 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 10

    Update Time : Mon Sep  5 23:36:23 2016
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 9d0ce487 - correct
         Events : 81626

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       26        1      active sync

   0     0       0        0        0      removed
   1     1       8       26        1      active sync
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
/dev/loop2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 9d37bc89:711887ae:a4d2adc2:26fd5302
  Creation Time : Wed Jan 25 09:08:11 2012
     Raid Level : raid5
  Used Dev Size : 1926247296 (1837.01 GiB 1972.48 GB)
     Array Size : 5778741888 (5511.04 GiB 5917.43 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 10

    Update Time : Mon Sep  5 23:29:23 2016
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 9d0ce291 - correct
         Events : 81589

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       42        2      active sync

   0     0       8       10        0      active sync
   1     1       8       26        1      active sync
   2     2       8       42        2      active sync
   3     3       0        0        3      faulty removed
================================================================================

There comes my mistake: I ran the --create command instead of --assemble :

================================================================================
# mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
--run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing

mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/loop0 appears to contain an ext2fs file system
       size=5778741888K  mtime=Sat Sep  3 11:00:22 2016
mdadm: /dev/loop0 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
mdadm: /dev/loop1 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
mdadm: /dev/loop2 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
mdadm: size set to 1926115840K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: creation continuing despite oddities due to --run
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
================================================================================

After that, mounting failed:

================================================================================
# mount /dev/md1 /raid/
mount: /dev/md1 is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/md1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
================================================================================

Here's more info about the new raid to be compared with the initial one:

================================================================================
# mdadm --examine /dev/loop[012]

/dev/loop0:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa56f42f:bb95fbde:11ce620e:878b2b1c
           Name : tucana.caoba.fr:1  (local to host tucana.caoba.fr)
  Creation Time : Mon Sep 19 23:17:04 2016
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3852232703 (1836.89 GiB 1972.34 GB)
     Array Size : 5778347520 (5510.66 GiB 5917.03 GB)
  Used Dev Size : 3852231680 (1836.89 GiB 1972.34 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : b4622e59:f0735f5a:825086d1:57f89efb

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Sep 19 23:17:04 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 3ec8dda7 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/loop1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa56f42f:bb95fbde:11ce620e:878b2b1c
           Name : tucana.caoba.fr:1  (local to host tucana.caoba.fr)
  Creation Time : Mon Sep 19 23:17:04 2016
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3852232703 (1836.89 GiB 1972.34 GB)
     Array Size : 5778347520 (5510.66 GiB 5917.03 GB)
  Used Dev Size : 3852231680 (1836.89 GiB 1972.34 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : 9d42153b:4173aeea:51f41ebc:3789f98a

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Sep 19 23:17:04 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2af1f191 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/loop2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa56f42f:bb95fbde:11ce620e:878b2b1c
           Name : tucana.caoba.fr:1  (local to host tucana.caoba.fr)
  Creation Time : Mon Sep 19 23:17:04 2016
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3852232703 (1836.89 GiB 1972.34 GB)
     Array Size : 5778347520 (5510.66 GiB 5917.03 GB)
  Used Dev Size : 3852231680 (1836.89 GiB 1972.34 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : ada52b0e:f2c4a680:ece59800:6425a9b2

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Sep 19 23:17:04 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2a341a - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
================================================================================

With the help of the initial mdadm --examine , is it possible to
recreate my raid in a way that I can read data out of it?


I also posted the question here:
http://www.unix.com/unix-for-advanced-and-expert-users/268771-how-fix-mistake-raid-mdadm-create-instead-assemble-post302983179.html#post302983179

Regards
-------------------------
Santiago DIEZ
CAOBA Conseil
Paris, France

Re: How to fix mistake on raid: mdadm create instead of assemble?

[Andreas Klauer]

On Fri, Oct 07, 2016 at 05:37:32PM +0200, Santiago DIEZ wrote:
> First thing I did is ddrescue the remaining partitions sd[abc]10 .
> ddrescue did not stumble into any read error so I assume all remaining
> partitions are perfectly safe.

So ... don't you still have a good copy?

You only killed one of them, right? Did not make same mistake twice?

> There comes my mistake: I ran the --create command instead of --assemble :
> 
> ================================================================================
> # mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
> --run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing
> 
> mdadm: layout defaults to left-symmetric
> mdadm: layout defaults to left-symmetric
> mdadm: chunk size defaults to 512K
> mdadm: /dev/loop0 appears to contain an ext2fs file system
>        size=5778741888K  mtime=Sat Sep  3 11:00:22 2016
> mdadm: /dev/loop0 appears to be part of a raid array:
>        level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
> mdadm: /dev/loop1 appears to be part of a raid array:
>        level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
> mdadm: /dev/loop2 appears to be part of a raid array:
>        level=raid5 devices=4 ctime=Wed Jan 25 09:08:11 2012
> mdadm: size set to 1926115840K
> mdadm: automatically enabling write-intent bitmap on large array
> mdadm: creation continuing despite oddities due to --run
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md1 started.
> ================================================================================

You had 0.90 metadata before, that is metadata at the end of the device. 
1.2 metadata is at the start of the device (4K from start). So you have 
overwritten your filesystem superblock...

You can --create again with --metadata=0.90 --chunk=64K options and 
see what is left to the rescue.

But it would be much better if you still had your good copy of the original.

Regards
Andreas Klauer

Re: How to fix mistake on raid: mdadm create instead of assemble?

[Wols Lists]

On 08/10/16 13:30, Andreas Klauer wrote:
> On Fri, Oct 07, 2016 at 05:37:32PM +0200, Santiago DIEZ wrote:
>> > First thing I did is ddrescue the remaining partitions sd[abc]10 .
>> > ddrescue did not stumble into any read error so I assume all remaining
>> > partitions are perfectly safe.
> So ... don't you still have a good copy?
> 
> You only killed one of them, right? Did not make same mistake twice?
> 
>> > There comes my mistake: I ran the --create command instead of --assemble :
>> > 
>> > ================================================================================
>> > # mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
>> > --run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing

One oddity I've noticed. You've created the array using loop devices.
What are these?

The reason I ask is that using loopback devices is a standard technique
for rebuilding a damaged array, specifically to prevent md from actually
writing to the drive. So is it possible that "mdadm --create" only wrote
to ram, and a reboot will recover your ddrescue copies untouched?

My raid-fu isn't enough to tell me whether I'm right or not ... :-)

If necessary you'll have to do another ddrescue from the original
drives, and you should then be able to assemble the array from the
copies. Don't use "missing", use "--force" and you should get a working,
degraded, array to which you can add a new drive and rebuild the array.

mdadm --assemble /dev/md0 /dev/sd[efg]10 --force

if I'm right ... so long as it's the copies, you can always recover
again from the original disks, and if there's a problem with the copies
mdadm should complain when it assembles the array.

Cheers,
Wol

Re: How to fix mistake on raid: mdadm create instead of assemble?

[Santiago DIEZ]

Hi,

Thanks Andreas,

Yes apparently, 3/4 of the original disks seem to be safe. But I'm
terrified at the idea of doing something wrong assembling them.
Incidentally, I indeed did a mistake trying to assemble the ddrescue
images of the 3 safe disks. I tried to create again with proper
metadata and chunck but it did not work. I'm still scared at the idea
of restarting the original raid. I'm currently ddrescuing again the 3
partitions to then try and *assemble* them rather than *create*.


Thanks Wol,

I use loop devices because I work on partition images, not on actual partitions:
I use ddrescue to copy data from /dev/sd[abc]10 to
some.other.server:/home/sd[abc].img
Then I go to some.other.server and turn the images into loop devices :
losetup /dev/loop0 /home/sda10.img
losetup /dev/loop1 /home/sdb10.img
losetup /dev/loop2 /home/sdc10.img
Then I tried to created the raid, it worked but as I said, the
filesystem was unreadable.
I know the idea of using loop devices works because I tested it before.
I'm doing the whole procedure all over again (takes 5 days to ddrescue
the 3 partitions to another server) and then I will use the command
you recommended :
mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1 /dev/loop2 --force


Will keep you posted

-------------------------
Santiago DIEZ
Quark Systems & CAOBA
23 rue du Buisson Saint-Louis, 75010 Paris
-------------------------

On Mon, Oct 10, 2016 at 12:39 AM, Wols Lists <antlists@youngman.org.uk> wrote:
>
> On 08/10/16 13:30, Andreas Klauer wrote:
> > On Fri, Oct 07, 2016 at 05:37:32PM +0200, Santiago DIEZ wrote:
> >> > First thing I did is ddrescue the remaining partitions sd[abc]10 .
> >> > ddrescue did not stumble into any read error so I assume all remaining
> >> > partitions are perfectly safe.
> > So ... don't you still have a good copy?
> >
> > You only killed one of them, right? Did not make same mistake twice?
> >
> >> > There comes my mistake: I ran the --create command instead of --assemble :
> >> >
> >> > ================================================================================
> >> > # mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
> >> > --run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing
>
> One oddity I've noticed. You've created the array using loop devices.
> What are these?
>
> The reason I ask is that using loopback devices is a standard technique
> for rebuilding a damaged array, specifically to prevent md from actually
> writing to the drive. So is it possible that "mdadm --create" only wrote
> to ram, and a reboot will recover your ddrescue copies untouched?
>
> My raid-fu isn't enough to tell me whether I'm right or not ... :-)
>
> If necessary you'll have to do another ddrescue from the original
> drives, and you should then be able to assemble the array from the
> copies. Don't use "missing", use "--force" and you should get a working,
> degraded, array to which you can add a new drive and rebuild the array.
>
> mdadm --assemble /dev/md0 /dev/sd[efg]10 --force
>
> if I'm right ... so long as it's the copies, you can always recover
> again from the original disks, and if there's a problem with the copies
> mdadm should complain when it assembles the array.
>
> Cheers,
> Wol

Re: How to fix mistake on raid: mdadm create instead of assemble?

[Shaohua Li]

On Fri, Oct 21, 2016 at 10:45:10AM +0200, Santiago DIEZ wrote:
> Hi,
> 
> Thanks Andreas,
> 
> Yes apparently, 3/4 of the original disks seem to be safe. But I'm
> terrified at the idea of doing something wrong assembling them.
> Incidentally, I indeed did a mistake trying to assemble the ddrescue
> images of the 3 safe disks. I tried to create again with proper
> metadata and chunck but it did not work. I'm still scared at the idea
> of restarting the original raid. I'm currently ddrescuing again the 3
> partitions to then try and *assemble* them rather than *create*.
> 
> 
> Thanks Wol,
> 
> I use loop devices because I work on partition images, not on actual partitions:
> I use ddrescue to copy data from /dev/sd[abc]10 to
> some.other.server:/home/sd[abc].img
> Then I go to some.other.server and turn the images into loop devices :
> losetup /dev/loop0 /home/sda10.img
> losetup /dev/loop1 /home/sdb10.img
> losetup /dev/loop2 /home/sdc10.img
> Then I tried to created the raid, it worked but as I said, the
> filesystem was unreadable.
> I know the idea of using loop devices works because I tested it before.
> I'm doing the whole procedure all over again (takes 5 days to ddrescue
> the 3 partitions to another server) and then I will use the command
> you recommended :
> mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1 /dev/loop2 --force
> 
> 
> Will keep you posted
> 
> -------------------------
> Santiago DIEZ
> Quark Systems & CAOBA
> 23 rue du Buisson Saint-Louis, 75010 Paris
> -------------------------
> 
> On Mon, Oct 10, 2016 at 12:39 AM, Wols Lists <antlists@youngman.org.uk> wrote:
> >
> > On 08/10/16 13:30, Andreas Klauer wrote:
> > > On Fri, Oct 07, 2016 at 05:37:32PM +0200, Santiago DIEZ wrote:
> > >> > First thing I did is ddrescue the remaining partitions sd[abc]10 .
> > >> > ddrescue did not stumble into any read error so I assume all remaining
> > >> > partitions are perfectly safe.
> > > So ... don't you still have a good copy?
> > >
> > > You only killed one of them, right? Did not make same mistake twice?
> > >
> > >> > There comes my mistake: I ran the --create command instead of --assemble :
> > >> >
> > >> > ================================================================================
> > >> > # mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
> > >> > --run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing
> >
> > One oddity I've noticed. You've created the array using loop devices.
> > What are these?
> >
> > The reason I ask is that using loopback devices is a standard technique
> > for rebuilding a damaged array, specifically to prevent md from actually
> > writing to the drive. So is it possible that "mdadm --create" only wrote
> > to ram, and a reboot will recover your ddrescue copies untouched?
> >
> > My raid-fu isn't enough to tell me whether I'm right or not ... :-)
> >
> > If necessary you'll have to do another ddrescue from the original
> > drives, and you should then be able to assemble the array from the
> > copies. Don't use "missing", use "--force" and you should get a working,
> > degraded, array to which you can add a new drive and rebuild the array.
> >
> > mdadm --assemble /dev/md0 /dev/sd[efg]10 --force
> >
> > if I'm right ... so long as it's the copies, you can always recover
> > again from the original disks, and if there's a problem with the copies
> > mdadm should complain when it assembles the array.

Hmm, those commands work for me. I'm adding Song and Jes if they have ideas.

Thanks,
Shaohua

Re: How to fix mistake on raid: mdadm create instead of assemble?

[Santiago DIEZ]

Fantastic! I ran the following command:

mdadm --assemble /dev/md10 --verbose --force --run /dev/loop0
/dev/loop1 /dev/loop2

and it returned

mdadm: looking for devices for /dev/md10
mdadm: /dev/loop0 is identified as a member of /dev/md10, slot 0.
mdadm: /dev/loop1 is identified as a member of /dev/md10, slot 1.
mdadm: /dev/loop2 is identified as a member of /dev/md10, slot 2.
mdadm: forcing event count in /dev/loop0(0) from 81589 upto 81626
mdadm: forcing event count in /dev/loop2(2) from 81589 upto 81626
mdadm: added /dev/loop1 to /dev/md10 as 1
mdadm: added /dev/loop2 to /dev/md10 as 2
mdadm: no uptodate device for slot 6 of /dev/md10
mdadm: added /dev/loop0 to /dev/md10 as 0
mdadm: /dev/md10 has been started with 3 drives (out of 4).

I can mount /dev/md10 and I have access to my data.

Now that I know my data is safe.
I need to restart the original array on the server.
The array is completely deactivated on the original server.
The company that I rent the dedicated server from is waiting for my GO
to replace disk sdd with a new one.
I'm giving then the GO now.
After I reboot the server, what is it I need to do to rebuild sdd as
the fourth disk of the array?

Regards
-------------------------
Santiago DIEZ
Quark Systems & CAOBA
23 rue du Buisson Saint-Louis, 75010 Paris
-------------------------


On Sat, Oct 22, 2016 at 12:35 AM, Shaohua Li <shli@kernel.org> wrote:
> On Fri, Oct 21, 2016 at 10:45:10AM +0200, Santiago DIEZ wrote:
>> Hi,
>>
>> Thanks Andreas,
>>
>> Yes apparently, 3/4 of the original disks seem to be safe. But I'm
>> terrified at the idea of doing something wrong assembling them.
>> Incidentally, I indeed did a mistake trying to assemble the ddrescue
>> images of the 3 safe disks. I tried to create again with proper
>> metadata and chunck but it did not work. I'm still scared at the idea
>> of restarting the original raid. I'm currently ddrescuing again the 3
>> partitions to then try and *assemble* them rather than *create*.
>>
>>
>> Thanks Wol,
>>
>> I use loop devices because I work on partition images, not on actual partitions:
>> I use ddrescue to copy data from /dev/sd[abc]10 to
>> some.other.server:/home/sd[abc].img
>> Then I go to some.other.server and turn the images into loop devices :
>> losetup /dev/loop0 /home/sda10.img
>> losetup /dev/loop1 /home/sdb10.img
>> losetup /dev/loop2 /home/sdc10.img
>> Then I tried to created the raid, it worked but as I said, the
>> filesystem was unreadable.
>> I know the idea of using loop devices works because I tested it before.
>> I'm doing the whole procedure all over again (takes 5 days to ddrescue
>> the 3 partitions to another server) and then I will use the command
>> you recommended :
>> mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1 /dev/loop2 --force
>>
>>
>> Will keep you posted
>>
>> -------------------------
>> Santiago DIEZ
>> Quark Systems & CAOBA
>> 23 rue du Buisson Saint-Louis, 75010 Paris
>> -------------------------
>>
>> On Mon, Oct 10, 2016 at 12:39 AM, Wols Lists <antlists@youngman.org.uk> wrote:
>> >
>> > On 08/10/16 13:30, Andreas Klauer wrote:
>> > > On Fri, Oct 07, 2016 at 05:37:32PM +0200, Santiago DIEZ wrote:
>> > >> > First thing I did is ddrescue the remaining partitions sd[abc]10 .
>> > >> > ddrescue did not stumble into any read error so I assume all remaining
>> > >> > partitions are perfectly safe.
>> > > So ... don't you still have a good copy?
>> > >
>> > > You only killed one of them, right? Did not make same mistake twice?
>> > >
>> > >> > There comes my mistake: I ran the --create command instead of --assemble :
>> > >> >
>> > >> > ================================================================================
>> > >> > # mdadm --create --verbose /dev/md1 --raid-devices=4 --level=raid5
>> > >> > --run --readonly /dev/loop0 /dev/loop1 /dev/loop2 missing
>> >
>> > One oddity I've noticed. You've created the array using loop devices.
>> > What are these?
>> >
>> > The reason I ask is that using loopback devices is a standard technique
>> > for rebuilding a damaged array, specifically to prevent md from actually
>> > writing to the drive. So is it possible that "mdadm --create" only wrote
>> > to ram, and a reboot will recover your ddrescue copies untouched?
>> >
>> > My raid-fu isn't enough to tell me whether I'm right or not ... :-)
>> >
>> > If necessary you'll have to do another ddrescue from the original
>> > drives, and you should then be able to assemble the array from the
>> > copies. Don't use "missing", use "--force" and you should get a working,
>> > degraded, array to which you can add a new drive and rebuild the array.
>> >
>> > mdadm --assemble /dev/md0 /dev/sd[efg]10 --force
>> >
>> > if I'm right ... so long as it's the copies, you can always recover
>> > again from the original disks, and if there's a problem with the copies
>> > mdadm should complain when it assembles the array.
>
> Hmm, those commands work for me. I'm adding Song and Jes if they have ideas.
>
> Thanks,
> Shaohua