Reassembling RAID1 after good drive was offline [newbie]

Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

I recently made a RAID1 array from a couple of extra usb drives:

$ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2

I'm using it for my backups.  I've run into a small problem though.
One of the drives for some reason hadn't come online when I tried to 
assemble the array.  Trying --assemble a second time after it came online 
apparently wasn't the right thing to do:

$ mdadm --assemble --scan
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ mdadm --assemble --scan
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ cat /proc/mdstat 
Personalities : [raid1] 
md126 : active (auto-read-only) raid1 sdc2[0]
      943587136 blocks super 1.2 [2/1] [U_]
      
md127 : active (auto-read-only) raid1 sdd2[1]
      943587136 blocks super 1.2 [2/1] [_U]


AFAIK both drives are healthy, but since that happened, it refuses to 
assemble them both in the array:

$ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed
mdadm: /dev/md/backup has been started with 1 drive (out of 2).

I haven't mounted the volume, so I believe the drives are in sync.

Is there any way to put the array back together without having to 
resync?  It looks like I want something like --assume-clean, but that 
flag isn't valid in assemble mode.

Please cc responses to me.

Thanks,
Aryeh Leib Taurog

Re: Reassembling RAID1 after good drive was offline [newbie]

[Robert L Mathews]

On 1/1/15 9:40 AM, Aryeh Leib Taurog wrote:

> I recently made a RAID1 array from a couple of extra usb drives:
> 
> $ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2

These are sdc2 and sdd2. Okay.


> Personalities : [raid1] 
> md126 : active (auto-read-only) raid1 sdc2[0]
>       943587136 blocks super 1.2 [2/1] [U_]
>       
> md127 : active (auto-read-only) raid1 sdd2[1]
>       943587136 blocks super 1.2 [2/1] [_U]

Still sdc2 and sdd2, although now in two arrays.


> AFAIK both drives are healthy, but since that happened, it refuses to 
> assemble them both in the array:
> 
> $ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
> mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed

Now you're working on sdb2 and sdd2. Is that intentional? Did sdc2
become sdb2 after a restart or something?


> Is there any way to put the array back together without having to 
> resync?

You should collect more data about what array each partition thinks it's
a member of, etc., before you try anything else. People can probably
help more if you report the output of these to the list:

 mdadm --detail /dev/md*
 mdadm --examine /dev/sd*

-- 
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

Re: Reassembling RAID1 after good drive was offline [newbie]

[Anthonys Lists]

On 01/01/2015 20:54, Robert L Mathews wrote:
>> Personalities : [raid1]
>> >md126 : active (auto-read-only) raid1 sdc2[0]
>> >       943587136 blocks super 1.2 [2/1] [U_]
>> >       
>> >md127 : active (auto-read-only) raid1 sdd2[1]
>> >       943587136 blocks super 1.2 [2/1] [_U]
> Still sdc2 and sdd2, although now in two arrays.
>
>
What version is mdadm? iirc there is a known bug in older versions which 
does this. (Splits one array into two arrays.)

Cheers,
Wol

Re: Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

On Thu, Jan 01, 2015 at 8:54 PM, Robert L Mathews wrote:
>> I recently made a RAID1 array from a couple of extra usb drives:
>> $ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2
>
> These are sdc2 and sdd2. Okay.
>
>> Personalities : [raid1] md126 : active (auto-read-only) raid1 
>> sdc2[0]
>>       943587136 blocks super 1.2 [2/1] [U_]
>>       
>> md127 : active (auto-read-only) raid1 sdd2[1]
>>       943587136 blocks super 1.2 [2/1] [_U]
>
> Still sdc2 and sdd2, although now in two arrays.
>
>> AFAIK both drives are healthy, but since that happened, it refuses 
>> to assemble them both in the array:
>> $ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
>> mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed
>
> Now you're working on sdb2 and sdd2. Is that intentional? Did sdc2
> become sdb2 after a restart or something?

Yes.  They're just two of three external usb disks.  The lettering
depends on the order in which I power up the computer and the drives.
I believe I read that mdadm works even when the device names change.
It did seem to be working before this issue arose.

>> Is there any way to put the array back together without having to 
>> resync?
>
> You should collect more data about what array each partition thinks  
> it's a member of, etc., before you try anything else. People can  
> probably help more if you report the output of these to the list:
> 
> mdadm --detail /dev/md*

I had to assemble the array, but as above, it only includes one device.
While doing this I discovered one of the usb cables is flaky, which 
explains why the device (sdc below) wasn't always coming on line.

$ mdadm --assemble --force /dev/md/backup /dev/sd[cd]2
mdadm: ignoring /dev/sdc2 as it reports /dev/sdd2 as failed
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ mdadm --detail /dev/md/*
/dev/md/backup:
        Version : 1.2
  Creation Time : Wed Dec 17 22:39:10 2014
     Raid Level : raid1
     Array Size : 943587136 (899.87 GiB 966.23 GB)
  Used Dev Size : 943587136 (899.87 GiB 966.23 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Dec 30 08:06:51 2014
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : deb76:backup  (local to host deb76)
           UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
         Events : 382

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       50        1      active sync   /dev/sdd2

> mdadm --examine /dev/sd*

I won't bore you with sda and sdb, there's nothing raid-related
on them.

$ mdadm --examine /dev/sd[cd]{,2}
/dev/sdc:
   MBR Magic : aa55
Partition[0] :     16777216 sectors at        16384 (type 83)
Partition[1] :   1887436800 sectors at     16793600 (type da)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :     16777216 sectors at        16384 (type 83)
Partition[1] :   1887436800 sectors at     16793600 (type da)
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
           Name : deb76:backup  (local to host deb76)
  Creation Time : Wed Dec 17 22:39:10 2014
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1887174656 (899.88 GiB 966.23 GB)
     Array Size : 943587136 (899.87 GiB 966.23 GB)
  Used Dev Size : 1887174272 (899.87 GiB 966.23 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 0347f019:c7911229:bde3bb2e:847aeebc

    Update Time : Sun Dec 28 21:20:46 2014
       Checksum : 7cd44677 - correct
         Events : 38


   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing)
/dev/sdd2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
           Name : deb76:backup  (local to host deb76)
  Creation Time : Wed Dec 17 22:39:10 2014
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1887174656 (899.88 GiB 966.23 GB)
     Array Size : 943587136 (899.87 GiB 966.23 GB)
  Used Dev Size : 1887174272 (899.87 GiB 966.23 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d2c29953:34be093c:00279411:674cd53e

    Update Time : Tue Dec 30 08:06:51 2014
       Checksum : 2e39ee1c - correct
         Events : 382


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)

Re: Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

On Fri, Jan 02, 2015 at 11:02, Anthonys Lists wrote:
>On 01/01/2015 20:54, Robert L Mathews wrote:
>>> >Personalities : [raid1]
>>> >md126 : active (auto-read-only) raid1 sdc2[0]
>>> >       943587136 blocks super 1.2 [2/1] [U_]
>>> >       
>>> >md127 : active (auto-read-only) raid1 sdd2[1]
>>> >       943587136 blocks super 1.2 [2/1] [_U]
>> Still sdc2 and sdd2, although now in two arrays.
>>
>>
> What version is mdadm? iirc there is a known bug in older versions 
> which does this. (Splits one array into two arrays.)

$ mdadm --version
mdadm - v3.2.5 - 18th May 2012

It's the mdadm packaged with Debian wheezy

Re: Reassembling RAID1 after good drive was offline [newbie]

[Robert L Mathews]

On 1/2/15 5:01 AM, Aryeh Leib Taurog wrote:

> While doing this I discovered one of the usb cables is flaky, which 
> explains why the device (sdc below) wasn't always coming on line.

Okay, that's probably the source of the problem, then. One of the times
when you restarted, only the non-flaky one was available. That made md
assemble the array with only that member, increasing the event count on
that partition. So now the partitions are "different" (although perhaps
you didn't change the data on them if you never mounted the array after
the problem started).

Your question is about how to reassemble them without a resync, which is
understandable, but the reason md isn't doing that automatically is that
it thinks they might be different.

I personally would always do a resync in such a case. A flaky cable
suggests that data on it is suspect, and this is one of the things RAID
1 is for: it allows you to copy clean data to suspect partitions.

But I'm known to be data-paranoid. Other people may have different
opinions/suggestions.

-- 
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

Re: Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

On Fri, Jan 02, 2015 at 6:38 PM, Robert L Mathews wrote:
> Your question is about how to reassemble them without a resync,  
> which is understandable, but the reason md isn't doing that 
> automatically is that it thinks they might be different.
 
> I personally would always do a resync in such a case. A flaky cable 
> suggests that data on it is suspect, and this is one of the things  
> RAID 1 is for: it allows you to copy clean data to suspect 
> partitions.

> But I'm known to be data-paranoid. Other people may have different 
> opinions/suggestions.

Thanks.  I tend to agree.  Would the resync just copy all the data 
from the "good" drive back to the "failed" drive?  For diagnostic 
purposes, it would actually be a lot more informative to compare the 
two drives and see if there really is data corruption on one of them 
or not.  Is there a way to do that?  If I were to demonstrate that the 
data are in sync, I would want to reassemble without resync.

Also, in my situation, since for now I'm just using a pair of external 
drives, I could easily imagine accidentally trying to assemble the 
array when one of the drives is powered down.  Then this situation 
would arise again without faulty hardware.  I suppose an 'assemble' 
script could help protect against the latter case by checking first 
that both devices are available.

Prudence notwithstanding, I do think there are valid cases for 
reassembling this array without resync.  If there's a way to do that, 
I'd still like to know.

With appreciation,
Aryeh Leib Taurog

Re: Reassembling RAID1 after good drive was offline [newbie]

[Peter Grandi]

Yet another of an endless (but not too frequent fortunately)
stream of "wildly optimistic" messages to this mailing list...

> Would the resync just copy all the data from the "good" drive
> back to the "failed" drive?

This seems to me quite "imaginative" based on the dream that
resync has psychic powers.

> For diagnostic purposes, it would actually be a lot more
> informative to compare the two drives and see if there really
> is data corruption on one of them or not.

This seems to me "wildly optimistic" that when two blocks differ
it is possible in the general case to determine whether one (and
which one) or both are "corrupted".

> Is there a way to do that?

'man cmp' may show a way to "compare the two drives".

> If I were to demonstrate that the data are in sync, I would
> want to reassemble without resync.

"Fantastic logic".

> Also, in my situation, since for now I'm just using a pair of
> external drives, I could easily imagine accidentally trying to
> assemble the array when one of the drives is powered down.
> Then this situation would arise again without faulty hardware.

This may be based on the "amazing insight" that differences in
content and event counts are the same as data corruption.

> Prudence notwithstanding, I do think there are valid cases for 
> reassembling this array without resync.

I believe that you think that but I also believe the manual when
it says "Use this only if you really know what you are doing";
because many MD users may not have the level of skills and
insight about RAID that you think you have, MD is designed to
protect users by default from their own "amazing optimism",
especially users that haven't read yet:

  https://raid.wiki.kernel.org/index.php/RAID_Recovery
  https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID

> If there's a way to do that, I'd still like to know.

'man mdadm' may show a way to assemble "without resync".

Re: Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

On Sun, 4 Jan 2015 at 11:10 Peter Grandi wrote:
> Yet another of an endless (but not too frequent fortunately)
> stream of "wildly optimistic" messages to this mailing list...

No intent to offend.  I specifically put "newbie" in the subject.

>> Would the resync just copy all the data from the "good" drive
>> back to the "failed" drive?
>
> This seems to me quite "imaginative" based on the dream that
> resync has psychic powers.

I am not sure what you mean.  Two drives in a RAID1 array.  At one 
point, one drive failed to come on line.  Now mdadm refuses to include 
that drive in the array.  So there's the "good" drive, which appears 
in the now degraded array, and the "failed" drive, which does not.  I 
have never done a resync, and I haven't seen a detailed description of 
what it does, but given that mdadm seems to have decided which drive 
is good and which not, and assuming mdadm doesn't know anything about 
the contents of the data, what is so "imaginative" about the notion 
that if I add the "failed" drive to the array, it would simply copy 
all the data on the "good" drive byte-by-byte onto the "failed" drive, 
overwriting whatever is currently on the "failed" drive?  I can't 
imagine how else a resync would work.  What am I missing?

>> For diagnostic purposes, it would actually be a lot more
>> informative to compare the two drives and see if there really
>> is data corruption on one of them or not.
>
> This seems to me "wildly optimistic" that when two blocks differ
> it is possible in the general case to determine whether one (and
> which one) or both are "corrupted".

I was only referring to my specific case.  One drive was found to have 
faulty hardware, one not.  The one with the faulty hardware is 
suspect, the other not.  If the two differ, then I, perhaps naively, 
would assume that the suspect drive experienced data corruption and 
the other not.  To my mind, whether they differ or not could indicate 
something about the condition of the hardware and may have potentially 
useful implications regarding data on drives previously used (without 
MD) in the suspect hardware.  If I'm wrong, I'd be thrilled to learn.

>> Is there a way to do that?
>
> 'man cmp' may show a way to "compare the two drives".

I am quite familiar with the unix toolset.  But I don't know enough 
about mdadm.  Where, for example, is the superblock, and where do my 
data begin?  I gather the superblock is expected to differ, since each 
device has a UUID and distinct event count.  I read the mdadm man 
page, but it doesn't seem to discuss implementation details such as 
this.  So if there's no mdadm-specific approach, it would help to know 
where on the device I could find the data that I should expect to 
match, or alternatively, why I should not interest myself in such a 
comparison.

>> If I were to demonstrate that the data are in sync, I would
>> want to reassemble without resync.
>
> "Fantastic logic".

Please help me understand.  It's a RAID1 array.  Doesn't that mean the 
devices are supposed to have identical copies of the data?  And if I 
can demonstrate that they in fact are identical, why would a resync be 
necessary?  I am sure I am missing something.  Please clue me in.

>> Also, in my situation, since for now I'm just using a pair of
>> external drives, I could easily imagine accidentally trying to
>> assemble the array when one of the drives is powered down.
>> Then this situation would arise again without faulty hardware.
>
> This may be based on the "amazing insight" that differences in
> content and event counts are the same as data corruption.

Again, I fail to comprehend.  If anything, I assumed the opposite.  It 
was my understanding that a difference in event counts could very 
easily arise without data corruption, as could differences in content.  
It also seems obvious that data corruption could occur in ways that 
would affect all drives in an array.  But I don't understand how any 
of this is relevant.

>> Prudence notwithstanding, I do think there are valid cases for 
>> reassembling this array without resync.
>
> I believe that you think that but I also believe the manual when
> it says "Use this only if you really know what you are doing";
> because many MD users may not have the level of skills and
> insight about RAID that you think you have, MD is designed to
> protect users by default from their own "amazing optimism",
> especially users that haven't read yet:
>
>  https://raid.wiki.kernel.org/index.php/RAID_Recovery
>  https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID

I don't think I have any skill or insight about RAID.  I would like to 
learn and understand.  I have read the linked material, as well as the 
mdadm man page a few times, but it seems to me there are some basic 
concepts which aren't explained clearly.  It does say that when in 
doubt one should contact "the friendly and helpful folk on the 
linux-raid mailing list."

>> If there's a way to do that, I'd still like to know.
>
> 'man mdadm' may show a way to assemble "without resync".

I didn't find it.  As I said, it seems to me one needs much more 
experience and understanding of MD than I have to understand most of 
that document.  There is an indication in the RAID_Recovery wiki page 
that I can reissue the original --create command with --assume-clean, 
but there's more discussion about why it might not work than about 
what it actually does.  It also encourages contacting the mailing list 
first, which I have done, hoping to gain some insight.

I'd like to think you didn't mean to be offensive, condescending, or 
hyperbolic, and I'm not trying to flame, but I'm feeling very 
defensive after reading your message, and none the wiser.  That may 
have come out in my response, despite my best efforts.  I read what I 
could find about MD before posting, but it seems to me there's some 
basic information which isn't documented in the places I've looked.  I 
tried searching the mailing list, but finding a needle in a haystack 
is much harder when you don't know what a needle looks like.

With appreciation,
Aryeh Leib Taurog

Please cc me on your responses, I'm not subscribed to the list and 
this thread keeps disappearing from the archives at marc.info

Re: Reassembling RAID1 after good drive was offline [newbie]

[Wols Lists]

On 04/01/15 21:07, Aryeh Leib Taurog wrote:
> On Sun, 4 Jan 2015 at 11:10 Peter Grandi wrote:
>> Yet another of an endless (but not too frequent fortunately)
>> stream of "wildly optimistic" messages to this mailing list...
> 
> No intent to offend.  I specifically put "newbie" in the subject.
> 
>>> Would the resync just copy all the data from the "good" drive
>>> back to the "failed" drive?
>>
>> This seems to me quite "imaginative" based on the dream that
>> resync has psychic powers.
> 
> I am not sure what you mean.  Two drives in a RAID1 array.  At one 
> point, one drive failed to come on line.  Now mdadm refuses to include 
> that drive in the array.  So there's the "good" drive, which appears 
> in the now degraded array, and the "failed" drive, which does not.  I 
> have never done a resync, and I haven't seen a detailed description of 
> what it does, but given that mdadm seems to have decided which drive 
> is good and which not, and assuming mdadm doesn't know anything about 
> the contents of the data, what is so "imaginative" about the notion 
> that if I add the "failed" drive to the array, it would simply copy 
> all the data on the "good" drive byte-by-byte onto the "failed" drive, 
> overwriting whatever is currently on the "failed" drive?  I can't 
> imagine how else a resync would work.  What am I missing?

That mirroring isn't fault-tolerant-raid? I know it's been given a raid
classification, but raids 1 and 0 really just give you a bigger faster
disk. It's only the other raids that have any error correction ability,
because they use parity etc to be able to tell which set of data is correct.
> 
>>> For diagnostic purposes, it would actually be a lot more
>>> informative to compare the two drives and see if there really
>>> is data corruption on one of them or not.
>>
>> This seems to me "wildly optimistic" that when two blocks differ
>> it is possible in the general case to determine whether one (and
>> which one) or both are "corrupted".
> 
> I was only referring to my specific case.  One drive was found to have 
> faulty hardware, one not.  The one with the faulty hardware is 
> suspect, the other not.  If the two differ, then I, perhaps naively, 
> would assume that the suspect drive experienced data corruption and 
> the other not.  To my mind, whether they differ or not could indicate 
> something about the condition of the hardware and may have potentially 
> useful implications regarding data on drives previously used (without 
> MD) in the suspect hardware.  If I'm wrong, I'd be thrilled to learn.
> 
>>> Is there a way to do that?
>>
>> 'man cmp' may show a way to "compare the two drives".
> 
> I am quite familiar with the unix toolset.  But I don't know enough 
> about mdadm.  Where, for example, is the superblock, and where do my 
> data begin?  I gather the superblock is expected to differ, since each 
> device has a UUID and distinct event count.  I read the mdadm man 
> page, but it doesn't seem to discuss implementation details such as 
> this.  So if there's no mdadm-specific approach, it would help to know 
> where on the device I could find the data that I should expect to 
> match, or alternatively, why I should not interest myself in such a 
> comparison.
> 
>>> If I were to demonstrate that the data are in sync, I would
>>> want to reassemble without resync.
>>
>> "Fantastic logic".
> 
> Please help me understand.  It's a RAID1 array.  Doesn't that mean the 
> devices are supposed to have identical copies of the data?  And if I 
> can demonstrate that they in fact are identical, why would a resync be 
> necessary?  I am sure I am missing something.  Please clue me in.

They are supposed to be identical. But since the raid came up with one
drive missing, the other drive may have been modified, so they're not
identical any more. And the raid array has NO WAY of knowing which
version is "correct". Sorry :-( but that's mirrors for you ...
> 
>>> Also, in my situation, since for now I'm just using a pair of
>>> external drives, I could easily imagine accidentally trying to
>>> assemble the array when one of the drives is powered down.
>>> Then this situation would arise again without faulty hardware.
>>
>> This may be based on the "amazing insight" that differences in
>> content and event counts are the same as data corruption.
> 
> Again, I fail to comprehend.  If anything, I assumed the opposite.  It 
> was my understanding that a difference in event counts could very 
> easily arise without data corruption, as could differences in content.  
> It also seems obvious that data corruption could occur in ways that 
> would affect all drives in an array.  But I don't understand how any 
> of this is relevant.

If there's differences in event counts and content, then the drives are
no longer identical. Reassemble the array, without a resync, and things
could get complicated (and nasty) VERY quickly :-(
> 
>>> Prudence notwithstanding, I do think there are valid cases for 
>>> reassembling this array without resync.
>>
>> I believe that you think that but I also believe the manual when
>> it says "Use this only if you really know what you are doing";
>> because many MD users may not have the level of skills and
>> insight about RAID that you think you have, MD is designed to
>> protect users by default from their own "amazing optimism",
>> especially users that haven't read yet:
>>
>>  https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>  https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> 
> I don't think I have any skill or insight about RAID.  I would like to 
> learn and understand.  I have read the linked material, as well as the 
> mdadm man page a few times, but it seems to me there are some basic 
> concepts which aren't explained clearly.  It does say that when in 
> doubt one should contact "the friendly and helpful folk on the 
> linux-raid mailing list."
> 
>>> If there's a way to do that, I'd still like to know.
>>
>> 'man mdadm' may show a way to assemble "without resync".
> 
> I didn't find it.  As I said, it seems to me one needs much more 
> experience and understanding of MD than I have to understand most of 
> that document.  There is an indication in the RAID_Recovery wiki page 
> that I can reissue the original --create command with --assume-clean, 
> but there's more discussion about why it might not work than about 
> what it actually does.  It also encourages contacting the mailing list 
> first, which I have done, hoping to gain some insight.

The problem with "--create --assume-clean", is that if the array isn't
clean (and in this case it isn't - we don't know what is different
between the drives but *something* is), then you're asking for trouble.
> 
> I'd like to think you didn't mean to be offensive, condescending, or 
> hyperbolic, and I'm not trying to flame, but I'm feeling very 
> defensive after reading your message, and none the wiser.  That may 
> have come out in my response, despite my best efforts.  I read what I 
> could find about MD before posting, but it seems to me there's some 
> basic information which isn't documented in the places I've looked.  I 
> tried searching the mailing list, but finding a needle in a haystack 
> is much harder when you don't know what a needle looks like.
> 
I know it's hard if you don't know what a needle looks like, but in this
case I think the analogy is very apt - if you go looking you are going
to stab yourself with it.

I think you are going to have to bite the bullet and resync. Maybe keep
the existing drive as a backup, get a new drive, and resync to that.
Make that you backup regime - once a month or whatever swap one drive
over as a backup.

The alternative is get another drive and go raid-5 - but see all the
threads warning about using raid 5 with consumer drives - you need to
tweak your system settings otherwise transient faults can break your system.

And at the end of the day, a resync may take time, but it won't have
much impact on you using your system. I've got mirrored 3TB drives, and
the resync when I brought the array up (I moved a non-mirrored system on
to one drive "--create /dev/disk1 missing" then added the other) just
ticked over in the background.

Cheers,
Wol

Re: Reassembling RAID1 after good drive was offline [newbie]

[Robert L Mathews]

On 1/4/15 1:45 PM, Wols Lists wrote:

> It's only the other raids that have any error correction ability,
> because they use parity etc to be able to tell which set of data is correct.

In theory, yes, but in practice, they don't -- or do they? I thought
that even if you use RAID 6, md doesn't check all members on each read
request, calculate parities, and mask silent platter-level corruption by
ignoring a member spewing bad bits. It just uses parities to fill in
missing data if a member fails.

Or does it actually do error detection, and I'm confused?

-- 
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

Re: Reassembling RAID1 after good drive was offline [newbie]

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

On Sun, 04 Jan 2015 21:45:22 +0000 Wols Lists <antlists@youngman.org.uk>
wrote:

> On 04/01/15 21:07, Aryeh Leib Taurog wrote:
> > On Sun, 4 Jan 2015 at 11:10 Peter Grandi wrote:
> >> Yet another of an endless (but not too frequent fortunately)
> >> stream of "wildly optimistic" messages to this mailing list...
> > 
> > No intent to offend.  I specifically put "newbie" in the subject.
> > 
> >>> Would the resync just copy all the data from the "good" drive
> >>> back to the "failed" drive?
> >>
> >> This seems to me quite "imaginative" based on the dream that
> >> resync has psychic powers.
> > 
> > I am not sure what you mean.  Two drives in a RAID1 array.  At one 
> > point, one drive failed to come on line.  Now mdadm refuses to include 
> > that drive in the array.  So there's the "good" drive, which appears 
> > in the now degraded array, and the "failed" drive, which does not.  I 
> > have never done a resync, and I haven't seen a detailed description of 
> > what it does, but given that mdadm seems to have decided which drive 
> > is good and which not, and assuming mdadm doesn't know anything about 
> > the contents of the data, what is so "imaginative" about the notion 
> > that if I add the "failed" drive to the array, it would simply copy 
> > all the data on the "good" drive byte-by-byte onto the "failed" drive, 
> > overwriting whatever is currently on the "failed" drive?  I can't 
> > imagine how else a resync would work.  What am I missing?
> 
> That mirroring isn't fault-tolerant-raid? I know it's been given a raid
> classification, but raids 1 and 0 really just give you a bigger faster
> disk. It's only the other raids that have any error correction ability,
> because they use parity etc to be able to tell which set of data is correct.

Bzzt.  You lose :-)

Of course RAID1 is fault tolerant!! (I agree that RAID0 isn't).

It cannot tolerate every conceivable fault (e.g. asteroid impact), but it
tolerates most single hardware faults.

The fault is detected by the drive, possibly using a CRC, or by the
controller (hmm.. the drive isn't responding, must be faulty!) and this fault
is communicated to md.  md then manages the fault by accesses the other
device.

*No* RAID level has error detection ability - *all* RAID levels (except zero)
have error correction - providing something else detects the error.
Parity vs mirroring makes no difference here.

(All non-zero RAID levels could try to detect faults by reading all blocks
in a stripe and comparing, but there is no threat-model which makes this a
worthwhile practice)


And to answer the original question: just let it resync.  Had you started
that when you asked the question it would be done by now :-).

To avoid similar problems in future:
 - use a newer mdadm (sorry, but there are bugs sometimes)
 - add an internal write-intent bitmap.  That makes the resync much faster
   when needed
 - Possibly as '--no-degraded' when assembling arrays.

NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Reassembling RAID1 after good drive was offline [newbie]

[Aryeh Leib Taurog]

On Tue, Jan 06, 2015 at 07:54:00AM +1300, NeilBrown wrote:
> The fault is detected by the drive, possibly using a CRC, or by the
> controller (hmm.. the drive isn't responding, must be faulty!) and this fault
> is communicated to md.  md then manages the fault by accesses the other
> device.

I imagine that RAID does introduce a risk here.  If the drive is fine 
and the other hardware isn't, one really could end up with disparate 
data.  I have had situations (perhaps related to the flaky cable I 
recently discovered) where I wrote "A" to the drive and then read back 
"B."  On a single drive system, it's possible to confirm a write by 
reading it, but with RAID that is not the case.  So RAID increases my 
safety against drive failure at the expense of increased reliance on 
the other hardware.  (Sorry if this is all obvious, I'm new to RAID 
and trying to get this clear for myself.)

My current procedure is to make a backup with dar, test the archive, 
and then generate par2 recovery files.  The par2 files give me some 
protection against data corruption, but that only helps if I can rely 
on the initial test.  So I guess my options are to use more reliable 
hardware and/or to backup first to a single drive, on which I test and 
generate recovery files, and then copy to RAID device.

> *No* RAID level has error detection ability - *all* RAID levels (except zero)
> have error correction - providing something else detects the error.
> Parity vs mirroring makes no difference here.
> 
> And to answer the original question: just let it resync.

Thank you all for your answers.

I still didn't get clear confirmation about what resync does, though.  
I understand that md doesn't have any way of knowing *which* drive is 
the "correct" drive, but it *has* decided somehow from which to 
assemble the array and which to ignore.  I am assuming that the 
procedure is to add it back with (the man page implies that -a would 
have the same effect in this case)

    mdadm /dev/md/backup --re-add /dev/sdc2

and that md will 'resync' i.e. make a byte-for-byte copy of the first 
device back onto the second device.  I thought this was obvious, but 
several objections were raised so I'm not entirely sure any more.  Is 
it any more or less than that?

I'm also not sure, given the objections about md not having psychic 
powers, how exactly md did decide which device to include and which to 
ignore, and I'm puzzled by this message:

    mdadm: ignoring /dev/sdc2 as it reports /dev/sdd2 as failed

The logic seems reversed to me.  Is this just the artifact of a 
possibly buggy version of md, or am I missing something here also?

> Had you started that when you asked the question it would be done by 
> now :-)

Of course, but the goal was to learn more about what I'm doing, not to 
save time.  I usually favor a deeper understanding over expedience.

> To avoid similar problems in future:
>  - use a newer mdadm (sorry, but there are bugs sometimes)
>  - add an internal write-intent bitmap.  That makes the resync much faster
>    when needed
>  - Possibly as '--no-degraded' when assembling arrays.

Thanks.  This is all quite helpful.