hung grow

hung grow

[Curt]

Hello all,

So, I got my raid6 is a half fucked state you could say. My raid6 lost
3 drives and then starting throwing I/O errors on the mount point.  If
I tried to restart with the good drives, I got too few to start raid.
So one that got marked faulty, almost matched the even count of the
others, only off by a few.  So I did a assemble force, which seemed to
work and I could see my data.  I should have just pulled it off at
that point and saved what I could, but no.

So I replaced 2 of the bad drives and added them to the raid, it went
through recovery, but only marked the 2 new drives as spares and
showed the bad removed, 2 spares, and it marked one faulty aging, see
the below.

uname -a
Linux dev 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux

mdadm --detail /dev/md127
/dev/md127:
           Version : 0.90
     Creation Time : Fri Jun 15 15:52:05 2012
        Raid Level : raid6
        Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
     Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
      Raid Devices : 7
     Total Devices : 7
   Preferred Minor : 127
       Persistence : Superblock is persistent

       Update Time : Tue Oct  3 22:22:06 2017
             State : clean, FAILED
    Active Devices : 4
   Working Devices : 6
    Failed Devices : 1
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

              UUID : 714a612d:9bd35197:36c91ae3:c168144d
            Events : 0.11559613

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       3       8        1        3      active sync   /dev/sda1
       -       0        0        4      removed
       -       0        0        5      removed
       -       0        0        6      removed

       7       8       80        -      spare   /dev/sdf
       8       8       16        -      spare   /dev/sdb
       9       8       65        -      faulty   /dev/sde1

after several tries to reassemble, the spares wouldn't got active.  So
on the advice of somenone, I set the raid to grow to 8, the theory
being it would make on spare active. Which somewhat worked, but grow
froze at 0% and when I do a detail on md127 it just hangs, it returned
once when this first started and showed the spare in spare rebuilding
status, but sync_action showed reshape and mdstat.


examine returns this, it's the same for all as far as I can see
mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 10:10:37 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71846f - correct
         Events : 11559679

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8        1        3      active sync   /dev/sda1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed


Is my raid completely fucked or can I still recover some data with
doing the create assume clean?

Cheers,
Curt

Re: hung grow

[Anthony Youngman]

On 04/10/17 18:18, Curt wrote:
> Is my raid completely fucked or can I still recover some data with
> doing the create assume clean?

PLEASE PLEASE PLEASE DON'T !!!!!!

I take it you haven't read the raid wiki?

https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn

The bad news is your array is well borked. The good news is I don't 
think you have - YET - managed to bork it irretrievably. A create will 
almost certainly trash it beyond recovery!!!

I think we can stop/revert the grow, and get the array back to a usable 
state, where we can force an assemble. If a bit of data gets lost, sorry.

Do you have spare SATA ports? So you have the bad drives you replaced 
(can you ddrescue them on to new drives?). What was the original 
configuration of the raid - you say you lost three drives, but how many 
did you have to start with?

I'll let the experts talk you through the actual recovery, but the steps 
need to be to revert the grow, ddrescue the best of your failed drives, 
force an assembly, and then replace the other two failed drives. No 
guarantees as to how much data will be left at the end, although 
hopefully we'll save most of it.

Cheers,
Wol

Re: hung grow

[Curt]

Hi,

I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery

I don't have any spare bays on that server...I'd have to make a trip
to my datacenter and bring the drives back to my house.  The bad thing
is the 2 drives I replaced, failed a while ago, so they were behind.
I was hoping I could still use the 4 drives I had before I did a grow
on them.  Do they need to be up-to-date or do I just need the config
from them to recover the 3 drives that were still good?

Oh, I originally started with 7, 2 failed a few moths back and the 3rd
one just recently. FML

Cheers,
Curt

On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 18:18, Curt wrote:
>>
>> Is my raid completely fucked or can I still recover some data with
>> doing the create assume clean?
>
>
> PLEASE PLEASE PLEASE DON'T !!!!!!
>
> I take it you haven't read the raid wiki?
>
> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>
> The bad news is your array is well borked. The good news is I don't think
> you have - YET - managed to bork it irretrievably. A create will almost
> certainly trash it beyond recovery!!!
>
> I think we can stop/revert the grow, and get the array back to a usable
> state, where we can force an assemble. If a bit of data gets lost, sorry.
>
> Do you have spare SATA ports? So you have the bad drives you replaced (can
> you ddrescue them on to new drives?). What was the original configuration of
> the raid - you say you lost three drives, but how many did you have to start
> with?
>
> I'll let the experts talk you through the actual recovery, but the steps
> need to be to revert the grow, ddrescue the best of your failed drives,
> force an assembly, and then replace the other two failed drives. No
> guarantees as to how much data will be left at the end, although hopefully
> we'll save most of it.
>
> Cheers,
> Wol

Re: hung grow

[Joe Landman]

On 10/04/2017 02:16 PM, Curt wrote:
> Hi,
>
> I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery
>
> I don't have any spare bays on that server...I'd have to make a trip
> to my datacenter and bring the drives back to my house.  The bad thing
> is the 2 drives I replaced, failed a while ago, so they were behind.
> I was hoping I could still use the 4 drives I had before I did a grow
> on them.  Do they need to be up-to-date or do I just need the config
> from them to recover the 3 drives that were still good?
>
> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
> one just recently. FML

Er ... honestly, I hope you have a backup.

If the drives are really dead, and can't be seen with lsscsi or cat 
/proc/scsi/scsi , then your raid is probably gone.

If they can be seen, the ddrescue is your best option right now.

Do not grow the system.  Stop that.  Do nothing that changes metadata.

You may (remotely possibly) recover if you can copy the "dead" drives to 
two new live ones.

>
> Cheers,
> Curt
>
> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 18:18, Curt wrote:
>>> Is my raid completely fucked or can I still recover some data with
>>> doing the create assume clean?
>>
>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>
>> I take it you haven't read the raid wiki?
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>
>> The bad news is your array is well borked. The good news is I don't think
>> you have - YET - managed to bork it irretrievably. A create will almost
>> certainly trash it beyond recovery!!!
>>
>> I think we can stop/revert the grow, and get the array back to a usable
>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>
>> Do you have spare SATA ports? So you have the bad drives you replaced (can
>> you ddrescue them on to new drives?). What was the original configuration of
>> the raid - you say you lost three drives, but how many did you have to start
>> with?
>>
>> I'll let the experts talk you through the actual recovery, but the steps
>> need to be to revert the grow, ddrescue the best of your failed drives,
>> force an assembly, and then replace the other two failed drives. No
>> guarantees as to how much data will be left at the end, although hopefully
>> we'll save most of it.
>>
>> Cheers,
>> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
  

Re: hung grow

[Curt]

Hi Joe,

To clarify, the drives aren't completely dead.  I can see/examine all
the drives currently in the array, the older ones I could see/examine,
but like I said they had been marked faulty for a while and event
count was way low.  The grow never went anywhere, just stayed at 0%
with 100% CPU usage on md127_raid process.  I have rebooted and am not
currently touching the drives.

Assuming I can do a dd on one of my failed drives, will I be able to
recover the data that's on the 4 that were good, before I took bad
advice?  Also, will I need to dd on the failed drives or can I do 2 of
the 3?

On Wed, Oct 4, 2017 at 2:29 PM, Joe Landman <joe.landman@gmail.com> wrote:
>
>
> On 10/04/2017 02:16 PM, Curt wrote:
>>
>> Hi,
>>
>> I was reading this one
>> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>
>> I don't have any spare bays on that server...I'd have to make a trip
>> to my datacenter and bring the drives back to my house.  The bad thing
>> is the 2 drives I replaced, failed a while ago, so they were behind.
>> I was hoping I could still use the 4 drives I had before I did a grow
>> on them.  Do they need to be up-to-date or do I just need the config
>> from them to recover the 3 drives that were still good?
>>
>> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
>> one just recently. FML
>
>
> Er ... honestly, I hope you have a backup.
>
> If the drives are really dead, and can't be seen with lsscsi or cat
> /proc/scsi/scsi , then your raid is probably gone.
>
> If they can be seen, the ddrescue is your best option right now.
>
> Do not grow the system.  Stop that.  Do nothing that changes metadata.
>
> You may (remotely possibly) recover if you can copy the "dead" drives to two
> new live ones.
>
>>
>> Cheers,
>> Curt
>>
>> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
>> <antlists@youngman.org.uk> wrote:
>>>
>>> On 04/10/17 18:18, Curt wrote:
>>>>
>>>> Is my raid completely fucked or can I still recover some data with
>>>> doing the create assume clean?
>>>
>>>
>>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>>
>>> I take it you haven't read the raid wiki?
>>>
>>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>>
>>> The bad news is your array is well borked. The good news is I don't think
>>> you have - YET - managed to bork it irretrievably. A create will almost
>>> certainly trash it beyond recovery!!!
>>>
>>> I think we can stop/revert the grow, and get the array back to a usable
>>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>>
>>> Do you have spare SATA ports? So you have the bad drives you replaced
>>> (can
>>> you ddrescue them on to new drives?). What was the original configuration
>>> of
>>> the raid - you say you lost three drives, but how many did you have to
>>> start
>>> with?
>>>
>>> I'll let the experts talk you through the actual recovery, but the steps
>>> need to be to revert the grow, ddrescue the best of your failed drives,
>>> force an assembly, and then replace the other two failed drives. No
>>> guarantees as to how much data will be left at the end, although
>>> hopefully
>>> we'll save most of it.
>>>
>>> Cheers,
>>> Wol
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> Joe Landman
> e: joe.landman@gmail.com
> t: @hpcjoe
> w: https://scalability.org
> g: https://github.com/joelandman
>

Re: hung grow

[Joe Landman]

On 10/04/2017 02:37 PM, Curt wrote:
> Hi Joe,
>
> To clarify, the drives aren't completely dead.  I can see/examine all
> the drives currently in the array, the older ones I could see/examine,
> but like I said they had been marked faulty for a while and event
> count was way low.  The grow never went anywhere, just stayed at 0%
> with 100% CPU usage on md127_raid process.  I have rebooted and am not
> currently touching the drives.
>
> Assuming I can do a dd on one of my failed drives, will I be able to
> recover the data that's on the 4 that were good, before I took bad
> advice?  Also, will I need to dd on the failed drives or can I do 2 of
> the 3?

Not sure.   You will need to try to get back as much as you can off the 
other original "bad" drives.  If those drives are not "bad", you can 
pull out the "new" drives, and put them in.  See if you can force an 
assembly of the RAID.  If that works, you may have data (if the grow 
didn't corrupt anything).

If this is the case, the very first thing you should do is find and copy 
the data that you cannot lose from those drives, to another location, 
quickly.

Before you take any more advice, I'd recommend seeing if you can 
actually recover what you have now.

Generally speaking 3 failed drives on a RAID6 is a dead RAID6.  You may 
get lucky, in that this may have been simply a timeout error (I've seen 
these on consumer grade drives), or an internal operation on the drive 
taking longer than normal, and been booted.  In which case, you'll get 
scary warning messages, but might get your data back.

Under no circumstances do anything to change RAID metadata right now 
(grow, shrink, etc.).  Start with basic assembly.  If you can do that, 
you are in good shape.  If you can't, recovery is unlikely, even with 
heroic intervention.

>
> On Wed, Oct 4, 2017 at 2:29 PM, Joe Landman <joe.landman@gmail.com> wrote:
>>
>> On 10/04/2017 02:16 PM, Curt wrote:
>>> Hi,
>>>
>>> I was reading this one
>>> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>>
>>> I don't have any spare bays on that server...I'd have to make a trip
>>> to my datacenter and bring the drives back to my house.  The bad thing
>>> is the 2 drives I replaced, failed a while ago, so they were behind.
>>> I was hoping I could still use the 4 drives I had before I did a grow
>>> on them.  Do they need to be up-to-date or do I just need the config
>>> from them to recover the 3 drives that were still good?
>>>
>>> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
>>> one just recently. FML
>>
>> Er ... honestly, I hope you have a backup.
>>
>> If the drives are really dead, and can't be seen with lsscsi or cat
>> /proc/scsi/scsi , then your raid is probably gone.
>>
>> If they can be seen, the ddrescue is your best option right now.
>>
>> Do not grow the system.  Stop that.  Do nothing that changes metadata.
>>
>> You may (remotely possibly) recover if you can copy the "dead" drives to two
>> new live ones.
>>
>>> Cheers,
>>> Curt
>>>
>>> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
>>> <antlists@youngman.org.uk> wrote:
>>>> On 04/10/17 18:18, Curt wrote:
>>>>> Is my raid completely fucked or can I still recover some data with
>>>>> doing the create assume clean?
>>>>
>>>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>>>
>>>> I take it you haven't read the raid wiki?
>>>>
>>>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>>>
>>>> The bad news is your array is well borked. The good news is I don't think
>>>> you have - YET - managed to bork it irretrievably. A create will almost
>>>> certainly trash it beyond recovery!!!
>>>>
>>>> I think we can stop/revert the grow, and get the array back to a usable
>>>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>>>
>>>> Do you have spare SATA ports? So you have the bad drives you replaced
>>>> (can
>>>> you ddrescue them on to new drives?). What was the original configuration
>>>> of
>>>> the raid - you say you lost three drives, but how many did you have to
>>>> start
>>>> with?
>>>>
>>>> I'll let the experts talk you through the actual recovery, but the steps
>>>> need to be to revert the grow, ddrescue the best of your failed drives,
>>>> force an assembly, and then replace the other two failed drives. No
>>>> guarantees as to how much data will be left at the end, although
>>>> hopefully
>>>> we'll save most of it.
>>>>
>>>> Cheers,
>>>> Wol
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> Joe Landman
>> e: joe.landman@gmail.com
>> t: @hpcjoe
>> w: https://scalability.org
>> g: https://github.com/joelandman
>>

-- 
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman

Re: hung grow

[Anthony Youngman]

On 04/10/17 19:44, Joe Landman wrote:
> Generally speaking 3 failed drives on a RAID6 is a dead RAID6.  You may 
> get lucky, in that this may have been simply a timeout error (I've seen 
> these on consumer grade drives), or an internal operation on the drive 
> taking longer than normal, and been booted.  In which case, you'll get 
> scary warning messages, but might get your data back.
> 
> Under no circumstances do anything to change RAID metadata right now 
> (grow, shrink, etc.).  Start with basic assembly.  If you can do that, 
> you are in good shape.  If you can't, recovery is unlikely, even with 
> heroic intervention.

No - Curt needs to stop the grow before anything else will work, I 
think. Fortunately, seeing as it's hung at 0% this shouldn't mess about 
with the data at all.

And you NEED to ddrescue that fifth drive. At which point basic assembly 
should hopefully work fine.

(Note that the grow quite likely failed because the fifth drive errored 
... Oh - and are they raid drives? What are they? You don't have a 
timeout mismatch?)

Cheers,
Wol

Re: hung grow

[Curt]

Ok, thanks.

I'm pretty sure I'll be able to DD from at least one of the failed
drives, as I could still query them before I yanked them.  Assuming I
can DD one of the old drives to one of my new ones.

I'd DDrescue old to new drive. Then do an assemble for force, with a
mix of the dd drives and my old good ones? So if sda/b are new DD'd
drives and sdc/d/e are hosed grow drives, I'd do an assemble force
revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
use my info from the DD drives to assemble the array back to 7 drives?
 Did I understand that right?

Oh and how can I tell if I have a timeout mismatch.  They should be raid drives.

Cheers,
Curt

On Wed, Oct 4, 2017 at 3:01 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 19:44, Joe Landman wrote:
>>
>> Generally speaking 3 failed drives on a RAID6 is a dead RAID6.  You may
>> get lucky, in that this may have been simply a timeout error (I've seen
>> these on consumer grade drives), or an internal operation on the drive
>> taking longer than normal, and been booted.  In which case, you'll get scary
>> warning messages, but might get your data back.
>>
>> Under no circumstances do anything to change RAID metadata right now
>> (grow, shrink, etc.).  Start with basic assembly.  If you can do that, you
>> are in good shape.  If you can't, recovery is unlikely, even with heroic
>> intervention.
>
>
> No - Curt needs to stop the grow before anything else will work, I think.
> Fortunately, seeing as it's hung at 0% this shouldn't mess about with the
> data at all.
>
> And you NEED to ddrescue that fifth drive. At which point basic assembly
> should hopefully work fine.
>
> (Note that the grow quite likely failed because the fifth drive errored ...
> Oh - and are they raid drives? What are they? You don't have a timeout
> mismatch?)
>
> Cheers,
> Wol

Re: hung grow

[Anthony Youngman]

On 04/10/17 20:09, Curt wrote:
> Ok, thanks.
> 
> I'm pretty sure I'll be able to DD from at least one of the failed
> drives, as I could still query them before I yanked them.  Assuming I
> can DD one of the old drives to one of my new ones.
> 
> I'd DDrescue old to new drive. Then do an assemble for force, with a
> mix of the dd drives and my old good ones? So if sda/b are new DD'd
> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
> use my info from the DD drives to assemble the array back to 7 drives?
>   Did I understand that right?

This sounds like you need to take a great big step backwards, and make 
sure you understand EXACTLY what is going on. We have a mix of good 
drives, copies of bad drives, and an array that doesn't know whether it 
is supposed to have 7 or 9 drives. One wrong step and your array will be 
toast.

You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.

But I think the first thing we need to do, is to wait for an expert like 
Phil to chime in and sort out that reshape. Your four good drives all 
think they are part of a 9-drive array. Your first two drives to fail 
think they are part of a 7-drive array. Does the third drive think it's 
part of a 7-drive or 9-drive array?

Can you do a --examine on this drive? I suspect the grow blew up because 
it couldn't access this drive. I this drive thinks it is part of a 
7-drive array, we have a bit of a problem on our hands.

I'm hoping it thinks it's part of a 9-drive array - I think we may be 
able to get out of this ...
> 
> Oh and how can I tell if I have a timeout mismatch.  They should be raid drives.

smartctl -x /dev/sdX

This will give you both the sort of drive you have - yes if it's in a 
datacentre chances are it is a raid drive - and then search the output 
for Error Recovery Control. This is from my hard drive...

SCT capabilities:              (0x003f) SCT Status supported.
                                         SCT Error Recovery Control 
supported.
                                         SCT Feature Control supported.
                                         SCT Data Table supported.

You need error recovery to be supported. If it isn't ...

> 
> Cheers,
> Curt

Cheers,
Wol

Re: hung grow

[Curt]

Hello,

Thanks for clarifying. All current good drives report that they are
part of a 8 drive array.  I only grew the raid by 1 device, so it
would from 7-8, which is what they all report.  The 3rd failed doesn't
report anything on examine, I haven't touched it at all and was not
included in my assemble.  The 2 I replaced, the original drives I
yanked,  think they are still part of a 7 drive array.

I'll be doing a ddrescue on the drives tonight, but will wait till
Phil or someone chimes in with my next steps after I do that.

lol, chalk one more up for FML. "SCT Error Recovery Control command
not supported".  I'm guessing this is a real bad thing now?  I didn't
buy these drives or org set it up.

On Wed, Oct 4, 2017 at 3:46 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 20:09, Curt wrote:
>>
>> Ok, thanks.
>>
>> I'm pretty sure I'll be able to DD from at least one of the failed
>> drives, as I could still query them before I yanked them.  Assuming I
>> can DD one of the old drives to one of my new ones.
>>
>> I'd DDrescue old to new drive. Then do an assemble for force, with a
>> mix of the dd drives and my old good ones? So if sda/b are new DD'd
>> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
>> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
>> use my info from the DD drives to assemble the array back to 7 drives?
>>   Did I understand that right?
>
>
> This sounds like you need to take a great big step backwards, and make sure
> you understand EXACTLY what is going on. We have a mix of good drives,
> copies of bad drives, and an array that doesn't know whether it is supposed
> to have 7 or 9 drives. One wrong step and your array will be toast.
>
> You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.
>
> But I think the first thing we need to do, is to wait for an expert like
> Phil to chime in and sort out that reshape. Your four good drives all think
> they are part of a 9-drive array. Your first two drives to fail think they
> are part of a 7-drive array. Does the third drive think it's part of a
> 7-drive or 9-drive array?
>
> Can you do a --examine on this drive? I suspect the grow blew up because it
> couldn't access this drive. I this drive thinks it is part of a 7-drive
> array, we have a bit of a problem on our hands.
>
> I'm hoping it thinks it's part of a 9-drive array - I think we may be able
> to get out of this ...
>>
>>
>> Oh and how can I tell if I have a timeout mismatch.  They should be raid
>> drives.
>
>
> smartctl -x /dev/sdX
>
> This will give you both the sort of drive you have - yes if it's in a
> datacentre chances are it is a raid drive - and then search the output for
> Error Recovery Control. This is from my hard drive...
>
> SCT capabilities:              (0x003f) SCT Status supported.
>                                         SCT Error Recovery Control
> supported.
>                                         SCT Feature Control supported.
>                                         SCT Data Table supported.
>
> You need error recovery to be supported. If it isn't ...
>
>>
>> Cheers,
>> Curt
>
>
> Cheers,
> Wol

Re: hung grow

[Anthony Youngman]

On 04/10/17 21:01, Curt wrote:
> Hello,
> 
> Thanks for clarifying. All current good drives report that they are
> part of a 8 drive array.  I only grew the raid by 1 device, so it
> would from 7-8, which is what they all report.

That's good.

> The 3rd failed doesn't
> report anything on examine, I haven't touched it at all and was not
> included in my assemble. 

So this wasn't part of your original "assemble --force"? This wasn't 
part of the array you managed to get working and then failed again?

> The 2 I replaced, the original drives I
> yanked,  think they are still part of a 7 drive array.

Okay. The remaining five original disks are the ones we NEED. I'm 
seriously concerned that if that third drive was one of these five, 
we're in big trouble - we've lost the superblock.
> 
> I'll be doing a ddrescue on the drives tonight, but will wait till
> Phil or someone chimes in with my next steps after I do that.

If you've got enough to ddrescue all of those five original drives, then 
that's absolutely great.

Remember - if we can't get five original drives (or copies thereof) the 
array is toast.
> 
> lol, chalk one more up for FML. "SCT Error Recovery Control command
> not supported".  I'm guessing this is a real bad thing now?  I didn't
> buy these drives or org set it up.
> 
I'm not sure whether this is good news or bad. Actually, it *could* be 
great news for the rescue! It's bad news for raid though, if you don't 
deal with it up front - I guess that wasn't done ...

*************

Go and read the wiki - the "When Things Go Wrogn" section. That will 
hopefully help a lot and it explains the Error Recovery stuff (the 
timeout mismatch page). Fix that problem and your dodgy drives will 
probably dd without trouble at all.

Hopefully with all copied drives, but if you have to mix dd'd and 
original drives you're probably okay, you should now be able to assemble 
a working array with five drives by doing an

mdadm --assemble blah blah blah --update=revert-reshape

That will put you back to a "5 drives out of 7" working array. The 
problem with this is that it will be a degraded, linear array.

I'm not sure whether a --display will list the failed drives - if it 
does you can now --remove them. So you'll now have a working, 7-drive 
array, with two drives missing.

Now --add in the two new drives. MAKE SURE you've read the section on 
timeout mismatch and dealt with it! The rebuild/recovery will ALMOST 
CERTAINLY FAIL if you don't! Also note that I am not sure about how 
those drives will display while rebuilding - they may well display as 
being spares during a rebuild.

Lastly, MAKE SURE you set up a regular scrub. There's a distinct 
possibility that this problem wouldn't have arisen (or would have been 
found quicker) if a scrub had been in place. And if you can set up a 
trigger that emails you the contents of /proc/mdstat every few days. 
It's far too easy to miss a failed drive if you don't have something 
shoving it in your face every few days.

Cheers,
Wol

> On Wed, Oct 4, 2017 at 3:46 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 20:09, Curt wrote:
>>>
>>> Ok, thanks.
>>>
>>> I'm pretty sure I'll be able to DD from at least one of the failed
>>> drives, as I could still query them before I yanked them.  Assuming I
>>> can DD one of the old drives to one of my new ones.
>>>
>>> I'd DDrescue old to new drive. Then do an assemble for force, with a
>>> mix of the dd drives and my old good ones? So if sda/b are new DD'd
>>> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
>>> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
>>> use my info from the DD drives to assemble the array back to 7 drives?
>>>    Did I understand that right?
>>
>>
>> This sounds like you need to take a great big step backwards, and make sure
>> you understand EXACTLY what is going on. We have a mix of good drives,
>> copies of bad drives, and an array that doesn't know whether it is supposed
>> to have 7 or 9 drives. One wrong step and your array will be toast.
>>
>> You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.
>>
>> But I think the first thing we need to do, is to wait for an expert like
>> Phil to chime in and sort out that reshape. Your four good drives all think
>> they are part of a 9-drive array. Your first two drives to fail think they
>> are part of a 7-drive array. Does the third drive think it's part of a
>> 7-drive or 9-drive array?
>>
>> Can you do a --examine on this drive? I suspect the grow blew up because it
>> couldn't access this drive. I this drive thinks it is part of a 7-drive
>> array, we have a bit of a problem on our hands.
>>
>> I'm hoping it thinks it's part of a 9-drive array - I think we may be able
>> to get out of this ...
>>>
>>>
>>> Oh and how can I tell if I have a timeout mismatch.  They should be raid
>>> drives.
>>
>>
>> smartctl -x /dev/sdX
>>
>> This will give you both the sort of drive you have - yes if it's in a
>> datacentre chances are it is a raid drive - and then search the output for
>> Error Recovery Control. This is from my hard drive...
>>
>> SCT capabilities:              (0x003f) SCT Status supported.
>>                                          SCT Error Recovery Control
>> supported.
>>                                          SCT Feature Control supported.
>>                                          SCT Data Table supported.
>>
>> You need error recovery to be supported. If it isn't ...
>>
>>>
>>> Cheers,
>>> Curt
>>
>>
>> Cheers,
>> Wol
> 

Re: hung grow

[Phil Turmel]

Hi Curt,

Let me endorse Wol's prescription, with a few comments:

On 10/04/2017 05:08 PM, Anthony Youngman wrote:
> On 04/10/17 21:01, Curt wrote:

{ Side note: what possessed you to do a grow operation? }

>> I'll be doing a ddrescue on the drives tonight, but will wait till
>> Phil or someone chimes in with my next steps after I do that.

I haven't seen complete mdadm -E reports for all of these devices, nor
mdadm -D for the array itself.  Please do so now.  If you have any of
that from before the crash, please post that too.  Run mdadm -E on the
two earliest failed drives.

Post the uncut output inline here on the list, in plain text mode, with
line wrap disabled, please.

> If you've got enough to ddrescue all of those five original drives, then
> that's absolutely great.
> 
> Remember - if we can't get five original drives (or copies thereof) the
> array is toast.
>>
>> lol, chalk one more up for FML. "SCT Error Recovery Control command
>> not supported".  I'm guessing this is a real bad thing now?  I didn't
>> buy these drives or org set it up.
>>
> I'm not sure whether this is good news or bad. Actually, it *could* be
> great news for the rescue! It's bad news for raid though, if you don't
> deal with it up front - I guess that wasn't done ...

It is mixed news.  It is almost certainly the reason you've had drives
bumped out of your arrays.  I suspect these drives all report *PASSED*
from smartctl.  Which means that the drives really are good, just
suffering from ordinary uncorrected errors.

You'll certainly have to use the 180 second driver timeout work-around
to get through this crisis.

In the meantime, please run "smartctl -iA -l scterc" on each of your
drives, including the failed ones, and post the uncut output here.
{ Include the device name with each }

> Go and read the wiki - the "When Things Go Wrogn" section. That will
> hopefully help a lot and it explains the Error Recovery stuff (the
> timeout mismatch page). Fix that problem and your dodgy drives will
> probably dd without trouble at all.

Let me emphasize this.  The timeout mismatch problem is so prevalent and
your experience so common that I thought to myself "I bet this one is
timeout mismatch" when I read your first mail.

> Hopefully with all copied drives, but if you have to mix dd'd and
> original drives you're probably okay, you should now be able to assemble
> a working array with five drives by doing an

As already noted, you definitely need to use ddrescue on the third
drive that failed.  You may also need to ddrescue your four remaining
good drives if they also have "Pending Sector" counts.

> mdadm --assemble blah blah blah --update=revert-reshape
> 
> That will put you back to a "5 drives out of 7" working array. The
> problem with this is that it will be a degraded, linear array.

This is the correct next step, after all required ddrescues.

> I'm not sure whether a --display will list the failed drives - if it
> does you can now --remove them. So you'll now have a working, 7-drive
> array, with two drives missing.

This is the time to grab any backups you need of critical content.  Do
*not* write to the array at this point.  Get all your data off.

Then:

> Now --add in the two new drives. MAKE SURE you've read the section on
> timeout mismatch and dealt with it! The rebuild/recovery will ALMOST
> CERTAINLY FAIL if you don't! Also note that I am not sure about how
> those drives will display while rebuilding - they may well display as
> being spares during a rebuild.

The timeout mismatch fixes won't help your case.  You have no redundancy
left, so the kickout scenarios involved no longer apply.  They applied
when your first two drives were kicked out.  When timeouts are not
mismatched, MD raid *fixes* the occasional bad sector.

> Lastly, MAKE SURE you set up a regular scrub. There's a distinct
> possibility that this problem wouldn't have arisen (or would have been
> found quicker) if a scrub had been in place. And if you can set up a
> trigger that emails you the contents of /proc/mdstat every few days.
> It's far too easy to miss a failed drive if you don't have something
> shoving it in your face every few days.

If you have a timeout mismatch problem, one's array will die much sooner
with scrubs.  Because MD raid will fail to fix UREs, and it will find
them right away.

But again, get us the detailed reports, and we'll help make sure your
commands are correct.

Phil

Re: hung grow

[Curt]

I just realized I forgot to do a reply all on this yesterday.  In case
anyone else in the group is interested. The ddrescue on the failed
drive is complete now.

Requested info pasted below, at least the info I got.

(Side note: Let's just go with stupidity)

Then depending on what your feedback is, theoretically I'll want to run

mdadm --assemble --force --update=revert-reshape /dev/md127 /dev/sda1
/dev/sdc1 /dev/sdd1 /dev/sdg1/ /dev/ddRescuePart

I appreciate all your help on this.

Cheers,
Curt

On Wed, Oct 4, 2017 at 5:53 PM, Phil Turmel <philip@turmel.org> wrote:
> Hi Curt,
>
> Let me endorse Wol's prescription, with a few comments:
>
> On 10/04/2017 05:08 PM, Anthony Youngman wrote:
>> On 04/10/17 21:01, Curt wrote:
>
> { Side note: what possessed you to do a grow operation? }
>
>>> I'll be doing a ddrescue on the drives tonight, but will wait till
>>> Phil or someone chimes in with my next steps after I do that.
>
> I haven't seen complete mdadm -E reports for all of these devices, nor
> mdadm -D for the array itself.  Please do so now.  If you have any of
> that from before the crash, please post that too.  Run mdadm -E on the
> two earliest failed drives.
>
Don't hate on me too bad.  I already know I made several very stupid
mistakes along the way.

Here's watch I got probably missing a few things that would be useful.
The array is currently stopped, so I can't get you the -D, but here's
what I got

Array Before Grow
/dev/md127:
           Version : 0.90
     Creation Time : Fri Jun 15 15:52:05 2012
        Raid Level : raid6
        Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
     Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
      Raid Devices : 7
     Total Devices : 7
   Preferred Minor : 127
       Persistence : Superblock is persistent

       Update Time : Tue Oct  3 21:13:32 2017
             State : clean, degraded, recovering
    Active Devices : 5
   Working Devices : 7
    Failed Devices : 0
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

    Rebuild Status : 84% complete

              UUID : 714a612d:9bd35197:36c91ae3:c168144d
            Events : 0.11559596

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       3       8        1        3      active sync   /dev/sda1
       4       8       65        4      active sync   /dev/sde1
       8       8       16        5      spare rebuilding   /dev/sdb
       7       8       80        6      spare rebuilding   /dev/sdf


Array After Grow:
mdadm --detail /dev/md127
/dev/md127:
           Version : 0.91
     Creation Time : Fri Jun 15 15:52:05 2012
        Raid Level : raid6
        Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
     Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
      Raid Devices : 8
     Total Devices : 7
   Preferred Minor : 127
       Persistence : Superblock is persistent

       Update Time : Tue Oct  3 23:10:32 2017
             State : clean, FAILED, reshaping
    Active Devices : 5
   Working Devices : 7
    Failed Devices : 0
     Spare Devices : 2

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

    Reshape Status : 0% complete
     Delta Devices : 1, (7->8)

              UUID : 714a612d:9bd35197:36c91ae3:c168144d
            Events : 0.11559671

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       3       8        1        3      active sync   /dev/sda1
       4       8       65        4      active sync   /dev/sde1
       5       8       16        5      spare rebuilding   /dev/sdb
       6       8       80        6      spare rebuilding   /dev/sdf
       -       0        0        7      removed


Here's the few I have from before.  I really shouldn't have been doing
this at 4am.
****************
mdadm --examine /dev/sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127

    Update Time : Tue Oct  3 22:38:22 2017
          State : clean
 Active Devices : 4
Working Devices : 6
 Failed Devices : 3
  Spare Devices : 2
       Checksum : cdfbf074 - correct
         Events : 11559615

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     7       8       80        7      spare   /dev/sdf

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       0        0        4      faulty removed
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8       80        7      spare   /dev/sdf
   8     8       8       16        8      spare   /dev/sdb

mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127

    Update Time : Tue Oct  3 22:38:22 2017
          State : clean
 Active Devices : 4
Working Devices : 6
 Failed Devices : 3
  Spare Devices : 2
       Checksum : cdfbf023 - correct
         Events : 11559615

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8        1        3      active sync   /dev/sda1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       0        0        4      faulty removed
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8       80        7      spare   /dev/sdf
   8     8       8       16        8      spare   /dev/sdb

Here's the 3 failed drives: NOTE: I only had one bay available, so
they all have the same drive letter

mdadm --examine /dev/sdz1
/dev/sdz1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 126

    Update Time : Mon Jul 11 16:54:15 2016
          State : active
 Active Devices : 6
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 0
       Checksum : ca7ec3b0 - correct
         Events : 3397832

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1      65       97        1      active sync

   0     0      65      113        0      active sync
   1     1      65       97        1      active sync
   2     2      65       81        2      active sync
   3     3       0        0        3      faulty removed
   4     4      65       49        4      active sync
   5     5      65       33        5      active sync
   6     6      65       17        6      active sync

**********************THE ONE BELOW I'M DOING A DDRESCUE FROM******

mdadm --examine /dev/sdz1
/dev/sdz1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127

    Update Time : Sat Sep  2 01:00:37 2017
          State : active
 Active Devices : 6
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 0
       Checksum : cd217ebc - correct
         Events : 11559404

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     5       8       65        5      active sync   /dev/sde1

   0     0       8       81        0      active sync
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync
   3     3      65      129        3      active sync   /dev/sdy1
   4     4       8       49        4      active sync   /dev/sdd1
   5     5       8       65        5      active sync   /dev/sde1
   6     6       0        0        6      faulty removed



***************

mdadm --examine /dev/sdz1
/dev/sdz1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127

    Update Time : Mon Nov  7 02:02:38 2016
          State : active
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0
       Checksum : cb1ec57d - correct
         Events : 3652739

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     6       8       97        6      active sync   /dev/sdg1

   0     0       8       81        0      active sync
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync
   3     3      65      129        3      active sync   /dev/sdy1
   4     4       8       49        4      active sync   /dev/sdd1
   5     5       8       65        5      active sync   /dev/sde1
   6     6       8       97        6      active sync   /dev/sdg1

CURRENT EXAMINE
*************************
mdadm -E /dev/sd[acdeg]1
/dev/sda1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71a9cb - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8        1        3      active sync   /dev/sda1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71a9e9 - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       33        2      active sync   /dev/sdc1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed
/dev/sdd1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71a9f7 - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       49        1      active sync   /dev/sdd1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed
/dev/sde1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71aa0d - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8       65        4      active sync   /dev/sde1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed
/dev/sdg1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71aa25 - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       97        0      active sync   /dev/sdg1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed

/dev/sdb:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71a9dc - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     6       8       16        6      active   /dev/sdb

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty remove

> Post the uncut output inline here on the list, in plain text mode, with
> line wrap disabled, please.
>
>> If you've got enough to ddrescue all of those five original drives, then
>> that's absolutely great.
>>
>> Remember - if we can't get five original drives (or copies thereof) the
>> array is toast.
>>>
>>> lol, chalk one more up for FML. "SCT Error Recovery Control command
>>> not supported".  I'm guessing this is a real bad thing now?  I didn't
>>> buy these drives or org set it up.
>>>
>> I'm not sure whether this is good news or bad. Actually, it *could* be
>> great news for the rescue! It's bad news for raid though, if you don't
>> deal with it up front - I guess that wasn't done ...
>
> It is mixed news.  It is almost certainly the reason you've had drives
> bumped out of your arrays.  I suspect these drives all report *PASSED*
> from smartctl.  Which means that the drives really are good, just
> suffering from ordinary uncorrected errors.
>
> You'll certainly have to use the 180 second driver timeout work-around
> to get through this crisis.
>
> In the meantime, please run "smartctl -iA -l scterc" on each of your
> drives, including the failed ones, and post the uncut output here.
> { Include the device name with each }
>
Sorry I don't have it for the failed ones, I forgot to run in before I
started ddrescue, here's the current drives
# smartctl -iA -l scterc /dev/sda
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1ER164
Serial Number:    W4Z14ZNW
LU WWN Device Id: 5 000c50 07d29ef14
Firmware Version: CC25
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:28:30 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   106   099   006    Pre-fail
Always       -       11140560
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail
Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age
Always       -       14
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x000f   089   060   030    Pre-fail
Always       -       827856598
  9 Power_On_Hours          0x0032   079   079   000    Old_age
Always       -       18858
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age
Always       -       14
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age
Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age
Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age
Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age
Always       -       0 0 0
189 High_Fly_Writes         0x003a   013   013   000    Old_age
Always       -       87
190 Airflow_Temperature_Cel 0x0022   071   063   045    Old_age
Always       -       29 (Min/Max 29/30)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age
Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age
Always       -       8
193 Load_Cycle_Count        0x0032   100   100   000    Old_age
Always       -       268
194 Temperature_Celsius     0x0022   029   040   000    Old_age
Always       -       29 (0 18 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age
Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age
Offline      -       18841h+49m+16.895s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age
Offline      -       84336821090
242 Total_LBAs_Read         0x0000   100   253   000    Old_age
Offline      -       4832824497202

SCT Error Recovery Control command not supported

# smartctl -iA -l scterc /dev/sdb
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1ER164
Serial Number:    Z4Z3Y7XM
LU WWN Device Id: 5 000c50 087461756
Firmware Version: CC26
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:28:44 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   117   099   006    Pre-fail
Always       -       157161144
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail
Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age
Always       -       15
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x000f   086   060   030    Pre-fail
Always       -       409701090
  9 Power_On_Hours          0x0032   086   086   000    Old_age
Always       -       12274
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age
Always       -       15
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age
Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age
Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age
Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age
Always       -       0 0 0
189 High_Fly_Writes         0x003a   092   092   000    Old_age
Always       -       8
190 Airflow_Temperature_Cel 0x0022   070   065   045    Old_age
Always       -       30 (Min/Max 28/33)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age
Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age
Always       -       10
193 Load_Cycle_Count        0x0032   100   100   000    Old_age
Always       -       142
194 Temperature_Celsius     0x0022   030   040   000    Old_age
Always       -       30 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age
Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age
Offline      -       12268h+22m+21.157s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age
Offline      -       83831274067
242 Total_LBAs_Read         0x0000   100   253   000    Old_age
Offline      -       124530518173

SCT Error Recovery Control command not supported

# smartctl -iA -l scterc /dev/sdc
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Black
Device Model:     WDC WD2002FAEX-007BA0
Serial Number:    WD-WMAY04949787
LU WWN Device Id: 5 0014ee 25c3e0682
Firmware Version: 05.01D05
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:28:46 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail
Always       -       2
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail
Always       -       8041
  4 Start_Stop_Count        0x0032   100   100   000    Old_age
Always       -       36
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age
Always       -       0
  9 Power_On_Hours          0x0032   071   071   000    Old_age
Always       -       21337
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age
Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age
Always       -       27
193 Load_Cycle_Count        0x0032   200   200   000    Old_age
Always       -       8
194 Temperature_Celsius     0x0022   117   107   000    Old_age
Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age
Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age
Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age
Offline      -       0

SCT Error Recovery Control command not supported

# smartctl -iA -l scterc /dev/sdd
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Black
Device Model:     WDC WD2002FAEX-007BA0
Serial Number:    WD-WMAY04912439
LU WWN Device Id: 5 0014ee 25c3f0960
Firmware Version: 05.01D05
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:29:33 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail
Always       -       8
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail
Always       -       7950
  4 Start_Stop_Count        0x0032   100   100   000    Old_age
Always       -       36
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age
Always       -       0
  9 Power_On_Hours          0x0032   071   071   000    Old_age
Always       -       21325
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age
Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age
Always       -       27
193 Load_Cycle_Count        0x0032   200   200   000    Old_age
Always       -       8
194 Temperature_Celsius     0x0022   116   106   000    Old_age
Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age
Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age
Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age
Offline      -       0

SCT Error Recovery Control command not supported

# smartctl -iA -l scterc /dev/sde
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Black
Device Model:     WDC WD2002FAEX-007BA0
Serial Number:    WD-WMAY05040774
LU WWN Device Id: 5 0014ee 2b1938a22
Firmware Version: 05.01D05
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:29:36 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail
Always       -       0
  3 Spin_Up_Time            0x0027   253   253   021    Pre-fail
Always       -       8083
  4 Start_Stop_Count        0x0032   100   100   000    Old_age
Always       -       36
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age
Always       -       0
  9 Power_On_Hours          0x0032   071   071   000    Old_age
Always       -       21328
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age
Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age
Always       -       27
193 Load_Cycle_Count        0x0032   200   200   000    Old_age
Always       -       8
194 Temperature_Celsius     0x0022   116   108   000    Old_age
Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age
Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age
Always       -       2
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age
Offline      -       2
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age
Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age
Offline      -       2

SCT Error Recovery Control command not supported

#smartctl -iA -l scterc /dev/sdg
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1ER164
Serial Number:    ZA5029A8
LU WWN Device Id: 5 000c50 0874eb397
Firmware Version: CC26
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct  4 20:29:42 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail
Always       -       232755000
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail
Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age
Always       -       10
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x000f   087   060   030    Pre-fail
Always       -       606406779
  9 Power_On_Hours          0x0032   086   086   000    Old_age
Always       -       13052
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age
Always       -       10
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age
Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age
Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age
Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age
Always       -       0 0 0
189 High_Fly_Writes         0x003a   098   098   000    Old_age
Always       -       2
190 Airflow_Temperature_Cel 0x0022   069   060   045    Old_age
Always       -       31 (Min/Max 29/34)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age
Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age
Always       -       4
193 Load_Cycle_Count        0x0032   100   100   000    Old_age
Always       -       310
194 Temperature_Celsius     0x0022   031   040   000    Old_age
Always       -       31 (0 25 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age
Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age
Offline      -       13039h+51m+52.726s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age
Offline      -       33464557056
242 Total_LBAs_Read         0x0000   100   253   000    Old_age
Offline      -       2634696436762

SCT Error Recovery Control command not supported


>> Go and read the wiki - the "When Things Go Wrogn" section. That will
>> hopefully help a lot and it explains the Error Recovery stuff (the
>> timeout mismatch page). Fix that problem and your dodgy drives will
>> probably dd without trouble at all.
>
> Let me emphasize this.  The timeout mismatch problem is so prevalent and
> your experience so common that I thought to myself "I bet this one is
> timeout mismatch" when I read your first mail.
>
>> Hopefully with all copied drives, but if you have to mix dd'd and
>> original drives you're probably okay, you should now be able to assemble
>> a working array with five drives by doing an
>
> As already noted, you definitely need to use ddrescue on the third
> drive that failed.  You may also need to ddrescue your four remaining
> good drives if they also have "Pending Sector" counts.
>
>> mdadm --assemble blah blah blah --update=revert-reshape
>>
>> That will put you back to a "5 drives out of 7" working array. The
>> problem with this is that it will be a degraded, linear array.
>
> This is the correct next step, after all required ddrescues.
>
>> I'm not sure whether a --display will list the failed drives - if it
>> does you can now --remove them. So you'll now have a working, 7-drive
>> array, with two drives missing.
>
> This is the time to grab any backups you need of critical content.  Do
> *not* write to the array at this point.  Get all your data off.
>
> Then:
>
>> Now --add in the two new drives. MAKE SURE you've read the section on
>> timeout mismatch and dealt with it! The rebuild/recovery will ALMOST
>> CERTAINLY FAIL if you don't! Also note that I am not sure about how
>> those drives will display while rebuilding - they may well display as
>> being spares during a rebuild.
>
> The timeout mismatch fixes won't help your case.  You have no redundancy
> left, so the kickout scenarios involved no longer apply.  They applied
> when your first two drives were kicked out.  When timeouts are not
> mismatched, MD raid *fixes* the occasional bad sector.
>
>> Lastly, MAKE SURE you set up a regular scrub. There's a distinct
>> possibility that this problem wouldn't have arisen (or would have been
>> found quicker) if a scrub had been in place. And if you can set up a
>> trigger that emails you the contents of /proc/mdstat every few days.
>> It's far too easy to miss a failed drive if you don't have something
>> shoving it in your face every few days.
>
> If you have a timeout mismatch problem, one's array will die much sooner
> with scrubs.  Because MD raid will fail to fix UREs, and it will find
> them right away.
>
> But again, get us the detailed reports, and we'll help make sure your
> commands are correct.
>
> Phil

Re: hung grow

[Wols Lists]

On 06/10/17 02:25, Curt wrote:
> Requested info pasted below, at least the info I got.

Seagate Barracudas, WD Blacks. NOT good raid drives, sorry. Once you get
it all working, then you need that timeout script configured to run
every boot, at which point your array should be safe.

Once it's all fine, if you're in the market for more drives look at
Seagate NAS or Ironwolf or Constellation, WD Reds, or Hitachi Ultrastar.
They're all advertised (Constellation excepted, but that's reported as
supporting ERC) as being suitable for raid. Expect to pay a 30-50%
premium over what a Barracuda will cost.

Cheers,
Wol

Re: hung grow

[Phil Turmel]

Hi Curt,

Sorry about the delay /-:

{ Convention on kernel.org lists is reply to all, bottom replies (or
interleaved, and trim unnecessary quoted material. Please help those
of us who follow many threads. }

On 10/05/2017 07:40 PM, Curt wrote:
> Hey Phil,
> 
> I got the drive mentioned ddrescued, I wanted to make sure you didn't see
> anything odd from my pastes before I try the revert.

Only one:

> # smartctl -iA -l scterc /dev/sde
> 197 Current_Pending_Sector  0x0032   200   200   000    Old_age
> Always       -       2
> 198 Offline_Uncorrectable   0x0030   200   200   000    Old_age
> Offline      -       2

You'll have to ddrescue /dev/sde1 as well.  Do that before proceeding.

Phil

Re: hung grow

[Curt]

On Fri, Oct 6, 2017 at 9:13 AM, Phil Turmel <philip@turmel.org> wrote:
> Hi Curt,
>
> Sorry about the delay /-:
>
> { Convention on kernel.org lists is reply to all, bottom replies (or
> interleaved, and trim unnecessary quoted material. Please help those
> of us who follow many threads. }
>
> On 10/05/2017 07:40 PM, Curt wrote:
>> Hey Phil,
>>
>> I got the drive mentioned ddrescued, I wanted to make sure you didn't see
>> anything odd from my pastes before I try the revert.
>
> Only one:
>
>> # smartctl -iA -l scterc /dev/sde
>> 197 Current_Pending_Sector  0x0032   200   200   000    Old_age
>> Always       -       2
>> 198 Offline_Uncorrectable   0x0030   200   200   000    Old_age
>> Offline      -       2
>
> You'll have to ddrescue /dev/sde1 as well.  Do that before proceeding.
>
> Phil

Hi Phil,

No worries, I appreciate the help.

(Noted and I will try to follow that standard in all future communication)

Ok, I will ddrescue that drive today, which should take about 10
hours.  I'm just doing a ddrescue -f -n /dev/bad /dev/new logFile.  Is
there any other suggested flags?

Once that is done, I'll run the before mentioned, mdadm assemble force
revert sda/c/d/g + rescued e and z.  Then start pulling whatever data
I can off it.

Cheers,
Curt

Re: hung grow

[Joe Landman]

On 10/06/2017 10:07 AM, Curt wrote:
> On Fri, Oct 6, 2017 at 9:13 AM, Phil Turmel <philip@turmel.org> wrote:

[...]

> Once that is done, I'll run the before mentioned, mdadm assemble force
> revert sda/c/d/g + rescued e and z.  Then start pulling whatever data
> I can off it.

I'd suggest that some data may be more highly valuable than other data 
on there.  So if you can pull that off and stash somewhere, that would 
be recommended before copying everything.  In the past I've made 
tarballs and moved copies around to various locations to preserve 
important data.  Once you have that, do the bigger copy of data.



-- 
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman

Re: hung grow

[Phil Turmel]

On 10/06/2017 10:07 AM, Curt wrote:
> Ok, I will ddrescue that drive today, which should take about 10
> hours.  I'm just doing a ddrescue -f -n /dev/bad /dev/new logFile.  Is
> there any other suggested flags?

No, that should do.  You can ddrescue just the partition if you like.

> Once that is done, I'll run the before mentioned, mdadm assemble force
> revert sda/c/d/g + rescued e and z.

Yes, just make sure you are at the MD member level, whether whole device
or partition for each component.  Your new devices were added as whole
devices, an inconsistency that could burn you later.

> Then start pulling whatever data
> I can off it.

Yes.

Re: hung grow

[Curt]

On Fri, Oct 6, 2017 at 10:27 AM, Phil Turmel <philip@turmel.org> wrote:
> On 10/06/2017 10:07 AM, Curt wrote:
>> Once that is done, I'll run the before mentioned, mdadm assemble force
>> revert sda/c/d/g + rescued e and z.
>
> Yes, just make sure you are at the MD member level, whether whole device
> or partition for each component.  Your new devices were added as whole
> devices, an inconsistency that could burn you later.

Ok, I tried to do an assemble including the failed drive I ddrescued
and get superblock on /dev/sdf1 doesn't match others assembly aborted.
Does that mean I won't be able to revert or recovery any data?  I hope
not.

Cheers,
Curt

Re: hung grow

[Curt]

> Ok, I tried to do an assemble including the failed drive I ddrescued
> and get superblock on /dev/sdf1 doesn't match others assembly aborted.
> Does that mean I won't be able to revert or recovery any data?  I hope
> not.
>
> Cheers,
> Curt

Just to clarify, sdf1 is the ddrescue of the failed drive.

Re: hung grow

[Curt]

Here's the output of -E for sdf1 and one other, in case it's needed.
As I understand it the UUID is what has to do with the superblock, but
UUID's match between them.

mdadm -E /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127

    Update Time : Sat Sep  2 01:00:37 2017
          State : active
 Active Devices : 6
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 0
       Checksum : cd217ebc - correct
         Events : 11559404

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     5       8       65        5      active sync   /dev/sde1

   0     0       8       81        0      active sync   /dev/sdf1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync
   3     3      65      129        3      active sync   /dev/sdy1
   4     4       8       49        4      active sync   /dev/sdd1
   5     5       8       65        5      active sync   /dev/sde1
   6     6       0        0        6      faulty removed

*****************************
mdadm -E /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : 714a612d:9bd35197:36c91ae3:c168144d
  Creation Time : Fri Jun 15 15:52:05 2012
     Raid Level : raid6
  Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
     Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

    Update Time : Wed Oct  4 12:49:57 2017
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce71a9cb - correct
         Events : 11559681

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8        1        3      active sync   /dev/sda1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8        1        3      active sync   /dev/sda1
   4     4       8       65        4      active sync   /dev/sde1
   5     5       0        0        5      faulty removed
   6     6       8       16        6      active   /dev/sdb
   7     7       0        0        7      faulty removed

Re: hung grow

[Phil Turmel]

On 10/07/2017 04:45 PM, Curt wrote:
> Here's the output of -E for sdf1 and one other, in case it's needed.
> As I understand it the UUID is what has to do with the superblock, but
> UUID's match between them.
> 
> mdadm -E /dev/sdf1
> /dev/sdf1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 714a612d:9bd35197:36c91ae3:c168144d
>   Creation Time : Fri Jun 15 15:52:05 2012
>      Raid Level : raid6
>   Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
>      Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
>    Raid Devices : 7
>   Total Devices : 7
> Preferred Minor : 127
> 
>     Update Time : Sat Sep  2 01:00:37 2017
>           State : active
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : cd217ebc - correct
>          Events : 11559404
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     5       8       65        5      active sync   /dev/sde1
> 
>    0     0       8       81        0      active sync   /dev/sdf1
>    1     1       8       33        1      active sync   /dev/sdc1
>    2     2       8       17        2      active sync
>    3     3      65      129        3      active sync   /dev/sdy1
>    4     4       8       49        4      active sync   /dev/sdd1
>    5     5       8       65        5      active sync   /dev/sde1
>    6     6       0        0        6      faulty removed
> 
> *****************************
> mdadm -E /dev/sda1
> /dev/sda1:
>           Magic : a92b4efc
>         Version : 0.91.00
>            UUID : 714a612d:9bd35197:36c91ae3:c168144d
>   Creation Time : Fri Jun 15 15:52:05 2012
>      Raid Level : raid6
>   Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
>      Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
>    Raid Devices : 8
>   Total Devices : 6
> Preferred Minor : 127
> 
>   Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
>   Delta Devices : 1 (7->8)
> 
>     Update Time : Wed Oct  4 12:49:57 2017
>           State : clean
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 2
>   Spare Devices : 0
>        Checksum : ce71a9cb - correct
>          Events : 11559681
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     3       8        1        3      active sync   /dev/sda1
> 
>    0     0       8       97        0      active sync   /dev/sdg1
>    1     1       8       49        1      active sync   /dev/sdd1
>    2     2       8       33        2      active sync   /dev/sdc1
>    3     3       8        1        3      active sync   /dev/sda1
>    4     4       8       65        4      active sync   /dev/sde1
>    5     5       0        0        5      faulty removed
>    6     6       8       16        6      active   /dev/sdb
>    7     7       0        0        7      faulty removed
> 

Hmm.  Please run the assemble again, with the --verbose option, and
paste all the output here so we can see what's happening.

Also grab the corresponding tail of dmesg (however many lines cover the
assembly attempt).

Phil

Re: hung grow

[Curt]

On Sat, Oct 7, 2017 at 5:29 PM, Phil Turmel <philip@turmel.org> wrote:
> On 10/07/2017 04:45 PM, Curt wrote:
>
> Hmm.  Please run the assemble again, with the --verbose option, and
> paste all the output here so we can see what's happening.
>
> Also grab the corresponding tail of dmesg (however many lines cover the
> assembly attempt).
>
> Phil

You wouldn't think reply all would be such a hard thing to remember to do.

verbose didn't really add anything, here's my full command, maybe I
miss typed something and didn't notice.

# mdadm --assemble --force --update=revert-reshape --verbose
/dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sdz1
mdadm: looking for devices for /dev/md127
mdadm: superblock on /dev/sdf1 doesn't match others - assembly aborted

I've got nothing in dmesg or messages

If I can't get the old drive to work.  Is there any chance to still
getting some data from the other drives? Either running the above
without sdf or worse comes to worse the assume clean someone mentioned
before.

Cheers,
Curt

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

On Sun, Oct 08 2017, Curt wrote:

> On Sat, Oct 7, 2017 at 5:29 PM, Phil Turmel <philip@turmel.org> wrote:
>> On 10/07/2017 04:45 PM, Curt wrote:
>>
>> Hmm.  Please run the assemble again, with the --verbose option, and
>> paste all the output here so we can see what's happening.
>>
>> Also grab the corresponding tail of dmesg (however many lines cover the
>> assembly attempt).
>>
>> Phil
>
> You wouldn't think reply all would be such a hard thing to remember to do.
>
> verbose didn't really add anything, here's my full command, maybe I
> miss typed something and didn't notice.
>
> # mdadm --assemble --force --update=revert-reshape --verbose
> /dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sdz1
> mdadm: looking for devices for /dev/md127
> mdadm: superblock on /dev/sdf1 doesn't match others - assembly aborted

You get this because sdf1 says:

   Raid Devices : 7
  Total Devices : 7

while sda1 (for example) says:

   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 127

  Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
  Delta Devices : 1 (7->8)

mdadm cannot reconcile this difference.

It appears that sdf1 was never involved in any reshape.
So you need to revert the reshape before trying to include sdf1 into the
array.  Clearly you need at least 6 devices that were involved in
the reshape to do this.
I haven't been following closely ... do you have 6 such devices?

NeilBrown


>
> I've got nothing in dmesg or messages
>
> If I can't get the old drive to work.  Is there any chance to still
> getting some data from the other drives? Either running the above
> without sdf or worse comes to worse the assume clean someone mentioned
> before.
>
> Cheers,
> Curt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

> You get this because sdf1 says:
>
>    Raid Devices : 7
>   Total Devices : 7
>
> while sda1 (for example) says:
>
>    Raid Devices : 8
>   Total Devices : 6
> Preferred Minor : 127
>
>   Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
>   Delta Devices : 1 (7->8)
>
> mdadm cannot reconcile this difference.
>
> It appears that sdf1 was never involved in any reshape.
> So you need to revert the reshape before trying to include sdf1 into the
> array.  Clearly you need at least 6 devices that were involved in
> the reshape to do this.
> I haven't been following closely ... do you have 6 such devices?
>
> NeilBrown
>

Correct.  Which I thought was sorta the point, but could have
completely misunderstood it, sdf was restored from a "faulty" drive
that was out before the reshape.  Whether I have 6 devices depends on
how picky things are.  I've got 5 that should be in sync, the 6th not,
but it was involved in the reshape.

Short version, is I shot myself in the foot on this one. Reshape never
got anywhere, but need to try to revert and save what data I can.

Cheers,
Curt

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

On Sun, Oct 08 2017, Curt wrote:

>> You get this because sdf1 says:
>>
>>    Raid Devices : 7
>>   Total Devices : 7
>>
>> while sda1 (for example) says:
>>
>>    Raid Devices : 8
>>   Total Devices : 6
>> Preferred Minor : 127
>>
>>   Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
>>   Delta Devices : 1 (7->8)
>>
>> mdadm cannot reconcile this difference.
>>
>> It appears that sdf1 was never involved in any reshape.
>> So you need to revert the reshape before trying to include sdf1 into the
>> array.  Clearly you need at least 6 devices that were involved in
>> the reshape to do this.
>> I haven't been following closely ... do you have 6 such devices?
>>
>> NeilBrown
>>
>
> Correct.  Which I thought was sorta the point, but could have
> completely misunderstood it, sdf was restored from a "faulty" drive
> that was out before the reshape.  Whether I have 6 devices depends on
> how picky things are.  I've got 5 that should be in sync, the 6th not,
> but it was involved in the reshape.
>
> Short version, is I shot myself in the foot on this one. Reshape never
> got anywhere, but need to try to revert and save what data I can.

Hmmm... (goes back and looks at more of thread..)

Ahhh .. you had an array which was rebuilding two spares, and you
told it to start reshaping... Interesting.
Theoretically that should work.  Was it deliberate? (I cannot seem to
find the start of the thread).

Looking at the list of "current --examine output", it appears that
 /dev/sdg1
 /dev/sdd1
 /dev/sdc1
 /dev/sda1
 /dev/sde1

 /dev/sdb

are all valid devices with the same event counts.  They are the six that
you need.
To confirm that names haven't changed, you can:
  mdadm --examine  /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
  /dev/sdb | grep Events

and confirm all the numbers are the same.
Then do the same and grep for "this" and confirm all the Raid Disk
numbers are different.

Interesting that /dev/sdb is a whole device and the rest are partitions.
I assume you know about that and why it is.

What happens if you run the --assemble --update=revert-reshape command on
these 6 devices (without --force)??

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
> On Sun, Oct 08 2017, Curt wrote:
> Theoretically that should work.  Was it deliberate? (I cannot seem to
> find the start of the thread).
Yes and no, I ran the command, but it wasn't what I really wanted
and/or needed, so to speak.


> are all valid devices with the same event counts.  They are the six that
> you need.
> To confirm that names haven't changed, you can:
>   mdadm --examine  /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
>   /dev/sdb | grep Events

Yes, the numbers all match.
>
> and confirm all the numbers are the same.
> Then do the same and grep for "this" and confirm all the Raid Disk
> numbers are different.
confirmed
>
> Interesting that /dev/sdb is a whole device and the rest are partitions.
> I assume you know about that and why it is.
Just a mistake when I added it.
>
> What happens if you run the --assemble --update=revert-reshape command on
> these 6 devices (without --force)??
>
> NeilBrown
I'll give it a try, but first a question and FYI.  Earlier Phil had me
ddrescue sde1 because it had pending sectors I believe.  Should I use
sde1 or sdz1(ddrescued version)?

Also when this whole thing started. md127_raid process was using 100%
cpu, but nothing seemed to be happening, No numbers changed, nothing
that I could tell anyway.  Any suggestions on if the same thing
happens when I run the revert-reshape? Could one of my libraries be
behind a version somewhere?

Cheers,
Curt

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]

On Mon, Oct 09 2017, Curt wrote:

> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>> On Sun, Oct 08 2017, Curt wrote:
>> Theoretically that should work.  Was it deliberate? (I cannot seem to
>> find the start of the thread).
> Yes and no, I ran the command, but it wasn't what I really wanted
> and/or needed, so to speak.
>
>
>> are all valid devices with the same event counts.  They are the six that
>> you need.
>> To confirm that names haven't changed, you can:
>>   mdadm --examine  /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
>>   /dev/sdb | grep Events
>
> Yes, the numbers all match.
>>
>> and confirm all the numbers are the same.
>> Then do the same and grep for "this" and confirm all the Raid Disk
>> numbers are different.
> confirmed
>>
>> Interesting that /dev/sdb is a whole device and the rest are partitions.
>> I assume you know about that and why it is.
> Just a mistake when I added it.
>>
>> What happens if you run the --assemble --update=revert-reshape command on
>> these 6 devices (without --force)??
>>
>> NeilBrown
> I'll give it a try, but first a question and FYI.  Earlier Phil had me
> ddrescue sde1 because it had pending sectors I believe.  Should I use
> sde1 or sdz1(ddrescued version)?

If sdz1 contains the same data as sde1, then use sdz1.

>
> Also when this whole thing started. md127_raid process was using 100%
> cpu, but nothing seemed to be happening, No numbers changed, nothing
> that I could tell anyway.  Any suggestions on if the same thing
> happens when I run the revert-reshape? Could one of my libraries be
> behind a version somewhere?

If md127_raid is using 100% then that is a kernel bug.
If that happens, then find all processes that are spinning or are in 'D'
state, and
   cat /proc/$PID/stack
for each pid, and report all the output.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
> What happens if you run the --assemble --update=revert-reshape command on
> these 6 devices (without --force)??

Here's the output
mdadm --assemble --verbose --update=revert-reshape /dev/md127
/dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
mdadm: looking for devices for /dev/md127
mdadm: Reshape position is not suitably aligned.
mdadm: Try normal assembly and stop again

Cheers,
Curt

Re: hung grow

[Curt]

On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>> What happens if you run the --assemble --update=revert-reshape command on
>> these 6 devices (without --force)??
>
> Here's the output
> mdadm --assemble --verbose --update=revert-reshape /dev/md127
> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
> mdadm: looking for devices for /dev/md127
> mdadm: Reshape position is not suitably aligned.
> mdadm: Try normal assembly and stop again
>

What are the steps I need to take to resume the grow, but just enough
to get it to the point I can revert it and see what data I have on the
drives? Or can I force the revert?

Cheers,
Curt

Re: hung grow

[Phil Turmel]

On 10/10/2017 08:08 AM, Curt wrote:
> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>>> What happens if you run the --assemble --update=revert-reshape command on
>>> these 6 devices (without --force)??
>>
>> Here's the output
>> mdadm --assemble --verbose --update=revert-reshape /dev/md127
>> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
>> mdadm: looking for devices for /dev/md127
>> mdadm: Reshape position is not suitably aligned.
>> mdadm: Try normal assembly and stop again
>>
> 
> What are the steps I need to take to resume the grow, but just enough
> to get it to the point I can revert it and see what data I have on the
> drives? Or can I force the revert?

I'm stumped.  Since your reshape position, while "0%", was 4GB into your
array, there's no way to use --create --assume-clean to fix this.

Unless Neil knows some magic, your data is toast.  Sorry.

Phil

Re: hung grow

[Anthony Youngman]

On 10/10/17 14:06, Phil Turmel wrote:
> On 10/10/2017 08:08 AM, Curt wrote:
>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>>> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>>>> What happens if you run the --assemble --update=revert-reshape command on
>>>> these 6 devices (without --force)??
>>>
>>> Here's the output
>>> mdadm --assemble --verbose --update=revert-reshape /dev/md127
>>> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
>>> mdadm: looking for devices for /dev/md127
>>> mdadm: Reshape position is not suitably aligned.
>>> mdadm: Try normal assembly and stop again
>>>
>>
>> What are the steps I need to take to resume the grow, but just enough
>> to get it to the point I can revert it and see what data I have on the
>> drives? Or can I force the revert?
> 
> I'm stumped.  Since your reshape position, while "0%", was 4GB into your
> array, there's no way to use --create --assume-clean to fix this.
> 
> Unless Neil knows some magic, your data is toast.  Sorry.
> 
I guess the only chance is to try and get all the drives that were 
reshaping (or copies of them) in a fit state to try and re-assemble the 
array. Would the revert work then? Or would the reshape crash on hitting 
the first bad sector?

Cheers,
Wol

Re: hung grow

[Phil Turmel]

On 10/10/2017 09:37 AM, Anthony Youngman wrote:
> On 10/10/17 14:06, Phil Turmel wrote:
>> On 10/10/2017 08:08 AM, Curt wrote:
>>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:

>>>> mdadm: Reshape position is not suitably aligned.

I've never seen this before, and don't have time at the moment
to read the code to understand what it means.

>> Unless Neil knows some magic, your data is toast.  Sorry.

> I guess the only chance is to try and get all the drives that were
> reshaping (or copies of them) in a fit state to try and re-assemble the
> array. Would the revert work then?

Probably.  But yeah, "fit state".  That's the magic. /-:

> Or would the reshape crash on hitting
> the first bad sector?

The two devices with pending sectors have been ddrescue'd, so no.

Phil

Re: hung grow

[Curt]

>
> I've never seen this before, and don't have time at the moment
> to read the code to understand what it means.
I did a search and here's the comment before the code I found spits
out that error message.  I don't know if it's useful to you
/* reshape_position is a little messy.
* Its value must be a multiple of the larger
* chunk size, and of the "after" data disks.
* So when reverting we need to change it to
* be a multiple of the new "after" data disks,
* which is the old "before".
* If it isn't already a multiple of 'before',
* the only thing we could do would be
* copy some block around on the disks, which
* is easy to get wrong.
* So we reject a revert-reshape unless the
* alignment is good.
*/
if (reshape_sectors % reshape_chunk)

That doesn't read good for me.
>
>
> Probably.  But yeah, "fit state".  That's the magic. /-:
>
What are the chances if I just do an assemble and let it try to grow
again that it will complete without issue? Then assuming it completes,
that I will have most my data still?

Cheers,
Curt

Re: hung grow

[Reindl Harald]

Am 10.10.2017 um 16:11 schrieb Curt:
> What are the chances if I just do an assemble and let it try to grow
> again that it will complete without issue? Then assuming it completes,
> that I will have most my data still?

low

what is "most of my data" in case of an array when your data are on top 
of that array in a filesystem which don't know anything about what 
happened with the phyiscal stuff underlying?

Re: hung grow

[Phil Turmel]

On 10/10/2017 10:11 AM, Curt wrote:
>>
>> I've never seen this before, and don't have time at the moment
>> to read the code to understand what it means.
> I did a search and here's the comment before the code I found spits
> out that error message.  I don't know if it's useful to you
> /* reshape_position is a little messy.
> * Its value must be a multiple of the larger
> * chunk size, and of the "after" data disks.
> * So when reverting we need to change it to
> * be a multiple of the new "after" data disks,
> * which is the old "before".
> * If it isn't already a multiple of 'before',
> * the only thing we could do would be
> * copy some block around on the disks, which
> * is easy to get wrong.
> * So we reject a revert-reshape unless the
> * alignment is good.
> */
> if (reshape_sectors % reshape_chunk)
> 
> That doesn't read good for me.

Yup.

>> Probably.  But yeah, "fit state".  That's the magic. /-:
>>
> What are the chances if I just do an assemble and let it try to grow
> again that it will complete without issue? Then assuming it completes,
> that I will have most my data still?

Yeah, try without the revert.  You don't have to wait for the reshape to
finish to grab your critical files off of it.  In fact, you might want
to freeze the reshape to minimize the chance of hitting a not-yet-found URE.

Phil

Re: hung grow

[Curt]

>
> Yeah, try without the revert.  You don't have to wait for the reshape to
> finish to grab your critical files off of it.  In fact, you might want
> to freeze the reshape to minimize the chance of hitting a not-yet-found URE.

Ok, so try the before command, but instead of --update=revert-reshape,
do a --update=freeze-reshape?

Re: hung grow

[Phil Turmel]

On 10/10/2017 10:23 AM, Curt wrote:
>>
>> Yeah, try without the revert.  You don't have to wait for the reshape to
>> finish to grab your critical files off of it.  In fact, you might want
>> to freeze the reshape to minimize the chance of hitting a not-yet-found URE.
> 
> Ok, so try the before command, but instead of --update=revert-reshape,
> do a --update=freeze-reshape?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Just --freeze-reshape, not --update.

Re: hung grow

[Curt]

>
> Just --freeze-reshape, not --update.
>
Ok, here's the output
mdadm --detail /dev/md127
/dev/md127:
           Version : 0.91
     Creation Time : Fri Jun 15 15:52:05 2012
        Raid Level : raid6
        Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
     Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
      Raid Devices : 8
     Total Devices : 6
   Preferred Minor : 127
       Persistence : Superblock is persistent

       Update Time : Tue Oct 10 15:11:26 2017
             State : clean, FAILED, reshaping
    Active Devices : 5
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

    Reshape Status : 0% complete
     Delta Devices : 1, (7->8)

              UUID : 714a612d:9bd35197:36c91ae3:c168144d
            Events : 0.11559682

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       3       8        1        3      active sync   /dev/sda1
       4      65      145        4      active sync   /dev/sdz1
       -       0        0        5      removed
       6       8       16        6      spare rebuilding   /dev/sdb
       -       0        0        7      removed

But in my dmesg, I'm seeing task md127_reshape blocked for 120
seconds, and when I cat sync_action, it shows reshape.  Which
shouldn't it be frozen or something like that?  Also md127_raid6 task
is using 100% cpu.  I was going to paste the assemble output, but hit
clear instead of copy.  It didn't show any errors I saw, just starting
with 6 drives. reshape isn't using any cpu

If I do a cat of /proc/pid/stack, all I get is
[<ffffffffffffffff>] 0xffffffffffffffff

Should I just let it run?


Here's dmesg output for the raid stuff
521267.896376]  sdb: unknown partition table
[521267.896766] md: bind<sdd1>
[521267.896900] md: bind<sdc1>
[521267.897089] md: bind<sda1>
[521267.897336] md: bind<sdz1>
[521267.902584] md: bind<sdb>
[521267.918722] md: bind<sdg1>
[521268.013150] md/raid:md127: reshape will continue
[521268.013168] md/raid:md127: device sdg1 operational as raid disk 0
[521268.013170] md/raid:md127: device sdz1 operational as raid disk 4
[521268.013172] md/raid:md127: device sda1 operational as raid disk 3
[521268.013173] md/raid:md127: device sdc1 operational as raid disk 2
[521268.013175] md/raid:md127: device sdd1 operational as raid disk 1
[521268.014065] md/raid:md127: allocated 0kB
[521268.014096] md/raid:md127: raid level 6 active with 6 out of 8
devices, algorithm 2
[521268.019158] RAID conf printout:
[521268.019160]  --- level:6 rd:8 wd:6
[521268.019162]  disk 0, o:1, dev:sdg1
[521268.019164]  disk 1, o:1, dev:sdd1
[521268.019166]  disk 2, o:1, dev:sdc1
[521268.019167]  disk 3, o:1, dev:sda1
[521268.019169]  disk 4, o:1, dev:sdz1
[521268.019171]  disk 6, o:1, dev:sdb
[521268.019208] md127: Warning: Device sdz1 is misaligned
[521268.019210] md127: Warning: Device sda1 is misaligned
[521268.019211] md127: Warning: Device sda1 is misaligned
[521268.019236] md127: detected capacity change from 0 to 10001939824640
[521268.019242] md: reshape of RAID array md127
[521268.019248] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[521268.019249] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for reshape.
[521268.019255] md: using 128k window, over a total of 1953503872k.

Re: hung grow

[Phil Turmel]

On 10/10/2017 03:25 PM, Curt wrote:
>>
>> Just --freeze-reshape, not --update.
>>
> Ok, here's the output

>              State : clean, FAILED, reshaping

Hmmm. Both clean and failed.  If you can mount it, do so and copy what
you can (in order of importance).

If you can't, I'm out of ideas.

Phil

Re: hung grow

[Curt]

On Tue, Oct 10, 2017 at 3:42 PM, Phil Turmel <philip@turmel.org> wrote:
> On 10/10/2017 03:25 PM, Curt wrote:
>>>
>>> Just --freeze-reshape, not --update.
>>>
>> Ok, here's the output
>
>>              State : clean, FAILED, reshaping
>
> Hmmm. Both clean and failed.  If you can mount it, do so and copy what
> you can (in order of importance).
>
> If you can't, I'm out of ideas.
>
> Phil

I tried a mount, it just sits there.  No errors, which I'm not sure if
that's good or bad, so it's almost like it's just waiting for the
device to not be busy.  I noticed it was lists both a reshape and
spare rebuild.  Could it be it's trying to rebuild the spare and
getting in some kind of dead lock?

Re: hung grow

[Curt]

> I tried a mount, it just sits there.  No errors, which I'm not sure if
> that's good or bad, so it's almost like it's just waiting for the
> device to not be busy.  I noticed it was lists both a reshape and
> spare rebuild.  Could it be it's trying to rebuild the spare and
> getting in some kind of dead lock?

Follow up, here's the top process, does this look right to you?
9771 root      20   0       0      0      0 R 100.0  0.0  38:27.01 md127_raid6

Re: hung grow

[Phil Turmel]

On 10/10/2017 03:51 PM, Curt wrote:
>> I tried a mount, it just sits there.  No errors, which I'm not sure if
>> that's good or bad, so it's almost like it's just waiting for the
>> device to not be busy.  I noticed it was lists both a reshape and
>> spare rebuild.  Could it be it's trying to rebuild the spare and
>> getting in some kind of dead lock?
> 
> Follow up, here's the top process, does this look right to you?
> 9771 root      20   0       0      0      0 R 100.0  0.0  38:27.01 md127_raid6

No, that's not good. And blocking the mount process is a disaster.

Please take notes on all device serials vs. MD device # and consider
trying everything with rescue distro.  I generally use SystemRescueCD,
but anything with the latest stable kernel and mdadm might help.

Phil

Re: hung grow

[Curt]

>
> No, that's not good. And blocking the mount process is a disaster.
>
> Please take notes on all device serials vs. MD device # and consider
> trying everything with rescue distro.  I generally use SystemRescueCD,
> but anything with the latest stable kernel and mdadm might help.
>
> Phil
>

Figured.  Doesn't seem like it's really doing anything, outside taking
100% cpu. Doing a cat of /proc/pid/io shows all 0's.

Anyway.  Should I just run an mdadm --stop on the md127 or doing you
think I'll have to do a reboot?

After it's stopped. I'm guessing just boot with a rescue CD and try to
do an assemble and then mount and hope it doesn't do the same thing?

Cheers,
Curt

Re: hung grow

[Phil Turmel]

On 10/10/2017 04:29 PM, Curt wrote:
>>
>> No, that's not good. And blocking the mount process is a disaster.
>>
>> Please take notes on all device serials vs. MD device # and consider
>> trying everything with rescue distro.  I generally use SystemRescueCD,
>> but anything with the latest stable kernel and mdadm might help.
>>
>> Phil
>>
> 
> Figured.  Doesn't seem like it's really doing anything, outside taking
> 100% cpu. Doing a cat of /proc/pid/io shows all 0's.
> 
> Anyway.  Should I just run an mdadm --stop on the md127 or doing you
> think I'll have to do a reboot?

Use --stop.  It might work and would be good if it did.

> After it's stopped. I'm guessing just boot with a rescue CD and try to
> do an assemble and then mount and hope it doesn't do the same thing?

Yes.  Device letters might change -- that's why you need serial numbers
before you shut down.

Phil

Re: hung grow

[Curt]

>
> Use --stop.  It might work and would be good if it did.

Ok, that's a negative on the stop. It just hung, can't break out.
Anything I should do before I have to reboot this server?

>
>> After it's stopped. I'm guessing just boot with a rescue CD and try to
>> do an assemble and then mount and hope it doesn't do the same thing?
>
> Yes.  Device letters might change -- that's why you need serial numbers
> before you shut down.
>
Ok. I'll probably just yank the drives and bring them back to my
place.  I have a spare server sitting on a work bench here, so I can
put them in that, so they'll be the only drives.  I'll note the serial
info before I do, though.
> Phil
>

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]

On Tue, Oct 10 2017, Curt wrote:

>>
>> Just --freeze-reshape, not --update.
>>
> Ok, here's the output
> mdadm --detail /dev/md127
> /dev/md127:
>            Version : 0.91
>      Creation Time : Fri Jun 15 15:52:05 2012
>         Raid Level : raid6
>         Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
>      Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
>       Raid Devices : 8
>      Total Devices : 6
>    Preferred Minor : 127
>        Persistence : Superblock is persistent
>
>        Update Time : Tue Oct 10 15:11:26 2017
>              State : clean, FAILED, reshaping
>     Active Devices : 5
>    Working Devices : 6
>     Failed Devices : 0
>      Spare Devices : 1
>
>             Layout : left-symmetric
>         Chunk Size : 64K
>
> Consistency Policy : unknown
>
>     Reshape Status : 0% complete
>      Delta Devices : 1, (7->8)
>
>               UUID : 714a612d:9bd35197:36c91ae3:c168144d
>             Events : 0.11559682
>
>     Number   Major   Minor   RaidDevice State
>        0       8       97        0      active sync   /dev/sdg1
>        1       8       49        1      active sync   /dev/sdd1
>        2       8       33        2      active sync   /dev/sdc1
>        3       8        1        3      active sync   /dev/sda1
>        4      65      145        4      active sync   /dev/sdz1
>        -       0        0        5      removed
>        6       8       16        6      spare rebuilding   /dev/sdb
>        -       0        0        7      removed
>
> But in my dmesg, I'm seeing task md127_reshape blocked for 120
> seconds, and when I cat sync_action, it shows reshape.  Which
> shouldn't it be frozen or something like that?  Also md127_raid6 task
> is using 100% cpu.  I was going to paste the assemble output, but hit
> clear instead of copy.  It didn't show any errors I saw, just starting
> with 6 drives. reshape isn't using any cpu
>
> If I do a cat of /proc/pid/stack, all I get is
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Should I just let it run?

Clearly a kernel bug.
What kernel are you using?  Can you try a newer one easily?

Can you please
  mkdir /tmp/dump
  mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
  tar czf --sparse /tmp/dump.tgz /tmp/dump

and send me /tmp/dump.tgz.  It will only contains the metadata.
I can then create an identical looking array and experiment.

I doubt if letting it run will bring benefits.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:

>
> Clearly a kernel bug.
> What kernel are you using?  Can you try a newer one easily?

3.10.0-229.el7.x86_64.  I could either update the OS or transfer to a
new server and use a rescue cd as mentioned before.
>
> Can you please
>   mkdir /tmp/dump
>   mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>   tar czf --sparse /tmp/dump.tgz /tmp/dump
>
 Yeah, I will get that for you once I reboot the server.

> and send me /tmp/dump.tgz.  It will only contains the metadata.
> I can then create an identical looking array and experiment.
>
> I doubt if letting it run will bring benefits.
>
> NeilBrown

Re: hung grow

[Curt]

On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:

>> Can you please
>>   mkdir /tmp/dump
>>   mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>   tar czf --sparse /tmp/dump.tgz /tmp/dump
>>
>  Yeah, I will get that for you once I reboot the server.
>
Quick question.  Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

On Tue, Oct 10 2017, Curt wrote:

> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>
>>> Can you please
>>>   mkdir /tmp/dump
>>>   mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>>   tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>
>>  Yeah, I will get that for you once I reboot the server.
>>
> Quick question.  Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?

"/tmp/dump" - yes,  I typed the wrong thing.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Tue, Oct 10, 2017 at 5:56 PM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Oct 10 2017, Curt wrote:
>
>> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>>
>>>> Can you please
>>>>   mkdir /tmp/dump
>>>>   mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>>>   tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>>
>>>  Yeah, I will get that for you once I reboot the server.
>>>
>> Quick question.  Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
>
> "/tmp/dump" - yes,  I typed the wrong thing.
>
> Thanks,
> NeilBrown

Out of curiosity, how long should it take to tar these files?  I get
the system reports them as 1.9T and there not really that big, hence
the sparse switch. I've just never tarred a sparse file file before.
It's been churning away for about 2 hours, which I'm going to assume
is normal.

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

On Tue, Oct 10 2017, Curt wrote:

> On Tue, Oct 10, 2017 at 5:56 PM, NeilBrown <neilb@suse.com> wrote:
>> On Tue, Oct 10 2017, Curt wrote:
>>
>>> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>>>
>>>>> Can you please
>>>>>   mkdir /tmp/dump
>>>>>   mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>>>>   tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>>>
>>>>  Yeah, I will get that for you once I reboot the server.
>>>>
>>> Quick question.  Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
>>
>> "/tmp/dump" - yes,  I typed the wrong thing.
>>
>> Thanks,
>> NeilBrown
>
> Out of curiosity, how long should it take to tar these files?  I get
> the system reports them as 1.9T and there not really that big, hence
> the sparse switch. I've just never tarred a sparse file file before.
> It's been churning away for about 2 hours, which I'm going to assume
> is normal.

With a sufficiently recent 'tar' and kernel (I don't know how recent
they need to be) it should be nearly instantaneous.  With older code it
could easily take hours.

The kernel needs to support SEEK_HOLE (v3.1 introduced that) and
tar need to support "--hole-detection=seek", which is the default if it
is supported.  I don't know what version.

NeilBrown


> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:
> Clearly a kernel bug.
> What kernel are you using?  Can you try a newer one easily?
>
 I just tried with Kernel 4.13 and got the same results.  Could it be
another library that wouldn't be updated with the kernel?

I'll see if I can't get to the data center tomorrow with a rescueCD
and try that. Any other suggestions or further information till then?

Cheers,
Curt

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Tue, Oct 10 2017, Curt wrote:

> On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:
>> Clearly a kernel bug.
>> What kernel are you using?  Can you try a newer one easily?
>>
>  I just tried with Kernel 4.13 and got the same results.  Could it be
> another library that wouldn't be updated with the kernel?

Nope.  It is a kernel bug that is still present.  I can easily reproduce
it on the mainline kernel now that I have your metadata.
I know generally what the problem is.  I'll try to craft a fix tomorrow.
You'll need to build and run a new kernel to be able to fix you array.
That might seem scary, but it really isn't that hard.

NeilBrown


>
> I'll see if I can't get to the data center tomorrow with a rescueCD
> and try that. Any other suggestions or further information till then?
>
> Cheers,
> Curt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Curt]

On Wed, Oct 11, 2017 at 12:49 AM, NeilBrown <neilb@suse.com> wrote:
>
> Nope.  It is a kernel bug that is still present.  I can easily reproduce
> it on the mainline kernel now that I have your metadata.
> I know generally what the problem is.  I'll try to craft a fix tomorrow.
> You'll need to build and run a new kernel to be able to fix you array.
> That might seem scary, but it really isn't that hard.
>
> NeilBrown
 Awesome thanks.  Just let me know what I need to do.

Cheers,
Curt

Re: hung grow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

On Wed, Oct 11 2017, Curt wrote:

> On Wed, Oct 11, 2017 at 12:49 AM, NeilBrown <neilb@suse.com> wrote:
>>
>> Nope.  It is a kernel bug that is still present.  I can easily reproduce
>> it on the mainline kernel now that I have your metadata.
>> I know generally what the problem is.  I'll try to craft a fix tomorrow.
>> You'll need to build and run a new kernel to be able to fix you array.
>> That might seem scary, but it really isn't that hard.
>>
>> NeilBrown
>  Awesome thanks.  Just let me know what I need to do.

I had hoped to have this for you today, but it didn't quite work out.  I
have a patch which I believe is correct, but I try not to trust myself
about such things after about 3pm.
Fixing the spin turned out to be very easy, but subsequent testing
showed another problem.  'sdb' is not fully recovered but after the
reverted reshape completed, md thought that it was.  It took a while to
figure out exactly what was going on there.

If nothing urgent intervenes tomorrow I'll review the patches and then
send them along with instructions on how to build a kernel.
It might help to know what distro you plan to do that on.
The key steps will be:
1/ get the kernel source
2/ apply the patch
3/ get a .config that matches what your distro kernel
4/ build the kernel
5/ make install modules_install
6/ reboot

The kernel source is simply https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.13.5.tar.xz
The "make install.." assumes your distro has the right helpers
installed.
The ".config" might be
   zcat /proc/config.gz > .config
If not, you'll need to hunt for it.  Someone can probably help.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hung grow

[Anthony Youngman]

On 10/10/17 15:00, Phil Turmel wrote:
> On 10/10/2017 09:37 AM, Anthony Youngman wrote:
>> On 10/10/17 14:06, Phil Turmel wrote:
>>> On 10/10/2017 08:08 AM, Curt wrote:
>>>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
> 
>>>>> mdadm: Reshape position is not suitably aligned.
> 
> I've never seen this before, and don't have time at the moment
> to read the code to understand what it means.

My reaction (having read the thread :-) is that one of the drives being 
used to try to re-assemble the array was dd'd from a drive that failed 
before the reshape. So that was "one of 7, no reshape". All of the 
others were "one of 9, reshape in progress, same position". So if we can 
get the drives that were reshaping, we should be able to revert it. 
Question is, can we?
> 
>>> Unless Neil knows some magic, your data is toast.  Sorry.
> 
>> I guess the only chance is to try and get all the drives that were
>> reshaping (or copies of them) in a fit state to try and re-assemble the
>> array. Would the revert work then?
> 
> Probably.  But yeah, "fit state".  That's the magic. /-:

So if Curt can do it, I think we stand a chance. A chance.
> 
>> Or would the reshape crash on hitting
>> the first bad sector?
> 
> The two devices with pending sectors have been ddrescue'd, so no.
> 
We'll just have a few bits of corruption, but then we knew that. Three 
drives gone from a raid-6, and any successful recovery is going to have 
*some* damage.

Cheers,
Wol

Re: hung grow

[Anthony Youngman]

On 04/10/17 19:37, Curt wrote:
> Assuming I can do a dd on one of my failed drives, will I be able to
> recover the data that's on the 4 that were good, before I took bad
> advice?  Also, will I need to dd on the failed drives or can I do 2 of
> the 3?

First, use ddrescue, not dd.

Secondly, you only need one drive to give you "5 out of 7". The most 
recent drive the better!

Cheers,
Wol

Re: hung grow

[Anthony Youngman]

On 04/10/17 19:16, Curt wrote:
> Hi,
> 
> I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery
> 
> I don't have any spare bays on that server...I'd have to make a trip
> to my datacenter and bring the drives back to my house.  The bad thing
> is the 2 drives I replaced, failed a while ago, so they were behind.
> I was hoping I could still use the 4 drives I had before I did a grow
> on them.  Do they need to be up-to-date or do I just need the config
> from them to recover the 3 drives that were still good?
> 
> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
> one just recently. FML
> 
Okay, that makes it a lot clearer what happened. First things first. 
That "grow" is trying to change your array from "7 with 3 failed" to "9 
with 3 failed". A complete balls-up. Sorry.

So firstly. We NEED to stop that grow. I think the option you want is 
--revert-reshape, but I'd rather an expert chimed in and said I've got 
it right.

Secondly, you NEED to ddrescue that failed drive number 5. If that drive 
is toast, then so is your array :-( If that means a trip to the 
datacentre then so be it. What you really don't want is for that drive 
to die beyond recovery before you get the chance to copy it. If it does 
appear to be toast, ime, leaving it powered off for a few days MAY give 
you the opportunity to recover it. Other people swear by putting it in 
the freezer. If it's dead, what have you got to lose?

Once you've got that far, you can now forcibly assemble your array, 
which will give you a "7 with 2 failed" working but degraded array.

Now you can do what you *should* have done with your two new drives - do 
a "--fail --remove --add" to delete the failed drives and put the new 
drives in.

Nice and simple, but it all depends on that failed drive ... if the 
worst does come to the worst, and the data is valuable enough, head 
crashes are rare nowadays and a specialist recovery firm may well be 
able to salvage it for you, but that won't be cheap ...

Another "worst case" recovery scenario is to force in your replacement 
drive with "--assume-clean" - I'm not sure how to do that ... But it 
will give you a working array with a LOT of damage. With five drives, 
though, that will give you a good chance of recovering a lot of the 
files with some data recovery software.

Cheers,
Wol

> Cheers,
> Curt
> 
> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 18:18, Curt wrote:
>>>
>>> Is my raid completely fucked or can I still recover some data with
>>> doing the create assume clean?
>>
>>
>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>
>> I take it you haven't read the raid wiki?
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>
>> The bad news is your array is well borked. The good news is I don't think
>> you have - YET - managed to bork it irretrievably. A create will almost
>> certainly trash it beyond recovery!!!
>>
>> I think we can stop/revert the grow, and get the array back to a usable
>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>
>> Do you have spare SATA ports? So you have the bad drives you replaced (can
>> you ddrescue them on to new drives?). What was the original configuration of
>> the raid - you say you lost three drives, but how many did you have to start
>> with?
>>
>> I'll let the experts talk you through the actual recovery, but the steps
>> need to be to revert the grow, ddrescue the best of your failed drives,
>> force an assembly, and then replace the other two failed drives. No
>> guarantees as to how much data will be left at the end, although hopefully
>> we'll save most of it.
>>
>> Cheers,
>> Wol
>