* hung grow
@ 2017-10-04 17:18 Curt
2017-10-04 17:51 ` Anthony Youngman
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-04 17:18 UTC (permalink / raw)
To: linux-raid
Hello all,
So, I got my raid6 is a half fucked state you could say. My raid6 lost
3 drives and then starting throwing I/O errors on the mount point. If
I tried to restart with the good drives, I got too few to start raid.
So one that got marked faulty, almost matched the even count of the
others, only off by a few. So I did a assemble force, which seemed to
work and I could see my data. I should have just pulled it off at
that point and saved what I could, but no.
So I replaced 2 of the bad drives and added them to the raid, it went
through recovery, but only marked the 2 new drives as spares and
showed the bad removed, 2 spares, and it marked one faulty aging, see
the below.
uname -a
Linux dev 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
mdadm --detail /dev/md127
/dev/md127:
Version : 0.90
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Persistence : Superblock is persistent
Update Time : Tue Oct 3 22:22:06 2017
State : clean, FAILED
Active Devices : 4
Working Devices : 6
Failed Devices : 1
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : unknown
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Events : 0.11559613
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 49 1 active sync /dev/sdd1
2 8 33 2 active sync /dev/sdc1
3 8 1 3 active sync /dev/sda1
- 0 0 4 removed
- 0 0 5 removed
- 0 0 6 removed
7 8 80 - spare /dev/sdf
8 8 16 - spare /dev/sdb
9 8 65 - faulty /dev/sde1
after several tries to reassemble, the spares wouldn't got active. So
on the advice of somenone, I set the raid to grow to 8, the theory
being it would make on spare active. Which somewhat worked, but grow
froze at 0% and when I do a detail on md127 it just hangs, it returned
once when this first started and showed the spare in spare rebuilding
status, but sync_action showed reshape and mdstat.
examine returns this, it's the same for all as far as I can see
mdadm --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 10:10:37 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71846f - correct
Events : 11559679
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 1 3 active sync /dev/sda1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
Is my raid completely fucked or can I still recover some data with
doing the create assume clean?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 17:18 hung grow Curt
@ 2017-10-04 17:51 ` Anthony Youngman
2017-10-04 18:16 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 17:51 UTC (permalink / raw)
To: Curt, linux-raid
On 04/10/17 18:18, Curt wrote:
> Is my raid completely fucked or can I still recover some data with
> doing the create assume clean?
PLEASE PLEASE PLEASE DON'T !!!!!!
I take it you haven't read the raid wiki?
https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
The bad news is your array is well borked. The good news is I don't
think you have - YET - managed to bork it irretrievably. A create will
almost certainly trash it beyond recovery!!!
I think we can stop/revert the grow, and get the array back to a usable
state, where we can force an assemble. If a bit of data gets lost, sorry.
Do you have spare SATA ports? So you have the bad drives you replaced
(can you ddrescue them on to new drives?). What was the original
configuration of the raid - you say you lost three drives, but how many
did you have to start with?
I'll let the experts talk you through the actual recovery, but the steps
need to be to revert the grow, ddrescue the best of your failed drives,
force an assembly, and then replace the other two failed drives. No
guarantees as to how much data will be left at the end, although
hopefully we'll save most of it.
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 17:51 ` Anthony Youngman
@ 2017-10-04 18:16 ` Curt
2017-10-04 18:29 ` Joe Landman
2017-10-04 18:57 ` Anthony Youngman
0 siblings, 2 replies; 59+ messages in thread
From: Curt @ 2017-10-04 18:16 UTC (permalink / raw)
To: Anthony Youngman; +Cc: linux-raid
Hi,
I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery
I don't have any spare bays on that server...I'd have to make a trip
to my datacenter and bring the drives back to my house. The bad thing
is the 2 drives I replaced, failed a while ago, so they were behind.
I was hoping I could still use the 4 drives I had before I did a grow
on them. Do they need to be up-to-date or do I just need the config
from them to recover the 3 drives that were still good?
Oh, I originally started with 7, 2 failed a few moths back and the 3rd
one just recently. FML
Cheers,
Curt
On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 18:18, Curt wrote:
>>
>> Is my raid completely fucked or can I still recover some data with
>> doing the create assume clean?
>
>
> PLEASE PLEASE PLEASE DON'T !!!!!!
>
> I take it you haven't read the raid wiki?
>
> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>
> The bad news is your array is well borked. The good news is I don't think
> you have - YET - managed to bork it irretrievably. A create will almost
> certainly trash it beyond recovery!!!
>
> I think we can stop/revert the grow, and get the array back to a usable
> state, where we can force an assemble. If a bit of data gets lost, sorry.
>
> Do you have spare SATA ports? So you have the bad drives you replaced (can
> you ddrescue them on to new drives?). What was the original configuration of
> the raid - you say you lost three drives, but how many did you have to start
> with?
>
> I'll let the experts talk you through the actual recovery, but the steps
> need to be to revert the grow, ddrescue the best of your failed drives,
> force an assembly, and then replace the other two failed drives. No
> guarantees as to how much data will be left at the end, although hopefully
> we'll save most of it.
>
> Cheers,
> Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:16 ` Curt
@ 2017-10-04 18:29 ` Joe Landman
2017-10-04 18:37 ` Curt
2017-10-04 18:57 ` Anthony Youngman
1 sibling, 1 reply; 59+ messages in thread
From: Joe Landman @ 2017-10-04 18:29 UTC (permalink / raw)
To: Curt, Anthony Youngman; +Cc: linux-raid
On 10/04/2017 02:16 PM, Curt wrote:
> Hi,
>
> I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery
>
> I don't have any spare bays on that server...I'd have to make a trip
> to my datacenter and bring the drives back to my house. The bad thing
> is the 2 drives I replaced, failed a while ago, so they were behind.
> I was hoping I could still use the 4 drives I had before I did a grow
> on them. Do they need to be up-to-date or do I just need the config
> from them to recover the 3 drives that were still good?
>
> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
> one just recently. FML
Er ... honestly, I hope you have a backup.
If the drives are really dead, and can't be seen with lsscsi or cat
/proc/scsi/scsi , then your raid is probably gone.
If they can be seen, the ddrescue is your best option right now.
Do not grow the system. Stop that. Do nothing that changes metadata.
You may (remotely possibly) recover if you can copy the "dead" drives to
two new live ones.
>
> Cheers,
> Curt
>
> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 18:18, Curt wrote:
>>> Is my raid completely fucked or can I still recover some data with
>>> doing the create assume clean?
>>
>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>
>> I take it you haven't read the raid wiki?
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>
>> The bad news is your array is well borked. The good news is I don't think
>> you have - YET - managed to bork it irretrievably. A create will almost
>> certainly trash it beyond recovery!!!
>>
>> I think we can stop/revert the grow, and get the array back to a usable
>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>
>> Do you have spare SATA ports? So you have the bad drives you replaced (can
>> you ddrescue them on to new drives?). What was the original configuration of
>> the raid - you say you lost three drives, but how many did you have to start
>> with?
>>
>> I'll let the experts talk you through the actual recovery, but the steps
>> need to be to revert the grow, ddrescue the best of your failed drives,
>> force an assembly, and then replace the other two failed drives. No
>> guarantees as to how much data will be left at the end, although hopefully
>> we'll save most of it.
>>
>> Cheers,
>> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:29 ` Joe Landman
@ 2017-10-04 18:37 ` Curt
2017-10-04 18:44 ` Joe Landman
2017-10-04 19:06 ` Anthony Youngman
0 siblings, 2 replies; 59+ messages in thread
From: Curt @ 2017-10-04 18:37 UTC (permalink / raw)
To: Joe Landman; +Cc: Anthony Youngman, linux-raid
Hi Joe,
To clarify, the drives aren't completely dead. I can see/examine all
the drives currently in the array, the older ones I could see/examine,
but like I said they had been marked faulty for a while and event
count was way low. The grow never went anywhere, just stayed at 0%
with 100% CPU usage on md127_raid process. I have rebooted and am not
currently touching the drives.
Assuming I can do a dd on one of my failed drives, will I be able to
recover the data that's on the 4 that were good, before I took bad
advice? Also, will I need to dd on the failed drives or can I do 2 of
the 3?
On Wed, Oct 4, 2017 at 2:29 PM, Joe Landman <joe.landman@gmail.com> wrote:
>
>
> On 10/04/2017 02:16 PM, Curt wrote:
>>
>> Hi,
>>
>> I was reading this one
>> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>
>> I don't have any spare bays on that server...I'd have to make a trip
>> to my datacenter and bring the drives back to my house. The bad thing
>> is the 2 drives I replaced, failed a while ago, so they were behind.
>> I was hoping I could still use the 4 drives I had before I did a grow
>> on them. Do they need to be up-to-date or do I just need the config
>> from them to recover the 3 drives that were still good?
>>
>> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
>> one just recently. FML
>
>
> Er ... honestly, I hope you have a backup.
>
> If the drives are really dead, and can't be seen with lsscsi or cat
> /proc/scsi/scsi , then your raid is probably gone.
>
> If they can be seen, the ddrescue is your best option right now.
>
> Do not grow the system. Stop that. Do nothing that changes metadata.
>
> You may (remotely possibly) recover if you can copy the "dead" drives to two
> new live ones.
>
>>
>> Cheers,
>> Curt
>>
>> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
>> <antlists@youngman.org.uk> wrote:
>>>
>>> On 04/10/17 18:18, Curt wrote:
>>>>
>>>> Is my raid completely fucked or can I still recover some data with
>>>> doing the create assume clean?
>>>
>>>
>>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>>
>>> I take it you haven't read the raid wiki?
>>>
>>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>>
>>> The bad news is your array is well borked. The good news is I don't think
>>> you have - YET - managed to bork it irretrievably. A create will almost
>>> certainly trash it beyond recovery!!!
>>>
>>> I think we can stop/revert the grow, and get the array back to a usable
>>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>>
>>> Do you have spare SATA ports? So you have the bad drives you replaced
>>> (can
>>> you ddrescue them on to new drives?). What was the original configuration
>>> of
>>> the raid - you say you lost three drives, but how many did you have to
>>> start
>>> with?
>>>
>>> I'll let the experts talk you through the actual recovery, but the steps
>>> need to be to revert the grow, ddrescue the best of your failed drives,
>>> force an assembly, and then replace the other two failed drives. No
>>> guarantees as to how much data will be left at the end, although
>>> hopefully
>>> we'll save most of it.
>>>
>>> Cheers,
>>> Wol
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> Joe Landman
> e: joe.landman@gmail.com
> t: @hpcjoe
> w: https://scalability.org
> g: https://github.com/joelandman
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:37 ` Curt
@ 2017-10-04 18:44 ` Joe Landman
2017-10-04 19:01 ` Anthony Youngman
2017-10-04 19:06 ` Anthony Youngman
1 sibling, 1 reply; 59+ messages in thread
From: Joe Landman @ 2017-10-04 18:44 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, linux-raid
On 10/04/2017 02:37 PM, Curt wrote:
> Hi Joe,
>
> To clarify, the drives aren't completely dead. I can see/examine all
> the drives currently in the array, the older ones I could see/examine,
> but like I said they had been marked faulty for a while and event
> count was way low. The grow never went anywhere, just stayed at 0%
> with 100% CPU usage on md127_raid process. I have rebooted and am not
> currently touching the drives.
>
> Assuming I can do a dd on one of my failed drives, will I be able to
> recover the data that's on the 4 that were good, before I took bad
> advice? Also, will I need to dd on the failed drives or can I do 2 of
> the 3?
Not sure. You will need to try to get back as much as you can off the
other original "bad" drives. If those drives are not "bad", you can
pull out the "new" drives, and put them in. See if you can force an
assembly of the RAID. If that works, you may have data (if the grow
didn't corrupt anything).
If this is the case, the very first thing you should do is find and copy
the data that you cannot lose from those drives, to another location,
quickly.
Before you take any more advice, I'd recommend seeing if you can
actually recover what you have now.
Generally speaking 3 failed drives on a RAID6 is a dead RAID6. You may
get lucky, in that this may have been simply a timeout error (I've seen
these on consumer grade drives), or an internal operation on the drive
taking longer than normal, and been booted. In which case, you'll get
scary warning messages, but might get your data back.
Under no circumstances do anything to change RAID metadata right now
(grow, shrink, etc.). Start with basic assembly. If you can do that,
you are in good shape. If you can't, recovery is unlikely, even with
heroic intervention.
>
> On Wed, Oct 4, 2017 at 2:29 PM, Joe Landman <joe.landman@gmail.com> wrote:
>>
>> On 10/04/2017 02:16 PM, Curt wrote:
>>> Hi,
>>>
>>> I was reading this one
>>> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>>
>>> I don't have any spare bays on that server...I'd have to make a trip
>>> to my datacenter and bring the drives back to my house. The bad thing
>>> is the 2 drives I replaced, failed a while ago, so they were behind.
>>> I was hoping I could still use the 4 drives I had before I did a grow
>>> on them. Do they need to be up-to-date or do I just need the config
>>> from them to recover the 3 drives that were still good?
>>>
>>> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
>>> one just recently. FML
>>
>> Er ... honestly, I hope you have a backup.
>>
>> If the drives are really dead, and can't be seen with lsscsi or cat
>> /proc/scsi/scsi , then your raid is probably gone.
>>
>> If they can be seen, the ddrescue is your best option right now.
>>
>> Do not grow the system. Stop that. Do nothing that changes metadata.
>>
>> You may (remotely possibly) recover if you can copy the "dead" drives to two
>> new live ones.
>>
>>> Cheers,
>>> Curt
>>>
>>> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
>>> <antlists@youngman.org.uk> wrote:
>>>> On 04/10/17 18:18, Curt wrote:
>>>>> Is my raid completely fucked or can I still recover some data with
>>>>> doing the create assume clean?
>>>>
>>>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>>>
>>>> I take it you haven't read the raid wiki?
>>>>
>>>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>>>
>>>> The bad news is your array is well borked. The good news is I don't think
>>>> you have - YET - managed to bork it irretrievably. A create will almost
>>>> certainly trash it beyond recovery!!!
>>>>
>>>> I think we can stop/revert the grow, and get the array back to a usable
>>>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>>>
>>>> Do you have spare SATA ports? So you have the bad drives you replaced
>>>> (can
>>>> you ddrescue them on to new drives?). What was the original configuration
>>>> of
>>>> the raid - you say you lost three drives, but how many did you have to
>>>> start
>>>> with?
>>>>
>>>> I'll let the experts talk you through the actual recovery, but the steps
>>>> need to be to revert the grow, ddrescue the best of your failed drives,
>>>> force an assembly, and then replace the other two failed drives. No
>>>> guarantees as to how much data will be left at the end, although
>>>> hopefully
>>>> we'll save most of it.
>>>>
>>>> Cheers,
>>>> Wol
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> --
>> Joe Landman
>> e: joe.landman@gmail.com
>> t: @hpcjoe
>> w: https://scalability.org
>> g: https://github.com/joelandman
>>
--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:16 ` Curt
2017-10-04 18:29 ` Joe Landman
@ 2017-10-04 18:57 ` Anthony Youngman
1 sibling, 0 replies; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 18:57 UTC (permalink / raw)
To: Curt; +Cc: linux-raid
On 04/10/17 19:16, Curt wrote:
> Hi,
>
> I was reading this one https://raid.wiki.kernel.org/index.php/RAID_Recovery
>
> I don't have any spare bays on that server...I'd have to make a trip
> to my datacenter and bring the drives back to my house. The bad thing
> is the 2 drives I replaced, failed a while ago, so they were behind.
> I was hoping I could still use the 4 drives I had before I did a grow
> on them. Do they need to be up-to-date or do I just need the config
> from them to recover the 3 drives that were still good?
>
> Oh, I originally started with 7, 2 failed a few moths back and the 3rd
> one just recently. FML
>
Okay, that makes it a lot clearer what happened. First things first.
That "grow" is trying to change your array from "7 with 3 failed" to "9
with 3 failed". A complete balls-up. Sorry.
So firstly. We NEED to stop that grow. I think the option you want is
--revert-reshape, but I'd rather an expert chimed in and said I've got
it right.
Secondly, you NEED to ddrescue that failed drive number 5. If that drive
is toast, then so is your array :-( If that means a trip to the
datacentre then so be it. What you really don't want is for that drive
to die beyond recovery before you get the chance to copy it. If it does
appear to be toast, ime, leaving it powered off for a few days MAY give
you the opportunity to recover it. Other people swear by putting it in
the freezer. If it's dead, what have you got to lose?
Once you've got that far, you can now forcibly assemble your array,
which will give you a "7 with 2 failed" working but degraded array.
Now you can do what you *should* have done with your two new drives - do
a "--fail --remove --add" to delete the failed drives and put the new
drives in.
Nice and simple, but it all depends on that failed drive ... if the
worst does come to the worst, and the data is valuable enough, head
crashes are rare nowadays and a specialist recovery firm may well be
able to salvage it for you, but that won't be cheap ...
Another "worst case" recovery scenario is to force in your replacement
drive with "--assume-clean" - I'm not sure how to do that ... But it
will give you a working array with a LOT of damage. With five drives,
though, that will give you a good chance of recovering a lot of the
files with some data recovery software.
Cheers,
Wol
> Cheers,
> Curt
>
> On Wed, Oct 4, 2017 at 1:51 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 18:18, Curt wrote:
>>>
>>> Is my raid completely fucked or can I still recover some data with
>>> doing the create assume clean?
>>
>>
>> PLEASE PLEASE PLEASE DON'T !!!!!!
>>
>> I take it you haven't read the raid wiki?
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>>
>> The bad news is your array is well borked. The good news is I don't think
>> you have - YET - managed to bork it irretrievably. A create will almost
>> certainly trash it beyond recovery!!!
>>
>> I think we can stop/revert the grow, and get the array back to a usable
>> state, where we can force an assemble. If a bit of data gets lost, sorry.
>>
>> Do you have spare SATA ports? So you have the bad drives you replaced (can
>> you ddrescue them on to new drives?). What was the original configuration of
>> the raid - you say you lost three drives, but how many did you have to start
>> with?
>>
>> I'll let the experts talk you through the actual recovery, but the steps
>> need to be to revert the grow, ddrescue the best of your failed drives,
>> force an assembly, and then replace the other two failed drives. No
>> guarantees as to how much data will be left at the end, although hopefully
>> we'll save most of it.
>>
>> Cheers,
>> Wol
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:44 ` Joe Landman
@ 2017-10-04 19:01 ` Anthony Youngman
2017-10-04 19:09 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 19:01 UTC (permalink / raw)
To: Joe Landman, Curt; +Cc: linux-raid
On 04/10/17 19:44, Joe Landman wrote:
> Generally speaking 3 failed drives on a RAID6 is a dead RAID6. You may
> get lucky, in that this may have been simply a timeout error (I've seen
> these on consumer grade drives), or an internal operation on the drive
> taking longer than normal, and been booted. In which case, you'll get
> scary warning messages, but might get your data back.
>
> Under no circumstances do anything to change RAID metadata right now
> (grow, shrink, etc.). Start with basic assembly. If you can do that,
> you are in good shape. If you can't, recovery is unlikely, even with
> heroic intervention.
No - Curt needs to stop the grow before anything else will work, I
think. Fortunately, seeing as it's hung at 0% this shouldn't mess about
with the data at all.
And you NEED to ddrescue that fifth drive. At which point basic assembly
should hopefully work fine.
(Note that the grow quite likely failed because the fifth drive errored
... Oh - and are they raid drives? What are they? You don't have a
timeout mismatch?)
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 18:37 ` Curt
2017-10-04 18:44 ` Joe Landman
@ 2017-10-04 19:06 ` Anthony Youngman
1 sibling, 0 replies; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 19:06 UTC (permalink / raw)
To: Curt, Joe Landman; +Cc: Anthony Youngman, linux-raid
On 04/10/17 19:37, Curt wrote:
> Assuming I can do a dd on one of my failed drives, will I be able to
> recover the data that's on the 4 that were good, before I took bad
> advice? Also, will I need to dd on the failed drives or can I do 2 of
> the 3?
First, use ddrescue, not dd.
Secondly, you only need one drive to give you "5 out of 7". The most
recent drive the better!
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 19:01 ` Anthony Youngman
@ 2017-10-04 19:09 ` Curt
2017-10-04 19:46 ` Anthony Youngman
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-04 19:09 UTC (permalink / raw)
To: Anthony Youngman; +Cc: Joe Landman, linux-raid
Ok, thanks.
I'm pretty sure I'll be able to DD from at least one of the failed
drives, as I could still query them before I yanked them. Assuming I
can DD one of the old drives to one of my new ones.
I'd DDrescue old to new drive. Then do an assemble for force, with a
mix of the dd drives and my old good ones? So if sda/b are new DD'd
drives and sdc/d/e are hosed grow drives, I'd do an assemble force
revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
use my info from the DD drives to assemble the array back to 7 drives?
Did I understand that right?
Oh and how can I tell if I have a timeout mismatch. They should be raid drives.
Cheers,
Curt
On Wed, Oct 4, 2017 at 3:01 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 19:44, Joe Landman wrote:
>>
>> Generally speaking 3 failed drives on a RAID6 is a dead RAID6. You may
>> get lucky, in that this may have been simply a timeout error (I've seen
>> these on consumer grade drives), or an internal operation on the drive
>> taking longer than normal, and been booted. In which case, you'll get scary
>> warning messages, but might get your data back.
>>
>> Under no circumstances do anything to change RAID metadata right now
>> (grow, shrink, etc.). Start with basic assembly. If you can do that, you
>> are in good shape. If you can't, recovery is unlikely, even with heroic
>> intervention.
>
>
> No - Curt needs to stop the grow before anything else will work, I think.
> Fortunately, seeing as it's hung at 0% this shouldn't mess about with the
> data at all.
>
> And you NEED to ddrescue that fifth drive. At which point basic assembly
> should hopefully work fine.
>
> (Note that the grow quite likely failed because the fifth drive errored ...
> Oh - and are they raid drives? What are they? You don't have a timeout
> mismatch?)
>
> Cheers,
> Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 19:09 ` Curt
@ 2017-10-04 19:46 ` Anthony Youngman
2017-10-04 20:01 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 19:46 UTC (permalink / raw)
To: Curt; +Cc: Joe Landman, linux-raid
On 04/10/17 20:09, Curt wrote:
> Ok, thanks.
>
> I'm pretty sure I'll be able to DD from at least one of the failed
> drives, as I could still query them before I yanked them. Assuming I
> can DD one of the old drives to one of my new ones.
>
> I'd DDrescue old to new drive. Then do an assemble for force, with a
> mix of the dd drives and my old good ones? So if sda/b are new DD'd
> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
> use my info from the DD drives to assemble the array back to 7 drives?
> Did I understand that right?
This sounds like you need to take a great big step backwards, and make
sure you understand EXACTLY what is going on. We have a mix of good
drives, copies of bad drives, and an array that doesn't know whether it
is supposed to have 7 or 9 drives. One wrong step and your array will be
toast.
You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.
But I think the first thing we need to do, is to wait for an expert like
Phil to chime in and sort out that reshape. Your four good drives all
think they are part of a 9-drive array. Your first two drives to fail
think they are part of a 7-drive array. Does the third drive think it's
part of a 7-drive or 9-drive array?
Can you do a --examine on this drive? I suspect the grow blew up because
it couldn't access this drive. I this drive thinks it is part of a
7-drive array, we have a bit of a problem on our hands.
I'm hoping it thinks it's part of a 9-drive array - I think we may be
able to get out of this ...
>
> Oh and how can I tell if I have a timeout mismatch. They should be raid drives.
smartctl -x /dev/sdX
This will give you both the sort of drive you have - yes if it's in a
datacentre chances are it is a raid drive - and then search the output
for Error Recovery Control. This is from my hard drive...
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control
supported.
SCT Feature Control supported.
SCT Data Table supported.
You need error recovery to be supported. If it isn't ...
>
> Cheers,
> Curt
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 19:46 ` Anthony Youngman
@ 2017-10-04 20:01 ` Curt
2017-10-04 21:08 ` Anthony Youngman
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-04 20:01 UTC (permalink / raw)
To: Anthony Youngman; +Cc: Joe Landman, linux-raid
Hello,
Thanks for clarifying. All current good drives report that they are
part of a 8 drive array. I only grew the raid by 1 device, so it
would from 7-8, which is what they all report. The 3rd failed doesn't
report anything on examine, I haven't touched it at all and was not
included in my assemble. The 2 I replaced, the original drives I
yanked, think they are still part of a 7 drive array.
I'll be doing a ddrescue on the drives tonight, but will wait till
Phil or someone chimes in with my next steps after I do that.
lol, chalk one more up for FML. "SCT Error Recovery Control command
not supported". I'm guessing this is a real bad thing now? I didn't
buy these drives or org set it up.
On Wed, Oct 4, 2017 at 3:46 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> On 04/10/17 20:09, Curt wrote:
>>
>> Ok, thanks.
>>
>> I'm pretty sure I'll be able to DD from at least one of the failed
>> drives, as I could still query them before I yanked them. Assuming I
>> can DD one of the old drives to one of my new ones.
>>
>> I'd DDrescue old to new drive. Then do an assemble for force, with a
>> mix of the dd drives and my old good ones? So if sda/b are new DD'd
>> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
>> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
>> use my info from the DD drives to assemble the array back to 7 drives?
>> Did I understand that right?
>
>
> This sounds like you need to take a great big step backwards, and make sure
> you understand EXACTLY what is going on. We have a mix of good drives,
> copies of bad drives, and an array that doesn't know whether it is supposed
> to have 7 or 9 drives. One wrong step and your array will be toast.
>
> You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.
>
> But I think the first thing we need to do, is to wait for an expert like
> Phil to chime in and sort out that reshape. Your four good drives all think
> they are part of a 9-drive array. Your first two drives to fail think they
> are part of a 7-drive array. Does the third drive think it's part of a
> 7-drive or 9-drive array?
>
> Can you do a --examine on this drive? I suspect the grow blew up because it
> couldn't access this drive. I this drive thinks it is part of a 7-drive
> array, we have a bit of a problem on our hands.
>
> I'm hoping it thinks it's part of a 9-drive array - I think we may be able
> to get out of this ...
>>
>>
>> Oh and how can I tell if I have a timeout mismatch. They should be raid
>> drives.
>
>
> smartctl -x /dev/sdX
>
> This will give you both the sort of drive you have - yes if it's in a
> datacentre chances are it is a raid drive - and then search the output for
> Error Recovery Control. This is from my hard drive...
>
> SCT capabilities: (0x003f) SCT Status supported.
> SCT Error Recovery Control
> supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> You need error recovery to be supported. If it isn't ...
>
>>
>> Cheers,
>> Curt
>
>
> Cheers,
> Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 20:01 ` Curt
@ 2017-10-04 21:08 ` Anthony Youngman
2017-10-04 21:53 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Anthony Youngman @ 2017-10-04 21:08 UTC (permalink / raw)
To: Curt; +Cc: Joe Landman, linux-raid
On 04/10/17 21:01, Curt wrote:
> Hello,
>
> Thanks for clarifying. All current good drives report that they are
> part of a 8 drive array. I only grew the raid by 1 device, so it
> would from 7-8, which is what they all report.
That's good.
> The 3rd failed doesn't
> report anything on examine, I haven't touched it at all and was not
> included in my assemble.
So this wasn't part of your original "assemble --force"? This wasn't
part of the array you managed to get working and then failed again?
> The 2 I replaced, the original drives I
> yanked, think they are still part of a 7 drive array.
Okay. The remaining five original disks are the ones we NEED. I'm
seriously concerned that if that third drive was one of these five,
we're in big trouble - we've lost the superblock.
>
> I'll be doing a ddrescue on the drives tonight, but will wait till
> Phil or someone chimes in with my next steps after I do that.
If you've got enough to ddrescue all of those five original drives, then
that's absolutely great.
Remember - if we can't get five original drives (or copies thereof) the
array is toast.
>
> lol, chalk one more up for FML. "SCT Error Recovery Control command
> not supported". I'm guessing this is a real bad thing now? I didn't
> buy these drives or org set it up.
>
I'm not sure whether this is good news or bad. Actually, it *could* be
great news for the rescue! It's bad news for raid though, if you don't
deal with it up front - I guess that wasn't done ...
*************
Go and read the wiki - the "When Things Go Wrogn" section. That will
hopefully help a lot and it explains the Error Recovery stuff (the
timeout mismatch page). Fix that problem and your dodgy drives will
probably dd without trouble at all.
Hopefully with all copied drives, but if you have to mix dd'd and
original drives you're probably okay, you should now be able to assemble
a working array with five drives by doing an
mdadm --assemble blah blah blah --update=revert-reshape
That will put you back to a "5 drives out of 7" working array. The
problem with this is that it will be a degraded, linear array.
I'm not sure whether a --display will list the failed drives - if it
does you can now --remove them. So you'll now have a working, 7-drive
array, with two drives missing.
Now --add in the two new drives. MAKE SURE you've read the section on
timeout mismatch and dealt with it! The rebuild/recovery will ALMOST
CERTAINLY FAIL if you don't! Also note that I am not sure about how
those drives will display while rebuilding - they may well display as
being spares during a rebuild.
Lastly, MAKE SURE you set up a regular scrub. There's a distinct
possibility that this problem wouldn't have arisen (or would have been
found quicker) if a scrub had been in place. And if you can set up a
trigger that emails you the contents of /proc/mdstat every few days.
It's far too easy to miss a failed drive if you don't have something
shoving it in your face every few days.
Cheers,
Wol
> On Wed, Oct 4, 2017 at 3:46 PM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>> On 04/10/17 20:09, Curt wrote:
>>>
>>> Ok, thanks.
>>>
>>> I'm pretty sure I'll be able to DD from at least one of the failed
>>> drives, as I could still query them before I yanked them. Assuming I
>>> can DD one of the old drives to one of my new ones.
>>>
>>> I'd DDrescue old to new drive. Then do an assemble for force, with a
>>> mix of the dd drives and my old good ones? So if sda/b are new DD'd
>>> drives and sdc/d/e are hosed grow drives, I'd do an assemble force
>>> revert-reshape /dev/md127 sda sdb sdc sdd and sde? Then assemble can
>>> use my info from the DD drives to assemble the array back to 7 drives?
>>> Did I understand that right?
>>
>>
>> This sounds like you need to take a great big step backwards, and make sure
>> you understand EXACTLY what is going on. We have a mix of good drives,
>> copies of bad drives, and an array that doesn't know whether it is supposed
>> to have 7 or 9 drives. One wrong step and your array will be toast.
>>
>> You want ALL FOUR KNOWN GOOD DRIVES. You want JUST ONE ddrescue'd drive.
>>
>> But I think the first thing we need to do, is to wait for an expert like
>> Phil to chime in and sort out that reshape. Your four good drives all think
>> they are part of a 9-drive array. Your first two drives to fail think they
>> are part of a 7-drive array. Does the third drive think it's part of a
>> 7-drive or 9-drive array?
>>
>> Can you do a --examine on this drive? I suspect the grow blew up because it
>> couldn't access this drive. I this drive thinks it is part of a 7-drive
>> array, we have a bit of a problem on our hands.
>>
>> I'm hoping it thinks it's part of a 9-drive array - I think we may be able
>> to get out of this ...
>>>
>>>
>>> Oh and how can I tell if I have a timeout mismatch. They should be raid
>>> drives.
>>
>>
>> smartctl -x /dev/sdX
>>
>> This will give you both the sort of drive you have - yes if it's in a
>> datacentre chances are it is a raid drive - and then search the output for
>> Error Recovery Control. This is from my hard drive...
>>
>> SCT capabilities: (0x003f) SCT Status supported.
>> SCT Error Recovery Control
>> supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> You need error recovery to be supported. If it isn't ...
>>
>>>
>>> Cheers,
>>> Curt
>>
>>
>> Cheers,
>> Wol
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-04 21:08 ` Anthony Youngman
@ 2017-10-04 21:53 ` Phil Turmel
[not found] ` <CADg2FGbnMzLBqWthKY5Uo__ANC2kAqH_8B1G23nhW+7hWJ=KeA@mail.gmail.com>
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-04 21:53 UTC (permalink / raw)
To: Anthony Youngman, Curt; +Cc: Joe Landman, linux-raid
Hi Curt,
Let me endorse Wol's prescription, with a few comments:
On 10/04/2017 05:08 PM, Anthony Youngman wrote:
> On 04/10/17 21:01, Curt wrote:
{ Side note: what possessed you to do a grow operation? }
>> I'll be doing a ddrescue on the drives tonight, but will wait till
>> Phil or someone chimes in with my next steps after I do that.
I haven't seen complete mdadm -E reports for all of these devices, nor
mdadm -D for the array itself. Please do so now. If you have any of
that from before the crash, please post that too. Run mdadm -E on the
two earliest failed drives.
Post the uncut output inline here on the list, in plain text mode, with
line wrap disabled, please.
> If you've got enough to ddrescue all of those five original drives, then
> that's absolutely great.
>
> Remember - if we can't get five original drives (or copies thereof) the
> array is toast.
>>
>> lol, chalk one more up for FML. "SCT Error Recovery Control command
>> not supported". I'm guessing this is a real bad thing now? I didn't
>> buy these drives or org set it up.
>>
> I'm not sure whether this is good news or bad. Actually, it *could* be
> great news for the rescue! It's bad news for raid though, if you don't
> deal with it up front - I guess that wasn't done ...
It is mixed news. It is almost certainly the reason you've had drives
bumped out of your arrays. I suspect these drives all report *PASSED*
from smartctl. Which means that the drives really are good, just
suffering from ordinary uncorrected errors.
You'll certainly have to use the 180 second driver timeout work-around
to get through this crisis.
In the meantime, please run "smartctl -iA -l scterc" on each of your
drives, including the failed ones, and post the uncut output here.
{ Include the device name with each }
> Go and read the wiki - the "When Things Go Wrogn" section. That will
> hopefully help a lot and it explains the Error Recovery stuff (the
> timeout mismatch page). Fix that problem and your dodgy drives will
> probably dd without trouble at all.
Let me emphasize this. The timeout mismatch problem is so prevalent and
your experience so common that I thought to myself "I bet this one is
timeout mismatch" when I read your first mail.
> Hopefully with all copied drives, but if you have to mix dd'd and
> original drives you're probably okay, you should now be able to assemble
> a working array with five drives by doing an
As already noted, you definitely need to use ddrescue on the third
drive that failed. You may also need to ddrescue your four remaining
good drives if they also have "Pending Sector" counts.
> mdadm --assemble blah blah blah --update=revert-reshape
>
> That will put you back to a "5 drives out of 7" working array. The
> problem with this is that it will be a degraded, linear array.
This is the correct next step, after all required ddrescues.
> I'm not sure whether a --display will list the failed drives - if it
> does you can now --remove them. So you'll now have a working, 7-drive
> array, with two drives missing.
This is the time to grab any backups you need of critical content. Do
*not* write to the array at this point. Get all your data off.
Then:
> Now --add in the two new drives. MAKE SURE you've read the section on
> timeout mismatch and dealt with it! The rebuild/recovery will ALMOST
> CERTAINLY FAIL if you don't! Also note that I am not sure about how
> those drives will display while rebuilding - they may well display as
> being spares during a rebuild.
The timeout mismatch fixes won't help your case. You have no redundancy
left, so the kickout scenarios involved no longer apply. They applied
when your first two drives were kicked out. When timeouts are not
mismatched, MD raid *fixes* the occasional bad sector.
> Lastly, MAKE SURE you set up a regular scrub. There's a distinct
> possibility that this problem wouldn't have arisen (or would have been
> found quicker) if a scrub had been in place. And if you can set up a
> trigger that emails you the contents of /proc/mdstat every few days.
> It's far too easy to miss a failed drive if you don't have something
> shoving it in your face every few days.
If you have a timeout mismatch problem, one's array will die much sooner
with scrubs. Because MD raid will fail to fix UREs, and it will find
them right away.
But again, get us the detailed reports, and we'll help make sure your
commands are correct.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
[not found] ` <CADg2FGbnMzLBqWthKY5Uo__ANC2kAqH_8B1G23nhW+7hWJ=KeA@mail.gmail.com>
@ 2017-10-06 1:25 ` Curt
2017-10-06 11:16 ` Wols Lists
[not found] ` <CADg2FGYc-sPjwukuhonoUUCr3ze3PQWv8gtZPnUT=E4CvsQftg@mail.gmail.com>
1 sibling, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-06 1:25 UTC (permalink / raw)
To: linux-raid
I just realized I forgot to do a reply all on this yesterday. In case
anyone else in the group is interested. The ddrescue on the failed
drive is complete now.
Requested info pasted below, at least the info I got.
(Side note: Let's just go with stupidity)
Then depending on what your feedback is, theoretically I'll want to run
mdadm --assemble --force --update=revert-reshape /dev/md127 /dev/sda1
/dev/sdc1 /dev/sdd1 /dev/sdg1/ /dev/ddRescuePart
I appreciate all your help on this.
Cheers,
Curt
On Wed, Oct 4, 2017 at 5:53 PM, Phil Turmel <philip@turmel.org> wrote:
> Hi Curt,
>
> Let me endorse Wol's prescription, with a few comments:
>
> On 10/04/2017 05:08 PM, Anthony Youngman wrote:
>> On 04/10/17 21:01, Curt wrote:
>
> { Side note: what possessed you to do a grow operation? }
>
>>> I'll be doing a ddrescue on the drives tonight, but will wait till
>>> Phil or someone chimes in with my next steps after I do that.
>
> I haven't seen complete mdadm -E reports for all of these devices, nor
> mdadm -D for the array itself. Please do so now. If you have any of
> that from before the crash, please post that too. Run mdadm -E on the
> two earliest failed drives.
>
Don't hate on me too bad. I already know I made several very stupid
mistakes along the way.
Here's watch I got probably missing a few things that would be useful.
The array is currently stopped, so I can't get you the -D, but here's
what I got
Array Before Grow
/dev/md127:
Version : 0.90
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Persistence : Superblock is persistent
Update Time : Tue Oct 3 21:13:32 2017
State : clean, degraded, recovering
Active Devices : 5
Working Devices : 7
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : unknown
Rebuild Status : 84% complete
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Events : 0.11559596
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 49 1 active sync /dev/sdd1
2 8 33 2 active sync /dev/sdc1
3 8 1 3 active sync /dev/sda1
4 8 65 4 active sync /dev/sde1
8 8 16 5 spare rebuilding /dev/sdb
7 8 80 6 spare rebuilding /dev/sdf
Array After Grow:
mdadm --detail /dev/md127
/dev/md127:
Version : 0.91
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 127
Persistence : Superblock is persistent
Update Time : Tue Oct 3 23:10:32 2017
State : clean, FAILED, reshaping
Active Devices : 5
Working Devices : 7
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : unknown
Reshape Status : 0% complete
Delta Devices : 1, (7->8)
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Events : 0.11559671
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 49 1 active sync /dev/sdd1
2 8 33 2 active sync /dev/sdc1
3 8 1 3 active sync /dev/sda1
4 8 65 4 active sync /dev/sde1
5 8 16 5 spare rebuilding /dev/sdb
6 8 80 6 spare rebuilding /dev/sdf
- 0 0 7 removed
Here's the few I have from before. I really shouldn't have been doing
this at 4am.
****************
mdadm --examine /dev/sdf
/dev/sdf:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Update Time : Tue Oct 3 22:38:22 2017
State : clean
Active Devices : 4
Working Devices : 6
Failed Devices : 3
Spare Devices : 2
Checksum : cdfbf074 - correct
Events : 11559615
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 80 7 spare /dev/sdf
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
7 7 8 80 7 spare /dev/sdf
8 8 8 16 8 spare /dev/sdb
mdadm --examine /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Update Time : Tue Oct 3 22:38:22 2017
State : clean
Active Devices : 4
Working Devices : 6
Failed Devices : 3
Spare Devices : 2
Checksum : cdfbf023 - correct
Events : 11559615
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 1 3 active sync /dev/sda1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
7 7 8 80 7 spare /dev/sdf
8 8 8 16 8 spare /dev/sdb
Here's the 3 failed drives: NOTE: I only had one bay available, so
they all have the same drive letter
mdadm --examine /dev/sdz1
/dev/sdz1:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 126
Update Time : Mon Jul 11 16:54:15 2016
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 1
Spare Devices : 0
Checksum : ca7ec3b0 - correct
Events : 3397832
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 65 97 1 active sync
0 0 65 113 0 active sync
1 1 65 97 1 active sync
2 2 65 81 2 active sync
3 3 0 0 3 faulty removed
4 4 65 49 4 active sync
5 5 65 33 5 active sync
6 6 65 17 6 active sync
**********************THE ONE BELOW I'M DOING A DDRESCUE FROM******
mdadm --examine /dev/sdz1
/dev/sdz1:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Update Time : Sat Sep 2 01:00:37 2017
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 1
Spare Devices : 0
Checksum : cd217ebc - correct
Events : 11559404
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 65 5 active sync /dev/sde1
0 0 8 81 0 active sync
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 active sync
3 3 65 129 3 active sync /dev/sdy1
4 4 8 49 4 active sync /dev/sdd1
5 5 8 65 5 active sync /dev/sde1
6 6 0 0 6 faulty removed
***************
mdadm --examine /dev/sdz1
/dev/sdz1:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Update Time : Mon Nov 7 02:02:38 2016
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : cb1ec57d - correct
Events : 3652739
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 97 6 active sync /dev/sdg1
0 0 8 81 0 active sync
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 active sync
3 3 65 129 3 active sync /dev/sdy1
4 4 8 49 4 active sync /dev/sdd1
5 5 8 65 5 active sync /dev/sde1
6 6 8 97 6 active sync /dev/sdg1
CURRENT EXAMINE
*************************
mdadm -E /dev/sd[acdeg]1
/dev/sda1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71a9cb - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 1 3 active sync /dev/sda1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
/dev/sdc1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71a9e9 - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 33 2 active sync /dev/sdc1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
/dev/sdd1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71a9f7 - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
/dev/sde1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71aa0d - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 65 4 active sync /dev/sde1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
/dev/sdg1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71aa25 - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 97 0 active sync /dev/sdg1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
/dev/sdb:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71a9dc - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 16 6 active /dev/sdb
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty remove
> Post the uncut output inline here on the list, in plain text mode, with
> line wrap disabled, please.
>
>> If you've got enough to ddrescue all of those five original drives, then
>> that's absolutely great.
>>
>> Remember - if we can't get five original drives (or copies thereof) the
>> array is toast.
>>>
>>> lol, chalk one more up for FML. "SCT Error Recovery Control command
>>> not supported". I'm guessing this is a real bad thing now? I didn't
>>> buy these drives or org set it up.
>>>
>> I'm not sure whether this is good news or bad. Actually, it *could* be
>> great news for the rescue! It's bad news for raid though, if you don't
>> deal with it up front - I guess that wasn't done ...
>
> It is mixed news. It is almost certainly the reason you've had drives
> bumped out of your arrays. I suspect these drives all report *PASSED*
> from smartctl. Which means that the drives really are good, just
> suffering from ordinary uncorrected errors.
>
> You'll certainly have to use the 180 second driver timeout work-around
> to get through this crisis.
>
> In the meantime, please run "smartctl -iA -l scterc" on each of your
> drives, including the failed ones, and post the uncut output here.
> { Include the device name with each }
>
Sorry I don't have it for the failed ones, I forgot to run in before I
started ddrescue, here's the current drives
# smartctl -iA -l scterc /dev/sda
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST2000DM001-1ER164
Serial Number: W4Z14ZNW
LU WWN Device Id: 5 000c50 07d29ef14
Firmware Version: CC25
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:28:30 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 106 099 006 Pre-fail
Always - 11140560
3 Spin_Up_Time 0x0003 096 096 000 Pre-fail
Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age
Always - 14
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail
Always - 0
7 Seek_Error_Rate 0x000f 089 060 030 Pre-fail
Always - 827856598
9 Power_On_Hours 0x0032 079 079 000 Old_age
Always - 18858
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail
Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age
Always - 14
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age
Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age
Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age
Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age
Always - 0 0 0
189 High_Fly_Writes 0x003a 013 013 000 Old_age
Always - 87
190 Airflow_Temperature_Cel 0x0022 071 063 045 Old_age
Always - 29 (Min/Max 29/30)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age
Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 8
193 Load_Cycle_Count 0x0032 100 100 000 Old_age
Always - 268
194 Temperature_Celsius 0x0022 029 040 000 Old_age
Always - 29 (0 18 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age
Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age
Offline - 18841h+49m+16.895s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age
Offline - 84336821090
242 Total_LBAs_Read 0x0000 100 253 000 Old_age
Offline - 4832824497202
SCT Error Recovery Control command not supported
# smartctl -iA -l scterc /dev/sdb
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST2000DM001-1ER164
Serial Number: Z4Z3Y7XM
LU WWN Device Id: 5 000c50 087461756
Firmware Version: CC26
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:28:44 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail
Always - 157161144
3 Spin_Up_Time 0x0003 096 096 000 Pre-fail
Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age
Always - 15
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail
Always - 0
7 Seek_Error_Rate 0x000f 086 060 030 Pre-fail
Always - 409701090
9 Power_On_Hours 0x0032 086 086 000 Old_age
Always - 12274
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail
Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age
Always - 15
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age
Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age
Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age
Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age
Always - 0 0 0
189 High_Fly_Writes 0x003a 092 092 000 Old_age
Always - 8
190 Airflow_Temperature_Cel 0x0022 070 065 045 Old_age
Always - 30 (Min/Max 28/33)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age
Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 10
193 Load_Cycle_Count 0x0032 100 100 000 Old_age
Always - 142
194 Temperature_Celsius 0x0022 030 040 000 Old_age
Always - 30 (0 21 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age
Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age
Offline - 12268h+22m+21.157s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age
Offline - 83831274067
242 Total_LBAs_Read 0x0000 100 253 000 Old_age
Offline - 124530518173
SCT Error Recovery Control command not supported
# smartctl -iA -l scterc /dev/sdc
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD2002FAEX-007BA0
Serial Number: WD-WMAY04949787
LU WWN Device Id: 5 0014ee 25c3e0682
Firmware Version: 05.01D05
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:28:46 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 2
3 Spin_Up_Time 0x0027 253 253 021 Pre-fail
Always - 8041
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 36
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 071 071 000 Old_age
Always - 21337
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 27
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 8
194 Temperature_Celsius 0x0022 117 107 000 Old_age
Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SCT Error Recovery Control command not supported
# smartctl -iA -l scterc /dev/sdd
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD2002FAEX-007BA0
Serial Number: WD-WMAY04912439
LU WWN Device Id: 5 0014ee 25c3f0960
Firmware Version: 05.01D05
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:29:33 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 8
3 Spin_Up_Time 0x0027 253 253 021 Pre-fail
Always - 7950
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 36
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 071 071 000 Old_age
Always - 21325
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 27
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 8
194 Temperature_Celsius 0x0022 116 106 000 Old_age
Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SCT Error Recovery Control command not supported
# smartctl -iA -l scterc /dev/sde
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD2002FAEX-007BA0
Serial Number: WD-WMAY05040774
LU WWN Device Id: 5 0014ee 2b1938a22
Firmware Version: 05.01D05
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:29:36 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 253 253 021 Pre-fail
Always - 8083
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 36
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 071 071 000 Old_age
Always - 21328
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 27
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 8
194 Temperature_Celsius 0x0022 116 108 000 Old_age
Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 2
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 2
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 2
SCT Error Recovery Control command not supported
#smartctl -iA -l scterc /dev/sdg
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-3.10.0-229.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST2000DM001-1ER164
Serial Number: ZA5029A8
LU WWN Device Id: 5 000c50 0874eb397
Firmware Version: CC26
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Oct 4 20:29:42 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 119 099 006 Pre-fail
Always - 232755000
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail
Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age
Always - 10
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail
Always - 0
7 Seek_Error_Rate 0x000f 087 060 030 Pre-fail
Always - 606406779
9 Power_On_Hours 0x0032 086 086 000 Old_age
Always - 13052
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail
Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age
Always - 10
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age
Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age
Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age
Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age
Always - 0 0 0
189 High_Fly_Writes 0x003a 098 098 000 Old_age
Always - 2
190 Airflow_Temperature_Cel 0x0022 069 060 045 Old_age
Always - 31 (Min/Max 29/34)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age
Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 4
193 Load_Cycle_Count 0x0032 100 100 000 Old_age
Always - 310
194 Temperature_Celsius 0x0022 031 040 000 Old_age
Always - 31 (0 25 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age
Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age
Offline - 13039h+51m+52.726s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age
Offline - 33464557056
242 Total_LBAs_Read 0x0000 100 253 000 Old_age
Offline - 2634696436762
SCT Error Recovery Control command not supported
>> Go and read the wiki - the "When Things Go Wrogn" section. That will
>> hopefully help a lot and it explains the Error Recovery stuff (the
>> timeout mismatch page). Fix that problem and your dodgy drives will
>> probably dd without trouble at all.
>
> Let me emphasize this. The timeout mismatch problem is so prevalent and
> your experience so common that I thought to myself "I bet this one is
> timeout mismatch" when I read your first mail.
>
>> Hopefully with all copied drives, but if you have to mix dd'd and
>> original drives you're probably okay, you should now be able to assemble
>> a working array with five drives by doing an
>
> As already noted, you definitely need to use ddrescue on the third
> drive that failed. You may also need to ddrescue your four remaining
> good drives if they also have "Pending Sector" counts.
>
>> mdadm --assemble blah blah blah --update=revert-reshape
>>
>> That will put you back to a "5 drives out of 7" working array. The
>> problem with this is that it will be a degraded, linear array.
>
> This is the correct next step, after all required ddrescues.
>
>> I'm not sure whether a --display will list the failed drives - if it
>> does you can now --remove them. So you'll now have a working, 7-drive
>> array, with two drives missing.
>
> This is the time to grab any backups you need of critical content. Do
> *not* write to the array at this point. Get all your data off.
>
> Then:
>
>> Now --add in the two new drives. MAKE SURE you've read the section on
>> timeout mismatch and dealt with it! The rebuild/recovery will ALMOST
>> CERTAINLY FAIL if you don't! Also note that I am not sure about how
>> those drives will display while rebuilding - they may well display as
>> being spares during a rebuild.
>
> The timeout mismatch fixes won't help your case. You have no redundancy
> left, so the kickout scenarios involved no longer apply. They applied
> when your first two drives were kicked out. When timeouts are not
> mismatched, MD raid *fixes* the occasional bad sector.
>
>> Lastly, MAKE SURE you set up a regular scrub. There's a distinct
>> possibility that this problem wouldn't have arisen (or would have been
>> found quicker) if a scrub had been in place. And if you can set up a
>> trigger that emails you the contents of /proc/mdstat every few days.
>> It's far too easy to miss a failed drive if you don't have something
>> shoving it in your face every few days.
>
> If you have a timeout mismatch problem, one's array will die much sooner
> with scrubs. Because MD raid will fail to fix UREs, and it will find
> them right away.
>
> But again, get us the detailed reports, and we'll help make sure your
> commands are correct.
>
> Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-06 1:25 ` Curt
@ 2017-10-06 11:16 ` Wols Lists
0 siblings, 0 replies; 59+ messages in thread
From: Wols Lists @ 2017-10-06 11:16 UTC (permalink / raw)
To: Curt, linux-raid
On 06/10/17 02:25, Curt wrote:
> Requested info pasted below, at least the info I got.
Seagate Barracudas, WD Blacks. NOT good raid drives, sorry. Once you get
it all working, then you need that timeout script configured to run
every boot, at which point your array should be safe.
Once it's all fine, if you're in the market for more drives look at
Seagate NAS or Ironwolf or Constellation, WD Reds, or Hitachi Ultrastar.
They're all advertised (Constellation excepted, but that's reported as
supporting ERC) as being suitable for raid. Expect to pay a 30-50%
premium over what a Barracuda will cost.
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
[not found] ` <CADg2FGYc-sPjwukuhonoUUCr3ze3PQWv8gtZPnUT=E4CvsQftg@mail.gmail.com>
@ 2017-10-06 13:13 ` Phil Turmel
2017-10-06 14:07 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-06 13:13 UTC (permalink / raw)
To: Curt, linux-raid
Hi Curt,
Sorry about the delay /-:
{ Convention on kernel.org lists is reply to all, bottom replies (or
interleaved, and trim unnecessary quoted material. Please help those
of us who follow many threads. }
On 10/05/2017 07:40 PM, Curt wrote:
> Hey Phil,
>
> I got the drive mentioned ddrescued, I wanted to make sure you didn't see
> anything odd from my pastes before I try the revert.
Only one:
> # smartctl -iA -l scterc /dev/sde
> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> Always - 2
> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> Offline - 2
You'll have to ddrescue /dev/sde1 as well. Do that before proceeding.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-06 13:13 ` Phil Turmel
@ 2017-10-06 14:07 ` Curt
2017-10-06 14:27 ` Joe Landman
2017-10-06 14:27 ` Phil Turmel
0 siblings, 2 replies; 59+ messages in thread
From: Curt @ 2017-10-06 14:07 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
On Fri, Oct 6, 2017 at 9:13 AM, Phil Turmel <philip@turmel.org> wrote:
> Hi Curt,
>
> Sorry about the delay /-:
>
> { Convention on kernel.org lists is reply to all, bottom replies (or
> interleaved, and trim unnecessary quoted material. Please help those
> of us who follow many threads. }
>
> On 10/05/2017 07:40 PM, Curt wrote:
>> Hey Phil,
>>
>> I got the drive mentioned ddrescued, I wanted to make sure you didn't see
>> anything odd from my pastes before I try the revert.
>
> Only one:
>
>> # smartctl -iA -l scterc /dev/sde
>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>> Always - 2
>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>> Offline - 2
>
> You'll have to ddrescue /dev/sde1 as well. Do that before proceeding.
>
> Phil
Hi Phil,
No worries, I appreciate the help.
(Noted and I will try to follow that standard in all future communication)
Ok, I will ddrescue that drive today, which should take about 10
hours. I'm just doing a ddrescue -f -n /dev/bad /dev/new logFile. Is
there any other suggested flags?
Once that is done, I'll run the before mentioned, mdadm assemble force
revert sda/c/d/g + rescued e and z. Then start pulling whatever data
I can off it.
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-06 14:07 ` Curt
@ 2017-10-06 14:27 ` Joe Landman
2017-10-06 14:27 ` Phil Turmel
1 sibling, 0 replies; 59+ messages in thread
From: Joe Landman @ 2017-10-06 14:27 UTC (permalink / raw)
To: Curt, Phil Turmel; +Cc: linux-raid
On 10/06/2017 10:07 AM, Curt wrote:
> On Fri, Oct 6, 2017 at 9:13 AM, Phil Turmel <philip@turmel.org> wrote:
[...]
> Once that is done, I'll run the before mentioned, mdadm assemble force
> revert sda/c/d/g + rescued e and z. Then start pulling whatever data
> I can off it.
I'd suggest that some data may be more highly valuable than other data
on there. So if you can pull that off and stash somewhere, that would
be recommended before copying everything. In the past I've made
tarballs and moved copies around to various locations to preserve
important data. Once you have that, do the bigger copy of data.
--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-06 14:07 ` Curt
2017-10-06 14:27 ` Joe Landman
@ 2017-10-06 14:27 ` Phil Turmel
2017-10-07 3:09 ` Curt
1 sibling, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-06 14:27 UTC (permalink / raw)
To: Curt; +Cc: linux-raid
On 10/06/2017 10:07 AM, Curt wrote:
> Ok, I will ddrescue that drive today, which should take about 10
> hours. I'm just doing a ddrescue -f -n /dev/bad /dev/new logFile. Is
> there any other suggested flags?
No, that should do. You can ddrescue just the partition if you like.
> Once that is done, I'll run the before mentioned, mdadm assemble force
> revert sda/c/d/g + rescued e and z.
Yes, just make sure you are at the MD member level, whether whole device
or partition for each component. Your new devices were added as whole
devices, an inconsistency that could burn you later.
> Then start pulling whatever data
> I can off it.
Yes.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-06 14:27 ` Phil Turmel
@ 2017-10-07 3:09 ` Curt
2017-10-07 3:15 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-07 3:09 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
On Fri, Oct 6, 2017 at 10:27 AM, Phil Turmel <philip@turmel.org> wrote:
> On 10/06/2017 10:07 AM, Curt wrote:
>> Once that is done, I'll run the before mentioned, mdadm assemble force
>> revert sda/c/d/g + rescued e and z.
>
> Yes, just make sure you are at the MD member level, whether whole device
> or partition for each component. Your new devices were added as whole
> devices, an inconsistency that could burn you later.
Ok, I tried to do an assemble including the failed drive I ddrescued
and get superblock on /dev/sdf1 doesn't match others assembly aborted.
Does that mean I won't be able to revert or recovery any data? I hope
not.
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-07 3:09 ` Curt
@ 2017-10-07 3:15 ` Curt
2017-10-07 20:45 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-07 3:15 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
> Ok, I tried to do an assemble including the failed drive I ddrescued
> and get superblock on /dev/sdf1 doesn't match others assembly aborted.
> Does that mean I won't be able to revert or recovery any data? I hope
> not.
>
> Cheers,
> Curt
Just to clarify, sdf1 is the ddrescue of the failed drive.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-07 3:15 ` Curt
@ 2017-10-07 20:45 ` Curt
2017-10-07 21:29 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-07 20:45 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
Here's the output of -E for sdf1 and one other, in case it's needed.
As I understand it the UUID is what has to do with the superblock, but
UUID's match between them.
mdadm -E /dev/sdf1
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 127
Update Time : Sat Sep 2 01:00:37 2017
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 1
Spare Devices : 0
Checksum : cd217ebc - correct
Events : 11559404
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 65 5 active sync /dev/sde1
0 0 8 81 0 active sync /dev/sdf1
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 active sync
3 3 65 129 3 active sync /dev/sdy1
4 4 8 49 4 active sync /dev/sdd1
5 5 8 65 5 active sync /dev/sde1
6 6 0 0 6 faulty removed
*****************************
mdadm -E /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 0.91.00
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
Update Time : Wed Oct 4 12:49:57 2017
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : ce71a9cb - correct
Events : 11559681
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 1 3 active sync /dev/sda1
0 0 8 97 0 active sync /dev/sdg1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 1 3 active sync /dev/sda1
4 4 8 65 4 active sync /dev/sde1
5 5 0 0 5 faulty removed
6 6 8 16 6 active /dev/sdb
7 7 0 0 7 faulty removed
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-07 20:45 ` Curt
@ 2017-10-07 21:29 ` Phil Turmel
2017-10-08 22:40 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-07 21:29 UTC (permalink / raw)
To: Curt; +Cc: linux-raid
On 10/07/2017 04:45 PM, Curt wrote:
> Here's the output of -E for sdf1 and one other, in case it's needed.
> As I understand it the UUID is what has to do with the superblock, but
> UUID's match between them.
>
> mdadm -E /dev/sdf1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : 714a612d:9bd35197:36c91ae3:c168144d
> Creation Time : Fri Jun 15 15:52:05 2012
> Raid Level : raid6
> Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
> Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
> Raid Devices : 7
> Total Devices : 7
> Preferred Minor : 127
>
> Update Time : Sat Sep 2 01:00:37 2017
> State : active
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 1
> Spare Devices : 0
> Checksum : cd217ebc - correct
> Events : 11559404
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 5 8 65 5 active sync /dev/sde1
>
> 0 0 8 81 0 active sync /dev/sdf1
> 1 1 8 33 1 active sync /dev/sdc1
> 2 2 8 17 2 active sync
> 3 3 65 129 3 active sync /dev/sdy1
> 4 4 8 49 4 active sync /dev/sdd1
> 5 5 8 65 5 active sync /dev/sde1
> 6 6 0 0 6 faulty removed
>
> *****************************
> mdadm -E /dev/sda1
> /dev/sda1:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 714a612d:9bd35197:36c91ae3:c168144d
> Creation Time : Fri Jun 15 15:52:05 2012
> Raid Level : raid6
> Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
> Array Size : 11721023232 (11178.04 GiB 12002.33 GB)
> Raid Devices : 8
> Total Devices : 6
> Preferred Minor : 127
>
> Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
> Delta Devices : 1 (7->8)
>
> Update Time : Wed Oct 4 12:49:57 2017
> State : clean
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 2
> Spare Devices : 0
> Checksum : ce71a9cb - correct
> Events : 11559681
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 1 3 active sync /dev/sda1
>
> 0 0 8 97 0 active sync /dev/sdg1
> 1 1 8 49 1 active sync /dev/sdd1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 1 3 active sync /dev/sda1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 0 0 5 faulty removed
> 6 6 8 16 6 active /dev/sdb
> 7 7 0 0 7 faulty removed
>
Hmm. Please run the assemble again, with the --verbose option, and
paste all the output here so we can see what's happening.
Also grab the corresponding tail of dmesg (however many lines cover the
assembly attempt).
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-07 21:29 ` Phil Turmel
@ 2017-10-08 22:40 ` Curt
2017-10-09 1:23 ` NeilBrown
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-08 22:40 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
On Sat, Oct 7, 2017 at 5:29 PM, Phil Turmel <philip@turmel.org> wrote:
> On 10/07/2017 04:45 PM, Curt wrote:
>
> Hmm. Please run the assemble again, with the --verbose option, and
> paste all the output here so we can see what's happening.
>
> Also grab the corresponding tail of dmesg (however many lines cover the
> assembly attempt).
>
> Phil
You wouldn't think reply all would be such a hard thing to remember to do.
verbose didn't really add anything, here's my full command, maybe I
miss typed something and didn't notice.
# mdadm --assemble --force --update=revert-reshape --verbose
/dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sdz1
mdadm: looking for devices for /dev/md127
mdadm: superblock on /dev/sdf1 doesn't match others - assembly aborted
I've got nothing in dmesg or messages
If I can't get the old drive to work. Is there any chance to still
getting some data from the other drives? Either running the above
without sdf or worse comes to worse the assume clean someone mentioned
before.
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-08 22:40 ` Curt
@ 2017-10-09 1:23 ` NeilBrown
2017-10-09 1:40 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: NeilBrown @ 2017-10-09 1:23 UTC (permalink / raw)
To: Curt, Phil Turmel; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]
On Sun, Oct 08 2017, Curt wrote:
> On Sat, Oct 7, 2017 at 5:29 PM, Phil Turmel <philip@turmel.org> wrote:
>> On 10/07/2017 04:45 PM, Curt wrote:
>>
>> Hmm. Please run the assemble again, with the --verbose option, and
>> paste all the output here so we can see what's happening.
>>
>> Also grab the corresponding tail of dmesg (however many lines cover the
>> assembly attempt).
>>
>> Phil
>
> You wouldn't think reply all would be such a hard thing to remember to do.
>
> verbose didn't really add anything, here's my full command, maybe I
> miss typed something and didn't notice.
>
> # mdadm --assemble --force --update=revert-reshape --verbose
> /dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdg1 /dev/sdf1 /dev/sdz1
> mdadm: looking for devices for /dev/md127
> mdadm: superblock on /dev/sdf1 doesn't match others - assembly aborted
You get this because sdf1 says:
Raid Devices : 7
Total Devices : 7
while sda1 (for example) says:
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
Delta Devices : 1 (7->8)
mdadm cannot reconcile this difference.
It appears that sdf1 was never involved in any reshape.
So you need to revert the reshape before trying to include sdf1 into the
array. Clearly you need at least 6 devices that were involved in
the reshape to do this.
I haven't been following closely ... do you have 6 such devices?
NeilBrown
>
> I've got nothing in dmesg or messages
>
> If I can't get the old drive to work. Is there any chance to still
> getting some data from the other drives? Either running the above
> without sdf or worse comes to worse the assume clean someone mentioned
> before.
>
> Cheers,
> Curt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 1:23 ` NeilBrown
@ 2017-10-09 1:40 ` Curt
2017-10-09 4:28 ` NeilBrown
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-09 1:40 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, linux-raid
> You get this because sdf1 says:
>
> Raid Devices : 7
> Total Devices : 7
>
> while sda1 (for example) says:
>
> Raid Devices : 8
> Total Devices : 6
> Preferred Minor : 127
>
> Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
> Delta Devices : 1 (7->8)
>
> mdadm cannot reconcile this difference.
>
> It appears that sdf1 was never involved in any reshape.
> So you need to revert the reshape before trying to include sdf1 into the
> array. Clearly you need at least 6 devices that were involved in
> the reshape to do this.
> I haven't been following closely ... do you have 6 such devices?
>
> NeilBrown
>
Correct. Which I thought was sorta the point, but could have
completely misunderstood it, sdf was restored from a "faulty" drive
that was out before the reshape. Whether I have 6 devices depends on
how picky things are. I've got 5 that should be in sync, the 6th not,
but it was involved in the reshape.
Short version, is I shot myself in the foot on this one. Reshape never
got anywhere, but need to try to revert and save what data I can.
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 1:40 ` Curt
@ 2017-10-09 4:28 ` NeilBrown
2017-10-09 4:59 ` Curt
2017-10-09 12:41 ` Curt
0 siblings, 2 replies; 59+ messages in thread
From: NeilBrown @ 2017-10-09 4:28 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, linux-raid
[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]
On Sun, Oct 08 2017, Curt wrote:
>> You get this because sdf1 says:
>>
>> Raid Devices : 7
>> Total Devices : 7
>>
>> while sda1 (for example) says:
>>
>> Raid Devices : 8
>> Total Devices : 6
>> Preferred Minor : 127
>>
>> Reshape pos'n : 3799296 (3.62 GiB 3.89 GB)
>> Delta Devices : 1 (7->8)
>>
>> mdadm cannot reconcile this difference.
>>
>> It appears that sdf1 was never involved in any reshape.
>> So you need to revert the reshape before trying to include sdf1 into the
>> array. Clearly you need at least 6 devices that were involved in
>> the reshape to do this.
>> I haven't been following closely ... do you have 6 such devices?
>>
>> NeilBrown
>>
>
> Correct. Which I thought was sorta the point, but could have
> completely misunderstood it, sdf was restored from a "faulty" drive
> that was out before the reshape. Whether I have 6 devices depends on
> how picky things are. I've got 5 that should be in sync, the 6th not,
> but it was involved in the reshape.
>
> Short version, is I shot myself in the foot on this one. Reshape never
> got anywhere, but need to try to revert and save what data I can.
Hmmm... (goes back and looks at more of thread..)
Ahhh .. you had an array which was rebuilding two spares, and you
told it to start reshaping... Interesting.
Theoretically that should work. Was it deliberate? (I cannot seem to
find the start of the thread).
Looking at the list of "current --examine output", it appears that
/dev/sdg1
/dev/sdd1
/dev/sdc1
/dev/sda1
/dev/sde1
/dev/sdb
are all valid devices with the same event counts. They are the six that
you need.
To confirm that names haven't changed, you can:
mdadm --examine /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
/dev/sdb | grep Events
and confirm all the numbers are the same.
Then do the same and grep for "this" and confirm all the Raid Disk
numbers are different.
Interesting that /dev/sdb is a whole device and the rest are partitions.
I assume you know about that and why it is.
What happens if you run the --assemble --update=revert-reshape command on
these 6 devices (without --force)??
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 4:28 ` NeilBrown
@ 2017-10-09 4:59 ` Curt
2017-10-09 5:47 ` NeilBrown
2017-10-09 12:41 ` Curt
1 sibling, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-09 4:59 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, linux-raid
On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
> On Sun, Oct 08 2017, Curt wrote:
> Theoretically that should work. Was it deliberate? (I cannot seem to
> find the start of the thread).
Yes and no, I ran the command, but it wasn't what I really wanted
and/or needed, so to speak.
> are all valid devices with the same event counts. They are the six that
> you need.
> To confirm that names haven't changed, you can:
> mdadm --examine /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
> /dev/sdb | grep Events
Yes, the numbers all match.
>
> and confirm all the numbers are the same.
> Then do the same and grep for "this" and confirm all the Raid Disk
> numbers are different.
confirmed
>
> Interesting that /dev/sdb is a whole device and the rest are partitions.
> I assume you know about that and why it is.
Just a mistake when I added it.
>
> What happens if you run the --assemble --update=revert-reshape command on
> these 6 devices (without --force)??
>
> NeilBrown
I'll give it a try, but first a question and FYI. Earlier Phil had me
ddrescue sde1 because it had pending sectors I believe. Should I use
sde1 or sdz1(ddrescued version)?
Also when this whole thing started. md127_raid process was using 100%
cpu, but nothing seemed to be happening, No numbers changed, nothing
that I could tell anyway. Any suggestions on if the same thing
happens when I run the revert-reshape? Could one of my libraries be
behind a version somewhere?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 4:59 ` Curt
@ 2017-10-09 5:47 ` NeilBrown
0 siblings, 0 replies; 59+ messages in thread
From: NeilBrown @ 2017-10-09 5:47 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]
On Mon, Oct 09 2017, Curt wrote:
> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>> On Sun, Oct 08 2017, Curt wrote:
>> Theoretically that should work. Was it deliberate? (I cannot seem to
>> find the start of the thread).
> Yes and no, I ran the command, but it wasn't what I really wanted
> and/or needed, so to speak.
>
>
>> are all valid devices with the same event counts. They are the six that
>> you need.
>> To confirm that names haven't changed, you can:
>> mdadm --examine /dev/sdg1 /dev/sdd1 /dev/sdc1 /dev/sda1 /dev/sde1 \
>> /dev/sdb | grep Events
>
> Yes, the numbers all match.
>>
>> and confirm all the numbers are the same.
>> Then do the same and grep for "this" and confirm all the Raid Disk
>> numbers are different.
> confirmed
>>
>> Interesting that /dev/sdb is a whole device and the rest are partitions.
>> I assume you know about that and why it is.
> Just a mistake when I added it.
>>
>> What happens if you run the --assemble --update=revert-reshape command on
>> these 6 devices (without --force)??
>>
>> NeilBrown
> I'll give it a try, but first a question and FYI. Earlier Phil had me
> ddrescue sde1 because it had pending sectors I believe. Should I use
> sde1 or sdz1(ddrescued version)?
If sdz1 contains the same data as sde1, then use sdz1.
>
> Also when this whole thing started. md127_raid process was using 100%
> cpu, but nothing seemed to be happening, No numbers changed, nothing
> that I could tell anyway. Any suggestions on if the same thing
> happens when I run the revert-reshape? Could one of my libraries be
> behind a version somewhere?
If md127_raid is using 100% then that is a kernel bug.
If that happens, then find all processes that are spinning or are in 'D'
state, and
cat /proc/$PID/stack
for each pid, and report all the output.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 4:28 ` NeilBrown
2017-10-09 4:59 ` Curt
@ 2017-10-09 12:41 ` Curt
2017-10-10 12:08 ` Curt
1 sibling, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-09 12:41 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, linux-raid
On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
> What happens if you run the --assemble --update=revert-reshape command on
> these 6 devices (without --force)??
Here's the output
mdadm --assemble --verbose --update=revert-reshape /dev/md127
/dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
mdadm: looking for devices for /dev/md127
mdadm: Reshape position is not suitably aligned.
mdadm: Try normal assembly and stop again
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-09 12:41 ` Curt
@ 2017-10-10 12:08 ` Curt
2017-10-10 13:06 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 12:08 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, linux-raid
On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>> What happens if you run the --assemble --update=revert-reshape command on
>> these 6 devices (without --force)??
>
> Here's the output
> mdadm --assemble --verbose --update=revert-reshape /dev/md127
> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
> mdadm: looking for devices for /dev/md127
> mdadm: Reshape position is not suitably aligned.
> mdadm: Try normal assembly and stop again
>
What are the steps I need to take to resume the grow, but just enough
to get it to the point I can revert it and see what data I have on the
drives? Or can I force the revert?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 12:08 ` Curt
@ 2017-10-10 13:06 ` Phil Turmel
2017-10-10 13:37 ` Anthony Youngman
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 13:06 UTC (permalink / raw)
To: Curt, NeilBrown; +Cc: linux-raid
On 10/10/2017 08:08 AM, Curt wrote:
> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>>> What happens if you run the --assemble --update=revert-reshape command on
>>> these 6 devices (without --force)??
>>
>> Here's the output
>> mdadm --assemble --verbose --update=revert-reshape /dev/md127
>> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
>> mdadm: looking for devices for /dev/md127
>> mdadm: Reshape position is not suitably aligned.
>> mdadm: Try normal assembly and stop again
>>
>
> What are the steps I need to take to resume the grow, but just enough
> to get it to the point I can revert it and see what data I have on the
> drives? Or can I force the revert?
I'm stumped. Since your reshape position, while "0%", was 4GB into your
array, there's no way to use --create --assume-clean to fix this.
Unless Neil knows some magic, your data is toast. Sorry.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 13:06 ` Phil Turmel
@ 2017-10-10 13:37 ` Anthony Youngman
2017-10-10 14:00 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Anthony Youngman @ 2017-10-10 13:37 UTC (permalink / raw)
To: Phil Turmel, Curt, NeilBrown; +Cc: linux-raid
On 10/10/17 14:06, Phil Turmel wrote:
> On 10/10/2017 08:08 AM, Curt wrote:
>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>>> On Mon, Oct 9, 2017 at 12:28 AM, NeilBrown <neilb@suse.com> wrote:
>>>> What happens if you run the --assemble --update=revert-reshape command on
>>>> these 6 devices (without --force)??
>>>
>>> Here's the output
>>> mdadm --assemble --verbose --update=revert-reshape /dev/md127
>>> /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sdb /dev/sdg1 /dev/sdz1
>>> mdadm: looking for devices for /dev/md127
>>> mdadm: Reshape position is not suitably aligned.
>>> mdadm: Try normal assembly and stop again
>>>
>>
>> What are the steps I need to take to resume the grow, but just enough
>> to get it to the point I can revert it and see what data I have on the
>> drives? Or can I force the revert?
>
> I'm stumped. Since your reshape position, while "0%", was 4GB into your
> array, there's no way to use --create --assume-clean to fix this.
>
> Unless Neil knows some magic, your data is toast. Sorry.
>
I guess the only chance is to try and get all the drives that were
reshaping (or copies of them) in a fit state to try and re-assemble the
array. Would the revert work then? Or would the reshape crash on hitting
the first bad sector?
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 13:37 ` Anthony Youngman
@ 2017-10-10 14:00 ` Phil Turmel
2017-10-10 14:11 ` Curt
2017-10-10 14:12 ` Anthony Youngman
0 siblings, 2 replies; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 14:00 UTC (permalink / raw)
To: Anthony Youngman, Curt, NeilBrown; +Cc: linux-raid
On 10/10/2017 09:37 AM, Anthony Youngman wrote:
> On 10/10/17 14:06, Phil Turmel wrote:
>> On 10/10/2017 08:08 AM, Curt wrote:
>>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>>>> mdadm: Reshape position is not suitably aligned.
I've never seen this before, and don't have time at the moment
to read the code to understand what it means.
>> Unless Neil knows some magic, your data is toast. Sorry.
> I guess the only chance is to try and get all the drives that were
> reshaping (or copies of them) in a fit state to try and re-assemble the
> array. Would the revert work then?
Probably. But yeah, "fit state". That's the magic. /-:
> Or would the reshape crash on hitting
> the first bad sector?
The two devices with pending sectors have been ddrescue'd, so no.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:00 ` Phil Turmel
@ 2017-10-10 14:11 ` Curt
2017-10-10 14:14 ` Reindl Harald
2017-10-10 14:15 ` Phil Turmel
2017-10-10 14:12 ` Anthony Youngman
1 sibling, 2 replies; 59+ messages in thread
From: Curt @ 2017-10-10 14:11 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
>
> I've never seen this before, and don't have time at the moment
> to read the code to understand what it means.
I did a search and here's the comment before the code I found spits
out that error message. I don't know if it's useful to you
/* reshape_position is a little messy.
* Its value must be a multiple of the larger
* chunk size, and of the "after" data disks.
* So when reverting we need to change it to
* be a multiple of the new "after" data disks,
* which is the old "before".
* If it isn't already a multiple of 'before',
* the only thing we could do would be
* copy some block around on the disks, which
* is easy to get wrong.
* So we reject a revert-reshape unless the
* alignment is good.
*/
if (reshape_sectors % reshape_chunk)
That doesn't read good for me.
>
>
> Probably. But yeah, "fit state". That's the magic. /-:
>
What are the chances if I just do an assemble and let it try to grow
again that it will complete without issue? Then assuming it completes,
that I will have most my data still?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:00 ` Phil Turmel
2017-10-10 14:11 ` Curt
@ 2017-10-10 14:12 ` Anthony Youngman
1 sibling, 0 replies; 59+ messages in thread
From: Anthony Youngman @ 2017-10-10 14:12 UTC (permalink / raw)
To: Phil Turmel, Curt, NeilBrown; +Cc: linux-raid
On 10/10/17 15:00, Phil Turmel wrote:
> On 10/10/2017 09:37 AM, Anthony Youngman wrote:
>> On 10/10/17 14:06, Phil Turmel wrote:
>>> On 10/10/2017 08:08 AM, Curt wrote:
>>>> On Mon, Oct 9, 2017 at 8:41 AM, Curt <lightspd@gmail.com> wrote:
>
>>>>> mdadm: Reshape position is not suitably aligned.
>
> I've never seen this before, and don't have time at the moment
> to read the code to understand what it means.
My reaction (having read the thread :-) is that one of the drives being
used to try to re-assemble the array was dd'd from a drive that failed
before the reshape. So that was "one of 7, no reshape". All of the
others were "one of 9, reshape in progress, same position". So if we can
get the drives that were reshaping, we should be able to revert it.
Question is, can we?
>
>>> Unless Neil knows some magic, your data is toast. Sorry.
>
>> I guess the only chance is to try and get all the drives that were
>> reshaping (or copies of them) in a fit state to try and re-assemble the
>> array. Would the revert work then?
>
> Probably. But yeah, "fit state". That's the magic. /-:
So if Curt can do it, I think we stand a chance. A chance.
>
>> Or would the reshape crash on hitting
>> the first bad sector?
>
> The two devices with pending sectors have been ddrescue'd, so no.
>
We'll just have a few bits of corruption, but then we knew that. Three
drives gone from a raid-6, and any successful recovery is going to have
*some* damage.
Cheers,
Wol
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:11 ` Curt
@ 2017-10-10 14:14 ` Reindl Harald
2017-10-10 14:15 ` Phil Turmel
1 sibling, 0 replies; 59+ messages in thread
From: Reindl Harald @ 2017-10-10 14:14 UTC (permalink / raw)
To: Curt, Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
Am 10.10.2017 um 16:11 schrieb Curt:
> What are the chances if I just do an assemble and let it try to grow
> again that it will complete without issue? Then assuming it completes,
> that I will have most my data still?
low
what is "most of my data" in case of an array when your data are on top
of that array in a filesystem which don't know anything about what
happened with the phyiscal stuff underlying?
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:11 ` Curt
2017-10-10 14:14 ` Reindl Harald
@ 2017-10-10 14:15 ` Phil Turmel
2017-10-10 14:23 ` Curt
1 sibling, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 14:15 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, NeilBrown, linux-raid
On 10/10/2017 10:11 AM, Curt wrote:
>>
>> I've never seen this before, and don't have time at the moment
>> to read the code to understand what it means.
> I did a search and here's the comment before the code I found spits
> out that error message. I don't know if it's useful to you
> /* reshape_position is a little messy.
> * Its value must be a multiple of the larger
> * chunk size, and of the "after" data disks.
> * So when reverting we need to change it to
> * be a multiple of the new "after" data disks,
> * which is the old "before".
> * If it isn't already a multiple of 'before',
> * the only thing we could do would be
> * copy some block around on the disks, which
> * is easy to get wrong.
> * So we reject a revert-reshape unless the
> * alignment is good.
> */
> if (reshape_sectors % reshape_chunk)
>
> That doesn't read good for me.
Yup.
>> Probably. But yeah, "fit state". That's the magic. /-:
>>
> What are the chances if I just do an assemble and let it try to grow
> again that it will complete without issue? Then assuming it completes,
> that I will have most my data still?
Yeah, try without the revert. You don't have to wait for the reshape to
finish to grab your critical files off of it. In fact, you might want
to freeze the reshape to minimize the chance of hitting a not-yet-found URE.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:15 ` Phil Turmel
@ 2017-10-10 14:23 ` Curt
2017-10-10 18:06 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 14:23 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
>
> Yeah, try without the revert. You don't have to wait for the reshape to
> finish to grab your critical files off of it. In fact, you might want
> to freeze the reshape to minimize the chance of hitting a not-yet-found URE.
Ok, so try the before command, but instead of --update=revert-reshape,
do a --update=freeze-reshape?
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 14:23 ` Curt
@ 2017-10-10 18:06 ` Phil Turmel
2017-10-10 19:25 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 18:06 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, NeilBrown, linux-raid
On 10/10/2017 10:23 AM, Curt wrote:
>>
>> Yeah, try without the revert. You don't have to wait for the reshape to
>> finish to grab your critical files off of it. In fact, you might want
>> to freeze the reshape to minimize the chance of hitting a not-yet-found URE.
>
> Ok, so try the before command, but instead of --update=revert-reshape,
> do a --update=freeze-reshape?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Just --freeze-reshape, not --update.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 18:06 ` Phil Turmel
@ 2017-10-10 19:25 ` Curt
2017-10-10 19:42 ` Phil Turmel
2017-10-10 20:47 ` NeilBrown
0 siblings, 2 replies; 59+ messages in thread
From: Curt @ 2017-10-10 19:25 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
>
> Just --freeze-reshape, not --update.
>
Ok, here's the output
mdadm --detail /dev/md127
/dev/md127:
Version : 0.91
Creation Time : Fri Jun 15 15:52:05 2012
Raid Level : raid6
Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 127
Persistence : Superblock is persistent
Update Time : Tue Oct 10 15:11:26 2017
State : clean, FAILED, reshaping
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : unknown
Reshape Status : 0% complete
Delta Devices : 1, (7->8)
UUID : 714a612d:9bd35197:36c91ae3:c168144d
Events : 0.11559682
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 49 1 active sync /dev/sdd1
2 8 33 2 active sync /dev/sdc1
3 8 1 3 active sync /dev/sda1
4 65 145 4 active sync /dev/sdz1
- 0 0 5 removed
6 8 16 6 spare rebuilding /dev/sdb
- 0 0 7 removed
But in my dmesg, I'm seeing task md127_reshape blocked for 120
seconds, and when I cat sync_action, it shows reshape. Which
shouldn't it be frozen or something like that? Also md127_raid6 task
is using 100% cpu. I was going to paste the assemble output, but hit
clear instead of copy. It didn't show any errors I saw, just starting
with 6 drives. reshape isn't using any cpu
If I do a cat of /proc/pid/stack, all I get is
[<ffffffffffffffff>] 0xffffffffffffffff
Should I just let it run?
Here's dmesg output for the raid stuff
521267.896376] sdb: unknown partition table
[521267.896766] md: bind<sdd1>
[521267.896900] md: bind<sdc1>
[521267.897089] md: bind<sda1>
[521267.897336] md: bind<sdz1>
[521267.902584] md: bind<sdb>
[521267.918722] md: bind<sdg1>
[521268.013150] md/raid:md127: reshape will continue
[521268.013168] md/raid:md127: device sdg1 operational as raid disk 0
[521268.013170] md/raid:md127: device sdz1 operational as raid disk 4
[521268.013172] md/raid:md127: device sda1 operational as raid disk 3
[521268.013173] md/raid:md127: device sdc1 operational as raid disk 2
[521268.013175] md/raid:md127: device sdd1 operational as raid disk 1
[521268.014065] md/raid:md127: allocated 0kB
[521268.014096] md/raid:md127: raid level 6 active with 6 out of 8
devices, algorithm 2
[521268.019158] RAID conf printout:
[521268.019160] --- level:6 rd:8 wd:6
[521268.019162] disk 0, o:1, dev:sdg1
[521268.019164] disk 1, o:1, dev:sdd1
[521268.019166] disk 2, o:1, dev:sdc1
[521268.019167] disk 3, o:1, dev:sda1
[521268.019169] disk 4, o:1, dev:sdz1
[521268.019171] disk 6, o:1, dev:sdb
[521268.019208] md127: Warning: Device sdz1 is misaligned
[521268.019210] md127: Warning: Device sda1 is misaligned
[521268.019211] md127: Warning: Device sda1 is misaligned
[521268.019236] md127: detected capacity change from 0 to 10001939824640
[521268.019242] md: reshape of RAID array md127
[521268.019248] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[521268.019249] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for reshape.
[521268.019255] md: using 128k window, over a total of 1953503872k.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 19:25 ` Curt
@ 2017-10-10 19:42 ` Phil Turmel
2017-10-10 19:49 ` Curt
2017-10-10 20:47 ` NeilBrown
1 sibling, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 19:42 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, NeilBrown, linux-raid
On 10/10/2017 03:25 PM, Curt wrote:
>>
>> Just --freeze-reshape, not --update.
>>
> Ok, here's the output
> State : clean, FAILED, reshaping
Hmmm. Both clean and failed. If you can mount it, do so and copy what
you can (in order of importance).
If you can't, I'm out of ideas.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 19:42 ` Phil Turmel
@ 2017-10-10 19:49 ` Curt
2017-10-10 19:51 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 19:49 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
On Tue, Oct 10, 2017 at 3:42 PM, Phil Turmel <philip@turmel.org> wrote:
> On 10/10/2017 03:25 PM, Curt wrote:
>>>
>>> Just --freeze-reshape, not --update.
>>>
>> Ok, here's the output
>
>> State : clean, FAILED, reshaping
>
> Hmmm. Both clean and failed. If you can mount it, do so and copy what
> you can (in order of importance).
>
> If you can't, I'm out of ideas.
>
> Phil
I tried a mount, it just sits there. No errors, which I'm not sure if
that's good or bad, so it's almost like it's just waiting for the
device to not be busy. I noticed it was lists both a reshape and
spare rebuild. Could it be it's trying to rebuild the spare and
getting in some kind of dead lock?
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 19:49 ` Curt
@ 2017-10-10 19:51 ` Curt
2017-10-10 20:18 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 19:51 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
> I tried a mount, it just sits there. No errors, which I'm not sure if
> that's good or bad, so it's almost like it's just waiting for the
> device to not be busy. I noticed it was lists both a reshape and
> spare rebuild. Could it be it's trying to rebuild the spare and
> getting in some kind of dead lock?
Follow up, here's the top process, does this look right to you?
9771 root 20 0 0 0 0 R 100.0 0.0 38:27.01 md127_raid6
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 19:51 ` Curt
@ 2017-10-10 20:18 ` Phil Turmel
2017-10-10 20:29 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 20:18 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, NeilBrown, linux-raid
On 10/10/2017 03:51 PM, Curt wrote:
>> I tried a mount, it just sits there. No errors, which I'm not sure if
>> that's good or bad, so it's almost like it's just waiting for the
>> device to not be busy. I noticed it was lists both a reshape and
>> spare rebuild. Could it be it's trying to rebuild the spare and
>> getting in some kind of dead lock?
>
> Follow up, here's the top process, does this look right to you?
> 9771 root 20 0 0 0 0 R 100.0 0.0 38:27.01 md127_raid6
No, that's not good. And blocking the mount process is a disaster.
Please take notes on all device serials vs. MD device # and consider
trying everything with rescue distro. I generally use SystemRescueCD,
but anything with the latest stable kernel and mdadm might help.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:18 ` Phil Turmel
@ 2017-10-10 20:29 ` Curt
2017-10-10 20:31 ` Phil Turmel
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 20:29 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
>
> No, that's not good. And blocking the mount process is a disaster.
>
> Please take notes on all device serials vs. MD device # and consider
> trying everything with rescue distro. I generally use SystemRescueCD,
> but anything with the latest stable kernel and mdadm might help.
>
> Phil
>
Figured. Doesn't seem like it's really doing anything, outside taking
100% cpu. Doing a cat of /proc/pid/io shows all 0's.
Anyway. Should I just run an mdadm --stop on the md127 or doing you
think I'll have to do a reboot?
After it's stopped. I'm guessing just boot with a rescue CD and try to
do an assemble and then mount and hope it doesn't do the same thing?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:29 ` Curt
@ 2017-10-10 20:31 ` Phil Turmel
2017-10-10 20:48 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: Phil Turmel @ 2017-10-10 20:31 UTC (permalink / raw)
To: Curt; +Cc: Anthony Youngman, NeilBrown, linux-raid
On 10/10/2017 04:29 PM, Curt wrote:
>>
>> No, that's not good. And blocking the mount process is a disaster.
>>
>> Please take notes on all device serials vs. MD device # and consider
>> trying everything with rescue distro. I generally use SystemRescueCD,
>> but anything with the latest stable kernel and mdadm might help.
>>
>> Phil
>>
>
> Figured. Doesn't seem like it's really doing anything, outside taking
> 100% cpu. Doing a cat of /proc/pid/io shows all 0's.
>
> Anyway. Should I just run an mdadm --stop on the md127 or doing you
> think I'll have to do a reboot?
Use --stop. It might work and would be good if it did.
> After it's stopped. I'm guessing just boot with a rescue CD and try to
> do an assemble and then mount and hope it doesn't do the same thing?
Yes. Device letters might change -- that's why you need serial numbers
before you shut down.
Phil
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 19:25 ` Curt
2017-10-10 19:42 ` Phil Turmel
@ 2017-10-10 20:47 ` NeilBrown
2017-10-10 20:58 ` Curt
2017-10-11 2:20 ` Curt
1 sibling, 2 replies; 59+ messages in thread
From: NeilBrown @ 2017-10-10 20:47 UTC (permalink / raw)
To: Curt, Phil Turmel; +Cc: Anthony Youngman, linux-raid
[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]
On Tue, Oct 10 2017, Curt wrote:
>>
>> Just --freeze-reshape, not --update.
>>
> Ok, here's the output
> mdadm --detail /dev/md127
> /dev/md127:
> Version : 0.91
> Creation Time : Fri Jun 15 15:52:05 2012
> Raid Level : raid6
> Array Size : 9767519360 (9315.03 GiB 10001.94 GB)
> Used Dev Size : 1953503872 (1863.01 GiB 2000.39 GB)
> Raid Devices : 8
> Total Devices : 6
> Preferred Minor : 127
> Persistence : Superblock is persistent
>
> Update Time : Tue Oct 10 15:11:26 2017
> State : clean, FAILED, reshaping
> Active Devices : 5
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Consistency Policy : unknown
>
> Reshape Status : 0% complete
> Delta Devices : 1, (7->8)
>
> UUID : 714a612d:9bd35197:36c91ae3:c168144d
> Events : 0.11559682
>
> Number Major Minor RaidDevice State
> 0 8 97 0 active sync /dev/sdg1
> 1 8 49 1 active sync /dev/sdd1
> 2 8 33 2 active sync /dev/sdc1
> 3 8 1 3 active sync /dev/sda1
> 4 65 145 4 active sync /dev/sdz1
> - 0 0 5 removed
> 6 8 16 6 spare rebuilding /dev/sdb
> - 0 0 7 removed
>
> But in my dmesg, I'm seeing task md127_reshape blocked for 120
> seconds, and when I cat sync_action, it shows reshape. Which
> shouldn't it be frozen or something like that? Also md127_raid6 task
> is using 100% cpu. I was going to paste the assemble output, but hit
> clear instead of copy. It didn't show any errors I saw, just starting
> with 6 drives. reshape isn't using any cpu
>
> If I do a cat of /proc/pid/stack, all I get is
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Should I just let it run?
Clearly a kernel bug.
What kernel are you using? Can you try a newer one easily?
Can you please
mkdir /tmp/dump
mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
tar czf --sparse /tmp/dump.tgz /tmp/dump
and send me /tmp/dump.tgz. It will only contains the metadata.
I can then create an identical looking array and experiment.
I doubt if letting it run will bring benefits.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:31 ` Phil Turmel
@ 2017-10-10 20:48 ` Curt
0 siblings, 0 replies; 59+ messages in thread
From: Curt @ 2017-10-10 20:48 UTC (permalink / raw)
To: Phil Turmel; +Cc: Anthony Youngman, NeilBrown, linux-raid
>
> Use --stop. It might work and would be good if it did.
Ok, that's a negative on the stop. It just hung, can't break out.
Anything I should do before I have to reboot this server?
>
>> After it's stopped. I'm guessing just boot with a rescue CD and try to
>> do an assemble and then mount and hope it doesn't do the same thing?
>
> Yes. Device letters might change -- that's why you need serial numbers
> before you shut down.
>
Ok. I'll probably just yank the drives and bring them back to my
place. I have a spare server sitting on a work bench here, so I can
put them in that, so they'll be the only drives. I'll note the serial
info before I do, though.
> Phil
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:47 ` NeilBrown
@ 2017-10-10 20:58 ` Curt
2017-10-10 21:23 ` Curt
2017-10-11 2:20 ` Curt
1 sibling, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 20:58 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, Anthony Youngman, linux-raid
On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:
>
> Clearly a kernel bug.
> What kernel are you using? Can you try a newer one easily?
3.10.0-229.el7.x86_64. I could either update the OS or transfer to a
new server and use a rescue cd as mentioned before.
>
> Can you please
> mkdir /tmp/dump
> mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
> tar czf --sparse /tmp/dump.tgz /tmp/dump
>
Yeah, I will get that for you once I reboot the server.
> and send me /tmp/dump.tgz. It will only contains the metadata.
> I can then create an identical looking array and experiment.
>
> I doubt if letting it run will bring benefits.
>
> NeilBrown
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:58 ` Curt
@ 2017-10-10 21:23 ` Curt
2017-10-10 21:56 ` NeilBrown
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-10 21:23 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, Anthony Youngman, linux-raid
On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>> Can you please
>> mkdir /tmp/dump
>> mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>> tar czf --sparse /tmp/dump.tgz /tmp/dump
>>
> Yeah, I will get that for you once I reboot the server.
>
Quick question. Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 21:23 ` Curt
@ 2017-10-10 21:56 ` NeilBrown
2017-10-11 0:26 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: NeilBrown @ 2017-10-10 21:56 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, Anthony Youngman, linux-raid
[-- Attachment #1: Type: text/plain, Size: 469 bytes --]
On Tue, Oct 10 2017, Curt wrote:
> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>
>>> Can you please
>>> mkdir /tmp/dump
>>> mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>> tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>
>> Yeah, I will get that for you once I reboot the server.
>>
> Quick question. Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
"/tmp/dump" - yes, I typed the wrong thing.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 21:56 ` NeilBrown
@ 2017-10-11 0:26 ` Curt
2017-10-11 4:46 ` NeilBrown
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-11 0:26 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, Anthony Youngman, linux-raid
On Tue, Oct 10, 2017 at 5:56 PM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Oct 10 2017, Curt wrote:
>
>> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>>
>>>> Can you please
>>>> mkdir /tmp/dump
>>>> mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>>> tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>>
>>> Yeah, I will get that for you once I reboot the server.
>>>
>> Quick question. Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
>
> "/tmp/dump" - yes, I typed the wrong thing.
>
> Thanks,
> NeilBrown
Out of curiosity, how long should it take to tar these files? I get
the system reports them as 1.9T and there not really that big, hence
the sparse switch. I've just never tarred a sparse file file before.
It's been churning away for about 2 hours, which I'm going to assume
is normal.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-10 20:47 ` NeilBrown
2017-10-10 20:58 ` Curt
@ 2017-10-11 2:20 ` Curt
2017-10-11 4:49 ` NeilBrown
1 sibling, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-11 2:20 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, Anthony Youngman, linux-raid
On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:
> Clearly a kernel bug.
> What kernel are you using? Can you try a newer one easily?
>
I just tried with Kernel 4.13 and got the same results. Could it be
another library that wouldn't be updated with the kernel?
I'll see if I can't get to the data center tomorrow with a rescueCD
and try that. Any other suggestions or further information till then?
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-11 0:26 ` Curt
@ 2017-10-11 4:46 ` NeilBrown
0 siblings, 0 replies; 59+ messages in thread
From: NeilBrown @ 2017-10-11 4:46 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, Anthony Youngman, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]
On Tue, Oct 10 2017, Curt wrote:
> On Tue, Oct 10, 2017 at 5:56 PM, NeilBrown <neilb@suse.com> wrote:
>> On Tue, Oct 10 2017, Curt wrote:
>>
>>> On Tue, Oct 10, 2017 at 4:58 PM, Curt <lightspd@gmail.com> wrote:
>>>
>>>>> Can you please
>>>>> mkdir /tmp/dump
>>>>> mdadm --dump=/dev/dump /dev...list.all.devices.in.the.array
>>>>> tar czf --sparse /tmp/dump.tgz /tmp/dump
>>>>>
>>>> Yeah, I will get that for you once I reboot the server.
>>>>
>>> Quick question. Is that supposed to be mdadm --dump=/dev/dump or /tmp/dump?
>>
>> "/tmp/dump" - yes, I typed the wrong thing.
>>
>> Thanks,
>> NeilBrown
>
> Out of curiosity, how long should it take to tar these files? I get
> the system reports them as 1.9T and there not really that big, hence
> the sparse switch. I've just never tarred a sparse file file before.
> It's been churning away for about 2 hours, which I'm going to assume
> is normal.
With a sufficiently recent 'tar' and kernel (I don't know how recent
they need to be) it should be nearly instantaneous. With older code it
could easily take hours.
The kernel needs to support SEEK_HOLE (v3.1 introduced that) and
tar need to support "--hole-detection=seek", which is the default if it
is supported. I don't know what version.
NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-11 2:20 ` Curt
@ 2017-10-11 4:49 ` NeilBrown
2017-10-11 15:38 ` Curt
0 siblings, 1 reply; 59+ messages in thread
From: NeilBrown @ 2017-10-11 4:49 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, Anthony Youngman, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]
On Tue, Oct 10 2017, Curt wrote:
> On Tue, Oct 10, 2017 at 4:47 PM, NeilBrown <neilb@suse.com> wrote:
>> Clearly a kernel bug.
>> What kernel are you using? Can you try a newer one easily?
>>
> I just tried with Kernel 4.13 and got the same results. Could it be
> another library that wouldn't be updated with the kernel?
Nope. It is a kernel bug that is still present. I can easily reproduce
it on the mainline kernel now that I have your metadata.
I know generally what the problem is. I'll try to craft a fix tomorrow.
You'll need to build and run a new kernel to be able to fix you array.
That might seem scary, but it really isn't that hard.
NeilBrown
>
> I'll see if I can't get to the data center tomorrow with a rescueCD
> and try that. Any other suggestions or further information till then?
>
> Cheers,
> Curt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-11 4:49 ` NeilBrown
@ 2017-10-11 15:38 ` Curt
2017-10-12 6:15 ` NeilBrown
0 siblings, 1 reply; 59+ messages in thread
From: Curt @ 2017-10-11 15:38 UTC (permalink / raw)
To: NeilBrown; +Cc: Phil Turmel, Anthony Youngman, linux-raid
On Wed, Oct 11, 2017 at 12:49 AM, NeilBrown <neilb@suse.com> wrote:
>
> Nope. It is a kernel bug that is still present. I can easily reproduce
> it on the mainline kernel now that I have your metadata.
> I know generally what the problem is. I'll try to craft a fix tomorrow.
> You'll need to build and run a new kernel to be able to fix you array.
> That might seem scary, but it really isn't that hard.
>
> NeilBrown
Awesome thanks. Just let me know what I need to do.
Cheers,
Curt
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: hung grow
2017-10-11 15:38 ` Curt
@ 2017-10-12 6:15 ` NeilBrown
0 siblings, 0 replies; 59+ messages in thread
From: NeilBrown @ 2017-10-12 6:15 UTC (permalink / raw)
To: Curt; +Cc: Phil Turmel, Anthony Youngman, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]
On Wed, Oct 11 2017, Curt wrote:
> On Wed, Oct 11, 2017 at 12:49 AM, NeilBrown <neilb@suse.com> wrote:
>>
>> Nope. It is a kernel bug that is still present. I can easily reproduce
>> it on the mainline kernel now that I have your metadata.
>> I know generally what the problem is. I'll try to craft a fix tomorrow.
>> You'll need to build and run a new kernel to be able to fix you array.
>> That might seem scary, but it really isn't that hard.
>>
>> NeilBrown
> Awesome thanks. Just let me know what I need to do.
I had hoped to have this for you today, but it didn't quite work out. I
have a patch which I believe is correct, but I try not to trust myself
about such things after about 3pm.
Fixing the spin turned out to be very easy, but subsequent testing
showed another problem. 'sdb' is not fully recovered but after the
reverted reshape completed, md thought that it was. It took a while to
figure out exactly what was going on there.
If nothing urgent intervenes tomorrow I'll review the patches and then
send them along with instructions on how to build a kernel.
It might help to know what distro you plan to do that on.
The key steps will be:
1/ get the kernel source
2/ apply the patch
3/ get a .config that matches what your distro kernel
4/ build the kernel
5/ make install modules_install
6/ reboot
The kernel source is simply https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.13.5.tar.xz
The "make install.." assumes your distro has the right helpers
installed.
The ".config" might be
zcat /proc/config.gz > .config
If not, you'll need to hunt for it. Someone can probably help.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
end of thread, other threads:[~2017-10-12 6:15 UTC | newest]
Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-04 17:18 hung grow Curt
2017-10-04 17:51 ` Anthony Youngman
2017-10-04 18:16 ` Curt
2017-10-04 18:29 ` Joe Landman
2017-10-04 18:37 ` Curt
2017-10-04 18:44 ` Joe Landman
2017-10-04 19:01 ` Anthony Youngman
2017-10-04 19:09 ` Curt
2017-10-04 19:46 ` Anthony Youngman
2017-10-04 20:01 ` Curt
2017-10-04 21:08 ` Anthony Youngman
2017-10-04 21:53 ` Phil Turmel
[not found] ` <CADg2FGbnMzLBqWthKY5Uo__ANC2kAqH_8B1G23nhW+7hWJ=KeA@mail.gmail.com>
2017-10-06 1:25 ` Curt
2017-10-06 11:16 ` Wols Lists
[not found] ` <CADg2FGYc-sPjwukuhonoUUCr3ze3PQWv8gtZPnUT=E4CvsQftg@mail.gmail.com>
2017-10-06 13:13 ` Phil Turmel
2017-10-06 14:07 ` Curt
2017-10-06 14:27 ` Joe Landman
2017-10-06 14:27 ` Phil Turmel
2017-10-07 3:09 ` Curt
2017-10-07 3:15 ` Curt
2017-10-07 20:45 ` Curt
2017-10-07 21:29 ` Phil Turmel
2017-10-08 22:40 ` Curt
2017-10-09 1:23 ` NeilBrown
2017-10-09 1:40 ` Curt
2017-10-09 4:28 ` NeilBrown
2017-10-09 4:59 ` Curt
2017-10-09 5:47 ` NeilBrown
2017-10-09 12:41 ` Curt
2017-10-10 12:08 ` Curt
2017-10-10 13:06 ` Phil Turmel
2017-10-10 13:37 ` Anthony Youngman
2017-10-10 14:00 ` Phil Turmel
2017-10-10 14:11 ` Curt
2017-10-10 14:14 ` Reindl Harald
2017-10-10 14:15 ` Phil Turmel
2017-10-10 14:23 ` Curt
2017-10-10 18:06 ` Phil Turmel
2017-10-10 19:25 ` Curt
2017-10-10 19:42 ` Phil Turmel
2017-10-10 19:49 ` Curt
2017-10-10 19:51 ` Curt
2017-10-10 20:18 ` Phil Turmel
2017-10-10 20:29 ` Curt
2017-10-10 20:31 ` Phil Turmel
2017-10-10 20:48 ` Curt
2017-10-10 20:47 ` NeilBrown
2017-10-10 20:58 ` Curt
2017-10-10 21:23 ` Curt
2017-10-10 21:56 ` NeilBrown
2017-10-11 0:26 ` Curt
2017-10-11 4:46 ` NeilBrown
2017-10-11 2:20 ` Curt
2017-10-11 4:49 ` NeilBrown
2017-10-11 15:38 ` Curt
2017-10-12 6:15 ` NeilBrown
2017-10-10 14:12 ` Anthony Youngman
2017-10-04 19:06 ` Anthony Youngman
2017-10-04 18:57 ` Anthony Youngman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.