* mdadm RAID6 "active" with spares and failed disks; need help
@ 2015-01-06 14:16 Matt Callaghan
0 siblings, 0 replies; 11+ messages in thread
From: Matt Callaghan @ 2015-01-06 14:16 UTC (permalink / raw)
To: linux-raid
I think I'm in a really bad state. Could an expert w/ mdadm please help?
I have a RAID6 mdadm device, and it got really messed up with spares:
{{{
md2000 : active raid6 sdm1[8](S) sdo1[3] sdi1[4] sdn1[2] sdk1[0](F)
sdl1[6] sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/5] [__UU_UUU]
}}}
And is now really broken (inactive)
{{{
md2000 : inactive sdn1[2](S) sdm1[8](S) sdl1[6](S) sdp1[7](S) sdi1[4](S)
sdo1[3](S) sdk1[0](S)
13674593976 blocks super 1.1
}}}
I have a forum post going w/ full details
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I /think/ I need to force re-assembly here, but I'd like some review
from the experts before proceeding.
Thank you in advance for your time,
~Matt/Fermulator
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
[not found] ` <BLU436-SMTP135A84B36120ACD11B2783D81F70@phx.gbl>
@ 2015-03-28 17:40 ` Phil Turmel
0 siblings, 0 replies; 11+ messages in thread
From: Phil Turmel @ 2015-03-28 17:40 UTC (permalink / raw)
To: Matt Callaghan, linux-raid
Hi Matt,
I didn't see this make it to linux-raid, so I'll quote more than normal.
Oh, and convention on kernel.org is to reply-to-all, trim unnecessary
quotes, and avoid top-posting. Please.
On 03/27/2015 11:10 PM, Matt Callaghan wrote:
> Just noticed the lsdrv [1] link to git; got it, here's the output
> {{{
> fermulator@fermmy-mdadm:~/downloads/lsdrv/lsdrv$ ./lsdrv
> PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40)
> ├scsi 0:x:x:x [Empty]
> └scsi 1:0:0:0 ATA Maxtor 6Y160M0
> └sda 152.67g [8:0] Empty/Unknown
> ├sda1 512.00m [8:1] Empty/Unknown
> │└Mounted as /dev/sda1 @ /boot/efi
> ├sda2 148.71g [8:2] Empty/Unknown
> │└Mounted as /dev/disk/by-uuid/5549ca2f-758a-4e04-8e36-cf4544bef4fb @ /
> └sda3 3.46g [8:3] Empty/Unknown
> PCI [mptsas] 05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
> ├scsi 2:0:0:0 ATA ST2000DL003-9VT1
> │└sdb 1.82t [8:16] Empty/Unknown
> │ └sdb1 1.82t [8:17] Empty/Unknown
> ├scsi 2:0:1:0 ATA ST2000DL003-9VT1
> │└sdc 1.82t [8:32] Empty/Unknown
> │ └sdc1 1.82t [8:33] Empty/Unknown
> ├scsi 2:0:2:0 ATA ST2000DL003-9VT1
> │└sdd 1.82t [8:48] Empty/Unknown
> │ └sdd1 1.82t [8:49] Empty/Unknown
> ├scsi 2:0:3:0 ATA ST2000VN000-1H31
> │└sde 1.82t [8:64] Empty/Unknown
> │ └sde1 1.82t [8:65] Empty/Unknown
> ├scsi 2:0:4:0 ATA ST2000DL003-9VT1
> │└sdf 1.82t [8:80] Empty/Unknown
> │ └sdf1 1.82t [8:81] Empty/Unknown
> ├scsi 2:0:5:0 ATA ST2000DL003-9VT1
> │└sdg 1.82t [8:96] Empty/Unknown
> │ └sdg1 1.82t [8:97] Empty/Unknown
> ├scsi 2:0:6:0 ATA ST2000DL003-9VT1
> │└sdh 1.82t [8:112] Empty/Unknown
> │ └sdh1 1.82t [8:113] Empty/Unknown
> ├scsi 2:0:7:0 ATA ST2000VN000-1H31
> │└sdi 1.82t [8:128] Empty/Unknown
> │ └sdi1 1.82t [8:129] Empty/Unknown
> └scsi 2:x:x:x [Empty]
> Other Block Devices
> ├loop0 0.00k [7:0] Empty/Unknown
> ├loop1 0.00k [7:1] Empty/Unknown
> ├loop2 0.00k [7:2] Empty/Unknown
> ├loop3 0.00k [7:3] Empty/Unknown
> ├loop4 0.00k [7:4] Empty/Unknown
> ├loop5 0.00k [7:5] Empty/Unknown
> ├loop6 0.00k [7:6] Empty/Unknown
> ├loop7 0.00k [7:7] Empty/Unknown
> ├ram0 64.00m [1:0] Empty/Unknown
> ├ram1 64.00m [1:1] Empty/Unknown
> ├ram2 64.00m [1:2] Empty/Unknown
> ├ram3 64.00m [1:3] Empty/Unknown
> ├ram4 64.00m [1:4] Empty/Unknown
> ├ram5 64.00m [1:5] Empty/Unknown
> ├ram6 64.00m [1:6] Empty/Unknown
> ├ram7 64.00m [1:7] Empty/Unknown
> ├ram8 64.00m [1:8] Empty/Unknown
> ├ram9 64.00m [1:9] Empty/Unknown
> ├ram10 64.00m [1:10] Empty/Unknown
> ├ram11 64.00m [1:11] Empty/Unknown
> ├ram12 64.00m [1:12] Empty/Unknown
> ├ram13 64.00m [1:13] Empty/Unknown
> ├ram14 64.00m [1:14] Empty/Unknown
> └ram15 64.00m [1:15] Empty/Unknown
> }}}
Ok. Not that helpful. I suspect you had error messages about missing
utilities. No serial numbers.
[trim /]
> mdadm output as of NOW. But note that the output here is likely useless
> since the last thing I was trying to was getting the array back together
> as per the forum posting... (it's definitely not in the original state
> anymore...)
Yep, useless.
[trim /]
> smartctl outputs are:
/dev/sdb:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD0XWHR
> LU WWN Device Id: 5 000c50 02f4197f5
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:05 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 113 099 006 - 51859880
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 422
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 072 060 030 - 17185766
> 9 Power_On_Hours -O--CK 061 061 000 - 34871
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 71
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 099 099 000 - 1
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 094 094 000 - 6
> 190 Airflow_Temperature_Cel -O---K 059 043 045 Past 41 (5 77 42 35 0)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 420
> 193 Load_Cycle_Count -O--CK 100 100 000 - 422
> 194 Temperature_Celsius -O---K 041 057 000 - 41 (0 13 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 51859880
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 16990890657845
> 241 Total_LBAs_Written ------ 100 253 000 - 731266756
> 242 Total_LBAs_Read ------ 100 253 000 - 1129016466
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
>
> SCT Error Recovery Control command not supported
Now we know why your array fell apart. Using green and/or desktop
drives without mitigating the timeout mismatch problem.
/dev/sdc:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD1B1ZJ
> LU WWN Device Id: 5 000c50 02f361865
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:06 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 112 090 006 - 44947192
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 68
> 5 Reallocated_Sector_Ct PO--CK 078 078 036 - 14728
> 7 Seek_Error_Rate POSR-- 072 066 030 - 15873942
> 9 Power_On_Hours -O--CK 061 061 000 - 34875
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 001 001 000 - 823
> 188 Command_Timeout -O--CK 100 099 000 - 65539
> 189 High_Fly_Writes -O-RCK 093 093 000 - 7
> 190 Airflow_Temperature_Cel -O---K 058 044 045 Past 42 (2 158 44 36 0)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 65
> 193 Load_Cycle_Count -O--CK 100 100 000 - 68
> 194 Temperature_Celsius -O---K 042 056 000 - 42 (0 13 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 016 003 000 - 44947192
> 197 Current_Pending_Sector -O--C- 089 089 000 - 952
^^^^^
Wow!
> 198 Offline_Uncorrectable ----C- 089 089 000 - 952
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 141149805250605
> 241 Total_LBAs_Written ------ 100 253 000 - 3292940140
> 242 Total_LBAs_Read ------ 100 253 000 - 496297916
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
>
> SCT Error Recovery Control command not supported
And again.
/dev/sdd:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD15M4K
> LU WWN Device Id: 5 000c50 02f386588
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 117 099 006 - 153485440
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 352
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 076 060 030 - 43819206
> 9 Power_On_Hours -O--CK 061 061 000 - 35013
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 097 097 000 - 3
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 099 099 000 - 1
> 190 Airflow_Temperature_Cel -O---K 057 046 045 - 43 (Min/Max 36/43)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 351
> 193 Load_Cycle_Count -O--CK 100 100 000 - 353
> 194 Temperature_Celsius -O---K 043 054 000 - 43 (0 11 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 021 003 000 - 153485440
> 197 Current_Pending_Sector -O--C- 100 100 000 - 8
More Pending sectors. These are locations where unrecoverable read
errors occurred that the firmware is waiting for an overwrite to decide
if they are fixable.
> 198 Offline_Uncorrectable ----C- 100 100 000 - 8
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 134501195876534
> 241 Total_LBAs_Written ------ 100 253 000 - 879538094
> 242 Total_LBAs_Read ------ 100 253 000 - 1783662156
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
Sigh.
/dev/sde:
> === START OF INFORMATION SECTION ===
> Device Model: ST2000VN000-1H3164
> Serial Number: W1H25K77
> LU WWN Device Id: 5 000c50 06a40c121
> Firmware Version: SC42
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5900 rpm
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM feature is: Unavailable
> APM level is: 254 (maximum performance)
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 116 099 006 - 117001736
> 3 Spin_Up_Time PO---- 096 095 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 21
> 5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
> 7 Seek_Error_Rate POSR-- 064 060 030 - 3017660
> 9 Power_On_Hours -O--CK 085 085 000 - 13146
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 21
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 058 058 000 - 42
> 190 Airflow_Temperature_Cel -O---K 065 056 045 - 35 (Min/Max 35/37)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
> 193 Load_Cycle_Count -O--CK 100 100 000 - 21
> 194 Temperature_Celsius -O---K 035 044 000 - 35 (0 16 0 0 0)
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control:
> Read: 1 (0.1 seconds)
> Write: 1 (0.1 seconds)
Interesting. Is this the device default? The drives I've seen that
have a default have either 4.0s or 7.0s.
/dev/sdf:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD18S73
> LU WWN Device Id: 5 000c50 02f3fab7d
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 109 099 006 - 23951160
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 39605538
> 9 Power_On_Hours -O--CK 061 061 000 - 34955
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 75
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 089 089 000 - 11
> 190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 34/42)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 013 003 000 - 23951160
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 194931385731211
> 241 Total_LBAs_Written ------ 100 253 000 - 4208935845
> 242 Total_LBAs_Read ------ 100 253 000 - 3841138908
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
And again.
/dev/sdg:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD1ACSD
> LU WWN Device Id: 5 000c50 02f31ac2f
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:08 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 113 099 006 - 50711848
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 41597886
> 9 Power_On_Hours -O--CK 061 061 000 - 34955
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 100 100 000 - 0
> 190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 36/43)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 50711848
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 121040768370827
> 241 Total_LBAs_Written ------ 100 253 000 - 1173584109
> 242 Total_LBAs_Read ------ 100 253 000 - 1269612579
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
And sigh again. Broken record, I know. But this is a big deal.
/dev/sdh:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD18S0M
> LU WWN Device Id: 5 000c50 02f3f4ec7
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:08 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 119 099 006 - 229878536
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 38838566
> 9 Power_On_Hours -O--CK 061 061 000 - 34957
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 76
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 094 094 000 - 6
> 190 Airflow_Temperature_Cel -O---K 061 051 045 - 39 (Min/Max 29/40)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 039 049 000 - 39 (0 11 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 023 003 000 - 229878536
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 30356828883085
> 241 Total_LBAs_Written ------ 100 253 000 - 16063676
> 242 Total_LBAs_Read ------ 100 253 000 - 2558000514
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
/dev/sdi:
> === START OF INFORMATION SECTION ===
> Device Model: ST2000VN000-1H3164
> Serial Number: W1H25JXM
> LU WWN Device Id: 5 000c50 06a406dab
> Firmware Version: SC42
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5900 rpm
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Fri Mar 27 22:57:09 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM feature is: Unavailable
> APM level is: 254 (maximum performance)
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 119 099 006 - 218566352
> 3 Spin_Up_Time PO---- 096 096 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 21
> 5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
> 7 Seek_Error_Rate POSR-- 064 060 030 - 3082219
> 9 Power_On_Hours -O--CK 085 085 000 - 13146
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 21
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 050 050 000 - 50
> 190 Airflow_Temperature_Cel -O---K 064 052 045 - 36 (Min/Max 36/38)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
> 193 Load_Cycle_Count -O--CK 100 100 000 - 21
> 194 Temperature_Celsius -O---K 036 048 000 - 36 (0 16 0 0 0)
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control:
> Read: 1 (0.1 seconds)
> Write: 1 (0.1 seconds)
So. You have eight devices that need to make a raid6, and you have no
order information. You have two devices with pending errors that cannot
help us without role #s.
First, you need to deal with the timeout mismatch problem. Only two of
your devices support ERC, so you will need to set long driver timeouts.
Some reading:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142504030927143&w=2
As for the latter link, I haven't tested that. When I needed such
features myself, I just put the appropriate commands into rc.local.
Since then, I've retired all of my non-raid-rated drives.
Next, you need to run numerous "mdadm --create --assume-clean" attempts
to figure out your device role order. You have 8-factorial permutations
to try (40,320). /dev/sdc and /dev/sdd have pending errors, so leave
them out (use "missing" in their places).
Your only info from the original post that shows all of the necessary
device characteristics is this:
> /dev/sdj1:
> Magic : a92b4efc
> Version : 1.1
> Feature Map : 0x2
> Array UUID : 15d2158f:5cf74d95:fd7f5607:0e447573
> Name : fermmy-server:2000 (local to host fermmy-server)
> Creation Time : Fri Apr 22 01:12:07 2011
> Raid Level : raid6
> Raid Devices : 8
>
> Avail Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
> Array Size : 11721080448 (11178.09 GiB 12002.39 GB)
> Data Offset : 304 sectors
> Super Offset : 0 sectors
> Recovery Offset : 2441891840 sectors
> State : clean
> Device UUID : eee3ae0e:f594fdba:58e19113:bc196464
>
> Update Time : Mon Jan 5 00:30:41 2015
> Checksum : 7a5a498d - correct
> Events : 42912
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 4
> Array State : A.AAAAAA ('A' == active, '.' == missing)
Note that the data offset is 304. Some of your devices reported a data
offset of 264. None of the reports were from original undisturbed
devices, so we really don't know what offset is correct. "mdadm --add"
will use that mdadm version's offset if it can.
I suggest you try to re-establish the distro you used at the time (April
2011) in a VM and create some test arrays with its version of mdadm to
get the offset to try first.
You then need to create a script that will perform the necessary "mdadm
--create --assume-clean" operations, followed by an "fsck -n" of the
device each time to see how messed up it is. Each attempt into its own
log file, so you can see (by size) which attempts were "cleanest".
Inspect the "best" log files manually to see what was found. With 40k
permutations, you may need to work out some grepping that will help
identify bad from possibly good.
If none of them come up relatively clean, try again with your next best
guess on the offset.
Good luck!
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-03-28 1:59 ` Phil Turmel
@ 2015-03-28 10:11 ` Roman Mamedov
[not found] ` <55161943.1090206@sympatico.ca>
1 sibling, 0 replies; 11+ messages in thread
From: Roman Mamedov @ 2015-03-28 10:11 UTC (permalink / raw)
To: Phil Turmel; +Cc: Matt Callaghan, linux-raid
[-- Attachment #1: Type: text/plain, Size: 607 bytes --]
On Fri, 27 Mar 2015 21:59:38 -0400
Phil Turmel <philip@turmel.org> wrote:
> I read through this. Given all of the destructive actions you took, I
> am doubtful you will ever get your data. Like mounting "readonly".
> That gives you a readonly filesystem, but it writes to the device.
> Possibly a great deal if there's a journal to replay.
Are you sure, which FS does that? I remember some discussion on FS lists
(Btrfs?), and IIRC the consensus and the implemented behavior was that the
device shouldn't ever be touched with writes on RO mounts, no matter what.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-03-27 23:48 ` Matt Callaghan
@ 2015-03-28 1:59 ` Phil Turmel
2015-03-28 10:11 ` Roman Mamedov
[not found] ` <55161943.1090206@sympatico.ca>
0 siblings, 2 replies; 11+ messages in thread
From: Phil Turmel @ 2015-03-28 1:59 UTC (permalink / raw)
To: Matt Callaghan, linux-raid
Hi Matt,
On 03/27/2015 07:48 PM, Matt Callaghan wrote:
> Back at it with fresh brain and fresh hardware. (several months ago I
> got part-way through Valentine's ideas but not all the way -- decided to
> get a clean setup before progressing further)
>
> I have built a new (fresh/clean) server, and compiled+installed the
> latest mdadm v3.3.2.
> The 8x drives from this RAID6 array have also been moved to the new
> temporary server.
>
> Now of course, in the new server, the device labels are different.
> I need to map the previous "known labels" in the old server (/dev/sdX)
> to the "new labels" in order to get the drive ordering for re-assembly
> right.
> http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I read through this. Given all of the destructive actions you took, I
am doubtful you will ever get your data. Like mounting "readonly".
That gives you a readonly filesystem, but it writes to the device.
Possibly a great deal if there's a journal to replay. You also trimmed
much useful data with "grep" that probably would help us save you now.
However, in the hope you might have useful data that can be correlated
with current status, start with lsdrv [1]. Paste the output in your
reply with word wrap turned off. That'll at least give us a correlation
between device name and serial number.
> e.g. before I had:
> {{{
> /dev/sd[nmlpiokj]1
> }}}
FWIW, it is not safe to use square bracket notation when order matters.
> , and now I have:
> {{{
> /dev/sd[abcdefghi]1
> }}}
The linux 'sd' driver has never guaranteed consistent device names.
It's merely an artifact of boot timing that makes it look that way.
Which is why array members have superblocks that record the roles. You
absolutely *must* have accurate role numbers to get your data back.
Show complete 'mdadm -E' output for all of your member partitions as
they stand now.
> Unfortunately I don't have any smartctl output saved from the previous
> server and I can't find a way to map device drive label to serial numbers.
> Any thoughts how I could do this based on the data I have saved in that
> forum post?
Please show current 'smartctl -x' output for all of these devices, too.
Just paste it all in your reply (with word wrap turned off).
Phil
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-22 9:47 ` Valentijn
@ 2015-03-27 23:48 ` Matt Callaghan
2015-03-28 1:59 ` Phil Turmel
0 siblings, 1 reply; 11+ messages in thread
From: Matt Callaghan @ 2015-03-27 23:48 UTC (permalink / raw)
To: linux-raid
Back at it with fresh brain and fresh hardware. (several months ago I
got part-way through Valentine's ideas but not all the way -- decided to
get a clean setup before progressing further)
I have built a new (fresh/clean) server, and compiled+installed the
latest mdadm v3.3.2.
The 8x drives from this RAID6 array have also been moved to the new
temporary server.
Now of course, in the new server, the device labels are different.
I need to map the previous "known labels" in the old server (/dev/sdX)
to the "new labels" in order to get the drive ordering for re-assembly
right.
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
e.g. before I had:
{{{
/dev/sd[nmlpiokj]1
}}}
, and now I have:
{{{
/dev/sd[abcdefghi]1
}}}
Unfortunately I don't have any smartctl output saved from the previous
server and I can't find a way to map device drive label to serial numbers.
Any thoughts how I could do this based on the data I have saved in that
forum post?
~Matt
-------- Original Message --------
From: Valentijn <v@lentijn.sess.ink>
Sent: 1/22/2015, 4:47:38 AM
To: Matt Callaghan <matt_callaghan@sympatico.ca>, Wols Lists
<antlists@youngman.org.uk>, linux-raid@vger.kernel.org
Cc:
Subject: Re: mdadm RAID6 "active" with spares and failed disks; need help
> Hi Matt,
As long as your data is still somewhere on these disks, all is not -
necessarily - lost. You could still try using dumpe2fs (and later
e2fsck) and/or dumpe2fs with different superblocks. And even if you
cannot find your file system by any means, you could try to use the
"foremost" utility to scrape off images, documents and the like from
these disks.
So I still don't think all is lost. However, I do think that will cost
more time. You may want to dedicate a spare machine to this task,
because of the resources.
I see that your mdadm says this, somewhere along your odyssee:
mdadm: /dev/sdk1 appears to contain an ext2fs file system
size=1695282944K mtime=Tue Apr 12 11:10:24 1977
... which could mean (I'm not sure, I'm just guessing) that due to the
internal bitmap, your fs has been overwritten.
Your new array in fact said:
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jan 7 09:46:44 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c7603819 - correct
Events : 0
... as far as I understand, this means that 8 blocks from the
superblock, some - whatever size - sectors were occupied by the Internal
Bitmap, which, in turn, would mean your filesystem superblock has been
overwritten.
The good news is: there is more than one superblock.
BTW, didn't you have the right raid drive ordering from the original
disks? You did have output of "mdadm --examine" after the array broke
down, didn't you? So your "create" statement is, by definition, correct
if a new "--examine" output shows the same output - hence the filesystem
is correct if the latter is the case?
So please try if "dumpe2fs -h -o superblock=32768" does anything. Or
98304, 163840, 229376. Dumpe2fs just dumps the fs header, nothing more.
If dumpe2fs doesn't do anything (but complain that it "Couldn't find
valid filesystem superblock"), then you could still try if "foremost"
finds anything. It's not that hard to use, you simply dedicate some
storage to it and tell it to scrape your array. It *will* find things
and it's up to you to see if
1) documents, images and the like are all 64K or 128K or less - and/or
contain large blocks of rubbish. This probably means you have the wrong
array config, because foremost in this case only finds single "chunks"
with correct data - if a file is longer, it doesn't find it and/or spews
out random data from other images
2) documents, images etcetera are OK. This means your array is OK. You
then can use foremost to scrape off everything (it may take weeks but it
could work), or simply try to find where the filesystem superblock hangs
out (if the array is in good order, the fs superblock must be somewhere,
right?)
Please, please try to do as little as possible on the real disks. Use
dmsetup to create snapshots. Copy the disks. Use hardware that is in
good state - you don't want to loose your data that you just found back
because the memory is flakey, do you? ;-)
I hope it's going to work.
Best regards,
Valentijn
On 01/21/15 01:34, Matt Callaghan wrote:
I tried again with the --bitmap=none, clearly that was a miss on my part.
However, still even with that correction, and attempting across varying
combinations of "drive ordering", the filesystem appears corrupt.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-21 0:34 ` Matt Callaghan
@ 2015-01-22 9:47 ` Valentijn
2015-03-27 23:48 ` Matt Callaghan
0 siblings, 1 reply; 11+ messages in thread
From: Valentijn @ 2015-01-22 9:47 UTC (permalink / raw)
To: Matt Callaghan, Wols Lists, linux-raid
Hi Matt,
As long as your data is still somewhere on these disks, all is not -
necessarily - lost. You could still try using dumpe2fs (and later
e2fsck) and/or dumpe2fs with different superblocks. And even if you
cannot find your file system by any means, you could try to use the
"foremost" utility to scrape off images, documents and the like from
these disks.
So I still don't think all is lost. However, I do think that will cost
more time. You may want to dedicate a spare machine to this task,
because of the resources.
I see that your mdadm says this, somewhere along your odyssee:
mdadm: /dev/sdk1 appears to contain an ext2fs file system
size=1695282944K mtime=Tue Apr 12 11:10:24 1977
... which could mean (I'm not sure, I'm just guessing) that due to the
internal bitmap, your fs has been overwritten.
Your new array in fact said:
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jan 7 09:46:44 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c7603819 - correct
Events : 0
... as far as I understand, this means that 8 blocks from the
superblock, some - whatever size - sectors were occupied by the Internal
Bitmap, which, in turn, would mean your filesystem superblock has been
overwritten.
The good news is: there is more than one superblock.
BTW, didn't you have the right raid drive ordering from the original
disks? You did have output of "mdadm --examine" after the array broke
down, didn't you? So your "create" statement is, by definition, correct
if a new "--examine" output shows the same output - hence the filesystem
is correct if the latter is the case?
So please try if "dumpe2fs -h -o superblock=32768" does anything. Or
98304, 163840, 229376. Dumpe2fs just dumps the fs header, nothing more.
If dumpe2fs doesn't do anything (but complain that it "Couldn't find
valid filesystem superblock"), then you could still try if "foremost"
finds anything. It's not that hard to use, you simply dedicate some
storage to it and tell it to scrape your array. It *will* find things
and it's up to you to see if
1) documents, images and the like are all 64K or 128K or less - and/or
contain large blocks of rubbish. This probably means you have the wrong
array config, because foremost in this case only finds single "chunks"
with correct data - if a file is longer, it doesn't find it and/or spews
out random data from other images
2) documents, images etcetera are OK. This means your array is OK. You
then can use foremost to scrape off everything (it may take weeks but it
could work), or simply try to find where the filesystem superblock hangs
out (if the array is in good order, the fs superblock must be somewhere,
right?)
Please, please try to do as little as possible on the real disks. Use
dmsetup to create snapshots. Copy the disks. Use hardware that is in
good state - you don't want to loose your data that you just found back
because the memory is flakey, do you? ;-)
I hope it's going to work.
Best regards,
Valentijn
On 01/21/15 01:34, Matt Callaghan wrote:
> I tried again with the --bitmap=none, clearly that was a miss on my part.
> However, still even with that correction, and attempting across varying
> combinations of "drive ordering", the filesystem appears corrupt.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-12 16:35 ` Wols Lists
@ 2015-01-21 0:34 ` Matt Callaghan
2015-01-22 9:47 ` Valentijn
0 siblings, 1 reply; 11+ messages in thread
From: Matt Callaghan @ 2015-01-21 0:34 UTC (permalink / raw)
To: Wols Lists, linux-raid
Thanks Wols and Valentijn for your input.
I tried again with the --bitmap=none, clearly that was a miss on my part.
However, still even with that correction, and attempting across varying
combinations of "drive ordering", the filesystem appears corrupt.
I think I have to accept entire data loss here. :(
~Matt
-------- Original Message --------
From: Wols Lists <antlists@youngman.org.uk>
Sent: 1/12/2015, 11:35:57 AM
To: linux-raid@vger.kernel.org
Cc:
Subject: Re: mdadm RAID6 "active" with spares and failed disks; need help
> On 11/01/15 23:22, Valentijn Sessink wrote:
Also, I would not have dared to run all these statements on "live" (or
dead, for that matter ;-) disks.
I'm no expert either, but looking at the blog, I'm worried he might have
trashed the array with almost the first thing he did :-(
"add" says it will add a drive as spare if it didn't originally belong
to the array. If it adds a spare to a degraded array, the array will
immediately start to repair itself.
OOPS!!! As it sounds like exactly this could have happened - mdadm
didn't recognise the disk as it added it.
Just speculating, but unfortunately this seems quite likely :-(
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-11 23:22 ` Valentijn Sessink
@ 2015-01-12 16:35 ` Wols Lists
2015-01-21 0:34 ` Matt Callaghan
0 siblings, 1 reply; 11+ messages in thread
From: Wols Lists @ 2015-01-12 16:35 UTC (permalink / raw)
To: linux-raid
On 11/01/15 23:22, Valentijn Sessink wrote:
> Also, I would not have dared to run all these statements on "live" (or
> dead, for that matter ;-) disks.
I'm no expert either, but looking at the blog, I'm worried he might have
trashed the array with almost the first thing he did :-(
"add" says it will add a drive as spare if it didn't originally belong
to the array. If it adds a spare to a degraded array, the array will
immediately start to repair itself.
OOPS!!! As it sounds like exactly this could have happened - mdadm
didn't recognise the disk as it added it.
Just speculating, but unfortunately this seems quite likely :-(
Cheers,
Wol
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-11 20:26 ` Matt Callaghan
@ 2015-01-11 23:22 ` Valentijn Sessink
2015-01-12 16:35 ` Wols Lists
0 siblings, 1 reply; 11+ messages in thread
From: Valentijn Sessink @ 2015-01-11 23:22 UTC (permalink / raw)
To: Matt Callaghan, linux-raid
Hi Matt,
I'm by no means a specialist, but I have been saving a few arrays
lately, so here's my 2 cents. From what I see, I'd say you're almost
there, but you didn't use "--bitmap=none" in your create-statement and
as far as I can see, there is no bitmap specified in the original raid
blocks but there is one in the newly created one. I may be wrong
though!, please take my advice with a grain of salt and at your own risk.
Also, I would not have dared to run all these statements on "live" (or
dead, for that matter ;-) disks. See my posting (that is unfinished, but
I'll add info as I have time) at
http://valentijn.sessink.nl/?p=557 where I use "dmsetup" to create a few
virtual disks - all writes are redirected to another device. Fun thing
is, that after that, you can mess up all you want. You just remove the
virtual disk and poof, everything is as it was before (failed raid and
all, isn't that funny? :)
I hope this helps.
Best regards,
Valentijn
On 11-01-15 21:26, Matt Callaghan wrote:
> Updating this e-mail thread. I got the latest mdadm version that
> supports data offset variance per device and attempted to reconstruct
> RAID6 according to previous data, but so far no luck.
> As far as I a can tell (sadly), all of my data is lost. I've updated the
> forum thread with the final details and failures.
> http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
--
✉ v@lentijn.sess.ink ☏ +31777777713 (31 7x7 13)
⌂ durgerdamstraat 29 zaandam
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
2015-01-07 13:34 ` Matt Callaghan
@ 2015-01-11 20:26 ` Matt Callaghan
2015-01-11 23:22 ` Valentijn Sessink
0 siblings, 1 reply; 11+ messages in thread
From: Matt Callaghan @ 2015-01-11 20:26 UTC (permalink / raw)
To: linux-raid
Updating this e-mail thread. I got the latest mdadm version that
supports data offset variance per device and attempted to reconstruct
RAID6 according to previous data, but so far no luck.
As far as I a can tell (sadly), all of my data is lost. I've updated the
forum thread with the final details and failures.
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I'll leave the drives "in this state" until end of the month in hopes
that someone has another idea on how to recover.
NOTE: I will pay $$$ if there is a person that helps me to recover the
data :)
~Matt
-------- Original Message --------
From: Matt Callaghan <matt_callaghan@sympatico.ca>
Sent: Wed 07 Jan 2015 08:34:01 AM EST
To: linux-raid@vger.kernel.org
Cc:
Subject: Re: mdadm RAID6 "active" with spares and failed disks; need help
> Just to give a small update (I realize many people may still be on
holidays)
I've tried to work with a few people on IRC, and in conjunction with
lots of reading from others' experiences attempting to recover the array
but no luck yet.
I /hope/ I haven't ruined anything.
The forum post referenced below has full details, but here's a summary
of "what happened"
notice how some drives are "moving" around :( [either due to a mistake I
made, or the server haulting/lockup during rebuilds, I'm not sure]
{{{
-------------------------------------------------------------------------------------------
| | Device Role #
-------------------------------------------------------------------------------------------
| DEVICE | COMMENTS | Dec GOOD | Jan4 6:28AM | 12:10PM | 12:40PM |
Jan5 12:30AM | 12:50AM | 8:30AM | 6:34PM | Jan6 6:45AM |
-------------------------------------------------------------------------------------------
| /dev/sdi | | 4 | 4 | 4 | 4 |
4 | 4 | 4 | 4 | 4 |
| /dev/sdj | failing | 5 | 5 FAIL | ( ) | 8 |
8 | 8 FAIL | ( ) | ( ) | ( ) |
| /dev/sdk | failing? | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 FAIL | 0 FAIL |
| /dev/sdl | | 6 | 6 | 6 | 6 |
6 | 6 | 6 | 6 | 6 |
| /dev/sdm | | 1 | 1 | 1 | 1 | (
) | ( ) | ( ) | 8 | 8 SPARE |
| /dev/sdn | | 2 | 2 | 2 | 2 |
2 | 2 | 2 | 2 | 2 |
| /dev/sdo | | 3 | 3 | 3 | 3 |
3 | 3 | 3 | 3 | 3 |
| /dev/sdp | | 7 | 7 | 7 | 7 |
7 | 7 | 7 | 7 | 7 |
-------------------------------------------------------------------------------------------
}}}
Full details from my e-mail notifications of /proc/mdstat (although
unfortunately I don't have FULL mdadm --detail/examine information per
state transition)
{{{
Dec GOOD
md2000 : active raid6 sdo1[3] sdj1[5] sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/8] [UUUUUUUU]
FAIL EVENT on Jan 4th @ 6:28AM
md2000 : active raid6 sdo1[3] sdj1[5](F) sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[==============>......] check = 73.6% (1439539228/1953513408)
finish=536.6min speed=15960K/sec
DEGRADED EVENT on Jan 4th @ 6:39AM
md2000 : active raid6 sdo1[3] sdj1[5](F) sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[==============>......] check = 73.6% (1439539228/1953513408)
finish=5091.8min speed=1682K/sec
DEGRADED EVENT on Jan 4th @ 12:10PM
md2000 : active raid6 sdo1[3] sdn1[2] sdi1[4] sdm1[1] sdk1[0] sdp1[7]
sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
DEGRADED EVENT on Jan 4th @ 12:21PM
md2000 : active raid6 sdk1[0] sdo1[3] sdm1[1] sdn1[2] sdi1[4] sdp1[7]
sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
DEGRADED EVENT on Jan 4th @ 12:40PM
md2000 : active raid6 sdj1[8] sdm1[1] sdo1[3] sdn1[2] sdk1[0] sdi1[4]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[>....................] recovery = 0.2% (5137892/1953513408)
finish=921.7min speed=35227K/sec
DEGRADED EVENT on Jan 5th @ 12:30AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8] sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[============>........] recovery = 62.9% (1229102028/1953513408)
finish=259.8min speed=46466K/sec
FAIL SPARE EVENT on Jan 5th @ 12:50AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8](F) sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[=============>.......] recovery = 68.1% (1332029020/1953513408)
finish=150.3min speed=68897K/sec
DEGRADED EVENT on Jan 5th @ 6:43AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8](F) sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[=============>.......] recovery = 68.1% (1332029020/1953513408)
finish=76028.6min speed=136K/sec
TEST MESSAGE on Jan 5th @ 8:30AM
md2000 : active raid6 sdo1[3] sdi1[4] sdn1[2] sdk1[0] sdl1[6] sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
}}}
I've tried mdadm --create --assume-clean for several combinations of the
"device role # ordering", but so far none have exposed a usable ext4
partition for /dev/md2000.
Was speaking with someone on IRC, and it's been shown that the data
offset for the devices has changed over time in mdadm, so I need to
recompile mdadm 3.3.x and attempt it that way.
I'll update when I get to trying that.
~Fermmy
-------- Original Message --------
From: Matt Callaghan <matt_callaghan@sympatico.ca>
Sent: Tue 06 Jan 2015 09:16:52 AM EST
To: linux-raid@vger.kernel.org
Cc:
Subject: mdadm RAID6 "active" with spares and failed disks; need help
I think I'm in a really bad state. Could an expert w/ mdadm please
help?
I have a RAID6 mdadm device, and it got really messed up with spares:
{{{
md2000 : active raid6 sdm1[8](S) sdo1[3] sdi1[4] sdn1[2] sdk1[0](F)
sdl1[6] sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/5] [__UU_UUU]
}}}
And is now really broken (inactive)
{{{
md2000 : inactive sdn1[2](S) sdm1[8](S) sdl1[6](S) sdp1[7](S) sdi1[4](S)
sdo1[3](S) sdk1[0](S)
13674593976 blocks super 1.1
}}}
I have a forum post going w/ full details
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I /think/ I need to force re-assembly here, but I'd like some review
from the experts before proceeding.
Thank you in advance for your time,
~Matt/Fermulator
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: mdadm RAID6 "active" with spares and failed disks; need help
[not found] <54ABEE54.6020707@sympatico.ca>
@ 2015-01-07 13:34 ` Matt Callaghan
2015-01-11 20:26 ` Matt Callaghan
0 siblings, 1 reply; 11+ messages in thread
From: Matt Callaghan @ 2015-01-07 13:34 UTC (permalink / raw)
To: linux-raid
Just to give a small update (I realize many people may still be on holidays)
I've tried to work with a few people on IRC, and in conjunction with
lots of reading from others' experiences attempting to recover the array
but no luck yet.
I /hope/ I haven't ruined anything.
The forum post referenced below has full details, but here's a summary
of "what happened"
notice how some drives are "moving" around :( [either due to a mistake I
made, or the server haulting/lockup during rebuilds, I'm not sure]
{{{
-------------------------------------------------------------------------------------------
| | Device Role #
-------------------------------------------------------------------------------------------
| DEVICE | COMMENTS | Dec GOOD | Jan4 6:28AM | 12:10PM | 12:40PM |
Jan5 12:30AM | 12:50AM | 8:30AM | 6:34PM | Jan6 6:45AM |
-------------------------------------------------------------------------------------------
| /dev/sdi | | 4 | 4 | 4 | 4 |
4 | 4 | 4 | 4 | 4 |
| /dev/sdj | failing | 5 | 5 FAIL | ( ) | 8 |
8 | 8 FAIL | ( ) | ( ) | ( ) |
| /dev/sdk | failing? | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 FAIL | 0 FAIL |
| /dev/sdl | | 6 | 6 | 6 | 6 |
6 | 6 | 6 | 6 | 6 |
| /dev/sdm | | 1 | 1 | 1 | 1 |
( ) | ( ) | ( ) | 8 | 8 SPARE |
| /dev/sdn | | 2 | 2 | 2 | 2 |
2 | 2 | 2 | 2 | 2 |
| /dev/sdo | | 3 | 3 | 3 | 3 |
3 | 3 | 3 | 3 | 3 |
| /dev/sdp | | 7 | 7 | 7 | 7 |
7 | 7 | 7 | 7 | 7 |
-------------------------------------------------------------------------------------------
}}}
Full details from my e-mail notifications of /proc/mdstat (although
unfortunately I don't have FULL mdadm --detail/examine information per
state transition)
{{{
Dec GOOD
md2000 : active raid6 sdo1[3] sdj1[5] sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/8] [UUUUUUUU]
FAIL EVENT on Jan 4th @ 6:28AM
md2000 : active raid6 sdo1[3] sdj1[5](F) sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[==============>......] check = 73.6% (1439539228/1953513408)
finish=536.6min speed=15960K/sec
DEGRADED EVENT on Jan 4th @ 6:39AM
md2000 : active raid6 sdo1[3] sdj1[5](F) sdk1[0] sdi1[4] sdn1[2] sdm1[1]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[==============>......] check = 73.6% (1439539228/1953513408)
finish=5091.8min speed=1682K/sec
DEGRADED EVENT on Jan 4th @ 12:10PM
md2000 : active raid6 sdo1[3] sdn1[2] sdi1[4] sdm1[1] sdk1[0] sdp1[7]
sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
DEGRADED EVENT on Jan 4th @ 12:21PM
md2000 : active raid6 sdk1[0] sdo1[3] sdm1[1] sdn1[2] sdi1[4] sdp1[7]
sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
DEGRADED EVENT on Jan 4th @ 12:40PM
md2000 : active raid6 sdj1[8] sdm1[1] sdo1[3] sdn1[2] sdk1[0] sdi1[4]
sdp1[7] sdl1[6]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/7] [UUUU_UUU]
[>....................] recovery = 0.2% (5137892/1953513408)
finish=921.7min speed=35227K/sec
DEGRADED EVENT on Jan 5th @ 12:30AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8] sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[============>........] recovery = 62.9% (1229102028/1953513408)
finish=259.8min speed=46466K/sec
FAIL SPARE EVENT on Jan 5th @ 12:50AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8](F) sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[=============>.......] recovery = 68.1% (1332029020/1953513408)
finish=150.3min speed=68897K/sec
DEGRADED EVENT on Jan 5th @ 6:43AM
md2000 : active raid6 sdk1[0] sdo1[3] sdn1[2] sdj1[8](F) sdi1[4] sdl1[6]
sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
[=============>.......] recovery = 68.1% (1332029020/1953513408)
finish=76028.6min speed=136K/sec
TEST MESSAGE on Jan 5th @ 8:30AM
md2000 : active raid6 sdo1[3] sdi1[4] sdn1[2] sdk1[0] sdl1[6] sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/6] [U_UU_UUU]
}}}
I've tried mdadm --create --assume-clean for several combinations of the
"device role # ordering", but so far none have exposed a usable ext4
partition for /dev/md2000.
Was speaking with someone on IRC, and it's been shown that the data
offset for the devices has changed over time in mdadm, so I need to
recompile mdadm 3.3.x and attempt it that way.
I'll update when I get to trying that.
~Fermmy
-------- Original Message --------
From: Matt Callaghan <matt_callaghan@sympatico.ca>
Sent: Tue 06 Jan 2015 09:16:52 AM EST
To: linux-raid@vger.kernel.org
Cc:
Subject: mdadm RAID6 "active" with spares and failed disks; need help
> I think I'm in a really bad state. Could an expert w/ mdadm please
> help?
I have a RAID6 mdadm device, and it got really messed up with spares:
{{{
md2000 : active raid6 sdm1[8](S) sdo1[3] sdi1[4] sdn1[2] sdk1[0](F)
sdl1[6] sdp1[7]
11721080448 blocks super 1.1 level 6, 64k chunk, algorithm 2
[8/5] [__UU_UUU]
}}}
And is now really broken (inactive)
{{{
md2000 : inactive sdn1[2](S) sdm1[8](S) sdl1[6](S) sdp1[7](S) sdi1[4](S)
sdo1[3](S) sdk1[0](S)
13674593976 blocks super 1.1
}}}
I have a forum post going w/ full details
http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I /think/ I need to force re-assembly here, but I'd like some review
from the experts before proceeding.
Thank you in advance for your time,
~Matt/Fermulator
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2015-03-28 17:40 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-06 14:16 mdadm RAID6 "active" with spares and failed disks; need help Matt Callaghan
[not found] <54ABEE54.6020707@sympatico.ca>
2015-01-07 13:34 ` Matt Callaghan
2015-01-11 20:26 ` Matt Callaghan
2015-01-11 23:22 ` Valentijn Sessink
2015-01-12 16:35 ` Wols Lists
2015-01-21 0:34 ` Matt Callaghan
2015-01-22 9:47 ` Valentijn
2015-03-27 23:48 ` Matt Callaghan
2015-03-28 1:59 ` Phil Turmel
2015-03-28 10:11 ` Roman Mamedov
[not found] ` <55161943.1090206@sympatico.ca>
[not found] ` <BLU436-SMTP135A84B36120ACD11B2783D81F70@phx.gbl>
2015-03-28 17:40 ` Phil Turmel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.