* RAID 6 Not Mounting (Block device is empty)
@ 2015-11-07 17:05 Francisco Parada
2015-11-07 18:49 ` Phil Turmel
0 siblings, 1 reply; 12+ messages in thread
From: Francisco Parada @ 2015-11-07 17:05 UTC (permalink / raw)
To: linux-raid
Hello,
I’m not sure if this is the right way to go about it, so let me give you my story. I had a 7 x 3TB RAID array (12TB in total), 6 drives in the array and 1 spare that was also part of the array but simply a spare waiting at the ready in case of a drive failure, is what I had running before my array broke. I added two new arrays to my system last night, in order to back up my current RAID 6. The first array was a 3 x 3TB RAID 0, for a total of 9TB. Then a 2 x 1TB RAID 0 array, for a total of 2TB. 9+2 = 11 and although I’m 1TB shy, I knew I had a bunch of crap and redundancy to get rid of, I just really needed a solid backup after I was going to clean up.
After creating the new arrays, I started transferring from my 12TB array, 2TB worth of data to the 2TB RAID 0 array. At some point during the transfer, rsync complained of an I/O error. It seemed to have transferred 500GB worth of data before this mishap. The following morning, I noticed that error, and saw that I couldn’t mount my 12TB array anymore. Mind you, I didn’t touch this original array, but I think what happened was that the I/O error blew 2 of my drives.
What I’m thinking of doing is the following, but I’m just looking for some advice in case I’m missing anything:
sudo mdadm create --assume-clean --level=6 --raid-devices=7 --size=2930135040 /dev/md127 /dev/sde /dev/sdf /dev/sdg /dev/sdh missing missing /dev/sdd
Here is the output of mdadm examination:
=======================================================================
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdb
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : active
Device UUID : b4f8af9e:5d486bf7:d86fba0d:0af9e3a5
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 03:14:41 2015
Checksum : 3c08c930 - correct
Events : 58344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdc
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : active
Device UUID : 822e5181:1c731970:3ae49e25:06a93621
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 03:14:41 2015
Checksum : 2a8f4e1b - correct
Events : 58344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : active
Device UUID : d8c57aac:91fb811c:eab9e82d:a60ca6bf
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 03:14:41 2015
Checksum : a8daa737 - correct
Events : 58344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sde
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : active
Device UUID : 46ae26e3:8fc7b451:548951c6:44426755
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 03:14:41 2015
Checksum : 42e360a7 - correct
Events : 58344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdf
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : clean
Device UUID : 9dc9106f:3a491437:d01b1b49:c7637240
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 04:11:15 2015
Checksum : 21f7bef6 - correct
Events : 58362
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : .AA... ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdg
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : clean
Device UUID : cd8e1df7:b6a8a95e:a744b6f0:27c6fa00
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Nov 4 04:11:15 2015
Checksum : 39bd6edb - correct
Events : 58362
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : .AA... ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sdh
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 4259892e:2291b1be:a1cac81d:81d59c9d
Name : Rex:en1
Creation Time : Mon Apr 20 19:11:03 2015
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : clean
Device UUID : a130d2b7:77f8fe81:48203bc8:d2413cae
Update Time : Sun Nov 1 01:00:56 2015
Checksum : a2939695 - expected a2939694
Events : 58339
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sd[b-h] |grep "Array State"
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Array State : .AA... ('A' == active, '.' == missing, 'R' == replacing)
Array State : .AA... ('A' == active, '.' == missing, 'R' == replacing)
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sd[b-h] |grep "Events"
Events : 58344
Events : 58344
Events : 58344
Events : 58344
Events : 58362
Events : 58362
Events : 58339
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sd[b-h] |grep "raid.status"
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sd[b-h] |grep "Role"
Device Role : Active device 3
Device Role : Active device 4
Device Role : Active device 5
Device Role : Active device 0
Device Role : Active device 1
Device Role : Active device 2
Device Role : spare
cisco@DoctorBanner:~$ sudo mdadm --examine /dev/sd[b-h] |grep "Used"
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
=======================================================================
Thanks for your help in advance, I’m hoping to be able to find something, but so far this is all I’ve got.
Much appreciated!
Cisco--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-07 17:05 RAID 6 Not Mounting (Block device is empty) Francisco Parada
@ 2015-11-07 18:49 ` Phil Turmel
2015-11-07 19:17 ` Francisco Parada
0 siblings, 1 reply; 12+ messages in thread
From: Phil Turmel @ 2015-11-07 18:49 UTC (permalink / raw)
To: Francisco Parada, linux-raid
On 11/07/2015 12:05 PM, Francisco Parada wrote:
> Hello,
>
> I’m not sure if this is the right way to go about it, so let me give
> you my story. I had a 7 x 3TB RAID array (12TB in total), 6 drives
> in the array and 1 spare that was also part of the array but simply a
> spare waiting at the ready in case of a drive failure, is what I had
> running before my array broke. I added two new arrays to my system
> last night, in order to back up my current RAID 6. The first array
> was a 3 x 3TB RAID 0, for a total of 9TB. Then a 2 x 1TB RAID 0
> array, for a total of 2TB. 9+2 = 11 and although I’m 1TB shy, I knew
> I had a bunch of crap and redundancy to get rid of, I just really
> needed a solid backup after I was going to clean up.
Yes, this is the right way to go about it. Missed just a few items that
would help. Good report.
> After creating the new arrays, I started transferring from my 12TB
> array, 2TB worth of data to the 2TB RAID 0 array. At some point
> during the transfer, rsync complained of an I/O error. It seemed to
> have transferred 500GB worth of data before this mishap. The
> following morning, I noticed that error, and saw that I couldn’t
> mount my 12TB array anymore. Mind you, I didn’t touch this original
> array, but I think what happened was that the I/O error blew 2 of my
> drives.
>
> What I’m thinking of doing is the following, but I’m just looking for
> some advice in case I’m missing anything:
>
> sudo mdadm create --assume-clean --level=6 --raid-devices=7 --size=2930135040 /dev/md127 /dev/sde /dev/sdf /dev/sdg /dev/sdh missing
missing /dev/sdd
Absolutely not!
Your array went from running to dead in minutes, so the variation of
event counts doesn't matter that much. You should forcibly re-assemble
with all devices.
However, before you do *anything*, you need to figure out why so many
devices were ejected from your array. Was it a controller glitch? A
power supply failure? Or, most likely, Unrecoverable Read Errors being
exposed by your first-ever backup, combined with timeout mismatch?
Any attempt to reassemble/recreate/recovery will simply blow up again if
the root cause isn't addressed.
In your next reply, please paste:
1) the dmesg from the time around the event, +/- a few minutes.
2) the output of the following drive diagnostics:
for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A -l scterc $x ; done
Do *not* perform any --create operation on your array.
*Do* read the list archives linked below -- if any part of it is
unclear, please ask in your next reply.
Phil
[1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
[2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
[3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
[4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
[5] http://marc.info/?l=linux-raid&m=132477199207506
[6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
[7] http://marc.info/?l=linux-raid&m=142487508806844&w=3
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-07 18:49 ` Phil Turmel
@ 2015-11-07 19:17 ` Francisco Parada
2015-11-07 21:08 ` Phil Turmel
0 siblings, 1 reply; 12+ messages in thread
From: Francisco Parada @ 2015-11-07 19:17 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
Hi Phil,
First, I want to thank you for taking the time to reply to me, I truly appreciate it. Secondly, I must correct my statement “I added two new arrays to my system last night” … I had started writing this email a few nights ago, and shut down the system in order to prevent me from getting frustrated and doing something stupid. So I walked away from all of it, and just forgot to amend my original email. So the system was off for a few days, and I turned it back on a few minutes before sending my email, and dmesg only shows today’s output, and interestingly enough, no timestamps are on there.
> 1) the dmesg from the time around the event, +/- a few minutes.
Having said that, dmesg isn’t showing me anything from that day either, and I just found out that /var/log/messages doesn’t even exist in my Ubuntu Server 15.04. It seems I have to enable that, so that’s one more thing I’m about to do now. Are there any other ways I could possibly retrieve that? I’m afraid the answer will be a solid “no”, but worth asking.
> 2) the output of the following drive diagnostics:
>
> for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A -l scterc $x ; done
==========================================================================================
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Blue Serial ATA
Device Model: WDC WD2500AAJS-00VTA0
Serial Number: WD-WMART0793031
LU WWN Device Id: 5 0014ee 0ab2d3f8e
Firmware Version: 01.01B01
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0003 177 177 021 Pre-fail Always - 2125
4 Start_Stop_Count 0x0032 094 094 000 Old_age Always - 6885
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x000e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 4641
10 Spin_Retry_Count 0x0012 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 226
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 163
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 226
194 Temperature_Celsius 0x0022 116 082 000 Old_age Always - 27
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 4
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00MMMB0
Serial Number: WD-WCAWZ2971763
LU WWN Device Id: 5 0014ee 2b2105a3c
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 1
3 Spin_Up_Time 0x0027 153 148 021 Pre-fail Always - 9350
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 122
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 070 070 000 Old_age Always - 22291
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 106
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 60
193 Load_Cycle_Count 0x0032 094 094 000 Old_age Always - 318888
194 Temperature_Celsius 0x0022 118 090 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 176 000 Old_age Always - 14946
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00D8PB0
Serial Number: WD-WMC4N0H5ZZ3V
LU WWN Device Id: 5 0014ee 6afd6e011
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 4
3 Spin_Up_Time 0x0027 181 178 021 Pre-fail Always - 5933
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 42
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 3994
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 13
193 Load_Cycle_Count 0x0032 172 172 000 Old_age Always - 86564
194 Temperature_Celsius 0x0022 118 110 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 895
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00MMMB0
Serial Number: WD-WCAWZ2957084
LU WWN Device Id: 5 0014ee 25cb87409
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 148 144 021 Pre-fail Always - 9600
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 117
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 070 070 000 Old_age Always - 22299
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 100
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 57
193 Load_Cycle_Count 0x0032 096 096 000 Old_age Always - 314144
194 Temperature_Celsius 0x0022 116 092 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 188 000 Old_age Always - 17745
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00MMMB0
Serial Number: WD-WCAWZ2968521
LU WWN Device Id: 5 0014ee 2b20fc98b
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 148 142 021 Pre-fail Always - 9583
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 123
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 070 070 000 Old_age Always - 22302
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 104
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 52
193 Load_Cycle_Count 0x0032 092 092 000 Old_age Always - 326940
194 Temperature_Celsius 0x0022 116 096 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 199 000 Old_age Always - 14499
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00SPEB0
Serial Number: WD-WCC4E0437242
LU WWN Device Id: 5 0014ee 25e91d5af
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 6
3 Spin_Up_Time 0x0027 176 172 021 Pre-fail Always - 8200
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 97
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 14450
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 81
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 47
193 Load_Cycle_Count 0x0032 159 159 000 Old_age Always - 124845
194 Temperature_Celsius 0x0022 118 099 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 199 000 Old_age Always - 12706
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00MMMB0
Serial Number: WD-WCAWZ2963057
LU WWN Device Id: 5 0014ee 20762f880
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 177 147 021 Pre-fail Always - 8108
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 124
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 070 070 000 Old_age Always - 22301
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 108
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 62
193 Load_Cycle_Count 0x0032 090 090 000 Old_age Always - 330837
194 Temperature_Celsius 0x0022 117 096 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 179 000 Old_age Always - 27256
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.19.0-31-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model: WDC WD30EZRX-00D8PB0
Serial Number: WD-WMC4N1816663
LU WWN Device Id: 5 0014ee 0592d9c0a
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 7 14:03:27 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 165 165 021 Pre-fail Always - 6733
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 36
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 2716
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 32
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 11
193 Load_Cycle_Count 0x0032 192 192 000 Old_age Always - 24669
194 Temperature_Celsius 0x0022 122 112 000 Old_age Always - 28
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 182 000 Old_age Always - 1640
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SCT Error Recovery Control command not supported
==========================================================================================
> Do *not* perform any --create operation on your array.
No worries, I’m not touching that. Thank you for your input.
> *Do* read the list archives linked below -- if any part of it is
> unclear, please ask in your next reply.
>
> Phil
>
> [1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
> [2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
> [3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
> [4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
> [5] http://marc.info/?l=linux-raid&m=132477199207506
> [6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
> [7] http://marc.info/?l=linux-raid&m=142487508806844&w=3
Much appreciated, giving those a look now.
Thank you,
Cisco--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-07 19:17 ` Francisco Parada
@ 2015-11-07 21:08 ` Phil Turmel
[not found] ` <C57F2260-D20C-4688-93F6-CB3CE8CF9F30@abitofthisabitofthat.com>
0 siblings, 1 reply; 12+ messages in thread
From: Phil Turmel @ 2015-11-07 21:08 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
On 11/07/2015 02:17 PM, Francisco Parada wrote:
> Hi Phil,
>
> First, I want to thank you for taking the time to reply to me, I truly appreciate it. Secondly, I must correct my statement “I added two new arrays to my system last night” … I had started writing this email a few nights ago, and shut down the system in order to prevent me from getting frustrated and doing something stupid. So I walked away from all of it, and just forgot to amend my original email. So the system was off for a few days, and I turned it back on a few minutes before sending my email, and dmesg only shows today’s output, and interestingly enough, no timestamps are on there.
That's ok. I was looking at the last update times on your mdadm -E reports.
>> 1) the dmesg from the time around the event, +/- a few minutes.
>
> Having said that, dmesg isn’t showing me anything from that day either, and I just found out that /var/log/messages doesn’t even exist in my Ubuntu Server 15.04. It seems I have to enable that, so that’s one more thing I’m about to do now. Are there any other ways I could possibly retrieve that? I’m afraid the answer will be a solid “no”, but worth asking.
Hmm. Did Ubuntu switch to systemd for that version? If so, you'll need to use journalctl. I'm only now learning that, so you'll have to research the options you need yourself. Or others here on the list will chime in :-)
>> 2) the output of the following drive diagnostics:
>>
>> for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A -l scterc $x ; done
You cut off the "echo $x" part. I very much wanted that to document which drives were which serial numbers.
If you want to be pedantic, combine mdadm -E and smartctl instead:
for x in /dev/sd[a-z] ; do mdadm -E $x ; smartctl -i -A -l scterc $x ; done
However, it is clear from these reports that you are in fact suffering from timeout mismatch:
> SCT Error Recovery Control:
> Read: Disabled
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
> SCT Error Recovery Control command not supported
You will need to apply the workarounds for these drives. The one with scterc disabled is a raid-capable drive that just powers up in desktop mode. Add "smartctl -l scterc,70,70 /dev/sdX" to your boot scripts for that one. For the others, you will need to set a long timeout. For now, before any more mdadm operations, just use the blanket work-around script:
for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
Now, I didn't see any "Current Pending Sector" counts, so I don't think you are suffering from UREs. In fact, with four of your drives dying together, I suspect you overloaded your power supply(ies) with the extra arrays, either electrically or thermally. The power was OK for idle and trivial operations but couldn't handle the load while copying.
Backup across a gigabit lan if you can't get all the necessary drives into the main case.
> ==========================================================================================
>
>> Do *not* perform any --create operation on your array.
> No worries, I’m not touching that. Thank you for your input.
At this point, I'm confident that your complete set of original drives should just be forcibly assembled:
mdadm -Afv /dev/mdX /dev/sd[b-h]
Replace the device letters if they've changed since your mdadm -E reports.
There might be minor filesystem damage if any blocks were in flight to the array when it died. fsck, then mount.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
[not found] ` <C57F2260-D20C-4688-93F6-CB3CE8CF9F30@abitofthisabitofthat.com>
@ 2015-11-07 23:07 ` Phil Turmel
2015-11-08 5:59 ` Francisco Parada
0 siblings, 1 reply; 12+ messages in thread
From: Phil Turmel @ 2015-11-07 23:07 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
On 11/07/2015 05:18 PM, Francisco Parada wrote:
> So it looks like /var/log/syslog is where it’s storing those values,
> I’ve included the full log for that day (gzip attached), but for
> completion, here are the times I see dropped drives first start happening:
[trim /]
Ok. Those four drives cut out within a few milliseconds of each other.
>> However, it is clear from these reports that you are in fact suffering
>> from timeout mismatch:
> Thanks, I didn’t realize that. Here is my command, because your’s
> wasn’t giving me the mdadm portion for some reason:
Because I do all such maintenance as root. In an ubuntu system, use
"sudo su -" to get a full root console.
[trim /]
Also, in the future, turn off word wrap when pasting. :-)
> Thanks, I’ll mount those and circle back. I’m going to head out due to
> a family dinner, so I’ll circle back once I’m done. Once again, thank
> you kindly for your insight!!!
No hurry. And you're welcome.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-07 23:07 ` Phil Turmel
@ 2015-11-08 5:59 ` Francisco Parada
2015-11-08 14:20 ` Phil Turmel
0 siblings, 1 reply; 12+ messages in thread
From: Francisco Parada @ 2015-11-08 5:59 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
> Ok. Those four drives cut out within a few milliseconds of each other.
So I checked the enclosure’s PSU voltages, and it looks like the +5V rail was running at around +3V and the +12V rail at almost +8V under load. So I think your original theory of there being a power issue is correct. It’s a bummer, because I have 2 of these Rosewill 8-bay enclosures, and they both show that same issue. This is very likely why they’ve been discontinued. Granted the speed on them is 1.5Gbps total per channel, but it’s really just for movies, so I’m not concerned about high speed read and writes. I’ll just have to McGuyver it and use different PSUs than the ones they came with. I would imagine, that after about 500GB worth of data being transferred, all the writing caused those drives to get pretty hot, making the PSU that much more likely to cut power.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816132016
> Because I do all such maintenance as root. In an ubuntu system, use
> "sudo su -" to get a full root console.
Ah, gotcha!
> Also, in the future, turn off word wrap when pasting. :-)
Boom! Done deal, there was a setting in my Apple Mail application under Mail -> Preferences -> Fonts & Colors -> “Use fixed-width font for plain text messages”. I think this might fix this word wrap issue, so my apologies. It’s either that, or the fact that I copied and pasted directly from my Terminal application, and it converted the email to HTML, which changed the format.
Now, onto the butter. I tried what you told me, and it showed that the devices were busy:
====================================================
cisco@DoctorBanner:~$ sudo mdadm -Afv /dev/md127 /dev/sd[b-h]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is busy - skipping
mdadm: /dev/sdc is busy - skipping
mdadm: /dev/sdd is busy - skipping
mdadm: /dev/sde is busy - skipping
mdadm: /dev/sdf is busy - skipping
mdadm: /dev/sdg is busy - skipping
mdadm: /dev/sdh is busy - skipping
====================================================
So I continued to stop the array:
====================================================
cisco@DoctorBanner:~$ sudo mdadm --stop /dev/md127
mdadm: stopped /dev/md127
====================================================
Then I tried it once again, but it seems that because they’re displaying that the Events are out of sync, that it’s not able to start the array. Any ideas?
====================================================
cisco@DoctorBanner:~$ sudo mdadm -Afv /dev/md127 /dev/sd[b-h]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sde is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdf is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdg is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdh is identified as a member of /dev/md127, slot -1.
mdadm: added /dev/sdf to /dev/md127 as 1
mdadm: added /dev/sdg to /dev/md127 as 2
mdadm: added /dev/sdb to /dev/md127 as 3 (possibly out of date)
mdadm: added /dev/sdc to /dev/md127 as 4 (possibly out of date)
mdadm: added /dev/sdd to /dev/md127 as 5 (possibly out of date)
mdadm: added /dev/sdh to /dev/md127 as -1
mdadm: added /dev/sde to /dev/md127 as 0
mdadm: /dev/md127 assembled from 3 drives and 1 spare - not enough to start the array.
====================================================
Thanks again,
Cisco----
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 5:59 ` Francisco Parada
@ 2015-11-08 14:20 ` Phil Turmel
2015-11-08 14:48 ` Francisco Parada
0 siblings, 1 reply; 12+ messages in thread
From: Phil Turmel @ 2015-11-08 14:20 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
On 11/08/2015 12:59 AM, Francisco Parada wrote:
>
>> Ok. Those four drives cut out within a few milliseconds of each other.
> So I checked the enclosure’s PSU voltages, and it looks like the +5V rail was running at around +3V and the +12V rail at almost +8V under load. So I think your original theory of there being a power issue is correct. It’s a bummer, because I have 2 of these Rosewill 8-bay enclosures, and they both show that same issue. This is very likely why they’ve been discontinued. Granted the speed on them is 1.5Gbps total per channel, but it’s really just for movies, so I’m not concerned about high speed read and writes. I’ll just have to McGuyver it and use different PSUs than the ones they came with. I would imagine, that after about 500GB worth of data being transferred, all the writing caused those drives to get pretty hot, making the PSU that much more likely to cut power.
> http://www.newegg.com/Product/Product.aspx?Item=N82E16816132016
I consider this *good* news :-)
>> Also, in the future, turn off word wrap when pasting. :-)
> Boom! Done deal, there was a setting in my Apple Mail application under Mail -> Preferences -> Fonts & Colors -> “Use fixed-width font for plain text messages”. I think this might fix this word wrap issue, so my apologies. It’s either that, or the fact that I copied and pasted directly from my Terminal application, and it converted the email to HTML, which changed the format.
Apple Mail? Eeeewwww! :-)
In Thunderbird, word wrap is a separate setting. You might want to
experiment a bit.
> Now, onto the butter. I tried what you told me, and it showed that the devices were busy:
Yeah, that's normal with partial successes. Also, consider not using
md127. That's mdadm's "I don't know what number to use" default.
(Followed by 126, then 125, as so on.)
> Then I tried it once again, but it seems that because they’re displaying that the Events are out of sync, that it’s not able to start the array. Any ideas?
The --force option is supposed to override that. I suspect since you
are running an old kernel, 3.19, you are also runnning an old mdadm. I
recall some assemble bugs being fixed in the past few years. Boot your
system temporarily with a rescue CD -- I recommend the one at
www.sysrescuecd.org -- that has current kernels and mdadm. That'll let
you complete the assembly. After that, with the array idle, a normal
shutdown will let your old kernel assemble it.
Consider upgrading your system when all is safe.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 14:20 ` Phil Turmel
@ 2015-11-08 14:48 ` Francisco Parada
2015-11-08 16:32 ` Francisco Parada
2015-11-08 17:48 ` Phil Turmel
0 siblings, 2 replies; 12+ messages in thread
From: Francisco Parada @ 2015-11-08 14:48 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
> Apple Mail? Eeeewwww! :-)
Hah! Jerk! ;-) Just kidding
> Yeah, that's normal with partial successes. Also, consider not using
> md127. That's mdadm's "I don't know what number to use" default.
> (Followed by 126, then 125, as so on.)
Just want to clarify this bit. Are you saying I should use a number as such to mount it now instead the 127 that it was called until the failure? Or in the future? For the two newer arrays, I had called them 500 and 400, just to be safe so there was no overlap.
[trim /]
> Consider upgrading your system when all is safe.
OK, I wasn't too sure how safe it was to upgrade the kernel to something that Canonical wasn't even using on 15.04 ... But I'll do some more research.
I'll use the recovery media option to give that a shot, and report back shortly.
You're a rockstar!
Thank you,
Cisco--
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 14:48 ` Francisco Parada
@ 2015-11-08 16:32 ` Francisco Parada
2015-11-08 17:25 ` Phil Turmel
2015-11-08 17:48 ` Phil Turmel
1 sibling, 1 reply; 12+ messages in thread
From: Francisco Parada @ 2015-11-08 16:32 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
Phil,
You fucking rock, dude! Systemrescd did the trick. I had to force the assembly twice, but after that, it worked like a charm. I want to thank you from the bottom of my geek's heart! I was struggling to find an answer online, and everyone pointed to recreating the array. I'm so happy I found this mailing list, but further more, I thank my lucky stars that I came across YOU during this treacherous endeavor. My friend, you're a gentlemen and a scholar, nay ... A fucking Viking Berserker, is what you are! You have no idea how much of a big help you were to me, thank you ever so much.
I really can't thank you enough. I have certainly met my match with mdadm, and was truly humbled by this experience.
Thank you, thank you, thank you!!!!!
Cisco
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 16:32 ` Francisco Parada
@ 2015-11-08 17:25 ` Phil Turmel
2015-11-08 17:32 ` Francisco Parada
0 siblings, 1 reply; 12+ messages in thread
From: Phil Turmel @ 2015-11-08 17:25 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
On 11/08/2015 11:32 AM, Francisco Parada wrote:
> Phil,
>
> You f***ing rock, dude! Systemrescd did the trick. I had to force
> the assembly twice, but after that, it worked like a charm. I want to
> thank you from the bottom of my geek's heart! I was struggling to
> find an answer online, and everyone pointed to recreating the array.
> I'm so happy I found this mailing list, but further more, I thank my
> lucky stars that I came across YOU during this treacherous endeavor.
> My friend, you're a gentlemen and a scholar, nay ... A f***ing Viking
> Berserker, is what you are! You have no idea how much of a big help
> you were to me, thank you ever so much.
Viking Berserker? Cool! Although, with my ancestry, Brittany Pirate
might be more appropriate.
{ I had to share this with my wife... She is amused, btw. }
> I really can't thank you enough. I have certainly met my match with
> mdadm, and was truly humbled by this experience.
>
> Thank you, thank you, thank you!!!!!
You're very welcome. Glad to help.
Phil
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 17:25 ` Phil Turmel
@ 2015-11-08 17:32 ` Francisco Parada
0 siblings, 0 replies; 12+ messages in thread
From: Francisco Parada @ 2015-11-08 17:32 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
> Viking Berserker? Cool! Although, with my ancestry, Brittany Pirate
> might be more appropriate.
You deserve whatever title you want to give yourself! I knight you, Sir Brittany Pirate of Tortuga
> { I had to share this with my wife... She is amused, btw. }
Haha!! I'm happy to return the joy you've given me as well.
Thanks!!!!!!
Cisco
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RAID 6 Not Mounting (Block device is empty)
2015-11-08 14:48 ` Francisco Parada
2015-11-08 16:32 ` Francisco Parada
@ 2015-11-08 17:48 ` Phil Turmel
1 sibling, 0 replies; 12+ messages in thread
From: Phil Turmel @ 2015-11-08 17:48 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
Forgot to answer this one.
On 11/08/2015 09:48 AM, Francisco Parada wrote:
>
>> Apple Mail? Eeeewwww! :-)
> Hah! Jerk! ;-) Just kidding
I'm a Linux enthusiast all the way -- servers, desktops, laptop, android
phone, ddwrt routers, etc. I suffer to run Windows for some work
applications, but only in a VM.
>> Yeah, that's normal with partial successes. Also, consider not using
>> md127. That's mdadm's "I don't know what number to use" default.
>> (Followed by 126, then 125, as so on.)
> Just want to clarify this bit. Are you saying I should use a number as such to mount it now instead the 127 that it was called until the failure? Or in the future? For the two newer arrays, I had called them 500 and 400, just to be safe so there was no overlap.
You need an mdadm.conf file that calls out your arrays by name and uuid.
Your distro probably put a template in either /etc/ or /etc/mdadm/.
You can append your current running setup with:
mdadm -Es >>/etc/mdadm/mdadm.conf
Then trim the ARRAY lines to just:
ARRAY /dev/md1 UUID=......
Then use 'update-initramfs' or whatever ubuntu is using in 15.10 to
install that into your early boot environment.
Don't forget to fix your timeout mismatch problem.
Phil
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-11-08 17:48 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-07 17:05 RAID 6 Not Mounting (Block device is empty) Francisco Parada
2015-11-07 18:49 ` Phil Turmel
2015-11-07 19:17 ` Francisco Parada
2015-11-07 21:08 ` Phil Turmel
[not found] ` <C57F2260-D20C-4688-93F6-CB3CE8CF9F30@abitofthisabitofthat.com>
2015-11-07 23:07 ` Phil Turmel
2015-11-08 5:59 ` Francisco Parada
2015-11-08 14:20 ` Phil Turmel
2015-11-08 14:48 ` Francisco Parada
2015-11-08 16:32 ` Francisco Parada
2015-11-08 17:25 ` Phil Turmel
2015-11-08 17:32 ` Francisco Parada
2015-11-08 17:48 ` Phil Turmel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.