* Superblock Missing
@ 2017-09-02 0:52 David Mitchell
2017-09-02 1:43 ` Andreas Klauer
0 siblings, 1 reply; 9+ messages in thread
From: David Mitchell @ 2017-09-02 0:52 UTC (permalink / raw)
To: linux-raid
Respectfully requesting some help.
Very similar to what MasterPrenium posted last month, I'm struggling
to recover from a lost superblock after a grow operation.
I had a raid 5 array running since 2012 (metadata version .90) I
recently grew it from 2TB drives to 4TB.(metadata version 1.2) The
grow was successful and the array ran for approximately 2 weeks.
After that I booted to a LiveCD, changed out a boot drive (not in the
array) reinstalled Grub, etc.
After doing the work I rebooted and attempted to load the array.
With the new boot drive I didn't realize it at the time, but I was
missing the raid456 module. The array wouldn't load. I fixed the
kernel & module issues, and attempted to load but it said missing
superblock info. I was dumbfounded because the array had nothing to
do with the boot/OS drive.
Problem: Superblock not found
The hard drives are all good. All the logs looked clean, and after
several troubleshooting steps I eventually I ran the command
#mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
The array immediately went into a sync which completed successfully
the next day. Still no superblock.
The array starts fine now, but still no superblock.
mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: /dev/md0 has been started with 3 drives.
virtual raid # dmesg
[177604.370207] md: md0 stopped.
[177604.385059] md: bind<sdd1>
[177604.385425] md: bind<sde1>
[177604.385591] md: bind<sdc1>
[177604.415558] md/raid:md0: device sdc1 operational as raid disk 0
[177604.415561] md/raid:md0: device sde1 operational as raid disk 2
[177604.415562] md/raid:md0: device sdd1 operational as raid disk 1
[177604.415850] md/raid:md0: allocated 3316kB
[177604.417538] md/raid:md0: raid level 5 active with 3 out of 3
devices, algorithm 2
[177604.417540] RAID conf printout:
[177604.417541] --- level:5 rd:3 wd:3
[177604.417543] disk 0, o:1, dev:sdc1
[177604.417544] disk 1, o:1, dev:sdd1
[177604.417545] disk 2, o:1, dev:sde1
[177604.417637] created bitmap (30 pages) for device md0
[177604.418110] md0: bitmap initialized from disk: read 2 pages, set 0
of 59518 bits
[177605.200213] md0: detected capacity change from 0 to 7988370735104
# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.
As a troubleshooting step I tried:
mdadm --grow /dev/md0 --size max
mdadm: component size of /dev/md0 unchanged at 3900571648K
I believe this shows the pre-grow size of 4TB, not the current correct
amount of 8TB
Any ideas (other than restore) would be appreciated.
Requested from "When Things Go Wrong"
-------------------------------------
# mdadm --examine /dev/sd[cde]1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
Name : virtual:0 (local to host virtual)
Creation Time : Sat Aug 26 16:21:20 2017
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 77580f12:adf88476:d9c1448b:b041443f
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Aug 31 22:21:32 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : adbbd4f2 - correct
Events : 6261
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
Name : virtual:0 (local to host virtual)
Creation Time : Sat Aug 26 16:21:20 2017
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 1cb11ccc:2aeb095d:6be1838b:7d7b33b7
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Aug 31 22:21:32 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c67c7975 - correct
Events : 6261
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
Name : virtual:0 (local to host virtual)
Creation Time : Sat Aug 26 16:21:20 2017
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c322a4fb:ec99f835:0ce9cb45:ce6b376d
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Aug 31 22:21:32 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 3f3e91d2 - correct
Events : 6261
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
------------------------------------------------
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Aug 26 16:21:20 2017
Raid Level : raid5
Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
Used Dev Size : 3900571648 (3719.88 GiB 3994.19 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Aug 31 22:21:32 2017
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : virtual:0 (local to host virtual)
UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
Events : 6261
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
3 8 65 2 active sync /dev/sde1
---------------------------------------------------------------------------
# ./lsdrv
PCI [ahci] 00:17.0 SATA controller: Intel Corporation Sunrise Point-H
SATA controller [AHCI mode] (rev 31)
├scsi 0:0:0:0 ATA HGST HTS541010A7 {S0A001SSGRU38K}
│└sda 931.51g [8:0] Partitioned (dos)
│ ├sda1 915.64g [8:1] ext4 {119f7409-bb57-4c82-a526-286a639779d6}
│ │└Mounted as /dev/sda1 @ /
│ ├sda2 1.00k [8:2] Partitioned (dos)
│ └sda5 15.87g [8:5] swap {f2217412-d4d6-4a6e-b699-7f2978db9423}
├scsi 1:0:0:0 ATA ST500LM012 HN-M5 {S2SVJ9CC509140}
│└sdb 465.76g [8:16] Partitioned (dos)
│ └sdb1 465.76g [8:17] ext4 {f155bd77-8924-400c-bab4-04b641338282}
├scsi 2:0:0:0 ATA WDC WD40EZRZ-00W {WD-WCC4E0XVVCXH}
│└sdc 3.64t [8:32] Partitioned (gpt)
│ └sdc1 3.63t [8:33] MD raid5 (0/3) (w/ sdd1,sde1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
│ └md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
│ Empty/Unknown
├scsi 3:x:x:x [Empty]
├scsi 4:0:0:0 ATA WDC WD40EZRX-00S {WD-WCC4E0992649}
│└sdd 3.64t [8:48] Partitioned (gpt)
│ └sdd1 3.63t [8:49] MD raid5 (1/3) (w/ sdc1,sde1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
│ └md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
│ Empty/Unknown
└scsi 5:0:0:0 ATA WDC WD40EZRZ-00W {WD-WCC4E6CV0S5R}
└sde 3.64t [8:64] Partitioned (gpt)
└sde1 3.63t [8:65] MD raid5 (2/3) (w/ sdc1,sdd1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
└md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
Empty/Unknown
PCI [pata_pdc2027x] 04:01.0 Mass storage controller: Promise
Technology, Inc. 20269 (rev 02)
├scsi 6:0:0:0 LITE-ON DVDRW SHW-160P6S {LITE-ON_DVDRW_SHW-160P6S}
│└sr0 1.00g [11:0] Empty/Unknown
└scsi 7:x:x:x [Empty]
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
├loop7 0.00k [7:7] Empty/Unknown
├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
│ Empty/Unknown
├ram0 64.00m [1:0] Empty/Unknown
├ram1 64.00m [1:1] Empty/Unknown
├ram2 64.00m [1:2] Empty/Unknown
├ram3 64.00m [1:3] Empty/Unknown
├ram4 64.00m [1:4] Empty/Unknown
├ram5 64.00m [1:5] Empty/Unknown
├ram6 64.00m [1:6] Empty/Unknown
├ram7 64.00m [1:7] Empty/Unknown
├ram8 64.00m [1:8] Empty/Unknown
├ram9 64.00m [1:9] Empty/Unknown
├ram10 64.00m [1:10] Empty/Unknown
├ram11 64.00m [1:11] Empty/Unknown
├ram12 64.00m [1:12] Empty/Unknown
├ram13 64.00m [1:13] Empty/Unknown
├ram14 64.00m [1:14] Empty/Unknown
└ram15 64.00m [1:15] Empty/Unknown
-----------------------------------------------------------------
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md0 : active raid5 sdc1[0] sde1[3] sdd1[1]
7801143296 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/30 pages [0KB], 65536KB chunk
unused devices: <none>
---------------------------------------------------------------
# smartctl --xall /dev/sdc
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EZRZ-00WN9B0
Serial Number: WD-WCC (removed)
LU WWN Device Id: 5 0014ee 261e42d67
Firmware Version: 80.00A80
User Capacity: 4,000,753,476,096 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 31 22:30:51 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (53280) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 532) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 204 181 021 - 6758
4 Start_Stop_Count -O--CK 100 100 000 - 120
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 079 079 000 - 15652
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 120
192 Power-Off_Retract_Count -O--CK 200 200 000 - 83
193 Load_Cycle_Count -O--CK 170 170 000 - 92101
194 Temperature_Celsius -O---K 123 109 000 - 29
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 SATA NCQ Queued Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 39 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xcf GPL VS 65535 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 29 Celsius
Power Cycle Min/Max Temperature: 29/32 Celsius
Lifetime Min/Max Temperature: 19/43 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (109)
Index Estimated Time Temperature Celsius
110 2017-08-31 14:33 29 **********
... ..(476 skipped). .. **********
109 2017-08-31 22:30 29 **********
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 10 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 10 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 178857 Vendor specific
# smartctl --xall /dev/sdd
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Green
Device Model: WDC WD40EZRX-00SPEB0
Serial Number: WD-WCC(removed)
LU WWN Device Id: 5 0014ee 2b47635c2
Firmware Version: 80.00A80
User Capacity: 4,000,753,476,096 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 31 22:30:54 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (52020) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 520) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 202 171 021 - 6900
4 Start_Stop_Count -O--CK 092 092 000 - 8907
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 100 253 000 - 0
9 Power_On_Hours -O--CK 075 075 000 - 18350
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 099 099 000 - 1075
192 Power-Off_Retract_Count -O--CK 200 200 000 - 88
193 Load_Cycle_Count -O--CK 185 185 000 - 46420
194 Temperature_Celsius -O---K 119 098 000 - 33
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 SATA NCQ Queued Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xcf GPL VS 65535 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 33 Celsius
Power Cycle Min/Max Temperature: 33/36 Celsius
Lifetime Min/Max Temperature: 13/54 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (443)
Index Estimated Time Temperature Celsius
444 2017-08-31 14:33 34 ***************
... ..(426 skipped). .. ***************
393 2017-08-31 21:40 34 ***************
394 2017-08-31 21:41 33 **************
... ..( 6 skipped). .. **************
401 2017-08-31 21:48 33 **************
402 2017-08-31 21:49 34 ***************
403 2017-08-31 21:50 34 ***************
404 2017-08-31 21:51 33 **************
... ..( 32 skipped). .. **************
437 2017-08-31 22:24 33 **************
438 2017-08-31 22:25 34 ***************
... ..( 4 skipped). .. ***************
443 2017-08-31 22:30 34 ***************
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 10 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 10 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 178859 Vendor specific
# smartctl --xall /dev/sde
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EZRZ-00WN9B0
Serial Number: WD-WCC(removed)
LU WWN Device Id: 5 0014ee 2b73a30c5
Firmware Version: 80.00A80
User Capacity: 4,000,753,476,096 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 31 22:30:56 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (50160) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 502) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 195 169 021 - 7208
4 Start_Stop_Count -O--CK 100 100 000 - 114
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 079 079 000 - 15651
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 114
192 Power-Off_Retract_Count -O--CK 200 200 000 - 77
193 Load_Cycle_Count -O--CK 170 170 000 - 92696
194 Temperature_Celsius -O---K 122 107 000 - 30
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 SATA NCQ Queued Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 39 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xcf GPL VS 65535 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 30 Celsius
Power Cycle Min/Max Temperature: 30/33 Celsius
Lifetime Min/Max Temperature: 20/45 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (52)
Index Estimated Time Temperature Celsius
53 2017-08-31 14:33 31 ************
... ..( 87 skipped). .. ************
141 2017-08-31 16:01 31 ************
142 2017-08-31 16:02 32 *************
... ..( 18 skipped). .. *************
161 2017-08-31 16:21 32 *************
162 2017-08-31 16:22 31 ************
... ..( 93 skipped). .. ************
256 2017-08-31 17:56 31 ************
257 2017-08-31 17:57 30 ***********
... ..( 7 skipped). .. ***********
265 2017-08-31 18:05 30 ***********
266 2017-08-31 18:06 31 ************
267 2017-08-31 18:07 31 ************
268 2017-08-31 18:08 30 ***********
... ..( 5 skipped). .. ***********
274 2017-08-31 18:14 30 ***********
275 2017-08-31 18:15 31 ************
... ..(254 skipped). .. ************
52 2017-08-31 22:30 31 ************
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 9 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 10 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 178860 Vendor specific
Sincerely,
David
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Superblock Missing
2017-09-02 0:52 Superblock Missing David Mitchell
@ 2017-09-02 1:43 ` Andreas Klauer
2017-09-02 15:43 ` David Mitchell
0 siblings, 1 reply; 9+ messages in thread
From: Andreas Klauer @ 2017-09-02 1:43 UTC (permalink / raw)
To: David Mitchell; +Cc: linux-raid
On Fri, Sep 01, 2017 at 08:52:23PM -0400, David Mitchell wrote:
> # mdadm --examine /dev/md0
> mdadm: No md superblock detected on /dev/md0.
Everyone will assume user error at this point.
/dev/md0 is not supposed to have a mdadm superblock to examine.
It has a filesystem or LUKS or LVM header or whatever it is you do.
For --examine you have to provide one or more member devices.
You can do mdadm --detail /dev/md* or mdadm --examine /dev/sd*.
> Requested from "When Things Go Wrong"
>
> -------------------------------------
> # mdadm --examine /dev/sd[cde]1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 1.2
Oh look. There it is. Your superblock...
Other than that I see no problems. Your Load Cycle Counts are high
and for WD there are tools to finetune that behaviour but it's
unclear whether that's actually a real problem.
> SMART Extended Self-test Log Version: 1 (1 sectors)
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Short offline Completed without error 00% 0 -
Consider running smart tests more regularly.
Regards
Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Superblock Missing
2017-09-02 1:43 ` Andreas Klauer
@ 2017-09-02 15:43 ` David Mitchell
2017-09-02 16:49 ` Andreas Klauer
0 siblings, 1 reply; 9+ messages in thread
From: David Mitchell @ 2017-09-02 15:43 UTC (permalink / raw)
To: linux-raid
Andreas, thank you for your response.
Sorry I'm still confused. While the array seems alive, the
filesystem seems to be missing?
I'm unable to mount the filesystem.
As a troubleshooting step I recoverjpeg and it did find a bunch of files.
#recoverjpeg -s 1m /dev/md0
# dumpe2fs /dev/md0
dumpe2fs 1.42.13 (17-May-2015)
dumpe2fs: Bad magic number in super-block while trying to open /dev/md0
Couldn't find valid filesystem superblock.
------------------
# mount /dev/md0 /mnt/raid/
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
----------------
dmesg:
[307811.909930] EXT4-fs (md0): VFS: Can't find ext4 filesystem
---------------
# mdadm --detail /dev/md*
/dev/md0:
Version : 1.2
Creation Time : Sat Aug 26 16:21:20 2017
Raid Level : raid5
Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
Used Dev Size : 3900571648 (3719.88 GiB 3994.19 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Aug 31 22:21:32 2017
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : virtual:0 (local to host virtual)
UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
Events : 6261
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
3 8 65 2 active sync /dev/sde1
/dev/md127:
Version :
Raid Level : raid0
Total Devices : 0
State : inactive
Number Major Minor RaidDevice
-----------------------------------------------------------------------
Sincerely,
David Mitchell
On Fri, Sep 1, 2017 at 9:43 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Fri, Sep 01, 2017 at 08:52:23PM -0400, David Mitchell wrote:
>> # mdadm --examine /dev/md0
>> mdadm: No md superblock detected on /dev/md0.
>
> Everyone will assume user error at this point.
> /dev/md0 is not supposed to have a mdadm superblock to examine.
> It has a filesystem or LUKS or LVM header or whatever it is you do.
>
> For --examine you have to provide one or more member devices.
>
> You can do mdadm --detail /dev/md* or mdadm --examine /dev/sd*.
>
>> Requested from "When Things Go Wrong"
>>
>> -------------------------------------
>> # mdadm --examine /dev/sd[cde]1
>> /dev/sdc1:
>> Magic : a92b4efc
>> Version : 1.2
>
> Oh look. There it is. Your superblock...
>
> Other than that I see no problems. Your Load Cycle Counts are high
> and for WD there are tools to finetune that behaviour but it's
> unclear whether that's actually a real problem.
>
>> SMART Extended Self-test Log Version: 1 (1 sectors)
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Short offline Completed without error 00% 0 -
>
> Consider running smart tests more regularly.
>
> Regards
> Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Superblock Missing
2017-09-02 15:43 ` David Mitchell
@ 2017-09-02 16:49 ` Andreas Klauer
2017-09-04 3:35 ` David Mitchell
0 siblings, 1 reply; 9+ messages in thread
From: Andreas Klauer @ 2017-09-02 16:49 UTC (permalink / raw)
To: David Mitchell; +Cc: linux-raid
On Sat, Sep 02, 2017 at 11:43:49AM -0400, David Mitchell wrote:
> Sorry I'm still confused.
> While the array seems alive, the filesystem seems to be missing?
Hello David,
I'm the one who is confused here.
You posted to the raid mailing list about a superblock, and you even
posted the mdadm command and error message to go along with it.
How did it turn into a filesystem problem now? In your last mail,
I did not see that at all.
This was what you posted and I thought you were referring to:
> # mdadm --examine /dev/md0
> mdadm: No md superblock detected on /dev/md0.
And thus that was the part I referred to in my answer.
----
> # mdadm --detail /dev/md*
> /dev/md0:
> Version : 1.2
> Creation Time : Sat Aug 26 16:21:20 2017
> Raid Level : raid5
So according to this, your RAID was created about a week ago.
You did write that you just migrated to 4TB drives. There are various ways
to go about that, making a new RAID and copying stuff over from the
old one is not unusual. So this creation time might be perfectly normal.
Of course if this migration happened more than one week ago then I can
only assume you used mdadm --create to fix some RAID issue or other?
mdadm --create is like mkfs: you don't expect data to exist afterwards.
For mdadm --create to retain data you have to get all settings perfectly
right and those settings change depending on when you originally created
it and what you did with it since (e.g. mdadm --grow might change offsets).
I'm not sure how to help you because nothing in your mails explains
how this issue came about.
There are previous discussions on this list about botched mdadm --create,
wrong data offsets, and searching for correct filesystem offsets.
Whatever you do, don't write on these drives while looking for lost data.
Regards
Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Superblock Missing
2017-09-02 16:49 ` Andreas Klauer
@ 2017-09-04 3:35 ` David Mitchell
2017-09-04 15:02 ` Wols Lists
2017-09-04 15:49 ` Andreas Klauer
0 siblings, 2 replies; 9+ messages in thread
From: David Mitchell @ 2017-09-04 3:35 UTC (permalink / raw)
To: linux-raid
Let me provide a more detailed history:
Original array 4 drive 2TB Raid 5 array created back in 2012. Over the
years as drives failed, I replaced the 2 of the 2 TB drives with 4TB
drives, so I had two 4TB and two 2TB drives providing a 6TB usable
array. (2x4 - P)
About a month ago I put a third 4TB drive in, and a "grow" from four
drives to three 4TB drives still in a Raid 5.
Once down to three drives, I then grew to the max size. Everything
worked perfectly, I had 8TB of usable raid 5 with 3 drives. The file
system passed a e2fsck no problem. Ran for about 2 weeks, with
absolutely no problems.
With a spare 2TB drive sitting on my desk, I decided to remove my 1tb
OS drive and put in the 2TB drive as the OS drive. Not to mess up the
array I made sure to zap the UUID, repartition, reformat, etc. Copied
the files from my 1TB OS drive to the "new" 2TB OS drive via a live
USB key.
Rebooted, ran into grub issues. Tried to reinstall grub from a live
USB, but was unable to make it work. I rebooted then did a new Mint
install on the 2TB drive using LVM for the first time. I was
extremely careful NOT to touch any of 4TB drives in the array during
the install. The system rebooted and the new OS worked fine. I did
not start up the array. I commented out the array in fstab and
mdadm.conf
I then copied my old files from the 1TB drive over to the new 2TB
drive with the exception of the kernel and initrd. Rebooted and
everything seemed to work ok with my old files. I went to assemble
and mount the array, specifying the drives. I would not assemble.
I wasn't sure if using LVM for the first time on my boot drive, but
not my array was the problem. What I forgot is I booted to the new
kernel and the old image, and the raid456 module wasn't loading. I
updated the kernel and modules, and rebooted. The raid456 was now
loading fine, but the array would not load and mount. Complained
about superblock. To rule out the LVM issue and kernel I reverted
back to the old 1TB OS drive, so there would be no changes.
The array still would not assemble. Raid456 did load correctly. I
looked at all the array logs and everything seemed clean. The
timestamps on each of the three 4TB raid drives all matched. There
were no apparent errors, but the array would not assemble. I really
do NOT remember running the --create command. What did work was the
force command:
mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
The array instantly loaded and went into a sync. I let the sync
complete overnight. The file system on the raid array seems to be
gone.
I'm looking for any hints on how I might get it back.
The array loads fine, but when I attempted to mount the array, it
complains about not finding the file system
#mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
As a troubleshooting step I ran testdisk to see if that might recover
anything. It doesn't find a partition on /dev/md0, but it did find a
few old floppy image files I had on the array leading me to believe
the data is there.
As a troubleshooting step I ran recoverjpeg which did find picture
files. The first time I ran it it showed a bunch of smaller pictures
and they seemed to be fine. I stopped it an searched only for
pictures above 512k. The pictures over 512k don't display
correctly. It only shows part of the pictures. I'm still searching
the array for other pictures.
At this point I'm hoping for help on next steps in
recovery/troubleshooting. I'm on the raid mailing list rather than
the file system mailing list because I'm assuming that something got
changed when I forced assembly. That perhaps the array is not
presenting the filesystem correctly. Not sure if LDM had anything to
do with this? At some point from the array metadata version changed.
I think this was necessary as I moved from 2TB to 4TB disks.
Below is some history out of the mdadm.conf. It may be missing lines
and steps because of the OS drive swaps, etc.
I've commented these out over time. The /dev/md/1 occurred after the
OS reinstall. I also see the UUID changed. I don't know whether or
not when I changed out the hard drives this caused the UUID's to
change? Or if the OS installation changed the UUID's. Or if
somewhere along the way I ran a create. I really don't remember doing
a --create. Is the change in UUID's the smoking gun that a --create
was done and my data is gone?
mdadm.conf:
# This file was auto-generated on Sun, 06 May 2012 22:55:33 -0400
# by mkconf $Id$
#ARRAY /dev/md0 metadata=0.90 UUID=f5a6ec06:b292289b:1fd984af:668516de
(original)
#ARRAY /dev/md/1 UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c
(after OS reinstall?)
#ARRAY /dev/md0 UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c
#ARRAY /dev/md0 metadata=1.2 name=virtual:1
UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c
# blkid
/dev/sda1: UUID="119f7409-bb57-4c82-a526-286a639779d6" TYPE="ext4"
PARTUUID="3b337c94-01"
/dev/sda5: UUID="f2217412-d4d6-4a6e-b699-7f2978db9423" TYPE="swap"
PARTUUID="3b337c94-05"
/dev/sdb1: UUID="f155bd77-8924-400c-bab4-04b641338282" TYPE="ext4"
PARTUUID="40303808-01"
/dev/sdc1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="77580f12-adf8-8476-d9c1-448bb041443f" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"
/dev/sdd1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="1cb11ccc-2aeb-095d-6be1-838b7d7b33b7" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"
/dev/sde1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="c322a4fb-ec99-f835-0ce9-cb45ce6b376d" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"
As a further troubleshooting step I recently updated to mdadm 4.0
Sincerely,
David Mitchell
On Sat, Sep 2, 2017 at 12:49 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Sat, Sep 02, 2017 at 11:43:49AM -0400, David Mitchell wrote:
>> Sorry I'm still confused.
>> While the array seems alive, the filesystem seems to be missing?
>
> Hello David,
>
> I'm the one who is confused here.
>
> You posted to the raid mailing list about a superblock, and you even
> posted the mdadm command and error message to go along with it.
> How did it turn into a filesystem problem now? In your last mail,
> I did not see that at all.
>
> This was what you posted and I thought you were referring to:
>
>> # mdadm --examine /dev/md0
>> mdadm: No md superblock detected on /dev/md0.
>
> And thus that was the part I referred to in my answer.
>
> ----
>
>> # mdadm --detail /dev/md*
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Sat Aug 26 16:21:20 2017
>> Raid Level : raid5
>
> So according to this, your RAID was created about a week ago.
>
> You did write that you just migrated to 4TB drives. There are various ways
> to go about that, making a new RAID and copying stuff over from the
> old one is not unusual. So this creation time might be perfectly normal.
>
> Of course if this migration happened more than one week ago then I can
> only assume you used mdadm --create to fix some RAID issue or other?
>
> mdadm --create is like mkfs: you don't expect data to exist afterwards.
> For mdadm --create to retain data you have to get all settings perfectly
> right and those settings change depending on when you originally created
> it and what you did with it since (e.g. mdadm --grow might change offsets).
>
> I'm not sure how to help you because nothing in your mails explains
> how this issue came about.
>
> There are previous discussions on this list about botched mdadm --create,
> wrong data offsets, and searching for correct filesystem offsets.
>
> Whatever you do, don't write on these drives while looking for lost data.
>
> Regards
> Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Superblock Missing
2017-09-04 3:35 ` David Mitchell
@ 2017-09-04 15:02 ` Wols Lists
2017-09-04 15:49 ` Andreas Klauer
1 sibling, 0 replies; 9+ messages in thread
From: Wols Lists @ 2017-09-04 15:02 UTC (permalink / raw)
To: David Mitchell, linux-raid
On 04/09/17 04:35, David Mitchell wrote:
> mdadm --stop /dev/md0
> mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
>
> The array instantly loaded and went into a sync. I let the sync
> complete overnight. The file system on the raid array seems to be
> gone.
>
> I'm looking for any hints on how I might get it back.
>
> The array loads fine, but when I attempted to mount the array, it
> complains about not finding the file system
OoWwww!
I think you've just trashed one of the drives :-( The sync will have
overwritten one of the drives with parity calculated from the other two.
You've got two drives of valid data from a 3-drive raid-5 set, so the
data could be recovered, but it's going to be a long job.
Somebody else is going to have to chime in and tell you what to do, but
now is the time you really want your backup :-(
Cheers,
Wol
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Superblock Missing
2017-09-04 3:35 ` David Mitchell
2017-09-04 15:02 ` Wols Lists
@ 2017-09-04 15:49 ` Andreas Klauer
2017-09-04 17:56 ` David Mitchell
1 sibling, 1 reply; 9+ messages in thread
From: Andreas Klauer @ 2017-09-04 15:49 UTC (permalink / raw)
To: David Mitchell; +Cc: linux-raid
On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:
> I really do NOT remember running the --create command.
There is no other explanation for it. It has happened somehow.
> The pictures over 512k don't display correctly.
So not only/necessarily a wrong data offset, but also wrong drive order.
> At this point I'm hoping for help on next steps in recovery/troubleshooting.
1. Use overlays.
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
That way you can safely experiment and create RAID with
different settings. (Use --assume-clean and/or missing)
2. Check first 128M of each drive (your current data offset).
See if you can find a valid filesystem header anywhere.
That way you could determine the correct data offset.
3. Find a JPEG header (any known file type, like a 2-3M file)
and look at the other drives for the same offset. You should be
able to deduce RAID layout, chunksize, drive order from that.
Instead of 3) you can also simply trial and error with overlays
until you find a setting that allows photorec to find larger files intact.
The resync might not necessarily have damaged your data. If the offsets
were the same and the RAID level was the same, and the drives were in
sync, a resync even with wrong settings would still produce the same data.
For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage
provided you don't write anything else...
Regards
Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Superblock Missing
2017-09-04 15:49 ` Andreas Klauer
@ 2017-09-04 17:56 ` David Mitchell
2017-09-06 3:07 ` David Mitchell
0 siblings, 1 reply; 9+ messages in thread
From: David Mitchell @ 2017-09-04 17:56 UTC (permalink / raw)
To: linux-raid
@Andreas Klauer a huge THANK YOU for taking the time to review my
issue. I really appreciate your help.
Sincerely,
David Mitchell
On Mon, Sep 4, 2017 at 11:49 AM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:
>> I really do NOT remember running the --create command.
>
> There is no other explanation for it. It has happened somehow.
>
>> The pictures over 512k don't display correctly.
>
> So not only/necessarily a wrong data offset, but also wrong drive order.
>
>> At this point I'm hoping for help on next steps in recovery/troubleshooting.
>
> 1. Use overlays.
>
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> That way you can safely experiment and create RAID with
> different settings. (Use --assume-clean and/or missing)
>
> 2. Check first 128M of each drive (your current data offset).
> See if you can find a valid filesystem header anywhere.
> That way you could determine the correct data offset.
>
> 3. Find a JPEG header (any known file type, like a 2-3M file)
> and look at the other drives for the same offset. You should be
> able to deduce RAID layout, chunksize, drive order from that.
>
> Instead of 3) you can also simply trial and error with overlays
> until you find a setting that allows photorec to find larger files intact.
>
> The resync might not necessarily have damaged your data. If the offsets
> were the same and the RAID level was the same, and the drives were in
> sync, a resync even with wrong settings would still produce the same data.
> For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage
> provided you don't write anything else...
>
> Regards
> Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Superblock Missing
2017-09-04 17:56 ` David Mitchell
@ 2017-09-06 3:07 ` David Mitchell
0 siblings, 0 replies; 9+ messages in thread
From: David Mitchell @ 2017-09-06 3:07 UTC (permalink / raw)
To: linux-raid
SUCCESSFUL RECOVERY!
@Andreas Klauer you are the man! A huge THANK YOU for taking the
time to review my
issue. I really appreciate your help. While I do back up there were
some recent pictures of my children that had not been backed up yet.
Thank you!
Recreating the array with a different drive order fixed my problem,
100% recovered!
After setting up overlays per your instructions:
# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
# mdadm --assemble --force /dev/md1 /dev/mapper/sde1 /dev/mapper/sdc1
/dev/mapper/sdd1
mdadm: /dev/md1 has been started with 3 drives.
# mount /dev/md1 /mnt/raid/
mount: wrong fs type, bad option, bad superblock on /dev/md1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
unused devices: <none>
# mdadm --create --verbose --assume-clean /dev/md1 --level=5
--raid-devices=3 /dev/mapper/sde1 /dev/mapper/sdc1 /dev/mapper/sdd1
/ NOTE the order changed from C,D,E (not working) to E, C, D (working)
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/mapper/sde1 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: partition table exists on /dev/mapper/sde1 but will be lost or
meaningless after creating array
mdadm: /dev/mapper/sdc1 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: partition table exists on /dev/mapper/sdc1 but will be lost or
meaningless after creating array
mdadm: /dev/mapper/sdd1 appears to be part of a raid array:
level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: size set to 3900571648K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? Y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
# mount /dev/md1 /mnt/raid/
# ls /mnt/raid/
kvm lost+found media my-docs test tmp
# umount /mnt/raid
# fsck.ext4 -f -n /dev/md1
e2fsck 1.42.13 (17-May-2015)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md1: 224614/487571456 files (0.9% non-contiguous),
690410561/1950285824 blocks
virtual mnt # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md1 : active raid5 dm-1[2] dm-0[1] dm-2[0]
7801143296 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
bitmap: 0/30 pages [0KB], 65536KB chunk
unused devices: <none>
Sincerely,
David Mitchell
On Mon, Sep 4, 2017 at 1:56 PM, David Mitchell
<mr.david.mitchell@gmail.com> wrote:
> @Andreas Klauer a huge THANK YOU for taking the time to review my
> issue. I really appreciate your help.
>
> Sincerely,
>
> David Mitchell
>
>
> On Mon, Sep 4, 2017 at 11:49 AM, Andreas Klauer
> <Andreas.Klauer@metamorpher.de> wrote:
>> On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:
>>> I really do NOT remember running the --create command.
>>
>> There is no other explanation for it. It has happened somehow.
>>
>>> The pictures over 512k don't display correctly.
>>
>> So not only/necessarily a wrong data offset, but also wrong drive order.
>>
>>> At this point I'm hoping for help on next steps in recovery/troubleshooting.
>>
>> 1. Use overlays.
>>
>> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>>
>> That way you can safely experiment and create RAID with
>> different settings. (Use --assume-clean and/or missing)
>>
>> 2. Check first 128M of each drive (your current data offset).
>> See if you can find a valid filesystem header anywhere.
>> That way you could determine the correct data offset.
>>
>> 3. Find a JPEG header (any known file type, like a 2-3M file)
>> and look at the other drives for the same offset. You should be
>> able to deduce RAID layout, chunksize, drive order from that.
>>
>> Instead of 3) you can also simply trial and error with overlays
>> until you find a setting that allows photorec to find larger files intact.
>>
>> The resync might not necessarily have damaged your data. If the offsets
>> were the same and the RAID level was the same, and the drives were in
>> sync, a resync even with wrong settings would still produce the same data.
>> For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage
>> provided you don't write anything else...
>>
>> Regards
>> Andreas Klauer
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-09-06 3:07 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-02 0:52 Superblock Missing David Mitchell
2017-09-02 1:43 ` Andreas Klauer
2017-09-02 15:43 ` David Mitchell
2017-09-02 16:49 ` Andreas Klauer
2017-09-04 3:35 ` David Mitchell
2017-09-04 15:02 ` Wols Lists
2017-09-04 15:49 ` Andreas Klauer
2017-09-04 17:56 ` David Mitchell
2017-09-06 3:07 ` David Mitchell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.