* RAID 6, 6 device array - all devices lost superblock
@ 2022-08-28 2:00 Peter Sanders
2022-08-28 9:14 ` Wols Lists
` (2 more replies)
0 siblings, 3 replies; 29+ messages in thread
From: Peter Sanders @ 2022-08-28 2:00 UTC (permalink / raw)
To: linux-raid
have a RAID 6 array, 6 devices. Been running it for years without much issue.
Had hardware issues with my system - ended up replacing the
motherboard, video card, and power supply and re-installing the OS
(Debian 11).
As the hardware issues evolved, I'd crash, reboot, un-mount the array,
run fsck, mount and continue on my way - no problems.
After the hardware was replaced, my array will not assemble - mdadm
assemble reports no RAID superblock on the devices.
root@superior:/etc/mdadm# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sr0: No medium found
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
Examine reports
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
Searching for these results indicate I can rebuild the superblock, but
details on how to do that are lacking, at least on the pages I found.
Currently I have no /dev/md* devices.
I have access to the old mdadm.conf file - have tried assembling with
it, with the default mdadm.conf, and with no mdadm.conf file in /etc
and /etc/mdadm.
Suggestions for how to get the array back would be most appreciated.
Thanks
- Peter
Here is the data suggested from the wiki page:
root@superior:/etc/mdadm# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sr0: No medium found
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sde
mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdf
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/nvme0n1p9 (Expected magic
a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme0n1p9
mdadm: No super block found on /dev/nvme0n1p8 (Expected magic
a92b4efc, got 0000040c)
mdadm: no RAID superblock on /dev/nvme0n1p8
mdadm: No super block found on /dev/nvme0n1p7 (Expected magic
a92b4efc, got 00002004)
mdadm: no RAID superblock on /dev/nvme0n1p7
mdadm: No super block found on /dev/nvme0n1p6 (Expected magic
a92b4efc, got 0000040d)
mdadm: no RAID superblock on /dev/nvme0n1p6
mdadm: No super block found on /dev/nvme0n1p5 (Expected magic
a92b4efc, got 00000409)
mdadm: no RAID superblock on /dev/nvme0n1p5
mdadm: /dev/nvme0n1p2 is too small for md: size is 2 sectors.
mdadm: no RAID superblock on /dev/nvme0n1p2
mdadm: No super block found on /dev/nvme0n1p1 (Expected magic
a92b4efc, got 00040001)
mdadm: no RAID superblock on /dev/nvme0n1p1
mdadm: No super block found on /dev/nvme0n1 (Expected magic a92b4efc,
got 7a78e8ed)
mdadm: no RAID superblock on /dev/nvme0n1
root@superior:/etc/mdadm#
uname -a
Linux superior 5.10.0-17-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13)
x86_64 GNU/Linux
mdadm --version
mdadm - v4.1 - 2018-10-01
smartctl devices ------------
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba P300
Device Model: TOSHIBA HDWD130
Serial Number: 477ALBNAS
LU WWN Device Id: 5 000039 fe6d2e832
Firmware Version: MX6OACF0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:49 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (21791) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 364) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0
2 Throughput_Performance P-S--- 141 141 054 - 66
3 Spin_Up_Time POS--- 160 160 024 - 361 (Average 357)
4 Start_Stop_Count -O--C- 100 100 000 - 204
5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0
7 Seek_Error_Rate PO-R-- 100 100 067 - 0
8 Seek_Time_Performance P-S--- 124 124 020 - 33
9 Power_On_Hours -O--C- 095 095 000 - 41740
10 Spin_Retry_Count PO--C- 100 100 060 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 204
192 Power-Off_Retract_Count -O--CK 100 100 000 - 759
193 Load_Cycle_Count -O--C- 100 100 000 - 759
194 Temperature_Celsius -O---- 181 181 000 - 33 (Min/Max 20/50)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 7 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x20 GPL R/O 1 Streaming performance log [OBS-8]
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
Device State: Active (0)
Current Temperature: 33 Celsius
Power Cycle Min/Max Temperature: 29/33 Celsius
Lifetime Min/Max Temperature: 20/50 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (51)
Index Estimated Time Temperature Celsius
52 2022-08-25 19:12 33 **************
... ..( 76 skipped). .. **************
1 2022-08-25 20:29 33 **************
2 2022-08-25 20:30 ? -
3 2022-08-25 20:31 33 **************
4 2022-08-25 20:32 34 ***************
5 2022-08-25 20:33 33 **************
6 2022-08-25 20:34 34 ***************
... ..( 2 skipped). .. ***************
9 2022-08-25 20:37 34 ***************
10 2022-08-25 20:38 ? -
11 2022-08-25 20:39 29 **********
12 2022-08-25 20:40 30 ***********
... ..( 2 skipped). .. ***********
15 2022-08-25 20:43 30 ***********
16 2022-08-25 20:44 31 ************
... ..( 3 skipped). .. ************
20 2022-08-25 20:48 31 ************
21 2022-08-25 20:49 32 *************
... ..( 9 skipped). .. *************
31 2022-08-25 20:59 32 *************
32 2022-08-25 21:00 33 **************
... ..( 18 skipped). .. **************
51 2022-08-25 21:19 33 **************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 204 --- Lifetime Power-On Resets
0x01 0x010 4 41740 --- Power-on Hours
0x01 0x018 6 20304278904 --- Logical Sectors Written
0x01 0x020 6 64656942 --- Number of Write Commands
0x01 0x028 6 350269182084 --- Logical Sectors Read
0x01 0x030 6 481405773 --- Number of Read Commands
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 41734 --- Spindle Motor Power-on Hours
0x03 0x010 4 41734 --- Head Flying Hours
0x03 0x018 4 759 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 22 --- Read Recovery Attempts
0x03 0x030 4 6 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 1 --- Resets Between Cmd Acceptance and
Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 33 --- Current Temperature
0x05 0x010 1 33 N-- Average Short Term Temperature
0x05 0x018 1 37 N-- Average Long Term Temperature
0x05 0x020 1 50 --- Highest Temperature
0x05 0x028 1 20 --- Lowest Temperature
0x05 0x030 1 46 N-- Highest Average Short Term Temperature
0x05 0x038 1 25 N-- Lowest Average Short Term Temperature
0x05 0x040 1 43 N-- Highest Average Long Term Temperature
0x05 0x048 1 25 N-- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 1006 --- Number of Hardware Resets
0x06 0x010 4 494 --- Number of ASR Events
0x06 0x018 4 0 --- Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0009 2 35 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 5 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Green
Device Model: WDC WD30EZRX-00DC0B0
Serial Number: WD-WCC1T0668790
LU WWN Device Id: 5 0014ee 2084d406a
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:51 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (40560) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 407) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70b5) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 181 178 021 - 5916
4 Start_Stop_Count -O--CK 100 100 000 - 377
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 007 007 000 - 68295
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 296
192 Power-Off_Retract_Count -O--CK 200 200 000 - 242
193 Load_Cycle_Count -O--CK 052 052 000 - 445057
194 Temperature_Celsius -O---K 121 102 000 - 29
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 7 -
# 2 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 29 Celsius
Power Cycle Min/Max Temperature: 28/29 Celsius
Lifetime Min/Max Temperature: 2/48 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (138)
Index Estimated Time Temperature Celsius
139 2022-08-25 13:22 30 ***********
140 2022-08-25 13:23 29 **********
... ..( 5 skipped). .. **********
146 2022-08-25 13:29 29 **********
147 2022-08-25 13:30 ? -
148 2022-08-25 13:31 26 *******
149 2022-08-25 13:32 ? -
150 2022-08-25 13:33 28 *********
151 2022-08-25 13:34 ? -
152 2022-08-25 13:35 28 *********
153 2022-08-25 13:36 28 *********
154 2022-08-25 13:37 29 **********
... ..( 55 skipped). .. **********
210 2022-08-25 14:33 29 **********
211 2022-08-25 14:34 30 ***********
... ..( 11 skipped). .. ***********
223 2022-08-25 14:46 30 ***********
224 2022-08-25 14:47 29 **********
... ..(103 skipped). .. **********
328 2022-08-25 16:31 29 **********
329 2022-08-25 16:32 30 ***********
... ..( 18 skipped). .. ***********
348 2022-08-25 16:51 30 ***********
349 2022-08-25 16:52 29 **********
... ..( 33 skipped). .. **********
383 2022-08-25 17:26 29 **********
384 2022-08-25 17:27 30 ***********
... ..( 10 skipped). .. ***********
395 2022-08-25 17:38 30 ***********
396 2022-08-25 17:39 29 **********
... ..(218 skipped). .. **********
137 2022-08-25 21:18 29 **********
138 2022-08-25 21:19 ? -
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 305 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 5 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 2491 Vendor specific
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba P300
Device Model: TOSHIBA HDWD130
Serial Number: Y7211KPAS
LU WWN Device Id: 5 000039 fe6dca946
Firmware Version: MX6OACF0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:51 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (21791) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 364) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0
2 Throughput_Performance P-S--- 139 139 054 - 71
3 Spin_Up_Time POS--- 160 160 024 - 361 (Average 355)
4 Start_Stop_Count -O--C- 100 100 000 - 189
5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0
7 Seek_Error_Rate PO-R-- 100 100 067 - 0
8 Seek_Time_Performance P-S--- 128 128 020 - 31
9 Power_On_Hours -O--C- 095 095 000 - 35428
10 Spin_Retry_Count PO--C- 100 100 060 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 189
192 Power-Off_Retract_Count -O--CK 100 100 000 - 599
193 Load_Cycle_Count -O--C- 100 100 000 - 599
194 Temperature_Celsius -O---- 176 176 000 - 34 (Min/Max 19/50)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 7 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x20 GPL R/O 1 Streaming performance log [OBS-8]
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
Device State: Active (0)
Current Temperature: 34 Celsius
Power Cycle Min/Max Temperature: 28/34 Celsius
Lifetime Min/Max Temperature: 19/50 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (15)
Index Estimated Time Temperature Celsius
16 2022-08-25 19:12 33 **************
... ..( 66 skipped). .. **************
83 2022-08-25 20:19 33 **************
84 2022-08-25 20:20 34 ***************
... ..( 8 skipped). .. ***************
93 2022-08-25 20:29 34 ***************
94 2022-08-25 20:30 ? -
95 2022-08-25 20:31 34 ***************
... ..( 5 skipped). .. ***************
101 2022-08-25 20:37 34 ***************
102 2022-08-25 20:38 ? -
103 2022-08-25 20:39 29 **********
104 2022-08-25 20:40 29 **********
105 2022-08-25 20:41 30 ***********
106 2022-08-25 20:42 30 ***********
107 2022-08-25 20:43 31 ************
... ..( 3 skipped). .. ************
111 2022-08-25 20:47 31 ************
112 2022-08-25 20:48 32 *************
... ..( 4 skipped). .. *************
117 2022-08-25 20:53 32 *************
118 2022-08-25 20:54 33 **************
... ..( 15 skipped). .. **************
6 2022-08-25 21:10 33 **************
7 2022-08-25 21:11 34 ***************
8 2022-08-25 21:12 33 **************
9 2022-08-25 21:13 34 ***************
... ..( 5 skipped). .. ***************
15 2022-08-25 21:19 34 ***************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 189 --- Lifetime Power-On Resets
0x01 0x010 4 35428 --- Power-on Hours
0x01 0x018 6 12728825059 --- Logical Sectors Written
0x01 0x020 6 36220308 --- Number of Write Commands
0x01 0x028 6 289884223915 --- Logical Sectors Read
0x01 0x030 6 321688917 --- Number of Read Commands
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 35423 --- Spindle Motor Power-on Hours
0x03 0x010 4 35423 --- Head Flying Hours
0x03 0x018 4 599 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 7 --- Read Recovery Attempts
0x03 0x030 4 6 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 1 --- Resets Between Cmd Acceptance and
Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 34 --- Current Temperature
0x05 0x010 1 34 N-- Average Short Term Temperature
0x05 0x018 1 37 N-- Average Long Term Temperature
0x05 0x020 1 50 --- Highest Temperature
0x05 0x028 1 19 --- Lowest Temperature
0x05 0x030 1 46 N-- Highest Average Short Term Temperature
0x05 0x038 1 25 N-- Lowest Average Short Term Temperature
0x05 0x040 1 43 N-- Highest Average Long Term Temperature
0x05 0x048 1 25 N-- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 147297 --- Number of Hardware Resets
0x06 0x010 4 8793 --- Number of ASR Events
0x06 0x018 4 0 --- Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0009 2 29 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 5 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Green
Device Model: WDC WD30EZRX-00D8PB0
Serial Number: WD-WCC4N0091255
LU WWN Device Id: 5 0014ee 2b3d4ffa1
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:53 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (42480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 426) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 2
3 Spin_Up_Time POS--K 184 181 021 - 5783
4 Start_Stop_Count -O--CK 100 100 000 - 275
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 039 039 000 - 44593
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 273
192 Power-Off_Retract_Count -O--CK 200 200 000 - 225
193 Load_Cycle_Count -O--CK 047 047 000 - 461100
194 Temperature_Celsius -O---K 122 105 000 - 28
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 28 Celsius
Power Cycle Min/Max Temperature: 27/28 Celsius
Lifetime Min/Max Temperature: 2/44 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (444)
Index Estimated Time Temperature Celsius
445 2022-08-25 13:22 29 **********
... ..( 33 skipped). .. **********
1 2022-08-25 13:56 29 **********
2 2022-08-25 13:57 ? -
3 2022-08-25 13:58 29 **********
... ..( 6 skipped). .. **********
10 2022-08-25 14:05 29 **********
11 2022-08-25 14:06 ? -
12 2022-08-25 14:07 26 *******
13 2022-08-25 14:08 ? -
14 2022-08-25 14:09 27 ********
15 2022-08-25 14:10 27 ********
16 2022-08-25 14:11 28 *********
... ..( 37 skipped). .. *********
54 2022-08-25 14:49 28 *********
55 2022-08-25 14:50 29 **********
... ..(388 skipped). .. **********
444 2022-08-25 21:19 29 **********
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 286 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 5 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 2493 Vendor specific
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Green
Device Model: WDC WD30EZRX-00MMMB0
Serial Number: WD-WCAWZ2669166
LU WWN Device Id: 5 0014ee 15a13d994
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:53 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (50160) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 482) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 153 138 021 - 9350
4 Start_Stop_Count -O--CK 100 100 000 - 297
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 040 040 000 - 44409
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 268
192 Power-Off_Retract_Count -O--CK 200 200 000 - 218
193 Load_Cycle_Count -O--CK 001 001 000 - 1082082
194 Temperature_Celsius -O---K 122 105 000 - 30
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 1
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
Device State: Active (0)
Current Temperature: 30 Celsius
Power Cycle Min/Max Temperature: 27/30 Celsius
Lifetime Min/Max Temperature: 0/47 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (88)
Index Estimated Time Temperature Celsius
89 2022-08-25 13:22 30 ***********
... ..( 33 skipped). .. ***********
123 2022-08-25 13:56 30 ***********
124 2022-08-25 13:57 ? -
125 2022-08-25 13:58 31 ************
126 2022-08-25 13:59 30 ***********
... ..( 5 skipped). .. ***********
132 2022-08-25 14:05 30 ***********
133 2022-08-25 14:06 ? -
134 2022-08-25 14:07 26 *******
135 2022-08-25 14:08 ? -
136 2022-08-25 14:09 27 ********
... ..( 3 skipped). .. ********
140 2022-08-25 14:13 27 ********
141 2022-08-25 14:14 28 *********
... ..( 3 skipped). .. *********
145 2022-08-25 14:18 28 *********
146 2022-08-25 14:19 29 **********
... ..( 13 skipped). .. **********
160 2022-08-25 14:33 29 **********
161 2022-08-25 14:34 30 ***********
... ..( 43 skipped). .. ***********
205 2022-08-25 15:18 30 ***********
206 2022-08-25 15:19 31 ************
207 2022-08-25 15:20 30 ***********
... ..(168 skipped). .. ***********
376 2022-08-25 18:09 30 ***********
377 2022-08-25 18:10 31 ************
378 2022-08-25 18:11 30 ***********
... ..( 34 skipped). .. ***********
413 2022-08-25 18:46 30 ***********
414 2022-08-25 18:47 31 ************
415 2022-08-25 18:48 30 ***********
... ..( 7 skipped). .. ***********
423 2022-08-25 18:56 30 ***********
424 2022-08-25 18:57 31 ************
425 2022-08-25 18:58 30 ***********
... ..( 7 skipped). .. ***********
433 2022-08-25 19:06 30 ***********
434 2022-08-25 19:07 31 ************
435 2022-08-25 19:08 30 ***********
... ..( 47 skipped). .. ***********
5 2022-08-25 19:56 30 ***********
6 2022-08-25 19:57 31 ************
7 2022-08-25 19:58 30 ***********
... ..( 80 skipped). .. ***********
88 2022-08-25 21:19 30 ***********
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x8000 4 2492 Vendor specific
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba P300
Device Model: TOSHIBA HDWD130
Serial Number: 477ABEJAS
LU WWN Device Id: 5 000039 fe6d2ce25
Firmware Version: MX6OACF0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Aug 25 21:19:53 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (23082) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 385) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0
2 Throughput_Performance P-S--- 140 140 054 - 68
3 Spin_Up_Time POS--- 161 161 024 - 358 (Average 354)
4 Start_Stop_Count -O--C- 100 100 000 - 243
5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0
7 Seek_Error_Rate PO-R-- 100 100 067 - 0
8 Seek_Time_Performance P-S--- 126 126 020 - 32
9 Power_On_Hours -O--C- 094 094 000 - 44046
10 Spin_Retry_Count PO--C- 100 100 060 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 243
192 Power-Off_Retract_Count -O--CK 100 100 000 - 912
193 Load_Cycle_Count -O--C- 100 100 000 - 912
194 Temperature_Celsius -O---- 193 193 000 - 31 (Min/Max 19/46)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 7 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x20 GPL R/O 1 Streaming performance log [OBS-8]
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
Device State: Active (0)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 28/32 Celsius
Lifetime Min/Max Temperature: 19/46 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (117)
Index Estimated Time Temperature Celsius
118 2022-08-25 19:12 31 ************
... ..( 76 skipped). .. ************
67 2022-08-25 20:29 31 ************
68 2022-08-25 20:30 ? -
69 2022-08-25 20:31 31 ************
70 2022-08-25 20:32 32 *************
71 2022-08-25 20:33 31 ************
72 2022-08-25 20:34 31 ************
73 2022-08-25 20:35 32 *************
74 2022-08-25 20:36 32 *************
75 2022-08-25 20:37 32 *************
76 2022-08-25 20:38 ? -
77 2022-08-25 20:39 28 *********
78 2022-08-25 20:40 29 **********
... ..( 2 skipped). .. **********
81 2022-08-25 20:43 29 **********
82 2022-08-25 20:44 30 ***********
... ..( 5 skipped). .. ***********
88 2022-08-25 20:50 30 ***********
89 2022-08-25 20:51 31 ************
... ..( 27 skipped). .. ************
117 2022-08-25 21:19 31 ************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 243 --- Lifetime Power-On Resets
0x01 0x010 4 44046 --- Power-on Hours
0x01 0x018 6 27756962802 --- Logical Sectors Written
0x01 0x020 6 86355955 --- Number of Write Commands
0x01 0x028 6 381193626849 --- Logical Sectors Read
0x01 0x030 6 791200694 --- Number of Read Commands
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 44040 --- Spindle Motor Power-on Hours
0x03 0x010 4 44040 --- Head Flying Hours
0x03 0x018 4 912 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 0 --- Read Recovery Attempts
0x03 0x030 4 6 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 0 --- Resets Between Cmd Acceptance and
Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 32 --- Current Temperature
0x05 0x010 1 31 N-- Average Short Term Temperature
0x05 0x018 1 35 N-- Average Long Term Temperature
0x05 0x020 1 46 --- Highest Temperature
0x05 0x028 1 19 --- Lowest Temperature
0x05 0x030 1 43 N-- Highest Average Short Term Temperature
0x05 0x038 1 25 N-- Lowest Average Short Term Temperature
0x05 0x040 1 41 N-- Highest Average Long Term Temperature
0x05 0x048 1 25 N-- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 4706 --- Number of Hardware Resets
0x06 0x010 4 3910 --- Number of ASR Events
0x06 0x018 4 0 --- Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0009 2 29 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 5 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
mdadm --examine devices -----
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdf:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm --detail /dev/md0 ------
lsdrv ------------------------
PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics
Corporation E12 NVMe Controller (rev 01)
└nvme nvme0 PCIe SSD {21112925606047}
└nvme0n1 238.47g [259:0] Partitioned (dos)
├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e}
│└Mounted as /dev/nvme0n1p1 @ /boot
├nvme0n1p2 1.00k [259:2] Partitioned (dos)
├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b}
│└Mounted as /dev/nvme0n1p5 @ /
├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff}
│└Mounted as /dev/nvme0n1p6 @ /var
├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee}
├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9}
│└Mounted as /dev/nvme0n1p8 @ /home
└nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c}
PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp.
JMB368 IDE controller (rev 10)
└scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M}
└sr0 1.00g [11:0] Empty/Unknown
PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363
SATA/IDE Controller (rev 10)
└scsi 2:x:x:x [Empty]
PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD]
FCH SATA Controller [AHCI mode] (rev 51)
├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS}
│└sda 2.73t [8:0] Partitioned (PMBR)
└scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS}
└sdc 2.73t [8:32] Partitioned (gpt)
PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD]
FCH SATA Controller [AHCI mode] (rev 51)
├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790}
│└sdb 2.73t [8:16] Partitioned (gpt)
├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255}
│└sdd 2.73t [8:48] Partitioned (gpt)
├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166}
│└sde 2.73t [8:64] Partitioned (gpt)
└scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS}
└sdf 2.73t [8:80] Partitioned (gpt)
cat /proc/mdstat -------------
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
unused devices: <none>
cat /etc/mdadm/mdadm.conf ----
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=109fa7b0:cf08fdba:e36284a9:5786ffff
name=superior:0
# This configuration was auto-generated on Sun, 26 Dec 2021 13:31:14
-0500 by mkconf
cat /proc/partitions ---------
major minor #blocks name
259 0 250059096 nvme0n1
259 1 496640 nvme0n1p1
259 2 1 nvme0n1p2
259 3 63475712 nvme0n1p5
259 4 97654784 nvme0n1p6
259 5 37888 nvme0n1p7
259 6 86913024 nvme0n1p8
259 7 1474560 nvme0n1p9
8 32 2930266584 sdc
8 80 2930266584 sdf
8 64 2930266584 sde
8 48 2930266584 sdd
8 16 2930266584 sdb
8 0 2930266584 sda
11 0 1048575 sr0
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 2:00 RAID 6, 6 device array - all devices lost superblock Peter Sanders @ 2022-08-28 9:14 ` Wols Lists 2022-08-28 9:54 ` Wols Lists 2022-08-28 15:10 ` John Stoffel 2022-08-28 17:11 ` Andy Smith 2 siblings, 1 reply; 29+ messages in thread From: Wols Lists @ 2022-08-28 9:14 UTC (permalink / raw) To: Peter Sanders, linux-raid; +Cc: Phil Turmel, NeilBrown On 28/08/2022 03:00, Peter Sanders wrote: > have a RAID 6 array, 6 devices. Been running it for years without much issue. > > Had hardware issues with my system - ended up replacing the > motherboard, video card, and power supply and re-installing the OS > (Debian 11). > > As the hardware issues evolved, I'd crash, reboot, un-mount the array, > run fsck, mount and continue on my way - no problems. > > After the hardware was replaced, my array will not assemble - mdadm > assemble reports no RAID superblock on the devices. > root@superior:/etc/mdadm# mdadm --assemble --scan --verbose > mdadm: looking for devices for /dev/md/0 > mdadm: cannot open device /dev/sr0: No medium found > mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000) > mdadm: no RAID superblock on /dev/sda > mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000) > mdadm: no RAID superblock on /dev/sdb > > Examine reports > /dev/sda: > MBR Magic : aa55 > Partition[0] : 4294967295 sectors at 1 (type ee) > > Searching for these results indicate I can rebuild the superblock, but > details on how to do that are lacking, at least on the pages I found. Ouch. That's not nice, but we should be able to get things back, I hope. I notice it's looking for your superblock on the drive itself. Were your drives partitioned? Because unfortunately, it's well known for drives moving between hardware to have their MBR/GPT wiped :-( Hopefully that's the case, and examining the drives with gdisk/fdisk will come up with "your GPT is damaged. Recover?". If so, you're probably good. If not, do you have a record of your partitions? Can you just recreate them? If you don't know what you're doing here, I'd wait for a bit more advice unless you can back the drives up first. Whatever happens, do you have a backup? Can you make one? If your drives were NOT partitioned, then I'm afraid we're into forensics here. Read up on overlays, so you can make the drives read-only, re-create the superblock, and check if you got it right. I've not done this myself, so I would hesitate to advise you, but loads of people have said the instructions do work, and they've recovered their arrays. > > Currently I have no /dev/md* devices. > I have access to the old mdadm.conf file - have tried assembling with > it, with the default mdadm.conf, and with no mdadm.conf file in /etc > and /etc/mdadm. It looks like the drives weren't partitioned :-( I think you're into forensics. > > Suggestions for how to get the array back would be most appreciated. Not what you're asking for, but another suggestion - DITCH THOSE DRIVES. WD Greens are just plain unsuitable for raid, and if your P300s are new, they are too :-( (Greens will be damaged as their optimisation is completely wrong for raid, the new P300s are SMR) I notice that ERC is disabled ... I'd get 4x6TB N300s or Seagate Ironwolves (if cost is an issue, you can get away with two). If you do get four, swap out two greens, and rebuild onto the 6TBs. If you can only afford two, swap out two P300s, raid-0 them and rebuild as raid-5 onto the 6TB/P300 drives, then you CAN go raid-6 raid-0ing a green and your last P300 together. Just get rid of the greens asap, and the P300s after. > > Thanks > - Peter > Cheers, Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 9:14 ` Wols Lists @ 2022-08-28 9:54 ` Wols Lists 2022-08-28 16:47 ` Phil Turmel 0 siblings, 1 reply; 29+ messages in thread From: Wols Lists @ 2022-08-28 9:54 UTC (permalink / raw) To: Peter Sanders, linux-raid; +Cc: Phil Turmel, NeilBrown On 28/08/2022 10:14, Wols Lists wrote: >> Currently I have no /dev/md* devices. >> I have access to the old mdadm.conf file - have tried assembling with >> it, with the default mdadm.conf, and with no mdadm.conf file in /etc >> and /etc/mdadm. > > It looks like the drives weren't partitioned :-( I think you're into > forensics. Whoops - my system froze while I was originally writing my reply, and I forgot to put this into my rewrite ... Look up overlays in the wiki. I've never done it myself, but a fair few people have said the instructions worked a treat. You're basically making the drives read-only (all writes get dumped into the overlay file), and then re-creating the array over the top, so you can test whether you got it right. If you don't, you just ditch the overlays and start again, if you did get it right you can recreate the array for real. Cheers, Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 9:54 ` Wols Lists @ 2022-08-28 16:47 ` Phil Turmel [not found] ` <CAKAPSkJAQYsec-4zzcePbkJ7Ee0=sd_QvHj4Stnyineq+T8BXw@mail.gmail.com> 0 siblings, 1 reply; 29+ messages in thread From: Phil Turmel @ 2022-08-28 16:47 UTC (permalink / raw) To: Wols Lists, Peter Sanders, John Stoffel; +Cc: NeilBrown, linux-raid Hi Peter, et al, On 8/28/22 05:54, Wols Lists wrote: > On 28/08/2022 10:14, Wols Lists wrote: >>> Currently I have no /dev/md* devices. >>> I have access to the old mdadm.conf file - have tried assembling with >>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc >>> and /etc/mdadm. >> >> It looks like the drives weren't partitioned :-( I think you're into >> forensics. It is too soon to say this. The supplied mdadm.conf file does not contain specific partition information. It is possible the partition tables have just been wiped. > Whoops - my system froze while I was originally writing my reply, and I > forgot to put this into my rewrite ... > > Look up overlays in the wiki. I've never done it myself, but a fair few > people have said the instructions worked a treat. > > You're basically making the drives read-only (all writes get dumped into > the overlay file), and then re-creating the array over the top, so you > can test whether you got it right. If you don't, you just ditch the > overlays and start again, if you did get it right you can recreate the > array for real. > > Cheers, > Wol On 8/28/22 11:10, John Stoffel wrote: >>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. > Peter> Had hardware issues with my system - ended up replacing the > Peter> motherboard, video card, and power supply and re-installing the OS > Peter> (Debian 11). > > Can you give us details on the old vs new motherboard/cpu? It might > be that you need to tweak the BIOS of the motherboard to expose the > old SATA formats as well. > > Did you install debian onto a fresh boot disk? Is your BIOS setup to > only do the new form of booting from UEFI devices, so maybe check your > BIOS settings that the data drives are all in AHCI mode, or possibly > even in IDE mode. It all depends on how old the original hardware > was. > > I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak > the BIOS defaults to see my disks. I guess I should do a clean > install from a blank disk, but I wanted to minimize downtime. It is important to end up in AHCI mode on all MOBO ports. If not set that way now, please change them. > Wols has some great advice here, and I heartily recommend that you use > overlayfs when doing your testing. Check the RAID WIKI for > suggestions. Concur. > And don't panic! Your data is probably there, but just missing the > super blocks or partition tables. Both, I suspect. On 8/27/22 22:00, Peter Sanders wrote: > lsdrv ------------------------ > PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics > Corporation E12 NVMe Controller (rev 01) > └nvme nvme0 PCIe SSD {21112925606047} > └nvme0n1 238.47g [259:0] Partitioned (dos) > ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} > │└Mounted as /dev/nvme0n1p1 @ /boot > ├nvme0n1p2 1.00k [259:2] Partitioned (dos) > ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} > │└Mounted as /dev/nvme0n1p5 @ / > ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} > │└Mounted as /dev/nvme0n1p6 @ /var > ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} > ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} > │└Mounted as /dev/nvme0n1p8 @ /home > └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} > PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. > JMB368 IDE controller (rev 10) > └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} > └sr0 1.00g [11:0] Empty/Unknown > PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 > SATA/IDE Controller (rev 10) > └scsi 2:x:x:x [Empty] > PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > FCH SATA Controller [AHCI mode] (rev 51) > ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} > │└sda 2.73t [8:0] Partitioned (PMBR) > └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} > └sdc 2.73t [8:32] Partitioned (gpt) > PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > FCH SATA Controller [AHCI mode] (rev 51) > ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} > │└sdb 2.73t [8:16] Partitioned (gpt) > ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} > │└sdd 2.73t [8:48] Partitioned (gpt) > ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} > │└sde 2.73t [8:64] Partitioned (gpt) > └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} > └sdf 2.73t [8:80] Partitioned (gpt) Unfortunately, my lsdrv tool is not able to reconstruct missing parts. It is most useful when used on a *good* system and *saved* for help diagnosing *future* problems. Please share your /etc/fstab, and if you were using LVM on top of the raid, share your lvm.conf and anything in /etc/lvm/backup. Please describe the layer(s) that were on top of the raid. We need to help you look for signatures, and it helps to be selective in what signatures to look for. After that, we will want to figure out your raid's chunk size and data offsets. If you know of a particular large file (8MB or larger) that is sure to be in the raid and you happen to have a copy tucked away, then my findHash[1] tool might be able to definitively determine those values. (Time consuming, though.) Meanwhile, don't do *anything* that would write to those drives. Phil [1] https://github.com/pturmel/findHash ^ permalink raw reply [flat|nested] 29+ messages in thread
[parent not found: <CAKAPSkJAQYsec-4zzcePbkJ7Ee0=sd_QvHj4Stnyineq+T8BXw@mail.gmail.com>]
* Re: RAID 6, 6 device array - all devices lost superblock [not found] ` <CAKAPSkJAQYsec-4zzcePbkJ7Ee0=sd_QvHj4Stnyineq+T8BXw@mail.gmail.com> @ 2022-08-28 17:16 ` Wols Lists 2022-08-28 18:45 ` John Stoffel 1 sibling, 0 replies; 29+ messages in thread From: Wols Lists @ 2022-08-28 17:16 UTC (permalink / raw) To: Peter Sanders, Phil Turmel; +Cc: John Stoffel, NeilBrown, linux-raid On 28/08/2022 18:01, Peter Sanders wrote: > It was set up on the device level, not partitions. (I remember getting > some advice on the web that device was better than partition... Yay for > internet advice) Well, it really SHOULDN'T matter. Except there's plenty of crap software that says "ooh a disk with no partition table - it must be empty - let's initialise it without asking the user whether it's okay". Or, as in your case, it seems like your mobo has wiped the start of the disk for some reason. We now recommend partitions, not because it's better, but as a defensive mechanism against the idiots out there ... :-( > > I'm surveying my other disks to see what I have available to do the > overlay attempt. > > What are the size of the overlay files going to end up being? I'm not sure what the recommendation is, I think it used to be about 10%, but I think you'll get away with much less. If you have the space ELSEWHERE eg your system partition, try and allow about 1% ie 60GB per drive. So you want about 360GB of free space if possible. I don't think it's dangerous if you don't allow enough space - you'll just hit a "disk full" on your overlays which will be a frustration but not a disaster. So just give it as much room as you can afford. > > I did run into UEFI vs AHCI issues early in the process.. they are all > set to non-UEFI. > > OS update was onto a new SSD... > Cheers, Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock [not found] ` <CAKAPSkJAQYsec-4zzcePbkJ7Ee0=sd_QvHj4Stnyineq+T8BXw@mail.gmail.com> 2022-08-28 17:16 ` Wols Lists @ 2022-08-28 18:45 ` John Stoffel 2022-08-28 19:36 ` Phil Turmel 1 sibling, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-08-28 18:45 UTC (permalink / raw) To: Peter Sanders Cc: Phil Turmel, Wols Lists, John Stoffel, NeilBrown, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: Peter> It was set up on the device level, not partitions. (I remember Peter> getting some advice on the web that device was better than Peter> partition... Yay for internet advice) Yeah, this is NOT good advice. Generally systems will not destroy existing partition tables, but if they see an empty (to them) disk... all bets are off. Peter> I'm surveying my other disks to see what I have available to do Peter> the overlay attempt. They're small. They are sparse files, so just follow the instructions. Peter> What are the size of the overlay files going to end up being? Not too large, but it depends on how much data is written to the overlayfs to get your data back. If you follow the instructions on this page: https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID It says to create a sparse file for each disk that is 1% of the size of the disk. This can add up... you might need to add a blank disk to your system to hold these. In this case, if you think you know which order the disks were in, you could try to create the RAID6 array (but only using the overlayfs devices!!!!!) I can't stress this enough. Peter> I did run into UEFI vs AHCI issues early in the process.. they Peter> are all set to non-UEFI. That's good. Peter> OS update was onto a new SSD... Ok. Do you have the old OS disk around by any chance? That might give some pointers to how the disks are setup.. You could look in /var/tmp/initrd/... for old mdadm.conf files, which might give more details. Peter> On Sun, Aug 28, 2022, 12:47 Phil Turmel <philip@turmel.org> wrote: Peter> Hi Peter, et al, Peter> On 8/28/22 05:54, Wols Lists wrote: >> On 28/08/2022 10:14, Wols Lists wrote: >>>> Currently I have no /dev/md* devices. >>>> I have access to the old mdadm.conf file - have tried assembling with >>>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc >>>> and /etc/mdadm. >>> >>> It looks like the drives weren't partitioned :-( I think you're into >>> forensics. Peter> It is too soon to say this. The supplied mdadm.conf file does not Peter> contain specific partition information. It is possible the partition Peter> tables have just been wiped. >> Whoops - my system froze while I was originally writing my reply, and I >> forgot to put this into my rewrite ... >> >> Look up overlays in the wiki. I've never done it myself, but a fair few >> people have said the instructions worked a treat. >> >> You're basically making the drives read-only (all writes get dumped into >> the overlay file), and then re-creating the array over the top, so you >> can test whether you got it right. If you don't, you just ditch the >> overlays and start again, if you did get it right you can recreate the >> array for real. >> >> Cheers, >> Wol Peter> On 8/28/22 11:10, John Stoffel wrote: >>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. Peter> Had hardware issues with my system - ended up replacing the Peter> motherboard, video card, and power supply and re-installing the OS Peter> (Debian 11). >> >> Can you give us details on the old vs new motherboard/cpu? It might >> be that you need to tweak the BIOS of the motherboard to expose the >> old SATA formats as well. >> >> Did you install debian onto a fresh boot disk? Is your BIOS setup to >> only do the new form of booting from UEFI devices, so maybe check your >> BIOS settings that the data drives are all in AHCI mode, or possibly >> even in IDE mode. It all depends on how old the original hardware >> was. >> >> I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak >> the BIOS defaults to see my disks. I guess I should do a clean >> install from a blank disk, but I wanted to minimize downtime. Peter> It is important to end up in AHCI mode on all MOBO ports. If not set Peter> that way now, please change them. >> Wols has some great advice here, and I heartily recommend that you use >> overlayfs when doing your testing. Check the RAID WIKI for >> suggestions. Peter> Concur. >> And don't panic! Your data is probably there, but just missing the >> super blocks or partition tables. Peter> Both, I suspect. Peter> On 8/27/22 22:00, Peter Sanders wrote: >> lsdrv ------------------------ >> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics >> Corporation E12 NVMe Controller (rev 01) >> └nvme nvme0 PCIe SSD {21112925606047} >> └nvme0n1 238.47g [259:0] Partitioned (dos) >> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} >> │└Mounted as /dev/nvme0n1p1 @ /boot >> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) >> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} >> │└Mounted as /dev/nvme0n1p5 @ / >> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} >> │└Mounted as /dev/nvme0n1p6 @ /var >> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} >> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} >> │└Mounted as /dev/nvme0n1p8 @ /home >> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} >> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. >> JMB368 IDE controller (rev 10) >> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} >> └sr0 1.00g [11:0] Empty/Unknown >> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 >> SATA/IDE Controller (rev 10) >> └scsi 2:x:x:x [Empty] >> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >> FCH SATA Controller [AHCI mode] (rev 51) >> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} >> │└sda 2.73t [8:0] Partitioned (PMBR) >> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} >> └sdc 2.73t [8:32] Partitioned (gpt) >> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >> FCH SATA Controller [AHCI mode] (rev 51) >> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} >> │└sdb 2.73t [8:16] Partitioned (gpt) >> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} >> │└sdd 2.73t [8:48] Partitioned (gpt) >> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} >> │└sde 2.73t [8:64] Partitioned (gpt) >> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} >> └sdf 2.73t [8:80] Partitioned (gpt) Peter> Unfortunately, my lsdrv tool is not able to reconstruct missing parts. Peter> It is most useful when used on a *good* system and *saved* for help Peter> diagnosing *future* problems. Peter> Please share your /etc/fstab, and if you were using LVM on top of the Peter> raid, share your lvm.conf and anything in /etc/lvm/backup. Peter> Please describe the layer(s) that were on top of the raid. Peter> We need to help you look for signatures, and it helps to be selective in Peter> what signatures to look for. Peter> After that, we will want to figure out your raid's chunk size and data Peter> offsets. If you know of a particular large file (8MB or larger) that is Peter> sure to be in the raid and you happen to have a copy tucked away, then Peter> my findHash[1] tool might be able to definitively determine those Peter> values. (Time consuming, though.) Peter> Meanwhile, don't do *anything* that would write to those drives. Peter> Phil Peter> [1] https://github.com/pturmel/findHash ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 18:45 ` John Stoffel @ 2022-08-28 19:36 ` Phil Turmel 2022-08-28 19:49 ` John Stoffel 0 siblings, 1 reply; 29+ messages in thread From: Phil Turmel @ 2022-08-28 19:36 UTC (permalink / raw) To: John Stoffel, Peter Sanders; +Cc: Wols Lists, NeilBrown, linux-raid Pssst! John, All of my comments were attributed to Peter by your mail client. ): On 8/28/22 14:45, John Stoffel wrote: >>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > Peter> It was set up on the device level, not partitions. (I remember > Peter> getting some advice on the web that device was better than > Peter> partition... Yay for internet advice) > > Yeah, this is NOT good advice. Generally systems will not destroy > existing partition tables, but if they see an empty (to them) > disk... all bets are off. > > Peter> I'm surveying my other disks to see what I have available to do > Peter> the overlay attempt. > > They're small. They are sparse files, so just follow the > instructions. > > Peter> What are the size of the overlay files going to end up being? > > Not too large, but it depends on how much data is written to the > overlayfs to get your data back. If you follow the instructions on > this page: > > https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID > > It says to create a sparse file for each disk that is 1% of the size > of the disk. This can add up... you might need to add a blank disk to > your system to hold these. > > In this case, if you think you know which order the disks were in, you > could try to create the RAID6 array (but only using the overlayfs > devices!!!!!) I can't stress this enough. > > > Peter> I did run into UEFI vs AHCI issues early in the process.. they > Peter> are all set to non-UEFI. > > That's good. > > Peter> OS update was onto a new SSD... > > Ok. Do you have the old OS disk around by any chance? That might > give some pointers to how the disks are setup.. You could look in > /var/tmp/initrd/... for old mdadm.conf files, which might give more > details. > > Peter> On Sun, Aug 28, 2022, 12:47 Phil Turmel <philip@turmel.org> wrote: > > Peter> Hi Peter, et al, > > Peter> On 8/28/22 05:54, Wols Lists wrote: >>> On 28/08/2022 10:14, Wols Lists wrote: >>>>> Currently I have no /dev/md* devices. >>>>> I have access to the old mdadm.conf file - have tried assembling with >>>>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc >>>>> and /etc/mdadm. >>>> >>>> It looks like the drives weren't partitioned :-( I think you're into >>>> forensics. > > Peter> It is too soon to say this. The supplied mdadm.conf file does not > Peter> contain specific partition information. It is possible the partition > Peter> tables have just been wiped. > >>> Whoops - my system froze while I was originally writing my reply, and I >>> forgot to put this into my rewrite ... >>> >>> Look up overlays in the wiki. I've never done it myself, but a fair few >>> people have said the instructions worked a treat. >>> >>> You're basically making the drives read-only (all writes get dumped into >>> the overlay file), and then re-creating the array over the top, so you >>> can test whether you got it right. If you don't, you just ditch the >>> overlays and start again, if you did get it right you can recreate the >>> array for real. >>> >>> Cheers, >>> Wol > > Peter> On 8/28/22 11:10, John Stoffel wrote: >>>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >>> > Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. > Peter> Had hardware issues with my system - ended up replacing the > Peter> motherboard, video card, and power supply and re-installing the OS > Peter> (Debian 11). >>> >>> Can you give us details on the old vs new motherboard/cpu? It might >>> be that you need to tweak the BIOS of the motherboard to expose the >>> old SATA formats as well. >>> >>> Did you install debian onto a fresh boot disk? Is your BIOS setup to >>> only do the new form of booting from UEFI devices, so maybe check your >>> BIOS settings that the data drives are all in AHCI mode, or possibly >>> even in IDE mode. It all depends on how old the original hardware >>> was. >>> >>> I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak >>> the BIOS defaults to see my disks. I guess I should do a clean >>> install from a blank disk, but I wanted to minimize downtime. > > Peter> It is important to end up in AHCI mode on all MOBO ports. If not set > Peter> that way now, please change them. > >>> Wols has some great advice here, and I heartily recommend that you use >>> overlayfs when doing your testing. Check the RAID WIKI for >>> suggestions. > > Peter> Concur. > >>> And don't panic! Your data is probably there, but just missing the >>> super blocks or partition tables. > > Peter> Both, I suspect. > > Peter> On 8/27/22 22:00, Peter Sanders wrote: >>> lsdrv ------------------------ >>> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics >>> Corporation E12 NVMe Controller (rev 01) >>> └nvme nvme0 PCIe SSD {21112925606047} >>> └nvme0n1 238.47g [259:0] Partitioned (dos) >>> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} >>> │└Mounted as /dev/nvme0n1p1 @ /boot >>> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) >>> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} >>> │└Mounted as /dev/nvme0n1p5 @ / >>> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} >>> │└Mounted as /dev/nvme0n1p6 @ /var >>> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} >>> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} >>> │└Mounted as /dev/nvme0n1p8 @ /home >>> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} >>> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. >>> JMB368 IDE controller (rev 10) >>> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} >>> └sr0 1.00g [11:0] Empty/Unknown >>> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 >>> SATA/IDE Controller (rev 10) >>> └scsi 2:x:x:x [Empty] >>> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >>> FCH SATA Controller [AHCI mode] (rev 51) >>> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} >>> │└sda 2.73t [8:0] Partitioned (PMBR) >>> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} >>> └sdc 2.73t [8:32] Partitioned (gpt) >>> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >>> FCH SATA Controller [AHCI mode] (rev 51) >>> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} >>> │└sdb 2.73t [8:16] Partitioned (gpt) >>> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} >>> │└sdd 2.73t [8:48] Partitioned (gpt) >>> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} >>> │└sde 2.73t [8:64] Partitioned (gpt) >>> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} >>> └sdf 2.73t [8:80] Partitioned (gpt) > > Peter> Unfortunately, my lsdrv tool is not able to reconstruct missing parts. > Peter> It is most useful when used on a *good* system and *saved* for help > Peter> diagnosing *future* problems. > > Peter> Please share your /etc/fstab, and if you were using LVM on top of the > Peter> raid, share your lvm.conf and anything in /etc/lvm/backup. > > Peter> Please describe the layer(s) that were on top of the raid. > > Peter> We need to help you look for signatures, and it helps to be selective in > Peter> what signatures to look for. > > Peter> After that, we will want to figure out your raid's chunk size and data > Peter> offsets. If you know of a particular large file (8MB or larger) that is > Peter> sure to be in the raid and you happen to have a copy tucked away, then > Peter> my findHash[1] tool might be able to definitively determine those > Peter> values. (Time consuming, though.) > > Peter> Meanwhile, don't do *anything* that would write to those drives. > > Peter> Phil > > Peter> [1] https://github.com/pturmel/findHash > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 19:36 ` Phil Turmel @ 2022-08-28 19:49 ` John Stoffel 2022-08-28 23:24 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-08-28 19:49 UTC (permalink / raw) To: Phil Turmel Cc: John Stoffel, Peter Sanders, Wols Lists, NeilBrown, linux-raid >>>>> "Phil" == Phil Turmel <philip@turmel.org> writes: Phil> Pssst! John, Phil> All of my comments were attributed to Peter by your mail client. ): Yeah... sometimes my mail reader gets confused when it cites previous emails. I shoudl probably just drop to > only from now on. Phil> On 8/28/22 14:45, John Stoffel wrote: >>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> Peter> It was set up on the device level, not partitions. (I remember Peter> getting some advice on the web that device was better than Peter> partition... Yay for internet advice) >> >> Yeah, this is NOT good advice. Generally systems will not destroy >> existing partition tables, but if they see an empty (to them) >> disk... all bets are off. >> Peter> I'm surveying my other disks to see what I have available to do Peter> the overlay attempt. >> >> They're small. They are sparse files, so just follow the >> instructions. >> Peter> What are the size of the overlay files going to end up being? >> >> Not too large, but it depends on how much data is written to the >> overlayfs to get your data back. If you follow the instructions on >> this page: >> >> https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID >> >> It says to create a sparse file for each disk that is 1% of the size >> of the disk. This can add up... you might need to add a blank disk to >> your system to hold these. >> >> In this case, if you think you know which order the disks were in, you >> could try to create the RAID6 array (but only using the overlayfs >> devices!!!!!) I can't stress this enough. >> >> Peter> I did run into UEFI vs AHCI issues early in the process.. they Peter> are all set to non-UEFI. >> >> That's good. >> Peter> OS update was onto a new SSD... >> >> Ok. Do you have the old OS disk around by any chance? That might >> give some pointers to how the disks are setup.. You could look in >> /var/tmp/initrd/... for old mdadm.conf files, which might give more >> details. >> Peter> On Sun, Aug 28, 2022, 12:47 Phil Turmel <philip@turmel.org> wrote: >> Peter> Hi Peter, et al, >> Peter> On 8/28/22 05:54, Wols Lists wrote: >>>> On 28/08/2022 10:14, Wols Lists wrote: >>>>> Currently I have no /dev/md* devices. >>>>> I have access to the old mdadm.conf file - have tried assembling with >>>>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc >>>>> and /etc/mdadm. >>>>> >>>>> It looks like the drives weren't partitioned :-( I think you're into >>>>> forensics. >> Peter> It is too soon to say this. The supplied mdadm.conf file does not Peter> contain specific partition information. It is possible the partition Peter> tables have just been wiped. >> >>>> Whoops - my system froze while I was originally writing my reply, and I >>>> forgot to put this into my rewrite ... >>>> >>>> Look up overlays in the wiki. I've never done it myself, but a fair few >>>> people have said the instructions worked a treat. >>>> >>>> You're basically making the drives read-only (all writes get dumped into >>>> the overlay file), and then re-creating the array over the top, so you >>>> can test whether you got it right. If you don't, you just ditch the >>>> overlays and start again, if you did get it right you can recreate the >>>> array for real. >>>> >>>> Cheers, >>>> Wol >> Peter> On 8/28/22 11:10, John Stoffel wrote: >>>>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >>>> Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. Peter> Had hardware issues with my system - ended up replacing the Peter> motherboard, video card, and power supply and re-installing the OS Peter> (Debian 11). >>>> >>>> Can you give us details on the old vs new motherboard/cpu? It might >>>> be that you need to tweak the BIOS of the motherboard to expose the >>>> old SATA formats as well. >>>> >>>> Did you install debian onto a fresh boot disk? Is your BIOS setup to >>>> only do the new form of booting from UEFI devices, so maybe check your >>>> BIOS settings that the data drives are all in AHCI mode, or possibly >>>> even in IDE mode. It all depends on how old the original hardware >>>> was. >>>> >>>> I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak >>>> the BIOS defaults to see my disks. I guess I should do a clean >>>> install from a blank disk, but I wanted to minimize downtime. >> Peter> It is important to end up in AHCI mode on all MOBO ports. If not set Peter> that way now, please change them. >> >>>> Wols has some great advice here, and I heartily recommend that you use >>>> overlayfs when doing your testing. Check the RAID WIKI for >>>> suggestions. >> Peter> Concur. >> >>>> And don't panic! Your data is probably there, but just missing the >>>> super blocks or partition tables. >> Peter> Both, I suspect. >> Peter> On 8/27/22 22:00, Peter Sanders wrote: >>>> lsdrv ------------------------ >>>> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics >>>> Corporation E12 NVMe Controller (rev 01) >>>> └nvme nvme0 PCIe SSD {21112925606047} >>>> └nvme0n1 238.47g [259:0] Partitioned (dos) >>>> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} >>>> │└Mounted as /dev/nvme0n1p1 @ /boot >>>> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) >>>> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} >>>> │└Mounted as /dev/nvme0n1p5 @ / >>>> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} >>>> │└Mounted as /dev/nvme0n1p6 @ /var >>>> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} >>>> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} >>>> │└Mounted as /dev/nvme0n1p8 @ /home >>>> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} >>>> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. >>>> JMB368 IDE controller (rev 10) >>>> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} >>>> └sr0 1.00g [11:0] Empty/Unknown >>>> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 >>>> SATA/IDE Controller (rev 10) >>>> └scsi 2:x:x:x [Empty] >>>> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >>>> FCH SATA Controller [AHCI mode] (rev 51) >>>> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} >>>> │└sda 2.73t [8:0] Partitioned (PMBR) >>>> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} >>>> └sdc 2.73t [8:32] Partitioned (gpt) >>>> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] >>>> FCH SATA Controller [AHCI mode] (rev 51) >>>> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} >>>> │└sdb 2.73t [8:16] Partitioned (gpt) >>>> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} >>>> │└sdd 2.73t [8:48] Partitioned (gpt) >>>> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} >>>> │└sde 2.73t [8:64] Partitioned (gpt) >>>> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} >>>> └sdf 2.73t [8:80] Partitioned (gpt) >> Peter> Unfortunately, my lsdrv tool is not able to reconstruct missing parts. Peter> It is most useful when used on a *good* system and *saved* for help Peter> diagnosing *future* problems. >> Peter> Please share your /etc/fstab, and if you were using LVM on top of the Peter> raid, share your lvm.conf and anything in /etc/lvm/backup. >> Peter> Please describe the layer(s) that were on top of the raid. >> Peter> We need to help you look for signatures, and it helps to be selective in Peter> what signatures to look for. >> Peter> After that, we will want to figure out your raid's chunk size and data Peter> offsets. If you know of a particular large file (8MB or larger) that is Peter> sure to be in the raid and you happen to have a copy tucked away, then Peter> my findHash[1] tool might be able to definitively determine those Peter> values. (Time consuming, though.) >> Peter> Meanwhile, don't do *anything* that would write to those drives. >> Peter> Phil >> Peter> [1] https://github.com/pturmel/findHash >> ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 19:49 ` John Stoffel @ 2022-08-28 23:24 ` Peter Sanders 2022-08-29 13:12 ` Peter Sanders 2022-08-29 21:45 ` John Stoffel 0 siblings, 2 replies; 29+ messages in thread From: Peter Sanders @ 2022-08-28 23:24 UTC (permalink / raw) To: John Stoffel; +Cc: Phil Turmel, Wols Lists, NeilBrown, linux-raid Phil, fstab from the working config - # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/sda1 during installation UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 errors=remount-ro 0 1 # /home was on /dev/sda6 during installation UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 defaults 0 2 # /var was on /dev/sda5 during installation UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 defaults 0 2 /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 # raid array /dev/md0 /mnt/raid6 ext4 defaults 0 2 No LVM, one large EXT4 partition I have several large files ( NEF and various mpg files) I can identify and have backup copies available. I have the overlays created. 300G for each of the six drives. - Peter On Sun, Aug 28, 2022 at 3:49 PM John Stoffel <john@stoffel.org> wrote: > > >>>>> "Phil" == Phil Turmel <philip@turmel.org> writes: > > Phil> Pssst! John, > Phil> All of my comments were attributed to Peter by your mail client. ): > > Yeah... sometimes my mail reader gets confused when it cites previous > emails. I shoudl probably just drop to > only from now on. > > > Phil> On 8/28/22 14:45, John Stoffel wrote: > >>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > >> > Peter> It was set up on the device level, not partitions. (I remember > Peter> getting some advice on the web that device was better than > Peter> partition... Yay for internet advice) > >> > >> Yeah, this is NOT good advice. Generally systems will not destroy > >> existing partition tables, but if they see an empty (to them) > >> disk... all bets are off. > >> > Peter> I'm surveying my other disks to see what I have available to do > Peter> the overlay attempt. > >> > >> They're small. They are sparse files, so just follow the > >> instructions. > >> > Peter> What are the size of the overlay files going to end up being? > >> > >> Not too large, but it depends on how much data is written to the > >> overlayfs to get your data back. If you follow the instructions on > >> this page: > >> > >> https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID > >> > >> It says to create a sparse file for each disk that is 1% of the size > >> of the disk. This can add up... you might need to add a blank disk to > >> your system to hold these. > >> > >> In this case, if you think you know which order the disks were in, you > >> could try to create the RAID6 array (but only using the overlayfs > >> devices!!!!!) I can't stress this enough. > >> > >> > Peter> I did run into UEFI vs AHCI issues early in the process.. they > Peter> are all set to non-UEFI. > >> > >> That's good. > >> > Peter> OS update was onto a new SSD... > >> > >> Ok. Do you have the old OS disk around by any chance? That might > >> give some pointers to how the disks are setup.. You could look in > >> /var/tmp/initrd/... for old mdadm.conf files, which might give more > >> details. > >> > Peter> On Sun, Aug 28, 2022, 12:47 Phil Turmel <philip@turmel.org> wrote: > >> > Peter> Hi Peter, et al, > >> > Peter> On 8/28/22 05:54, Wols Lists wrote: > >>>> On 28/08/2022 10:14, Wols Lists wrote: > >>>>> Currently I have no /dev/md* devices. > >>>>> I have access to the old mdadm.conf file - have tried assembling with > >>>>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc > >>>>> and /etc/mdadm. > >>>>> > >>>>> It looks like the drives weren't partitioned :-( I think you're into > >>>>> forensics. > >> > Peter> It is too soon to say this. The supplied mdadm.conf file does not > Peter> contain specific partition information. It is possible the partition > Peter> tables have just been wiped. > >> > >>>> Whoops - my system froze while I was originally writing my reply, and I > >>>> forgot to put this into my rewrite ... > >>>> > >>>> Look up overlays in the wiki. I've never done it myself, but a fair few > >>>> people have said the instructions worked a treat. > >>>> > >>>> You're basically making the drives read-only (all writes get dumped into > >>>> the overlay file), and then re-creating the array over the top, so you > >>>> can test whether you got it right. If you don't, you just ditch the > >>>> overlays and start again, if you did get it right you can recreate the > >>>> array for real. > >>>> > >>>> Cheers, > >>>> Wol > >> > Peter> On 8/28/22 11:10, John Stoffel wrote: > >>>>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > >>>> > Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. > Peter> Had hardware issues with my system - ended up replacing the > Peter> motherboard, video card, and power supply and re-installing the OS > Peter> (Debian 11). > >>>> > >>>> Can you give us details on the old vs new motherboard/cpu? It might > >>>> be that you need to tweak the BIOS of the motherboard to expose the > >>>> old SATA formats as well. > >>>> > >>>> Did you install debian onto a fresh boot disk? Is your BIOS setup to > >>>> only do the new form of booting from UEFI devices, so maybe check your > >>>> BIOS settings that the data drives are all in AHCI mode, or possibly > >>>> even in IDE mode. It all depends on how old the original hardware > >>>> was. > >>>> > >>>> I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak > >>>> the BIOS defaults to see my disks. I guess I should do a clean > >>>> install from a blank disk, but I wanted to minimize downtime. > >> > Peter> It is important to end up in AHCI mode on all MOBO ports. If not set > Peter> that way now, please change them. > >> > >>>> Wols has some great advice here, and I heartily recommend that you use > >>>> overlayfs when doing your testing. Check the RAID WIKI for > >>>> suggestions. > >> > Peter> Concur. > >> > >>>> And don't panic! Your data is probably there, but just missing the > >>>> super blocks or partition tables. > >> > Peter> Both, I suspect. > >> > Peter> On 8/27/22 22:00, Peter Sanders wrote: > >>>> lsdrv ------------------------ > >>>> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics > >>>> Corporation E12 NVMe Controller (rev 01) > >>>> └nvme nvme0 PCIe SSD {21112925606047} > >>>> └nvme0n1 238.47g [259:0] Partitioned (dos) > >>>> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} > >>>> │└Mounted as /dev/nvme0n1p1 @ /boot > >>>> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) > >>>> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} > >>>> │└Mounted as /dev/nvme0n1p5 @ / > >>>> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} > >>>> │└Mounted as /dev/nvme0n1p6 @ /var > >>>> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} > >>>> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} > >>>> │└Mounted as /dev/nvme0n1p8 @ /home > >>>> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} > >>>> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. > >>>> JMB368 IDE controller (rev 10) > >>>> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} > >>>> └sr0 1.00g [11:0] Empty/Unknown > >>>> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 > >>>> SATA/IDE Controller (rev 10) > >>>> └scsi 2:x:x:x [Empty] > >>>> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > >>>> FCH SATA Controller [AHCI mode] (rev 51) > >>>> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} > >>>> │└sda 2.73t [8:0] Partitioned (PMBR) > >>>> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} > >>>> └sdc 2.73t [8:32] Partitioned (gpt) > >>>> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > >>>> FCH SATA Controller [AHCI mode] (rev 51) > >>>> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} > >>>> │└sdb 2.73t [8:16] Partitioned (gpt) > >>>> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} > >>>> │└sdd 2.73t [8:48] Partitioned (gpt) > >>>> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} > >>>> │└sde 2.73t [8:64] Partitioned (gpt) > >>>> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} > >>>> └sdf 2.73t [8:80] Partitioned (gpt) > >> > Peter> Unfortunately, my lsdrv tool is not able to reconstruct missing parts. > Peter> It is most useful when used on a *good* system and *saved* for help > Peter> diagnosing *future* problems. > >> > Peter> Please share your /etc/fstab, and if you were using LVM on top of the > Peter> raid, share your lvm.conf and anything in /etc/lvm/backup. > >> > Peter> Please describe the layer(s) that were on top of the raid. > >> > Peter> We need to help you look for signatures, and it helps to be selective in > Peter> what signatures to look for. > >> > Peter> After that, we will want to figure out your raid's chunk size and data > Peter> offsets. If you know of a particular large file (8MB or larger) that is > Peter> sure to be in the raid and you happen to have a copy tucked away, then > Peter> my findHash[1] tool might be able to definitively determine those > Peter> values. (Time consuming, though.) > >> > Peter> Meanwhile, don't do *anything* that would write to those drives. > >> > Peter> Phil > >> > Peter> [1] https://github.com/pturmel/findHash > >> > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 23:24 ` Peter Sanders @ 2022-08-29 13:12 ` Peter Sanders 2022-08-29 21:45 ` John Stoffel 1 sibling, 0 replies; 29+ messages in thread From: Peter Sanders @ 2022-08-29 13:12 UTC (permalink / raw) To: John Stoffel; +Cc: Phil Turmel, Wols Lists, NeilBrown, linux-raid Phil, The correct findHash - python tool on github - pturmel/findHash? - Peter On Sun, Aug 28, 2022 at 7:24 PM Peter Sanders <plsander@gmail.com> wrote: > > Phil, > > fstab from the working config - > > # <file system> <mount point> <type> <options> <dump> <pass> > # / was on /dev/sda1 during installation > UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 > errors=remount-ro 0 1 > # /home was on /dev/sda6 during installation > UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 > defaults 0 2 > # /var was on /dev/sda5 during installation > UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 > defaults 0 2 > /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 > /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 > # raid array > /dev/md0 /mnt/raid6 ext4 defaults 0 2 > > No LVM, one large EXT4 partition > > I have several large files ( NEF and various mpg files) I can identify > and have backup copies available. > > I have the overlays created. 300G for each of the six drives. > > - Peter > > On Sun, Aug 28, 2022 at 3:49 PM John Stoffel <john@stoffel.org> wrote: > > > > >>>>> "Phil" == Phil Turmel <philip@turmel.org> writes: > > > > Phil> Pssst! John, > > Phil> All of my comments were attributed to Peter by your mail client. ): > > > > Yeah... sometimes my mail reader gets confused when it cites previous > > emails. I shoudl probably just drop to > only from now on. > > > > > > Phil> On 8/28/22 14:45, John Stoffel wrote: > > >>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > >> > > Peter> It was set up on the device level, not partitions. (I remember > > Peter> getting some advice on the web that device was better than > > Peter> partition... Yay for internet advice) > > >> > > >> Yeah, this is NOT good advice. Generally systems will not destroy > > >> existing partition tables, but if they see an empty (to them) > > >> disk... all bets are off. > > >> > > Peter> I'm surveying my other disks to see what I have available to do > > Peter> the overlay attempt. > > >> > > >> They're small. They are sparse files, so just follow the > > >> instructions. > > >> > > Peter> What are the size of the overlay files going to end up being? > > >> > > >> Not too large, but it depends on how much data is written to the > > >> overlayfs to get your data back. If you follow the instructions on > > >> this page: > > >> > > >> https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID > > >> > > >> It says to create a sparse file for each disk that is 1% of the size > > >> of the disk. This can add up... you might need to add a blank disk to > > >> your system to hold these. > > >> > > >> In this case, if you think you know which order the disks were in, you > > >> could try to create the RAID6 array (but only using the overlayfs > > >> devices!!!!!) I can't stress this enough. > > >> > > >> > > Peter> I did run into UEFI vs AHCI issues early in the process.. they > > Peter> are all set to non-UEFI. > > >> > > >> That's good. > > >> > > Peter> OS update was onto a new SSD... > > >> > > >> Ok. Do you have the old OS disk around by any chance? That might > > >> give some pointers to how the disks are setup.. You could look in > > >> /var/tmp/initrd/... for old mdadm.conf files, which might give more > > >> details. > > >> > > Peter> On Sun, Aug 28, 2022, 12:47 Phil Turmel <philip@turmel.org> wrote: > > >> > > Peter> Hi Peter, et al, > > >> > > Peter> On 8/28/22 05:54, Wols Lists wrote: > > >>>> On 28/08/2022 10:14, Wols Lists wrote: > > >>>>> Currently I have no /dev/md* devices. > > >>>>> I have access to the old mdadm.conf file - have tried assembling with > > >>>>> it, with the default mdadm.conf, and with no mdadm.conf file in /etc > > >>>>> and /etc/mdadm. > > >>>>> > > >>>>> It looks like the drives weren't partitioned :-( I think you're into > > >>>>> forensics. > > >> > > Peter> It is too soon to say this. The supplied mdadm.conf file does not > > Peter> contain specific partition information. It is possible the partition > > Peter> tables have just been wiped. > > >> > > >>>> Whoops - my system froze while I was originally writing my reply, and I > > >>>> forgot to put this into my rewrite ... > > >>>> > > >>>> Look up overlays in the wiki. I've never done it myself, but a fair few > > >>>> people have said the instructions worked a treat. > > >>>> > > >>>> You're basically making the drives read-only (all writes get dumped into > > >>>> the overlay file), and then re-creating the array over the top, so you > > >>>> can test whether you got it right. If you don't, you just ditch the > > >>>> overlays and start again, if you did get it right you can recreate the > > >>>> array for real. > > >>>> > > >>>> Cheers, > > >>>> Wol > > >> > > Peter> On 8/28/22 11:10, John Stoffel wrote: > > >>>>>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > >>>> > > Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. > > Peter> Had hardware issues with my system - ended up replacing the > > Peter> motherboard, video card, and power supply and re-installing the OS > > Peter> (Debian 11). > > >>>> > > >>>> Can you give us details on the old vs new motherboard/cpu? It might > > >>>> be that you need to tweak the BIOS of the motherboard to expose the > > >>>> old SATA formats as well. > > >>>> > > >>>> Did you install debian onto a fresh boot disk? Is your BIOS setup to > > >>>> only do the new form of booting from UEFI devices, so maybe check your > > >>>> BIOS settings that the data drives are all in AHCI mode, or possibly > > >>>> even in IDE mode. It all depends on how old the original hardware > > >>>> was. > > >>>> > > >>>> I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak > > >>>> the BIOS defaults to see my disks. I guess I should do a clean > > >>>> install from a blank disk, but I wanted to minimize downtime. > > >> > > Peter> It is important to end up in AHCI mode on all MOBO ports. If not set > > Peter> that way now, please change them. > > >> > > >>>> Wols has some great advice here, and I heartily recommend that you use > > >>>> overlayfs when doing your testing. Check the RAID WIKI for > > >>>> suggestions. > > >> > > Peter> Concur. > > >> > > >>>> And don't panic! Your data is probably there, but just missing the > > >>>> super blocks or partition tables. > > >> > > Peter> Both, I suspect. > > >> > > Peter> On 8/27/22 22:00, Peter Sanders wrote: > > >>>> lsdrv ------------------------ > > >>>> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics > > >>>> Corporation E12 NVMe Controller (rev 01) > > >>>> └nvme nvme0 PCIe SSD {21112925606047} > > >>>> └nvme0n1 238.47g [259:0] Partitioned (dos) > > >>>> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} > > >>>> │└Mounted as /dev/nvme0n1p1 @ /boot > > >>>> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) > > >>>> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} > > >>>> │└Mounted as /dev/nvme0n1p5 @ / > > >>>> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} > > >>>> │└Mounted as /dev/nvme0n1p6 @ /var > > >>>> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} > > >>>> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} > > >>>> │└Mounted as /dev/nvme0n1p8 @ /home > > >>>> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} > > >>>> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. > > >>>> JMB368 IDE controller (rev 10) > > >>>> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} > > >>>> └sr0 1.00g [11:0] Empty/Unknown > > >>>> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 > > >>>> SATA/IDE Controller (rev 10) > > >>>> └scsi 2:x:x:x [Empty] > > >>>> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > > >>>> FCH SATA Controller [AHCI mode] (rev 51) > > >>>> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} > > >>>> │└sda 2.73t [8:0] Partitioned (PMBR) > > >>>> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} > > >>>> └sdc 2.73t [8:32] Partitioned (gpt) > > >>>> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] > > >>>> FCH SATA Controller [AHCI mode] (rev 51) > > >>>> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} > > >>>> │└sdb 2.73t [8:16] Partitioned (gpt) > > >>>> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} > > >>>> │└sdd 2.73t [8:48] Partitioned (gpt) > > >>>> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} > > >>>> │└sde 2.73t [8:64] Partitioned (gpt) > > >>>> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} > > >>>> └sdf 2.73t [8:80] Partitioned (gpt) > > >> > > Peter> Unfortunately, my lsdrv tool is not able to reconstruct missing parts. > > Peter> It is most useful when used on a *good* system and *saved* for help > > Peter> diagnosing *future* problems. > > >> > > Peter> Please share your /etc/fstab, and if you were using LVM on top of the > > Peter> raid, share your lvm.conf and anything in /etc/lvm/backup. > > >> > > Peter> Please describe the layer(s) that were on top of the raid. > > >> > > Peter> We need to help you look for signatures, and it helps to be selective in > > Peter> what signatures to look for. > > >> > > Peter> After that, we will want to figure out your raid's chunk size and data > > Peter> offsets. If you know of a particular large file (8MB or larger) that is > > Peter> sure to be in the raid and you happen to have a copy tucked away, then > > Peter> my findHash[1] tool might be able to definitively determine those > > Peter> values. (Time consuming, though.) > > >> > > Peter> Meanwhile, don't do *anything* that would write to those drives. > > >> > > Peter> Phil > > >> > > Peter> [1] https://github.com/pturmel/findHash > > >> > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 23:24 ` Peter Sanders 2022-08-29 13:12 ` Peter Sanders @ 2022-08-29 21:45 ` John Stoffel 2022-08-29 22:29 ` Eyal Lebedinsky 1 sibling, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-08-29 21:45 UTC (permalink / raw) To: Peter Sanders Cc: John Stoffel, Phil Turmel, Wols Lists, NeilBrown, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: Peter> Phil, Peter> fstab from the working config - Peter> # <file system> <mount point> <type> <options> <dump> <pass> Peter> # / was on /dev/sda1 during installation Peter> UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 Peter> errors=remount-ro 0 1 Peter> # /home was on /dev/sda6 during installation Peter> UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 Peter> defaults 0 2 Peter> # /var was on /dev/sda5 during installation Peter> UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 Peter> defaults 0 2 Peter> /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 Peter> /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 Peter> # raid array Peter> /dev/md0 /mnt/raid6 ext4 defaults 0 2 Peter> No LVM, one large EXT4 partition Peter> I have several large files ( NEF and various mpg files) I can identify Peter> and have backup copies available. Peter> I have the overlays created. 300G for each of the six drives. So that's good. Now you have to try and figure out which order they were created in. As the docs show, you setup the overlayfs on top of each of the six drives. Keep track by noting the drive serial numbers, since Linux can move them around and change drive letters on reboots. Then using the overlays, do an: mdadm --create /dev/md0 --level=raid6 -n 6 /dev/sd[bcdefg] fsck -n /dev/md0 and see what you get. If it doesn't look like a real filesystem, then you can break it down, and then modify the order you give the drive letters, like: /dev/sd[cdefge] and rinse and repeat as it goes. Not fun... but should hopefully fix things for you. John ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-29 21:45 ` John Stoffel @ 2022-08-29 22:29 ` Eyal Lebedinsky 2022-08-29 23:53 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: Eyal Lebedinsky @ 2022-08-29 22:29 UTC (permalink / raw) To: linux-raid On 30/08/2022 07.45, John Stoffel wrote: >>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > Peter> Phil, > Peter> fstab from the working config - > > Peter> # <file system> <mount point> <type> <options> <dump> <pass> > Peter> # / was on /dev/sda1 during installation > Peter> UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 > Peter> errors=remount-ro 0 1 > Peter> # /home was on /dev/sda6 during installation > Peter> UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 > Peter> defaults 0 2 > Peter> # /var was on /dev/sda5 during installation > Peter> UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 > Peter> defaults 0 2 > Peter> /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 > Peter> /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 > Peter> # raid array > Peter> /dev/md0 /mnt/raid6 ext4 defaults 0 2 > > Peter> No LVM, one large EXT4 partition > > Peter> I have several large files ( NEF and various mpg files) I can identify > Peter> and have backup copies available. > > Peter> I have the overlays created. 300G for each of the six drives. > > So that's good. Now you have to try and figure out which order they > were created in. As the docs show, you setup the overlayfs on top of > each of the six drives. > > Keep track by noting the drive serial numbers, since Linux can move > them around and change drive letters on reboots. > > > Then using the overlays, do an: > > mdadm --create /dev/md0 --level=raid6 -n 6 /dev/sd[bcdefg] > fsck -n /dev/md0 > > and see what you get. If it doesn't look like a real filesystem, then > you can break it down, and then modify the order you give the drive > letters, like: > > /dev/sd[cdefge] > > and rinse and repeat as it goes. Not fun... but should hopefully fix > things for you. > > John An aside, I would think the way to specify a list in a nominated order is something like $ echo /dev/sd{c,d,a,b} /dev/sdc /dev/sdd /dev/sda /dev/sdb rather than $ echo /dev/sd[cdab] /dev/sda /dev/sdb /dev/sdc /dev/sdd which will be in sorting order, regardless of the order of the letter. -- Eyal Lebedinsky (fedora@eyal.emu.id.au) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-29 22:29 ` Eyal Lebedinsky @ 2022-08-29 23:53 ` Peter Sanders 2022-08-30 13:27 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-08-29 23:53 UTC (permalink / raw) To: Eyal Lebedinsky; +Cc: linux-raid Couple more questions. Mdadm -create ... Do I use the /dev/sdx or /dev/mapper/sdx name for the overlayed device? And reset the mapping between each create attempt by doing: remove the loop-device/overlay association dmsetup remove on all devices remove the overlay files rm remove the loop back devices losetup -d ... rebuild the loop back devices mknod -m 660 ... build the overlay files truncate -s 300G overlay-... reassociate the loop-devices and the overlays losetup... dmsetup.. and try again. (Yeah, I recognize that there is code to do this (I think) in the article, but my script-fu is not up to fully understanding those examples.) On Mon, Aug 29, 2022 at 6:58 PM Eyal Lebedinsky <fedora@eyal.emu.id.au> wrote: > > > On 30/08/2022 07.45, John Stoffel wrote: > >>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > > Peter> Phil, > > Peter> fstab from the working config - > > > > Peter> # <file system> <mount point> <type> <options> <dump> <pass> > > Peter> # / was on /dev/sda1 during installation > > Peter> UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 > > Peter> errors=remount-ro 0 1 > > Peter> # /home was on /dev/sda6 during installation > > Peter> UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 > > Peter> defaults 0 2 > > Peter> # /var was on /dev/sda5 during installation > > Peter> UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 > > Peter> defaults 0 2 > > Peter> /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 > > Peter> /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 > > Peter> # raid array > > Peter> /dev/md0 /mnt/raid6 ext4 defaults 0 2 > > > > Peter> No LVM, one large EXT4 partition > > > > Peter> I have several large files ( NEF and various mpg files) I can identify > > Peter> and have backup copies available. > > > > Peter> I have the overlays created. 300G for each of the six drives. > > > > So that's good. Now you have to try and figure out which order they > > were created in. As the docs show, you setup the overlayfs on top of > > each of the six drives. > > > > Keep track by noting the drive serial numbers, since Linux can move > > them around and change drive letters on reboots. > > > > > > Then using the overlays, do an: > > > > mdadm --create /dev/md0 --level=raid6 -n 6 /dev/sd[bcdefg] > > fsck -n /dev/md0 > > > > and see what you get. If it doesn't look like a real filesystem, then > > you can break it down, and then modify the order you give the drive > > letters, like: > > > > /dev/sd[cdefge] > > > > and rinse and repeat as it goes. Not fun... but should hopefully fix > > things for you. > > > > John > > An aside, I would think the way to specify a list in a nominated order is something like > > $ echo /dev/sd{c,d,a,b} > /dev/sdc /dev/sdd /dev/sda /dev/sdb > > rather than > > $ echo /dev/sd[cdab] > /dev/sda /dev/sdb /dev/sdc /dev/sdd > > which will be in sorting order, regardless of the order of the letter. > > -- > Eyal Lebedinsky (fedora@eyal.emu.id.au) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-29 23:53 ` Peter Sanders @ 2022-08-30 13:27 ` Peter Sanders 2022-08-30 18:03 ` Wols Lists 0 siblings, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-08-30 13:27 UTC (permalink / raw) To: Eyal Lebedinsky; +Cc: linux-raid Tried with the /dev/mapper/sdx devices. root@superior:/mnt/backup# mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg mdadm: partition table exists on /dev/mapper/sdb mdadm: partition table exists on /dev/mapper/sdc mdadm: partition table exists on /dev/mapper/sdc but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdd mdadm: partition table exists on /dev/mapper/sdd but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sde mdadm: partition table exists on /dev/mapper/sde but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdf mdadm: partition table exists on /dev/mapper/sdf but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdg mdadm: partition table exists on /dev/mapper/sdg but will be lost or meaningless after creating array Continue creating array? n mdadm: create aborted. root@superior:/mnt/backup# Chickened out and aborted the create. Are those expected messages for the mess I am in? And the victory conditions would be a mountable file system that passes a fsck? On Mon, Aug 29, 2022 at 7:53 PM Peter Sanders <plsander@gmail.com> wrote: > > Couple more questions. > > Mdadm -create ... Do I use the /dev/sdx or /dev/mapper/sdx name for > the overlayed device? > > And reset the mapping between each create attempt by doing: > remove the loop-device/overlay association > dmsetup remove on all devices > remove the overlay files > rm > remove the loop back devices > losetup -d ... > rebuild the loop back devices > mknod -m 660 ... > build the overlay files > truncate -s 300G overlay-... > reassociate the loop-devices and the overlays > losetup... dmsetup.. > > and try again. > > (Yeah, I recognize that there is code to do this (I think) in the > article, but my script-fu is not up to fully understanding those > examples.) > > On Mon, Aug 29, 2022 at 6:58 PM Eyal Lebedinsky <fedora@eyal.emu.id.au> wrote: > > > > > > On 30/08/2022 07.45, John Stoffel wrote: > > >>>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > > > > Peter> Phil, > > > Peter> fstab from the working config - > > > > > > Peter> # <file system> <mount point> <type> <options> <dump> <pass> > > > Peter> # / was on /dev/sda1 during installation > > > Peter> UUID=50976432-b750-4809-80ac-3bbdd2773163 / ext4 > > > Peter> errors=remount-ro 0 1 > > > Peter> # /home was on /dev/sda6 during installation > > > Peter> UUID=eb93a2c4-0190-41fa-a41d-7a5966c6bc47 /home ext4 > > > Peter> defaults 0 2 > > > Peter> # /var was on /dev/sda5 during installation > > > Peter> UUID=d1aa6d1f-3ee9-48a8-9350-b15149f738c4 /var ext4 > > > Peter> defaults 0 2 > > > Peter> /dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 > > > Peter> /dev/sr1 /media/cdrom1 udf,iso9660 user,noauto 0 0 > > > Peter> # raid array > > > Peter> /dev/md0 /mnt/raid6 ext4 defaults 0 2 > > > > > > Peter> No LVM, one large EXT4 partition > > > > > > Peter> I have several large files ( NEF and various mpg files) I can identify > > > Peter> and have backup copies available. > > > > > > Peter> I have the overlays created. 300G for each of the six drives. > > > > > > So that's good. Now you have to try and figure out which order they > > > were created in. As the docs show, you setup the overlayfs on top of > > > each of the six drives. > > > > > > Keep track by noting the drive serial numbers, since Linux can move > > > them around and change drive letters on reboots. > > > > > > > > > Then using the overlays, do an: > > > > > > mdadm --create /dev/md0 --level=raid6 -n 6 /dev/sd[bcdefg] > > > fsck -n /dev/md0 > > > > > > and see what you get. If it doesn't look like a real filesystem, then > > > you can break it down, and then modify the order you give the drive > > > letters, like: > > > > > > /dev/sd[cdefge] > > > > > > and rinse and repeat as it goes. Not fun... but should hopefully fix > > > things for you. > > > > > > John > > > > An aside, I would think the way to specify a list in a nominated order is something like > > > > $ echo /dev/sd{c,d,a,b} > > /dev/sdc /dev/sdd /dev/sda /dev/sdb > > > > rather than > > > > $ echo /dev/sd[cdab] > > /dev/sda /dev/sdb /dev/sdc /dev/sdd > > > > which will be in sorting order, regardless of the order of the letter. > > > > -- > > Eyal Lebedinsky (fedora@eyal.emu.id.au) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-30 13:27 ` Peter Sanders @ 2022-08-30 18:03 ` Wols Lists 2022-08-31 17:48 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: Wols Lists @ 2022-08-30 18:03 UTC (permalink / raw) To: Peter Sanders, Eyal Lebedinsky; +Cc: linux-raid On 30/08/2022 14:27, Peter Sanders wrote: > > And the victory conditions would be a mountable file system that passes a fsck? Yes. Just make sure you delve through the file system a bit and satisfy yourself it looks good, too ... Cheers, Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-30 18:03 ` Wols Lists @ 2022-08-31 17:48 ` Peter Sanders 2022-08-31 20:37 ` John Stoffel 0 siblings, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-08-31 17:48 UTC (permalink / raw) To: Wols Lists; +Cc: Eyal Lebedinsky, linux-raid encountering a puzzling situation. dmsetup is failing to return. root@superior:/mnt/backup# dmsetup status sdg: 0 5860533168 snapshot 16/8388608000 16 sdf: 0 5860533168 snapshot 16/8388608000 16 sde: 0 5860533168 snapshot 16/8388608000 16 sdd: 0 5860533168 snapshot 16/8388608000 16 sdc: 0 5860533168 snapshot 16/8388608000 16 sdb: 0 5860533168 snapshot 16/8388608000 16 dmsetup remove sdg runs for hours. Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. dmsetup status shows: sdf: 0 5860533168 snapshot 16/8388608000 16 sde: 0 5860533168 snapshot 16/8388608000 16 sdd: 0 5860533168 snapshot 16/8388608000 16 sdc: 0 5860533168 snapshot 16/8388608000 16 sdb: 0 5860533168 snapshot 16/8388608000 16 dmsetup ls --tree root@superior:/mnt/backup# dmsetup ls --tree sdf (253:3) ├─ (7:3) └─ (8:80) sde (253:1) ├─ (7:1) └─ (8:64) sdd (253:2) ├─ (7:2) └─ (8:48) sdc (253:0) ├─ (7:0) └─ (8:32) sdb (253:5) ├─ (7:5) └─ (8:16) any suggestions? On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > > On 30/08/2022 14:27, Peter Sanders wrote: > > > > And the victory conditions would be a mountable file system that passes a fsck? > > Yes. Just make sure you delve through the file system a bit and satisfy > yourself it looks good, too ... > > Cheers, > Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-31 17:48 ` Peter Sanders @ 2022-08-31 20:37 ` John Stoffel 2022-09-02 14:56 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-08-31 20:37 UTC (permalink / raw) To: Peter Sanders; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > encountering a puzzling situation. > dmsetup is failing to return. I don't think you need to use dmsetup in your case, but can you post *all* the commands you ran before you got to this point, and the output of cat /proc/mdstat as well? Thinking on this some more, you might need to actually also add: --assume-clean to the 'mdadm create ....' string, since you don't want it to zero the array or anything. Sorry for not remembering this at the time! So if you can, please just start over from scratch, showing the setup of the loop devices, the overlayfs setup, and the building the RAID6 array, along with the cat /proc/mdstat after you do the initial build. John P.S. For those who hated my email citing tool, I pulled it out for now. Only citing with > now. :-) > root@superior:/mnt/backup# dmsetup status > sdg: 0 5860533168 snapshot 16/8388608000 16 > sdf: 0 5860533168 snapshot 16/8388608000 16 > sde: 0 5860533168 snapshot 16/8388608000 16 > sdd: 0 5860533168 snapshot 16/8388608000 16 > sdc: 0 5860533168 snapshot 16/8388608000 16 > sdb: 0 5860533168 snapshot 16/8388608000 16 > dmsetup remove sdg runs for hours. > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > dmsetup status shows: > sdf: 0 5860533168 snapshot 16/8388608000 16 > sde: 0 5860533168 snapshot 16/8388608000 16 > sdd: 0 5860533168 snapshot 16/8388608000 16 > sdc: 0 5860533168 snapshot 16/8388608000 16 > sdb: 0 5860533168 snapshot 16/8388608000 16 > dmsetup ls --tree > root@superior:/mnt/backup# dmsetup ls --tree > sdf (253:3) > ├─ (7:3) > └─ (8:80) > sde (253:1) > ├─ (7:1) > └─ (8:64) > sdd (253:2) > ├─ (7:2) > └─ (8:48) > sdc (253:0) > ├─ (7:0) > └─ (8:32) > sdb (253:5) > ├─ (7:5) > └─ (8:16) > any suggestions? > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: >> >> On 30/08/2022 14:27, Peter Sanders wrote: >> > >> > And the victory conditions would be a mountable file system that passes a fsck? >> >> Yes. Just make sure you delve through the file system a bit and satisfy >> yourself it looks good, too ... >> >> Cheers, >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-31 20:37 ` John Stoffel @ 2022-09-02 14:56 ` Peter Sanders 2022-09-02 18:52 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-09-02 14:56 UTC (permalink / raw) To: John Stoffel; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid contents of /proc/mdstat root@superior:/mnt/backup# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none> root@superior:/mnt/backup# Here are the steps I ran (minus some mounting other devices and looking around for mdadm tracks on the old os disk) 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1}) 411 apt install parallel 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1}) 413 echo $DEVICES 414 cat /proc/partitions 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" 417 echo $DEVICES 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 419 ls /dev/loop* 420 dc 421 cd /mnt/backup/ 422 ls 423 parallel truncate -s300G overlay-{/} ::: $DEVICES 424 ls 425 ls -la 426 df -h 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 428 ls /dev/mapper/ 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) 430 echo $OVERLAYS 431 dmsetup status 432 mdadm --assemble --force /dev/md1 $OVERLAYS 433 history 434 dmsetup status 435 echo $OVERLAYS 436 mdadm --assemble --force /dev/md0 $OVERLAYS 437 cat /proc/partitions 438 mkdir /mnt/oldroot << look for inird mdadm files >> 484 echo $OVERLAYS 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg << cancelled out of 485, review instructions... >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg 487 fsck -n /dev/md0 488 mdadm --stop /dev/md0 489 echo $DEVICES 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES 491 dmsetup status 492 ls 493 rm overlay-* 494 ls 495 parallel losetup -d ::: /dev/loop[0-9]* 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 497 parallel truncate -s300G overlay-{/} ::: $DEVICES 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 499 dmsetup status 500 /sbin/reboot 501 history 502 dmsetup status 503 mount 504 cat /proc/partitions 505 nano /etc/fstab 506 mount /mnt/backup/ 507 ls /mnt/backup/ 508 rm /mnt/backup/ 509 rm /mnt/backup/overlay-sd* 510 emacs setupOverlay & 511 ps auxww | grep emacs 512 kill 65017 513 ls /dev/loo* 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' 515 echo $DEVICES 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 517 ls /dev/loo* 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES 519 ls 520 rm overlay-sd* 521 cd /mnt/bak 522 cd /mnt/backup/ 523 ls 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES 525 ls -la 526 blockdev --getsize /dev/sdb 527 man losetup 528 man losetup 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 530 dmsetup status 531 history | grep mdadm 532 history 533 dmsetup status 534 history | grep dmsetup 535 dmsetup status 536 dmsetup remove sdg 537 dmsetup ls --tree 538 lsof 539 dmsetup ls --tre 540 dmsetup ls --tree 541 lsof | grep -i sdg 542 lsof | grep -i sdf 543 history |grep dmsetup | less 544 dmsetup status 545 history > ~plsander/Documents/raidIssues/joblog On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: > > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > encountering a puzzling situation. > > dmsetup is failing to return. > > I don't think you need to use dmsetup in your case, but can you post > *all* the commands you ran before you got to this point, and the > output of > > cat /proc/mdstat > > as well? Thinking on this some more, you might need to actually also > add: > > --assume-clean > > to the 'mdadm create ....' string, since you don't want it to zero the > array or anything. > > Sorry for not remembering this at the time! > > So if you can, please just start over from scratch, showing the setup > of the loop devices, the overlayfs setup, and the building the RAID6 > array, along with the cat /proc/mdstat after you do the initial build. > > John > > P.S. For those who hated my email citing tool, I pulled it out for > now. Only citing with > now. :-) > > > root@superior:/mnt/backup# dmsetup status > > sdg: 0 5860533168 snapshot 16/8388608000 16 > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > sde: 0 5860533168 snapshot 16/8388608000 16 > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > dmsetup remove sdg runs for hours. > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > > > dmsetup status shows: > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > sde: 0 5860533168 snapshot 16/8388608000 16 > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > dmsetup ls --tree > > root@superior:/mnt/backup# dmsetup ls --tree > > sdf (253:3) > > ├─ (7:3) > > └─ (8:80) > > sde (253:1) > > ├─ (7:1) > > └─ (8:64) > > sdd (253:2) > > ├─ (7:2) > > └─ (8:48) > > sdc (253:0) > > ├─ (7:0) > > └─ (8:32) > > sdb (253:5) > > ├─ (7:5) > > └─ (8:16) > > > any suggestions? > > > > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: > >> > > >> > And the victory conditions would be a mountable file system that passes a fsck? > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy > >> yourself it looks good, too ... > >> > >> Cheers, > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-02 14:56 ` Peter Sanders @ 2022-09-02 18:52 ` Peter Sanders 2022-09-02 19:12 ` John Stoffel 0 siblings, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-09-02 18:52 UTC (permalink / raw) To: John Stoffel; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid Question on restarting from scratch... How to reset to the starting point? dmsetup, both for remove and create of the overlay seems to be hanging. On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: > > contents of /proc/mdstat > > root@superior:/mnt/backup# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > unused devices: <none> > root@superior:/mnt/backup# > > > > Here are the steps I ran (minus some mounting other devices and > looking around for mdadm tracks on the old os disk) > > 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > echo /dev/{1}) > 411 apt install parallel > 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > echo /dev/{1}) > 413 echo $DEVICES > 414 cat /proc/partitions > 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > 417 echo $DEVICES > 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > {#}' ::: $DEVICES > 419 ls /dev/loop* > 420 dc > 421 cd /mnt/backup/ > 422 ls > 423 parallel truncate -s300G overlay-{/} ::: $DEVICES > 424 ls > 425 ls -la > 426 df -h > 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 428 ls /dev/mapper/ > 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) > 430 echo $OVERLAYS > 431 dmsetup status > 432 mdadm --assemble --force /dev/md1 $OVERLAYS > 433 history > 434 dmsetup status > 435 echo $OVERLAYS > 436 mdadm --assemble --force /dev/md0 $OVERLAYS > 437 cat /proc/partitions > 438 mkdir /mnt/oldroot > << look for inird mdadm files >> > 484 echo $OVERLAYS > 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > /dev/mapper/sdg > << cancelled out of 485, review instructions... >> > 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > /dev/mapper/sdg > 487 fsck -n /dev/md0 > 488 mdadm --stop /dev/md0 > 489 echo $DEVICES > 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES > 491 dmsetup status > 492 ls > 493 rm overlay-* > 494 ls > 495 parallel losetup -d ::: /dev/loop[0-9]* > 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > {#}' ::: $DEVICES > 497 parallel truncate -s300G overlay-{/} ::: $DEVICES > 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 499 dmsetup status > 500 /sbin/reboot > 501 history > 502 dmsetup status > 503 mount > 504 cat /proc/partitions > 505 nano /etc/fstab > 506 mount /mnt/backup/ > 507 ls /mnt/backup/ > 508 rm /mnt/backup/ > 509 rm /mnt/backup/overlay-sd* > 510 emacs setupOverlay & > 511 ps auxww | grep emacs > 512 kill 65017 > 513 ls /dev/loo* > 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' > 515 echo $DEVICES > 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b > 7 {#}' ::: $DEVICES > 517 ls /dev/loo* > 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES > 519 ls > 520 rm overlay-sd* > 521 cd /mnt/bak > 522 cd /mnt/backup/ > 523 ls > 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES > 525 ls -la > 526 blockdev --getsize /dev/sdb > 527 man losetup > 528 man losetup > 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > create {/}' ::: $DEVICES > 530 dmsetup status > 531 history | grep mdadm > 532 history > 533 dmsetup status > 534 history | grep dmsetup > 535 dmsetup status > 536 dmsetup remove sdg > 537 dmsetup ls --tree > 538 lsof > 539 dmsetup ls --tre > 540 dmsetup ls --tree > 541 lsof | grep -i sdg > 542 lsof | grep -i sdf > 543 history |grep dmsetup | less > 544 dmsetup status > 545 history > ~plsander/Documents/raidIssues/joblog > > On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: > > > > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > > > encountering a puzzling situation. > > > dmsetup is failing to return. > > > > I don't think you need to use dmsetup in your case, but can you post > > *all* the commands you ran before you got to this point, and the > > output of > > > > cat /proc/mdstat > > > > as well? Thinking on this some more, you might need to actually also > > add: > > > > --assume-clean > > > > to the 'mdadm create ....' string, since you don't want it to zero the > > array or anything. > > > > Sorry for not remembering this at the time! > > > > So if you can, please just start over from scratch, showing the setup > > of the loop devices, the overlayfs setup, and the building the RAID6 > > array, along with the cat /proc/mdstat after you do the initial build. > > > > John > > > > P.S. For those who hated my email citing tool, I pulled it out for > > now. Only citing with > now. :-) > > > > > root@superior:/mnt/backup# dmsetup status > > > sdg: 0 5860533168 snapshot 16/8388608000 16 > > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > > sde: 0 5860533168 snapshot 16/8388608000 16 > > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > > > dmsetup remove sdg runs for hours. > > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > > > > > dmsetup status shows: > > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > > sde: 0 5860533168 snapshot 16/8388608000 16 > > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > > > dmsetup ls --tree > > > root@superior:/mnt/backup# dmsetup ls --tree > > > sdf (253:3) > > > ├─ (7:3) > > > └─ (8:80) > > > sde (253:1) > > > ├─ (7:1) > > > └─ (8:64) > > > sdd (253:2) > > > ├─ (7:2) > > > └─ (8:48) > > > sdc (253:0) > > > ├─ (7:0) > > > └─ (8:32) > > > sdb (253:5) > > > ├─ (7:5) > > > └─ (8:16) > > > > > any suggestions? > > > > > > > > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > > >> > > >> On 30/08/2022 14:27, Peter Sanders wrote: > > >> > > > >> > And the victory conditions would be a mountable file system that passes a fsck? > > >> > > >> Yes. Just make sure you delve through the file system a bit and satisfy > > >> yourself it looks good, too ... > > >> > > >> Cheers, > > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-02 18:52 ` Peter Sanders @ 2022-09-02 19:12 ` John Stoffel 2022-09-03 0:39 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-09-02 19:12 UTC (permalink / raw) To: Peter Sanders; +Cc: John Stoffel, Wols Lists, Eyal Lebedinsky, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: Peter, please include the output of all the commands, not just the commands themselves. See my comments below. > Question on restarting from scratch... > How to reset to the starting point? I think you need to blow away the loop devices and re-create them. Or at least blow away the dmsetup devices you just created. It might be quickest to just reboot. What OS are you using for the recovery? Is it a recent live image? Sorry for asking so many questions... some of this is new to me too. > dmsetup, both for remove and create of the overlay seems to be hanging. > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: >> >> contents of /proc/mdstat >> >> root@superior:/mnt/backup# cat /proc/mdstat >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] >> [raid4] [raid10] >> unused devices: <none> >> root@superior:/mnt/backup# >> >> >> >> Here are the steps I ran (minus some mounting other devices and >> looking around for mdadm tracks on the old os disk) >> >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> echo /dev/{1}) >> 411 apt install parallel >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> echo /dev/{1}) >> 413 echo $DEVICES So you found no MD RAID super blocks on any of the base devices. You can skip this step moving forward. >> 414 cat /proc/partitions >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" >> 417 echo $DEVICES >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> {#}' ::: $DEVICES >> 419 ls /dev/loop* Can you show the output of all these commands, not just the commands please? >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> create {/}' ::: $DEVICES >> 428 ls /dev/mapper/ This is some key output to view. >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) >> 430 echo $OVERLAYS What are the overlays? >> 431 dmsetup status What did this command show? >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS And here is where I think you need to put --assume-clean when using 'create' command instead. It's not going to assemble anything because the info was wiped. I *think* you really want: mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS And once you do this above command and it comes back, do: cat /proc/mdstat and show all the output please! >> 433 history >> 434 dmsetup status >> 435 echo $OVERLAYS >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS >> 437 cat /proc/partitions >> 438 mkdir /mnt/oldroot >> << look for inird mdadm files >> >> 484 echo $OVERLAYS >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> /dev/mapper/sdg I'm confused here, what is the difference between the md1 you assembled above, and the md0 you're doing here? >> << cancelled out of 485, review instructions... >> >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> /dev/mapper/sdg >> 487 fsck -n /dev/md0 And what output did you get here? Did it find a filesystem? You might want to try: blkid /dev/md0 >> 488 mdadm --stop /dev/md0 >> 489 echo $DEVICES >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES >> 491 dmsetup status This all worked properly? No errors? I gave up after this because it's not clear what the results really are. If you don't find a filesystem that fsck's cleanly, then you should just need to stop the array, then re-create it but shuffle the order of the devices. Instead of disk in order of "sdb sdc sdd... sdN", you would try the order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the list of devices? With six disks, you have I think 6 factorial options to try. Which is alot of options to go though, and why you need to automate this more. But also keep a log and show the output! John >> 492 ls >> 493 rm overlay-* >> 494 ls >> 495 parallel losetup -d ::: /dev/loop[0-9]* >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> {#}' ::: $DEVICES >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> create {/}' ::: $DEVICES >> 499 dmsetup status >> 500 /sbin/reboot >> 501 history >> 502 dmsetup status >> 503 mount >> 504 cat /proc/partitions >> 505 nano /etc/fstab >> 506 mount /mnt/backup/ >> 507 ls /mnt/backup/ >> 508 rm /mnt/backup/ >> 509 rm /mnt/backup/overlay-sd* >> 510 emacs setupOverlay & >> 511 ps auxww | grep emacs >> 512 kill 65017 >> 513 ls /dev/loo* >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' >> 515 echo $DEVICES >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b >> 7 {#}' ::: $DEVICES >> 517 ls /dev/loo* >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> 519 ls >> 520 rm overlay-sd* >> 521 cd /mnt/bak >> 522 cd /mnt/backup/ >> 523 ls >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> 525 ls -la >> 526 blockdev --getsize /dev/sdb >> 527 man losetup >> 528 man losetup >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> create {/}' ::: $DEVICES >> 530 dmsetup status >> 531 history | grep mdadm >> 532 history >> 533 dmsetup status >> 534 history | grep dmsetup >> 535 dmsetup status >> 536 dmsetup remove sdg >> 537 dmsetup ls --tree >> 538 lsof >> 539 dmsetup ls --tre >> 540 dmsetup ls --tree >> 541 lsof | grep -i sdg >> 542 lsof | grep -i sdf >> 543 history |grep dmsetup | less >> 544 dmsetup status >> 545 history > ~plsander/Documents/raidIssues/joblog >> >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: >> > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> > >> > > encountering a puzzling situation. >> > > dmsetup is failing to return. >> > >> > I don't think you need to use dmsetup in your case, but can you post >> > *all* the commands you ran before you got to this point, and the >> > output of >> > >> > cat /proc/mdstat >> > >> > as well? Thinking on this some more, you might need to actually also >> > add: >> > >> > --assume-clean >> > >> > to the 'mdadm create ....' string, since you don't want it to zero the >> > array or anything. >> > >> > Sorry for not remembering this at the time! >> > >> > So if you can, please just start over from scratch, showing the setup >> > of the loop devices, the overlayfs setup, and the building the RAID6 >> > array, along with the cat /proc/mdstat after you do the initial build. >> > >> > John >> > >> > P.S. For those who hated my email citing tool, I pulled it out for >> > now. Only citing with > now. :-) >> > >> > > root@superior:/mnt/backup# dmsetup status >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > dmsetup remove sdg runs for hours. >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. >> > >> > > dmsetup status shows: >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > dmsetup ls --tree >> > > root@superior:/mnt/backup# dmsetup ls --tree >> > > sdf (253:3) >> > > ├─ (7:3) >> > > └─ (8:80) >> > > sde (253:1) >> > > ├─ (7:1) >> > > └─ (8:64) >> > > sdd (253:2) >> > > ├─ (7:2) >> > > └─ (8:48) >> > > sdc (253:0) >> > > ├─ (7:0) >> > > └─ (8:32) >> > > sdb (253:5) >> > > ├─ (7:5) >> > > └─ (8:16) >> > >> > > any suggestions? >> > >> > >> > >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: >> > >> >> > >> On 30/08/2022 14:27, Peter Sanders wrote: >> > >> > >> > >> > And the victory conditions would be a mountable file system that passes a fsck? >> > >> >> > >> Yes. Just make sure you delve through the file system a bit and satisfy >> > >> yourself it looks good, too ... >> > >> >> > >> Cheers, >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-02 19:12 ` John Stoffel @ 2022-09-03 0:39 ` Peter Sanders 2022-09-03 5:51 ` Peter Sanders 2022-09-05 19:25 ` John Stoffel 0 siblings, 2 replies; 29+ messages in thread From: Peter Sanders @ 2022-09-03 0:39 UTC (permalink / raw) To: John Stoffel; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid Repeat of run 1 plsander@superior:~$ su - Password: root@superior:~# cat /proc/partitions major minor #blocks name 259 0 250059096 nvme0n1 259 1 496640 nvme0n1p1 259 2 1 nvme0n1p2 259 3 63475712 nvme0n1p5 259 4 97654784 nvme0n1p6 259 5 37888 nvme0n1p7 259 6 86913024 nvme0n1p8 259 7 1474560 nvme0n1p9 8 16 2930266584 sdb 8 80 2930266584 sdf 8 0 1953514584 sda 8 1 1953513472 sda1 8 32 2930266584 sdc 8 96 2930266584 sdg 8 64 2930266584 sde 8 48 2930266584 sdd 11 0 1048575 sr0 root@superior:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none> root@superior:~# DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" root@superior:~# echo $DEVICES /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg root@superior:~# parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES root@superior:~# ls /dev/lo log loop2 loop4 loop6 loop1 loop3 loop5 loop-control root@superior:~# ls /dev/lo* /dev/log /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop-control root@superior:~# ls -l /dev/loop* brw-rw---- 1 root root 7, 1 Sep 2 20:30 /dev/loop1 brw-rw---- 1 root root 7, 2 Sep 2 20:30 /dev/loop2 brw-rw---- 1 root root 7, 3 Sep 2 20:30 /dev/loop3 brw-rw---- 1 root root 7, 4 Sep 2 20:30 /dev/loop4 brw-rw---- 1 root root 7, 5 Sep 2 20:30 /dev/loop5 brw-rw---- 1 root root 7, 6 Sep 2 20:30 /dev/loop6 crw-rw---- 1 root disk 10, 237 Sep 2 20:22 /dev/loop-control root@superior:~# cd /mnt/backup/ root@superior:/mnt/backup# parallel truncate -s4000G overlay-{/} ::: $DEVICES root@superior:/mnt/backup# ls -l total 16 drwx------ 2 root root 16384 Aug 28 18:50 lost+found -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdb -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdc -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdd -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sde -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdf -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdg root@superior:/mnt/backup# rm over* root@superior:/mnt/backup# parallel truncate -s300G overlay-{/} ::: $DEVICES root@superior:/mnt/backup# ls -la total 24 drwxr-xr-x 3 root root 4096 Sep 2 20:31 . drwxr-xr-x 7 root root 4096 Aug 29 09:17 .. drwx------ 2 root root 16384 Aug 28 18:50 lost+found -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdb -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdc -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdd -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sde -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdf -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdg root@superior:/mnt/backup# dmsetup status No devices found root@superior:/mnt/backup# date Fri 02 Sep 2022 08:32:11 PM EDT root@superior:/mnt/backup# parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES root@superior:/mnt/backup# date Fri 02 Sep 2022 08:32:20 PM EDT root@superior:/mnt/backup# dmsetup status sdg: 0 5860533168 snapshot 16/629145600 16 sdf: 0 5860533168 snapshot 16/629145600 16 sde: 0 5860533168 snapshot 16/629145600 16 sdd: 0 5860533168 snapshot 16/629145600 16 sdc: 0 5860533168 snapshot 16/629145600 16 sdb: 0 5860533168 snapshot 16/629145600 16 root@superior:/mnt/backup# OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) root@superior:/mnt/backup# echo $OVERLAYS /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg root@superior:/mnt/backup# mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS mdadm: partition table exists on /dev/mapper/sdb mdadm: partition table exists on /dev/mapper/sdc mdadm: partition table exists on /dev/mapper/sdc but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdd mdadm: partition table exists on /dev/mapper/sdd but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sde mdadm: partition table exists on /dev/mapper/sde but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdf mdadm: partition table exists on /dev/mapper/sdf but will be lost or meaningless after creating array mdadm: partition table exists on /dev/mapper/sdg mdadm: partition table exists on /dev/mapper/sdg but will be lost or meaningless after creating array Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started. root@superior:/mnt/backup# ls -l /dev/md* brw-rw---- 1 root disk 9, 1 Sep 2 20:34 /dev/md1 root@superior:/mnt/backup# fsck /dev/md1 fsck from util-linux 2.36.1 e2fsck 1.46.2 (28-Feb-2021) ext2fs_open2: Bad magic number in super-block fsck.ext2: Superblock invalid, trying backup blocks... fsck.ext2: Bad magic number in super-block while trying to open /dev/md1 The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem. If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> or e2fsck -b 32768 <device> root@superior:/mnt/backup# blkid /dev/md1 root@superior:/mnt/backup# root@superior:/mnt/backup# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid6 dm-3[5] dm-2[4] dm-1[3] dm-5[2] dm-0[1] dm-4[0] 11720536064 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none> root@superior:/mnt/backup# Some questions - - is the easiest 'reset for next run' to reboot and rebuild? On Fri, Sep 2, 2022 at 3:12 PM John Stoffel <john@stoffel.org> wrote: > > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > Peter, please include the output of all the commands, not just the > commands themselves. See my comments below. > > > > Question on restarting from scratch... > > How to reset to the starting point? > > I think you need to blow away the loop devices and re-create them. > > Or at least blow away the dmsetup devices you just created. > > It might be quickest to just reboot. What OS are you using for the > recovery? Is it a recent live image? Sorry for asking so many > questions... some of this is new to me too. > > > > dmsetup, both for remove and create of the overlay seems to be hanging. > > > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: > >> > >> contents of /proc/mdstat > >> > >> root@superior:/mnt/backup# cat /proc/mdstat > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > >> [raid4] [raid10] > >> unused devices: <none> > >> root@superior:/mnt/backup# > >> > >> > >> > >> Here are the steps I ran (minus some mounting other devices and > >> looking around for mdadm tracks on the old os disk) > >> > >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > >> echo /dev/{1}) > >> 411 apt install parallel > >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > >> echo /dev/{1}) > >> 413 echo $DEVICES > > So you found no MD RAID super blocks on any of the base devices. You > can skip this step moving forward. > > >> 414 cat /proc/partitions > >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > >> 417 echo $DEVICES > >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > >> {#}' ::: $DEVICES > >> 419 ls /dev/loop* > > Can you show the output of all these commands, not just the commands please? > > >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES > > >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> create {/}' ::: $DEVICES > >> 428 ls /dev/mapper/ > > This is some key output to view. > > >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) > >> 430 echo $OVERLAYS > > What are the overlays? > > >> 431 dmsetup status > > What did this command show? > > >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS > > And here is where I think you need to put --assume-clean when using > 'create' command instead. It's not going to assemble anything because > the info was wiped. I *think* you really want: > > mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS > > And once you do this above command and it comes back, do: > > cat /proc/mdstat > > and show all the output please! > > >> 433 history > >> 434 dmsetup status > >> 435 echo $OVERLAYS > >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS > >> 437 cat /proc/partitions > >> 438 mkdir /mnt/oldroot > >> << look for inird mdadm files >> > >> 484 echo $OVERLAYS > >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > >> /dev/mapper/sdg > > I'm confused here, what is the difference between the md1 you > assembled above, and the md0 you're doing here? > > >> << cancelled out of 485, review instructions... >> > >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > >> /dev/mapper/sdg > >> 487 fsck -n /dev/md0 > > And what output did you get here? Did it find a filesystem? You might want > to try: > > blkid /dev/md0 > > > >> 488 mdadm --stop /dev/md0 > >> 489 echo $DEVICES > >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES > >> 491 dmsetup status > > This all worked properly? No errors? > > I gave up after this because it's not clear what the results really > are. If you don't find a filesystem that fsck's cleanly, then you > should just need to stop the array, then re-create it but shuffle the > order of the devices. > > Instead of disk in order of "sdb sdc sdd... sdN", you would try the > order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the > list of devices? With six disks, you have I think 6 factorial options > to try. Which is alot of options to go though, and why you need to > automate this more. But also keep a log and show the output! > > John > > > >> 492 ls > >> 493 rm overlay-* > >> 494 ls > >> 495 parallel losetup -d ::: /dev/loop[0-9]* > >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > >> {#}' ::: $DEVICES > >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES > >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> create {/}' ::: $DEVICES > >> 499 dmsetup status > >> 500 /sbin/reboot > >> 501 history > >> 502 dmsetup status > >> 503 mount > >> 504 cat /proc/partitions > >> 505 nano /etc/fstab > >> 506 mount /mnt/backup/ > >> 507 ls /mnt/backup/ > >> 508 rm /mnt/backup/ > >> 509 rm /mnt/backup/overlay-sd* > >> 510 emacs setupOverlay & > >> 511 ps auxww | grep emacs > >> 512 kill 65017 > >> 513 ls /dev/loo* > >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' > >> 515 echo $DEVICES > >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b > >> 7 {#}' ::: $DEVICES > >> 517 ls /dev/loo* > >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES > >> 519 ls > >> 520 rm overlay-sd* > >> 521 cd /mnt/bak > >> 522 cd /mnt/backup/ > >> 523 ls > >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES > >> 525 ls -la > >> 526 blockdev --getsize /dev/sdb > >> 527 man losetup > >> 528 man losetup > >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> create {/}' ::: $DEVICES > >> 530 dmsetup status > >> 531 history | grep mdadm > >> 532 history > >> 533 dmsetup status > >> 534 history | grep dmsetup > >> 535 dmsetup status > >> 536 dmsetup remove sdg > >> 537 dmsetup ls --tree > >> 538 lsof > >> 539 dmsetup ls --tre > >> 540 dmsetup ls --tree > >> 541 lsof | grep -i sdg > >> 542 lsof | grep -i sdf > >> 543 history |grep dmsetup | less > >> 544 dmsetup status > >> 545 history > ~plsander/Documents/raidIssues/joblog > >> > >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: > >> > > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > >> > > >> > > encountering a puzzling situation. > >> > > dmsetup is failing to return. > >> > > >> > I don't think you need to use dmsetup in your case, but can you post > >> > *all* the commands you ran before you got to this point, and the > >> > output of > >> > > >> > cat /proc/mdstat > >> > > >> > as well? Thinking on this some more, you might need to actually also > >> > add: > >> > > >> > --assume-clean > >> > > >> > to the 'mdadm create ....' string, since you don't want it to zero the > >> > array or anything. > >> > > >> > Sorry for not remembering this at the time! > >> > > >> > So if you can, please just start over from scratch, showing the setup > >> > of the loop devices, the overlayfs setup, and the building the RAID6 > >> > array, along with the cat /proc/mdstat after you do the initial build. > >> > > >> > John > >> > > >> > P.S. For those who hated my email citing tool, I pulled it out for > >> > now. Only citing with > now. :-) > >> > > >> > > root@superior:/mnt/backup# dmsetup status > >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > >> > > >> > > dmsetup remove sdg runs for hours. > >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > >> > > >> > > dmsetup status shows: > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > >> > > >> > > dmsetup ls --tree > >> > > root@superior:/mnt/backup# dmsetup ls --tree > >> > > sdf (253:3) > >> > > ├─ (7:3) > >> > > └─ (8:80) > >> > > sde (253:1) > >> > > ├─ (7:1) > >> > > └─ (8:64) > >> > > sdd (253:2) > >> > > ├─ (7:2) > >> > > └─ (8:48) > >> > > sdc (253:0) > >> > > ├─ (7:0) > >> > > └─ (8:32) > >> > > sdb (253:5) > >> > > ├─ (7:5) > >> > > └─ (8:16) > >> > > >> > > any suggestions? > >> > > >> > > >> > > >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > >> > >> > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: > >> > >> > > >> > >> > And the victory conditions would be a mountable file system that passes a fsck? > >> > >> > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy > >> > >> yourself it looks good, too ... > >> > >> > >> > >> Cheers, > >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-03 0:39 ` Peter Sanders @ 2022-09-03 5:51 ` Peter Sanders 2022-09-05 19:36 ` John Stoffel 2022-09-05 19:25 ` John Stoffel 1 sibling, 1 reply; 29+ messages in thread From: Peter Sanders @ 2022-09-03 5:51 UTC (permalink / raw) To: John Stoffel; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid tried removing the setup: root@superior:/mnt/backup# mdadm --stop /dev/md1 mdadm: stopped /dev/md1 root@superior:/mnt/backup# parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES ^C (ran for an hour before cancel... ) root@superior:/mnt/backup# dmsetup status No devices found root@superior:/mnt/backupls lost+found overlay-sdb overlay-sdc overlay-sdd overlay-sde overlay-sdf overlay-sdg root@superior:/mnt/backup# rm overlay-sd* root@superior:/mnt/backup# ls /dev/loop ls: cannot access '/dev/loop': No such file or directory root@superior:/mnt/backup# ls /dev/loop* /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 root@superior:/mnt/backup# parallel losetup -d ::: /dev/loop[0-9]* losetup: /dev/loop6: detach failed: No such device or address losetup: /dev/loop7: detach failed: No such device or address root@superior:/mnt/backup# ls /dev/loop* /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 root@superior:/mnt/backup# ls -la /dev/lo* lrwxrwxrwx 1 root root 28 Sep 2 20:22 /dev/log -> /run/systemd/journal/dev-log brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control root@superior:/mnt/backup# losetup -d ::: /dev/loop70 losetup: :::: failed to use device: No such device root@superior:/mnt/backup# losetup -d ::: /dev/loop7 losetup: :::: failed to use device: No such device root@superior:/mnt/backup# losetup -d /dev/loop7 losetup: /dev/loop7: detach failed: No such device or address root@superior:/mnt/backup# ls -la /dev/loop* brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control root@superior:/mnt/backup# losetup /dev/loop7 losetup: /dev/loop7: No such file or directory root@superior:/mnt/backup# losetup /dev/loop5 losetup: /dev/loop5: No such file or directory root@superior:/mnt/backup# not sure why losetup cannot see the existing /dev/loopx devices. On Fri, Sep 2, 2022 at 8:39 PM Peter Sanders <plsander@gmail.com> wrote: > > Repeat of run 1 > > plsander@superior:~$ su - > Password: > root@superior:~# cat /proc/partitions > major minor #blocks name > > 259 0 250059096 nvme0n1 > 259 1 496640 nvme0n1p1 > 259 2 1 nvme0n1p2 > 259 3 63475712 nvme0n1p5 > 259 4 97654784 nvme0n1p6 > 259 5 37888 nvme0n1p7 > 259 6 86913024 nvme0n1p8 > 259 7 1474560 nvme0n1p9 > 8 16 2930266584 sdb > 8 80 2930266584 sdf > 8 0 1953514584 sda > 8 1 1953513472 sda1 > 8 32 2930266584 sdc > 8 96 2930266584 sdg > 8 64 2930266584 sde > 8 48 2930266584 sdd > 11 0 1048575 sr0 > root@superior:~# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > unused devices: <none> > root@superior:~# DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > root@superior:~# echo $DEVICES > /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > root@superior:~# parallel 'test -e /dev/loop{#} || mknod -m 660 > /dev/loop{#} b 7 {#}' ::: $DEVICES > root@superior:~# ls /dev/lo > log loop2 loop4 loop6 > loop1 loop3 loop5 loop-control > root@superior:~# ls /dev/lo* > /dev/log /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 > /dev/loop6 /dev/loop-control > root@superior:~# ls -l /dev/loop* > brw-rw---- 1 root root 7, 1 Sep 2 20:30 /dev/loop1 > brw-rw---- 1 root root 7, 2 Sep 2 20:30 /dev/loop2 > brw-rw---- 1 root root 7, 3 Sep 2 20:30 /dev/loop3 > brw-rw---- 1 root root 7, 4 Sep 2 20:30 /dev/loop4 > brw-rw---- 1 root root 7, 5 Sep 2 20:30 /dev/loop5 > brw-rw---- 1 root root 7, 6 Sep 2 20:30 /dev/loop6 > crw-rw---- 1 root disk 10, 237 Sep 2 20:22 /dev/loop-control > root@superior:~# cd /mnt/backup/ > root@superior:/mnt/backup# parallel truncate -s4000G overlay-{/} ::: $DEVICES > root@superior:/mnt/backup# ls -l > total 16 > drwx------ 2 root root 16384 Aug 28 18:50 lost+found > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdb > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdc > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdd > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sde > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdf > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdg > root@superior:/mnt/backup# rm over* > root@superior:/mnt/backup# parallel truncate -s300G overlay-{/} ::: $DEVICES > root@superior:/mnt/backup# ls -la > total 24 > drwxr-xr-x 3 root root 4096 Sep 2 20:31 . > drwxr-xr-x 7 root root 4096 Aug 29 09:17 .. > drwx------ 2 root root 16384 Aug 28 18:50 lost+found > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdb > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdc > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdd > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sde > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdf > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdg > root@superior:/mnt/backup# dmsetup status > No devices found > root@superior:/mnt/backup# date > Fri 02 Sep 2022 08:32:11 PM EDT > root@superior:/mnt/backup# parallel 'size=$(blockdev --getsize {}); > loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} > $loop P 8 | dmsetup create {/}' ::: $DEVICES > root@superior:/mnt/backup# date > Fri 02 Sep 2022 08:32:20 PM EDT > root@superior:/mnt/backup# dmsetup status > sdg: 0 5860533168 snapshot 16/629145600 16 > sdf: 0 5860533168 snapshot 16/629145600 16 > sde: 0 5860533168 snapshot 16/629145600 16 > sdd: 0 5860533168 snapshot 16/629145600 16 > sdc: 0 5860533168 snapshot 16/629145600 16 > sdb: 0 5860533168 snapshot 16/629145600 16 > root@superior:/mnt/backup# OVERLAYS=$(parallel echo /dev/mapper/{/} > ::: $DEVICES) > root@superior:/mnt/backup# echo $OVERLAYS > /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde > /dev/mapper/sdf /dev/mapper/sdg > root@superior:/mnt/backup# mdadm --create /dev/md1 --level=raid6 -n 6 > --assume-clean $OVERLAYS > mdadm: partition table exists on /dev/mapper/sdb > mdadm: partition table exists on /dev/mapper/sdc > mdadm: partition table exists on /dev/mapper/sdc but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdd > mdadm: partition table exists on /dev/mapper/sdd but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sde > mdadm: partition table exists on /dev/mapper/sde but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdf > mdadm: partition table exists on /dev/mapper/sdf but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdg > mdadm: partition table exists on /dev/mapper/sdg but will be lost or > meaningless after creating array > Continue creating array? y > mdadm: Defaulting to version 1.2 metadata > mdadm: array /dev/md1 started. > root@superior:/mnt/backup# ls -l /dev/md* > brw-rw---- 1 root disk 9, 1 Sep 2 20:34 /dev/md1 > root@superior:/mnt/backup# fsck /dev/md1 > fsck from util-linux 2.36.1 > e2fsck 1.46.2 (28-Feb-2021) > ext2fs_open2: Bad magic number in super-block > fsck.ext2: Superblock invalid, trying backup blocks... > fsck.ext2: Bad magic number in super-block while trying to open /dev/md1 > > The superblock could not be read or does not describe a valid ext2/ext3/ext4 > filesystem. If the device is valid and it really contains an ext2/ext3/ext4 > filesystem (and not swap or ufs or something else), then the superblock > is corrupt, and you might try running e2fsck with an alternate superblock: > e2fsck -b 8193 <device> > or > e2fsck -b 32768 <device> > > root@superior:/mnt/backup# blkid /dev/md1 > root@superior:/mnt/backup# > root@superior:/mnt/backup# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > md1 : active raid6 dm-3[5] dm-2[4] dm-1[3] dm-5[2] dm-0[1] dm-4[0] > 11720536064 blocks super 1.2 level 6, 512k chunk, algorithm 2 > [6/6] [UUUUUU] > bitmap: 0/22 pages [0KB], 65536KB chunk > > unused devices: <none> > root@superior:/mnt/backup# > > Some questions - > - is the easiest 'reset for next run' to reboot and rebuild? > > > On Fri, Sep 2, 2022 at 3:12 PM John Stoffel <john@stoffel.org> wrote: > > > > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > > Peter, please include the output of all the commands, not just the > > commands themselves. See my comments below. > > > > > > > Question on restarting from scratch... > > > How to reset to the starting point? > > > > I think you need to blow away the loop devices and re-create them. > > > > Or at least blow away the dmsetup devices you just created. > > > > It might be quickest to just reboot. What OS are you using for the > > recovery? Is it a recent live image? Sorry for asking so many > > questions... some of this is new to me too. > > > > > > > dmsetup, both for remove and create of the overlay seems to be hanging. > > > > > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: > > >> > > >> contents of /proc/mdstat > > >> > > >> root@superior:/mnt/backup# cat /proc/mdstat > > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > > >> [raid4] [raid10] > > >> unused devices: <none> > > >> root@superior:/mnt/backup# > > >> > > >> > > >> > > >> Here are the steps I ran (minus some mounting other devices and > > >> looking around for mdadm tracks on the old os disk) > > >> > > >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > > >> echo /dev/{1}) > > >> 411 apt install parallel > > >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > > >> echo /dev/{1}) > > >> 413 echo $DEVICES > > > > So you found no MD RAID super blocks on any of the base devices. You > > can skip this step moving forward. > > > > >> 414 cat /proc/partitions > > >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > > >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > > >> 417 echo $DEVICES > > >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > > >> {#}' ::: $DEVICES > > >> 419 ls /dev/loop* > > > > Can you show the output of all these commands, not just the commands please? > > > > >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES > > > > >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > > >> create {/}' ::: $DEVICES > > >> 428 ls /dev/mapper/ > > > > This is some key output to view. > > > > >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) > > >> 430 echo $OVERLAYS > > > > What are the overlays? > > > > >> 431 dmsetup status > > > > What did this command show? > > > > >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS > > > > And here is where I think you need to put --assume-clean when using > > 'create' command instead. It's not going to assemble anything because > > the info was wiped. I *think* you really want: > > > > mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS > > > > And once you do this above command and it comes back, do: > > > > cat /proc/mdstat > > > > and show all the output please! > > > > >> 433 history > > >> 434 dmsetup status > > >> 435 echo $OVERLAYS > > >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS > > >> 437 cat /proc/partitions > > >> 438 mkdir /mnt/oldroot > > >> << look for inird mdadm files >> > > >> 484 echo $OVERLAYS > > >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > > >> /dev/mapper/sdg > > > > I'm confused here, what is the difference between the md1 you > > assembled above, and the md0 you're doing here? > > > > >> << cancelled out of 485, review instructions... >> > > >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > > >> /dev/mapper/sdg > > >> 487 fsck -n /dev/md0 > > > > And what output did you get here? Did it find a filesystem? You might want > > to try: > > > > blkid /dev/md0 > > > > > > >> 488 mdadm --stop /dev/md0 > > >> 489 echo $DEVICES > > >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES > > >> 491 dmsetup status > > > > This all worked properly? No errors? > > > > I gave up after this because it's not clear what the results really > > are. If you don't find a filesystem that fsck's cleanly, then you > > should just need to stop the array, then re-create it but shuffle the > > order of the devices. > > > > Instead of disk in order of "sdb sdc sdd... sdN", you would try the > > order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the > > list of devices? With six disks, you have I think 6 factorial options > > to try. Which is alot of options to go though, and why you need to > > automate this more. But also keep a log and show the output! > > > > John > > > > > > >> 492 ls > > >> 493 rm overlay-* > > >> 494 ls > > >> 495 parallel losetup -d ::: /dev/loop[0-9]* > > >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > > >> {#}' ::: $DEVICES > > >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES > > >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > > >> create {/}' ::: $DEVICES > > >> 499 dmsetup status > > >> 500 /sbin/reboot > > >> 501 history > > >> 502 dmsetup status > > >> 503 mount > > >> 504 cat /proc/partitions > > >> 505 nano /etc/fstab > > >> 506 mount /mnt/backup/ > > >> 507 ls /mnt/backup/ > > >> 508 rm /mnt/backup/ > > >> 509 rm /mnt/backup/overlay-sd* > > >> 510 emacs setupOverlay & > > >> 511 ps auxww | grep emacs > > >> 512 kill 65017 > > >> 513 ls /dev/loo* > > >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' > > >> 515 echo $DEVICES > > >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b > > >> 7 {#}' ::: $DEVICES > > >> 517 ls /dev/loo* > > >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES > > >> 519 ls > > >> 520 rm overlay-sd* > > >> 521 cd /mnt/bak > > >> 522 cd /mnt/backup/ > > >> 523 ls > > >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES > > >> 525 ls -la > > >> 526 blockdev --getsize /dev/sdb > > >> 527 man losetup > > >> 528 man losetup > > >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > > >> create {/}' ::: $DEVICES > > >> 530 dmsetup status > > >> 531 history | grep mdadm > > >> 532 history > > >> 533 dmsetup status > > >> 534 history | grep dmsetup > > >> 535 dmsetup status > > >> 536 dmsetup remove sdg > > >> 537 dmsetup ls --tree > > >> 538 lsof > > >> 539 dmsetup ls --tre > > >> 540 dmsetup ls --tree > > >> 541 lsof | grep -i sdg > > >> 542 lsof | grep -i sdf > > >> 543 history |grep dmsetup | less > > >> 544 dmsetup status > > >> 545 history > ~plsander/Documents/raidIssues/joblog > > >> > > >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: > > >> > > > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > >> > > > >> > > encountering a puzzling situation. > > >> > > dmsetup is failing to return. > > >> > > > >> > I don't think you need to use dmsetup in your case, but can you post > > >> > *all* the commands you ran before you got to this point, and the > > >> > output of > > >> > > > >> > cat /proc/mdstat > > >> > > > >> > as well? Thinking on this some more, you might need to actually also > > >> > add: > > >> > > > >> > --assume-clean > > >> > > > >> > to the 'mdadm create ....' string, since you don't want it to zero the > > >> > array or anything. > > >> > > > >> > Sorry for not remembering this at the time! > > >> > > > >> > So if you can, please just start over from scratch, showing the setup > > >> > of the loop devices, the overlayfs setup, and the building the RAID6 > > >> > array, along with the cat /proc/mdstat after you do the initial build. > > >> > > > >> > John > > >> > > > >> > P.S. For those who hated my email citing tool, I pulled it out for > > >> > now. Only citing with > now. :-) > > >> > > > >> > > root@superior:/mnt/backup# dmsetup status > > >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > >> > > > >> > > dmsetup remove sdg runs for hours. > > >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > > >> > > > >> > > dmsetup status shows: > > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > >> > > > >> > > dmsetup ls --tree > > >> > > root@superior:/mnt/backup# dmsetup ls --tree > > >> > > sdf (253:3) > > >> > > ├─ (7:3) > > >> > > └─ (8:80) > > >> > > sde (253:1) > > >> > > ├─ (7:1) > > >> > > └─ (8:64) > > >> > > sdd (253:2) > > >> > > ├─ (7:2) > > >> > > └─ (8:48) > > >> > > sdc (253:0) > > >> > > ├─ (7:0) > > >> > > └─ (8:32) > > >> > > sdb (253:5) > > >> > > ├─ (7:5) > > >> > > └─ (8:16) > > >> > > > >> > > any suggestions? > > >> > > > >> > > > >> > > > >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > > >> > >> > > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: > > >> > >> > > > >> > >> > And the victory conditions would be a mountable file system that passes a fsck? > > >> > >> > > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy > > >> > >> yourself it looks good, too ... > > >> > >> > > >> > >> Cheers, > > >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-03 5:51 ` Peter Sanders @ 2022-09-05 19:36 ` John Stoffel 2022-09-05 20:16 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: John Stoffel @ 2022-09-05 19:36 UTC (permalink / raw) To: Peter Sanders; +Cc: John Stoffel, Wols Lists, Eyal Lebedinsky, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > tried removing the setup: > root@superior:/mnt/backup# mdadm --stop /dev/md1 > mdadm: stopped /dev/md1 > root@superior:/mnt/backup# parallel 'dmsetup remove {/}; rm > overlay-{/}' ::: $DEVICES > ^C > (ran for an hour before cancel... ) > root@superior:/mnt/backup# dmsetup status > No devices found > root@superior:/mnt/backupls > lost+found overlay-sdb overlay-sdc overlay-sdd overlay-sde > overlay-sdf overlay-sdg > root@superior:/mnt/backup# rm overlay-sd* > root@superior:/mnt/backup# ls /dev/loop > ls: cannot access '/dev/loop': No such file or directory > root@superior:/mnt/backup# ls /dev/loop* > /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control > /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 > root@superior:/mnt/backup# parallel losetup -d ::: /dev/loop[0-9]* > losetup: /dev/loop6: detach failed: No such device or address > losetup: /dev/loop7: detach failed: No such device or address > root@superior:/mnt/backup# ls /dev/loop* > /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control > /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 > root@superior:/mnt/backup# ls -la /dev/lo* > lrwxrwxrwx 1 root root 28 Sep 2 20:22 /dev/log -> > /run/systemd/journal/dev-log > brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 > brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 > brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 > brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 > brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 > brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 > brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 > brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 > crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control > root@superior:/mnt/backup# losetup -d ::: /dev/loop70 > losetup: :::: failed to use device: No such device > root@superior:/mnt/backup# losetup -d ::: /dev/loop7 > losetup: :::: failed to use device: No such device > root@superior:/mnt/backup# losetup -d /dev/loop7 > losetup: /dev/loop7: detach failed: No such device or address > root@superior:/mnt/backup# ls -la /dev/loop* > brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 > brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 > brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 > brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 > brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 > brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 > brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 > brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 > crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control > root@superior:/mnt/backup# losetup /dev/loop7 > losetup: /dev/loop7: No such file or directory > root@superior:/mnt/backup# losetup /dev/loop5 > losetup: /dev/loop5: No such file or directory > root@superior:/mnt/backup# > not sure why losetup cannot see the existing /dev/loopx devices. I was reading the dmsetup man page and it said that if the devices are open, when you do a remove, it sorta fails them and then blocks them as unable to have more IO sent to them. But honestly I'm not an expert on dmsetup. But sure, try to reboot each time, but you also need to make sure the disks are in the same positions each time after reboot, so look at the serial numbers. lsscsi -g -l might be enough to give you unique or instead use: hdparm -i /dev/sda | grep SerialNo to get the info and keep track of which disk is in which order. John > On Fri, Sep 2, 2022 at 8:39 PM Peter Sanders <plsander@gmail.com> wrote: >> >> Repeat of run 1 >> >> plsander@superior:~$ su - >> Password: >> root@superior:~# cat /proc/partitions >> major minor #blocks name >> >> 259 0 250059096 nvme0n1 >> 259 1 496640 nvme0n1p1 >> 259 2 1 nvme0n1p2 >> 259 3 63475712 nvme0n1p5 >> 259 4 97654784 nvme0n1p6 >> 259 5 37888 nvme0n1p7 >> 259 6 86913024 nvme0n1p8 >> 259 7 1474560 nvme0n1p9 >> 8 16 2930266584 sdb >> 8 80 2930266584 sdf >> 8 0 1953514584 sda >> 8 1 1953513472 sda1 >> 8 32 2930266584 sdc >> 8 96 2930266584 sdg >> 8 64 2930266584 sde >> 8 48 2930266584 sdd >> 11 0 1048575 sr0 >> root@superior:~# cat /proc/mdstat >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] >> [raid4] [raid10] >> unused devices: <none> >> root@superior:~# DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" >> root@superior:~# echo $DEVICES >> /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg >> root@superior:~# parallel 'test -e /dev/loop{#} || mknod -m 660 >> /dev/loop{#} b 7 {#}' ::: $DEVICES >> root@superior:~# ls /dev/lo >> log loop2 loop4 loop6 >> loop1 loop3 loop5 loop-control >> root@superior:~# ls /dev/lo* >> /dev/log /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 >> /dev/loop6 /dev/loop-control >> root@superior:~# ls -l /dev/loop* >> brw-rw---- 1 root root 7, 1 Sep 2 20:30 /dev/loop1 >> brw-rw---- 1 root root 7, 2 Sep 2 20:30 /dev/loop2 >> brw-rw---- 1 root root 7, 3 Sep 2 20:30 /dev/loop3 >> brw-rw---- 1 root root 7, 4 Sep 2 20:30 /dev/loop4 >> brw-rw---- 1 root root 7, 5 Sep 2 20:30 /dev/loop5 >> brw-rw---- 1 root root 7, 6 Sep 2 20:30 /dev/loop6 >> crw-rw---- 1 root disk 10, 237 Sep 2 20:22 /dev/loop-control >> root@superior:~# cd /mnt/backup/ >> root@superior:/mnt/backup# parallel truncate -s4000G overlay-{/} ::: $DEVICES >> root@superior:/mnt/backup# ls -l >> total 16 >> drwx------ 2 root root 16384 Aug 28 18:50 lost+found >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdb >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdc >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdd >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sde >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdf >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdg >> root@superior:/mnt/backup# rm over* >> root@superior:/mnt/backup# parallel truncate -s300G overlay-{/} ::: $DEVICES >> root@superior:/mnt/backup# ls -la >> total 24 >> drwxr-xr-x 3 root root 4096 Sep 2 20:31 . >> drwxr-xr-x 7 root root 4096 Aug 29 09:17 .. >> drwx------ 2 root root 16384 Aug 28 18:50 lost+found >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdb >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdc >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdd >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sde >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdf >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdg >> root@superior:/mnt/backup# dmsetup status >> No devices found >> root@superior:/mnt/backup# date >> Fri 02 Sep 2022 08:32:11 PM EDT >> root@superior:/mnt/backup# parallel 'size=$(blockdev --getsize {}); >> loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} >> $loop P 8 | dmsetup create {/}' ::: $DEVICES >> root@superior:/mnt/backup# date >> Fri 02 Sep 2022 08:32:20 PM EDT >> root@superior:/mnt/backup# dmsetup status >> sdg: 0 5860533168 snapshot 16/629145600 16 >> sdf: 0 5860533168 snapshot 16/629145600 16 >> sde: 0 5860533168 snapshot 16/629145600 16 >> sdd: 0 5860533168 snapshot 16/629145600 16 >> sdc: 0 5860533168 snapshot 16/629145600 16 >> sdb: 0 5860533168 snapshot 16/629145600 16 >> root@superior:/mnt/backup# OVERLAYS=$(parallel echo /dev/mapper/{/} >> ::: $DEVICES) >> root@superior:/mnt/backup# echo $OVERLAYS >> /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde >> /dev/mapper/sdf /dev/mapper/sdg >> root@superior:/mnt/backup# mdadm --create /dev/md1 --level=raid6 -n 6 >> --assume-clean $OVERLAYS >> mdadm: partition table exists on /dev/mapper/sdb >> mdadm: partition table exists on /dev/mapper/sdc >> mdadm: partition table exists on /dev/mapper/sdc but will be lost or >> meaningless after creating array >> mdadm: partition table exists on /dev/mapper/sdd >> mdadm: partition table exists on /dev/mapper/sdd but will be lost or >> meaningless after creating array >> mdadm: partition table exists on /dev/mapper/sde >> mdadm: partition table exists on /dev/mapper/sde but will be lost or >> meaningless after creating array >> mdadm: partition table exists on /dev/mapper/sdf >> mdadm: partition table exists on /dev/mapper/sdf but will be lost or >> meaningless after creating array >> mdadm: partition table exists on /dev/mapper/sdg >> mdadm: partition table exists on /dev/mapper/sdg but will be lost or >> meaningless after creating array >> Continue creating array? y >> mdadm: Defaulting to version 1.2 metadata >> mdadm: array /dev/md1 started. >> root@superior:/mnt/backup# ls -l /dev/md* >> brw-rw---- 1 root disk 9, 1 Sep 2 20:34 /dev/md1 >> root@superior:/mnt/backup# fsck /dev/md1 >> fsck from util-linux 2.36.1 >> e2fsck 1.46.2 (28-Feb-2021) >> ext2fs_open2: Bad magic number in super-block >> fsck.ext2: Superblock invalid, trying backup blocks... >> fsck.ext2: Bad magic number in super-block while trying to open /dev/md1 >> >> The superblock could not be read or does not describe a valid ext2/ext3/ext4 >> filesystem. If the device is valid and it really contains an ext2/ext3/ext4 >> filesystem (and not swap or ufs or something else), then the superblock >> is corrupt, and you might try running e2fsck with an alternate superblock: >> e2fsck -b 8193 <device> >> or >> e2fsck -b 32768 <device> >> >> root@superior:/mnt/backup# blkid /dev/md1 >> root@superior:/mnt/backup# >> root@superior:/mnt/backup# cat /proc/mdstat >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] >> [raid4] [raid10] >> md1 : active raid6 dm-3[5] dm-2[4] dm-1[3] dm-5[2] dm-0[1] dm-4[0] >> 11720536064 blocks super 1.2 level 6, 512k chunk, algorithm 2 >> [6/6] [UUUUUU] >> bitmap: 0/22 pages [0KB], 65536KB chunk >> >> unused devices: <none> >> root@superior:/mnt/backup# >> >> Some questions - >> - is the easiest 'reset for next run' to reboot and rebuild? >> >> >> On Fri, Sep 2, 2022 at 3:12 PM John Stoffel <john@stoffel.org> wrote: >> > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> > >> > Peter, please include the output of all the commands, not just the >> > commands themselves. See my comments below. >> > >> > >> > > Question on restarting from scratch... >> > > How to reset to the starting point? >> > >> > I think you need to blow away the loop devices and re-create them. >> > >> > Or at least blow away the dmsetup devices you just created. >> > >> > It might be quickest to just reboot. What OS are you using for the >> > recovery? Is it a recent live image? Sorry for asking so many >> > questions... some of this is new to me too. >> > >> > >> > > dmsetup, both for remove and create of the overlay seems to be hanging. >> > >> > > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: >> > >> >> > >> contents of /proc/mdstat >> > >> >> > >> root@superior:/mnt/backup# cat /proc/mdstat >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] >> > >> [raid4] [raid10] >> > >> unused devices: <none> >> > >> root@superior:/mnt/backup# >> > >> >> > >> >> > >> >> > >> Here are the steps I ran (minus some mounting other devices and >> > >> looking around for mdadm tracks on the old os disk) >> > >> >> > >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> > >> echo /dev/{1}) >> > >> 411 apt install parallel >> > >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> > >> echo /dev/{1}) >> > >> 413 echo $DEVICES >> > >> > So you found no MD RAID super blocks on any of the base devices. You >> > can skip this step moving forward. >> > >> > >> 414 cat /proc/partitions >> > >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg >> > >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" >> > >> 417 echo $DEVICES >> > >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> > >> {#}' ::: $DEVICES >> > >> 419 ls /dev/loop* >> > >> > Can you show the output of all these commands, not just the commands please? >> > >> > >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES >> > >> > >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> > >> create {/}' ::: $DEVICES >> > >> 428 ls /dev/mapper/ >> > >> > This is some key output to view. >> > >> > >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) >> > >> 430 echo $OVERLAYS >> > >> > What are the overlays? >> > >> > >> 431 dmsetup status >> > >> > What did this command show? >> > >> > >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS >> > >> > And here is where I think you need to put --assume-clean when using >> > 'create' command instead. It's not going to assemble anything because >> > the info was wiped. I *think* you really want: >> > >> > mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS >> > >> > And once you do this above command and it comes back, do: >> > >> > cat /proc/mdstat >> > >> > and show all the output please! >> > >> > >> 433 history >> > >> 434 dmsetup status >> > >> 435 echo $OVERLAYS >> > >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS >> > >> 437 cat /proc/partitions >> > >> 438 mkdir /mnt/oldroot >> > >> << look for inird mdadm files >> >> > >> 484 echo $OVERLAYS >> > >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> > >> /dev/mapper/sdg >> > >> > I'm confused here, what is the difference between the md1 you >> > assembled above, and the md0 you're doing here? >> > >> > >> << cancelled out of 485, review instructions... >> >> > >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> > >> /dev/mapper/sdg >> > >> 487 fsck -n /dev/md0 >> > >> > And what output did you get here? Did it find a filesystem? You might want >> > to try: >> > >> > blkid /dev/md0 >> > >> > >> > >> 488 mdadm --stop /dev/md0 >> > >> 489 echo $DEVICES >> > >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES >> > >> 491 dmsetup status >> > >> > This all worked properly? No errors? >> > >> > I gave up after this because it's not clear what the results really >> > are. If you don't find a filesystem that fsck's cleanly, then you >> > should just need to stop the array, then re-create it but shuffle the >> > order of the devices. >> > >> > Instead of disk in order of "sdb sdc sdd... sdN", you would try the >> > order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the >> > list of devices? With six disks, you have I think 6 factorial options >> > to try. Which is alot of options to go though, and why you need to >> > automate this more. But also keep a log and show the output! >> > >> > John >> > >> > >> > >> 492 ls >> > >> 493 rm overlay-* >> > >> 494 ls >> > >> 495 parallel losetup -d ::: /dev/loop[0-9]* >> > >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> > >> {#}' ::: $DEVICES >> > >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES >> > >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> > >> create {/}' ::: $DEVICES >> > >> 499 dmsetup status >> > >> 500 /sbin/reboot >> > >> 501 history >> > >> 502 dmsetup status >> > >> 503 mount >> > >> 504 cat /proc/partitions >> > >> 505 nano /etc/fstab >> > >> 506 mount /mnt/backup/ >> > >> 507 ls /mnt/backup/ >> > >> 508 rm /mnt/backup/ >> > >> 509 rm /mnt/backup/overlay-sd* >> > >> 510 emacs setupOverlay & >> > >> 511 ps auxww | grep emacs >> > >> 512 kill 65017 >> > >> 513 ls /dev/loo* >> > >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' >> > >> 515 echo $DEVICES >> > >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b >> > >> 7 {#}' ::: $DEVICES >> > >> 517 ls /dev/loo* >> > >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> > >> 519 ls >> > >> 520 rm overlay-sd* >> > >> 521 cd /mnt/bak >> > >> 522 cd /mnt/backup/ >> > >> 523 ls >> > >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> > >> 525 ls -la >> > >> 526 blockdev --getsize /dev/sdb >> > >> 527 man losetup >> > >> 528 man losetup >> > >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> > >> create {/}' ::: $DEVICES >> > >> 530 dmsetup status >> > >> 531 history | grep mdadm >> > >> 532 history >> > >> 533 dmsetup status >> > >> 534 history | grep dmsetup >> > >> 535 dmsetup status >> > >> 536 dmsetup remove sdg >> > >> 537 dmsetup ls --tree >> > >> 538 lsof >> > >> 539 dmsetup ls --tre >> > >> 540 dmsetup ls --tree >> > >> 541 lsof | grep -i sdg >> > >> 542 lsof | grep -i sdf >> > >> 543 history |grep dmsetup | less >> > >> 544 dmsetup status >> > >> 545 history > ~plsander/Documents/raidIssues/joblog >> > >> >> > >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: >> > >> > >> > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> > >> > >> > >> > > encountering a puzzling situation. >> > >> > > dmsetup is failing to return. >> > >> > >> > >> > I don't think you need to use dmsetup in your case, but can you post >> > >> > *all* the commands you ran before you got to this point, and the >> > >> > output of >> > >> > >> > >> > cat /proc/mdstat >> > >> > >> > >> > as well? Thinking on this some more, you might need to actually also >> > >> > add: >> > >> > >> > >> > --assume-clean >> > >> > >> > >> > to the 'mdadm create ....' string, since you don't want it to zero the >> > >> > array or anything. >> > >> > >> > >> > Sorry for not remembering this at the time! >> > >> > >> > >> > So if you can, please just start over from scratch, showing the setup >> > >> > of the loop devices, the overlayfs setup, and the building the RAID6 >> > >> > array, along with the cat /proc/mdstat after you do the initial build. >> > >> > >> > >> > John >> > >> > >> > >> > P.S. For those who hated my email citing tool, I pulled it out for >> > >> > now. Only citing with > now. :-) >> > >> > >> > >> > > root@superior:/mnt/backup# dmsetup status >> > >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> > >> > >> > >> > > dmsetup remove sdg runs for hours. >> > >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. >> > >> > >> > >> > > dmsetup status shows: >> > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> > >> > >> > >> > > dmsetup ls --tree >> > >> > > root@superior:/mnt/backup# dmsetup ls --tree >> > >> > > sdf (253:3) >> > >> > > ├─ (7:3) >> > >> > > └─ (8:80) >> > >> > > sde (253:1) >> > >> > > ├─ (7:1) >> > >> > > └─ (8:64) >> > >> > > sdd (253:2) >> > >> > > ├─ (7:2) >> > >> > > └─ (8:48) >> > >> > > sdc (253:0) >> > >> > > ├─ (7:0) >> > >> > > └─ (8:32) >> > >> > > sdb (253:5) >> > >> > > ├─ (7:5) >> > >> > > └─ (8:16) >> > >> > >> > >> > > any suggestions? >> > >> > >> > >> > >> > >> > >> > >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: >> > >> > >> >> > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: >> > >> > >> > >> > >> > >> > And the victory conditions would be a mountable file system that passes a fsck? >> > >> > >> >> > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy >> > >> > >> yourself it looks good, too ... >> > >> > >> >> > >> > >> Cheers, >> > >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-05 19:36 ` John Stoffel @ 2022-09-05 20:16 ` Peter Sanders 0 siblings, 0 replies; 29+ messages in thread From: Peter Sanders @ 2022-09-05 20:16 UTC (permalink / raw) To: John Stoffel; +Cc: Wols Lists, Eyal Lebedinsky, linux-raid Wrote a script to do the setup: ----- #! /usr/bin/bash cd /mnt/backup; DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg'; echo $DEVICES; date; parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES; ls -l /dev/loop*; date; parallel truncate -s300G overlay-{/} ::: $DEVICES; ls -l /mnt/backup; date; parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES; date; OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES); dmsetup status; ----- Ran that, then ran pvs, etc Mon 05 Sep 2022 04:05:50 PM EDT sdg: 0 5860533168 snapshot 16/629145600 16 sdf: 0 5860533168 snapshot 16/629145600 16 sde: 0 5860533168 snapshot 16/629145600 16 sdd: 0 5860533168 snapshot 16/629145600 16 sdc: 0 5860533168 snapshot 16/629145600 16 sdb: 0 5860533168 snapshot 16/629145600 16 root@superior:~# pvs pvs pvscan root@superior:~# pvs root@superior:~# vg vgcfgbackup vgck vgdisplay vgimport vgmknodes vgrename vgsplit vgcfgrestore vgconvert vgexport vgimportclone vgreduce vgs vgchange vgcreate vgextend vgmerge vgremove vgscan root@superior:~# vgs root@superior:~# pvs -a PV VG Fmt Attr PSize PFree /dev/loop0 --- 0 0 /dev/loop1 --- 0 0 /dev/loop2 --- 0 0 /dev/loop3 --- 0 0 /dev/loop4 --- 0 0 /dev/loop5 --- 0 0 /dev/nvme0n1 --- 0 0 /dev/nvme0n1p1 --- 0 0 /dev/nvme0n1p5 --- 0 0 /dev/nvme0n1p6 --- 0 0 /dev/nvme0n1p7 --- 0 0 /dev/nvme0n1p8 --- 0 0 /dev/nvme0n1p9 --- 0 0 /dev/sda1 --- 0 0 root@superior:~# vgs -a root@superior:~# vgs -a root@superior:~# dmsetup remove No device specified. Command failed. root@superior:~# echo $DEVICES root@superior:~# dmsetup remove /dev/sdb Device sdb not found Command failed. root@superior:~# cat /proc/partitions major minor #blocks name 259 0 250059096 nvme0n1 259 1 496640 nvme0n1p1 259 2 1 nvme0n1p2 259 3 63475712 nvme0n1p5 259 4 97654784 nvme0n1p6 259 5 37888 nvme0n1p7 259 6 86913024 nvme0n1p8 259 7 1474560 nvme0n1p9 11 0 1048575 sr0 8 48 2930266584 sdd 8 16 2930266584 sdb 8 80 2930266584 sdf 8 0 1953514584 sda 8 1 1953513472 sda1 8 32 2930266584 sdc 8 64 2930266584 sde 8 96 2930266584 sdg 7 0 314572800 loop0 7 1 314572800 loop1 7 2 314572800 loop2 7 3 314572800 loop3 7 4 314572800 loop4 7 5 314572800 loop5 253 0 2930266584 dm-0 253 1 2930266584 dm-1 253 2 2930266584 dm-2 253 3 2930266584 dm-3 253 4 2930266584 dm-4 253 5 2930266584 dm-5 root@superior:~# Should I be calling dmsetup remove for /dev/sdx or /dev/dm-n? Since dmsetup remove fails to find /dev/sdb... for device order -- will ls -l /dev/disk/by-id work? These first nine lines appear to be show serial numbers. root@superior:~# ls -l /dev/disk/by-id/ total 0 lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-MAD_DOG_LS-DVDRW_TSH652M -> ../../sr0 lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-TOSHIBA_HDWD130_477ABEJAS -> ../../sdg lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-TOSHIBA_HDWD130_477ALBNAS -> ../../sdb lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-TOSHIBA_HDWD130_Y7211KPAS -> ../../sdc lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-WDC_WD20EARX-00PASB0_WD-WMAZA6843376 -> ../../sda lrwxrwxrwx 1 root root 10 Sep 5 16:02 ata-WDC_WD20EARX-00PASB0_WD-WMAZA6843376-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-WDC_WD30EZRX-00D8PB0_WD-WCC4N0091255 -> ../../sde lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-WDC_WD30EZRX-00DC0B0_WD-WCC1T0668790 -> ../../sdd lrwxrwxrwx 1 root root 9 Sep 5 16:02 ata-WDC_WD30EZRX-00MMMB0_WD-WCAWZ2669166 -> ../../sdf lrwxrwxrwx 1 root root 13 Sep 5 16:02 nvme-eui.6479a75970c003ee -> ../../nvme0n1 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part5 -> ../../nvme0n1p5 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part6 -> ../../nvme0n1p6 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part7 -> ../../nvme0n1p7 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part8 -> ../../nvme0n1p8 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-eui.6479a75970c003ee-part9 -> ../../nvme0n1p9 lrwxrwxrwx 1 root root 13 Sep 5 16:02 nvme-PCIe_SSD_21112925606047 -> ../../nvme0n1 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part5 -> ../../nvme0n1p5 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part6 -> ../../nvme0n1p6 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part7 -> ../../nvme0n1p7 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part8 -> ../../nvme0n1p8 lrwxrwxrwx 1 root root 15 Sep 5 16:02 nvme-PCIe_SSD_21112925606047-part9 -> ../../nvme0n1p9 lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x5000039fe6d2ce25 -> ../../sdg lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x5000039fe6d2e832 -> ../../sdb lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x5000039fe6dca946 -> ../../sdc lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x50014ee15a13d994 -> ../../sdf lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x50014ee206a417d2 -> ../../sda lrwxrwxrwx 1 root root 10 Sep 5 16:02 wwn-0x50014ee206a417d2-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x50014ee2084d406a -> ../../sdd lrwxrwxrwx 1 root root 9 Sep 5 16:02 wwn-0x50014ee2b3d4ffa1 -> ../../sde On Mon, Sep 5, 2022 at 3:36 PM John Stoffel <john@stoffel.org> wrote: > > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > > > tried removing the setup: > > root@superior:/mnt/backup# mdadm --stop /dev/md1 > > mdadm: stopped /dev/md1 > > root@superior:/mnt/backup# parallel 'dmsetup remove {/}; rm > > overlay-{/}' ::: $DEVICES > > ^C > > > (ran for an hour before cancel... ) > > > root@superior:/mnt/backup# dmsetup status > > No devices found > > root@superior:/mnt/backupls > > lost+found overlay-sdb overlay-sdc overlay-sdd overlay-sde > > overlay-sdf overlay-sdg > > root@superior:/mnt/backup# rm overlay-sd* > > root@superior:/mnt/backup# ls /dev/loop > > ls: cannot access '/dev/loop': No such file or directory > > root@superior:/mnt/backup# ls /dev/loop* > > /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control > > /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 > > root@superior:/mnt/backup# parallel losetup -d ::: /dev/loop[0-9]* > > losetup: /dev/loop6: detach failed: No such device or address > > losetup: /dev/loop7: detach failed: No such device or address > > root@superior:/mnt/backup# ls /dev/loop* > > /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control > > /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 > > root@superior:/mnt/backup# ls -la /dev/lo* > > lrwxrwxrwx 1 root root 28 Sep 2 20:22 /dev/log -> > > /run/systemd/journal/dev-log > > brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 > > brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 > > brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 > > brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 > > brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 > > brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 > > brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 > > brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 > > crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control > > root@superior:/mnt/backup# losetup -d ::: /dev/loop70 > > losetup: :::: failed to use device: No such device > > root@superior:/mnt/backup# losetup -d ::: /dev/loop7 > > losetup: :::: failed to use device: No such device > > root@superior:/mnt/backup# losetup -d /dev/loop7 > > losetup: /dev/loop7: detach failed: No such device or address > > root@superior:/mnt/backup# ls -la /dev/loop* > > brw-rw---- 1 root disk 7, 0 Sep 2 20:32 /dev/loop0 > > brw-rw---- 1 root disk 7, 1 Sep 2 20:32 /dev/loop1 > > brw-rw---- 1 root disk 7, 2 Sep 2 20:32 /dev/loop2 > > brw-rw---- 1 root disk 7, 3 Sep 2 20:32 /dev/loop3 > > brw-rw---- 1 root disk 7, 4 Sep 2 20:32 /dev/loop4 > > brw-rw---- 1 root disk 7, 5 Sep 2 20:32 /dev/loop5 > > brw-rw---- 1 root disk 7, 6 Sep 2 20:32 /dev/loop6 > > brw-rw---- 1 root disk 7, 7 Sep 2 20:32 /dev/loop7 > > crw-rw---- 1 root disk 10, 237 Sep 2 20:32 /dev/loop-control > > root@superior:/mnt/backup# losetup /dev/loop7 > > losetup: /dev/loop7: No such file or directory > > root@superior:/mnt/backup# losetup /dev/loop5 > > losetup: /dev/loop5: No such file or directory > > root@superior:/mnt/backup# > > > > not sure why losetup cannot see the existing /dev/loopx devices. > > I was reading the dmsetup man page and it said that if the devices are > open, when you do a remove, it sorta fails them and then blocks them > as unable to have more IO sent to them. But honestly I'm not an > expert on dmsetup. > > But sure, try to reboot each time, but you also need to make sure the > disks are in the same positions each time after reboot, so look at the > serial numbers. > > lsscsi -g -l > > might be enough to give you unique or instead use: > > hdparm -i /dev/sda | grep SerialNo > > to get the info and keep track of which disk is in which order. > > John > > > > > On Fri, Sep 2, 2022 at 8:39 PM Peter Sanders <plsander@gmail.com> wrote: > >> > >> Repeat of run 1 > >> > >> plsander@superior:~$ su - > >> Password: > >> root@superior:~# cat /proc/partitions > >> major minor #blocks name > >> > >> 259 0 250059096 nvme0n1 > >> 259 1 496640 nvme0n1p1 > >> 259 2 1 nvme0n1p2 > >> 259 3 63475712 nvme0n1p5 > >> 259 4 97654784 nvme0n1p6 > >> 259 5 37888 nvme0n1p7 > >> 259 6 86913024 nvme0n1p8 > >> 259 7 1474560 nvme0n1p9 > >> 8 16 2930266584 sdb > >> 8 80 2930266584 sdf > >> 8 0 1953514584 sda > >> 8 1 1953513472 sda1 > >> 8 32 2930266584 sdc > >> 8 96 2930266584 sdg > >> 8 64 2930266584 sde > >> 8 48 2930266584 sdd > >> 11 0 1048575 sr0 > >> root@superior:~# cat /proc/mdstat > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > >> [raid4] [raid10] > >> unused devices: <none> > >> root@superior:~# DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > >> root@superior:~# echo $DEVICES > >> /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > >> root@superior:~# parallel 'test -e /dev/loop{#} || mknod -m 660 > >> /dev/loop{#} b 7 {#}' ::: $DEVICES > >> root@superior:~# ls /dev/lo > >> log loop2 loop4 loop6 > >> loop1 loop3 loop5 loop-control > >> root@superior:~# ls /dev/lo* > >> /dev/log /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 > >> /dev/loop6 /dev/loop-control > >> root@superior:~# ls -l /dev/loop* > >> brw-rw---- 1 root root 7, 1 Sep 2 20:30 /dev/loop1 > >> brw-rw---- 1 root root 7, 2 Sep 2 20:30 /dev/loop2 > >> brw-rw---- 1 root root 7, 3 Sep 2 20:30 /dev/loop3 > >> brw-rw---- 1 root root 7, 4 Sep 2 20:30 /dev/loop4 > >> brw-rw---- 1 root root 7, 5 Sep 2 20:30 /dev/loop5 > >> brw-rw---- 1 root root 7, 6 Sep 2 20:30 /dev/loop6 > >> crw-rw---- 1 root disk 10, 237 Sep 2 20:22 /dev/loop-control > >> root@superior:~# cd /mnt/backup/ > >> root@superior:/mnt/backup# parallel truncate -s4000G overlay-{/} ::: $DEVICES > >> root@superior:/mnt/backup# ls -l > >> total 16 > >> drwx------ 2 root root 16384 Aug 28 18:50 lost+found > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdb > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdc > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdd > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sde > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdf > >> -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdg > >> root@superior:/mnt/backup# rm over* > >> root@superior:/mnt/backup# parallel truncate -s300G overlay-{/} ::: $DEVICES > >> root@superior:/mnt/backup# ls -la > >> total 24 > >> drwxr-xr-x 3 root root 4096 Sep 2 20:31 . > >> drwxr-xr-x 7 root root 4096 Aug 29 09:17 .. > >> drwx------ 2 root root 16384 Aug 28 18:50 lost+found > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdb > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdc > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdd > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sde > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdf > >> -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdg > >> root@superior:/mnt/backup# dmsetup status > >> No devices found > >> root@superior:/mnt/backup# date > >> Fri 02 Sep 2022 08:32:11 PM EDT > >> root@superior:/mnt/backup# parallel 'size=$(blockdev --getsize {}); > >> loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} > >> $loop P 8 | dmsetup create {/}' ::: $DEVICES > >> root@superior:/mnt/backup# date > >> Fri 02 Sep 2022 08:32:20 PM EDT > >> root@superior:/mnt/backup# dmsetup status > >> sdg: 0 5860533168 snapshot 16/629145600 16 > >> sdf: 0 5860533168 snapshot 16/629145600 16 > >> sde: 0 5860533168 snapshot 16/629145600 16 > >> sdd: 0 5860533168 snapshot 16/629145600 16 > >> sdc: 0 5860533168 snapshot 16/629145600 16 > >> sdb: 0 5860533168 snapshot 16/629145600 16 > >> root@superior:/mnt/backup# OVERLAYS=$(parallel echo /dev/mapper/{/} > >> ::: $DEVICES) > >> root@superior:/mnt/backup# echo $OVERLAYS > >> /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde > >> /dev/mapper/sdf /dev/mapper/sdg > >> root@superior:/mnt/backup# mdadm --create /dev/md1 --level=raid6 -n 6 > >> --assume-clean $OVERLAYS > >> mdadm: partition table exists on /dev/mapper/sdb > >> mdadm: partition table exists on /dev/mapper/sdc > >> mdadm: partition table exists on /dev/mapper/sdc but will be lost or > >> meaningless after creating array > >> mdadm: partition table exists on /dev/mapper/sdd > >> mdadm: partition table exists on /dev/mapper/sdd but will be lost or > >> meaningless after creating array > >> mdadm: partition table exists on /dev/mapper/sde > >> mdadm: partition table exists on /dev/mapper/sde but will be lost or > >> meaningless after creating array > >> mdadm: partition table exists on /dev/mapper/sdf > >> mdadm: partition table exists on /dev/mapper/sdf but will be lost or > >> meaningless after creating array > >> mdadm: partition table exists on /dev/mapper/sdg > >> mdadm: partition table exists on /dev/mapper/sdg but will be lost or > >> meaningless after creating array > >> Continue creating array? y > >> mdadm: Defaulting to version 1.2 metadata > >> mdadm: array /dev/md1 started. > >> root@superior:/mnt/backup# ls -l /dev/md* > >> brw-rw---- 1 root disk 9, 1 Sep 2 20:34 /dev/md1 > >> root@superior:/mnt/backup# fsck /dev/md1 > >> fsck from util-linux 2.36.1 > >> e2fsck 1.46.2 (28-Feb-2021) > >> ext2fs_open2: Bad magic number in super-block > >> fsck.ext2: Superblock invalid, trying backup blocks... > >> fsck.ext2: Bad magic number in super-block while trying to open /dev/md1 > >> > >> The superblock could not be read or does not describe a valid ext2/ext3/ext4 > >> filesystem. If the device is valid and it really contains an ext2/ext3/ext4 > >> filesystem (and not swap or ufs or something else), then the superblock > >> is corrupt, and you might try running e2fsck with an alternate superblock: > >> e2fsck -b 8193 <device> > >> or > >> e2fsck -b 32768 <device> > >> > >> root@superior:/mnt/backup# blkid /dev/md1 > >> root@superior:/mnt/backup# > >> root@superior:/mnt/backup# cat /proc/mdstat > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > >> [raid4] [raid10] > >> md1 : active raid6 dm-3[5] dm-2[4] dm-1[3] dm-5[2] dm-0[1] dm-4[0] > >> 11720536064 blocks super 1.2 level 6, 512k chunk, algorithm 2 > >> [6/6] [UUUUUU] > >> bitmap: 0/22 pages [0KB], 65536KB chunk > >> > >> unused devices: <none> > >> root@superior:/mnt/backup# > >> > >> Some questions - > >> - is the easiest 'reset for next run' to reboot and rebuild? > >> > >> > >> On Fri, Sep 2, 2022 at 3:12 PM John Stoffel <john@stoffel.org> wrote: > >> > > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > >> > > >> > Peter, please include the output of all the commands, not just the > >> > commands themselves. See my comments below. > >> > > >> > > >> > > Question on restarting from scratch... > >> > > How to reset to the starting point? > >> > > >> > I think you need to blow away the loop devices and re-create them. > >> > > >> > Or at least blow away the dmsetup devices you just created. > >> > > >> > It might be quickest to just reboot. What OS are you using for the > >> > recovery? Is it a recent live image? Sorry for asking so many > >> > questions... some of this is new to me too. > >> > > >> > > >> > > dmsetup, both for remove and create of the overlay seems to be hanging. > >> > > >> > > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: > >> > >> > >> > >> contents of /proc/mdstat > >> > >> > >> > >> root@superior:/mnt/backup# cat /proc/mdstat > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > >> > >> [raid4] [raid10] > >> > >> unused devices: <none> > >> > >> root@superior:/mnt/backup# > >> > >> > >> > >> > >> > >> > >> > >> Here are the steps I ran (minus some mounting other devices and > >> > >> looking around for mdadm tracks on the old os disk) > >> > >> > >> > >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > >> > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > >> > >> echo /dev/{1}) > >> > >> 411 apt install parallel > >> > >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} > >> > >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' > >> > >> echo /dev/{1}) > >> > >> 413 echo $DEVICES > >> > > >> > So you found no MD RAID super blocks on any of the base devices. You > >> > can skip this step moving forward. > >> > > >> > >> 414 cat /proc/partitions > >> > >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > >> > >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > >> > >> 417 echo $DEVICES > >> > >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > >> > >> {#}' ::: $DEVICES > >> > >> 419 ls /dev/loop* > >> > > >> > Can you show the output of all these commands, not just the commands please? > >> > > >> > >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES > >> > > >> > >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> > >> create {/}' ::: $DEVICES > >> > >> 428 ls /dev/mapper/ > >> > > >> > This is some key output to view. > >> > > >> > >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) > >> > >> 430 echo $OVERLAYS > >> > > >> > What are the overlays? > >> > > >> > >> 431 dmsetup status > >> > > >> > What did this command show? > >> > > >> > >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS > >> > > >> > And here is where I think you need to put --assume-clean when using > >> > 'create' command instead. It's not going to assemble anything because > >> > the info was wiped. I *think* you really want: > >> > > >> > mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS > >> > > >> > And once you do this above command and it comes back, do: > >> > > >> > cat /proc/mdstat > >> > > >> > and show all the output please! > >> > > >> > >> 433 history > >> > >> 434 dmsetup status > >> > >> 435 echo $OVERLAYS > >> > >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS > >> > >> 437 cat /proc/partitions > >> > >> 438 mkdir /mnt/oldroot > >> > >> << look for inird mdadm files >> > >> > >> 484 echo $OVERLAYS > >> > >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > >> > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > >> > >> /dev/mapper/sdg > >> > > >> > I'm confused here, what is the difference between the md1 you > >> > assembled above, and the md0 you're doing here? > >> > > >> > >> << cancelled out of 485, review instructions... >> > >> > >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb > >> > >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf > >> > >> /dev/mapper/sdg > >> > >> 487 fsck -n /dev/md0 > >> > > >> > And what output did you get here? Did it find a filesystem? You might want > >> > to try: > >> > > >> > blkid /dev/md0 > >> > > >> > > >> > >> 488 mdadm --stop /dev/md0 > >> > >> 489 echo $DEVICES > >> > >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES > >> > >> 491 dmsetup status > >> > > >> > This all worked properly? No errors? > >> > > >> > I gave up after this because it's not clear what the results really > >> > are. If you don't find a filesystem that fsck's cleanly, then you > >> > should just need to stop the array, then re-create it but shuffle the > >> > order of the devices. > >> > > >> > Instead of disk in order of "sdb sdc sdd... sdN", you would try the > >> > order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the > >> > list of devices? With six disks, you have I think 6 factorial options > >> > to try. Which is alot of options to go though, and why you need to > >> > automate this more. But also keep a log and show the output! > >> > > >> > John > >> > > >> > > >> > >> 492 ls > >> > >> 493 rm overlay-* > >> > >> 494 ls > >> > >> 495 parallel losetup -d ::: /dev/loop[0-9]* > >> > >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 > >> > >> {#}' ::: $DEVICES > >> > >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES > >> > >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> > >> create {/}' ::: $DEVICES > >> > >> 499 dmsetup status > >> > >> 500 /sbin/reboot > >> > >> 501 history > >> > >> 502 dmsetup status > >> > >> 503 mount > >> > >> 504 cat /proc/partitions > >> > >> 505 nano /etc/fstab > >> > >> 506 mount /mnt/backup/ > >> > >> 507 ls /mnt/backup/ > >> > >> 508 rm /mnt/backup/ > >> > >> 509 rm /mnt/backup/overlay-sd* > >> > >> 510 emacs setupOverlay & > >> > >> 511 ps auxww | grep emacs > >> > >> 512 kill 65017 > >> > >> 513 ls /dev/loo* > >> > >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' > >> > >> 515 echo $DEVICES > >> > >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b > >> > >> 7 {#}' ::: $DEVICES > >> > >> 517 ls /dev/loo* > >> > >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES > >> > >> 519 ls > >> > >> 520 rm overlay-sd* > >> > >> 521 cd /mnt/bak > >> > >> 522 cd /mnt/backup/ > >> > >> 523 ls > >> > >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES > >> > >> 525 ls -la > >> > >> 526 blockdev --getsize /dev/sdb > >> > >> 527 man losetup > >> > >> 528 man losetup > >> > >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f > >> > >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup > >> > >> create {/}' ::: $DEVICES > >> > >> 530 dmsetup status > >> > >> 531 history | grep mdadm > >> > >> 532 history > >> > >> 533 dmsetup status > >> > >> 534 history | grep dmsetup > >> > >> 535 dmsetup status > >> > >> 536 dmsetup remove sdg > >> > >> 537 dmsetup ls --tree > >> > >> 538 lsof > >> > >> 539 dmsetup ls --tre > >> > >> 540 dmsetup ls --tree > >> > >> 541 lsof | grep -i sdg > >> > >> 542 lsof | grep -i sdf > >> > >> 543 history |grep dmsetup | less > >> > >> 544 dmsetup status > >> > >> 545 history > ~plsander/Documents/raidIssues/joblog > >> > >> > >> > >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: > >> > >> > > >> > >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > >> > >> > > >> > >> > > encountering a puzzling situation. > >> > >> > > dmsetup is failing to return. > >> > >> > > >> > >> > I don't think you need to use dmsetup in your case, but can you post > >> > >> > *all* the commands you ran before you got to this point, and the > >> > >> > output of > >> > >> > > >> > >> > cat /proc/mdstat > >> > >> > > >> > >> > as well? Thinking on this some more, you might need to actually also > >> > >> > add: > >> > >> > > >> > >> > --assume-clean > >> > >> > > >> > >> > to the 'mdadm create ....' string, since you don't want it to zero the > >> > >> > array or anything. > >> > >> > > >> > >> > Sorry for not remembering this at the time! > >> > >> > > >> > >> > So if you can, please just start over from scratch, showing the setup > >> > >> > of the loop devices, the overlayfs setup, and the building the RAID6 > >> > >> > array, along with the cat /proc/mdstat after you do the initial build. > >> > >> > > >> > >> > John > >> > >> > > >> > >> > P.S. For those who hated my email citing tool, I pulled it out for > >> > >> > now. Only citing with > now. :-) > >> > >> > > >> > >> > > root@superior:/mnt/backup# dmsetup status > >> > >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > >> > >> > > dmsetup remove sdg runs for hours. > >> > >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > >> > >> > > >> > >> > > dmsetup status shows: > >> > >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sde: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 > >> > >> > > >> > >> > > dmsetup ls --tree > >> > >> > > root@superior:/mnt/backup# dmsetup ls --tree > >> > >> > > sdf (253:3) > >> > >> > > ├─ (7:3) > >> > >> > > └─ (8:80) > >> > >> > > sde (253:1) > >> > >> > > ├─ (7:1) > >> > >> > > └─ (8:64) > >> > >> > > sdd (253:2) > >> > >> > > ├─ (7:2) > >> > >> > > └─ (8:48) > >> > >> > > sdc (253:0) > >> > >> > > ├─ (7:0) > >> > >> > > └─ (8:32) > >> > >> > > sdb (253:5) > >> > >> > > ├─ (7:5) > >> > >> > > └─ (8:16) > >> > >> > > >> > >> > > any suggestions? > >> > >> > > >> > >> > > >> > >> > > >> > >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: > >> > >> > >> > >> > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: > >> > >> > >> > > >> > >> > >> > And the victory conditions would be a mountable file system that passes a fsck? > >> > >> > >> > >> > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy > >> > >> > >> yourself it looks good, too ... > >> > >> > >> > >> > >> > >> Cheers, > >> > >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-09-03 0:39 ` Peter Sanders 2022-09-03 5:51 ` Peter Sanders @ 2022-09-05 19:25 ` John Stoffel 1 sibling, 0 replies; 29+ messages in thread From: John Stoffel @ 2022-09-05 19:25 UTC (permalink / raw) To: Peter Sanders; +Cc: John Stoffel, Wols Lists, Eyal Lebedinsky, linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: > Repeat of run 1 > plsander@superior:~$ su - > Password: > root@superior:~# cat /proc/partitions > major minor #blocks name > 259 0 250059096 nvme0n1 > 259 1 496640 nvme0n1p1 > 259 2 1 nvme0n1p2 > 259 3 63475712 nvme0n1p5 > 259 4 97654784 nvme0n1p6 > 259 5 37888 nvme0n1p7 > 259 6 86913024 nvme0n1p8 > 259 7 1474560 nvme0n1p9 > 8 16 2930266584 sdb > 8 80 2930266584 sdf > 8 0 1953514584 sda > 8 1 1953513472 sda1 > 8 32 2930266584 sdc > 8 96 2930266584 sdg > 8 64 2930266584 sde > 8 48 2930266584 sdd > 11 0 1048575 sr0 > root@superior:~# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > unused devices: <none> > root@superior:~# DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" > root@superior:~# echo $DEVICES > /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg > root@superior:~# parallel 'test -e /dev/loop{#} || mknod -m 660 > /dev/loop{#} b 7 {#}' ::: $DEVICES > root@superior:~# ls /dev/lo > log loop2 loop4 loop6 > loop1 loop3 loop5 loop-control > root@superior:~# ls /dev/lo* > /dev/log /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 > /dev/loop6 /dev/loop-control > root@superior:~# ls -l /dev/loop* > brw-rw---- 1 root root 7, 1 Sep 2 20:30 /dev/loop1 > brw-rw---- 1 root root 7, 2 Sep 2 20:30 /dev/loop2 > brw-rw---- 1 root root 7, 3 Sep 2 20:30 /dev/loop3 > brw-rw---- 1 root root 7, 4 Sep 2 20:30 /dev/loop4 > brw-rw---- 1 root root 7, 5 Sep 2 20:30 /dev/loop5 > brw-rw---- 1 root root 7, 6 Sep 2 20:30 /dev/loop6 > crw-rw---- 1 root disk 10, 237 Sep 2 20:22 /dev/loop-control > root@superior:~# cd /mnt/backup/ > root@superior:/mnt/backup# parallel truncate -s4000G overlay-{/} ::: $DEVICES > root@superior:/mnt/backup# ls -l > total 16 > drwx------ 2 root root 16384 Aug 28 18:50 lost+found > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdb > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdc > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdd > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sde > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdf > -rw-r--r-- 1 root root 4294967296000 Sep 2 20:31 overlay-sdg > root@superior:/mnt/backup# rm over* So why did you remove these overlays? Too big for some reason? > root@superior:/mnt/backup# parallel truncate -s300G overlay-{/} ::: $DEVICES > root@superior:/mnt/backup# ls -la > total 24 > drwxr-xr-x 3 root root 4096 Sep 2 20:31 . > drwxr-xr-x 7 root root 4096 Aug 29 09:17 .. > drwx------ 2 root root 16384 Aug 28 18:50 lost+found > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdb > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdc > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdd > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sde > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdf > -rw-r--r-- 1 root root 322122547200 Sep 2 20:31 overlay-sdg > root@superior:/mnt/backup# dmsetup status > No devices found > root@superior:/mnt/backup# date > Fri 02 Sep 2022 08:32:11 PM EDT This looks good. > root@superior:/mnt/backup# parallel 'size=$(blockdev --getsize {}); > loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} > $loop P 8 | dmsetup create {/}' ::: $DEVICES > root@superior:/mnt/backup# date > Fri 02 Sep 2022 08:32:20 PM EDT > root@superior:/mnt/backup# dmsetup status > sdg: 0 5860533168 snapshot 16/629145600 16 > sdf: 0 5860533168 snapshot 16/629145600 16 > sde: 0 5860533168 snapshot 16/629145600 16 > sdd: 0 5860533168 snapshot 16/629145600 16 > sdc: 0 5860533168 snapshot 16/629145600 16 > sdb: 0 5860533168 snapshot 16/629145600 16 Here's where I might want to see the output of the commands: pvs, vgs and lvs. I'm not wild about the 'dmsetup status' command and it's output. > root@superior:/mnt/backup# OVERLAYS=$(parallel echo /dev/mapper/{/} > ::: $DEVICES) > root@superior:/mnt/backup# echo $OVERLAYS > /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde > /dev/mapper/sdf /dev/mapper/sdg > root@superior:/mnt/backup# mdadm --create /dev/md1 --level=raid6 -n 6 > --assume-clean $OVERLAYS > mdadm: partition table exists on /dev/mapper/sdb > mdadm: partition table exists on /dev/mapper/sdc > mdadm: partition table exists on /dev/mapper/sdc but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdd > mdadm: partition table exists on /dev/mapper/sdd but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sde > mdadm: partition table exists on /dev/mapper/sde but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdf > mdadm: partition table exists on /dev/mapper/sdf but will be lost or > meaningless after creating array > mdadm: partition table exists on /dev/mapper/sdg > mdadm: partition table exists on /dev/mapper/sdg but will be lost or > meaningless after creating array > Continue creating array? y > mdadm: Defaulting to version 1.2 metadata > mdadm: array /dev/md1 started. > root@superior:/mnt/backup# ls -l /dev/md* > brw-rw---- 1 root disk 9, 1 Sep 2 20:34 /dev/md1 > root@superior:/mnt/backup# fsck /dev/md1 > fsck from util-linux 2.36.1 > e2fsck 1.46.2 (28-Feb-2021) > ext2fs_open2: Bad magic number in super-block > fsck.ext2: Superblock invalid, trying backup blocks... > fsck.ext2: Bad magic number in super-block while trying to open /dev/md1 > The superblock could not be read or does not describe a valid ext2/ext3/ext4 > filesystem. If the device is valid and it really contains an ext2/ext3/ext4 > filesystem (and not swap or ufs or something else), then the superblock > is corrupt, and you might try running e2fsck with an alternate superblock: > e2fsck -b 8193 <device> > or > e2fsck -b 32768 <device> Did you try these various e2fsck -b numbers? You might need to look at the ext2 man page for if there are higher numbers. It all depends on how much of the start of the disk(s) has been wiped here. > root@superior:/mnt/backup# blkid /dev/md1 > root@superior:/mnt/backup# > root@superior:/mnt/backup# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > md1 : active raid6 dm-3[5] dm-2[4] dm-1[3] dm-5[2] dm-0[1] dm-4[0] > 11720536064 blocks super 1.2 level 6, 512k chunk, algorithm 2 > [6/6] [UUUUUU] > bitmap: 0/22 pages [0KB], 65536KB chunk > unused devices: <none> > root@superior:/mnt/backup# > Some questions - > - is the easiest 'reset for next run' to reboot and rebuild? I think if you do: mdadm stop /dev/md1 dmsetup remove ... And it should release the disks. You might need to do the '-force' flag to dmsetup remove though. It should work. I'm just worried that each time you reboot the new motherboard's bios is doing something to the disks and maybe re-initilizing the disks. Could you maybe dump the first couple of MBs of one of the disks and then hexdump it with ASCII to look for strings you might recognize? Then you can also keep that and see if there's any changes when you reboot, just as a sanity check. > On Fri, Sep 2, 2022 at 3:12 PM John Stoffel <john@stoffel.org> wrote: >> >> >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> >> Peter, please include the output of all the commands, not just the >> commands themselves. See my comments below. >> >> >> > Question on restarting from scratch... >> > How to reset to the starting point? >> >> I think you need to blow away the loop devices and re-create them. >> >> Or at least blow away the dmsetup devices you just created. >> >> It might be quickest to just reboot. What OS are you using for the >> recovery? Is it a recent live image? Sorry for asking so many >> questions... some of this is new to me too. >> >> >> > dmsetup, both for remove and create of the overlay seems to be hanging. >> >> > On Fri, Sep 2, 2022 at 10:56 AM Peter Sanders <plsander@gmail.com> wrote: >> >> >> >> contents of /proc/mdstat >> >> >> >> root@superior:/mnt/backup# cat /proc/mdstat >> >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] >> >> [raid4] [raid10] >> >> unused devices: <none> >> >> root@superior:/mnt/backup# >> >> >> >> >> >> >> >> Here are the steps I ran (minus some mounting other devices and >> >> looking around for mdadm tracks on the old os disk) >> >> >> >> 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> >> echo /dev/{1}) >> >> 411 apt install parallel >> >> 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} >> >> --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' >> >> echo /dev/{1}) >> >> 413 echo $DEVICES >> >> So you found no MD RAID super blocks on any of the base devices. You >> can skip this step moving forward. >> >> >> 414 cat /proc/partitions >> >> 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg >> >> 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" >> >> 417 echo $DEVICES >> >> 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> >> {#}' ::: $DEVICES >> >> 419 ls /dev/loop* >> >> Can you show the output of all these commands, not just the commands please? >> >> >> 423 parallel truncate -s300G overlay-{/} ::: $DEVICES >> >> >> 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> >> create {/}' ::: $DEVICES >> >> 428 ls /dev/mapper/ >> >> This is some key output to view. >> >> >> 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) >> >> 430 echo $OVERLAYS >> >> What are the overlays? >> >> >> 431 dmsetup status >> >> What did this command show? >> >> >> 432 mdadm --assemble --force /dev/md1 $OVERLAYS >> >> And here is where I think you need to put --assume-clean when using >> 'create' command instead. It's not going to assemble anything because >> the info was wiped. I *think* you really want: >> >> mdadm --create /dev/md1 --level=raid6 -n 6 --assume-clean $OVERLAYS >> >> And once you do this above command and it comes back, do: >> >> cat /proc/mdstat >> >> and show all the output please! >> >> >> 433 history >> >> 434 dmsetup status >> >> 435 echo $OVERLAYS >> >> 436 mdadm --assemble --force /dev/md0 $OVERLAYS >> >> 437 cat /proc/partitions >> >> 438 mkdir /mnt/oldroot >> >> << look for inird mdadm files >> >> >> 484 echo $OVERLAYS >> >> 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> >> /dev/mapper/sdg >> >> I'm confused here, what is the difference between the md1 you >> assembled above, and the md0 you're doing here? >> >> >> << cancelled out of 485, review instructions... >> >> >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb >> >> /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf >> >> /dev/mapper/sdg >> >> 487 fsck -n /dev/md0 >> >> And what output did you get here? Did it find a filesystem? You might want >> to try: >> >> blkid /dev/md0 >> >> >> >> 488 mdadm --stop /dev/md0 >> >> 489 echo $DEVICES >> >> 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES >> >> 491 dmsetup status >> >> This all worked properly? No errors? >> >> I gave up after this because it's not clear what the results really >> are. If you don't find a filesystem that fsck's cleanly, then you >> should just need to stop the array, then re-create it but shuffle the >> order of the devices. >> >> Instead of disk in order of "sdb sdc sdd... sdN", you would try the >> order "sdc sdd ... sdN sdb". See how I moved sdb to the end of the >> list of devices? With six disks, you have I think 6 factorial options >> to try. Which is alot of options to go though, and why you need to >> automate this more. But also keep a log and show the output! >> >> John >> >> >> >> 492 ls >> >> 493 rm overlay-* >> >> 494 ls >> >> 495 parallel losetup -d ::: /dev/loop[0-9]* >> >> 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 >> >> {#}' ::: $DEVICES >> >> 497 parallel truncate -s300G overlay-{/} ::: $DEVICES >> >> 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> >> create {/}' ::: $DEVICES >> >> 499 dmsetup status >> >> 500 /sbin/reboot >> >> 501 history >> >> 502 dmsetup status >> >> 503 mount >> >> 504 cat /proc/partitions >> >> 505 nano /etc/fstab >> >> 506 mount /mnt/backup/ >> >> 507 ls /mnt/backup/ >> >> 508 rm /mnt/backup/ >> >> 509 rm /mnt/backup/overlay-sd* >> >> 510 emacs setupOverlay & >> >> 511 ps auxww | grep emacs >> >> 512 kill 65017 >> >> 513 ls /dev/loo* >> >> 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' >> >> 515 echo $DEVICES >> >> 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b >> >> 7 {#}' ::: $DEVICES >> >> 517 ls /dev/loo* >> >> 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> >> 519 ls >> >> 520 rm overlay-sd* >> >> 521 cd /mnt/bak >> >> 522 cd /mnt/backup/ >> >> 523 ls >> >> 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES >> >> 525 ls -la >> >> 526 blockdev --getsize /dev/sdb >> >> 527 man losetup >> >> 528 man losetup >> >> 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f >> >> --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup >> >> create {/}' ::: $DEVICES >> >> 530 dmsetup status >> >> 531 history | grep mdadm >> >> 532 history >> >> 533 dmsetup status >> >> 534 history | grep dmsetup >> >> 535 dmsetup status >> >> 536 dmsetup remove sdg >> >> 537 dmsetup ls --tree >> >> 538 lsof >> >> 539 dmsetup ls --tre >> >> 540 dmsetup ls --tree >> >> 541 lsof | grep -i sdg >> >> 542 lsof | grep -i sdf >> >> 543 history |grep dmsetup | less >> >> 544 dmsetup status >> >> 545 history > ~plsander/Documents/raidIssues/joblog >> >> >> >> On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote: >> >> > >> >> > >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: >> >> > >> >> > > encountering a puzzling situation. >> >> > > dmsetup is failing to return. >> >> > >> >> > I don't think you need to use dmsetup in your case, but can you post >> >> > *all* the commands you ran before you got to this point, and the >> >> > output of >> >> > >> >> > cat /proc/mdstat >> >> > >> >> > as well? Thinking on this some more, you might need to actually also >> >> > add: >> >> > >> >> > --assume-clean >> >> > >> >> > to the 'mdadm create ....' string, since you don't want it to zero the >> >> > array or anything. >> >> > >> >> > Sorry for not remembering this at the time! >> >> > >> >> > So if you can, please just start over from scratch, showing the setup >> >> > of the loop devices, the overlayfs setup, and the building the RAID6 >> >> > array, along with the cat /proc/mdstat after you do the initial build. >> >> > >> >> > John >> >> > >> >> > P.S. For those who hated my email citing tool, I pulled it out for >> >> > now. Only citing with > now. :-) >> >> > >> >> > > root@superior:/mnt/backup# dmsetup status >> >> > > sdg: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> >> > >> >> > > dmsetup remove sdg runs for hours. >> >> > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. >> >> > >> >> > > dmsetup status shows: >> >> > > sdf: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sde: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdd: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdc: 0 5860533168 snapshot 16/8388608000 16 >> >> > > sdb: 0 5860533168 snapshot 16/8388608000 16 >> >> > >> >> > > dmsetup ls --tree >> >> > > root@superior:/mnt/backup# dmsetup ls --tree >> >> > > sdf (253:3) >> >> > > ├─ (7:3) >> >> > > └─ (8:80) >> >> > > sde (253:1) >> >> > > ├─ (7:1) >> >> > > └─ (8:64) >> >> > > sdd (253:2) >> >> > > ├─ (7:2) >> >> > > └─ (8:48) >> >> > > sdc (253:0) >> >> > > ├─ (7:0) >> >> > > └─ (8:32) >> >> > > sdb (253:5) >> >> > > ├─ (7:5) >> >> > > └─ (8:16) >> >> > >> >> > > any suggestions? >> >> > >> >> > >> >> > >> >> > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote: >> >> > >> >> >> > >> On 30/08/2022 14:27, Peter Sanders wrote: >> >> > >> > >> >> > >> > And the victory conditions would be a mountable file system that passes a fsck? >> >> > >> >> >> > >> Yes. Just make sure you delve through the file system a bit and satisfy >> >> > >> yourself it looks good, too ... >> >> > >> >> >> > >> Cheers, >> >> > >> Wol ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 2:00 RAID 6, 6 device array - all devices lost superblock Peter Sanders 2022-08-28 9:14 ` Wols Lists @ 2022-08-28 15:10 ` John Stoffel 2022-08-28 17:11 ` Andy Smith 2 siblings, 0 replies; 29+ messages in thread From: John Stoffel @ 2022-08-28 15:10 UTC (permalink / raw) To: Peter Sanders; +Cc: linux-raid >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes: Peter> have a RAID 6 array, 6 devices. Been running it for years without much issue. Peter> Had hardware issues with my system - ended up replacing the Peter> motherboard, video card, and power supply and re-installing the OS Peter> (Debian 11). Can you give us details on the old vs new motherboard/cpu? It might be that you need to tweak the BIOS of the motherboard to expose the old SATA formats as well. Did you install debian onto a fresh boot disk? Is your BIOS setup to only do the new form of booting from UEFI devices, so maybe check your BIOS settings that the data drives are all in AHCI mode, or possibly even in IDE mode. It all depends on how old the original hardware was. I just recenly upgraded from a 2010 MB/CPU combo and I had to tweak the BIOS defaults to see my disks. I guess I should do a clean install from a blank disk, but I wanted to minimize downtime. Wols has some great advice here, and I heartily recommend that you use overlayfs when doing your testing. Check the RAID WIKI for suggestions. And don't panic! Your data is probably there, but just missing the super blocks or partition tables. John Peter> As the hardware issues evolved, I'd crash, reboot, un-mount the array, Peter> run fsck, mount and continue on my way - no problems. Peter> After the hardware was replaced, my array will not assemble - mdadm Peter> assemble reports no RAID superblock on the devices. Peter> root@superior:/etc/mdadm# mdadm --assemble --scan --verbose Peter> mdadm: looking for devices for /dev/md/0 Peter> mdadm: cannot open device /dev/sr0: No medium found Peter> mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sda Peter> mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sdb Peter> Examine reports Peter> /dev/sda: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> Searching for these results indicate I can rebuild the superblock, but Peter> details on how to do that are lacking, at least on the pages I found. Peter> Currently I have no /dev/md* devices. Peter> I have access to the old mdadm.conf file - have tried assembling with Peter> it, with the default mdadm.conf, and with no mdadm.conf file in /etc Peter> and /etc/mdadm. Peter> Suggestions for how to get the array back would be most appreciated. Peter> Thanks Peter> - Peter Peter> Here is the data suggested from the wiki page: Peter> root@superior:/etc/mdadm# mdadm --assemble --scan --verbose Peter> mdadm: looking for devices for /dev/md/0 Peter> mdadm: cannot open device /dev/sr0: No medium found Peter> mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sda Peter> mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sdb Peter> mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sdd Peter> mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sde Peter> mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sdf Peter> mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/sdc Peter> mdadm: No super block found on /dev/nvme0n1p9 (Expected magic Peter> a92b4efc, got 00000000) Peter> mdadm: no RAID superblock on /dev/nvme0n1p9 Peter> mdadm: No super block found on /dev/nvme0n1p8 (Expected magic Peter> a92b4efc, got 0000040c) Peter> mdadm: no RAID superblock on /dev/nvme0n1p8 Peter> mdadm: No super block found on /dev/nvme0n1p7 (Expected magic Peter> a92b4efc, got 00002004) Peter> mdadm: no RAID superblock on /dev/nvme0n1p7 Peter> mdadm: No super block found on /dev/nvme0n1p6 (Expected magic Peter> a92b4efc, got 0000040d) Peter> mdadm: no RAID superblock on /dev/nvme0n1p6 Peter> mdadm: No super block found on /dev/nvme0n1p5 (Expected magic Peter> a92b4efc, got 00000409) Peter> mdadm: no RAID superblock on /dev/nvme0n1p5 Peter> mdadm: /dev/nvme0n1p2 is too small for md: size is 2 sectors. Peter> mdadm: no RAID superblock on /dev/nvme0n1p2 Peter> mdadm: No super block found on /dev/nvme0n1p1 (Expected magic Peter> a92b4efc, got 00040001) Peter> mdadm: no RAID superblock on /dev/nvme0n1p1 Peter> mdadm: No super block found on /dev/nvme0n1 (Expected magic a92b4efc, Peter> got 7a78e8ed) Peter> mdadm: no RAID superblock on /dev/nvme0n1 Peter> root@superior:/etc/mdadm# Peter> uname -a Peter> Linux superior 5.10.0-17-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13) Peter> x86_64 GNU/Linux Peter> mdadm --version Peter> mdadm - v4.1 - 2018-10-01 Peter> smartctl devices ------------ Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Toshiba P300 Peter> Device Model: TOSHIBA HDWD130 Peter> Serial Number: 477ALBNAS Peter> LU WWN Device Id: 5 000039 fe6d2e832 Peter> Firmware Version: MX6OACF0 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Rotation Rate: 7200 rpm Peter> Form Factor: 3.5 inches Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ATA8-ACS T13/1699-D revision 4 Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:49 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Disabled Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x80) Offline data collection activity Peter> was never started. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (21791) seconds. Peter> Offline data collection Peter> capabilities: (0x5b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> No Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 1) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 364) minutes. Peter> SCT capabilities: (0x003d) SCT Status supported. Peter> SCT Error Recovery Control supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 Peter> 2 Throughput_Performance P-S--- 141 141 054 - 66 Peter> 3 Spin_Up_Time POS--- 160 160 024 - 361 (Average 357) Peter> 4 Start_Stop_Count -O--C- 100 100 000 - 204 Peter> 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 Peter> 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 Peter> 8 Seek_Time_Performance P-S--- 124 124 020 - 33 Peter> 9 Power_On_Hours -O--C- 095 095 000 - 41740 Peter> 10 Spin_Retry_Count PO--C- 100 100 060 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 204 Peter> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 759 Peter> 193 Load_Cycle_Count -O--C- 100 100 000 - 759 Peter> 194 Temperature_Celsius -O---- 181 181 000 - 33 (Min/Max 20/50) Peter> 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 Peter> 197 Current_Pending_Sector -O---K 100 100 000 - 0 Peter> 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x03 GPL R/O 1 Ext. Comprehensive SMART error log Peter> 0x04 GPL R/O 7 Device Statistics log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x08 GPL R/O 2 Power Conditions log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x20 GPL R/O 1 Streaming performance log [OBS-8] Peter> 0x21 GPL R/O 1 Write stream error log Peter> 0x22 GPL R/O 1 Read stream error log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> No self-tests have been logged. [To run self-tests, use: smartctl -t] Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 256 (0x0100) Peter> Device State: Active (0) Peter> Current Temperature: 33 Celsius Peter> Power Cycle Min/Max Temperature: 29/33 Celsius Peter> Lifetime Min/Max Temperature: 20/50 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -40/70 Celsius Peter> Temperature History Size (Index): 128 (51) Peter> Index Estimated Time Temperature Celsius Peter> 52 2022-08-25 19:12 33 ************** Peter> ... ..( 76 skipped). .. ************** Peter> 1 2022-08-25 20:29 33 ************** Peter> 2 2022-08-25 20:30 ? - Peter> 3 2022-08-25 20:31 33 ************** Peter> 4 2022-08-25 20:32 34 *************** Peter> 5 2022-08-25 20:33 33 ************** Peter> 6 2022-08-25 20:34 34 *************** Peter> ... ..( 2 skipped). .. *************** Peter> 9 2022-08-25 20:37 34 *************** Peter> 10 2022-08-25 20:38 ? - Peter> 11 2022-08-25 20:39 29 ********** Peter> 12 2022-08-25 20:40 30 *********** Peter> ... ..( 2 skipped). .. *********** Peter> 15 2022-08-25 20:43 30 *********** Peter> 16 2022-08-25 20:44 31 ************ Peter> ... ..( 3 skipped). .. ************ Peter> 20 2022-08-25 20:48 31 ************ Peter> 21 2022-08-25 20:49 32 ************* Peter> ... ..( 9 skipped). .. ************* Peter> 31 2022-08-25 20:59 32 ************* Peter> 32 2022-08-25 21:00 33 ************** Peter> ... ..( 18 skipped). .. ************** Peter> 51 2022-08-25 21:19 33 ************** Peter> SCT Error Recovery Control: Peter> Read: Disabled Peter> Write: Disabled Peter> Device Statistics (GP Log 0x04) Peter> Page Offset Size Value Flags Description Peter> 0x01 ===== = = === == General Statistics (rev 1) == Peter> 0x01 0x008 4 204 --- Lifetime Power-On Resets Peter> 0x01 0x010 4 41740 --- Power-on Hours Peter> 0x01 0x018 6 20304278904 --- Logical Sectors Written Peter> 0x01 0x020 6 64656942 --- Number of Write Commands Peter> 0x01 0x028 6 350269182084 --- Logical Sectors Read Peter> 0x01 0x030 6 481405773 --- Number of Read Commands Peter> 0x03 ===== = = === == Rotating Media Statistics (rev 1) == Peter> 0x03 0x008 4 41734 --- Spindle Motor Power-on Hours Peter> 0x03 0x010 4 41734 --- Head Flying Hours Peter> 0x03 0x018 4 759 --- Head Load Events Peter> 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors Peter> 0x03 0x028 4 22 --- Read Recovery Attempts Peter> 0x03 0x030 4 6 --- Number of Mechanical Start Failures Peter> 0x04 ===== = = === == General Errors Statistics (rev 1) == Peter> 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors Peter> 0x04 0x010 4 1 --- Resets Between Cmd Acceptance and Peter> Completion Peter> 0x05 ===== = = === == Temperature Statistics (rev 1) == Peter> 0x05 0x008 1 33 --- Current Temperature Peter> 0x05 0x010 1 33 N-- Average Short Term Temperature Peter> 0x05 0x018 1 37 N-- Average Long Term Temperature Peter> 0x05 0x020 1 50 --- Highest Temperature Peter> 0x05 0x028 1 20 --- Lowest Temperature Peter> 0x05 0x030 1 46 N-- Highest Average Short Term Temperature Peter> 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature Peter> 0x05 0x040 1 43 N-- Highest Average Long Term Temperature Peter> 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature Peter> 0x05 0x050 4 0 --- Time in Over-Temperature Peter> 0x05 0x058 1 60 --- Specified Maximum Operating Temperature Peter> 0x05 0x060 4 0 --- Time in Under-Temperature Peter> 0x05 0x068 1 0 --- Specified Minimum Operating Temperature Peter> 0x06 ===== = = === == Transport Statistics (rev 1) == Peter> 0x06 0x008 4 1006 --- Number of Hardware Resets Peter> 0x06 0x010 4 494 --- Number of ASR Events Peter> 0x06 0x018 4 0 --- Number of Interface CRC Errors Peter> |||_ C monitored condition met Peter> ||__ D supports DSN Peter> |___ N normalized value Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x0009 2 35 Transition from drive PhyRdy to drive PhyNRdy Peter> 0x000a 2 5 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x000d 2 0 Non-CRC errors within host-to-device FIS Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Western Digital Green Peter> Device Model: WDC WD30EZRX-00DC0B0 Peter> Serial Number: WD-WCC1T0668790 Peter> LU WWN Device Id: 5 0014ee 2084d406a Peter> Firmware Version: 80.00A80 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ACS-2 (minor revision not indicated) Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:51 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Unavailable Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x82) Offline data collection activity Peter> was completed without error. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (40560) seconds. Peter> Offline data collection Peter> capabilities: (0x7b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 2) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 407) minutes. Peter> Conveyance self-test routine Peter> recommended polling time: ( 5) minutes. Peter> SCT capabilities: (0x70b5) SCT Status supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0 Peter> 3 Spin_Up_Time POS--K 181 178 021 - 5916 Peter> 4 Start_Stop_Count -O--CK 100 100 000 - 377 Peter> 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 Peter> 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 Peter> 9 Power_On_Hours -O--CK 007 007 000 - 68295 Peter> 10 Spin_Retry_Count -O--CK 100 100 000 - 0 Peter> 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 296 Peter> 192 Power-Off_Retract_Count -O--CK 200 200 000 - 242 Peter> 193 Load_Cycle_Count -O--CK 052 052 000 - 445057 Peter> 194 Temperature_Celsius -O---K 121 102 000 - 29 Peter> 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 Peter> 197 Current_Pending_Sector -O--CK 200 200 000 - 0 Peter> 198 Offline_Uncorrectable ----CK 200 200 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 Peter> 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x02 SL R/O 5 Comprehensive SMART error log Peter> 0x03 GPL R/O 6 Ext. Comprehensive SMART error log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log Peter> 0xa8-0xb7 GPL,SL VS 1 Device vendor specific log Peter> 0xbd GPL,SL VS 1 Device vendor specific log Peter> 0xc0 GPL,SL VS 1 Device vendor specific log Peter> 0xc1 GPL VS 93 Device vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (6 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> Num Test_Description Status Remaining Peter> LifeTime(hours) LBA_of_first_error Peter> # 1 Extended offline Completed without error 00% 7 - Peter> # 2 Short offline Completed without error 00% 0 - Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 258 (0x0102) Peter> Device State: Active (0) Peter> Current Temperature: 29 Celsius Peter> Power Cycle Min/Max Temperature: 28/29 Celsius Peter> Lifetime Min/Max Temperature: 2/48 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> Vendor specific: Peter> 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Peter> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -41/85 Celsius Peter> Temperature History Size (Index): 478 (138) Peter> Index Estimated Time Temperature Celsius Peter> 139 2022-08-25 13:22 30 *********** Peter> 140 2022-08-25 13:23 29 ********** Peter> ... ..( 5 skipped). .. ********** Peter> 146 2022-08-25 13:29 29 ********** Peter> 147 2022-08-25 13:30 ? - Peter> 148 2022-08-25 13:31 26 ******* Peter> 149 2022-08-25 13:32 ? - Peter> 150 2022-08-25 13:33 28 ********* Peter> 151 2022-08-25 13:34 ? - Peter> 152 2022-08-25 13:35 28 ********* Peter> 153 2022-08-25 13:36 28 ********* Peter> 154 2022-08-25 13:37 29 ********** Peter> ... ..( 55 skipped). .. ********** Peter> 210 2022-08-25 14:33 29 ********** Peter> 211 2022-08-25 14:34 30 *********** Peter> ... ..( 11 skipped). .. *********** Peter> 223 2022-08-25 14:46 30 *********** Peter> 224 2022-08-25 14:47 29 ********** Peter> ... ..(103 skipped). .. ********** Peter> 328 2022-08-25 16:31 29 ********** Peter> 329 2022-08-25 16:32 30 *********** Peter> ... ..( 18 skipped). .. *********** Peter> 348 2022-08-25 16:51 30 *********** Peter> 349 2022-08-25 16:52 29 ********** Peter> ... ..( 33 skipped). .. ********** Peter> 383 2022-08-25 17:26 29 ********** Peter> 384 2022-08-25 17:27 30 *********** Peter> ... ..( 10 skipped). .. *********** Peter> 395 2022-08-25 17:38 30 *********** Peter> 396 2022-08-25 17:39 29 ********** Peter> ... ..(218 skipped). .. ********** Peter> 137 2022-08-25 21:18 29 ********** Peter> 138 2022-08-25 21:19 ? - Peter> SCT Error Recovery Control command not supported Peter> Device Statistics (GP/SMART Log 0x04) not supported Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x0008 2 0 Device-to-host non-data FIS retries Peter> 0x0009 2 305 Transition from drive PhyRdy to drive PhyNRdy Peter> 0x000a 2 5 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC Peter> 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC Peter> 0x8000 4 2491 Vendor specific Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Toshiba P300 Peter> Device Model: TOSHIBA HDWD130 Peter> Serial Number: Y7211KPAS Peter> LU WWN Device Id: 5 000039 fe6dca946 Peter> Firmware Version: MX6OACF0 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Rotation Rate: 7200 rpm Peter> Form Factor: 3.5 inches Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ATA8-ACS T13/1699-D revision 4 Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:51 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Disabled Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x80) Offline data collection activity Peter> was never started. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (21791) seconds. Peter> Offline data collection Peter> capabilities: (0x5b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> No Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 1) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 364) minutes. Peter> SCT capabilities: (0x003d) SCT Status supported. Peter> SCT Error Recovery Control supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 Peter> 2 Throughput_Performance P-S--- 139 139 054 - 71 Peter> 3 Spin_Up_Time POS--- 160 160 024 - 361 (Average 355) Peter> 4 Start_Stop_Count -O--C- 100 100 000 - 189 Peter> 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 Peter> 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 Peter> 8 Seek_Time_Performance P-S--- 128 128 020 - 31 Peter> 9 Power_On_Hours -O--C- 095 095 000 - 35428 Peter> 10 Spin_Retry_Count PO--C- 100 100 060 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 189 Peter> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 599 Peter> 193 Load_Cycle_Count -O--C- 100 100 000 - 599 Peter> 194 Temperature_Celsius -O---- 176 176 000 - 34 (Min/Max 19/50) Peter> 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 Peter> 197 Current_Pending_Sector -O---K 100 100 000 - 0 Peter> 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x03 GPL R/O 1 Ext. Comprehensive SMART error log Peter> 0x04 GPL R/O 7 Device Statistics log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x08 GPL R/O 2 Power Conditions log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x20 GPL R/O 1 Streaming performance log [OBS-8] Peter> 0x21 GPL R/O 1 Write stream error log Peter> 0x22 GPL R/O 1 Read stream error log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> No self-tests have been logged. [To run self-tests, use: smartctl -t] Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 256 (0x0100) Peter> Device State: Active (0) Peter> Current Temperature: 34 Celsius Peter> Power Cycle Min/Max Temperature: 28/34 Celsius Peter> Lifetime Min/Max Temperature: 19/50 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -40/70 Celsius Peter> Temperature History Size (Index): 128 (15) Peter> Index Estimated Time Temperature Celsius Peter> 16 2022-08-25 19:12 33 ************** Peter> ... ..( 66 skipped). .. ************** Peter> 83 2022-08-25 20:19 33 ************** Peter> 84 2022-08-25 20:20 34 *************** Peter> ... ..( 8 skipped). .. *************** Peter> 93 2022-08-25 20:29 34 *************** Peter> 94 2022-08-25 20:30 ? - Peter> 95 2022-08-25 20:31 34 *************** Peter> ... ..( 5 skipped). .. *************** Peter> 101 2022-08-25 20:37 34 *************** Peter> 102 2022-08-25 20:38 ? - Peter> 103 2022-08-25 20:39 29 ********** Peter> 104 2022-08-25 20:40 29 ********** Peter> 105 2022-08-25 20:41 30 *********** Peter> 106 2022-08-25 20:42 30 *********** Peter> 107 2022-08-25 20:43 31 ************ Peter> ... ..( 3 skipped). .. ************ Peter> 111 2022-08-25 20:47 31 ************ Peter> 112 2022-08-25 20:48 32 ************* Peter> ... ..( 4 skipped). .. ************* Peter> 117 2022-08-25 20:53 32 ************* Peter> 118 2022-08-25 20:54 33 ************** Peter> ... ..( 15 skipped). .. ************** Peter> 6 2022-08-25 21:10 33 ************** Peter> 7 2022-08-25 21:11 34 *************** Peter> 8 2022-08-25 21:12 33 ************** Peter> 9 2022-08-25 21:13 34 *************** Peter> ... ..( 5 skipped). .. *************** Peter> 15 2022-08-25 21:19 34 *************** Peter> SCT Error Recovery Control: Peter> Read: Disabled Peter> Write: Disabled Peter> Device Statistics (GP Log 0x04) Peter> Page Offset Size Value Flags Description Peter> 0x01 ===== = = === == General Statistics (rev 1) == Peter> 0x01 0x008 4 189 --- Lifetime Power-On Resets Peter> 0x01 0x010 4 35428 --- Power-on Hours Peter> 0x01 0x018 6 12728825059 --- Logical Sectors Written Peter> 0x01 0x020 6 36220308 --- Number of Write Commands Peter> 0x01 0x028 6 289884223915 --- Logical Sectors Read Peter> 0x01 0x030 6 321688917 --- Number of Read Commands Peter> 0x03 ===== = = === == Rotating Media Statistics (rev 1) == Peter> 0x03 0x008 4 35423 --- Spindle Motor Power-on Hours Peter> 0x03 0x010 4 35423 --- Head Flying Hours Peter> 0x03 0x018 4 599 --- Head Load Events Peter> 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors Peter> 0x03 0x028 4 7 --- Read Recovery Attempts Peter> 0x03 0x030 4 6 --- Number of Mechanical Start Failures Peter> 0x04 ===== = = === == General Errors Statistics (rev 1) == Peter> 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors Peter> 0x04 0x010 4 1 --- Resets Between Cmd Acceptance and Peter> Completion Peter> 0x05 ===== = = === == Temperature Statistics (rev 1) == Peter> 0x05 0x008 1 34 --- Current Temperature Peter> 0x05 0x010 1 34 N-- Average Short Term Temperature Peter> 0x05 0x018 1 37 N-- Average Long Term Temperature Peter> 0x05 0x020 1 50 --- Highest Temperature Peter> 0x05 0x028 1 19 --- Lowest Temperature Peter> 0x05 0x030 1 46 N-- Highest Average Short Term Temperature Peter> 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature Peter> 0x05 0x040 1 43 N-- Highest Average Long Term Temperature Peter> 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature Peter> 0x05 0x050 4 0 --- Time in Over-Temperature Peter> 0x05 0x058 1 60 --- Specified Maximum Operating Temperature Peter> 0x05 0x060 4 0 --- Time in Under-Temperature Peter> 0x05 0x068 1 0 --- Specified Minimum Operating Temperature Peter> 0x06 ===== = = === == Transport Statistics (rev 1) == Peter> 0x06 0x008 4 147297 --- Number of Hardware Resets Peter> 0x06 0x010 4 8793 --- Number of ASR Events Peter> 0x06 0x018 4 0 --- Number of Interface CRC Errors Peter> |||_ C monitored condition met Peter> ||__ D supports DSN Peter> |___ N normalized value Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x0009 2 29 Transition from drive PhyRdy to drive PhyNRdy Peter> 0x000a 2 5 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x000d 2 0 Non-CRC errors within host-to-device FIS Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Western Digital Green Peter> Device Model: WDC WD30EZRX-00D8PB0 Peter> Serial Number: WD-WCC4N0091255 Peter> LU WWN Device Id: 5 0014ee 2b3d4ffa1 Peter> Firmware Version: 80.00A80 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Rotation Rate: 5400 rpm Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ACS-2 (minor revision not indicated) Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:53 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Unavailable Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x82) Offline data collection activity Peter> was completed without error. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (42480) seconds. Peter> Offline data collection Peter> capabilities: (0x7b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 2) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 426) minutes. Peter> Conveyance self-test routine Peter> recommended polling time: ( 5) minutes. Peter> SCT capabilities: (0x7035) SCT Status supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 2 Peter> 3 Spin_Up_Time POS--K 184 181 021 - 5783 Peter> 4 Start_Stop_Count -O--CK 100 100 000 - 275 Peter> 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 Peter> 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 Peter> 9 Power_On_Hours -O--CK 039 039 000 - 44593 Peter> 10 Spin_Retry_Count -O--CK 100 100 000 - 0 Peter> 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 273 Peter> 192 Power-Off_Retract_Count -O--CK 200 200 000 - 225 Peter> 193 Load_Cycle_Count -O--CK 047 047 000 - 461100 Peter> 194 Temperature_Celsius -O---K 122 105 000 - 28 Peter> 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 Peter> 197 Current_Pending_Sector -O--CK 200 200 000 - 0 Peter> 198 Offline_Uncorrectable ----CK 200 200 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 Peter> 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x02 SL R/O 5 Comprehensive SMART error log Peter> 0x03 GPL R/O 6 Ext. Comprehensive SMART error log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log Peter> 0xa8-0xb7 GPL,SL VS 1 Device vendor specific log Peter> 0xbd GPL,SL VS 1 Device vendor specific log Peter> 0xc0 GPL,SL VS 1 Device vendor specific log Peter> 0xc1 GPL VS 93 Device vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (6 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> No self-tests have been logged. [To run self-tests, use: smartctl -t] Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 258 (0x0102) Peter> Device State: Active (0) Peter> Current Temperature: 28 Celsius Peter> Power Cycle Min/Max Temperature: 27/28 Celsius Peter> Lifetime Min/Max Temperature: 2/44 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> Vendor specific: Peter> 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Peter> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -41/85 Celsius Peter> Temperature History Size (Index): 478 (444) Peter> Index Estimated Time Temperature Celsius Peter> 445 2022-08-25 13:22 29 ********** Peter> ... ..( 33 skipped). .. ********** Peter> 1 2022-08-25 13:56 29 ********** Peter> 2 2022-08-25 13:57 ? - Peter> 3 2022-08-25 13:58 29 ********** Peter> ... ..( 6 skipped). .. ********** Peter> 10 2022-08-25 14:05 29 ********** Peter> 11 2022-08-25 14:06 ? - Peter> 12 2022-08-25 14:07 26 ******* Peter> 13 2022-08-25 14:08 ? - Peter> 14 2022-08-25 14:09 27 ******** Peter> 15 2022-08-25 14:10 27 ******** Peter> 16 2022-08-25 14:11 28 ********* Peter> ... ..( 37 skipped). .. ********* Peter> 54 2022-08-25 14:49 28 ********* Peter> 55 2022-08-25 14:50 29 ********** Peter> ... ..(388 skipped). .. ********** Peter> 444 2022-08-25 21:19 29 ********** Peter> SCT Error Recovery Control command not supported Peter> Device Statistics (GP/SMART Log 0x04) not supported Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x0008 2 0 Device-to-host non-data FIS retries Peter> 0x0009 2 286 Transition from drive PhyRdy to drive PhyNRdy Peter> 0x000a 2 5 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC Peter> 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC Peter> 0x8000 4 2493 Vendor specific Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Western Digital Green Peter> Device Model: WDC WD30EZRX-00MMMB0 Peter> Serial Number: WD-WCAWZ2669166 Peter> LU WWN Device Id: 5 0014ee 15a13d994 Peter> Firmware Version: 80.00A80 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ATA8-ACS (minor revision not indicated) Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:53 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Unavailable Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x82) Offline data collection activity Peter> was completed without error. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (50160) seconds. Peter> Offline data collection Peter> capabilities: (0x7b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 2) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 482) minutes. Peter> Conveyance self-test routine Peter> recommended polling time: ( 5) minutes. Peter> SCT capabilities: (0x3035) SCT Status supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0 Peter> 3 Spin_Up_Time POS--K 153 138 021 - 9350 Peter> 4 Start_Stop_Count -O--CK 100 100 000 - 297 Peter> 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 Peter> 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 Peter> 9 Power_On_Hours -O--CK 040 040 000 - 44409 Peter> 10 Spin_Retry_Count -O--CK 100 100 000 - 0 Peter> 11 Calibration_Retry_Count -O--CK 100 100 000 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 268 Peter> 192 Power-Off_Retract_Count -O--CK 200 200 000 - 218 Peter> 193 Load_Cycle_Count -O--CK 001 001 000 - 1082082 Peter> 194 Temperature_Celsius -O---K 122 105 000 - 30 Peter> 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 Peter> 197 Current_Pending_Sector -O--CK 200 200 000 - 0 Peter> 198 Offline_Uncorrectable ----CK 200 200 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 Peter> 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 1 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x02 SL R/O 5 Comprehensive SMART error log Peter> 0x03 GPL R/O 6 Ext. Comprehensive SMART error log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log Peter> 0xa8-0xb7 GPL,SL VS 1 Device vendor specific log Peter> 0xbd GPL,SL VS 1 Device vendor specific log Peter> 0xc0 GPL,SL VS 1 Device vendor specific log Peter> 0xc1 GPL VS 93 Device vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (6 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> No self-tests have been logged. [To run self-tests, use: smartctl -t] Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 258 (0x0102) Peter> Device State: Active (0) Peter> Current Temperature: 30 Celsius Peter> Power Cycle Min/Max Temperature: 27/30 Celsius Peter> Lifetime Min/Max Temperature: 0/47 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -41/85 Celsius Peter> Temperature History Size (Index): 478 (88) Peter> Index Estimated Time Temperature Celsius Peter> 89 2022-08-25 13:22 30 *********** Peter> ... ..( 33 skipped). .. *********** Peter> 123 2022-08-25 13:56 30 *********** Peter> 124 2022-08-25 13:57 ? - Peter> 125 2022-08-25 13:58 31 ************ Peter> 126 2022-08-25 13:59 30 *********** Peter> ... ..( 5 skipped). .. *********** Peter> 132 2022-08-25 14:05 30 *********** Peter> 133 2022-08-25 14:06 ? - Peter> 134 2022-08-25 14:07 26 ******* Peter> 135 2022-08-25 14:08 ? - Peter> 136 2022-08-25 14:09 27 ******** Peter> ... ..( 3 skipped). .. ******** Peter> 140 2022-08-25 14:13 27 ******** Peter> 141 2022-08-25 14:14 28 ********* Peter> ... ..( 3 skipped). .. ********* Peter> 145 2022-08-25 14:18 28 ********* Peter> 146 2022-08-25 14:19 29 ********** Peter> ... ..( 13 skipped). .. ********** Peter> 160 2022-08-25 14:33 29 ********** Peter> 161 2022-08-25 14:34 30 *********** Peter> ... ..( 43 skipped). .. *********** Peter> 205 2022-08-25 15:18 30 *********** Peter> 206 2022-08-25 15:19 31 ************ Peter> 207 2022-08-25 15:20 30 *********** Peter> ... ..(168 skipped). .. *********** Peter> 376 2022-08-25 18:09 30 *********** Peter> 377 2022-08-25 18:10 31 ************ Peter> 378 2022-08-25 18:11 30 *********** Peter> ... ..( 34 skipped). .. *********** Peter> 413 2022-08-25 18:46 30 *********** Peter> 414 2022-08-25 18:47 31 ************ Peter> 415 2022-08-25 18:48 30 *********** Peter> ... ..( 7 skipped). .. *********** Peter> 423 2022-08-25 18:56 30 *********** Peter> 424 2022-08-25 18:57 31 ************ Peter> 425 2022-08-25 18:58 30 *********** Peter> ... ..( 7 skipped). .. *********** Peter> 433 2022-08-25 19:06 30 *********** Peter> 434 2022-08-25 19:07 31 ************ Peter> 435 2022-08-25 19:08 30 *********** Peter> ... ..( 47 skipped). .. *********** Peter> 5 2022-08-25 19:56 30 *********** Peter> 6 2022-08-25 19:57 31 ************ Peter> 7 2022-08-25 19:58 30 *********** Peter> ... ..( 80 skipped). .. *********** Peter> 88 2022-08-25 21:19 30 *********** Peter> SCT Error Recovery Control command not supported Peter> Device Statistics (GP/SMART Log 0x04) not supported Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x000a 2 3 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x8000 4 2492 Vendor specific Peter> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-17-amd64] (local build) Peter> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Peter> === START OF INFORMATION SECTION === Peter> Model Family: Toshiba P300 Peter> Device Model: TOSHIBA HDWD130 Peter> Serial Number: 477ABEJAS Peter> LU WWN Device Id: 5 000039 fe6d2ce25 Peter> Firmware Version: MX6OACF0 Peter> User Capacity: 3,000,592,982,016 bytes [3.00 TB] Peter> Sector Sizes: 512 bytes logical, 4096 bytes physical Peter> Rotation Rate: 7200 rpm Peter> Form Factor: 3.5 inches Peter> Device is: In smartctl database [for details use: -P show] Peter> ATA Version is: ATA8-ACS T13/1699-D revision 4 Peter> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Peter> Local Time is: Thu Aug 25 21:19:53 2022 EDT Peter> SMART support is: Available - device has SMART capability. Peter> SMART support is: Enabled Peter> AAM feature is: Unavailable Peter> APM feature is: Disabled Peter> Rd look-ahead is: Enabled Peter> Write cache is: Enabled Peter> DSN feature is: Unavailable Peter> ATA Security is: Disabled, frozen [SEC2] Peter> Wt Cache Reorder: Enabled Peter> === START OF READ SMART DATA SECTION === Peter> SMART overall-health self-assessment test result: PASSED Peter> General SMART Values: Peter> Offline data collection status: (0x80) Offline data collection activity Peter> was never started. Peter> Auto Offline Data Collection: Enabled. Peter> Self-test execution status: ( 0) The previous self-test Peter> routine completed Peter> without error or no self-test has ever Peter> been run. Peter> Total time to complete Offline Peter> data collection: (23082) seconds. Peter> Offline data collection Peter> capabilities: (0x5b) SMART execute Offline immediate. Peter> Auto Offline data collection on/off support. Peter> Suspend Offline collection upon new Peter> command. Peter> Offline surface scan supported. Peter> Self-test supported. Peter> No Conveyance Self-test supported. Peter> Selective Self-test supported. Peter> SMART capabilities: (0x0003) Saves SMART data before entering Peter> power-saving mode. Peter> Supports SMART auto save timer. Peter> Error logging capability: (0x01) Error logging supported. Peter> General Purpose Logging supported. Peter> Short self-test routine Peter> recommended polling time: ( 1) minutes. Peter> Extended self-test routine Peter> recommended polling time: ( 385) minutes. Peter> SCT capabilities: (0x003d) SCT Status supported. Peter> SCT Error Recovery Control supported. Peter> SCT Feature Control supported. Peter> SCT Data Table supported. Peter> SMART Attributes Data Structure revision number: 16 Peter> Vendor Specific SMART Attributes with Thresholds: Peter> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE Peter> 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 Peter> 2 Throughput_Performance P-S--- 140 140 054 - 68 Peter> 3 Spin_Up_Time POS--- 161 161 024 - 358 (Average 354) Peter> 4 Start_Stop_Count -O--C- 100 100 000 - 243 Peter> 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 Peter> 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 Peter> 8 Seek_Time_Performance P-S--- 126 126 020 - 32 Peter> 9 Power_On_Hours -O--C- 094 094 000 - 44046 Peter> 10 Spin_Retry_Count PO--C- 100 100 060 - 0 Peter> 12 Power_Cycle_Count -O--CK 100 100 000 - 243 Peter> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 912 Peter> 193 Load_Cycle_Count -O--C- 100 100 000 - 912 Peter> 194 Temperature_Celsius -O---- 193 193 000 - 31 (Min/Max 19/46) Peter> 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 Peter> 197 Current_Pending_Sector -O---K 100 100 000 - 0 Peter> 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 Peter> 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0 Peter> ||||||_ K auto-keep Peter> |||||__ C event count Peter> ||||___ R error rate Peter> |||____ S speed/performance Peter> ||_____ O updated online Peter> |______ P prefailure warning Peter> General Purpose Log Directory Version 1 Peter> SMART Log Directory Version 1 [multi-sector log support] Peter> Address Access R/W Size Description Peter> 0x00 GPL,SL R/O 1 Log Directory Peter> 0x01 SL R/O 1 Summary SMART error log Peter> 0x03 GPL R/O 1 Ext. Comprehensive SMART error log Peter> 0x04 GPL R/O 7 Device Statistics log Peter> 0x06 SL R/O 1 SMART self-test log Peter> 0x07 GPL R/O 1 Extended self-test log Peter> 0x08 GPL R/O 2 Power Conditions log Peter> 0x09 SL R/W 1 Selective self-test log Peter> 0x10 GPL R/O 1 NCQ Command Error log Peter> 0x11 GPL R/O 1 SATA Phy Event Counters log Peter> 0x20 GPL R/O 1 Streaming performance log [OBS-8] Peter> 0x21 GPL R/O 1 Write stream error log Peter> 0x22 GPL R/O 1 Read stream error log Peter> 0x80-0x9f GPL,SL R/W 16 Host vendor specific log Peter> 0xe0 GPL,SL R/W 1 SCT Command/Status Peter> 0xe1 GPL,SL R/W 1 SCT Data Transfer Peter> SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Peter> No Errors Logged Peter> SMART Extended Self-test Log Version: 1 (1 sectors) Peter> No self-tests have been logged. [To run self-tests, use: smartctl -t] Peter> SMART Selective self-test log data structure revision number 1 Peter> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS Peter> 1 0 0 Not_testing Peter> 2 0 0 Not_testing Peter> 3 0 0 Not_testing Peter> 4 0 0 Not_testing Peter> 5 0 0 Not_testing Peter> Selective self-test flags (0x0): Peter> After scanning selected spans, do NOT read-scan remainder of disk. Peter> If Selective self-test is pending on power-up, resume after 0 minute delay. Peter> SCT Status Version: 3 Peter> SCT Version (vendor specific): 256 (0x0100) Peter> Device State: Active (0) Peter> Current Temperature: 31 Celsius Peter> Power Cycle Min/Max Temperature: 28/32 Celsius Peter> Lifetime Min/Max Temperature: 19/46 Celsius Peter> Under/Over Temperature Limit Count: 0/0 Peter> SCT Temperature History Version: 2 Peter> Temperature Sampling Period: 1 minute Peter> Temperature Logging Interval: 1 minute Peter> Min/Max recommended Temperature: 0/60 Celsius Peter> Min/Max Temperature Limit: -40/70 Celsius Peter> Temperature History Size (Index): 128 (117) Peter> Index Estimated Time Temperature Celsius Peter> 118 2022-08-25 19:12 31 ************ Peter> ... ..( 76 skipped). .. ************ Peter> 67 2022-08-25 20:29 31 ************ Peter> 68 2022-08-25 20:30 ? - Peter> 69 2022-08-25 20:31 31 ************ Peter> 70 2022-08-25 20:32 32 ************* Peter> 71 2022-08-25 20:33 31 ************ Peter> 72 2022-08-25 20:34 31 ************ Peter> 73 2022-08-25 20:35 32 ************* Peter> 74 2022-08-25 20:36 32 ************* Peter> 75 2022-08-25 20:37 32 ************* Peter> 76 2022-08-25 20:38 ? - Peter> 77 2022-08-25 20:39 28 ********* Peter> 78 2022-08-25 20:40 29 ********** Peter> ... ..( 2 skipped). .. ********** Peter> 81 2022-08-25 20:43 29 ********** Peter> 82 2022-08-25 20:44 30 *********** Peter> ... ..( 5 skipped). .. *********** Peter> 88 2022-08-25 20:50 30 *********** Peter> 89 2022-08-25 20:51 31 ************ Peter> ... ..( 27 skipped). .. ************ Peter> 117 2022-08-25 21:19 31 ************ Peter> SCT Error Recovery Control: Peter> Read: Disabled Peter> Write: Disabled Peter> Device Statistics (GP Log 0x04) Peter> Page Offset Size Value Flags Description Peter> 0x01 ===== = = === == General Statistics (rev 1) == Peter> 0x01 0x008 4 243 --- Lifetime Power-On Resets Peter> 0x01 0x010 4 44046 --- Power-on Hours Peter> 0x01 0x018 6 27756962802 --- Logical Sectors Written Peter> 0x01 0x020 6 86355955 --- Number of Write Commands Peter> 0x01 0x028 6 381193626849 --- Logical Sectors Read Peter> 0x01 0x030 6 791200694 --- Number of Read Commands Peter> 0x03 ===== = = === == Rotating Media Statistics (rev 1) == Peter> 0x03 0x008 4 44040 --- Spindle Motor Power-on Hours Peter> 0x03 0x010 4 44040 --- Head Flying Hours Peter> 0x03 0x018 4 912 --- Head Load Events Peter> 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors Peter> 0x03 0x028 4 0 --- Read Recovery Attempts Peter> 0x03 0x030 4 6 --- Number of Mechanical Start Failures Peter> 0x04 ===== = = === == General Errors Statistics (rev 1) == Peter> 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors Peter> 0x04 0x010 4 0 --- Resets Between Cmd Acceptance and Peter> Completion Peter> 0x05 ===== = = === == Temperature Statistics (rev 1) == Peter> 0x05 0x008 1 32 --- Current Temperature Peter> 0x05 0x010 1 31 N-- Average Short Term Temperature Peter> 0x05 0x018 1 35 N-- Average Long Term Temperature Peter> 0x05 0x020 1 46 --- Highest Temperature Peter> 0x05 0x028 1 19 --- Lowest Temperature Peter> 0x05 0x030 1 43 N-- Highest Average Short Term Temperature Peter> 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature Peter> 0x05 0x040 1 41 N-- Highest Average Long Term Temperature Peter> 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature Peter> 0x05 0x050 4 0 --- Time in Over-Temperature Peter> 0x05 0x058 1 60 --- Specified Maximum Operating Temperature Peter> 0x05 0x060 4 0 --- Time in Under-Temperature Peter> 0x05 0x068 1 0 --- Specified Minimum Operating Temperature Peter> 0x06 ===== = = === == Transport Statistics (rev 1) == Peter> 0x06 0x008 4 4706 --- Number of Hardware Resets Peter> 0x06 0x010 4 3910 --- Number of ASR Events Peter> 0x06 0x018 4 0 --- Number of Interface CRC Errors Peter> |||_ C monitored condition met Peter> ||__ D supports DSN Peter> |___ N normalized value Peter> Pending Defects log (GP Log 0x0c) not supported Peter> SATA Phy Event Counters (GP Log 0x11) Peter> ID Size Value Description Peter> 0x0001 2 0 Command failed due to ICRC error Peter> 0x0002 2 0 R_ERR response for data FIS Peter> 0x0003 2 0 R_ERR response for device-to-host data FIS Peter> 0x0004 2 0 R_ERR response for host-to-device data FIS Peter> 0x0005 2 0 R_ERR response for non-data FIS Peter> 0x0006 2 0 R_ERR response for device-to-host non-data FIS Peter> 0x0007 2 0 R_ERR response for host-to-device non-data FIS Peter> 0x0009 2 29 Transition from drive PhyRdy to drive PhyNRdy Peter> 0x000a 2 5 Device-to-host register FISes sent due to a COMRESET Peter> 0x000b 2 0 CRC errors within host-to-device FIS Peter> 0x000d 2 0 Non-CRC errors within host-to-device FIS Peter> mdadm --examine devices ----- Peter> /dev/sda: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> /dev/sdb: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> /dev/sdc: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> /dev/sdd: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> /dev/sde: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> /dev/sdf: Peter> MBR Magic : aa55 Peter> Partition[0] : 4294967295 sectors at 1 (type ee) Peter> mdadm --detail /dev/md0 ------ Peter> lsdrv ------------------------ Peter> PCI [nvme] 01:00.0 Non-Volatile memory controller: Phison Electronics Peter> Corporation E12 NVMe Controller (rev 01) Peter> └nvme nvme0 PCIe SSD {21112925606047} Peter> └nvme0n1 238.47g [259:0] Partitioned (dos) Peter> ├nvme0n1p1 485.00m [259:1] ext4 {f38776ac-1ce9-4fc8-ba50-94844b9f504e} Peter> │└Mounted as /dev/nvme0n1p1 @ /boot Peter> ├nvme0n1p2 1.00k [259:2] Partitioned (dos) Peter> ├nvme0n1p5 60.54g [259:3] ext4 {5ee1c3c0-3a05-466c-9f98-f5807c8d813b} Peter> │└Mounted as /dev/nvme0n1p5 @ / Peter> ├nvme0n1p6 93.13g [259:4] ext4 {9064169f-4fe3-4836-a906-28c1b445cdff} Peter> │└Mounted as /dev/nvme0n1p6 @ /var Peter> ├nvme0n1p7 37.00m [259:5] ext4 {25e161ad-94a0-4298-afaf-18e2433766ee} Peter> ├nvme0n1p8 82.89g [259:6] ext4 {ac874071-d759-4d33-b32f-83272f3eacd9} Peter> │└Mounted as /dev/nvme0n1p8 @ /home Peter> └nvme0n1p9 1.41g [259:7] swap {02cef84b-9a9d-4a0a-973c-fda1a78c533c} Peter> PCI [pata_jmicron] 26:00.1 IDE interface: JMicron Technology Corp. Peter> JMB368 IDE controller (rev 10) Peter> └scsi 0:0:0:0 MAD DOG LS-DVDRW TSH652M {MAD_DOG_LS-DVDRW_TSH652M} Peter> └sr0 1.00g [11:0] Empty/Unknown Peter> PCI [ahci] 26:00.0 SATA controller: JMicron Technology Corp. JMB363 Peter> SATA/IDE Controller (rev 10) Peter> └scsi 2:x:x:x [Empty] Peter> PCI [ahci] 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] Peter> FCH SATA Controller [AHCI mode] (rev 51) Peter> ├scsi 6:0:0:0 ATA TOSHIBA HDWD130 {477ALBNAS} Peter> │└sda 2.73t [8:0] Partitioned (PMBR) Peter> └scsi 7:0:0:0 ATA TOSHIBA HDWD130 {Y7211KPAS} Peter> └sdc 2.73t [8:32] Partitioned (gpt) Peter> PCI [ahci] 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] Peter> FCH SATA Controller [AHCI mode] (rev 51) Peter> ├scsi 8:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC1T0668790} Peter> │└sdb 2.73t [8:16] Partitioned (gpt) Peter> ├scsi 9:0:0:0 ATA WDC WD30EZRX-00D {WD-WCC4N0091255} Peter> │└sdd 2.73t [8:48] Partitioned (gpt) Peter> ├scsi 12:0:0:0 ATA WDC WD30EZRX-00M {WD-WCAWZ2669166} Peter> │└sde 2.73t [8:64] Partitioned (gpt) Peter> └scsi 13:0:0:0 ATA TOSHIBA HDWD130 {477ABEJAS} Peter> └sdf 2.73t [8:80] Partitioned (gpt) Peter> cat /proc/mdstat ------------- Peter> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] Peter> [raid4] [raid10] Peter> unused devices: <none> Peter> cat /etc/mdadm/mdadm.conf ---- Peter> # mdadm.conf Peter> # Peter> # !NB! Run update-initramfs -u after updating this file. Peter> # !NB! This will ensure that initramfs has an uptodate copy. Peter> # Peter> # Please refer to mdadm.conf(5) for information about this file. Peter> # Peter> # by default (built-in), scan all partitions (/proc/partitions) and all Peter> # containers for MD superblocks. alternatively, specify devices to scan, using Peter> # wildcards if desired. Peter> #DEVICE partitions containers Peter> # automatically tag new arrays as belonging to the local system Peter> HOMEHOST <system> Peter> # instruct the monitoring daemon where to send mail alerts Peter> MAILADDR root Peter> # definitions of existing MD arrays Peter> ARRAY /dev/md/0 metadata=1.2 UUID=109fa7b0:cf08fdba:e36284a9:5786ffff Peter> name=superior:0 Peter> # This configuration was auto-generated on Sun, 26 Dec 2021 13:31:14 Peter> -0500 by mkconf Peter> cat /proc/partitions --------- Peter> major minor #blocks name Peter> 259 0 250059096 nvme0n1 Peter> 259 1 496640 nvme0n1p1 Peter> 259 2 1 nvme0n1p2 Peter> 259 3 63475712 nvme0n1p5 Peter> 259 4 97654784 nvme0n1p6 Peter> 259 5 37888 nvme0n1p7 Peter> 259 6 86913024 nvme0n1p8 Peter> 259 7 1474560 nvme0n1p9 Peter> 8 32 2930266584 sdc Peter> 8 80 2930266584 sdf Peter> 8 64 2930266584 sde Peter> 8 48 2930266584 sdd Peter> 8 16 2930266584 sdb Peter> 8 0 2930266584 sda Peter> 11 0 1048575 sr0 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 2:00 RAID 6, 6 device array - all devices lost superblock Peter Sanders 2022-08-28 9:14 ` Wols Lists 2022-08-28 15:10 ` John Stoffel @ 2022-08-28 17:11 ` Andy Smith 2022-08-28 17:22 ` Andy Smith 2 siblings, 1 reply; 29+ messages in thread From: Andy Smith @ 2022-08-28 17:11 UTC (permalink / raw) To: linux-raid Hi Peter, On Sat, Aug 27, 2022 at 10:00:32PM -0400, Peter Sanders wrote: > After the hardware was replaced, my array will not assemble - mdadm > assemble reports no RAID superblock on the devices. > root@superior:/etc/mdadm# mdadm --assemble --scan --verbose > mdadm: looking for devices for /dev/md/0 > mdadm: cannot open device /dev/sr0: No medium found > mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000) > mdadm: no RAID superblock on /dev/sda > mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000) > mdadm: no RAID superblock on /dev/sdb I'm wondering if this is one of those motherboards that at boot helpfully writes a new empty GPT on any drive that it thinks doesn't have any kind of partitioning. I say this because: - It looks like you're using sd{a,b} etc with no partitions - I've heard of motherboards that do this - You say you just switched to a new motherboard Cheers, Andy ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 17:11 ` Andy Smith @ 2022-08-28 17:22 ` Andy Smith 2022-08-28 17:34 ` Peter Sanders 0 siblings, 1 reply; 29+ messages in thread From: Andy Smith @ 2022-08-28 17:22 UTC (permalink / raw) To: linux-raid On Sun, Aug 28, 2022 at 05:11:56PM +0000, Andy Smith wrote: > I'm wondering if this is one of those motherboards that at boot > helpfully writes a new empty GPT Sorry, all the other replies speculating the same have just arrived in my inbox! ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: RAID 6, 6 device array - all devices lost superblock 2022-08-28 17:22 ` Andy Smith @ 2022-08-28 17:34 ` Peter Sanders 0 siblings, 0 replies; 29+ messages in thread From: Peter Sanders @ 2022-08-28 17:34 UTC (permalink / raw) To: linux-raid It was set up on the device level, not partitions. (I remember getting some advice on the web that device was better than partition... Yay for internet advice) I'm surveying my other disks to see what I have available to do the overlay attempt. What are the size of the overlay files going to end up being? I did run into UEFI vs AHCI issues early in the process.. they are all set to non-UEFI. OS update was onto a new SSD... On Sun, Aug 28, 2022 at 1:23 PM Andy Smith <andy@strugglers.net> wrote: > > On Sun, Aug 28, 2022 at 05:11:56PM +0000, Andy Smith wrote: > > I'm wondering if this is one of those motherboards that at boot > > helpfully writes a new empty GPT > > Sorry, all the other replies speculating the same have just arrived > in my inbox! ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2022-09-05 20:16 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-28 2:00 RAID 6, 6 device array - all devices lost superblock Peter Sanders
2022-08-28 9:14 ` Wols Lists
2022-08-28 9:54 ` Wols Lists
2022-08-28 16:47 ` Phil Turmel
[not found] ` <CAKAPSkJAQYsec-4zzcePbkJ7Ee0=sd_QvHj4Stnyineq+T8BXw@mail.gmail.com>
2022-08-28 17:16 ` Wols Lists
2022-08-28 18:45 ` John Stoffel
2022-08-28 19:36 ` Phil Turmel
2022-08-28 19:49 ` John Stoffel
2022-08-28 23:24 ` Peter Sanders
2022-08-29 13:12 ` Peter Sanders
2022-08-29 21:45 ` John Stoffel
2022-08-29 22:29 ` Eyal Lebedinsky
2022-08-29 23:53 ` Peter Sanders
2022-08-30 13:27 ` Peter Sanders
2022-08-30 18:03 ` Wols Lists
2022-08-31 17:48 ` Peter Sanders
2022-08-31 20:37 ` John Stoffel
2022-09-02 14:56 ` Peter Sanders
2022-09-02 18:52 ` Peter Sanders
2022-09-02 19:12 ` John Stoffel
2022-09-03 0:39 ` Peter Sanders
2022-09-03 5:51 ` Peter Sanders
2022-09-05 19:36 ` John Stoffel
2022-09-05 20:16 ` Peter Sanders
2022-09-05 19:25 ` John Stoffel
2022-08-28 15:10 ` John Stoffel
2022-08-28 17:11 ` Andy Smith
2022-08-28 17:22 ` Andy Smith
2022-08-28 17:34 ` Peter Sanders
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.