What to do about Offline_Uncorrectable and Pending_Sector in RAID1

What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

Hi

I'm running software RAID1 across two drives in my home machine (LVM
on LUKS on RAID1). I've just installed smartmontools and run short
tests, and promptly received emails to tell me that one of the drives
has 4 offline uncorrectable sectors and 3 current pending sectors.
I've attached smartctl --xall output for sda (good) and sdb (bad).

These drives are pretty old (over 5 years) so I'm going to replace
them as soon as I have time (and yes, I have backups), but in the
meantime I'd like advice on:

1. What exactly this means. My understanding is that some data has
been lost (or may have been lost) on the drive, but the drive still
has spare sectors to remap things once the failed sectors are written
to. Is that correct?

2. How can I tell which sectors are problematic? If it's in the swap
partition I'm far less worried. I can see two LBAs for offline
uncorrectable errors in the --xall output, but that still leaves
another two at large.

3. Assuming my understanding is correct, and that the sector falls
within the RAID1 partition on the drive, is there some way I can
recover the sectors from the other drive in the RAID1? As a last
resort I imagine I could wipe the suspect drive and then rebuild it
from the good one, but I'm hoping there's something less risky I can
do.

Thanks in advance
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

[-- Attachment #2: sda.txt --]
[-- Type: text/plain, Size: 16559 bytes --]

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZA9626479
LU WWN Device Id: 5 0014ee 2b0e3fa4c
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Nov 13 20:30:04 2016 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(36780) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 355) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   172   162   021    -    6400
  4 Start_Stop_Count        -O--CK   097   097   000    -    3688
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   081   081   000    -    13891
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   097   097   000    -    3683
192 Power-Off_Retract_Count -O--CK   200   200   000    -    50
193 Load_Cycle_Count        -O--CK   001   001   000    -    912124
194 Temperature_Celsius     -O---K   120   109   000    -    30
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     13888         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     22/30 Celsius
Lifetime    Min/Max Temperature:     22/41 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (462)

Index    Estimated Time   Temperature Celsius
 463    2016-11-13 12:33    31  ************
 ...    ..( 16 skipped).    ..  ************
   2    2016-11-13 12:50    31  ************
   3    2016-11-13 12:51    30  ***********
   4    2016-11-13 12:52    30  ***********
   5    2016-11-13 12:53    30  ***********
   6    2016-11-13 12:54    31  ************
   7    2016-11-13 12:55    30  ***********
 ...    ..( 21 skipped).    ..  ***********
  29    2016-11-13 13:17    30  ***********
  30    2016-11-13 13:18    31  ************
 ...    ..( 14 skipped).    ..  ************
  45    2016-11-13 13:33    31  ************
  46    2016-11-13 13:34     ?  -
  47    2016-11-13 13:35    22  ***
  48    2016-11-13 13:36    22  ***
  49    2016-11-13 13:37    22  ***
  50    2016-11-13 13:38    23  ****
  51    2016-11-13 13:39    23  ****
  52    2016-11-13 13:40    24  *****
  53    2016-11-13 13:41    24  *****
  54    2016-11-13 13:42    25  ******
  55    2016-11-13 13:43    25  ******
  56    2016-11-13 13:44    26  *******
 ...    ..(  2 skipped).    ..  *******
  59    2016-11-13 13:47    26  *******
  60    2016-11-13 13:48    27  ********
 ...    ..(  4 skipped).    ..  ********
  65    2016-11-13 13:53    27  ********
  66    2016-11-13 13:54    28  *********
 ...    ..(  4 skipped).    ..  *********
  71    2016-11-13 13:59    28  *********
  72    2016-11-13 14:00    29  **********
 ...    ..( 19 skipped).    ..  **********
  92    2016-11-13 14:20    29  **********
  93    2016-11-13 14:21    30  ***********
 ...    ..( 12 skipped).    ..  ***********
 106    2016-11-13 14:34    30  ***********
 107    2016-11-13 14:35    31  ************
 ...    ..(  2 skipped).    ..  ************
 110    2016-11-13 14:38    31  ************
 111    2016-11-13 14:39    32  *************
 112    2016-11-13 14:40    31  ************
 ...    ..( 18 skipped).    ..  ************
 131    2016-11-13 14:59    31  ************
 132    2016-11-13 15:00    32  *************
 133    2016-11-13 15:01    32  *************
 134    2016-11-13 15:02    31  ************
 135    2016-11-13 15:03    32  *************
 136    2016-11-13 15:04    31  ************
 ...    ..( 10 skipped).    ..  ************
 147    2016-11-13 15:15    31  ************
 148    2016-11-13 15:16    32  *************
 149    2016-11-13 15:17    31  ************
 150    2016-11-13 15:18    31  ************
 151    2016-11-13 15:19    32  *************
 152    2016-11-13 15:20    31  ************
 ...    ..( 10 skipped).    ..  ************
 163    2016-11-13 15:31    31  ************
 164    2016-11-13 15:32     ?  -
 165    2016-11-13 15:33    20  *
 166    2016-11-13 15:34    21  **
 167    2016-11-13 15:35    21  **
 168    2016-11-13 15:36    21  **
 169    2016-11-13 15:37    22  ***
 170    2016-11-13 15:38    22  ***
 171    2016-11-13 15:39    22  ***
 172    2016-11-13 15:40    23  ****
 173    2016-11-13 15:41    24  *****
 174    2016-11-13 15:42    24  *****
 175    2016-11-13 15:43    24  *****
 176    2016-11-13 15:44    25  ******
 177    2016-11-13 15:45    25  ******
 178    2016-11-13 15:46    25  ******
 179    2016-11-13 15:47    26  *******
 ...    ..(  2 skipped).    ..  *******
 182    2016-11-13 15:50    26  *******
 183    2016-11-13 15:51    27  ********
 ...    ..(  7 skipped).    ..  ********
 191    2016-11-13 15:59    27  ********
 192    2016-11-13 16:00    28  *********
 ...    ..(  4 skipped).    ..  *********
 197    2016-11-13 16:05    28  *********
 198    2016-11-13 16:06    29  **********
 ...    ..( 13 skipped).    ..  **********
 212    2016-11-13 16:20    29  **********
 213    2016-11-13 16:21    30  ***********
 ...    ..(  5 skipped).    ..  ***********
 219    2016-11-13 16:27    30  ***********
 220    2016-11-13 16:28    31  ************
 221    2016-11-13 16:29    31  ************
 222    2016-11-13 16:30    31  ************
 223    2016-11-13 16:31    30  ***********
 224    2016-11-13 16:32    30  ***********
 225    2016-11-13 16:33    31  ************
 ...    ..(  2 skipped).    ..  ************
 228    2016-11-13 16:36    31  ************
 229    2016-11-13 16:37    30  ***********
 ...    ..(  5 skipped).    ..  ***********
 235    2016-11-13 16:43    30  ***********
 236    2016-11-13 16:44    31  ************
 237    2016-11-13 16:45    30  ***********
 ...    ..(  8 skipped).    ..  ***********
 246    2016-11-13 16:54    30  ***********
 247    2016-11-13 16:55    31  ************
 248    2016-11-13 16:56    30  ***********
 ...    ..(  9 skipped).    ..  ***********
 258    2016-11-13 17:06    30  ***********
 259    2016-11-13 17:07    31  ************
 260    2016-11-13 17:08    30  ***********
 ...    ..(  8 skipped).    ..  ***********
 269    2016-11-13 17:17    30  ***********
 270    2016-11-13 17:18    31  ************
 271    2016-11-13 17:19    31  ************
 272    2016-11-13 17:20    31  ************
 273    2016-11-13 17:21    30  ***********
 274    2016-11-13 17:22    30  ***********
 275    2016-11-13 17:23    31  ************
 276    2016-11-13 17:24    31  ************
 277    2016-11-13 17:25    30  ***********
 ...    ..(  7 skipped).    ..  ***********
 285    2016-11-13 17:33    30  ***********
 286    2016-11-13 17:34    31  ************
 ...    ..( 17 skipped).    ..  ************
 304    2016-11-13 17:52    31  ************
 305    2016-11-13 17:53    30  ***********
 306    2016-11-13 17:54    31  ************
 307    2016-11-13 17:55    30  ***********
 ...    ..(  5 skipped).    ..  ***********
 313    2016-11-13 18:01    30  ***********
 314    2016-11-13 18:02    31  ************
 315    2016-11-13 18:03    31  ************
 316    2016-11-13 18:04    30  ***********
 ...    ..(  3 skipped).    ..  ***********
 320    2016-11-13 18:08    30  ***********
 321    2016-11-13 18:09    31  ************
 ...    ..( 11 skipped).    ..  ************
 333    2016-11-13 18:21    31  ************
 334    2016-11-13 18:22    32  *************
 335    2016-11-13 18:23    31  ************
 336    2016-11-13 18:24    31  ************
 337    2016-11-13 18:25    32  *************
 338    2016-11-13 18:26    31  ************
 ...    ..( 11 skipped).    ..  ************
 350    2016-11-13 18:38    31  ************
 351    2016-11-13 18:39    32  *************
 352    2016-11-13 18:40    31  ************
 ...    ..(  5 skipped).    ..  ************
 358    2016-11-13 18:46    31  ************
 359    2016-11-13 18:47    32  *************
 360    2016-11-13 18:48    32  *************
 361    2016-11-13 18:49    32  *************
 362    2016-11-13 18:50    31  ************
 ...    ..( 14 skipped).    ..  ************
 377    2016-11-13 19:05    31  ************
 378    2016-11-13 19:06    30  ***********
 379    2016-11-13 19:07    31  ************
 380    2016-11-13 19:08    30  ***********
 ...    ..(  4 skipped).    ..  ***********
 385    2016-11-13 19:13    30  ***********
 386    2016-11-13 19:14    31  ************
 387    2016-11-13 19:15    31  ************
 388    2016-11-13 19:16    30  ***********
 ...    ..(  4 skipped).    ..  ***********
 393    2016-11-13 19:21    30  ***********
 394    2016-11-13 19:22    31  ************
 395    2016-11-13 19:23    30  ***********
 396    2016-11-13 19:24    31  ************
 ...    ..( 10 skipped).    ..  ************
 407    2016-11-13 19:35    31  ************
 408    2016-11-13 19:36    32  *************
 409    2016-11-13 19:37     ?  -
 410    2016-11-13 19:38    32  *************
 411    2016-11-13 19:39    31  ************
 ...    ..(  2 skipped).    ..  ************
 414    2016-11-13 19:42    31  ************
 415    2016-11-13 19:43    32  *************
 416    2016-11-13 19:44    31  ************
 417    2016-11-13 19:45    32  *************
 418    2016-11-13 19:46    31  ************
 419    2016-11-13 19:47    32  *************
 420    2016-11-13 19:48    31  ************
 421    2016-11-13 19:49    32  *************
 422    2016-11-13 19:50    31  ************
 423    2016-11-13 19:51    31  ************
 424    2016-11-13 19:52    31  ************
 425    2016-11-13 19:53    32  *************
 426    2016-11-13 19:54    31  ************
 ...    ..(  4 skipped).    ..  ************
 431    2016-11-13 19:59    31  ************
 432    2016-11-13 20:00    32  *************
 433    2016-11-13 20:01    32  *************
 434    2016-11-13 20:02    31  ************
 ...    ..(  4 skipped).    ..  ************
 439    2016-11-13 20:07    31  ************
 440    2016-11-13 20:08    32  *************
 441    2016-11-13 20:09    31  ************
 442    2016-11-13 20:10    32  *************
 443    2016-11-13 20:11    31  ************
 ...    ..(  2 skipped).    ..  ************
 446    2016-11-13 20:14    31  ************
 447    2016-11-13 20:15    32  *************
 448    2016-11-13 20:16    31  ************
 ...    ..(  5 skipped).    ..  ************
 454    2016-11-13 20:22    31  ************
 455    2016-11-13 20:23    32  *************
 456    2016-11-13 20:24    31  ************
 457    2016-11-13 20:25    32  *************
 458    2016-11-13 20:26    32  *************
 459    2016-11-13 20:27    31  ************
 ...    ..(  2 skipped).    ..  ************
 462    2016-11-13 20:30    31  ************

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x000a  2            3  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x8000  4         3578  Vendor specific


[-- Attachment #3: sdb.txt --]
[-- Type: text/plain, Size: 17448 bytes --]

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZA9552721
LU WWN Device Id: 5 0014ee 2b0e3f07a
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Nov 13 20:30:01 2016 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(37260) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 360) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   168   162   021    -    6566
  4 Start_Stop_Count        -O--CK   097   097   000    -    3748
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   081   081   000    -    13883
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   097   097   000    -    3683
192 Power-Off_Retract_Count -O--CK   200   200   000    -    49
193 Load_Cycle_Count        -O--CK   001   001   000    -    837570
194 Temperature_Celsius     -O---K   119   108   000    -    31
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   001   000    -    3
198 Offline_Uncorrectable   ----CK   200   200   000    -    4
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    3
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 2
	CR     = Command Register
	FEATR  = Features Register
	COUNT  = Count (was: Sector Count) Register
	LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
	LH     = LBA High (was: Cylinder High) Register    ]   LBA
	LM     = LBA Mid (was: Cylinder Low) Register      ] Register
	LL     = LBA Low (was: Sector Number) Register     ]
	DV     = Device (was: Device/Head) Register
	DC     = Device Control Register
	ER     = Error register
	ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 [1] occurred at disk power-on lifetime: 9505 hours (396 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 25 b3 05 58 40 00  Error: UNC at LBA = 0x25b30558 = 632489304

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 80 00 48 00 00 25 b3 0b 80 40 08     17:33:58.845  READ FPDMA QUEUED
  60 00 80 00 50 00 00 25 b3 0b 00 40 08     17:33:58.845  READ FPDMA QUEUED
  60 00 80 00 58 00 00 25 b3 0a 80 40 08     17:33:58.844  READ FPDMA QUEUED
  60 00 80 00 60 00 00 25 b3 0a 00 40 08     17:33:58.844  READ FPDMA QUEUED
  60 00 80 00 80 00 00 25 b3 09 80 40 08     17:33:58.832  READ FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 9505 hours (396 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 fe 00 00 25 b2 fa 58 40 00  Error: UNC at LBA = 0x25b2fa58 = 632486488

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 80 00 d8 00 00 25 b2 ff 00 40 08     17:33:56.041  READ FPDMA QUEUED
  60 00 80 00 d0 00 00 25 b2 fe 80 40 08     17:33:56.041  READ FPDMA QUEUED
  60 00 80 00 c8 00 00 25 b2 fe 00 40 08     17:33:56.041  READ FPDMA QUEUED
  60 00 80 00 c0 00 00 25 b2 fd 80 40 08     17:33:56.041  READ FPDMA QUEUED
  60 00 80 00 b8 00 00 25 b2 fd 00 40 08     17:33:56.040  READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     13880         -
# 2  Short offline       Aborted by host               10%     13880         -
# 3  Short offline       Interrupted (host reset)      10%     13880         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     22/31 Celsius
Lifetime    Min/Max Temperature:     21/42 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (433)

Index    Estimated Time   Temperature Celsius
 434    2016-11-13 12:33    32  *************
 ...    ..( 15 skipped).    ..  *************
 450    2016-11-13 12:49    32  *************
 451    2016-11-13 12:50    31  ************
 452    2016-11-13 12:51    32  *************
 ...    ..(  2 skipped).    ..  *************
 455    2016-11-13 12:54    32  *************
 456    2016-11-13 12:55    31  ************
 ...    ..( 16 skipped).    ..  ************
 473    2016-11-13 13:12    31  ************
 474    2016-11-13 13:13    32  *************
 475    2016-11-13 13:14    32  *************
 476    2016-11-13 13:15    32  *************
 477    2016-11-13 13:16    31  ************
   0    2016-11-13 13:17    31  ************
   1    2016-11-13 13:18    32  *************
 ...    ..( 14 skipped).    ..  *************
  16    2016-11-13 13:33    32  *************
  17    2016-11-13 13:34     ?  -
  18    2016-11-13 13:35    22  ***
  19    2016-11-13 13:36    22  ***
  20    2016-11-13 13:37    22  ***
  21    2016-11-13 13:38    23  ****
  22    2016-11-13 13:39    24  *****
  23    2016-11-13 13:40    24  *****
  24    2016-11-13 13:41    25  ******
  25    2016-11-13 13:42    25  ******
  26    2016-11-13 13:43    25  ******
  27    2016-11-13 13:44    26  *******
  28    2016-11-13 13:45    26  *******
  29    2016-11-13 13:46    27  ********
 ...    ..(  5 skipped).    ..  ********
  35    2016-11-13 13:52    27  ********
  36    2016-11-13 13:53    28  *********
  37    2016-11-13 13:54    28  *********
  38    2016-11-13 13:55    29  **********
  39    2016-11-13 13:56    28  *********
  40    2016-11-13 13:57    29  **********
 ...    ..( 11 skipped).    ..  **********
  52    2016-11-13 14:09    29  **********
  53    2016-11-13 14:10    30  ***********
 ...    ..( 11 skipped).    ..  ***********
  65    2016-11-13 14:22    30  ***********
  66    2016-11-13 14:23    31  ************
 ...    ..( 10 skipped).    ..  ************
  77    2016-11-13 14:34    31  ************
  78    2016-11-13 14:35    32  *************
 ...    ..( 26 skipped).    ..  *************
 105    2016-11-13 15:02    32  *************
 106    2016-11-13 15:03    33  **************
 107    2016-11-13 15:04    32  *************
 ...    ..( 10 skipped).    ..  *************
 118    2016-11-13 15:15    32  *************
 119    2016-11-13 15:16    33  **************
 120    2016-11-13 15:17    32  *************
 121    2016-11-13 15:18    32  *************
 122    2016-11-13 15:19    33  **************
 123    2016-11-13 15:20    32  *************
 ...    ..(  2 skipped).    ..  *************
 126    2016-11-13 15:23    32  *************
 127    2016-11-13 15:24    33  **************
 128    2016-11-13 15:25    32  *************
 ...    ..(  5 skipped).    ..  *************
 134    2016-11-13 15:31    32  *************
 135    2016-11-13 15:32     ?  -
 136    2016-11-13 15:33    21  **
 137    2016-11-13 15:34    21  **
 138    2016-11-13 15:35    21  **
 139    2016-11-13 15:36    22  ***
 140    2016-11-13 15:37    22  ***
 141    2016-11-13 15:38    23  ****
 142    2016-11-13 15:39    23  ****
 143    2016-11-13 15:40    23  ****
 144    2016-11-13 15:41    24  *****
 145    2016-11-13 15:42    25  ******
 ...    ..(  2 skipped).    ..  ******
 148    2016-11-13 15:45    25  ******
 149    2016-11-13 15:46    26  *******
 150    2016-11-13 15:47    26  *******
 151    2016-11-13 15:48    26  *******
 152    2016-11-13 15:49    27  ********
 ...    ..(  5 skipped).    ..  ********
 158    2016-11-13 15:55    27  ********
 159    2016-11-13 15:56    28  *********
 ...    ..(  5 skipped).    ..  *********
 165    2016-11-13 16:02    28  *********
 166    2016-11-13 16:03    29  **********
 ...    ..( 10 skipped).    ..  **********
 177    2016-11-13 16:14    29  **********
 178    2016-11-13 16:15    30  ***********
 ...    ..(  6 skipped).    ..  ***********
 185    2016-11-13 16:22    30  ***********
 186    2016-11-13 16:23    31  ************
 ...    ..(  9 skipped).    ..  ************
 196    2016-11-13 16:33    31  ************
 197    2016-11-13 16:34    32  *************
 ...    ..(  8 skipped).    ..  *************
 206    2016-11-13 16:43    32  *************
 207    2016-11-13 16:44    31  ************
 208    2016-11-13 16:45    31  ************
 209    2016-11-13 16:46    31  ************
 210    2016-11-13 16:47    32  *************
 211    2016-11-13 16:48    31  ************
 ...    ..(  2 skipped).    ..  ************
 214    2016-11-13 16:51    31  ************
 215    2016-11-13 16:52    32  *************
 216    2016-11-13 16:53    32  *************
 217    2016-11-13 16:54    31  ************
 218    2016-11-13 16:55    32  *************
 219    2016-11-13 16:56    31  ************
 ...    ..(  3 skipped).    ..  ************
 223    2016-11-13 17:00    31  ************
 224    2016-11-13 17:01    32  *************
 225    2016-11-13 17:02    32  *************
 226    2016-11-13 17:03    31  ************
 ...    ..(  2 skipped).    ..  ************
 229    2016-11-13 17:06    31  ************
 230    2016-11-13 17:07    32  *************
 231    2016-11-13 17:08    32  *************
 232    2016-11-13 17:09    31  ************
 ...    ..(  2 skipped).    ..  ************
 235    2016-11-13 17:12    31  ************
 236    2016-11-13 17:13    32  *************
 ...    ..( 39 skipped).    ..  *************
 276    2016-11-13 17:53    32  *************
 277    2016-11-13 17:54    31  ************
 ...    ..(  7 skipped).    ..  ************
 285    2016-11-13 18:02    31  ************
 286    2016-11-13 18:03    32  *************
 287    2016-11-13 18:04    32  *************
 288    2016-11-13 18:05    31  ************
 289    2016-11-13 18:06    32  *************
 ...    ..( 14 skipped).    ..  *************
 304    2016-11-13 18:21    32  *************
 305    2016-11-13 18:22    33  **************
 306    2016-11-13 18:23    32  *************
 307    2016-11-13 18:24    32  *************
 308    2016-11-13 18:25    33  **************
 309    2016-11-13 18:26    32  *************
 ...    ..( 19 skipped).    ..  *************
 329    2016-11-13 18:46    32  *************
 330    2016-11-13 18:47    33  **************
 331    2016-11-13 18:48    32  *************
 332    2016-11-13 18:49    32  *************
 333    2016-11-13 18:50    33  **************
 334    2016-11-13 18:51    32  *************
 ...    ..(  9 skipped).    ..  *************
 344    2016-11-13 19:01    32  *************
 345    2016-11-13 19:02    31  ************
 ...    ..( 16 skipped).    ..  ************
 362    2016-11-13 19:19    31  ************
 363    2016-11-13 19:20    32  *************
 ...    ..( 15 skipped).    ..  *************
 379    2016-11-13 19:36    32  *************
 380    2016-11-13 19:37     ?  -
 381    2016-11-13 19:38    33  **************
 382    2016-11-13 19:39    32  *************
 ...    ..( 29 skipped).    ..  *************
 412    2016-11-13 20:09    32  *************
 413    2016-11-13 20:10    33  **************
 ...    ..( 14 skipped).    ..  **************
 428    2016-11-13 20:25    33  **************
 429    2016-11-13 20:26    32  *************
 430    2016-11-13 20:27    33  **************
 431    2016-11-13 20:28    32  *************
 432    2016-11-13 20:29    32  *************
 433    2016-11-13 20:30    32  *************

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x000a  2            2  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x8000  4         3575  Vendor specific

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Anthony Youngman]

Quick first response ...

On 13/11/16 18:46, Bruce Merry wrote:
> Hi
>
> I'm running software RAID1 across two drives in my home machine (LVM
> on LUKS on RAID1). I've just installed smartmontools and run short
> tests, and promptly received emails to tell me that one of the drives
> has 4 offline uncorrectable sectors and 3 current pending sectors.
> I've attached smartctl --xall output for sda (good) and sdb (bad).
>
> These drives are pretty old (over 5 years) so I'm going to replace
> them as soon as I have time (and yes, I have backups), but in the
> meantime I'd like advice on:
>
What drives are they? I'm guessing they're hunky-dory, but they don't 
fall foul of timeout mismatch, do they?

https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

> 1. What exactly this means. My understanding is that some data has
> been lost (or may have been lost) on the drive, but the drive still
> has spare sectors to remap things once the failed sectors are written
> to. Is that correct?

It may also mean that the four sectors at least, have already been 
remapped ... I'll let the experts confirm. The three pending errors 
might be where a read has failed but there's not yet been a re-write - 
and you won't have noticed because the raid dealt with it.
>
> 2. How can I tell which sectors are problematic? If it's in the swap
> partition I'm far less worried. I can see two LBAs for offline
> uncorrectable errors in the --xall output, but that still leaves
> another two at large.

I don't think you need to be worried at all. It's only a few sectors, 
there's no sign of any further trouble? and as it's raided, when the 
drive returns an error the raid code will sort it out for you.
>
> 3. Assuming my understanding is correct, and that the sector falls
> within the RAID1 partition on the drive, is there some way I can
> recover the sectors from the other drive in the RAID1? As a last
> resort I imagine I could wipe the suspect drive and then rebuild it
> from the good one, but I'm hoping there's something less risky I can
> do.

Do a scrub? You've got seven errors total, which some people will say 
"panic on the first error" and others will say "so what, the odd error 
every now and then is nothing to worry about". The point of a scrub is 
it will background-scan the entire array, and if it can't read anything, 
it will re-calculate and re-write it.

Just make sure you've not got that timeout problem, or a scrub will make 
matters a whole lot worse ...
>
> Thanks in advance
> Bruce
>
Cheers,
Wol

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

On 13 November 2016 at 22:18, Anthony Youngman <antlists@youngman.org.uk> wrote:
> Quick first response ...
>
> On 13/11/16 18:46, Bruce Merry wrote:
>>
>> Hi
>>
>> I'm running software RAID1 across two drives in my home machine (LVM
>> on LUKS on RAID1). I've just installed smartmontools and run short
>> tests, and promptly received emails to tell me that one of the drives
>> has 4 offline uncorrectable sectors and 3 current pending sectors.
>> I've attached smartctl --xall output for sda (good) and sdb (bad).
>>
>> These drives are pretty old (over 5 years) so I'm going to replace
>> them as soon as I have time (and yes, I have backups), but in the
>> meantime I'd like advice on:
>>
> What drives are they? I'm guessing they're hunky-dory, but they don't fall
> foul of timeout mismatch, do they?
>
> https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

smartctl reports "SCT Error Recovery Control command not supported".
Does that mean I should be worried? Is there any way to tell whether a
given drive I can buy online supports it?

>> 1. What exactly this means. My understanding is that some data has
>> been lost (or may have been lost) on the drive, but the drive still
>> has spare sectors to remap things once the failed sectors are written
>> to. Is that correct?
>
>
> It may also mean that the four sectors at least, have already been remapped
> ... I'll let the experts confirm. The three pending errors might be where a
> read has failed but there's not yet been a re-write - and you won't have
> noticed because the raid dealt with it.

I'm guessing nothing has been remapped yet, because the
Reallocated_Sector_Ct and Reallocator_Event_ct are both zero.

>> 3. Assuming my understanding is correct, and that the sector falls
>> within the RAID1 partition on the drive, is there some way I can
>> recover the sectors from the other drive in the RAID1? As a last
>> resort I imagine I could wipe the suspect drive and then rebuild it
>> from the good one, but I'm hoping there's something less risky I can
>> do.
>
>
> Do a scrub? You've got seven errors total, which some people will say "panic
> on the first error" and others will say "so what, the odd error every now
> and then is nothing to worry about". The point of a scrub is it will
> background-scan the entire array, and if it can't read anything, it will
> re-calculate and re-write it.

Yes, that sounds like what I need. Thanks to Google I found
/usr/share/mdadm/checkarray to trigger this. It still has a few hours
to go, but now the bad drive has pending sectors == 65535 (which is
suspiciously power-of-two and I assume means it's actually higher and
is being clamped), and /sys/block/md0/md/mismatch_cnt is currently at
1408. If scrubbing is supposed to rewrite on failed reads I would have
expected pending sectors to go down rather than up, so I'm not sure
what's happening.

Thanks
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Wols Lists]

On 13/11/16 20:51, Bruce Merry wrote:
> On 13 November 2016 at 22:18, Anthony Youngman <antlists@youngman.org.uk> wrote:
>> Quick first response ...
>>
>> On 13/11/16 18:46, Bruce Merry wrote:
>>>
>>> Hi
>>>
>>> I'm running software RAID1 across two drives in my home machine (LVM
>>> on LUKS on RAID1). I've just installed smartmontools and run short
>>> tests, and promptly received emails to tell me that one of the drives
>>> has 4 offline uncorrectable sectors and 3 current pending sectors.
>>> I've attached smartctl --xall output for sda (good) and sdb (bad).
>>>
>>> These drives are pretty old (over 5 years) so I'm going to replace
>>> them as soon as I have time (and yes, I have backups), but in the
>>> meantime I'd like advice on:
>>>
>> What drives are they? I'm guessing they're hunky-dory, but they don't fall
>> foul of timeout mismatch, do they?
>>
>> https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
> 
> smartctl reports "SCT Error Recovery Control command not supported".
> Does that mean I should be worried? Is there any way to tell whether a
> given drive I can buy online supports it?

You need drives that explicitly support raid. WD Reds, Seagate NAS, some
Toshibas - my 2TB laptop drive does ... Try and find a friend with a
drive you like, and check it out, or ask on this list :-)

Did you run that script to increase the kernel timeout?
> 
>>> 1. What exactly this means. My understanding is that some data has
>>> been lost (or may have been lost) on the drive, but the drive still
>>> has spare sectors to remap things once the failed sectors are written
>>> to. Is that correct?
>>
>>
>> It may also mean that the four sectors at least, have already been remapped
>> ... I'll let the experts confirm. The three pending errors might be where a
>> read has failed but there's not yet been a re-write - and you won't have
>> noticed because the raid dealt with it.
> 
> I'm guessing nothing has been remapped yet, because the
> Reallocated_Sector_Ct and Reallocator_Event_ct are both zero.
> 
>>> 3. Assuming my understanding is correct, and that the sector falls
>>> within the RAID1 partition on the drive, is there some way I can
>>> recover the sectors from the other drive in the RAID1? As a last
>>> resort I imagine I could wipe the suspect drive and then rebuild it
>>> from the good one, but I'm hoping there's something less risky I can
>>> do.
>>
>>
>> Do a scrub? You've got seven errors total, which some people will say "panic
>> on the first error" and others will say "so what, the odd error every now
>> and then is nothing to worry about". The point of a scrub is it will
>> background-scan the entire array, and if it can't read anything, it will
>> re-calculate and re-write it.
> 
> Yes, that sounds like what I need. Thanks to Google I found
> /usr/share/mdadm/checkarray to trigger this. It still has a few hours
> to go, but now the bad drive has pending sectors == 65535 (which is
> suspiciously power-of-two and I assume means it's actually higher and
> is being clamped), and /sys/block/md0/md/mismatch_cnt is currently at
> 1408. If scrubbing is supposed to rewrite on failed reads I would have
> expected pending sectors to go down rather than up, so I'm not sure
> what's happening.
> 
Ummm....

Sounds like that drive could need replacing. I'd get a new drive and do
that as soon as possible - use the --replace option of mdadm - don't
fail the old drive and add the new. Dunno where you're based, but 5mins
on the internet ordering a new drive is probably time well spent.

Note that Seagate Barracudas don't have the best of reputations if
they're the drive you've already got, and the 3TB drives are best
avoided. Sod's law, I've got two of them ...

Advice I always give ... if you're getting new drives, always consider
increasing capacity. I don't know what size your current drives are, but
look at prices of drives a bit larger than what they are, and is it
worth paying the extra?

If you do get bigger drives, there's nothing stopping you making the
paritions on it bigger before you add them in to the array. It'll be
wasted space until you increase the size of all the drives, but once
you've replaced both drives, you can use mdadm to increase the array
size. I don't know about LUKS, but I would expect you can grow that, and
then you can expand your data partitions within that.

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Phil Turmel]

Hi Bruce,

On 11/13/2016 04:06 PM, Wols Lists wrote:
> On 13/11/16 20:51, Bruce Merry wrote:
>> On 13 November 2016 at 22:18, Anthony Youngman <antlists@youngman.org.uk> wrote:
>>> Quick first response ...

>>> https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
>>
>> smartctl reports "SCT Error Recovery Control command not supported".
>> Does that mean I should be worried? Is there any way to tell whether a
>> given drive I can buy online supports it?

You should be worried.  It is vital for proper MD raid operation that
drive timeouts be shorter than the kernel timeout for that device.  If
you can't make the drive timeout short, you *must* make the kernel
timeout long.

> You need drives that explicitly support raid. WD Reds, Seagate NAS, some
> Toshibas - my 2TB laptop drive does ... Try and find a friend with a
> drive you like, and check it out, or ask on this list :-)

Manufacturers' data sheets for system builders usually contain enough
information to determine if ERC is supported.  Nowadays, the "NAS"
families work out of the box.  And enterprise drives, too, of course.

>>>> 1. What exactly this means. My understanding is that some data has
>>>> been lost (or may have been lost) on the drive, but the drive still
>>>> has spare sectors to remap things once the failed sectors are written
>>>> to. Is that correct?

Generally, yes.

>>> Do a scrub? You've got seven errors total, which some people will say "panic
>>> on the first error" and others will say "so what, the odd error every now
>>> and then is nothing to worry about". The point of a scrub is it will
>>> background-scan the entire array, and if it can't read anything, it will
>>> re-calculate and re-write it.
>>
>> Yes, that sounds like what I need. Thanks to Google I found
>> /usr/share/mdadm/checkarray to trigger this. It still has a few hours
>> to go, but now the bad drive has pending sectors == 65535 (which is
>> suspiciously power-of-two and I assume means it's actually higher and
>> is being clamped), and /sys/block/md0/md/mismatch_cnt is currently at
>> 1408. If scrubbing is supposed to rewrite on failed reads I would have
>> expected pending sectors to go down rather than up, so I'm not sure
>> what's happening.
>>
> Ummm....
> 
> Sounds like that drive could need replacing. I'd get a new drive and do
> that as soon as possible - use the --replace option of mdadm - don't
> fail the old drive and add the new. Dunno where you're based, but 5mins
> on the internet ordering a new drive is probably time well spent.

You have two other possibilities:

1) Swap volumes in the raid.  These are known to drop unneeded writes
when the data isn't needed, even if it made it to one of the mirrors.
That makes harmless mismatches.

2) Trim.  Well-behaved drive firmware guarantees zeros for trimmed
sectors, but many drives return random data instead.  Zing, mismatches.
It's often unhelpful with encrypted volumes, as even well-behaved
firmware can't deliver zeroed sectors *inside* the encryption.

I wouldn't panic just yet.  The check scrub should (with mitigated
timeouts) fix all of your pending sectors.  Then look at your actual
relocations to determine if you really do have a problem.

Phil

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

On 14 November 2016 at 01:03, Phil Turmel <philip@turmel.org> wrote:
> Hi Bruce,
>
> On 11/13/2016 04:06 PM, Wols Lists wrote:
>> On 13/11/16 20:51, Bruce Merry wrote:
>>> On 13 November 2016 at 22:18, Anthony Youngman <antlists@youngman.org.uk> wrote:
>>>> Quick first response ...
>
>>>> https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
>>>
>>> smartctl reports "SCT Error Recovery Control command not supported".
>>> Does that mean I should be worried? Is there any way to tell whether a
>>> given drive I can buy online supports it?
>
> You should be worried.  It is vital for proper MD raid operation that
> drive timeouts be shorter than the kernel timeout for that device.  If
> you can't make the drive timeout short, you *must* make the kernel
> timeout long.

Okay, I'll give that script a go to increase my kernel timeout. If I
understand correctly, it's not the end of the world if the drive
doesn't support SCTERC, provided I have a long kernel timeout (and
when things go wrong it might take much longer to recover, but I can
live with that). Is that correct?

>>> Yes, that sounds like what I need. Thanks to Google I found
>>> /usr/share/mdadm/checkarray to trigger this. It still has a few hours
>>> to go, but now the bad drive has pending sectors == 65535 (which is
>>> suspiciously power-of-two and I assume means it's actually higher and
>>> is being clamped), and /sys/block/md0/md/mismatch_cnt is currently at
>>> 1408. If scrubbing is supposed to rewrite on failed reads I would have
>>> expected pending sectors to go down rather than up, so I'm not sure
>>> what's happening.
>>>
>> Ummm....
>>
>> Sounds like that drive could need replacing. I'd get a new drive and do
>> that as soon as possible - use the --replace option of mdadm - don't
>> fail the old drive and add the new. Dunno where you're based, but 5mins
>> on the internet ordering a new drive is probably time well spent.

Oh don't worry, I wasted no time in ordering new drives already.

> You have two other possibilities:
>
> 1) Swap volumes in the raid.  These are known to drop unneeded writes
> when the data isn't needed, even if it made it to one of the mirrors.
> That makes harmless mismatches.

It won't be that - I keep have separate non-RAIDed partitions for swap.

> 2) Trim.  Well-behaved drive firmware guarantees zeros for trimmed
> sectors, but many drives return random data instead.  Zing, mismatches.
> It's often unhelpful with encrypted volumes, as even well-behaved
> firmware can't deliver zeroed sectors *inside* the encryption.

Weee, sounds like fun. I hope it's that. Is there any way to tell
which blocks mismatch, so that I can tell which files are in trouble
(assuming I can figure out how to map through LVM, LUKS and
debuge2fs).

Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Phil Turmel]

Hi Bruce,

On 11/14/2016 01:50 AM, Bruce Merry wrote:

> Okay, I'll give that script a go to increase my kernel timeout. If I
> understand correctly, it's not the end of the world if the drive
> doesn't support SCTERC, provided I have a long kernel timeout (and
> when things go wrong it might take much longer to recover, but I can
> live with that). Is that correct?

Yes.

>> 2) Trim.  Well-behaved drive firmware guarantees zeros for trimmed
>> sectors, but many drives return random data instead.  Zing, mismatches.
>> It's often unhelpful with encrypted volumes, as even well-behaved
>> firmware can't deliver zeroed sectors *inside* the encryption.
> 
> Weee, sounds like fun. I hope it's that. Is there any way to tell
> which blocks mismatch, so that I can tell which files are in trouble
> (assuming I can figure out how to map through LVM, LUKS and
> debuge2fs).

The check operation doesn't log the sector addresses, unfortunately.  At
least I don't see any such operation in the code that increments
mismatch count.  Not even a tracepoint.  Hmmm.

In the meantime, run a "repair" scrub instead of a "check" scrub to
affirmatively force no mismatches.  (Writes first member of mirrors to
the others.)

Phil

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

On 14 November 2016 at 18:01, Phil Turmel <philip@turmel.org> wrote:
> In the meantime, run a "repair" scrub instead of a "check" scrub to
> affirmatively force no mismatches.  (Writes first member of mirrors to
> the others.)

I think that's good news. I wasn't sure which direction "repair" would
copy, but the good drive is the first member ("Device Role : Active
device 0"). I'll give that a go once the current scrub is done.

Thanks
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Phil Turmel]

On 11/14/2016 11:09 AM, Bruce Merry wrote:
> On 14 November 2016 at 18:01, Phil Turmel <philip@turmel.org> wrote:
>> In the meantime, run a "repair" scrub instead of a "check" scrub to
>> affirmatively force no mismatches.  (Writes first member of mirrors to
>> the others.)
> 
> I think that's good news. I wasn't sure which direction "repair" would
> copy, but the good drive is the first member ("Device Role : Active
> device 0"). I'll give that a go once the current scrub is done.

Direction is truly immaterial, as any bad sectors on device 0 will be
reconstructed from the other.  Neil Brown has a blog entry on this topic
that applies here:

http://neil.brown.name/blog/20100211050355

Phil

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
> Sounds like that drive could need replacing. I'd get a new drive and do
> that as soon as possible - use the --replace option of mdadm - don't
> fail the old drive and add the new.

Would you mind explaining why I should use --replace instead of taking
out the suspect drive? I guess I lose redundancy for any writes that
occur while the rebuild is happening, but I'd plan to do this with the
filesystem unmounted so there wouldn't be any writes.

What I'd quite like to do is treat the "good" drive as the source for
all the data (unless it turns out to have bad sectors too...), even if
the other drive doesn't have a read error; which I'd achieve by
failing the "bad" drive. I want to do this because after doing one
scrub and starting on a second, I'm still seeing non-zero
mismatch_cnt. Does --replace do anything clever about using the old
drive only when it must, or does it just read from the whole array
like normal, which might mean taking the mismatched data from the bad
drive?

Thanks
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Wols Lists]

On 14/11/16 15:52, Bruce Merry wrote:
> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>> > Sounds like that drive could need replacing. I'd get a new drive and do
>> > that as soon as possible - use the --replace option of mdadm - don't
>> > fail the old drive and add the new.
> Would you mind explaining why I should use --replace instead of taking
> out the suspect drive? I guess I lose redundancy for any writes that
> occur while the rebuild is happening, but I'd plan to do this with the
> filesystem unmounted so there wouldn't be any writes.

Because a replace will copy from the old drive to the new, recovering
any failures from the rest of the array. A fail-and-add will have to
rebuild the entire new array from what's left of the old, stressing the
old array much more.

Okay, in your case, it probably won't make an awful lot of difference,
but it does make you vulnerable to problems on the "good" drive. To
alter your wording slightly, you lose redundancy for writes AND READS
that occur while the array is rebuilding. It's just good practice (and I
point it out because --replace is new and not well known at present).

Cheers,
Wol

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Bruce Merry]

On 14 November 2016 at 17:58, Wols Lists <antlists@youngman.org.uk> wrote:
> On 14/11/16 15:52, Bruce Merry wrote:
>> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>>> > Sounds like that drive could need replacing. I'd get a new drive and do
>>> > that as soon as possible - use the --replace option of mdadm - don't
>>> > fail the old drive and add the new.
>> Would you mind explaining why I should use --replace instead of taking
>> out the suspect drive? I guess I lose redundancy for any writes that
>> occur while the rebuild is happening, but I'd plan to do this with the
>> filesystem unmounted so there wouldn't be any writes.
>
> Because a replace will copy from the old drive to the new, recovering
> any failures from the rest of the array. A fail-and-add will have to
> rebuild the entire new array from what's left of the old, stressing the
> old array much more.

Okay, I can see how for RAID5 that might be a bad thing.

In my case however, it sounds like --replace will copy everything from
the failing drive, whereas I'd rather it copied everything from the
good drive. Same stress on the array, less chance of copying dodgy
data.

Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Wols Lists]

On 14/11/16 16:03, Bruce Merry wrote:
> On 14 November 2016 at 17:58, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 14/11/16 15:52, Bruce Merry wrote:
>>> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>>>>> Sounds like that drive could need replacing. I'd get a new drive and do
>>>>> that as soon as possible - use the --replace option of mdadm - don't
>>>>> fail the old drive and add the new.
>>> Would you mind explaining why I should use --replace instead of taking
>>> out the suspect drive? I guess I lose redundancy for any writes that
>>> occur while the rebuild is happening, but I'd plan to do this with the
>>> filesystem unmounted so there wouldn't be any writes.
>>
>> Because a replace will copy from the old drive to the new, recovering
>> any failures from the rest of the array. A fail-and-add will have to
>> rebuild the entire new array from what's left of the old, stressing the
>> old array much more.
> 
> Okay, I can see how for RAID5 that might be a bad thing.
> 
> In my case however, it sounds like --replace will copy everything from
> the failing drive, whereas I'd rather it copied everything from the
> good drive. Same stress on the array, less chance of copying dodgy
> data.
> 
So long as the data on the drive is correct (it should be) and the drive
reports a fault where it can't read it, it'll only copy good data off
the bad drive. It'll copy it from the other drive if it's dud.

Cheers,
Wol

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Phil Turmel]

On 11/14/2016 11:03 AM, Bruce Merry wrote:
> On 14 November 2016 at 17:58, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 14/11/16 15:52, Bruce Merry wrote:
>>> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>>>>> Sounds like that drive could need replacing. I'd get a new drive and do
>>>>> that as soon as possible - use the --replace option of mdadm - don't
>>>>> fail the old drive and add the new.
>>> Would you mind explaining why I should use --replace instead of taking
>>> out the suspect drive? I guess I lose redundancy for any writes that
>>> occur while the rebuild is happening, but I'd plan to do this with the
>>> filesystem unmounted so there wouldn't be any writes.
>>
>> Because a replace will copy from the old drive to the new, recovering
>> any failures from the rest of the array. A fail-and-add will have to
>> rebuild the entire new array from what's left of the old, stressing the
>> old array much more.

I entirely endorse Anthony's advice on this one.  You are at great risk
of not completing a fail/add resync with the new drive.

> Okay, I can see how for RAID5 that might be a bad thing.
> 
> In my case however, it sounds like --replace will copy everything from
> the failing drive, whereas I'd rather it copied everything from the
> good drive. Same stress on the array, less chance of copying dodgy
> data.

You simply don't have that choice, sorry.  And drives returning dodgy
data is ungodly rare.  The sector checksum algorithms are that good.
You have a URE crisis in your array that is far more significant.

Phil

RE: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Peter Sangas]

Hi Wol,


-----Original Message-----
From: Wols Lists [mailto:antlists@youngman.org.uk] 
Sent: Monday, November 14, 2016 7:58 AM
To: Bruce Merry
Cc: linux-raid@vger.kernel.org
Subject: Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

On 14/11/16 15:52, Bruce Merry wrote:
> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>> > Sounds like that drive could need replacing. I'd get a new drive 
>> > and do that as soon as possible - use the --replace option of mdadm 
>> > - don't fail the old drive and add the new.
> Would you mind explaining why I should use --replace instead of taking 
> out the suspect drive? I guess I lose redundancy for any writes that 
> occur while the rebuild is happening, but I'd plan to do this with the 
> filesystem unmounted so there wouldn't be any writes.

>Because a replace will copy from the old drive to the new, recovering any failures from the rest of the array. A fail-and-add will have to rebuild the entire new array >from what's left of the old, stressing the old array much more.

>Okay, in your case, it probably won't make an awful lot of difference, but it does make you vulnerable to problems on the "good" drive. To alter your wording >slightly, you lose redundancy for writes AND READS that occur while the array is rebuilding. It's just good practice (and I point it out because --replace is new and >not well known at present).

>Cheers,
>Wol

With respect to the --replace switch and "replacing a failed drive" documented on the wiki here:
https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive  Can you clear a few things up for me ?

1. If I just want to replace a working drive in a RAID1 and the array is still redundant I can 
issue the following command as in your example:

mdadm /dev/mdN [--fail /dev/sdx1] --remove /dev/sdx1 --add /dev/sdy1

which fails and removes sdx1 and replaces it with sdy1.

Question1. How is this different from first doing a fail/remove on sdx1, physically replacing sdx1 with sdy1 and doing an add on sdy1?


2. If one of the drives as an error in a RAID1 and gets kicked out of the array and the array loses redundancy the wiki has the following example:

mdmad /dev/mdN --re-add /dev/sdX1
mdadm /dev/mdN --add /dev/sdY1 --replace /dev/sdX1 --with /dev/sdY1

Question2.   Is this point here to first try and re-add sdX1 with the "--re-add" (first line above) and if that fails do a replace (second line above)?


Thanks,
Peter


To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Anthony Youngman]

On 15/11/16 18:14, Peter Sangas wrote:
> Hi Wol,
>
>
> -----Original Message-----
> From: Wols Lists [mailto:antlists@youngman.org.uk]
> Sent: Monday, November 14, 2016 7:58 AM
> To: Bruce Merry
> Cc: linux-raid@vger.kernel.org
> Subject: Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1
>
> On 14/11/16 15:52, Bruce Merry wrote:
>> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>>>> Sounds like that drive could need replacing. I'd get a new drive
>>>> and do that as soon as possible - use the --replace option of mdadm
>>>> - don't fail the old drive and add the new.
>> Would you mind explaining why I should use --replace instead of taking
>> out the suspect drive? I guess I lose redundancy for any writes that
>> occur while the rebuild is happening, but I'd plan to do this with the
>> filesystem unmounted so there wouldn't be any writes.
>
>> Because a replace will copy from the old drive to the new, recovering any failures from the rest of the array. A fail-and-add will have to rebuild the entire new array >from what's left of the old, stressing the old array much more.
>
>> Okay, in your case, it probably won't make an awful lot of difference, but it does make you vulnerable to problems on the "good" drive. To alter your wording >slightly, you lose redundancy for writes AND READS that occur while the array is rebuilding. It's just good practice (and I point it out because --replace is new and >not well known at present).
>
>> Cheers,
>> Wol
>
> With respect to the --replace switch and "replacing a failed drive" documented on the wiki here:
> https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive  Can you clear a few things up for me ?
>
> 1. If I just want to replace a working drive in a RAID1 and the array is still redundant I can
> issue the following command as in your example:
>
> mdadm /dev/mdN [--fail /dev/sdx1] --remove /dev/sdx1 --add /dev/sdy1
>
> which fails and removes sdx1 and replaces it with sdy1.
>
> Question1. How is this different from first doing a fail/remove on sdx1, physically replacing sdx1 with sdy1 and doing an add on sdy1?
>
Not really different at all. It's just that you (obviously) can't do the 
remove and add in the same command if you physically swap the drive in 
the middle.

But I bang on a bit about having access to spare port to stick a drive 
on, so I've assumed you can have both the old and the new drive 
physically (and logically) in the system at the same time.
>
> 2. If one of the drives as an error in a RAID1 and gets kicked out of the array and the array loses redundancy the wiki has the following example:
>
> mdmad /dev/mdN --re-add /dev/sdX1
> mdadm /dev/mdN --add /dev/sdY1 --replace /dev/sdX1 --with /dev/sdY1
>
> Question2.   Is this point here to first try and re-add sdX1 with the "--re-add" (first line above) and if that fails do a replace (second line above)?
>
Correct. You've lost redundancy, and (you NEED a bitmap here) the idea 
is to get sdX1 back in to the array to restore redundancy before you 
copy its contents to sdY1.

You need the bitmap because, without it, a re-add becomes a normal add, 
and it's not only a waste of time, it adds stress to the array and 
increases your chances of a total failure.
>
> Thanks,
> Peter
>
Cheers,
Wol

RE: What to do about Offline_Uncorrectable and Pending_Sector in RAID1

[Peter Sangas]

Got it. Thanks again.


-----Original Message-----
From: Anthony Youngman [mailto:antlists@youngman.org.uk] 
Sent: Tuesday, November 15, 2016 10:50 AM
To: Peter Sangas
Cc: linux-raid@vger.kernel.org
Subject: Re: What to do about Offline_Uncorrectable and Pending_Sector in RAID1



On 15/11/16 18:14, Peter Sangas wrote:
> Hi Wol,
>
>
> -----Original Message-----
> From: Wols Lists [mailto:antlists@youngman.org.uk]
> Sent: Monday, November 14, 2016 7:58 AM
> To: Bruce Merry
> Cc: linux-raid@vger.kernel.org
> Subject: Re: What to do about Offline_Uncorrectable and Pending_Sector 
> in RAID1
>
> On 14/11/16 15:52, Bruce Merry wrote:
>> On 13 November 2016 at 23:06, Wols Lists <antlists@youngman.org.uk> wrote:
>>>> Sounds like that drive could need replacing. I'd get a new drive 
>>>> and do that as soon as possible - use the --replace option of mdadm
>>>> - don't fail the old drive and add the new.
>> Would you mind explaining why I should use --replace instead of 
>> taking out the suspect drive? I guess I lose redundancy for any 
>> writes that occur while the rebuild is happening, but I'd plan to do 
>> this with the filesystem unmounted so there wouldn't be any writes.
>
>> Because a replace will copy from the old drive to the new, recovering any failures from the rest of the array. A fail-and-add will have to rebuild the entire new array >from what's left of the old, stressing the old array much more.
>
>> Okay, in your case, it probably won't make an awful lot of difference, but it does make you vulnerable to problems on the "good" drive. To alter your wording >slightly, you lose redundancy for writes AND READS that occur while the array is rebuilding. It's just good practice (and I point it out because --replace is new and >not well known at present).
>
>> Cheers,
>> Wol
>
> With respect to the --replace switch and "replacing a failed drive" documented on the wiki here:
> https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive  Can you clear a few things up for me ?
>
> 1. If I just want to replace a working drive in a RAID1 and the array 
> is still redundant I can issue the following command as in your example:
>
> mdadm /dev/mdN [--fail /dev/sdx1] --remove /dev/sdx1 --add /dev/sdy1
>
> which fails and removes sdx1 and replaces it with sdy1.
>
> Question1. How is this different from first doing a fail/remove on sdx1, physically replacing sdx1 with sdy1 and doing an add on sdy1?
>
Not really different at all. It's just that you (obviously) can't do the remove and add in the same command if you physically swap the drive in the middle.

But I bang on a bit about having access to spare port to stick a drive on, so I've assumed you can have both the old and the new drive physically (and logically) in the system at the same time.
>
> 2. If one of the drives as an error in a RAID1 and gets kicked out of the array and the array loses redundancy the wiki has the following example:
>
> mdmad /dev/mdN --re-add /dev/sdX1
> mdadm /dev/mdN --add /dev/sdY1 --replace /dev/sdX1 --with /dev/sdY1
>
> Question2.   Is this point here to first try and re-add sdX1 with the "--re-add" (first line above) and if that fails do a replace (second line above)?
>
Correct. You've lost redundancy, and (you NEED a bitmap here) the idea is to get sdX1 back in to the array to restore redundancy before you copy its contents to sdY1.

You need the bitmap because, without it, a re-add becomes a normal add, and it's not only a waste of time, it adds stress to the array and increases your chances of a total failure.
>
> Thanks,
> Peter
>
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html