* do i need to give up on this setup
@ 2020-10-05 13:10 Daniel Sanabria
2020-10-05 13:17 ` Reindl Harald
2020-10-05 13:44 ` Roman Mamedov
0 siblings, 2 replies; 16+ messages in thread
From: Daniel Sanabria @ 2020-10-05 13:10 UTC (permalink / raw)
To: Linux-RAID
Hi all,
Scrubbing ( # echo check >
/sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
I'm attaching details of the array and disks (bloody wd greens) as
well as journalctl errors providing some details about the issue.
If you have any pointers on what might be the cause of this as well as
any recommendations on how to improve things please let me thank you
in advance ...
I have backups of the data so happy to move this to a different setup
you might recommend (apps will be mostly reading from the array via
NFS since most of the content will be media).
My suspicion is that a timer service is kicking in and disrupting the
scrubbing somehow but can't pinpoint what causes this.
Thanks again,
Dan
PD. Apologies for the verbosity of the logs but wasn't really sure if
you guys accept links from paste services
[dan@lamachine ~]$ sudo mdadm --detail /dev/md1
[sudo] password for dan:
/dev/md1:
Version : 1.2
Creation Time : Fri Feb 15 12:26:56 2019
Raid Level : raid5
Array Size : 4194039808 (3.91 TiB 4.29 TB)
Used Dev Size : 2097019904 (1999.87 GiB 2147.35 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Oct 5 11:35:31 2020
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
- 0 0 2 removed
3 8 65 - faulty /dev/sde1
[dan@lamachine ~]$
[dan@lamachine ~]$ sudo hdparm -I /dev/sdc
[sudo] password for dan:
/dev/sdc:
ATA device, with non-removable media
Model Number: WDC WD30EZRX-00D8PB0
Serial Number: WD-WCC4NCWT13RF
Firmware Revision: 80.00A80
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev
2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 5860531055
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
device size with M = 1024*1024: 2861587 MBytes
device size with M = 1000*1000: 3000591 MBytes (3000 GB)
cache/buffer size = unknown
Nominal Media Rotation Rate: 5400
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 0
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* 64-bit World wide name
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* NCQ priority information
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12] (vendor specific)
unknown 206[13] (vendor specific)
unknown 206[14] (vendor specific)
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
414min for SECURITY ERASE UNIT. 414min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee25fc9e460
NAA : 5
IEEE OUI : 0014ee
Unique ID : 25fc9e460
Checksum: correct
[dan@lamachine ~]$ sudo hdparm -I /dev/sde
/dev/sde:
ATA device, with non-removable media
Model Number: WDC WD30EZRX-00D8PB0
Serial Number: WD-WCC4N1294906
Firmware Revision: 80.00A80
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev
2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 5860531055
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
device size with M = 1024*1024: 2861587 MBytes
device size with M = 1000*1000: 3000591 MBytes (3000 GB)
cache/buffer size = unknown
Nominal Media Rotation Rate: 5400
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 0
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* 64-bit World wide name
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* NCQ priority information
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12] (vendor specific)
unknown 206[13] (vendor specific)
unknown 206[14] (vendor specific)
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
458min for SECURITY ERASE UNIT. 458min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee25f968120
NAA : 5
IEEE OUI : 0014ee
Unique ID : 25f968120
Checksum: correct
[dan@lamachine ~]$ sudo hdparm -I /dev/sdd
/dev/sdd:
ATA device, with non-removable media
Model Number: WDC WD30EZRX-00D8PB0
Serial Number: WD-WCC4NPRDD6D7
Firmware Revision: 80.00A80
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev
2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 5860533168
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
device size with M = 1024*1024: 2861588 MBytes
device size with M = 1000*1000: 3000592 MBytes (3000 GB)
cache/buffer size = unknown
Nominal Media Rotation Rate: 5400
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 0
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* 64-bit World wide name
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* NCQ priority information
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12] (vendor specific)
unknown 206[13] (vendor specific)
unknown 206[14] (vendor specific)
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
414min for SECURITY ERASE UNIT. 414min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee25fca27b1
NAA : 5
IEEE OUI : 0014ee
Unique ID : 25fca27b1
Checksum: correct
[dan@lamachine ~]$
truncated journalctl logs:
Oct 05 10:57:11 lamachine systemd-logind[1571]: Session 8 logged out.
Waiting for processes to exit.
Oct 05 10:57:11 lamachine systemd-logind[1571]: Removed session 8.
Oct 05 11:00:35 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 300)
Oct 05 11:00:35 lamachine smartd[1480]: Device: /dev/sdc [SAT], failed
to read SMART Attribute Data
Oct 05 11:00:35 lamachine kernel: ata7.00: configured for UDMA/133
Oct 05 11:00:35 lamachine smartd[1480]: Sending warning via
/usr/libexec/smartmontools/smartdnotify to root ...
Oct 05 11:00:35 lamachine smartd[1480]: Warning via
/usr/libexec/smartmontools/smartdnotify to root: successful
Oct 05 11:00:35 lamachine postfix/pickup[2347]: EFF87608EF11: uid=0 from=<root>
Oct 05 11:00:35 lamachine postfix/cleanup[4225]: EFF87608EF11:
message-id=<20201005100035.EFF87608EF11@lamachine.localdomain>
Oct 05 11:00:36 lamachine postfix/qmgr[2080]: EFF87608EF11:
from=<root@lamachine.localdomain>, size=524, nrcpt=1 (queue active)
Oct 05 11:00:36 lamachine postfix/local[4228]: EFF87608EF11:
to=<root@lamachine.localdomain>, orig_to=<root>, relay=local,
delay=0.15, delays=0.09/0.02/0/0.04, dsn=2.0.0, status=sent (delivered
to mailbox)
Oct 05 11:00:36 lamachine postfix/qmgr[2080]: EFF87608EF11: removed
Oct 05 11:08:36 lamachine sshd[3936]: Timeout, client not responding
from user dan 192.168.1.113 port 54226
Oct 05 11:08:36 lamachine sshd[3933]: pam_unix(sshd:session): session
closed for user dan
Oct 05 11:08:36 lamachine systemd-logind[1571]: Session 7 logged out.
Waiting for processes to exit.
Oct 05 11:08:36 lamachine sudo[3990]: pam_unix(sudo:session): session
closed for user root
Oct 05 11:08:36 lamachine systemd-logind[1571]: Removed session 7.
Oct 05 11:29:33 lamachine smartd[1480]: Device: /dev/sdc [SAT], read
SMART Attribute Data worked again, warning condition reset after 1
email
Oct 05 11:30:43 lamachine kernel: ata9: link is slow to respond,
please be patient (ready=0)
Oct 05 11:30:47 lamachine kernel: ata9: COMRESET failed (errno=-16)
Oct 05 11:30:53 lamachine kernel: ata9: link is slow to respond,
please be patient (ready=0)
Oct 05 11:30:57 lamachine kernel: ata9: COMRESET failed (errno=-16)
Oct 05 11:31:03 lamachine kernel: ata9: link is slow to respond,
please be patient (ready=0)
Oct 05 11:31:32 lamachine kernel: ata9: COMRESET failed (errno=-16)
Oct 05 11:31:32 lamachine kernel: ata9: limiting SATA link speed to 3.0 Gbps
Oct 05 11:31:37 lamachine smartd[1480]: Device: /dev/sde [SAT], failed
to read SMART Attribute Data
Oct 05 11:31:37 lamachine smartd[1480]: Sending warning via
/usr/libexec/smartmontools/smartdnotify to root ...
Oct 05 11:31:37 lamachine kernel: ata9: COMRESET failed (errno=-16)
Oct 05 11:31:37 lamachine kernel: ata9: reset failed, giving up
Oct 05 11:31:37 lamachine kernel: ata9.00: disabled
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#7 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#6 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=124s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#6 CDB:
Read(16) 88 00 00 00 00 00 45 29 f4 d8 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#7 CDB:
Read(16) 88 00 00 00 00 00 45 2a 78 18 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160411160 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160377560 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#9 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#16 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#9 CDB:
Read(16) 88 00 00 00 00 00 45 29 fa 18 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#16 CDB:
Read(16) 88 00 00 00 00 00 45 2a 82 98 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160413848 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160378904 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#10 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#18 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#10 CDB:
Read(16) 88 00 00 00 00 00 45 29 ff 58 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#18 CDB:
Read(16) 88 00 00 00 00 00 45 2a 8d 18 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160416536 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160380248 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#11 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#31 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#11 CDB:
Read(16) 88 00 00 00 00 00 45 2a 04 98 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#31 CDB:
Read(16) 88 00 00 00 00 00 45 2a 97 98 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160419224 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160381592 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#18 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#18 CDB:
Read(16) 88 00 00 00 00 00 45 2a b7 18 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160427288 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#12 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:31:37 lamachine kernel: sd 8:0:0:0: [sde] tag#12 CDB:
Read(16) 88 00 00 00 00 00 45 2a 09 d8 00 00 05 40 00 00
Oct 05 11:31:37 lamachine kernel: blk_update_request: I/O error, dev
sde, sector 1160382936 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:31:37 lamachine smartd[1480]: Warning via
/usr/libexec/smartmontools/smartdnotify to root: successful
Oct 05 11:31:38 lamachine postfix/pickup[4381]: 00E50608EF11: uid=0 from=<root>
Oct 05 11:31:38 lamachine postfix/cleanup[4522]: 00E50608EF11:
message-id=<20201005103138.00E50608EF11@lamachine.localdomain>
Oct 05 11:31:38 lamachine postfix/qmgr[2080]: 00E50608EF11:
from=<root@lamachine.localdomain>, size=524, nrcpt=1 (queue active)
Oct 05 11:31:38 lamachine postfix/local[4524]: 00E50608EF11:
to=<root@lamachine.localdomain>, orig_to=<root>, relay=local,
delay=0.11, delays=0.08/0.01/0/0.03, dsn=2.0.0, status=sent (delivered
to mailbox)
Oct 05 11:31:38 lamachine postfix/qmgr[2080]: 00E50608EF11: removed
Oct 05 11:31:47 lamachine kernel: INFO: task md1_resync:3091 blocked
for more than 120 seconds.
Oct 05 11:31:47 lamachine kernel: Not tainted
4.18.0-193.14.2.el8_2.x86_64 #1
Oct 05 11:31:47 lamachine kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 05 11:31:47 lamachine kernel: md1_resync D 0 3091 2 0x80004080
Oct 05 11:31:47 lamachine kernel: Call Trace:
Oct 05 11:31:47 lamachine kernel: ? __schedule+0x24f/0x650
Oct 05 11:31:47 lamachine kernel: schedule+0x2f/0xa0
Oct 05 11:31:47 lamachine kernel: raid5_get_active_stripe+0x469/0x5f0 [raid456]
Oct 05 11:31:47 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:31:47 lamachine kernel: raid5_sync_request+0x387/0x3b0 [raid456]
Oct 05 11:31:47 lamachine kernel: ? cpumask_next+0x17/0x20
Oct 05 11:31:47 lamachine kernel: ? is_mddev_idle+0xcc/0x12a
Oct 05 11:31:47 lamachine kernel: md_do_sync.cold.83+0x424/0x953
Oct 05 11:31:47 lamachine kernel: ? xfrm_user_net_init+0x90/0xa0
Oct 05 11:31:47 lamachine kernel: ? __switch_to_asm+0x41/0x70
Oct 05 11:31:47 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:31:47 lamachine kernel: ? md_register_thread+0xd0/0xd0
Oct 05 11:31:47 lamachine kernel: md_thread+0x94/0x150
Oct 05 11:31:47 lamachine kernel: kthread+0x112/0x130
Oct 05 11:31:47 lamachine kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 05 11:31:47 lamachine kernel: ret_from_fork+0x35/0x40
Oct 05 11:32:41 lamachine kernel: ata10: SATA link up 3.0 Gbps
(SStatus 123 SControl 300)
Oct 05 11:32:46 lamachine kernel: ata10.00: qc timeout (cmd 0xec)
Oct 05 11:32:47 lamachine kernel: ata10.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:32:47 lamachine kernel: ata10.00: revalidation failed (errno=-5)
Oct 05 11:32:48 lamachine kernel: ata10: SATA link up 3.0 Gbps
(SStatus 123 SControl 300)
Oct 05 11:32:58 lamachine kernel: ata10.00: qc timeout (cmd 0xec)
Oct 05 11:32:59 lamachine kernel: ata10.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:32:59 lamachine kernel: ata10.00: revalidation failed (errno=-5)
Oct 05 11:32:59 lamachine kernel: ata10: limiting SATA link speed to 1.5 Gbps
Oct 05 11:32:59 lamachine kernel: ata10: SATA link up 3.0 Gbps
(SStatus 123 SControl 310)
Oct 05 11:33:30 lamachine kernel: ata10.00: qc timeout (cmd 0xec)
Oct 05 11:33:30 lamachine kernel: ata10.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:33:30 lamachine kernel: ata10.00: revalidation failed (errno=-5)
Oct 05 11:33:30 lamachine kernel: ata10.00: disabled
Oct 05 11:33:32 lamachine kernel: ata10: SATA link up 3.0 Gbps
(SStatus 123 SControl 310)
Oct 05 11:33:50 lamachine kernel: INFO: task md1_raid5:1304 blocked
for more than 120 seconds.
Oct 05 11:33:50 lamachine kernel: Not tainted
4.18.0-193.14.2.el8_2.x86_64 #1
Oct 05 11:33:50 lamachine kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 05 11:33:50 lamachine kernel: md1_raid5 D 0 1304 2 0x80004000
Oct 05 11:33:50 lamachine kernel: Call Trace:
Oct 05 11:33:50 lamachine kernel: ? __schedule+0x24f/0x650
Oct 05 11:33:50 lamachine kernel: schedule+0x2f/0xa0
Oct 05 11:33:50 lamachine kernel: io_schedule+0x12/0x40
Oct 05 11:33:50 lamachine kernel: blk_mq_get_tag+0x119/0x250
Oct 05 11:33:50 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:33:50 lamachine kernel: blk_mq_get_request+0xb7/0x3c0
Oct 05 11:33:50 lamachine kernel: blk_mq_make_request+0x134/0x5a0
Oct 05 11:33:50 lamachine kernel: generic_make_request+0xcf/0x310
Oct 05 11:33:50 lamachine kernel: ops_run_io+0x881/0xd30 [raid456]
Oct 05 11:33:50 lamachine kernel: ? ops_complete_check+0x50/0x50 [raid456]
Oct 05 11:33:50 lamachine kernel: handle_stripe+0xc47/0x1f80 [raid456]
Oct 05 11:33:50 lamachine kernel: ? __wake_up_common+0x7a/0x190
Oct 05 11:33:50 lamachine kernel:
handle_active_stripes.isra.73+0x3e7/0x5c0 [raid456]
Oct 05 11:33:50 lamachine kernel: raid5d+0x392/0x5b0 [raid456]
Oct 05 11:33:50 lamachine kernel: ? schedule_timeout+0x20d/0x310
Oct 05 11:33:50 lamachine kernel: ? _raw_spin_unlock_irqrestore+0x11/0x20
Oct 05 11:33:50 lamachine kernel: ? md_register_thread+0xd0/0xd0
Oct 05 11:33:50 lamachine kernel: md_thread+0x94/0x150
Oct 05 11:33:50 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:33:50 lamachine kernel: kthread+0x112/0x130
Oct 05 11:33:50 lamachine kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 05 11:33:50 lamachine kernel: ret_from_fork+0x35/0x40
Oct 05 11:33:50 lamachine kernel: INFO: task md1_resync:3091 blocked
for more than 120 seconds.
Oct 05 11:33:50 lamachine kernel: Not tainted
4.18.0-193.14.2.el8_2.x86_64 #1
Oct 05 11:33:50 lamachine kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 05 11:33:50 lamachine kernel: md1_resync D 0 3091 2 0x80004080
Oct 05 11:33:50 lamachine kernel: Call Trace:
Oct 05 11:33:50 lamachine kernel: ? __schedule+0x24f/0x650
Oct 05 11:33:50 lamachine kernel: schedule+0x2f/0xa0
Oct 05 11:33:50 lamachine kernel: raid5_get_active_stripe+0x469/0x5f0 [raid456]
Oct 05 11:33:50 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:33:50 lamachine kernel: raid5_sync_request+0x387/0x3b0 [raid456]
Oct 05 11:33:50 lamachine kernel: ? cpumask_next+0x17/0x20
Oct 05 11:33:50 lamachine kernel: ? is_mddev_idle+0xcc/0x12a
Oct 05 11:33:50 lamachine kernel: md_do_sync.cold.83+0x424/0x953
Oct 05 11:33:50 lamachine kernel: ? xfrm_user_net_init+0x90/0xa0
Oct 05 11:33:50 lamachine kernel: ? __switch_to_asm+0x41/0x70
Oct 05 11:33:50 lamachine kernel: ? finish_wait+0x80/0x80
Oct 05 11:33:50 lamachine kernel: ? md_register_thread+0xd0/0xd0
Oct 05 11:33:50 lamachine kernel: md_thread+0x94/0x150
Oct 05 11:33:50 lamachine kernel: kthread+0x112/0x130
Oct 05 11:33:50 lamachine kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 05 11:33:50 lamachine kernel: ret_from_fork+0x35/0x40
Oct 05 11:34:39 lamachine kernel: ata8.00: exception Emask 0x0 SAct
0xffffffff SErr 0x0 action 0x6 frozen
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:00:d8:2f:2b/05:00:45:00:00/40 tag 0 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:08:18:35:2b/05:00:45:00:00/40 tag 1 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:10:58:3a:2b/05:00:45:00:00/40 tag 2 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:18:98:3f:2b/05:00:45:00:00/40 tag 3 ncq dma 688128 in
res
40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:20:d8:44:2b/05:00:45:00:00/40 tag 4 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:28:18:4a:2b/05:00:45:00:00/40 tag 5 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:30:58:4f:2b/05:00:45:00:00/40 tag 6 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:38:98:54:2b/05:00:45:00:00/40 tag 7 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:40:d8:59:2b/05:00:45:00:00/40 tag 8 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:48:18:5f:2b/05:00:45:00:00/40 tag 9 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:50:58:64:2b/05:00:45:00:00/40 tag 10 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:58:98:69:2b/05:00:45:00:00/40 tag 11 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:60:d8:6e:2b/05:00:45:00:00/40 tag 12 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:68:18:74:2b/05:00:45:00:00/40 tag 13 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:70:58:79:2b/05:00:45:00:00/40 tag 14 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:78:98:7e:2b/05:00:45:00:00/40 tag 15 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:80:18:b3:2b/05:00:45:00:00/40 tag 16 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:88:58:b8:2b/05:00:45:00:00/40 tag 17 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:90:98:bd:2b/05:00:45:00:00/40 tag 18 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:98:d8:c2:2b/05:00:45:00:00/40 tag 19 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:a0:18:c8:2b/05:00:45:00:00/40 tag 20 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:a8:58:cd:2b/05:00:45:00:00/40 tag 21 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:b0:d8:83:2b/05:00:45:00:00/40 tag 22 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:b8:18:89:2b/05:00:45:00:00/40 tag 23 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:c0:58:8e:2b/05:00:45:00:00/40 tag 24 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:c8:98:93:2b/05:00:45:00:00/40 tag 25 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:d0:d8:98:2b/05:00:45:00:00/40 tag 26 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:d8:18:9e:2b/05:00:45:00:00/40 tag 27 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:e0:58:a3:2b/05:00:45:00:00/40 tag 28 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:e8:98:a8:2b/05:00:45:00:00/40 tag 29 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:f0:d8:ad:2b/05:00:45:00:00/40 tag 30 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:f8:98:d2:2b/05:00:45:00:00/40 tag 31 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8: hard resetting link
Oct 05 11:34:39 lamachine kernel: ata7.00: exception Emask 0x0 SAct
0xffffffff SErr 0x0 action 0x6 frozen
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:00:d8:98:2b/05:00:45:00:00/40 tag 0 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:08:18:9e:2b/05:00:45:00:00/40 tag 1 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:10:58:a3:2b/05:00:45:00:00/40 tag 2 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:18:d8:2f:2b/05:00:45:00:00/40 tag 3 ncq dma 688128 in
res
40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:20:18:35:2b/05:00:45:00:00/40 tag 4 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:28:58:3a:2b/05:00:45:00:00/40 tag 5 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:30:98:3f:2b/05:00:45:00:00/40 tag 6 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:38:d8:44:2b/05:00:45:00:00/40 tag 7 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:40:18:4a:2b/05:00:45:00:00/40 tag 8 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:48:58:4f:2b/05:00:45:00:00/40 tag 9 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:50:98:54:2b/05:00:45:00:00/40 tag 10 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:58:d8:59:2b/05:00:45:00:00/40 tag 11 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:60:18:5f:2b/05:00:45:00:00/40 tag 12 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:68:58:64:2b/05:00:45:00:00/40 tag 13 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:70:98:69:2b/05:00:45:00:00/40 tag 14 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:78:d8:6e:2b/05:00:45:00:00/40 tag 15 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:80:98:a8:2b/05:00:45:00:00/40 tag 16 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:88:d8:ad:2b/05:00:45:00:00/40 tag 17 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:90:18:74:2b/05:00:45:00:00/40 tag 18 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:98:58:79:2b/05:00:45:00:00/40 tag 19 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:a0:98:7e:2b/05:00:45:00:00/40 tag 20 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:a8:d8:83:2b/05:00:45:00:00/40 tag 21 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:b0:18:89:2b/05:00:45:00:00/40 tag 22 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:b8:58:8e:2b/05:00:45:00:00/40 tag 23 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:c0:18:b3:2b/05:00:45:00:00/40 tag 24 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:c8:58:b8:2b/05:00:45:00:00/40 tag 25 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:d0:98:bd:2b/05:00:45:00:00/40 tag 26 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:d8:d8:c2:2b/05:00:45:00:00/40 tag 27 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:e0:18:c8:2b/05:00:45:00:00/40 tag 28 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:e8:58:cd:2b/05:00:45:00:00/40 tag 29 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:f0:98:d2:2b/05:00:45:00:00/40 tag 30 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata7.00: cmd
60/40:f8:98:93:2b/05:00:45:00:00/40 tag 31 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata7.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata7: hard resetting link
Oct 05 11:34:40 lamachine kernel: ata8: SATA link up 6.0 Gbps (SStatus
133 SControl 300)
Oct 05 11:34:40 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 300)
Oct 05 11:34:45 lamachine kernel: ata7.00: qc timeout (cmd 0xec)
Oct 05 11:34:45 lamachine kernel: ata8.00: qc timeout (cmd 0xec)
Oct 05 11:34:46 lamachine kernel: ata7.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:34:46 lamachine kernel: ata7.00: revalidation failed (errno=-5)
Oct 05 11:34:46 lamachine kernel: ata7: hard resetting link
Oct 05 11:34:46 lamachine kernel: ata8.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:34:46 lamachine kernel: ata8.00: revalidation failed (errno=-5)
Oct 05 11:34:46 lamachine kernel: ata8: hard resetting link
Oct 05 11:34:46 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 300)
Oct 05 11:34:46 lamachine kernel: ata8: SATA link up 6.0 Gbps (SStatus
133 SControl 300)
Oct 05 11:34:57 lamachine kernel: ata7.00: qc timeout (cmd 0xec)
Oct 05 11:34:57 lamachine kernel: ata8.00: qc timeout (cmd 0xec)
Oct 05 11:34:57 lamachine kernel: ata8.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:34:57 lamachine kernel: ata8.00: revalidation failed (errno=-5)
Oct 05 11:34:57 lamachine kernel: ata8: limiting SATA link speed to 3.0 Gbps
Oct 05 11:34:57 lamachine kernel: ata8: hard resetting link
Oct 05 11:34:57 lamachine kernel: ata7.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:34:57 lamachine kernel: ata7.00: revalidation failed (errno=-5)
Oct 05 11:34:57 lamachine kernel: ata7: limiting SATA link speed to 3.0 Gbps
Oct 05 11:34:57 lamachine kernel: ata7: hard resetting link
Oct 05 11:34:58 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:34:58 lamachine kernel: ata8: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:35:29 lamachine kernel: ata7.00: qc timeout (cmd 0xec)
Oct 05 11:35:29 lamachine kernel: ata8.00: qc timeout (cmd 0xec)
Oct 05 11:35:29 lamachine kernel: ata8.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:35:29 lamachine kernel: ata8.00: revalidation failed (errno=-5)
Oct 05 11:35:29 lamachine kernel: ata8.00: disabled
Oct 05 11:35:29 lamachine kernel: ata7.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:35:29 lamachine kernel: ata7.00: revalidation failed (errno=-5)
Oct 05 11:35:29 lamachine kernel: ata7.00: disabled
Oct 05 11:35:30 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:35:30 lamachine kernel: ata8: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:35:31 lamachine kernel: ata8: EH complete
Oct 05 11:35:31 lamachine kernel: ata7: EH complete
Oct 05 11:35:31 lamachine kernel: scsi_io_completion_action: 115
callbacks suppressed
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#12 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#12 CDB:
Read(16) 88 00 00 00 00 00 45 2b d7 d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: print_req_error: 115 callbacks suppressed
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160501208 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#18 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#18 CDB:
Read(16) 88 00 00 00 00 00 45 2b d7 d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160501208 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#13 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#13 CDB:
Read(16) 88 00 00 00 00 00 45 2b dd 18 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160502552 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#19 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#19 CDB:
Read(16) 88 00 00 00 00 00 45 2b dd 18 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160502552 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#14 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#14 CDB:
Read(16) 88 00 00 00 00 00 45 2b e2 58 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160503896 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#20 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#20 CDB:
Read(16) 88 00 00 00 00 00 45 2b e2 58 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160503896 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#15 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#15 CDB:
Read(16) 88 00 00 00 00 00 45 2b e7 98 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160505240 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#21 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#21 CDB:
Read(16) 88 00 00 00 00 00 45 2b e7 98 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160505240 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#16 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#16 CDB:
Read(16) 88 00 00 00 00 00 45 2b ec d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160506584 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#22 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#22 CDB:
Read(16) 88 00 00 00 00 00 45 2b ec d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160506584 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: md/raid:md1: 23277 read_errors > 23276 stripes
Oct 05 11:35:31 lamachine kernel: md/raid:md1: Too many read errors,
failing device sde1.
Oct 05 11:35:31 lamachine kernel: md/raid:md1: Disk failure on sde1,
disabling device.
md/raid:md1: Operation continuing on
2 devices.
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433448 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433456 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433336 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433464 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433472 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433344 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433480 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433352 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433488 on sde1).
Oct 05 11:35:31 lamachine kernel: md/raid:md1: read error not
correctable (sector 1160433360 on sde1).
Oct 05 11:35:31 lamachine kernel: md: super_written gets error=10
Oct 05 11:35:31 lamachine kernel: md: super_written gets error=10
Oct 05 11:35:31 lamachine kernel: md: super_written gets error=10
Oct 05 11:35:31 lamachine kernel: md: super_written gets error=10
Oct 05 11:35:31 lamachine kernel: md: super_written gets error=10
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 13:10 do i need to give up on this setup Daniel Sanabria
@ 2020-10-05 13:17 ` Reindl Harald
2020-10-05 13:44 ` Roman Mamedov
1 sibling, 0 replies; 16+ messages in thread
From: Reindl Harald @ 2020-10-05 13:17 UTC (permalink / raw)
To: Daniel Sanabria, Linux-RAID
Am 05.10.20 um 15:10 schrieb Daniel Sanabria:
> Scrubbing ( # echo check >
> /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
>
> I'm attaching details of the array and disks (bloody wd greens) as
> well as journalctl errors providing some details about the issue.
>
> If you have any pointers on what might be the cause of this as well as
> any recommendations on how to improve things please let me thank you
> in advance ...
>
> 3 8 65 - faulty /dev/sde1
why would you scrub an array when you *clearly* lost a whole disk
instead first replace that one and rebuild the array?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 13:10 do i need to give up on this setup Daniel Sanabria
2020-10-05 13:17 ` Reindl Harald
@ 2020-10-05 13:44 ` Roman Mamedov
[not found] ` <CAHscji0pNezf6xCpjWto5-21ayoCeLWm34GTYh5TSgxkOw90mw@mail.gmail.com>
1 sibling, 1 reply; 16+ messages in thread
From: Roman Mamedov @ 2020-10-05 13:44 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: Linux-RAID
On Mon, 5 Oct 2020 14:10:25 +0100
Daniel Sanabria <sanabria.d@gmail.com> wrote:
> Hi all,
>
> Scrubbing ( # echo check >
> /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
>
> I'm attaching details of the array and disks (bloody wd greens) as
> well as journalctl errors providing some details about the issue.
>
> If you have any pointers on what might be the cause of this as well as
> any recommendations on how to improve things please let me thank you
> in advance ...
>
> I have backups of the data so happy to move this to a different setup
> you might recommend (apps will be mostly reading from the array via
> NFS since most of the content will be media).
>
> My suspicion is that a timer service is kicking in and disrupting the
> scrubbing somehow but can't pinpoint what causes this.
It looks like a drive is dropping off the bus and then failing to reidentify,
could be bad cabling/controller/PSU, or just a bad drive. You should post
"smartctl -a" of all drives as well.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
[not found] ` <CAHscji0pNezf6xCpjWto5-21ayoCeLWm34GTYh5TSgxkOw90mw@mail.gmail.com>
@ 2020-10-05 14:04 ` Roman Mamedov
2020-10-05 14:10 ` Reindl Harald
2020-10-05 14:28 ` Daniel Sanabria
0 siblings, 2 replies; 16+ messages in thread
From: Roman Mamedov @ 2020-10-05 14:04 UTC (permalink / raw)
To: Daniel Sanabria, Linux-RAID
On Mon, 5 Oct 2020 14:59:35 +0100
Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > It looks like a drive is dropping off the bus and then failing to reidentify,
> > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > "smartctl -a" of all drives as well.
I meant not to me personally, but to the mailing list. The drives seem OK
though, even sde.
> [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
> [sudo] password for dan:
> smartctl 6.6 2017-11-05 r4594
> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Western Digital Green
> Device Model: WDC WD30EZRX-00D8PB0
> Serial Number: WD-WCC4NCWT13RF
> LU WWN Device Id: 5 0014ee 25fc9e460
> Firmware Version: 80.00A80
> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5400 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-2 (minor revision not indicated)
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Oct 5 14:58:34 2020 BST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 0) The previous self-test routine completed
> without error or no self-test has ever
> been run.
> Total time to complete Offline
> data collection: (38940) seconds.
> Offline data collection
> capabilities: (0x7b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> General Purpose Logging supported.
> Short self-test routine
> recommended polling time: ( 2) minutes.
> Extended self-test routine
> recommended polling time: ( 391) minutes.
> Conveyance self-test routine
> recommended polling time: ( 5) minutes.
> SCT capabilities: (0x7035) SCT Status supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> Always - 0
> 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
> Always - 6075
> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> Always - 81
> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> Always - 0
> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> Always - 0
> 9 Power_On_Hours 0x0032 075 075 000 Old_age
> Always - 18577
> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 81
> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> Always - 46
> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> Always - 176661
> 194 Temperature_Celsius 0x0022 122 109 000 Old_age
> Always - 28
> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> Always - 0
> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> Always - 0
> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> Offline - 0
> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> Always - 0
> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> Offline - 0
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Completed without error 00% 17479 -
> # 2 Short offline Completed without error 00% 15531 -
>
> SMART Selective self-test log data structure revision number 1
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Not_testing
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
> smartctl 6.6 2017-11-05 r4594
> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Western Digital Green
> Device Model: WDC WD30EZRX-00D8PB0
> Serial Number: WD-WCC4NPRDD6D7
> LU WWN Device Id: 5 0014ee 25fca27b1
> Firmware Version: 80.00A80
> User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5400 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-2 (minor revision not indicated)
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Oct 5 14:58:54 2020 BST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 0) The previous self-test routine completed
> without error or no self-test has ever
> been run.
> Total time to complete Offline
> data collection: (39060) seconds.
> Offline data collection
> capabilities: (0x7b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> General Purpose Logging supported.
> Short self-test routine
> recommended polling time: ( 2) minutes.
> Extended self-test routine
> recommended polling time: ( 392) minutes.
> Conveyance self-test routine
> recommended polling time: ( 5) minutes.
> SCT capabilities: (0x7035) SCT Status supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> Always - 0
> 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
> Always - 6100
> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> Always - 81
> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> Always - 0
> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> Always - 0
> 9 Power_On_Hours 0x0032 075 075 000 Old_age
> Always - 18580
> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 81
> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> Always - 53
> 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
> Always - 192427
> 194 Temperature_Celsius 0x0022 121 108 000 Old_age
> Always - 29
> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> Always - 0
> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> Always - 0
> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> Offline - 0
> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> Always - 0
> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> Offline - 0
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Completed without error 00% 17481 -
> # 2 Short offline Completed without error 00% 15534 -
>
> SMART Selective self-test log data structure revision number 1
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Not_testing
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> [dan@lamachine ~]$ sudo smartctl -a /dev/sde
> smartctl 6.6 2017-11-05 r4594
> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Western Digital Green
> Device Model: WDC WD30EZRX-00D8PB0
> Serial Number: WD-WCC4N1294906
> LU WWN Device Id: 5 0014ee 25f968120
> Firmware Version: 80.00A80
> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5400 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-2 (minor revision not indicated)
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Oct 5 14:58:57 2020 BST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 0) The previous self-test routine completed
> without error or no self-test has ever
> been run.
> Total time to complete Offline
> data collection: (43200) seconds.
> Offline data collection
> capabilities: (0x7b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> General Purpose Logging supported.
> Short self-test routine
> recommended polling time: ( 2) minutes.
> Extended self-test routine
> recommended polling time: ( 433) minutes.
> Conveyance self-test routine
> recommended polling time: ( 5) minutes.
> SCT capabilities: (0x7035) SCT Status supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> Always - 0
> 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
> Always - 6158
> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> Always - 80
> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> Always - 0
> 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
> Always - 0
> 9 Power_On_Hours 0x0032 075 075 000 Old_age
> Always - 18465
> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 80
> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> Always - 53
> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> Always - 174015
> 194 Temperature_Celsius 0x0022 121 107 000 Old_age
> Always - 29
> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> Always - 0
> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> Always - 0
> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> Offline - 0
> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> Always - 0
> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> Offline - 0
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Completed without error 00% 17347 -
> # 2 Short offline Completed without error 00% 15414 -
>
> SMART Selective self-test log data structure revision number 1
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Not_testing
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> [dan@lamachine ~]$
>
>
> On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
> >
> > On Mon, 5 Oct 2020 14:10:25 +0100
> > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Scrubbing ( # echo check >
> > > /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
> > >
> > > I'm attaching details of the array and disks (bloody wd greens) as
> > > well as journalctl errors providing some details about the issue.
> > >
> > > If you have any pointers on what might be the cause of this as well as
> > > any recommendations on how to improve things please let me thank you
> > > in advance ...
> > >
> > > I have backups of the data so happy to move this to a different setup
> > > you might recommend (apps will be mostly reading from the array via
> > > NFS since most of the content will be media).
> > >
> > > My suspicion is that a timer service is kicking in and disrupting the
> > > scrubbing somehow but can't pinpoint what causes this.
> >
> > It looks like a drive is dropping off the bus and then failing to reidentify,
> > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > "smartctl -a" of all drives as well.
> >
> > --
> > With respect,
> > Roman
--
With respect,
Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 14:04 ` Roman Mamedov
@ 2020-10-05 14:10 ` Reindl Harald
2020-10-05 14:28 ` Daniel Sanabria
1 sibling, 0 replies; 16+ messages in thread
From: Reindl Harald @ 2020-10-05 14:10 UTC (permalink / raw)
To: Roman Mamedov, Daniel Sanabria, Linux-RAID
Am 05.10.20 um 16:04 schrieb Roman Mamedov:
> On Mon, 5 Oct 2020 14:59:35 +0100
> Daniel Sanabria <sanabria.d@gmail.com> wrote:
>
>>> It looks like a drive is dropping off the bus and then failing to reidentify,
>>> could be bad cabling/controller/PSU, or just a bad drive. You should post
>>> "smartctl -a" of all drives as well.
>
> I meant not to me personally, but to the mailing list. The drives seem OK
> though, even sde.
you have a hardware problem and it#s no uncommon when one of your disks
is going crazy under laod that due the reset of the crontroller a second
one on the same bus is also reset
either one of your disks is faulty, the controller is faulty or you have
an issue with your cables
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:48:18:5f:2b/05:00:45:00:00/40 tag 9 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:50:58:64:2b/05:00:45:00:00/40 tag 10 ncq dma 688128 in
res
40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:34:39 lamachine kernel: ata8.00: status: { DRDY }
Oct 05 11:34:39 lamachine kernel: ata8.00: failed command: READ FPDMA QUEUED
Oct 05 11:34:39 lamachine kernel: ata8.00: cmd
60/40:58:98:69:2b/05:00:45:00:00/40 tag 11 ncq dma 688128 in
res
40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 05 11:35:29 lamachine kernel: ata7.00: qc timeout (cmd 0xec)
Oct 05 11:35:29 lamachine kernel: ata8.00: qc timeout (cmd 0xec)
Oct 05 11:35:29 lamachine kernel: ata8.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:35:29 lamachine kernel: ata8.00: revalidation failed (errno=-5)
Oct 05 11:35:29 lamachine kernel: ata8.00: disabled
Oct 05 11:35:29 lamachine kernel: ata7.00: failed to IDENTIFY (I/O
error, err_mask=0x4)
Oct 05 11:35:29 lamachine kernel: ata7.00: revalidation failed (errno=-5)
Oct 05 11:35:29 lamachine kernel: ata7.00: disabled
Oct 05 11:35:30 lamachine kernel: ata7: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:35:30 lamachine kernel: ata8: SATA link up 6.0 Gbps (SStatus
133 SControl 320)
Oct 05 11:35:31 lamachine kernel: ata8: EH complete
Oct 05 11:35:31 lamachine kernel: ata7: EH complete
Oct 05 11:35:31 lamachine kernel: scsi_io_completion_action: 115
callbacks suppressed
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#12 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 7:0:0:0: [sdd] tag#12 CDB:
Read(16) 88 00 00 00 00 00 45 2b d7 d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: print_req_error: 115 callbacks suppressed
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdd, sector 1160501208 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
class 0
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#18 FAILED
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Oct 05 11:35:31 lamachine kernel: sd 6:0:0:0: [sdc] tag#18 CDB:
Read(16) 88 00 00 00 00 00 45 2b d7 d8 00 00 05 40 00 00
Oct 05 11:35:31 lamachine kernel: blk_update_request: I/O error, dev
sdc, sector 1160501208 op 0x0:(READ) flags 0x4000 phys_seg 168 prio
>> [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
>> [sudo] password for dan:
>> smartctl 6.6 2017-11-05 r4594
>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: Western Digital Green
>> Device Model: WDC WD30EZRX-00D8PB0
>> Serial Number: WD-WCC4NCWT13RF
>> LU WWN Device Id: 5 0014ee 25fc9e460
>> Firmware Version: 80.00A80
>> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>> Rotation Rate: 5400 rpm
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: ACS-2 (minor revision not indicated)
>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is: Mon Oct 5 14:58:34 2020 BST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> General SMART Values:
>> Offline data collection status: (0x82) Offline data collection activity
>> was completed without error.
>> Auto Offline Data Collection: Enabled.
>> Self-test execution status: ( 0) The previous self-test routine completed
>> without error or no self-test has ever
>> been run.
>> Total time to complete Offline
>> data collection: (38940) seconds.
>> Offline data collection
>> capabilities: (0x7b) SMART execute Offline immediate.
>> Auto Offline data collection on/off support.
>> Suspend Offline collection upon new
>> command.
>> Offline surface scan supported.
>> Self-test supported.
>> Conveyance Self-test supported.
>> Selective Self-test supported.
>> SMART capabilities: (0x0003) Saves SMART data before entering
>> power-saving mode.
>> Supports SMART auto save timer.
>> Error logging capability: (0x01) Error logging supported.
>> General Purpose Logging supported.
>> Short self-test routine
>> recommended polling time: ( 2) minutes.
>> Extended self-test routine
>> recommended polling time: ( 391) minutes.
>> Conveyance self-test routine
>> recommended polling time: ( 5) minutes.
>> SCT capabilities: (0x7035) SCT Status supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> SMART Attributes Data Structure revision number: 16
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>> UPDATED WHEN_FAILED RAW_VALUE
>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>> Always - 0
>> 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
>> Always - 6075
>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>> Always - 81
>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>> Always - 0
>> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
>> Always - 0
>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>> Always - 18577
>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 81
>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>> Always - 46
>> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
>> Always - 176661
>> 194 Temperature_Celsius 0x0022 122 109 000 Old_age
>> Always - 28
>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>> Always - 0
>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>> Offline - 0
>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>> Offline - 0
>>
>> SMART Error Log Version: 1
>> No Errors Logged
>>
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Extended offline Completed without error 00% 17479 -
>> # 2 Short offline Completed without error 00% 15531 -
>>
>> SMART Selective self-test log data structure revision number 1
>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>> 1 0 0 Not_testing
>> 2 0 0 Not_testing
>> 3 0 0 Not_testing
>> 4 0 0 Not_testing
>> 5 0 0 Not_testing
>> Selective self-test flags (0x0):
>> After scanning selected spans, do NOT read-scan remainder of disk.
>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>
>> [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
>> smartctl 6.6 2017-11-05 r4594
>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: Western Digital Green
>> Device Model: WDC WD30EZRX-00D8PB0
>> Serial Number: WD-WCC4NPRDD6D7
>> LU WWN Device Id: 5 0014ee 25fca27b1
>> Firmware Version: 80.00A80
>> User Capacity: 3,000,592,982,016 bytes [3.00 TB]
>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>> Rotation Rate: 5400 rpm
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: ACS-2 (minor revision not indicated)
>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is: Mon Oct 5 14:58:54 2020 BST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> General SMART Values:
>> Offline data collection status: (0x82) Offline data collection activity
>> was completed without error.
>> Auto Offline Data Collection: Enabled.
>> Self-test execution status: ( 0) The previous self-test routine completed
>> without error or no self-test has ever
>> been run.
>> Total time to complete Offline
>> data collection: (39060) seconds.
>> Offline data collection
>> capabilities: (0x7b) SMART execute Offline immediate.
>> Auto Offline data collection on/off support.
>> Suspend Offline collection upon new
>> command.
>> Offline surface scan supported.
>> Self-test supported.
>> Conveyance Self-test supported.
>> Selective Self-test supported.
>> SMART capabilities: (0x0003) Saves SMART data before entering
>> power-saving mode.
>> Supports SMART auto save timer.
>> Error logging capability: (0x01) Error logging supported.
>> General Purpose Logging supported.
>> Short self-test routine
>> recommended polling time: ( 2) minutes.
>> Extended self-test routine
>> recommended polling time: ( 392) minutes.
>> Conveyance self-test routine
>> recommended polling time: ( 5) minutes.
>> SCT capabilities: (0x7035) SCT Status supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> SMART Attributes Data Structure revision number: 16
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>> UPDATED WHEN_FAILED RAW_VALUE
>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>> Always - 0
>> 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
>> Always - 6100
>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>> Always - 81
>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>> Always - 0
>> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
>> Always - 0
>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>> Always - 18580
>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 81
>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>> Always - 53
>> 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
>> Always - 192427
>> 194 Temperature_Celsius 0x0022 121 108 000 Old_age
>> Always - 29
>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>> Always - 0
>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>> Offline - 0
>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>> Offline - 0
>>
>> SMART Error Log Version: 1
>> No Errors Logged
>>
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Extended offline Completed without error 00% 17481 -
>> # 2 Short offline Completed without error 00% 15534 -
>>
>> SMART Selective self-test log data structure revision number 1
>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>> 1 0 0 Not_testing
>> 2 0 0 Not_testing
>> 3 0 0 Not_testing
>> 4 0 0 Not_testing
>> 5 0 0 Not_testing
>> Selective self-test flags (0x0):
>> After scanning selected spans, do NOT read-scan remainder of disk.
>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>
>> [dan@lamachine ~]$ sudo smartctl -a /dev/sde
>> smartctl 6.6 2017-11-05 r4594
>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: Western Digital Green
>> Device Model: WDC WD30EZRX-00D8PB0
>> Serial Number: WD-WCC4N1294906
>> LU WWN Device Id: 5 0014ee 25f968120
>> Firmware Version: 80.00A80
>> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>> Rotation Rate: 5400 rpm
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: ACS-2 (minor revision not indicated)
>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is: Mon Oct 5 14:58:57 2020 BST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> General SMART Values:
>> Offline data collection status: (0x82) Offline data collection activity
>> was completed without error.
>> Auto Offline Data Collection: Enabled.
>> Self-test execution status: ( 0) The previous self-test routine completed
>> without error or no self-test has ever
>> been run.
>> Total time to complete Offline
>> data collection: (43200) seconds.
>> Offline data collection
>> capabilities: (0x7b) SMART execute Offline immediate.
>> Auto Offline data collection on/off support.
>> Suspend Offline collection upon new
>> command.
>> Offline surface scan supported.
>> Self-test supported.
>> Conveyance Self-test supported.
>> Selective Self-test supported.
>> SMART capabilities: (0x0003) Saves SMART data before entering
>> power-saving mode.
>> Supports SMART auto save timer.
>> Error logging capability: (0x01) Error logging supported.
>> General Purpose Logging supported.
>> Short self-test routine
>> recommended polling time: ( 2) minutes.
>> Extended self-test routine
>> recommended polling time: ( 433) minutes.
>> Conveyance self-test routine
>> recommended polling time: ( 5) minutes.
>> SCT capabilities: (0x7035) SCT Status supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> SMART Attributes Data Structure revision number: 16
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>> UPDATED WHEN_FAILED RAW_VALUE
>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>> Always - 0
>> 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
>> Always - 6158
>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>> Always - 80
>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>> Always - 0
>> 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
>> Always - 0
>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>> Always - 18465
>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>> Always - 0
>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 80
>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>> Always - 53
>> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
>> Always - 174015
>> 194 Temperature_Celsius 0x0022 121 107 000 Old_age
>> Always - 29
>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>> Always - 0
>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>> Offline - 0
>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>> Always - 0
>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>> Offline - 0
>>
>> SMART Error Log Version: 1
>> No Errors Logged
>>
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Extended offline Completed without error 00% 17347 -
>> # 2 Short offline Completed without error 00% 15414 -
>>
>> SMART Selective self-test log data structure revision number 1
>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>> 1 0 0 Not_testing
>> 2 0 0 Not_testing
>> 3 0 0 Not_testing
>> 4 0 0 Not_testing
>> 5 0 0 Not_testing
>> Selective self-test flags (0x0):
>> After scanning selected spans, do NOT read-scan remainder of disk.
>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>
>> [dan@lamachine ~]$
>>
>>
>> On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
>>>
>>> On Mon, 5 Oct 2020 14:10:25 +0100
>>> Daniel Sanabria <sanabria.d@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Scrubbing ( # echo check >
>>>> /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
>>>>
>>>> I'm attaching details of the array and disks (bloody wd greens) as
>>>> well as journalctl errors providing some details about the issue.
>>>>
>>>> If you have any pointers on what might be the cause of this as well as
>>>> any recommendations on how to improve things please let me thank you
>>>> in advance ...
>>>>
>>>> I have backups of the data so happy to move this to a different setup
>>>> you might recommend (apps will be mostly reading from the array via
>>>> NFS since most of the content will be media).
>>>>
>>>> My suspicion is that a timer service is kicking in and disrupting the
>>>> scrubbing somehow but can't pinpoint what causes this.
>>>
>>> It looks like a drive is dropping off the bus and then failing to reidentify,
>>> could be bad cabling/controller/PSU, or just a bad drive. You should post
>>> "smartctl -a" of all drives as well.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 14:04 ` Roman Mamedov
2020-10-05 14:10 ` Reindl Harald
@ 2020-10-05 14:28 ` Daniel Sanabria
2020-10-05 15:58 ` Roger Heflin
1 sibling, 1 reply; 16+ messages in thread
From: Daniel Sanabria @ 2020-10-05 14:28 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
> I meant not to me personally, but to the mailing list. The drives seem OK
> though, even sde.
Sorry missed the reply-all button
On Mon, 5 Oct 2020 at 15:04, Roman Mamedov <rm@romanrm.net> wrote:
>
> On Mon, 5 Oct 2020 14:59:35 +0100
> Daniel Sanabria <sanabria.d@gmail.com> wrote:
>
> > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > "smartctl -a" of all drives as well.
>
> I meant not to me personally, but to the mailing list. The drives seem OK
> though, even sde.
>
> > [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
> > [sudo] password for dan:
> > smartctl 6.6 2017-11-05 r4594
> > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> >
> > === START OF INFORMATION SECTION ===
> > Model Family: Western Digital Green
> > Device Model: WDC WD30EZRX-00D8PB0
> > Serial Number: WD-WCC4NCWT13RF
> > LU WWN Device Id: 5 0014ee 25fc9e460
> > Firmware Version: 80.00A80
> > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > Rotation Rate: 5400 rpm
> > Device is: In smartctl database [for details use: -P show]
> > ATA Version is: ACS-2 (minor revision not indicated)
> > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > Local Time is: Mon Oct 5 14:58:34 2020 BST
> > SMART support is: Available - device has SMART capability.
> > SMART support is: Enabled
> >
> > === START OF READ SMART DATA SECTION ===
> > SMART overall-health self-assessment test result: PASSED
> >
> > General SMART Values:
> > Offline data collection status: (0x82) Offline data collection activity
> > was completed without error.
> > Auto Offline Data Collection: Enabled.
> > Self-test execution status: ( 0) The previous self-test routine completed
> > without error or no self-test has ever
> > been run.
> > Total time to complete Offline
> > data collection: (38940) seconds.
> > Offline data collection
> > capabilities: (0x7b) SMART execute Offline immediate.
> > Auto Offline data collection on/off support.
> > Suspend Offline collection upon new
> > command.
> > Offline surface scan supported.
> > Self-test supported.
> > Conveyance Self-test supported.
> > Selective Self-test supported.
> > SMART capabilities: (0x0003) Saves SMART data before entering
> > power-saving mode.
> > Supports SMART auto save timer.
> > Error logging capability: (0x01) Error logging supported.
> > General Purpose Logging supported.
> > Short self-test routine
> > recommended polling time: ( 2) minutes.
> > Extended self-test routine
> > recommended polling time: ( 391) minutes.
> > Conveyance self-test routine
> > recommended polling time: ( 5) minutes.
> > SCT capabilities: (0x7035) SCT Status supported.
> > SCT Feature Control supported.
> > SCT Data Table supported.
> >
> > SMART Attributes Data Structure revision number: 16
> > Vendor Specific SMART Attributes with Thresholds:
> > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > UPDATED WHEN_FAILED RAW_VALUE
> > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > Always - 0
> > 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
> > Always - 6075
> > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > Always - 81
> > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > Always - 0
> > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > Always - 0
> > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > Always - 18577
> > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > Always - 81
> > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > Always - 46
> > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > Always - 176661
> > 194 Temperature_Celsius 0x0022 122 109 000 Old_age
> > Always - 28
> > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > Always - 0
> > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > Offline - 0
> > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > Offline - 0
> >
> > SMART Error Log Version: 1
> > No Errors Logged
> >
> > SMART Self-test log structure revision number 1
> > Num Test_Description Status Remaining
> > LifeTime(hours) LBA_of_first_error
> > # 1 Extended offline Completed without error 00% 17479 -
> > # 2 Short offline Completed without error 00% 15531 -
> >
> > SMART Selective self-test log data structure revision number 1
> > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > 1 0 0 Not_testing
> > 2 0 0 Not_testing
> > 3 0 0 Not_testing
> > 4 0 0 Not_testing
> > 5 0 0 Not_testing
> > Selective self-test flags (0x0):
> > After scanning selected spans, do NOT read-scan remainder of disk.
> > If Selective self-test is pending on power-up, resume after 0 minute delay.
> >
> > [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
> > smartctl 6.6 2017-11-05 r4594
> > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> >
> > === START OF INFORMATION SECTION ===
> > Model Family: Western Digital Green
> > Device Model: WDC WD30EZRX-00D8PB0
> > Serial Number: WD-WCC4NPRDD6D7
> > LU WWN Device Id: 5 0014ee 25fca27b1
> > Firmware Version: 80.00A80
> > User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > Rotation Rate: 5400 rpm
> > Device is: In smartctl database [for details use: -P show]
> > ATA Version is: ACS-2 (minor revision not indicated)
> > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > Local Time is: Mon Oct 5 14:58:54 2020 BST
> > SMART support is: Available - device has SMART capability.
> > SMART support is: Enabled
> >
> > === START OF READ SMART DATA SECTION ===
> > SMART overall-health self-assessment test result: PASSED
> >
> > General SMART Values:
> > Offline data collection status: (0x82) Offline data collection activity
> > was completed without error.
> > Auto Offline Data Collection: Enabled.
> > Self-test execution status: ( 0) The previous self-test routine completed
> > without error or no self-test has ever
> > been run.
> > Total time to complete Offline
> > data collection: (39060) seconds.
> > Offline data collection
> > capabilities: (0x7b) SMART execute Offline immediate.
> > Auto Offline data collection on/off support.
> > Suspend Offline collection upon new
> > command.
> > Offline surface scan supported.
> > Self-test supported.
> > Conveyance Self-test supported.
> > Selective Self-test supported.
> > SMART capabilities: (0x0003) Saves SMART data before entering
> > power-saving mode.
> > Supports SMART auto save timer.
> > Error logging capability: (0x01) Error logging supported.
> > General Purpose Logging supported.
> > Short self-test routine
> > recommended polling time: ( 2) minutes.
> > Extended self-test routine
> > recommended polling time: ( 392) minutes.
> > Conveyance self-test routine
> > recommended polling time: ( 5) minutes.
> > SCT capabilities: (0x7035) SCT Status supported.
> > SCT Feature Control supported.
> > SCT Data Table supported.
> >
> > SMART Attributes Data Structure revision number: 16
> > Vendor Specific SMART Attributes with Thresholds:
> > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > UPDATED WHEN_FAILED RAW_VALUE
> > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > Always - 0
> > 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
> > Always - 6100
> > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > Always - 81
> > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > Always - 0
> > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > Always - 0
> > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > Always - 18580
> > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > Always - 81
> > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > Always - 53
> > 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
> > Always - 192427
> > 194 Temperature_Celsius 0x0022 121 108 000 Old_age
> > Always - 29
> > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > Always - 0
> > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > Offline - 0
> > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > Offline - 0
> >
> > SMART Error Log Version: 1
> > No Errors Logged
> >
> > SMART Self-test log structure revision number 1
> > Num Test_Description Status Remaining
> > LifeTime(hours) LBA_of_first_error
> > # 1 Extended offline Completed without error 00% 17481 -
> > # 2 Short offline Completed without error 00% 15534 -
> >
> > SMART Selective self-test log data structure revision number 1
> > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > 1 0 0 Not_testing
> > 2 0 0 Not_testing
> > 3 0 0 Not_testing
> > 4 0 0 Not_testing
> > 5 0 0 Not_testing
> > Selective self-test flags (0x0):
> > After scanning selected spans, do NOT read-scan remainder of disk.
> > If Selective self-test is pending on power-up, resume after 0 minute delay.
> >
> > [dan@lamachine ~]$ sudo smartctl -a /dev/sde
> > smartctl 6.6 2017-11-05 r4594
> > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> >
> > === START OF INFORMATION SECTION ===
> > Model Family: Western Digital Green
> > Device Model: WDC WD30EZRX-00D8PB0
> > Serial Number: WD-WCC4N1294906
> > LU WWN Device Id: 5 0014ee 25f968120
> > Firmware Version: 80.00A80
> > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > Rotation Rate: 5400 rpm
> > Device is: In smartctl database [for details use: -P show]
> > ATA Version is: ACS-2 (minor revision not indicated)
> > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > Local Time is: Mon Oct 5 14:58:57 2020 BST
> > SMART support is: Available - device has SMART capability.
> > SMART support is: Enabled
> >
> > === START OF READ SMART DATA SECTION ===
> > SMART overall-health self-assessment test result: PASSED
> >
> > General SMART Values:
> > Offline data collection status: (0x82) Offline data collection activity
> > was completed without error.
> > Auto Offline Data Collection: Enabled.
> > Self-test execution status: ( 0) The previous self-test routine completed
> > without error or no self-test has ever
> > been run.
> > Total time to complete Offline
> > data collection: (43200) seconds.
> > Offline data collection
> > capabilities: (0x7b) SMART execute Offline immediate.
> > Auto Offline data collection on/off support.
> > Suspend Offline collection upon new
> > command.
> > Offline surface scan supported.
> > Self-test supported.
> > Conveyance Self-test supported.
> > Selective Self-test supported.
> > SMART capabilities: (0x0003) Saves SMART data before entering
> > power-saving mode.
> > Supports SMART auto save timer.
> > Error logging capability: (0x01) Error logging supported.
> > General Purpose Logging supported.
> > Short self-test routine
> > recommended polling time: ( 2) minutes.
> > Extended self-test routine
> > recommended polling time: ( 433) minutes.
> > Conveyance self-test routine
> > recommended polling time: ( 5) minutes.
> > SCT capabilities: (0x7035) SCT Status supported.
> > SCT Feature Control supported.
> > SCT Data Table supported.
> >
> > SMART Attributes Data Structure revision number: 16
> > Vendor Specific SMART Attributes with Thresholds:
> > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > UPDATED WHEN_FAILED RAW_VALUE
> > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > Always - 0
> > 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
> > Always - 6158
> > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > Always - 80
> > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > Always - 0
> > 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
> > Always - 0
> > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > Always - 18465
> > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > Always - 0
> > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > Always - 80
> > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > Always - 53
> > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > Always - 174015
> > 194 Temperature_Celsius 0x0022 121 107 000 Old_age
> > Always - 29
> > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > Always - 0
> > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > Offline - 0
> > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > Always - 0
> > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > Offline - 0
> >
> > SMART Error Log Version: 1
> > No Errors Logged
> >
> > SMART Self-test log structure revision number 1
> > Num Test_Description Status Remaining
> > LifeTime(hours) LBA_of_first_error
> > # 1 Extended offline Completed without error 00% 17347 -
> > # 2 Short offline Completed without error 00% 15414 -
> >
> > SMART Selective self-test log data structure revision number 1
> > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > 1 0 0 Not_testing
> > 2 0 0 Not_testing
> > 3 0 0 Not_testing
> > 4 0 0 Not_testing
> > 5 0 0 Not_testing
> > Selective self-test flags (0x0):
> > After scanning selected spans, do NOT read-scan remainder of disk.
> > If Selective self-test is pending on power-up, resume after 0 minute delay.
> >
> > [dan@lamachine ~]$
> >
> >
> > On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
> > >
> > > On Mon, 5 Oct 2020 14:10:25 +0100
> > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Scrubbing ( # echo check >
> > > > /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
> > > >
> > > > I'm attaching details of the array and disks (bloody wd greens) as
> > > > well as journalctl errors providing some details about the issue.
> > > >
> > > > If you have any pointers on what might be the cause of this as well as
> > > > any recommendations on how to improve things please let me thank you
> > > > in advance ...
> > > >
> > > > I have backups of the data so happy to move this to a different setup
> > > > you might recommend (apps will be mostly reading from the array via
> > > > NFS since most of the content will be media).
> > > >
> > > > My suspicion is that a timer service is kicking in and disrupting the
> > > > scrubbing somehow but can't pinpoint what causes this.
> > >
> > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > "smartctl -a" of all drives as well.
> > >
> > > --
> > > With respect,
> > > Roman
>
>
> --
> With respect,
> Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 14:28 ` Daniel Sanabria
@ 2020-10-05 15:58 ` Roger Heflin
2020-10-06 7:56 ` Daniel Sanabria
0 siblings, 1 reply; 16+ messages in thread
From: Roger Heflin @ 2020-10-05 15:58 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: Roman Mamedov, Linux-RAID
what they said you have a hardware problem.
it could be about anything previously mentioned and could also be the
power supply being unable to provide a stable 12V for the disks.
You should provide the list more specifics on your hw setup, of
interest are what kind of SATA/SAS ports you are using and how the
disk are cabled in.
Note that there are a number of controllers that aren't the most
reliable and some of those controllers when something happens will
stop responding for all disks connected to it.
I have also seen badly designed motherboards have
build-in(non-AMD/non-Intel chips) sata ports that don't work under any
load that uses more than a single disk at a time, and/or acts badly
when given smart commands.
On Mon, Oct 5, 2020 at 9:30 AM Daniel Sanabria <sanabria.d@gmail.com> wrote:
>
> > I meant not to me personally, but to the mailing list. The drives seem OK
> > though, even sde.
>
> Sorry missed the reply-all button
>
> On Mon, 5 Oct 2020 at 15:04, Roman Mamedov <rm@romanrm.net> wrote:
> >
> > On Mon, 5 Oct 2020 14:59:35 +0100
> > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> >
> > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > "smartctl -a" of all drives as well.
> >
> > I meant not to me personally, but to the mailing list. The drives seem OK
> > though, even sde.
> >
> > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
> > > [sudo] password for dan:
> > > smartctl 6.6 2017-11-05 r4594
> > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > >
> > > === START OF INFORMATION SECTION ===
> > > Model Family: Western Digital Green
> > > Device Model: WDC WD30EZRX-00D8PB0
> > > Serial Number: WD-WCC4NCWT13RF
> > > LU WWN Device Id: 5 0014ee 25fc9e460
> > > Firmware Version: 80.00A80
> > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > Rotation Rate: 5400 rpm
> > > Device is: In smartctl database [for details use: -P show]
> > > ATA Version is: ACS-2 (minor revision not indicated)
> > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > Local Time is: Mon Oct 5 14:58:34 2020 BST
> > > SMART support is: Available - device has SMART capability.
> > > SMART support is: Enabled
> > >
> > > === START OF READ SMART DATA SECTION ===
> > > SMART overall-health self-assessment test result: PASSED
> > >
> > > General SMART Values:
> > > Offline data collection status: (0x82) Offline data collection activity
> > > was completed without error.
> > > Auto Offline Data Collection: Enabled.
> > > Self-test execution status: ( 0) The previous self-test routine completed
> > > without error or no self-test has ever
> > > been run.
> > > Total time to complete Offline
> > > data collection: (38940) seconds.
> > > Offline data collection
> > > capabilities: (0x7b) SMART execute Offline immediate.
> > > Auto Offline data collection on/off support.
> > > Suspend Offline collection upon new
> > > command.
> > > Offline surface scan supported.
> > > Self-test supported.
> > > Conveyance Self-test supported.
> > > Selective Self-test supported.
> > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > power-saving mode.
> > > Supports SMART auto save timer.
> > > Error logging capability: (0x01) Error logging supported.
> > > General Purpose Logging supported.
> > > Short self-test routine
> > > recommended polling time: ( 2) minutes.
> > > Extended self-test routine
> > > recommended polling time: ( 391) minutes.
> > > Conveyance self-test routine
> > > recommended polling time: ( 5) minutes.
> > > SCT capabilities: (0x7035) SCT Status supported.
> > > SCT Feature Control supported.
> > > SCT Data Table supported.
> > >
> > > SMART Attributes Data Structure revision number: 16
> > > Vendor Specific SMART Attributes with Thresholds:
> > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > UPDATED WHEN_FAILED RAW_VALUE
> > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > Always - 0
> > > 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
> > > Always - 6075
> > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > Always - 81
> > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > Always - 0
> > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > Always - 0
> > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > Always - 18577
> > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > Always - 81
> > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > Always - 46
> > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > Always - 176661
> > > 194 Temperature_Celsius 0x0022 122 109 000 Old_age
> > > Always - 28
> > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > Offline - 0
> > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > Offline - 0
> > >
> > > SMART Error Log Version: 1
> > > No Errors Logged
> > >
> > > SMART Self-test log structure revision number 1
> > > Num Test_Description Status Remaining
> > > LifeTime(hours) LBA_of_first_error
> > > # 1 Extended offline Completed without error 00% 17479 -
> > > # 2 Short offline Completed without error 00% 15531 -
> > >
> > > SMART Selective self-test log data structure revision number 1
> > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > 1 0 0 Not_testing
> > > 2 0 0 Not_testing
> > > 3 0 0 Not_testing
> > > 4 0 0 Not_testing
> > > 5 0 0 Not_testing
> > > Selective self-test flags (0x0):
> > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > >
> > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
> > > smartctl 6.6 2017-11-05 r4594
> > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > >
> > > === START OF INFORMATION SECTION ===
> > > Model Family: Western Digital Green
> > > Device Model: WDC WD30EZRX-00D8PB0
> > > Serial Number: WD-WCC4NPRDD6D7
> > > LU WWN Device Id: 5 0014ee 25fca27b1
> > > Firmware Version: 80.00A80
> > > User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > Rotation Rate: 5400 rpm
> > > Device is: In smartctl database [for details use: -P show]
> > > ATA Version is: ACS-2 (minor revision not indicated)
> > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > Local Time is: Mon Oct 5 14:58:54 2020 BST
> > > SMART support is: Available - device has SMART capability.
> > > SMART support is: Enabled
> > >
> > > === START OF READ SMART DATA SECTION ===
> > > SMART overall-health self-assessment test result: PASSED
> > >
> > > General SMART Values:
> > > Offline data collection status: (0x82) Offline data collection activity
> > > was completed without error.
> > > Auto Offline Data Collection: Enabled.
> > > Self-test execution status: ( 0) The previous self-test routine completed
> > > without error or no self-test has ever
> > > been run.
> > > Total time to complete Offline
> > > data collection: (39060) seconds.
> > > Offline data collection
> > > capabilities: (0x7b) SMART execute Offline immediate.
> > > Auto Offline data collection on/off support.
> > > Suspend Offline collection upon new
> > > command.
> > > Offline surface scan supported.
> > > Self-test supported.
> > > Conveyance Self-test supported.
> > > Selective Self-test supported.
> > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > power-saving mode.
> > > Supports SMART auto save timer.
> > > Error logging capability: (0x01) Error logging supported.
> > > General Purpose Logging supported.
> > > Short self-test routine
> > > recommended polling time: ( 2) minutes.
> > > Extended self-test routine
> > > recommended polling time: ( 392) minutes.
> > > Conveyance self-test routine
> > > recommended polling time: ( 5) minutes.
> > > SCT capabilities: (0x7035) SCT Status supported.
> > > SCT Feature Control supported.
> > > SCT Data Table supported.
> > >
> > > SMART Attributes Data Structure revision number: 16
> > > Vendor Specific SMART Attributes with Thresholds:
> > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > UPDATED WHEN_FAILED RAW_VALUE
> > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > Always - 0
> > > 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
> > > Always - 6100
> > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > Always - 81
> > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > Always - 0
> > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > Always - 0
> > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > Always - 18580
> > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > Always - 81
> > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > Always - 53
> > > 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
> > > Always - 192427
> > > 194 Temperature_Celsius 0x0022 121 108 000 Old_age
> > > Always - 29
> > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > Offline - 0
> > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > Offline - 0
> > >
> > > SMART Error Log Version: 1
> > > No Errors Logged
> > >
> > > SMART Self-test log structure revision number 1
> > > Num Test_Description Status Remaining
> > > LifeTime(hours) LBA_of_first_error
> > > # 1 Extended offline Completed without error 00% 17481 -
> > > # 2 Short offline Completed without error 00% 15534 -
> > >
> > > SMART Selective self-test log data structure revision number 1
> > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > 1 0 0 Not_testing
> > > 2 0 0 Not_testing
> > > 3 0 0 Not_testing
> > > 4 0 0 Not_testing
> > > 5 0 0 Not_testing
> > > Selective self-test flags (0x0):
> > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > >
> > > [dan@lamachine ~]$ sudo smartctl -a /dev/sde
> > > smartctl 6.6 2017-11-05 r4594
> > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > >
> > > === START OF INFORMATION SECTION ===
> > > Model Family: Western Digital Green
> > > Device Model: WDC WD30EZRX-00D8PB0
> > > Serial Number: WD-WCC4N1294906
> > > LU WWN Device Id: 5 0014ee 25f968120
> > > Firmware Version: 80.00A80
> > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > Rotation Rate: 5400 rpm
> > > Device is: In smartctl database [for details use: -P show]
> > > ATA Version is: ACS-2 (minor revision not indicated)
> > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > Local Time is: Mon Oct 5 14:58:57 2020 BST
> > > SMART support is: Available - device has SMART capability.
> > > SMART support is: Enabled
> > >
> > > === START OF READ SMART DATA SECTION ===
> > > SMART overall-health self-assessment test result: PASSED
> > >
> > > General SMART Values:
> > > Offline data collection status: (0x82) Offline data collection activity
> > > was completed without error.
> > > Auto Offline Data Collection: Enabled.
> > > Self-test execution status: ( 0) The previous self-test routine completed
> > > without error or no self-test has ever
> > > been run.
> > > Total time to complete Offline
> > > data collection: (43200) seconds.
> > > Offline data collection
> > > capabilities: (0x7b) SMART execute Offline immediate.
> > > Auto Offline data collection on/off support.
> > > Suspend Offline collection upon new
> > > command.
> > > Offline surface scan supported.
> > > Self-test supported.
> > > Conveyance Self-test supported.
> > > Selective Self-test supported.
> > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > power-saving mode.
> > > Supports SMART auto save timer.
> > > Error logging capability: (0x01) Error logging supported.
> > > General Purpose Logging supported.
> > > Short self-test routine
> > > recommended polling time: ( 2) minutes.
> > > Extended self-test routine
> > > recommended polling time: ( 433) minutes.
> > > Conveyance self-test routine
> > > recommended polling time: ( 5) minutes.
> > > SCT capabilities: (0x7035) SCT Status supported.
> > > SCT Feature Control supported.
> > > SCT Data Table supported.
> > >
> > > SMART Attributes Data Structure revision number: 16
> > > Vendor Specific SMART Attributes with Thresholds:
> > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > UPDATED WHEN_FAILED RAW_VALUE
> > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > Always - 0
> > > 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
> > > Always - 6158
> > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > Always - 80
> > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > Always - 0
> > > 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
> > > Always - 0
> > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > Always - 18465
> > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > Always - 0
> > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > Always - 80
> > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > Always - 53
> > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > Always - 174015
> > > 194 Temperature_Celsius 0x0022 121 107 000 Old_age
> > > Always - 29
> > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > Offline - 0
> > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > Always - 0
> > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > Offline - 0
> > >
> > > SMART Error Log Version: 1
> > > No Errors Logged
> > >
> > > SMART Self-test log structure revision number 1
> > > Num Test_Description Status Remaining
> > > LifeTime(hours) LBA_of_first_error
> > > # 1 Extended offline Completed without error 00% 17347 -
> > > # 2 Short offline Completed without error 00% 15414 -
> > >
> > > SMART Selective self-test log data structure revision number 1
> > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > 1 0 0 Not_testing
> > > 2 0 0 Not_testing
> > > 3 0 0 Not_testing
> > > 4 0 0 Not_testing
> > > 5 0 0 Not_testing
> > > Selective self-test flags (0x0):
> > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > >
> > > [dan@lamachine ~]$
> > >
> > >
> > > On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
> > > >
> > > > On Mon, 5 Oct 2020 14:10:25 +0100
> > > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Scrubbing ( # echo check >
> > > > > /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
> > > > >
> > > > > I'm attaching details of the array and disks (bloody wd greens) as
> > > > > well as journalctl errors providing some details about the issue.
> > > > >
> > > > > If you have any pointers on what might be the cause of this as well as
> > > > > any recommendations on how to improve things please let me thank you
> > > > > in advance ...
> > > > >
> > > > > I have backups of the data so happy to move this to a different setup
> > > > > you might recommend (apps will be mostly reading from the array via
> > > > > NFS since most of the content will be media).
> > > > >
> > > > > My suspicion is that a timer service is kicking in and disrupting the
> > > > > scrubbing somehow but can't pinpoint what causes this.
> > > >
> > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > "smartctl -a" of all drives as well.
> > > >
> > > > --
> > > > With respect,
> > > > Roman
> >
> >
> > --
> > With respect,
> > Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-05 15:58 ` Roger Heflin
@ 2020-10-06 7:56 ` Daniel Sanabria
2020-10-06 8:24 ` Reindl Harald
2020-10-06 10:53 ` Roger Heflin
0 siblings, 2 replies; 16+ messages in thread
From: Daniel Sanabria @ 2020-10-06 7:56 UTC (permalink / raw)
To: Roger Heflin; +Cc: Roman Mamedov, Linux-RAID
Yeah it is quite possible that the hardware can't support the setup
because this array was an afterthought and considered an upgrade to
the system.
For the record here are some more details about the setup:
Motherboard: ASROCK EP2C602-4L/D16
PSU: 850W Corsair RM Series
I have 6 drives connected to the motherboard. The 3 drives forming the
array are Western Digital Green WDC WD30EZRX-00D8PB0 and are connected
to 3 ports of the Marvell 9230 SATA controller. The other 3 drives are
Western Digital Caviar Blue (SATA) WDC WD5000AAKS-00A7B2 are connected
to the spare Marvell port and the 2 SATA/SAS Motherboard ports
available.
On Mon, 5 Oct 2020 at 16:58, Roger Heflin <rogerheflin@gmail.com> wrote:
>
> what they said you have a hardware problem.
>
> it could be about anything previously mentioned and could also be the
> power supply being unable to provide a stable 12V for the disks.
>
> You should provide the list more specifics on your hw setup, of
> interest are what kind of SATA/SAS ports you are using and how the
> disk are cabled in.
>
> Note that there are a number of controllers that aren't the most
> reliable and some of those controllers when something happens will
> stop responding for all disks connected to it.
>
> I have also seen badly designed motherboards have
> build-in(non-AMD/non-Intel chips) sata ports that don't work under any
> load that uses more than a single disk at a time, and/or acts badly
> when given smart commands.
>
> On Mon, Oct 5, 2020 at 9:30 AM Daniel Sanabria <sanabria.d@gmail.com> wrote:
> >
> > > I meant not to me personally, but to the mailing list. The drives seem OK
> > > though, even sde.
> >
> > Sorry missed the reply-all button
> >
> > On Mon, 5 Oct 2020 at 15:04, Roman Mamedov <rm@romanrm.net> wrote:
> > >
> > > On Mon, 5 Oct 2020 14:59:35 +0100
> > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > >
> > > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > > "smartctl -a" of all drives as well.
> > >
> > > I meant not to me personally, but to the mailing list. The drives seem OK
> > > though, even sde.
> > >
> > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
> > > > [sudo] password for dan:
> > > > smartctl 6.6 2017-11-05 r4594
> > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > >
> > > > === START OF INFORMATION SECTION ===
> > > > Model Family: Western Digital Green
> > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > Serial Number: WD-WCC4NCWT13RF
> > > > LU WWN Device Id: 5 0014ee 25fc9e460
> > > > Firmware Version: 80.00A80
> > > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > Rotation Rate: 5400 rpm
> > > > Device is: In smartctl database [for details use: -P show]
> > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > Local Time is: Mon Oct 5 14:58:34 2020 BST
> > > > SMART support is: Available - device has SMART capability.
> > > > SMART support is: Enabled
> > > >
> > > > === START OF READ SMART DATA SECTION ===
> > > > SMART overall-health self-assessment test result: PASSED
> > > >
> > > > General SMART Values:
> > > > Offline data collection status: (0x82) Offline data collection activity
> > > > was completed without error.
> > > > Auto Offline Data Collection: Enabled.
> > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > without error or no self-test has ever
> > > > been run.
> > > > Total time to complete Offline
> > > > data collection: (38940) seconds.
> > > > Offline data collection
> > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > Auto Offline data collection on/off support.
> > > > Suspend Offline collection upon new
> > > > command.
> > > > Offline surface scan supported.
> > > > Self-test supported.
> > > > Conveyance Self-test supported.
> > > > Selective Self-test supported.
> > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > power-saving mode.
> > > > Supports SMART auto save timer.
> > > > Error logging capability: (0x01) Error logging supported.
> > > > General Purpose Logging supported.
> > > > Short self-test routine
> > > > recommended polling time: ( 2) minutes.
> > > > Extended self-test routine
> > > > recommended polling time: ( 391) minutes.
> > > > Conveyance self-test routine
> > > > recommended polling time: ( 5) minutes.
> > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > SCT Feature Control supported.
> > > > SCT Data Table supported.
> > > >
> > > > SMART Attributes Data Structure revision number: 16
> > > > Vendor Specific SMART Attributes with Thresholds:
> > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > Always - 0
> > > > 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
> > > > Always - 6075
> > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > Always - 81
> > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > Always - 0
> > > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > > Always - 0
> > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > Always - 18577
> > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > Always - 81
> > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > Always - 46
> > > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > > Always - 176661
> > > > 194 Temperature_Celsius 0x0022 122 109 000 Old_age
> > > > Always - 28
> > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > Offline - 0
> > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > Offline - 0
> > > >
> > > > SMART Error Log Version: 1
> > > > No Errors Logged
> > > >
> > > > SMART Self-test log structure revision number 1
> > > > Num Test_Description Status Remaining
> > > > LifeTime(hours) LBA_of_first_error
> > > > # 1 Extended offline Completed without error 00% 17479 -
> > > > # 2 Short offline Completed without error 00% 15531 -
> > > >
> > > > SMART Selective self-test log data structure revision number 1
> > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > 1 0 0 Not_testing
> > > > 2 0 0 Not_testing
> > > > 3 0 0 Not_testing
> > > > 4 0 0 Not_testing
> > > > 5 0 0 Not_testing
> > > > Selective self-test flags (0x0):
> > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > >
> > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
> > > > smartctl 6.6 2017-11-05 r4594
> > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > >
> > > > === START OF INFORMATION SECTION ===
> > > > Model Family: Western Digital Green
> > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > Serial Number: WD-WCC4NPRDD6D7
> > > > LU WWN Device Id: 5 0014ee 25fca27b1
> > > > Firmware Version: 80.00A80
> > > > User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > Rotation Rate: 5400 rpm
> > > > Device is: In smartctl database [for details use: -P show]
> > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > Local Time is: Mon Oct 5 14:58:54 2020 BST
> > > > SMART support is: Available - device has SMART capability.
> > > > SMART support is: Enabled
> > > >
> > > > === START OF READ SMART DATA SECTION ===
> > > > SMART overall-health self-assessment test result: PASSED
> > > >
> > > > General SMART Values:
> > > > Offline data collection status: (0x82) Offline data collection activity
> > > > was completed without error.
> > > > Auto Offline Data Collection: Enabled.
> > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > without error or no self-test has ever
> > > > been run.
> > > > Total time to complete Offline
> > > > data collection: (39060) seconds.
> > > > Offline data collection
> > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > Auto Offline data collection on/off support.
> > > > Suspend Offline collection upon new
> > > > command.
> > > > Offline surface scan supported.
> > > > Self-test supported.
> > > > Conveyance Self-test supported.
> > > > Selective Self-test supported.
> > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > power-saving mode.
> > > > Supports SMART auto save timer.
> > > > Error logging capability: (0x01) Error logging supported.
> > > > General Purpose Logging supported.
> > > > Short self-test routine
> > > > recommended polling time: ( 2) minutes.
> > > > Extended self-test routine
> > > > recommended polling time: ( 392) minutes.
> > > > Conveyance self-test routine
> > > > recommended polling time: ( 5) minutes.
> > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > SCT Feature Control supported.
> > > > SCT Data Table supported.
> > > >
> > > > SMART Attributes Data Structure revision number: 16
> > > > Vendor Specific SMART Attributes with Thresholds:
> > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > Always - 0
> > > > 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
> > > > Always - 6100
> > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > Always - 81
> > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > Always - 0
> > > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > > Always - 0
> > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > Always - 18580
> > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > Always - 81
> > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > Always - 53
> > > > 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
> > > > Always - 192427
> > > > 194 Temperature_Celsius 0x0022 121 108 000 Old_age
> > > > Always - 29
> > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > Offline - 0
> > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > Offline - 0
> > > >
> > > > SMART Error Log Version: 1
> > > > No Errors Logged
> > > >
> > > > SMART Self-test log structure revision number 1
> > > > Num Test_Description Status Remaining
> > > > LifeTime(hours) LBA_of_first_error
> > > > # 1 Extended offline Completed without error 00% 17481 -
> > > > # 2 Short offline Completed without error 00% 15534 -
> > > >
> > > > SMART Selective self-test log data structure revision number 1
> > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > 1 0 0 Not_testing
> > > > 2 0 0 Not_testing
> > > > 3 0 0 Not_testing
> > > > 4 0 0 Not_testing
> > > > 5 0 0 Not_testing
> > > > Selective self-test flags (0x0):
> > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > >
> > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sde
> > > > smartctl 6.6 2017-11-05 r4594
> > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > >
> > > > === START OF INFORMATION SECTION ===
> > > > Model Family: Western Digital Green
> > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > Serial Number: WD-WCC4N1294906
> > > > LU WWN Device Id: 5 0014ee 25f968120
> > > > Firmware Version: 80.00A80
> > > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > Rotation Rate: 5400 rpm
> > > > Device is: In smartctl database [for details use: -P show]
> > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > Local Time is: Mon Oct 5 14:58:57 2020 BST
> > > > SMART support is: Available - device has SMART capability.
> > > > SMART support is: Enabled
> > > >
> > > > === START OF READ SMART DATA SECTION ===
> > > > SMART overall-health self-assessment test result: PASSED
> > > >
> > > > General SMART Values:
> > > > Offline data collection status: (0x82) Offline data collection activity
> > > > was completed without error.
> > > > Auto Offline Data Collection: Enabled.
> > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > without error or no self-test has ever
> > > > been run.
> > > > Total time to complete Offline
> > > > data collection: (43200) seconds.
> > > > Offline data collection
> > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > Auto Offline data collection on/off support.
> > > > Suspend Offline collection upon new
> > > > command.
> > > > Offline surface scan supported.
> > > > Self-test supported.
> > > > Conveyance Self-test supported.
> > > > Selective Self-test supported.
> > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > power-saving mode.
> > > > Supports SMART auto save timer.
> > > > Error logging capability: (0x01) Error logging supported.
> > > > General Purpose Logging supported.
> > > > Short self-test routine
> > > > recommended polling time: ( 2) minutes.
> > > > Extended self-test routine
> > > > recommended polling time: ( 433) minutes.
> > > > Conveyance self-test routine
> > > > recommended polling time: ( 5) minutes.
> > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > SCT Feature Control supported.
> > > > SCT Data Table supported.
> > > >
> > > > SMART Attributes Data Structure revision number: 16
> > > > Vendor Specific SMART Attributes with Thresholds:
> > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > Always - 0
> > > > 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
> > > > Always - 6158
> > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > Always - 80
> > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > Always - 0
> > > > 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
> > > > Always - 0
> > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > Always - 18465
> > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > Always - 0
> > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > Always - 80
> > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > Always - 53
> > > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > > Always - 174015
> > > > 194 Temperature_Celsius 0x0022 121 107 000 Old_age
> > > > Always - 29
> > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > Offline - 0
> > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > Always - 0
> > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > Offline - 0
> > > >
> > > > SMART Error Log Version: 1
> > > > No Errors Logged
> > > >
> > > > SMART Self-test log structure revision number 1
> > > > Num Test_Description Status Remaining
> > > > LifeTime(hours) LBA_of_first_error
> > > > # 1 Extended offline Completed without error 00% 17347 -
> > > > # 2 Short offline Completed without error 00% 15414 -
> > > >
> > > > SMART Selective self-test log data structure revision number 1
> > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > 1 0 0 Not_testing
> > > > 2 0 0 Not_testing
> > > > 3 0 0 Not_testing
> > > > 4 0 0 Not_testing
> > > > 5 0 0 Not_testing
> > > > Selective self-test flags (0x0):
> > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > >
> > > > [dan@lamachine ~]$
> > > >
> > > >
> > > > On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
> > > > >
> > > > > On Mon, 5 Oct 2020 14:10:25 +0100
> > > > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Scrubbing ( # echo check >
> > > > > > /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
> > > > > >
> > > > > > I'm attaching details of the array and disks (bloody wd greens) as
> > > > > > well as journalctl errors providing some details about the issue.
> > > > > >
> > > > > > If you have any pointers on what might be the cause of this as well as
> > > > > > any recommendations on how to improve things please let me thank you
> > > > > > in advance ...
> > > > > >
> > > > > > I have backups of the data so happy to move this to a different setup
> > > > > > you might recommend (apps will be mostly reading from the array via
> > > > > > NFS since most of the content will be media).
> > > > > >
> > > > > > My suspicion is that a timer service is kicking in and disrupting the
> > > > > > scrubbing somehow but can't pinpoint what causes this.
> > > > >
> > > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > > "smartctl -a" of all drives as well.
> > > > >
> > > > > --
> > > > > With respect,
> > > > > Roman
> > >
> > >
> > > --
> > > With respect,
> > > Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 7:56 ` Daniel Sanabria
@ 2020-10-06 8:24 ` Reindl Harald
2020-10-06 10:53 ` Roger Heflin
1 sibling, 0 replies; 16+ messages in thread
From: Reindl Harald @ 2020-10-06 8:24 UTC (permalink / raw)
To: Daniel Sanabria, Roger Heflin; +Cc: Roman Mamedov, Linux-RAID
Am 06.10.20 um 09:56 schrieb Daniel Sanabria:
> Yeah it is quite possible that the hardware can't support the setup
> because this array was an afterthought and considered an upgrade to
> the system.
>
> For the record here are some more details about the setup:
>
> Motherboard: ASROCK EP2C602-4L/D16
> PSU: 850W Corsair RM Series
> I have 6 drives connected to the motherboard. The 3 drives forming the
> array are Western Digital Green WDC WD30EZRX-00D8PB0 and are connected
> to 3 ports of the Marvell 9230 SATA controller. The other 3 drives are
> Western Digital Caviar Blue (SATA) WDC WD5000AAKS-00A7B2 are connected
> to the spare Marvell port and the 2 SATA/SAS Motherboard ports
> available.
yeah, unreliable desktop disks without at least increase the timeouts -
currently the only WD disks for a RAID setip are the "WD Gold" after
they lost theri brain and starting SMR on "WD Red"
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
-------------------------
[root@srv-rhsoft:~]$ cat /etc/systemd/system/disk-timeout.service
[Unit]
Description=SCSI Timeouts
[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-timeout.sh
[Install]
WantedBy=multi-user.target
-------------------------
[root@srv-rhsoft:~]$ cat /usr/local/bin/disk-timeout.sh
#!/usr/bin/dash
echo 180 > "/sys/block/sda/device/timeout"
echo 180 > "/sys/block/sdb/device/timeout"
echo 180 > "/sys/block/sdc/device/timeout"
echo 180 > "/sys/block/sdd/device/timeout"
-------------------------
> On Mon, 5 Oct 2020 at 16:58, Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>> what they said you have a hardware problem.
>>
>> it could be about anything previously mentioned and could also be the
>> power supply being unable to provide a stable 12V for the disks.
>>
>> You should provide the list more specifics on your hw setup, of
>> interest are what kind of SATA/SAS ports you are using and how the
>> disk are cabled in.
>>
>> Note that there are a number of controllers that aren't the most
>> reliable and some of those controllers when something happens will
>> stop responding for all disks connected to it.
>>
>> I have also seen badly designed motherboards have
>> build-in(non-AMD/non-Intel chips) sata ports that don't work under any
>> load that uses more than a single disk at a time, and/or acts badly
>> when given smart commands.
>>
>> On Mon, Oct 5, 2020 at 9:30 AM Daniel Sanabria <sanabria.d@gmail.com> wrote:
>>>
>>>> I meant not to me personally, but to the mailing list. The drives seem OK
>>>> though, even sde.
>>>
>>> Sorry missed the reply-all button
>>>
>>> On Mon, 5 Oct 2020 at 15:04, Roman Mamedov <rm@romanrm.net> wrote:
>>>>
>>>> On Mon, 5 Oct 2020 14:59:35 +0100
>>>> Daniel Sanabria <sanabria.d@gmail.com> wrote:
>>>>
>>>>>> It looks like a drive is dropping off the bus and then failing to reidentify,
>>>>>> could be bad cabling/controller/PSU, or just a bad drive. You should post
>>>>>> "smartctl -a" of all drives as well.
>>>>
>>>> I meant not to me personally, but to the mailing list. The drives seem OK
>>>> though, even sde.
>>>>
>>>>> [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
>>>>> [sudo] password for dan:
>>>>> smartctl 6.6 2017-11-05 r4594
>>>>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>>>>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>>>>
>>>>> === START OF INFORMATION SECTION ===
>>>>> Model Family: Western Digital Green
>>>>> Device Model: WDC WD30EZRX-00D8PB0
>>>>> Serial Number: WD-WCC4NCWT13RF
>>>>> LU WWN Device Id: 5 0014ee 25fc9e460
>>>>> Firmware Version: 80.00A80
>>>>> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
>>>>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>>>>> Rotation Rate: 5400 rpm
>>>>> Device is: In smartctl database [for details use: -P show]
>>>>> ATA Version is: ACS-2 (minor revision not indicated)
>>>>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>>>>> Local Time is: Mon Oct 5 14:58:34 2020 BST
>>>>> SMART support is: Available - device has SMART capability.
>>>>> SMART support is: Enabled
>>>>>
>>>>> === START OF READ SMART DATA SECTION ===
>>>>> SMART overall-health self-assessment test result: PASSED
>>>>>
>>>>> General SMART Values:
>>>>> Offline data collection status: (0x82) Offline data collection activity
>>>>> was completed without error.
>>>>> Auto Offline Data Collection: Enabled.
>>>>> Self-test execution status: ( 0) The previous self-test routine completed
>>>>> without error or no self-test has ever
>>>>> been run.
>>>>> Total time to complete Offline
>>>>> data collection: (38940) seconds.
>>>>> Offline data collection
>>>>> capabilities: (0x7b) SMART execute Offline immediate.
>>>>> Auto Offline data collection on/off support.
>>>>> Suspend Offline collection upon new
>>>>> command.
>>>>> Offline surface scan supported.
>>>>> Self-test supported.
>>>>> Conveyance Self-test supported.
>>>>> Selective Self-test supported.
>>>>> SMART capabilities: (0x0003) Saves SMART data before entering
>>>>> power-saving mode.
>>>>> Supports SMART auto save timer.
>>>>> Error logging capability: (0x01) Error logging supported.
>>>>> General Purpose Logging supported.
>>>>> Short self-test routine
>>>>> recommended polling time: ( 2) minutes.
>>>>> Extended self-test routine
>>>>> recommended polling time: ( 391) minutes.
>>>>> Conveyance self-test routine
>>>>> recommended polling time: ( 5) minutes.
>>>>> SCT capabilities: (0x7035) SCT Status supported.
>>>>> SCT Feature Control supported.
>>>>> SCT Data Table supported.
>>>>>
>>>>> SMART Attributes Data Structure revision number: 16
>>>>> Vendor Specific SMART Attributes with Thresholds:
>>>>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>>>>> UPDATED WHEN_FAILED RAW_VALUE
>>>>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>>>>> Always - 0
>>>>> 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
>>>>> Always - 6075
>>>>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>>>>> Always - 81
>>>>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>>>>> Always - 0
>>>>> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
>>>>> Always - 0
>>>>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>>>>> Always - 18577
>>>>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>>>>> Always - 81
>>>>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>>>>> Always - 46
>>>>> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
>>>>> Always - 176661
>>>>> 194 Temperature_Celsius 0x0022 122 109 000 Old_age
>>>>> Always - 28
>>>>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>>>>> Offline - 0
>>>>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>>>>> Offline - 0
>>>>>
>>>>> SMART Error Log Version: 1
>>>>> No Errors Logged
>>>>>
>>>>> SMART Self-test log structure revision number 1
>>>>> Num Test_Description Status Remaining
>>>>> LifeTime(hours) LBA_of_first_error
>>>>> # 1 Extended offline Completed without error 00% 17479 -
>>>>> # 2 Short offline Completed without error 00% 15531 -
>>>>>
>>>>> SMART Selective self-test log data structure revision number 1
>>>>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>>>>> 1 0 0 Not_testing
>>>>> 2 0 0 Not_testing
>>>>> 3 0 0 Not_testing
>>>>> 4 0 0 Not_testing
>>>>> 5 0 0 Not_testing
>>>>> Selective self-test flags (0x0):
>>>>> After scanning selected spans, do NOT read-scan remainder of disk.
>>>>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>>>>
>>>>> [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
>>>>> smartctl 6.6 2017-11-05 r4594
>>>>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>>>>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>>>>
>>>>> === START OF INFORMATION SECTION ===
>>>>> Model Family: Western Digital Green
>>>>> Device Model: WDC WD30EZRX-00D8PB0
>>>>> Serial Number: WD-WCC4NPRDD6D7
>>>>> LU WWN Device Id: 5 0014ee 25fca27b1
>>>>> Firmware Version: 80.00A80
>>>>> User Capacity: 3,000,592,982,016 bytes [3.00 TB]
>>>>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>>>>> Rotation Rate: 5400 rpm
>>>>> Device is: In smartctl database [for details use: -P show]
>>>>> ATA Version is: ACS-2 (minor revision not indicated)
>>>>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>>>>> Local Time is: Mon Oct 5 14:58:54 2020 BST
>>>>> SMART support is: Available - device has SMART capability.
>>>>> SMART support is: Enabled
>>>>>
>>>>> === START OF READ SMART DATA SECTION ===
>>>>> SMART overall-health self-assessment test result: PASSED
>>>>>
>>>>> General SMART Values:
>>>>> Offline data collection status: (0x82) Offline data collection activity
>>>>> was completed without error.
>>>>> Auto Offline Data Collection: Enabled.
>>>>> Self-test execution status: ( 0) The previous self-test routine completed
>>>>> without error or no self-test has ever
>>>>> been run.
>>>>> Total time to complete Offline
>>>>> data collection: (39060) seconds.
>>>>> Offline data collection
>>>>> capabilities: (0x7b) SMART execute Offline immediate.
>>>>> Auto Offline data collection on/off support.
>>>>> Suspend Offline collection upon new
>>>>> command.
>>>>> Offline surface scan supported.
>>>>> Self-test supported.
>>>>> Conveyance Self-test supported.
>>>>> Selective Self-test supported.
>>>>> SMART capabilities: (0x0003) Saves SMART data before entering
>>>>> power-saving mode.
>>>>> Supports SMART auto save timer.
>>>>> Error logging capability: (0x01) Error logging supported.
>>>>> General Purpose Logging supported.
>>>>> Short self-test routine
>>>>> recommended polling time: ( 2) minutes.
>>>>> Extended self-test routine
>>>>> recommended polling time: ( 392) minutes.
>>>>> Conveyance self-test routine
>>>>> recommended polling time: ( 5) minutes.
>>>>> SCT capabilities: (0x7035) SCT Status supported.
>>>>> SCT Feature Control supported.
>>>>> SCT Data Table supported.
>>>>>
>>>>> SMART Attributes Data Structure revision number: 16
>>>>> Vendor Specific SMART Attributes with Thresholds:
>>>>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>>>>> UPDATED WHEN_FAILED RAW_VALUE
>>>>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>>>>> Always - 0
>>>>> 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
>>>>> Always - 6100
>>>>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>>>>> Always - 81
>>>>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>>>>> Always - 0
>>>>> 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
>>>>> Always - 0
>>>>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>>>>> Always - 18580
>>>>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>>>>> Always - 81
>>>>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>>>>> Always - 53
>>>>> 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
>>>>> Always - 192427
>>>>> 194 Temperature_Celsius 0x0022 121 108 000 Old_age
>>>>> Always - 29
>>>>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>>>>> Offline - 0
>>>>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>>>>> Offline - 0
>>>>>
>>>>> SMART Error Log Version: 1
>>>>> No Errors Logged
>>>>>
>>>>> SMART Self-test log structure revision number 1
>>>>> Num Test_Description Status Remaining
>>>>> LifeTime(hours) LBA_of_first_error
>>>>> # 1 Extended offline Completed without error 00% 17481 -
>>>>> # 2 Short offline Completed without error 00% 15534 -
>>>>>
>>>>> SMART Selective self-test log data structure revision number 1
>>>>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>>>>> 1 0 0 Not_testing
>>>>> 2 0 0 Not_testing
>>>>> 3 0 0 Not_testing
>>>>> 4 0 0 Not_testing
>>>>> 5 0 0 Not_testing
>>>>> Selective self-test flags (0x0):
>>>>> After scanning selected spans, do NOT read-scan remainder of disk.
>>>>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>>>>
>>>>> [dan@lamachine ~]$ sudo smartctl -a /dev/sde
>>>>> smartctl 6.6 2017-11-05 r4594
>>>>> [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
>>>>> Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
>>>>>
>>>>> === START OF INFORMATION SECTION ===
>>>>> Model Family: Western Digital Green
>>>>> Device Model: WDC WD30EZRX-00D8PB0
>>>>> Serial Number: WD-WCC4N1294906
>>>>> LU WWN Device Id: 5 0014ee 25f968120
>>>>> Firmware Version: 80.00A80
>>>>> User Capacity: 3,000,591,900,160 bytes [3.00 TB]
>>>>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>>>>> Rotation Rate: 5400 rpm
>>>>> Device is: In smartctl database [for details use: -P show]
>>>>> ATA Version is: ACS-2 (minor revision not indicated)
>>>>> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>>>>> Local Time is: Mon Oct 5 14:58:57 2020 BST
>>>>> SMART support is: Available - device has SMART capability.
>>>>> SMART support is: Enabled
>>>>>
>>>>> === START OF READ SMART DATA SECTION ===
>>>>> SMART overall-health self-assessment test result: PASSED
>>>>>
>>>>> General SMART Values:
>>>>> Offline data collection status: (0x82) Offline data collection activity
>>>>> was completed without error.
>>>>> Auto Offline Data Collection: Enabled.
>>>>> Self-test execution status: ( 0) The previous self-test routine completed
>>>>> without error or no self-test has ever
>>>>> been run.
>>>>> Total time to complete Offline
>>>>> data collection: (43200) seconds.
>>>>> Offline data collection
>>>>> capabilities: (0x7b) SMART execute Offline immediate.
>>>>> Auto Offline data collection on/off support.
>>>>> Suspend Offline collection upon new
>>>>> command.
>>>>> Offline surface scan supported.
>>>>> Self-test supported.
>>>>> Conveyance Self-test supported.
>>>>> Selective Self-test supported.
>>>>> SMART capabilities: (0x0003) Saves SMART data before entering
>>>>> power-saving mode.
>>>>> Supports SMART auto save timer.
>>>>> Error logging capability: (0x01) Error logging supported.
>>>>> General Purpose Logging supported.
>>>>> Short self-test routine
>>>>> recommended polling time: ( 2) minutes.
>>>>> Extended self-test routine
>>>>> recommended polling time: ( 433) minutes.
>>>>> Conveyance self-test routine
>>>>> recommended polling time: ( 5) minutes.
>>>>> SCT capabilities: (0x7035) SCT Status supported.
>>>>> SCT Feature Control supported.
>>>>> SCT Data Table supported.
>>>>>
>>>>> SMART Attributes Data Structure revision number: 16
>>>>> Vendor Specific SMART Attributes with Thresholds:
>>>>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>>>>> UPDATED WHEN_FAILED RAW_VALUE
>>>>> 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
>>>>> Always - 0
>>>>> 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
>>>>> Always - 6158
>>>>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>>>>> Always - 80
>>>>> 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
>>>>> Always - 0
>>>>> 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
>>>>> Always - 0
>>>>> 9 Power_On_Hours 0x0032 075 075 000 Old_age
>>>>> Always - 18465
>>>>> 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
>>>>> Always - 0
>>>>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>>>>> Always - 80
>>>>> 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
>>>>> Always - 53
>>>>> 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
>>>>> Always - 174015
>>>>> 194 Temperature_Celsius 0x0022 121 107 000 Old_age
>>>>> Always - 29
>>>>> 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
>>>>> Offline - 0
>>>>> 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
>>>>> Always - 0
>>>>> 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
>>>>> Offline - 0
>>>>>
>>>>> SMART Error Log Version: 1
>>>>> No Errors Logged
>>>>>
>>>>> SMART Self-test log structure revision number 1
>>>>> Num Test_Description Status Remaining
>>>>> LifeTime(hours) LBA_of_first_error
>>>>> # 1 Extended offline Completed without error 00% 17347 -
>>>>> # 2 Short offline Completed without error 00% 15414 -
>>>>>
>>>>> SMART Selective self-test log data structure revision number 1
>>>>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>>>>> 1 0 0 Not_testing
>>>>> 2 0 0 Not_testing
>>>>> 3 0 0 Not_testing
>>>>> 4 0 0 Not_testing
>>>>> 5 0 0 Not_testing
>>>>> Selective self-test flags (0x0):
>>>>> After scanning selected spans, do NOT read-scan remainder of disk.
>>>>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>>>>
>>>>> [dan@lamachine ~]$
>>>>>
>>>>>
>>>>> On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
>>>>>>
>>>>>> On Mon, 5 Oct 2020 14:10:25 +0100
>>>>>> Daniel Sanabria <sanabria.d@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Scrubbing ( # echo check >
>>>>>>> /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
>>>>>>>
>>>>>>> I'm attaching details of the array and disks (bloody wd greens) as
>>>>>>> well as journalctl errors providing some details about the issue.
>>>>>>>
>>>>>>> If you have any pointers on what might be the cause of this as well as
>>>>>>> any recommendations on how to improve things please let me thank you
>>>>>>> in advance ...
>>>>>>>
>>>>>>> I have backups of the data so happy to move this to a different setup
>>>>>>> you might recommend (apps will be mostly reading from the array via
>>>>>>> NFS since most of the content will be media).
>>>>>>>
>>>>>>> My suspicion is that a timer service is kicking in and disrupting the
>>>>>>> scrubbing somehow but can't pinpoint what causes this.
>>>>>>
>>>>>> It looks like a drive is dropping off the bus and then failing to reidentify,
>>>>>> could be bad cabling/controller/PSU, or just a bad drive. You should post
>>>>>> "smartctl -a" of all drives as well.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 7:56 ` Daniel Sanabria
2020-10-06 8:24 ` Reindl Harald
@ 2020-10-06 10:53 ` Roger Heflin
2020-10-06 11:29 ` antlists
2020-10-06 15:03 ` Tim Small
1 sibling, 2 replies; 16+ messages in thread
From: Roger Heflin @ 2020-10-06 10:53 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: Roman Mamedov, Linux-RAID
Ok. That is kind of what I was fearing based on the behavior. I know
first hand that the marvel 9230 sata card is a POS.
When I was running it I noted that if you ran smart commands against
it it would go offline quicker. If you don't run smart commands at
all then it is more stable, but still will have serious issues
sometimes during raid syncs and/or rebuilds. When the card has an
issue all of the ports seem to stop responding to commands. I am
guessing the firmware on the card somehow crashes or gets into some
sort of endless loop. I reported it to marvel, they blamed the OS'es
ACHI drivers,even though the AHCI drivers worked perfectly fine with
the built in AMD ports. Gotta love engineers and support people that
have absolutely no idea what they are doing, nor how to validate a
design works. I have been burned by marvell cards enough I will not
buy any marvell product as I know they have zero idea how to validate
designs.
I moved to a used LSI SAS card (4 internal ports, 4 external ports
also, needs non-raid bios installed to be a dumb card). Outside of
the enterprise type cards, I have yet found a stable PCIE card, and in
one of my backup machines still use a Sata Sil (old pci) card as while
it is slow (all 4 ports limited to a total of about 120MB-real is
90MB), it does consistently work right.
On Tue, Oct 6, 2020 at 2:56 AM Daniel Sanabria <sanabria.d@gmail.com> wrote:
>
> Yeah it is quite possible that the hardware can't support the setup
> because this array was an afterthought and considered an upgrade to
> the system.
>
> For the record here are some more details about the setup:
>
> Motherboard: ASROCK EP2C602-4L/D16
> PSU: 850W Corsair RM Series
> I have 6 drives connected to the motherboard. The 3 drives forming the
> array are Western Digital Green WDC WD30EZRX-00D8PB0 and are connected
> to 3 ports of the Marvell 9230 SATA controller. The other 3 drives are
> Western Digital Caviar Blue (SATA) WDC WD5000AAKS-00A7B2 are connected
> to the spare Marvell port and the 2 SATA/SAS Motherboard ports
> available.
>
> On Mon, 5 Oct 2020 at 16:58, Roger Heflin <rogerheflin@gmail.com> wrote:
> >
> > what they said you have a hardware problem.
> >
> > it could be about anything previously mentioned and could also be the
> > power supply being unable to provide a stable 12V for the disks.
> >
> > You should provide the list more specifics on your hw setup, of
> > interest are what kind of SATA/SAS ports you are using and how the
> > disk are cabled in.
> >
> > Note that there are a number of controllers that aren't the most
> > reliable and some of those controllers when something happens will
> > stop responding for all disks connected to it.
> >
> > I have also seen badly designed motherboards have
> > build-in(non-AMD/non-Intel chips) sata ports that don't work under any
> > load that uses more than a single disk at a time, and/or acts badly
> > when given smart commands.
> >
> > On Mon, Oct 5, 2020 at 9:30 AM Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > >
> > > > I meant not to me personally, but to the mailing list. The drives seem OK
> > > > though, even sde.
> > >
> > > Sorry missed the reply-all button
> > >
> > > On Mon, 5 Oct 2020 at 15:04, Roman Mamedov <rm@romanrm.net> wrote:
> > > >
> > > > On Mon, 5 Oct 2020 14:59:35 +0100
> > > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > > >
> > > > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > > > "smartctl -a" of all drives as well.
> > > >
> > > > I meant not to me personally, but to the mailing list. The drives seem OK
> > > > though, even sde.
> > > >
> > > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdc
> > > > > [sudo] password for dan:
> > > > > smartctl 6.6 2017-11-05 r4594
> > > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > > >
> > > > > === START OF INFORMATION SECTION ===
> > > > > Model Family: Western Digital Green
> > > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > > Serial Number: WD-WCC4NCWT13RF
> > > > > LU WWN Device Id: 5 0014ee 25fc9e460
> > > > > Firmware Version: 80.00A80
> > > > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > > Rotation Rate: 5400 rpm
> > > > > Device is: In smartctl database [for details use: -P show]
> > > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > > Local Time is: Mon Oct 5 14:58:34 2020 BST
> > > > > SMART support is: Available - device has SMART capability.
> > > > > SMART support is: Enabled
> > > > >
> > > > > === START OF READ SMART DATA SECTION ===
> > > > > SMART overall-health self-assessment test result: PASSED
> > > > >
> > > > > General SMART Values:
> > > > > Offline data collection status: (0x82) Offline data collection activity
> > > > > was completed without error.
> > > > > Auto Offline Data Collection: Enabled.
> > > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > > without error or no self-test has ever
> > > > > been run.
> > > > > Total time to complete Offline
> > > > > data collection: (38940) seconds.
> > > > > Offline data collection
> > > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > > Auto Offline data collection on/off support.
> > > > > Suspend Offline collection upon new
> > > > > command.
> > > > > Offline surface scan supported.
> > > > > Self-test supported.
> > > > > Conveyance Self-test supported.
> > > > > Selective Self-test supported.
> > > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > > power-saving mode.
> > > > > Supports SMART auto save timer.
> > > > > Error logging capability: (0x01) Error logging supported.
> > > > > General Purpose Logging supported.
> > > > > Short self-test routine
> > > > > recommended polling time: ( 2) minutes.
> > > > > Extended self-test routine
> > > > > recommended polling time: ( 391) minutes.
> > > > > Conveyance self-test routine
> > > > > recommended polling time: ( 5) minutes.
> > > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > > SCT Feature Control supported.
> > > > > SCT Data Table supported.
> > > > >
> > > > > SMART Attributes Data Structure revision number: 16
> > > > > Vendor Specific SMART Attributes with Thresholds:
> > > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > > Always - 0
> > > > > 3 Spin_Up_Time 0x0027 178 165 021 Pre-fail
> > > > > Always - 6075
> > > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > > Always - 81
> > > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > > Always - 0
> > > > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > > > Always - 0
> > > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > > Always - 18577
> > > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > > Always - 81
> > > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > > Always - 46
> > > > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > > > Always - 176661
> > > > > 194 Temperature_Celsius 0x0022 122 109 000 Old_age
> > > > > Always - 28
> > > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > > Offline - 0
> > > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > > Offline - 0
> > > > >
> > > > > SMART Error Log Version: 1
> > > > > No Errors Logged
> > > > >
> > > > > SMART Self-test log structure revision number 1
> > > > > Num Test_Description Status Remaining
> > > > > LifeTime(hours) LBA_of_first_error
> > > > > # 1 Extended offline Completed without error 00% 17479 -
> > > > > # 2 Short offline Completed without error 00% 15531 -
> > > > >
> > > > > SMART Selective self-test log data structure revision number 1
> > > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > > 1 0 0 Not_testing
> > > > > 2 0 0 Not_testing
> > > > > 3 0 0 Not_testing
> > > > > 4 0 0 Not_testing
> > > > > 5 0 0 Not_testing
> > > > > Selective self-test flags (0x0):
> > > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > > >
> > > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sdd
> > > > > smartctl 6.6 2017-11-05 r4594
> > > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > > >
> > > > > === START OF INFORMATION SECTION ===
> > > > > Model Family: Western Digital Green
> > > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > > Serial Number: WD-WCC4NPRDD6D7
> > > > > LU WWN Device Id: 5 0014ee 25fca27b1
> > > > > Firmware Version: 80.00A80
> > > > > User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> > > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > > Rotation Rate: 5400 rpm
> > > > > Device is: In smartctl database [for details use: -P show]
> > > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > > Local Time is: Mon Oct 5 14:58:54 2020 BST
> > > > > SMART support is: Available - device has SMART capability.
> > > > > SMART support is: Enabled
> > > > >
> > > > > === START OF READ SMART DATA SECTION ===
> > > > > SMART overall-health self-assessment test result: PASSED
> > > > >
> > > > > General SMART Values:
> > > > > Offline data collection status: (0x82) Offline data collection activity
> > > > > was completed without error.
> > > > > Auto Offline Data Collection: Enabled.
> > > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > > without error or no self-test has ever
> > > > > been run.
> > > > > Total time to complete Offline
> > > > > data collection: (39060) seconds.
> > > > > Offline data collection
> > > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > > Auto Offline data collection on/off support.
> > > > > Suspend Offline collection upon new
> > > > > command.
> > > > > Offline surface scan supported.
> > > > > Self-test supported.
> > > > > Conveyance Self-test supported.
> > > > > Selective Self-test supported.
> > > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > > power-saving mode.
> > > > > Supports SMART auto save timer.
> > > > > Error logging capability: (0x01) Error logging supported.
> > > > > General Purpose Logging supported.
> > > > > Short self-test routine
> > > > > recommended polling time: ( 2) minutes.
> > > > > Extended self-test routine
> > > > > recommended polling time: ( 392) minutes.
> > > > > Conveyance self-test routine
> > > > > recommended polling time: ( 5) minutes.
> > > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > > SCT Feature Control supported.
> > > > > SCT Data Table supported.
> > > > >
> > > > > SMART Attributes Data Structure revision number: 16
> > > > > Vendor Specific SMART Attributes with Thresholds:
> > > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > > Always - 0
> > > > > 3 Spin_Up_Time 0x0027 178 164 021 Pre-fail
> > > > > Always - 6100
> > > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > > Always - 81
> > > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > > Always - 0
> > > > > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age
> > > > > Always - 0
> > > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > > Always - 18580
> > > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > > Always - 81
> > > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > > Always - 53
> > > > > 193 Load_Cycle_Count 0x0032 136 136 000 Old_age
> > > > > Always - 192427
> > > > > 194 Temperature_Celsius 0x0022 121 108 000 Old_age
> > > > > Always - 29
> > > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > > Offline - 0
> > > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > > Offline - 0
> > > > >
> > > > > SMART Error Log Version: 1
> > > > > No Errors Logged
> > > > >
> > > > > SMART Self-test log structure revision number 1
> > > > > Num Test_Description Status Remaining
> > > > > LifeTime(hours) LBA_of_first_error
> > > > > # 1 Extended offline Completed without error 00% 17481 -
> > > > > # 2 Short offline Completed without error 00% 15534 -
> > > > >
> > > > > SMART Selective self-test log data structure revision number 1
> > > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > > 1 0 0 Not_testing
> > > > > 2 0 0 Not_testing
> > > > > 3 0 0 Not_testing
> > > > > 4 0 0 Not_testing
> > > > > 5 0 0 Not_testing
> > > > > Selective self-test flags (0x0):
> > > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > > >
> > > > > [dan@lamachine ~]$ sudo smartctl -a /dev/sde
> > > > > smartctl 6.6 2017-11-05 r4594
> > > > > [x86_64-linux-4.18.0-193.14.2.el8_2.x86_64] (local build)
> > > > > Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
> > > > >
> > > > > === START OF INFORMATION SECTION ===
> > > > > Model Family: Western Digital Green
> > > > > Device Model: WDC WD30EZRX-00D8PB0
> > > > > Serial Number: WD-WCC4N1294906
> > > > > LU WWN Device Id: 5 0014ee 25f968120
> > > > > Firmware Version: 80.00A80
> > > > > User Capacity: 3,000,591,900,160 bytes [3.00 TB]
> > > > > Sector Sizes: 512 bytes logical, 4096 bytes physical
> > > > > Rotation Rate: 5400 rpm
> > > > > Device is: In smartctl database [for details use: -P show]
> > > > > ATA Version is: ACS-2 (minor revision not indicated)
> > > > > SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> > > > > Local Time is: Mon Oct 5 14:58:57 2020 BST
> > > > > SMART support is: Available - device has SMART capability.
> > > > > SMART support is: Enabled
> > > > >
> > > > > === START OF READ SMART DATA SECTION ===
> > > > > SMART overall-health self-assessment test result: PASSED
> > > > >
> > > > > General SMART Values:
> > > > > Offline data collection status: (0x82) Offline data collection activity
> > > > > was completed without error.
> > > > > Auto Offline Data Collection: Enabled.
> > > > > Self-test execution status: ( 0) The previous self-test routine completed
> > > > > without error or no self-test has ever
> > > > > been run.
> > > > > Total time to complete Offline
> > > > > data collection: (43200) seconds.
> > > > > Offline data collection
> > > > > capabilities: (0x7b) SMART execute Offline immediate.
> > > > > Auto Offline data collection on/off support.
> > > > > Suspend Offline collection upon new
> > > > > command.
> > > > > Offline surface scan supported.
> > > > > Self-test supported.
> > > > > Conveyance Self-test supported.
> > > > > Selective Self-test supported.
> > > > > SMART capabilities: (0x0003) Saves SMART data before entering
> > > > > power-saving mode.
> > > > > Supports SMART auto save timer.
> > > > > Error logging capability: (0x01) Error logging supported.
> > > > > General Purpose Logging supported.
> > > > > Short self-test routine
> > > > > recommended polling time: ( 2) minutes.
> > > > > Extended self-test routine
> > > > > recommended polling time: ( 433) minutes.
> > > > > Conveyance self-test routine
> > > > > recommended polling time: ( 5) minutes.
> > > > > SCT capabilities: (0x7035) SCT Status supported.
> > > > > SCT Feature Control supported.
> > > > > SCT Data Table supported.
> > > > >
> > > > > SMART Attributes Data Structure revision number: 16
> > > > > Vendor Specific SMART Attributes with Thresholds:
> > > > > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> > > > > UPDATED WHEN_FAILED RAW_VALUE
> > > > > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
> > > > > Always - 0
> > > > > 3 Spin_Up_Time 0x0027 176 166 021 Pre-fail
> > > > > Always - 6158
> > > > > 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> > > > > Always - 80
> > > > > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
> > > > > Always - 0
> > > > > 7 Seek_Error_Rate 0x002e 200 200 000 Old_age
> > > > > Always - 0
> > > > > 9 Power_On_Hours 0x0032 075 075 000 Old_age
> > > > > Always - 18465
> > > > > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
> > > > > Always - 0
> > > > > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> > > > > Always - 80
> > > > > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
> > > > > Always - 53
> > > > > 193 Load_Cycle_Count 0x0032 142 142 000 Old_age
> > > > > Always - 174015
> > > > > 194 Temperature_Celsius 0x0022 121 107 000 Old_age
> > > > > Always - 29
> > > > > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
> > > > > Offline - 0
> > > > > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
> > > > > Always - 0
> > > > > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
> > > > > Offline - 0
> > > > >
> > > > > SMART Error Log Version: 1
> > > > > No Errors Logged
> > > > >
> > > > > SMART Self-test log structure revision number 1
> > > > > Num Test_Description Status Remaining
> > > > > LifeTime(hours) LBA_of_first_error
> > > > > # 1 Extended offline Completed without error 00% 17347 -
> > > > > # 2 Short offline Completed without error 00% 15414 -
> > > > >
> > > > > SMART Selective self-test log data structure revision number 1
> > > > > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> > > > > 1 0 0 Not_testing
> > > > > 2 0 0 Not_testing
> > > > > 3 0 0 Not_testing
> > > > > 4 0 0 Not_testing
> > > > > 5 0 0 Not_testing
> > > > > Selective self-test flags (0x0):
> > > > > After scanning selected spans, do NOT read-scan remainder of disk.
> > > > > If Selective self-test is pending on power-up, resume after 0 minute delay.
> > > > >
> > > > > [dan@lamachine ~]$
> > > > >
> > > > >
> > > > > On Mon, 5 Oct 2020 at 14:44, Roman Mamedov <rm@romanrm.net> wrote:
> > > > > >
> > > > > > On Mon, 5 Oct 2020 14:10:25 +0100
> > > > > > Daniel Sanabria <sanabria.d@gmail.com> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Scrubbing ( # echo check >
> > > > > > > /sys/devices/virtual/block/md1/md/sync_action) is killing my array :(
> > > > > > >
> > > > > > > I'm attaching details of the array and disks (bloody wd greens) as
> > > > > > > well as journalctl errors providing some details about the issue.
> > > > > > >
> > > > > > > If you have any pointers on what might be the cause of this as well as
> > > > > > > any recommendations on how to improve things please let me thank you
> > > > > > > in advance ...
> > > > > > >
> > > > > > > I have backups of the data so happy to move this to a different setup
> > > > > > > you might recommend (apps will be mostly reading from the array via
> > > > > > > NFS since most of the content will be media).
> > > > > > >
> > > > > > > My suspicion is that a timer service is kicking in and disrupting the
> > > > > > > scrubbing somehow but can't pinpoint what causes this.
> > > > > >
> > > > > > It looks like a drive is dropping off the bus and then failing to reidentify,
> > > > > > could be bad cabling/controller/PSU, or just a bad drive. You should post
> > > > > > "smartctl -a" of all drives as well.
> > > > > >
> > > > > > --
> > > > > > With respect,
> > > > > > Roman
> > > >
> > > >
> > > > --
> > > > With respect,
> > > > Roman
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 10:53 ` Roger Heflin
@ 2020-10-06 11:29 ` antlists
2020-10-06 14:59 ` Roger Heflin
2020-10-06 15:03 ` Tim Small
1 sibling, 1 reply; 16+ messages in thread
From: antlists @ 2020-10-06 11:29 UTC (permalink / raw)
To: Roger Heflin, Daniel Sanabria; +Cc: Roman Mamedov, Linux-RAID
On 06/10/2020 11:53, Roger Heflin wrote:
> When the card has an
> issue all of the ports seem to stop responding to commands. I am
> guessing the firmware on the card somehow crashes or gets into some
> sort of endless loop. I reported it to marvel, they blamed the OS'es
> ACHI drivers,even though the AHCI drivers worked perfectly fine with
> the built in AMD ports.
So we've got the crap drives on the crap controllers ... would it make
any difference if you put the Greens on the motherboard, and the Caviars
on the Marvell? Caviars I believe are good quality drives that might
take enough load off the Marvell to enable it to work sort-of okay ...
Oh - and replace the Greens pretty soon - I don't know how they compare
against other drives quality-wise, but they are optimised in a raid
anti-pattern.
Cheers,
Wol
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 11:29 ` antlists
@ 2020-10-06 14:59 ` Roger Heflin
2020-10-09 1:00 ` John Stoffel
0 siblings, 1 reply; 16+ messages in thread
From: Roger Heflin @ 2020-10-06 14:59 UTC (permalink / raw)
To: antlists; +Cc: Daniel Sanabria, Roman Mamedov, Linux-RAID
The controller is crap, and is expected to have serious issues no
matter what drives are on the controller.
Given the green's don't seem to have any reallocated blocks, I am
guessing the controller is 90%+ of the problem, and right now may be
all of the problem. If you lose all of the disks on the marvell
controller at roughly the same time, that is the controller bug and
not a disk issue. It also does not seem to be caused by a disk issue,
the controller just seems to have a race condition when multiple
operations are being done at the same time the controller just
"crashes" and stops responding to all drives on that controller.
On Tue, Oct 6, 2020 at 6:29 AM antlists <antlists@youngman.org.uk> wrote:
>
> On 06/10/2020 11:53, Roger Heflin wrote:
> > When the card has an
> > issue all of the ports seem to stop responding to commands. I am
> > guessing the firmware on the card somehow crashes or gets into some
> > sort of endless loop. I reported it to marvel, they blamed the OS'es
> > ACHI drivers,even though the AHCI drivers worked perfectly fine with
> > the built in AMD ports.
>
> So we've got the crap drives on the crap controllers ... would it make
> any difference if you put the Greens on the motherboard, and the Caviars
> on the Marvell? Caviars I believe are good quality drives that might
> take enough load off the Marvell to enable it to work sort-of okay ...
>
> Oh - and replace the Greens pretty soon - I don't know how they compare
> against other drives quality-wise, but they are optimised in a raid
> anti-pattern.
>
> Cheers,
> Wol
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 10:53 ` Roger Heflin
2020-10-06 11:29 ` antlists
@ 2020-10-06 15:03 ` Tim Small
2020-10-06 16:01 ` Daniel Sanabria
1 sibling, 1 reply; 16+ messages in thread
From: Tim Small @ 2020-10-06 15:03 UTC (permalink / raw)
To: Roger Heflin, Daniel Sanabria; +Cc: Roman Mamedov, Linux-RAID
On 06/10/2020 11:53, Roger Heflin wrote:
> Outside of the enterprise type cards, I have yet found a stable PCIE card
>
I've also had numerous problems with Marvell SATA controllers.
I've generally found the ASMedia ASM1083 / ASM1085 AHCI controllers
stable and reliable.
ASMedia is part of Asus, and from Wikipedia:
"[ASMedia] produces designs for USB, PCI Express and SATA controllers.
Excluding the X570 chipset, all of the AM4 chipsets for AMD's Zen
micro-architecture were designed by ASMedia"
The ASM108x are only PCIe 2.0 x1 <-> 2 SATA port cards, however there
are designs (e.g. "SA3008" - around $45 online) which incorporate
multiple ASM108x behind an ASMedia PCIe 2.0 bridge if number of
available PCIe slots are an issue.
Tim.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 15:03 ` Tim Small
@ 2020-10-06 16:01 ` Daniel Sanabria
2020-10-07 7:26 ` Tim Small
0 siblings, 1 reply; 16+ messages in thread
From: Daniel Sanabria @ 2020-10-06 16:01 UTC (permalink / raw)
To: Tim Small; +Cc: Roger Heflin, Roman Mamedov, Linux-RAID
Thank you very much Guys. This is one of the best email lists I'm
subscribed to so thanks to you all !
I've decided to dissolve this array and will use the disks as stand
alone drives. I have another array (raid0) using a pair of the WD
caviar blues and the pair of non-marvell ports and haven't had any
issues in years so will keep that one.
Thanks again,
Dan
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 16:01 ` Daniel Sanabria
@ 2020-10-07 7:26 ` Tim Small
0 siblings, 0 replies; 16+ messages in thread
From: Tim Small @ 2020-10-07 7:26 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: Roger Heflin, Roman Mamedov, Linux-RAID
If you want to keep the Marvell controller in use, then I found them a
lot more stable with Tagged Command Queueing disabled:
echo 1 > /sys/block/sdX/device/queue_depth
Otherwise you might also look for a firmware update for the WD Green
drives which you are having problems with (they might be hidden on
vendor support sites like those of Dell and Lenovo, if you can find
particular PC models which shipped with the same models of WD drives
that you have).
Also if possible, consider switching for the ASMedia controllers if
possible - the two port versions are available for under €10.
Tim.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: do i need to give up on this setup
2020-10-06 14:59 ` Roger Heflin
@ 2020-10-09 1:00 ` John Stoffel
0 siblings, 0 replies; 16+ messages in thread
From: John Stoffel @ 2020-10-09 1:00 UTC (permalink / raw)
To: Roger Heflin; +Cc: antlists, Daniel Sanabria, Roman Mamedov, Linux-RAID
>>>>> "Roger" == Roger Heflin <rogerheflin@gmail.com> writes:
Roger> The controller is crap, and is expected to have serious issues no
Roger> matter what drives are on the controller.
I can't say enough good things about the LSI SATA RAID controllers.
You can usually get them pretty cheap on eBay, and just flash them
with the JBOD firmware and they do great.
LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT S)
It's been a while, but I think it was an IBM branded card at the
time. 8 ports, easy setup, works well. And looking on ebay, they're
cheap now, around $50 though you might have to pay for the 1-to-4
cables to goto SATA drives. 9211, 9341, stuff like that. Here's an
eBay listing with cables:
https://www.ebay.com/p/1404809612?iid=133501363959&_trkparms=aid%3D555018%26algo%3DPL.SIM%26ao%3D1%26asc%3D20170810093926%26meid%3De20ed20fb9634e328a31bca5c9e2063c%26pid%3D100854%26rk%3D1%26rkt%3D1%26itm%3D133501363959%26pmt%3D1%26noa%3D0%26pg%3D2322090%26algv%3DSimplAMLSeedlessV2&_trksid=p2322090.c100854.m4779
In my mind, the other advantage of these cards is that you can get two
of them, and split your data across two controllers. But it also gets
your data disks off the internal controllers, which means you don't
run into nearly as many problems where the system tries to boot off
your data drives, or you have to put boot blocks on them, etc.
This controller would be more tolerant of your existing drives, since
you have the 850w power supply, it's almost certainly not power
problems either.
John
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-10-09 1:07 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-05 13:10 do i need to give up on this setup Daniel Sanabria
2020-10-05 13:17 ` Reindl Harald
2020-10-05 13:44 ` Roman Mamedov
[not found] ` <CAHscji0pNezf6xCpjWto5-21ayoCeLWm34GTYh5TSgxkOw90mw@mail.gmail.com>
2020-10-05 14:04 ` Roman Mamedov
2020-10-05 14:10 ` Reindl Harald
2020-10-05 14:28 ` Daniel Sanabria
2020-10-05 15:58 ` Roger Heflin
2020-10-06 7:56 ` Daniel Sanabria
2020-10-06 8:24 ` Reindl Harald
2020-10-06 10:53 ` Roger Heflin
2020-10-06 11:29 ` antlists
2020-10-06 14:59 ` Roger Heflin
2020-10-09 1:00 ` John Stoffel
2020-10-06 15:03 ` Tim Small
2020-10-06 16:01 ` Daniel Sanabria
2020-10-07 7:26 ` Tim Small
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).