IO errors when building RAID1.... ?

IO errors when building RAID1.... ?

[Pierre Couderc]

When trying to build a RAID1 on main fs. After  normal debian stretch 
install :

<code>

root@server:/home/nous# btrfs device add /dev/sdb1 /
root@server:/home/nous# btrfs fi show
Label: none  uuid: ef0b9dad-c0eb-4a3b-9b41-e5e249363abc
         Total devices 2 FS bytes used 824.60MiB
         devid    1 size 1.82TiB used 3.02GiB path /dev/sda1
         devid    2 size 1.82TiB used 0.00B path /dev/sdb1

root@server:/home/nous# btrfs balance start -v -mconvert=raid1 
-dconvert=raid1 /
Dumping filters: flags 0x7, state 0x0, force is off
   DATA (flags 0x100): converting, target=16, soft is off
   METADATA (flags 0x100): converting, target=16, soft is off
   SYSTEM (flags 0x100): converting, target=16, soft is off
Killed
root@server:/home/nous# btrfs fi show
Label: none  uuid: ef0b9dad-c0eb-4a3b-9b41-e5e249363abc
         Total devices 2 FS bytes used 1.29GiB
         devid    2 size 1.82TiB used 1.00GiB path /dev/sdb1
         *** Some devices missing
</code>

Some IO errors on /dev/sda are found in journalctl (see them below)

I cannot believe that /dev/sda has no hard disk errors when installing 
without problems, but has many ones when I "btrfs device add /dev/sdb1 /".

I can reproduce the problem : reinstall (3times...) and try "btrfs 
device add /dev/sdb1 /" with the same results...


<journalctl>

Aug 31 17:34:55 server su[559]: Successful su for root by nous
Aug 31 17:34:55 server su[559]: + /dev/pts/1 nous:root
Aug 31 17:34:55 server su[559]: pam_unix(su:session): session opened for 
user root by nous(uid=1000)
Aug 31 17:34:55 server su[559]: pam_systemd(su:session): Cannot create 
session: Already running in a session
Aug 31 17:35:03 server kernel: BTRFS info (device sda1): disk added 
/dev/sdb1
Aug 31 17:35:40 server kernel: BTRFS info (device sda1): relocating 
block group 1103101952 flags 1
Aug 31 17:36:12 server sshd[572]: Accepted password for nous from 
2a01:e34:eeaf:c5f0:e54:15ff:feb1:b1c9 port 49308 ssh2
Aug 31 17:36:12 server sshd[572]: pam_unix(sshd:session): session opened 
for user nous by (uid=0)
Aug 31 17:36:12 server systemd-logind[415]: New session 4 of user nous.
Aug 31 17:36:12 server systemd[1]: Started Session 4 of user nous.
Aug 31 17:36:16 server kernel: ata1: lost interrupt (Status 0x50)
Aug 31 17:36:16 server kernel: ata1.00: exception Emask 0x50 SAct 0x0 
SErr 0x40d0802 action 0xe frozen
Aug 31 17:36:16 server kernel: ata1.00: SError: { RecovComm HostInt 
PHYRdyChg CommWake 10B8B DevExch }
Aug 31 17:36:16 server kernel: ata1.00: failed command: READ DMA
Aug 31 17:36:16 server kernel: ata1.00: cmd 
c8/00:60:00:cd:02/00:00:00:00:00/e0 tag 0 dma 49152 in
                                         res 
40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x54 (ATA bus error)
Aug 31 17:36:16 server kernel: ata1.00: status: { DRDY }
Aug 31 17:36:16 server kernel: ata1.00: hard resetting link
Aug 31 17:36:17 server kernel: ata1.01: hard resetting link
Aug 31 17:36:18 server kernel: ata1.01: failed to resume link (SControl 0)
Aug 31 17:36:18 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 
133 SControl 300)
Aug 31 17:36:18 server kernel: ata1.01: SATA link down (SStatus 4 
SControl 0)
Aug 31 17:36:18 server kernel: ata1.00: NODEV after polling detection
Aug 31 17:36:18 server kernel: ata1.00: revalidation failed (errno=-2)
Aug 31 17:36:20 server su[590]: Successful su for root by nous
Aug 31 17:36:20 server su[590]: + /dev/pts/2 nous:root
Aug 31 17:36:20 server su[590]: pam_unix(su:session): session opened for 
user root by nous(uid=1000)
Aug 31 17:36:20 server su[590]: pam_systemd(su:session): Cannot create 
session: Already running in a session
Aug 31 17:36:23 server kernel: ata1.00: hard resetting link
Aug 31 17:36:23 server kernel: ata1.01: hard resetting link
Aug 31 17:36:24 server kernel: ata1.01: failed to resume link (SControl 0)
Aug 31 17:36:25 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 
133 SControl 300)
Aug 31 17:36:25 server kernel: ata1.01: SATA link down (SStatus 4 
SControl 0)
Aug 31 17:36:25 server kernel: ata1.00: NODEV after polling detection
Aug 31 17:36:25 server kernel: ata1.00: revalidation failed (errno=-2)
Aug 31 17:36:30 server kernel: ata1.00: hard resetting link
Aug 31 17:36:30 server kernel: ata1.01: hard resetting link
Aug 31 17:36:31 server kernel: ata1.01: failed to resume link (SControl 0)
Aug 31 17:36:31 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 
133 SControl 300)
Aug 31 17:36:31 server kernel: ata1.01: SATA link down (SStatus 4 
SControl 0)
Aug 31 17:36:31 server kernel: ata1.00: NODEV after polling detection
Aug 31 17:36:31 server kernel: ata1.00: revalidation failed (errno=-2)
Aug 31 17:36:31 server kernel: ata1.00: disabled
Aug 31 17:36:36 server kernel: ata1.00: hard resetting link
Aug 31 17:36:37 server kernel: ata1.01: hard resetting link
Aug 31 17:36:38 server kernel: ata1.01: failed to resume link (SControl 0)
Aug 31 17:36:38 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 
133 SControl 300)
Aug 31 17:36:38 server kernel: ata1.01: SATA link down (SStatus 4 
SControl 0)
Aug 31 17:36:38 server kernel: ata1.00: NODEV after polling detection
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 FAILED Result: 
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 
Illegal Request [current]
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 Add. Sense: 
Unaligned write command
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 CDB: Read(10) 28 
00 00 02 cd 00 00 00 60 00
Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda, 
sector 183552
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] killing request
Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 1, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 2, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 3, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 4, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 5, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 6, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1 
errs: wr 7, rd 3, flush 0, corrupt 0, gen 0
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] FAILED Result: 
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] CDB: Write(10) 2a 00 00 
61 9c 00 00 0a 00 00
Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda, 
sector 6396928
Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device

more than 100 identical lines...

Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
Aug 31 17:36:38 server kernel: ata1: EH complete
Aug 31 17:36:38 server kernel: ata1.00: detaching (SCSI 0:0:0:0)
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Synchronize Cache(10) 
failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Stopping disk
Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Start/Stop Unit failed: 
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Aug 31 17:36:38 server kernel: Buffer I/O error on dev sda1, logical 
block 488378352, async page read
Aug 31 17:36:38 server kernel: scsi 0:0:0:0: rejecting I/O to dead device
Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda, 
sector 6762624
Aug 31 17:36:38 server kernel: BTRFS: error (device sda1) in 
btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing 
out transaction)
Aug 31 17:36:38 server kernel: BTRFS info (device sda1): forced readonly
Aug 31 17:36:38 server kernel: BTRFS warning (device sda1): Skipping 
commit of aborted transaction.
Aug 31 17:36:38 server kernel: ------------[ cut here ]------------
Aug 31 17:36:38 server kernel: WARNING: CPU: 1 PID: 159 at 
/build/linux-cRtIym/linux-4.9.30/fs/btrfs/transaction.c:1850 
cleanup_transaction+0x1f0/0x2e0 [btrfs]
Aug 31 17:36:38 server kernel: BTRFS: Transaction aborted (error -5)
Aug 31 17:36:38 server kernel: Modules linked in: intel_rapl 
x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass eeepc_wmi 
asus_wmi crct10dif_pclmul sparse_keymap crc32_pclmul g

Re: IO errors when building RAID1.... ?

[Chris Murphy]

On Fri, Aug 31, 2018 at 10:35 AM, Pierre Couderc <pierre@couderc.eu> wrote:


>
> Aug 31 17:34:55 server su[559]: Successful su for root by nous
> Aug 31 17:34:55 server su[559]: + /dev/pts/1 nous:root
> Aug 31 17:34:55 server su[559]: pam_unix(su:session): session opened for
> user root by nous(uid=1000)
> Aug 31 17:34:55 server su[559]: pam_systemd(su:session): Cannot create
> session: Already running in a session
> Aug 31 17:35:03 server kernel: BTRFS info (device sda1): disk added
> /dev/sdb1
> Aug 31 17:35:40 server kernel: BTRFS info (device sda1): relocating block
> group 1103101952 flags 1
> Aug 31 17:36:12 server sshd[572]: Accepted password for nous from
> 2a01:e34:eeaf:c5f0:e54:15ff:feb1:b1c9 port 49308 ssh2
> Aug 31 17:36:12 server sshd[572]: pam_unix(sshd:session): session opened for
> user nous by (uid=0)
> Aug 31 17:36:12 server systemd-logind[415]: New session 4 of user nous.
> Aug 31 17:36:12 server systemd[1]: Started Session 4 of user nous.
> Aug 31 17:36:16 server kernel: ata1: lost interrupt (Status 0x50)
> Aug 31 17:36:16 server kernel: ata1.00: exception Emask 0x50 SAct 0x0 SErr
> 0x40d0802 action 0xe frozen
> Aug 31 17:36:16 server kernel: ata1.00: SError: { RecovComm HostInt
> PHYRdyChg CommWake 10B8B DevExch }
> Aug 31 17:36:16 server kernel: ata1.00: failed command: READ DMA
> Aug 31 17:36:16 server kernel: ata1.00: cmd
> c8/00:60:00:cd:02/00:00:00:00:00/e0 tag 0 dma 49152 in
>                                         res
> 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x54 (ATA bus error)
> Aug 31 17:36:16 server kernel: ata1.00: status: { DRDY }
> Aug 31 17:36:16 server kernel: ata1.00: hard resetting link
> Aug 31 17:36:17 server kernel: ata1.01: hard resetting link
> Aug 31 17:36:18 server kernel: ata1.01: failed to resume link (SControl 0)
> Aug 31 17:36:18 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 133
> SControl 300)
> Aug 31 17:36:18 server kernel: ata1.01: SATA link down (SStatus 4 SControl
> 0)
> Aug 31 17:36:18 server kernel: ata1.00: NODEV after polling detection
> Aug 31 17:36:18 server kernel: ata1.00: revalidation failed (errno=-2)
> Aug 31 17:36:20 server su[590]: Successful su for root by nous
> Aug 31 17:36:20 server su[590]: + /dev/pts/2 nous:root
> Aug 31 17:36:20 server su[590]: pam_unix(su:session): session opened for
> user root by nous(uid=1000)
> Aug 31 17:36:20 server su[590]: pam_systemd(su:session): Cannot create
> session: Already running in a session
> Aug 31 17:36:23 server kernel: ata1.00: hard resetting link
> Aug 31 17:36:23 server kernel: ata1.01: hard resetting link
> Aug 31 17:36:24 server kernel: ata1.01: failed to resume link (SControl 0)
> Aug 31 17:36:25 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 133
> SControl 300)
> Aug 31 17:36:25 server kernel: ata1.01: SATA link down (SStatus 4 SControl
> 0)
> Aug 31 17:36:25 server kernel: ata1.00: NODEV after polling detection
> Aug 31 17:36:25 server kernel: ata1.00: revalidation failed (errno=-2)
> Aug 31 17:36:30 server kernel: ata1.00: hard resetting link
> Aug 31 17:36:30 server kernel: ata1.01: hard resetting link
> Aug 31 17:36:31 server kernel: ata1.01: failed to resume link (SControl 0)
> Aug 31 17:36:31 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 133
> SControl 300)
> Aug 31 17:36:31 server kernel: ata1.01: SATA link down (SStatus 4 SControl
> 0)
> Aug 31 17:36:31 server kernel: ata1.00: NODEV after polling detection
> Aug 31 17:36:31 server kernel: ata1.00: revalidation failed (errno=-2)
> Aug 31 17:36:31 server kernel: ata1.00: disabled
> Aug 31 17:36:36 server kernel: ata1.00: hard resetting link
> Aug 31 17:36:37 server kernel: ata1.01: hard resetting link
> Aug 31 17:36:38 server kernel: ata1.01: failed to resume link (SControl 0)
> Aug 31 17:36:38 server kernel: ata1.00: SATA link up 6.0 Gbps (SStatus 133
> SControl 300)
> Aug 31 17:36:38 server kernel: ata1.01: SATA link down (SStatus 4 SControl
> 0)
> Aug 31 17:36:38 server kernel: ata1.00: NODEV after polling detection
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : Illegal
> Request [current]
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 Add. Sense: Unaligned
> write command
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] tag#0 CDB: Read(10) 28 00
> 00 02 cd 00 00 00 60 00
> Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda,
> sector 183552
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] killing request
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 1, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 2, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 3, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 4, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 5, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 6, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: BTRFS error (device sda1): bdev /dev/sda1
> errs: wr 7, rd 3, flush 0, corrupt 0, gen 0
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] FAILED Result:
> hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] CDB: Write(10) 2a 00 00 61
> 9c 00 00 0a 00 00
> Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda,
> sector 6396928
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
>
> more than 100 identical lines...
>
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: rejecting I/O to offline device
> Aug 31 17:36:38 server kernel: ata1: EH complete
> Aug 31 17:36:38 server kernel: ata1.00: detaching (SCSI 0:0:0:0)
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Synchronize Cache(10)
> failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Stopping disk
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] Start/Stop Unit failed:
> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> Aug 31 17:36:38 server kernel: Buffer I/O error on dev sda1, logical block
> 488378352, async page read
> Aug 31 17:36:38 server kernel: scsi 0:0:0:0: rejecting I/O to dead device
> Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda,
> sector 6762624
> Aug 31 17:36:38 server kernel: BTRFS: error (device sda1) in
> btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out
> transaction)
> Aug 31 17:36:38 server kernel: BTRFS info (device sda1): forced readonly
> Aug 31 17:36:38 server kernel: BTRFS warning (device sda1): Skipping commit
> of aborted transaction.
> Aug 31 17:36:38 server kernel: ------------[ cut here ]------------
> Aug 31 17:36:38 server kernel: WARNING: CPU: 1 PID: 159 at
> /build/linux-cRtIym/linux-4.9.30/fs/btrfs/transaction.c:1850
> cleanup_transaction+0x1f0/0x2e0 [btrfs]
> Aug 31 17:36:38 server kernel: BTRFS: Transaction aborted (error -5)
> Aug 31 17:36:38 server kernel: Modules linked in: intel_rapl
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass eeepc_wmi
> asus_wmi crct10dif_pclmul sparse_keymap crc32_pclmul g
>
>


These are hardware problems and aren't related to Btrfs.

>sd 0:0:0:0: [sda] FAILED Result:
> hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> Aug 31 17:36:38 server kernel: sd 0:0:0:0: [sda] CDB: Write(10) 2a 00 00 61
> 9c 00 00 0a 00 00
> Aug 31 17:36:38 server kernel: blk_update_request: I/O error, dev sda,
> sector 6396928

Bad sector which is failing write. This is fatal, there isn't anything
the block layer or Btrfs (or ext4 or XFS) can do about it. Well,
ext234 do have an option to scan for bad sectors and create a bad
sector map which then can be used at mkfs time, and ext234 will avoid
using those sectors. And also the md driver has a bad sector option
for the same, and does remapping. But XFS and Btrfs don't do that.

If the drive is under warranty, get it swapped out, this is definitely
a warranty covered problem.




-- 
Chris Murphy

Re: IO errors when building RAID1.... ?

[Chris Murphy]

If you want you can post the output from 'sudo smartctl -x /dev/sda'
which will contain more information... but this is in some sense
superfluous. The problem is very clearly a bad drive, the drive
explicitly report to libata a write error, and included the sector LBA
affected, and only the drive firmware would know that. It's not likely
a cable problem or something like. And that the write error is
reported at all means it's persistent, not transient.


Chris Murphy

Re: IO errors when building RAID1.... ?

[Duncan]

Chris Murphy posted on Fri, 31 Aug 2018 13:02:16 -0600 as excerpted:

> If you want you can post the output from 'sudo smartctl -x /dev/sda'
> which will contain more information... but this is in some sense
> superfluous. The problem is very clearly a bad drive, the drive
> explicitly report to libata a write error, and included the sector LBA
> affected, and only the drive firmware would know that. It's not likely a
> cable problem or something like. And that the write error is reported at
> all means it's persistent, not transient.

Two points:

1) Does this happen to be an archive/SMR (shingled magnetic recording) 
device?  If so that might be the problem as such devices really aren't 
suited to normal usage (they really are designed for archiving), and 
btrfs' COW patterns can exacerbate the issue.  It's quite possible that 
the original install didn't load up the IO as heavily as the balance-
convert does, so the problem appears with convert but not for install.

2) Assuming it's /not/ an SMR issue, and smartctl doesn't say it's dying, 
I'd suggest running badblocks -w (make sure the device doesn't have 
anything valuable on it!) on the device -- note that this will take 
awhile, probably a couple days perhaps longer, as it writes four 
different patterns to the entire device one at a time, reading everything 
back to verify the pattern was written correctly, so it's actually going 
over the entire device 8 times, alternating write and read, but it should 
settle the issue of the reliability of the device.

Or if you'd rather spend the money than the time and it's not under 
warrantee still, just replace it, or at least buy a new one to use while 
you run the tests on that one.  I fully understand that tying up the 
thing running tests on it for days straight may not be viable.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: IO errors when building RAID1.... ?

[Pierre Couderc]

On 09/01/2018 03:35 AM, Duncan wrote:
> Chris Murphy posted on Fri, 31 Aug 2018 13:02:16 -0600 as excerpted:
>
>> If you want you can post the output from 'sudo smartctl -x /dev/sda'
>> which will contain more information... but this is in some sense
>> superfluous. The problem is very clearly a bad drive, the drive
>> explicitly report to libata a write error, and included the sector LBA
>> affected, and only the drive firmware would know that. It's not likely a
>> cable problem or something like. And that the write error is reported at
>> all means it's persistent, not transient.
> Two points:
>
> 1) Does this happen to be an archive/SMR (shingled magnetic recording)
> device?  If so that might be the problem as such devices really aren't
> suited to normal usage (they really are designed for archiving), and
> btrfs' COW patterns can exacerbate the issue.  It's quite possible that
> the original install didn't load up the IO as heavily as the balance-
> convert does, so the problem appears with convert but not for install.
>
> 2) Assuming it's /not/ an SMR issue, and smartctl doesn't say it's dying,
> I'd suggest running badblocks -w (make sure the device doesn't have
> anything valuable on it!) on the device -- note that this will take
> awhile, probably a couple days perhaps longer, as it writes four
> different patterns to the entire device one at a time, reading everything
> back to verify the pattern was written correctly, so it's actually going
> over the entire device 8 times, alternating write and read, but it should
> settle the issue of the reliability of the device.
>
> Or if you'd rather spend the money than the time and it's not under
> warrantee still, just replace it, or at least buy a new one to use while
> you run the tests on that one.  I fully understand that tying up the
> thing running tests on it for days straight may not be viable.
>
Thank you

sudo smartctl -x /dev/sda

finds 4 errors...

Thank you, I shall spend no more time o that and replace the HDD...

Anyway, I should expect a less cryptic message from btrfs...

PC

Re: IO errors when building RAID1.... ?

[Pierre Couderc]

On 08/31/2018 08:52 PM, Chris Murphy wrote:
>
> Bad sector which is failing write. This is fatal, there isn't anything
> the block layer or Btrfs (or ext4 or XFS) can do about it. Well,
> ext234 do have an option to scan for bad sectors and create a bad
> sector map which then can be used at mkfs time, and ext234 will avoid
> using those sectors. And also the md driver has a bad sector option
> for the same, and does remapping. But XFS and Btrfs don't do that.
>
> If the drive is under warranty, get it swapped out, this is definitely
> a warranty covered problem.
>
>
>
>
Thank you very much.

Once upon a time...(I am old), there were lists of bad sectors, and the 
software did avoid wrting in them. It seems to have disappeared. For 
which reason ? Maybe because these errors occur so  rarely, that it is 
not worth the trouble ?

Re: IO errors when building RAID1.... ?

[Chris Murphy]

On Sat, Sep 1, 2018 at 1:03 AM, Pierre Couderc <pierre@couderc.eu> wrote:
>
>
> On 08/31/2018 08:52 PM, Chris Murphy wrote:
>>
>>
>> Bad sector which is failing write. This is fatal, there isn't anything
>> the block layer or Btrfs (or ext4 or XFS) can do about it. Well,
>> ext234 do have an option to scan for bad sectors and create a bad
>> sector map which then can be used at mkfs time, and ext234 will avoid
>> using those sectors. And also the md driver has a bad sector option
>> for the same, and does remapping. But XFS and Btrfs don't do that.
>>
>> If the drive is under warranty, get it swapped out, this is definitely
>> a warranty covered problem.
>>
>>
>>
>>
> Thank you very much.
>
> Once upon a time...(I am old), there were lists of bad sectors, and the
> software did avoid wrting in them. It seems to have disappeared. For which
> reason ? Maybe because these errors occur so  rarely, that it is not worth
> the trouble ?

For > 10 years drive firmware handles bad sector remapping internally.
It remaps the sector logical address to a reserve physical sector.

NTFS and ext[234] have a means of accepting a list of bad sectors, and
will avoid using them. Btrfs doesn't. But also ZFS, XFS, APFS, HFS+
and I think even FAT, lack this capability. I'm not aware of any file
system that once had bad sector tracking, that has since dropped the
capability.

-- 
Chris Murphy

Re: IO errors when building RAID1.... ?

[Pierre Couderc]

On 09/03/2018 05:15 AM, Chris Murphy wrote:
> On Sat, Sep 1, 2018 at 1:03 AM, Pierre Couderc <pierre@couderc.eu> wrote:
>>
>> On 08/31/2018 08:52 PM, Chris Murphy wrote:
>>>
>>> Bad sector which is failing write. This is fatal, there isn't anything
>>> the block layer or Btrfs (or ext4 or XFS) can do about it. Well,
>>> ext234 do have an option to scan for bad sectors and create a bad
>>> sector map which then can be used at mkfs time, and ext234 will avoid
>>> using those sectors. And also the md driver has a bad sector option
>>> for the same, and does remapping. But XFS and Btrfs don't do that.
>>>
>>> If the drive is under warranty, get it swapped out, this is definitely
>>> a warranty covered problem.
>>>
>>>
>>>
>>>
>> Thank you very much.
>>
>> Once upon a time...(I am old), there were lists of bad sectors, and the
>> software did avoid wrting in them. It seems to have disappeared. For which
>> reason ? Maybe because these errors occur so  rarely, that it is not worth
>> the trouble ?
> For > 10 years drive firmware handles bad sector remapping internally.
> It remaps the sector logical address to a reserve physical sector.
>
> NTFS and ext[234] have a means of accepting a list of bad sectors, and
> will avoid using them. Btrfs doesn't. But also ZFS, XFS, APFS, HFS+
> and I think even FAT, lack this capability. I'm not aware of any file
> system that once had bad sector tracking, that has since dropped the
> capability.
>
Thank you, you are very clear.

Re: IO errors when building RAID1.... ?

[Adam Borowski]

On Sun, Sep 02, 2018 at 09:15:25PM -0600, Chris Murphy wrote:
> For > 10 years drive firmware handles bad sector remapping internally.
> It remaps the sector logical address to a reserve physical sector.
> 
> NTFS and ext[234] have a means of accepting a list of bad sectors, and
> will avoid using them. Btrfs doesn't. But also ZFS, XFS, APFS, HFS+
> and I think even FAT, lack this capability.

<nitpick>
FAT entry FF7 (FAT12)/FFF7 (FAT16)/...
</nitpick>


-- 
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]

Re: IO errors when building RAID1.... ?

[Chris Murphy]

On Mon, Sep 3, 2018 at 4:23 AM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Sun, Sep 02, 2018 at 09:15:25PM -0600, Chris Murphy wrote:
>> For > 10 years drive firmware handles bad sector remapping internally.
>> It remaps the sector logical address to a reserve physical sector.
>>
>> NTFS and ext[234] have a means of accepting a list of bad sectors, and
>> will avoid using them. Btrfs doesn't. But also ZFS, XFS, APFS, HFS+
>> and I think even FAT, lack this capability.
>
> <nitpick>
> FAT entry FF7 (FAT12)/FFF7 (FAT16)/...
> </nitpick>

Oh yeah even Linux mkdosfs does have -c option to check for bad
sectors and presumably will remove them from use. It doesn't accept a
separate list though, like badblocks + mke2fs.

-- 
Chris Murphy