* High IO Wait with RAID 1
@ 2009-03-12 23:46 Ryan Wagoner
2009-03-13 0:48 ` Alain Williams
2009-03-13 14:48 ` John Robinson
0 siblings, 2 replies; 14+ messages in thread
From: Ryan Wagoner @ 2009-03-12 23:46 UTC (permalink / raw)
To: linux-raid
From what I can tell the issue here lies with mdadm and/or its
interaction with CentOS 5.2. Let me first go over the configuration of
both systems.
System 1 - CentOS 5.2 x86_64
2x Seagate 7200.9 160GB in RAID 1
2x Seagate 7200.10 320GB in RAID 1
3x Hitachi Deskstar 7K1000 1TB in RAID 5
All attached to Supermicro LSI 1068 PCI Express controller
System 2 - CentOS 5.2 x86
1x Non Raid System Drive
2x Hitachi Deskstart 7K1000 1TB in RAID 1
Attached to onboard ICH controller
Both systems exhibit the same issues on the RAID 1 drives. That rules
out the drive brand and controller card. During any IO intensive
process the IO wait will raise and the system load will climb. I've
had the IO wait as high as 70% and the load at 13+ while migrating a
vmdk file with vmware-vdiskmanager. You can easily recreate the issue
with bonnie++.
I can perform the same disk intensive operation on the RAID 5 array
with almost no io wait or load. What is the deal with this? Is there
something I can tweak?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-12 23:46 High IO Wait with RAID 1 Ryan Wagoner
@ 2009-03-13 0:48 ` Alain Williams
2009-03-13 3:21 ` Ryan Wagoner
2009-03-13 18:42 ` David Rees
2009-03-13 14:48 ` John Robinson
1 sibling, 2 replies; 14+ messages in thread
From: Alain Williams @ 2009-03-13 0:48 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: linux-raid
On Thu, Mar 12, 2009 at 06:46:45PM -0500, Ryan Wagoner wrote:
> From what I can tell the issue here lies with mdadm and/or its
> interaction with CentOS 5.2. Let me first go over the configuration of
> both systems.
>
> System 1 - CentOS 5.2 x86_64
> 2x Seagate 7200.9 160GB in RAID 1
> 2x Seagate 7200.10 320GB in RAID 1
> 3x Hitachi Deskstar 7K1000 1TB in RAID 5
> All attached to Supermicro LSI 1068 PCI Express controller
>
> System 2 - CentOS 5.2 x86
> 1x Non Raid System Drive
> 2x Hitachi Deskstart 7K1000 1TB in RAID 1
> Attached to onboard ICH controller
>
> Both systems exhibit the same issues on the RAID 1 drives. That rules
> out the drive brand and controller card. During any IO intensive
> process the IO wait will raise and the system load will climb. I've
> had the IO wait as high as 70% and the load at 13+ while migrating a
> vmdk file with vmware-vdiskmanager. You can easily recreate the issue
> with bonnie++.
I suspect that the answer is 'no', however I am seeing problems with raid 1
on CentOS 5.2 x86_64. The system worked nicely for some 2 months, then apparently
a disk died and it's mirror appeared to have problems before the first could be
replaced. The motherboard & both disks have now been replaced (data saved with a bit
of luck & juggling). I have been assuming hardware, but there seems little else
to change... and you report long I/O waits that I saw and still see
(even when I don't see the kernel error messages below).
Disks have been Seagate & Samsung, but now both ST31000333AS (1TB) as raid 1.
Adaptec AIC7902 Ultra320 SCSI adapter
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
Executing 'w' or 'cat /proc/mdstat' can take several seconds,
failing sdb with mdadm and system performance becomes great again.
I am seeing this sort of thing in /var/log/messages:
Mar 12 09:21:58 BFPS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
Mar 12 09:21:58 BFPS kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Mar 12 09:21:58 BFPS kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 12 09:21:58 BFPS kernel: ata2.00: status: { DRDY }
Mar 12 09:22:03 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xd0)
Mar 12 09:22:08 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
Mar 12 09:22:08 BFPS kernel: ata2: hard resetting link
Mar 12 09:22:08 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 09:22:39 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
Mar 12 09:22:39 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Mar 12 09:22:39 BFPS kernel: ata2.00: revalidation failed (errno=-5)
Mar 12 09:22:39 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
Mar 12 09:22:44 BFPS kernel: ata2: hard resetting link
Mar 12 09:24:02 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
Mar 12 09:24:06 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
Mar 12 09:24:06 BFPS kernel: ata2.00: disabled
Mar 12 09:24:06 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xff)
Mar 12 09:24:06 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 12 09:24:06 BFPS kernel: ata2: EH complete
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 1953519821
Mar 12 09:24:06 BFPS kernel: raid1: Disk failure on sdb2, disabling device.
Mar 12 09:24:06 BFPS kernel: Operation continuing on 1 devices
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975018957
Mar 12 09:24:06 BFPS kernel: md: md3: sync done.
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975019981
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975021005
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022029
Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022157
Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
Mar 12 09:24:06 BFPS kernel: disk 1, wo:1, o:0, dev:sdb2
Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, not capable of SMART self-check
Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, failed to read SMART Attribute Data
Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
--
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256 http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
Past chairman of UKUUG: http://www.ukuug.org/
#include <std_disclaimer.h>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 0:48 ` Alain Williams
@ 2009-03-13 3:21 ` Ryan Wagoner
2009-03-13 9:39 ` Robin Hill
` (2 more replies)
2009-03-13 18:42 ` David Rees
1 sibling, 3 replies; 14+ messages in thread
From: Ryan Wagoner @ 2009-03-13 3:21 UTC (permalink / raw)
To: Alain Williams; +Cc: linux-raid
I'm glad I'm not the only one experiencing the issue. Luckily the
issues on both my systems aren't as bad. I don't have any errors
showing in /var/log/messages on either system. I've been trying to
track down this issue for about a year now. I just recently my the
connection with RAID 1 and mdadm when copying data on the second
system.
Unfortunately it looks like the fix is to avoid software RAID 1. I
prefer software RAID over hardware RAID on my home systems for the
flexibility it offers, especially since I can easily move the disks
between systems in the case of hardware failure.
If I can find time to migrate the VMs, which run my web sites and
email to another machine, I'll reinstall the one system utilizing RAID
1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
just pass the remaining disks through.
You would think that software RAID 1 would be much simpler to
implement than RAID 5 performance wise.
Ryan
On Thu, Mar 12, 2009 at 7:48 PM, Alain Williams <addw@phcomp.co.uk> wrote:
> On Thu, Mar 12, 2009 at 06:46:45PM -0500, Ryan Wagoner wrote:
>> From what I can tell the issue here lies with mdadm and/or its
>> interaction with CentOS 5.2. Let me first go over the configuration of
>> both systems.
>>
>> System 1 - CentOS 5.2 x86_64
>> 2x Seagate 7200.9 160GB in RAID 1
>> 2x Seagate 7200.10 320GB in RAID 1
>> 3x Hitachi Deskstar 7K1000 1TB in RAID 5
>> All attached to Supermicro LSI 1068 PCI Express controller
>>
>> System 2 - CentOS 5.2 x86
>> 1x Non Raid System Drive
>> 2x Hitachi Deskstart 7K1000 1TB in RAID 1
>> Attached to onboard ICH controller
>>
>> Both systems exhibit the same issues on the RAID 1 drives. That rules
>> out the drive brand and controller card. During any IO intensive
>> process the IO wait will raise and the system load will climb. I've
>> had the IO wait as high as 70% and the load at 13+ while migrating a
>> vmdk file with vmware-vdiskmanager. You can easily recreate the issue
>> with bonnie++.
>
> I suspect that the answer is 'no', however I am seeing problems with raid 1
> on CentOS 5.2 x86_64. The system worked nicely for some 2 months, then apparently
> a disk died and it's mirror appeared to have problems before the first could be
> replaced. The motherboard & both disks have now been replaced (data saved with a bit
> of luck & juggling). I have been assuming hardware, but there seems little else
> to change... and you report long I/O waits that I saw and still see
> (even when I don't see the kernel error messages below).
>
> Disks have been Seagate & Samsung, but now both ST31000333AS (1TB) as raid 1.
> Adaptec AIC7902 Ultra320 SCSI adapter
> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
>
> Executing 'w' or 'cat /proc/mdstat' can take several seconds,
> failing sdb with mdadm and system performance becomes great again.
>
> I am seeing this sort of thing in /var/log/messages:
> Mar 12 09:21:58 BFPS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
> Mar 12 09:21:58 BFPS kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> Mar 12 09:21:58 BFPS kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> Mar 12 09:21:58 BFPS kernel: ata2.00: status: { DRDY }
> Mar 12 09:22:03 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xd0)
> Mar 12 09:22:08 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
> Mar 12 09:22:08 BFPS kernel: ata2: hard resetting link
> Mar 12 09:22:08 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Mar 12 09:22:39 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
> Mar 12 09:22:39 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
> Mar 12 09:22:39 BFPS kernel: ata2.00: revalidation failed (errno=-5)
> Mar 12 09:22:39 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
> Mar 12 09:22:44 BFPS kernel: ata2: hard resetting link
> Mar 12 09:24:02 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
> Mar 12 09:24:06 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
> Mar 12 09:24:06 BFPS kernel: ata2.00: disabled
> Mar 12 09:24:06 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xff)
> Mar 12 09:24:06 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Mar 12 09:24:06 BFPS kernel: ata2: EH complete
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 1953519821
> Mar 12 09:24:06 BFPS kernel: raid1: Disk failure on sdb2, disabling device.
> Mar 12 09:24:06 BFPS kernel: Operation continuing on 1 devices
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975018957
> Mar 12 09:24:06 BFPS kernel: md: md3: sync done.
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975019981
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975021005
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022029
> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022157
> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
> Mar 12 09:24:06 BFPS kernel: disk 1, wo:1, o:0, dev:sdb2
> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
>
> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, not capable of SMART self-check
> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, failed to read SMART Attribute Data
> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
>
>
>
> --
> Alain Williams
> Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
> +44 (0) 787 668 0256 http://www.phcomp.co.uk/
> Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
> Past chairman of UKUUG: http://www.ukuug.org/
> #include <std_disclaimer.h>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 3:21 ` Ryan Wagoner
@ 2009-03-13 9:39 ` Robin Hill
2009-03-13 10:17 ` Alain Williams
2009-03-13 16:22 ` Bill Davidsen
2 siblings, 0 replies; 14+ messages in thread
From: Robin Hill @ 2009-03-13 9:39 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]
On Thu Mar 12, 2009 at 10:21:28PM -0500, Ryan Wagoner wrote:
> I'm glad I'm not the only one experiencing the issue. Luckily the
> issues on both my systems aren't as bad. I don't have any errors
> showing in /var/log/messages on either system. I've been trying to
> track down this issue for about a year now. I just recently my the
> connection with RAID 1 and mdadm when copying data on the second
> system.
>
> Unfortunately it looks like the fix is to avoid software RAID 1. I
> prefer software RAID over hardware RAID on my home systems for the
> flexibility it offers, especially since I can easily move the disks
> between systems in the case of hardware failure.
>
> If I can find time to migrate the VMs, which run my web sites and
> email to another machine, I'll reinstall the one system utilizing RAID
> 1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
> just pass the remaining disks through.
>
You could also try using software RAID 10 or RAID 5 on the two drives.
Both of these will be effectively the same as a RAID 1 (ISTR you can
actually just update the metadata to convert from a RAID 1 to a RAID 5)
but will use a different code path so should eliminate the issue you're
seeing. A 2 drive RAID 10 with far layout will also give some read
performance improvements (at the cost of slightly slower writes).
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 3:21 ` Ryan Wagoner
2009-03-13 9:39 ` Robin Hill
@ 2009-03-13 10:17 ` Alain Williams
[not found] ` <7d86ddb90903130519p4268dc33vc8ad42b53aefa2e2@mail.gmail.com>
2009-03-13 16:22 ` Bill Davidsen
2 siblings, 1 reply; 14+ messages in thread
From: Alain Williams @ 2009-03-13 10:17 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: linux-raid
On Thu, Mar 12, 2009 at 10:21:28PM -0500, Ryan Wagoner wrote:
> I'm glad I'm not the only one experiencing the issue. Luckily the
> issues on both my systems aren't as bad. I don't have any errors
> showing in /var/log/messages on either system. I've been trying to
> track down this issue for about a year now. I just recently my the
> connection with RAID 1 and mdadm when copying data on the second
> system.
Did you have the problem straight from install, or perhaps when a new
kernel started being used ?
My system worked well for some months, there was no kernel update and
it started to go wrong a couple of weeks ago. I also see errors
when I run 'badblocks' -- which makes it smell of a hardware issue,
but the disks were tested, on return, by the hardware supplier and
they did not find any problem with them.
Given that you are not seeing anything in /var/log/messages make me
think that I do have some other problem -- perhaps in addition to
what you have.
--
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256 http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
Past chairman of UKUUG: http://www.ukuug.org/
#include <std_disclaimer.h>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Fwd: High IO Wait with RAID 1
[not found] ` <7d86ddb90903130519p4268dc33vc8ad42b53aefa2e2@mail.gmail.com>
@ 2009-03-13 12:21 ` Ryan Wagoner
0 siblings, 0 replies; 14+ messages in thread
From: Ryan Wagoner @ 2009-03-13 12:21 UTC (permalink / raw)
To: linux-raid
I tried rolling back the kernal and have the same issue. Here is an
example of the dstat output when writing with bonnie++ on RAID 1. As
soon as the write buffer fills up the wait climbs as it is waiting to
write to the disk. The output looks the same on both systems.
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
0 0 100 0 0 0| 0 0 | 586B 1682B| 0 0 |1015 110
0 0 100 0 0 0| 0 0 | 64B 412B| 0 0 |1022 96
4 1 96 1 0 0| 40k 0 | 238B 664B| 0 0 |1011 124
43 6 50 1 0 0|4096B 0 | 375B 428B| 0 0 |1026 90
43 7 50 0 0 0| 0 0 | 64B 412B| 0 0 |1005 60
43 8 50 0 0 0| 0 0 | 64B 412B| 0 0 |1023 91
43 6 50 0 0 0|4096B 0 | 64B 412B| 0 0 |1006 77
40 14 44 0 0 1| 0 62M| 158B 396B| 0 0 |1194 160
40 10 0 46 0 3| 0 145M| 158B 522B| 0 0 |1297 128
38 8 0 52 0 3|4096B 127M| 64B 412B| 0 0 |1276 147
41 9 1 48 0 3|4096B 120M| 174B 366B| 0 0 |1252 129
43 8 3 45 0 0| 0 16k| 158B 412B| 0 0 |1012 113
40 16 6 36 0 1|4096B 41M| 64B 318B| 0 0 |1142 188
42 11 0 45 0 2| 0 130M| 64B 675B| 0 0 |1327 276
43 9 0 44 0 4| 0 138M| 64B 412B| 0 0 |1280 130
34 9 16 38 0 2|4096B 107M| 64B 412B| 0 0 |1229 120
44 9 4 44 0 0| 0 8192B| 64B 412B| 0 0 |1024 175
41 17 0 41 0 0| 0 33M| 192B 366B| 0 0 |1096 193
37 9 1 51 0 3|4096B 126M| 64B 428B| 0 0 |1288 173
44 8 0 44 0 3| 0 142M| 64B 412B| 0 0 |1289 164
Here is the dstat output with the same bonnie command on the RAID 5
volume. This machine has two VMware guests running so the system
wasn't idle when grabbing the output.
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 14 83 2 0 0| 120k 537k| 0 0 | 0 0.2 |3084 10k
18 11 68 1 0 0| 40k 8192B|9180B 1192B| 0 0 |3074 12k
36 18 43 2 0 0| 272k 24M| 123k 4865B| 0 0 |3858 14k
30 26 42 1 1 2| 808k 67M| 21k 1418B| 0 0 |5253 18k
39 19 43 0 0 0|4096B 0 |4600B 692B| 0 0 |3079 11k
36 19 29 17 0 0| 116k 1104k|3024B 2464B| 0 0 |3221 10k
40 17 14 30 0 1| 136k 400k| 86k 5828B| 0 0 |3189 10k
37 21 17 23 0 0| 380k 35M| 30k 1708B| 0 0 |4223 16k
30 29 37 2 2 2|1160k 115M| 390B 550B| 0 0 |6647 24k
31 29 37 1 1 2|1112k 127M| 664B 314B| 0 0 |6745 24k
33 26 28 11 0 1| 728k 71M|3074B 526B| 0 0 |4608 16k
37 24 2 37 0 0| 0 16k|1616B 14k| 0 0 |3086 10k
34 21 11 33 1 1| 388k 33M| 26k 1280B| 0 0 |3939 13k
30 32 36 1 1 1|1304k 111M| 60B 420B| 0 0 |5083 19k
31 35 30 2 1 2|1296k 125M| 19k 2051B| 0 0 |5987 20k
38 22 19 22 0 1| 692k 28M|3084B 2480B| 0 0 |3744 11k
41 17 38 3 0 0| 736k 2184k| 120B 298B| 0 0 |3785 11k
34 30 21 14 0 0| 360k 35M| 48k 2862B| 0 0 |4178 12k
37 26 35 1 1 1|1056k 136M| 13k 1394B| 0 0 |4331 11k
34 28 33 2 0 1|1228k 134M| 30k 1658B| 0 0 |4132 11k
36 21 28 14 0 0| 332k 23M| 151k 5798B| 0 0 |3368 9166
37 18 18 28 0 0| 16k 88k| 13k 990B| 0 0 |3092 8403
38 23 23 16 1 0| 316k 39M| 30k 1920B| 0 0 |3635 9723
32 33 33 2 0 1|1180k 132M| 295B 404B| 0 0 |3907 9935
31 31 35 2 1 1|1120k 123M| 43k 2424B| 0 0 |4746 14k
32 29 37 2 1 1|1380k 71M|3084B 2440B| 0 0 |5341 19k
37 24 36 1 0 0| 700k 53M| 459B 496B| 0 0 |4402 20k
35 20 29 14 1 1|1808k 61M|4596B 500B| 0 0 |5551 19k
30 30 35 2 1 3|1076k 107M| 246B 620B| 0 0 |6769 24k
36 25 30 7 1 2|1088k 66M| 165k 10k| 0 0 |5093 17k
On Fri, Mar 13, 2009 at 5:17 AM, Alain Williams <addw@phcomp.co.uk> wrote:
> On Thu, Mar 12, 2009 at 10:21:28PM -0500, Ryan Wagoner wrote:
>> I'm glad I'm not the only one experiencing the issue. Luckily the
>> issues on both my systems aren't as bad. I don't have any errors
>> showing in /var/log/messages on either system. I've been trying to
>> track down this issue for about a year now. I just recently my the
>> connection with RAID 1 and mdadm when copying data on the second
>> system.
>
> Did you have the problem straight from install, or perhaps when a new
> kernel started being used ?
>
> My system worked well for some months, there was no kernel update and
> it started to go wrong a couple of weeks ago. I also see errors
> when I run 'badblocks' -- which makes it smell of a hardware issue,
> but the disks were tested, on return, by the hardware supplier and
> they did not find any problem with them.
>
> Given that you are not seeing anything in /var/log/messages make me
> think that I do have some other problem -- perhaps in addition to
> what you have.
>
> --
> Alain Williams
> Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
> +44 (0) 787 668 0256 http://www.phcomp.co.uk/
> Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
> Past chairman of UKUUG: http://www.ukuug.org/
> #include <std_disclaimer.h>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-12 23:46 High IO Wait with RAID 1 Ryan Wagoner
2009-03-13 0:48 ` Alain Williams
@ 2009-03-13 14:48 ` John Robinson
1 sibling, 0 replies; 14+ messages in thread
From: John Robinson @ 2009-03-13 14:48 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: linux-raid
On 12/03/2009 23:46, Ryan Wagoner wrote:
[...]
> Both systems exhibit the same issues on the RAID 1 drives. That rules
> out the drive brand and controller card. During any IO intensive
> process the IO wait will raise and the system load will climb.
I'm sorry if this sounds flippant, but what did you expect?
> I've
> had the IO wait as high as 70% and the load at 13+ while migrating a
> vmdk file with vmware-vdiskmanager. You can easily recreate the issue
> with bonnie++.
>
> I can perform the same disk intensive operation on the RAID 5 array
> with almost no io wait or load. What is the deal with this? Is there
> something I can tweak?
I suspect your RAID-5 array has higher thoughput available - reading and
writing across 3 discs rather than 2 - and what you're doing isn't quite
maxing out its throughput, whereas it does on the 2-disc RAID-1. I
imagine you would see the same thing on a single disc.
Bear in mind that iowait time needn't be wasted; run a few copies of
something CPU intensive at the same time and you will see zero iowait.
High system load isn't necessarily a problem either; I once had a busy
web server where the system load went from about 5 one day to over 200
the next. Cue much panicking trying to find out what had happened.
Turned out the web designers had made the home page 70K instead of 60K,
so a lot of httpd processes sending to slow links were stalled when
previously the whole lot would have been in network buffers. It still
worked fine in that there was no detectable performance problem, it was
just an alarming-looking system load.
Cheers,
John.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 3:21 ` Ryan Wagoner
2009-03-13 9:39 ` Robin Hill
2009-03-13 10:17 ` Alain Williams
@ 2009-03-13 16:22 ` Bill Davidsen
2009-03-13 17:42 ` Ryan Wagoner
2 siblings, 1 reply; 14+ messages in thread
From: Bill Davidsen @ 2009-03-13 16:22 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: Alain Williams, linux-raid
Ryan Wagoner wrote:
> I'm glad I'm not the only one experiencing the issue. Luckily the
> issues on both my systems aren't as bad. I don't have any errors
> showing in /var/log/messages on either system. I've been trying to
> track down this issue for about a year now. I just recently my the
> connection with RAID 1 and mdadm when copying data on the second
> system.
>
> Unfortunately it looks like the fix is to avoid software RAID 1. I
> prefer software RAID over hardware RAID on my home systems for the
> flexibility it offers, especially since I can easily move the disks
> between systems in the case of hardware failure.
>
> If I can find time to migrate the VMs, which run my web sites and
> email to another machine, I'll reinstall the one system utilizing RAID
> 1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
> just pass the remaining disks through.
>
> You would think that software RAID 1 would be much simpler to
> implement than RAID 5 performance wise.
>
> Ryan
>
> On Thu, Mar 12, 2009 at 7:48 PM, Alain Williams <addw@phcomp.co.uk> wrote:
>
>> On Thu, Mar 12, 2009 at 06:46:45PM -0500, Ryan Wagoner wrote:
>>
>>> From what I can tell the issue here lies with mdadm and/or its
>>> interaction with CentOS 5.2. Let me first go over the configuration of
>>> both systems.
>>>
>>> System 1 - CentOS 5.2 x86_64
>>> 2x Seagate 7200.9 160GB in RAID 1
>>> 2x Seagate 7200.10 320GB in RAID 1
>>> 3x Hitachi Deskstar 7K1000 1TB in RAID 5
>>> All attached to Supermicro LSI 1068 PCI Express controller
>>>
>>> System 2 - CentOS 5.2 x86
>>> 1x Non Raid System Drive
>>> 2x Hitachi Deskstart 7K1000 1TB in RAID 1
>>> Attached to onboard ICH controller
>>>
>>> Both systems exhibit the same issues on the RAID 1 drives. That rules
>>> out the drive brand and controller card. During any IO intensive
>>> process the IO wait will raise and the system load will climb. I've
>>> had the IO wait as high as 70% and the load at 13+ while migrating a
>>> vmdk file with vmware-vdiskmanager. You can easily recreate the issue
>>> with bonnie++.
>>>
>> I suspect that the answer is 'no', however I am seeing problems with raid 1
>> on CentOS 5.2 x86_64. The system worked nicely for some 2 months, then apparently
>> a disk died and it's mirror appeared to have problems before the first could be
>> replaced. The motherboard & both disks have now been replaced (data saved with a bit
>> of luck & juggling). I have been assuming hardware, but there seems little else
>> to change... and you report long I/O waits that I saw and still see
>> (even when I don't see the kernel error messages below).
>>
>> Disks have been Seagate & Samsung, but now both ST31000333AS (1TB) as raid 1.
>> Adaptec AIC7902 Ultra320 SCSI adapter
>> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
>>
>> Executing 'w' or 'cat /proc/mdstat' can take several seconds,
>> failing sdb with mdadm and system performance becomes great again.
>>
>> I am seeing this sort of thing in /var/log/messages:
>> Mar 12 09:21:58 BFPS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
>> Mar 12 09:21:58 BFPS kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>> Mar 12 09:21:58 BFPS kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
>> Mar 12 09:21:58 BFPS kernel: ata2.00: status: { DRDY }
>> Mar 12 09:22:03 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xd0)
>> Mar 12 09:22:08 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
>> Mar 12 09:22:08 BFPS kernel: ata2: hard resetting link
>> Mar 12 09:22:08 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Mar 12 09:22:39 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>> Mar 12 09:22:39 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
>> Mar 12 09:22:39 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>> Mar 12 09:22:39 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
>> Mar 12 09:22:44 BFPS kernel: ata2: hard resetting link
>> Mar 12 09:24:02 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>> Mar 12 09:24:06 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
>> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
>> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>> Mar 12 09:24:06 BFPS kernel: ata2.00: disabled
>> Mar 12 09:24:06 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xff)
>> Mar 12 09:24:06 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
>> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
>> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Mar 12 09:24:06 BFPS kernel: ata2: EH complete
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 1953519821
>> Mar 12 09:24:06 BFPS kernel: raid1: Disk failure on sdb2, disabling device.
>> Mar 12 09:24:06 BFPS kernel: Operation continuing on 1 devices
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975018957
>> Mar 12 09:24:06 BFPS kernel: md: md3: sync done.
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975019981
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975021005
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022029
>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code = 0x00040000
>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector 975022157
>> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
>> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
>> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
>> Mar 12 09:24:06 BFPS kernel: disk 1, wo:1, o:0, dev:sdb2
>> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
>> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
>> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
>>
>> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, not capable of SMART self-check
>> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
>> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
>> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, failed to read SMART Attribute Data
>> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
>> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
>>
Part of the issue with software RAID is that when you do two writes, be
it mirrors or CRC, you actually have to push the data over the system
bus to the controller. With hardware RAID you push it to the controller,
freeing the bus.
"But wait, there's more" because the controller on a motherboard may not
have enough cache to hold the i/o, may not optimize the access, etc,
etc. And worst case it may write one drive then the other (serial
access) instead of writing both at once. So performance may not be a bit
better with the motherboard controller, and even an independent
controller may not help much under load, since the limits here are the
memory for cache (software has more) and the smartness of the write
decisions (software is usually at least as good).
Going to hardware RAID buys only one thing in my experience, and that is
it works at boot time, before the kernel gets into memory.
The other issue is that hardware RAID is per-disk, while software can
allow selection of multiple RAID types by partition, allowing arrays to
match use when that's appropriate.
--
Bill Davidsen <davidsen@tmr.com>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 16:22 ` Bill Davidsen
@ 2009-03-13 17:42 ` Ryan Wagoner
2009-03-13 18:37 ` David Rees
0 siblings, 1 reply; 14+ messages in thread
From: Ryan Wagoner @ 2009-03-13 17:42 UTC (permalink / raw)
To: Bill Davidsen; +Cc: Alain Williams, linux-raid
Yeah I understand the basics to RAID and the effect cache has on
performance. It just seems that RAID 1 should offer better write
performance than a 3 drive RAID 5 array. However I haven't run the
numbers so I could be wrong.
It could be just that I expect too much from RAID 1. I'm debating
about reloading the box with RAID 10 across 160GB of the 4 drives
(160GB and 320GB) and a mirror on the remaining space. In theory this
should gain me write performance.
Thanks,
Ryan
On Fri, Mar 13, 2009 at 11:22 AM, Bill Davidsen <davidsen@tmr.com> wrote:
> Ryan Wagoner wrote:
>>
>> I'm glad I'm not the only one experiencing the issue. Luckily the
>> issues on both my systems aren't as bad. I don't have any errors
>> showing in /var/log/messages on either system. I've been trying to
>> track down this issue for about a year now. I just recently my the
>> connection with RAID 1 and mdadm when copying data on the second
>> system.
>>
>> Unfortunately it looks like the fix is to avoid software RAID 1. I
>> prefer software RAID over hardware RAID on my home systems for the
>> flexibility it offers, especially since I can easily move the disks
>> between systems in the case of hardware failure.
>>
>> If I can find time to migrate the VMs, which run my web sites and
>> email to another machine, I'll reinstall the one system utilizing RAID
>> 1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
>> just pass the remaining disks through.
>>
>> You would think that software RAID 1 would be much simpler to
>> implement than RAID 5 performance wise.
>>
>> Ryan
>>
>> On Thu, Mar 12, 2009 at 7:48 PM, Alain Williams <addw@phcomp.co.uk> wrote:
>>
>>>
>>> On Thu, Mar 12, 2009 at 06:46:45PM -0500, Ryan Wagoner wrote:
>>>
>>>>
>>>> From what I can tell the issue here lies with mdadm and/or its
>>>> interaction with CentOS 5.2. Let me first go over the configuration of
>>>> both systems.
>>>>
>>>> System 1 - CentOS 5.2 x86_64
>>>> 2x Seagate 7200.9 160GB in RAID 1
>>>> 2x Seagate 7200.10 320GB in RAID 1
>>>> 3x Hitachi Deskstar 7K1000 1TB in RAID 5
>>>> All attached to Supermicro LSI 1068 PCI Express controller
>>>>
>>>> System 2 - CentOS 5.2 x86
>>>> 1x Non Raid System Drive
>>>> 2x Hitachi Deskstart 7K1000 1TB in RAID 1
>>>> Attached to onboard ICH controller
>>>>
>>>> Both systems exhibit the same issues on the RAID 1 drives. That rules
>>>> out the drive brand and controller card. During any IO intensive
>>>> process the IO wait will raise and the system load will climb. I've
>>>> had the IO wait as high as 70% and the load at 13+ while migrating a
>>>> vmdk file with vmware-vdiskmanager. You can easily recreate the issue
>>>> with bonnie++.
>>>>
>>>
>>> I suspect that the answer is 'no', however I am seeing problems with raid
>>> 1
>>> on CentOS 5.2 x86_64. The system worked nicely for some 2 months, then
>>> apparently
>>> a disk died and it's mirror appeared to have problems before the first
>>> could be
>>> replaced. The motherboard & both disks have now been replaced (data saved
>>> with a bit
>>> of luck & juggling). I have been assuming hardware, but there seems
>>> little else
>>> to change... and you report long I/O waits that I saw and still see
>>> (even when I don't see the kernel error messages below).
>>>
>>> Disks have been Seagate & Samsung, but now both ST31000333AS (1TB) as
>>> raid 1.
>>> Adaptec AIC7902 Ultra320 SCSI adapter
>>> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
>>>
>>> Executing 'w' or 'cat /proc/mdstat' can take several seconds,
>>> failing sdb with mdadm and system performance becomes great again.
>>>
>>> I am seeing this sort of thing in /var/log/messages:
>>> Mar 12 09:21:58 BFPS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr
>>> 0x0 action 0x2 frozen
>>> Mar 12 09:21:58 BFPS kernel: ata2.00: cmd
>>> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>>> Mar 12 09:21:58 BFPS kernel: res
>>> 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
>>> Mar 12 09:21:58 BFPS kernel: ata2.00: status: { DRDY }
>>> Mar 12 09:22:03 BFPS kernel: ata2: port is slow to respond, please be
>>> patient (Status 0xd0)
>>> Mar 12 09:22:08 BFPS kernel: ata2: device not ready (errno=-16), forcing
>>> hardreset
>>> Mar 12 09:22:08 BFPS kernel: ata2: hard resetting link
>>> Mar 12 09:22:08 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123
>>> SControl 300)
>>> Mar 12 09:22:39 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>>> Mar 12 09:22:39 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error,
>>> err_mask=0x5)
>>> Mar 12 09:22:39 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>>> Mar 12 09:22:39 BFPS kernel: ata2: failed to recover some devices,
>>> retrying in 5 secs
>>> Mar 12 09:22:44 BFPS kernel: ata2: hard resetting link
>>> Mar 12 09:24:02 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123
>>> SControl 300)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error,
>>> err_mask=0x5)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>>> Mar 12 09:24:06 BFPS kernel: ata2: failed to recover some devices,
>>> retrying in 5 secs
>>> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
>>> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123
>>> SControl 300)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error,
>>> err_mask=0x5)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: revalidation failed (errno=-5)
>>> Mar 12 09:24:06 BFPS kernel: ata2.00: disabled
>>> Mar 12 09:24:06 BFPS kernel: ata2: port is slow to respond, please be
>>> patient (Status 0xff)
>>> Mar 12 09:24:06 BFPS kernel: ata2: device not ready (errno=-16), forcing
>>> hardreset
>>> Mar 12 09:24:06 BFPS kernel: ata2: hard resetting link
>>> Mar 12 09:24:06 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123
>>> SControl 300)
>>> Mar 12 09:24:06 BFPS kernel: ata2: EH complete
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 1953519821
>>> Mar 12 09:24:06 BFPS kernel: raid1: Disk failure on sdb2, disabling
>>> device.
>>> Mar 12 09:24:06 BFPS kernel: Operation continuing on 1 devices
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 975018957
>>> Mar 12 09:24:06 BFPS kernel: md: md3: sync done.
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 975019981
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 975021005
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 975022029
>>> Mar 12 09:24:06 BFPS kernel: sd 1:0:0:0: SCSI error: return code =
>>> 0x00040000
>>> Mar 12 09:24:06 BFPS kernel: end_request: I/O error, dev sdb, sector
>>> 975022157
>>> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
>>> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
>>> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
>>> Mar 12 09:24:06 BFPS kernel: disk 1, wo:1, o:0, dev:sdb2
>>> Mar 12 09:24:06 BFPS kernel: RAID1 conf printout:
>>> Mar 12 09:24:06 BFPS kernel: --- wd:1 rd:2
>>> Mar 12 09:24:06 BFPS kernel: disk 0, wo:0, o:1, dev:sda2
>>>
>>> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, not capable of SMART
>>> self-check
>>> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
>>> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
>>> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, failed to read SMART
>>> Attribute Data
>>> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
>>> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
>>>
>
> Part of the issue with software RAID is that when you do two writes, be it
> mirrors or CRC, you actually have to push the data over the system bus to
> the controller. With hardware RAID you push it to the controller, freeing
> the bus.
>
> "But wait, there's more" because the controller on a motherboard may not
> have enough cache to hold the i/o, may not optimize the access, etc, etc.
> And worst case it may write one drive then the other (serial access) instead
> of writing both at once. So performance may not be a bit better with the
> motherboard controller, and even an independent controller may not help much
> under load, since the limits here are the memory for cache (software has
> more) and the smartness of the write decisions (software is usually at least
> as good).
>
> Going to hardware RAID buys only one thing in my experience, and that is it
> works at boot time, before the kernel gets into memory.
>
> The other issue is that hardware RAID is per-disk, while software can allow
> selection of multiple RAID types by partition, allowing arrays to match use
> when that's appropriate.
>
> --
> Bill Davidsen <davidsen@tmr.com>
> "Woe unto the statesman who makes war without a reason that will still
> be valid when the war is over..." Otto von Bismark
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 17:42 ` Ryan Wagoner
@ 2009-03-13 18:37 ` David Rees
0 siblings, 0 replies; 14+ messages in thread
From: David Rees @ 2009-03-13 18:37 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: Bill Davidsen, Alain Williams, linux-raid
On Fri, Mar 13, 2009 at 10:42 AM, Ryan Wagoner <rswagoner@gmail.com> wrote:
> Yeah I understand the basics to RAID and the effect cache has on
> performance. It just seems that RAID 1 should offer better write
> performance than a 3 drive RAID 5 array. However I haven't run the
> numbers so I could be wrong.
If we just look at best case scenario write throughput where N is the
number of disks in the array:
RAID1 = 1
RAID5 = N-1
So for your case, your RAID5 array should be able to write up to twice
as fast as your RAID1 array. This could be enough to explain the
difference in load.
Where RAID5 writes slow down significantly is if you are writing
chunks smaller than the stripe size and the affected stripe isn't in
cache and a read has to be performed before the write to recalculate
the parity.
But for your use case of transferring large VM images, this is
unlikely to be the case.
In otherwords, what you are experiencing (high IO wait writing to the
RAID1 arrays) is to be expected. From the stats you posted earlier,
it looks like you are maxing out the write speed of the RAID1 arrays -
thus you end up with high IO wait times.
> It could be just that I expect too much from RAID 1. I'm debating
> about reloading the box with RAID 10 across 160GB of the 4 drives
> (160GB and 320GB) and a mirror on the remaining space. In theory this
> should gain me write performance.
Yes, a RAID10 with 4 disks will get you up to twice the write
performance and up to 4 times the read performance without the
drawback of small writes losing performance due to parity reads.
-Dave
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 0:48 ` Alain Williams
2009-03-13 3:21 ` Ryan Wagoner
@ 2009-03-13 18:42 ` David Rees
1 sibling, 0 replies; 14+ messages in thread
From: David Rees @ 2009-03-13 18:42 UTC (permalink / raw)
To: Alain Williams; +Cc: Ryan Wagoner, linux-raid
On Thu, Mar 12, 2009 at 5:48 PM, Alain Williams <addw@phcomp.co.uk> wrote:
> I suspect that the answer is 'no', however I am seeing problems with raid 1
> on CentOS 5.2 x86_64. The system worked nicely for some 2 months, then apparently
> a disk died and it's mirror appeared to have problems before the first could be
> replaced. The motherboard & both disks have now been replaced (data saved with a bit
> of luck & juggling). I have been assuming hardware, but there seems little else
> to change... and you report long I/O waits that I saw and still see
> (even when I don't see the kernel error messages below).
>
> Disks have been Seagate & Samsung, but now both ST31000333AS (1TB) as raid 1.
> Adaptec AIC7902 Ultra320 SCSI adapter
> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
Uh, how are you using SATA disks on a SCSI controller?
> I am seeing this sort of thing in /var/log/messages:
> Mar 12 09:21:58 BFPS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
> Mar 12 09:21:58 BFPS kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> Mar 12 09:21:58 BFPS kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> Mar 12 09:21:58 BFPS kernel: ata2.00: status: { DRDY }
> Mar 12 09:22:03 BFPS kernel: ata2: port is slow to respond, please be patient (Status 0xd0)
> Mar 12 09:22:08 BFPS kernel: ata2: device not ready (errno=-16), forcing hardreset
> Mar 12 09:22:08 BFPS kernel: ata2: hard resetting link
> Mar 12 09:22:08 BFPS kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Mar 12 09:22:39 BFPS kernel: ata2.00: qc timeout (cmd 0xec)
> Mar 12 09:22:39 BFPS kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x5)
> Mar 12 09:22:39 BFPS kernel: ata2.00: revalidation failed (errno=-5)
> Mar 12 09:22:39 BFPS kernel: ata2: failed to recover some devices, retrying in 5 secs
> Mar 12 09:22:44 BFPS kernel: ata2: hard resetting link
You are experiencing one of the following:
Drive failure
Cabling failure
Controller failure
Buggy drive firmware
Buggy drivers
BTW, the ST31000333AS is one of the drives that were shipped with
buggy firmware causing timeouts like the ones you are seeing. You
should check with Seagate that you have the latest firmware flashed to
those drives.
> Mar 12 09:28:07 BFPS smartd[3183]: Device: /dev/sdb, not capable of SMART self-check
> Mar 12 09:28:07 BFPS smartd[3183]: Sending warning via mail to root ...
> Mar 12 09:28:07 BFPS smartd[3183]: Warning via mail to root: successful
You really need to get SMART checking working as well. Checking the
status of the drive (try smartctl) may help you isolate your problem.
Which, BTW, does not appear to be the same as the thread starter's.
-Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 18:29 ` Ryan Wagoner
@ 2009-03-13 22:10 ` David Rees
0 siblings, 0 replies; 14+ messages in thread
From: David Rees @ 2009-03-13 22:10 UTC (permalink / raw)
To: Ryan Wagoner; +Cc: David Lethe, Bill Davidsen, Alain Williams, linux-raid
On Fri, Mar 13, 2009 at 11:29 AM, Ryan Wagoner <rswagoner@gmail.com> wrote:
> The other system with the onboard ICH controller exhibits the same
> symptoms so I think my card is configured correctly. The interesting
> part of this is I can kick off a resync on all three RAID volumes and
> the system load and io wait is low. The rebuild rate is 85M/s for the
> RAID 5 volume and 65M/s for the RAID 1 volumes, which is the max each
> individual drive can do.
The load from rebuilding the raid doesn't show up in system load or io
wait as it's done in-kernel. If you run some other IO intensive
processes during the rebuild you should notice that it will run a bit
slower and io wait should be higher.
-Dave
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
2009-03-13 18:02 David Lethe
@ 2009-03-13 18:29 ` Ryan Wagoner
2009-03-13 22:10 ` David Rees
0 siblings, 1 reply; 14+ messages in thread
From: Ryan Wagoner @ 2009-03-13 18:29 UTC (permalink / raw)
To: David Lethe; +Cc: Bill Davidsen, Alain Williams, linux-raid
The card has the latest non RAID firmware loaded on it. The LSI 1068
model comes default from Supermicro with the non RAID firmware. It
only has an ARM processor capable of RAID 0 or 1 if you load the RAID
firmware.
The other system with the onboard ICH controller exhibits the same
symptoms so I think my card is configured correctly. The interesting
part of this is I can kick off a resync on all three RAID volumes and
the system load and io wait is low. The rebuild rate is 85M/s for the
RAID 5 volume and 65M/s for the RAID 1 volumes, which is the max each
individual drive can do.
Ryan
On Fri, Mar 13, 2009 at 1:02 PM, David Lethe <david@santools.com> wrote:
> -----Original Message-----
>
> From: "Ryan Wagoner" <rswagoner@gmail.com>
> Subj: Re: High IO Wait with RAID 1
> Date: Fri Mar 13, 2009 12:45 pm
> Size: 2K
> To: "Bill Davidsen" <davidsen@tmr.com>
> cc: "Alain Williams" <addw@phcomp.co.uk>; "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
>
> Yeah I understand the basics to RAID and the effect cache has on
> performance. It just seems that RAID 1 should offer better write
> performance than a 3 drive RAID 5 array. However I haven't run the
> numbers so I could be wrong.
>
> It could be just that I expect too much from RAID 1. I'm debating
> about reloading the box with RAID 10 across 160GB of the 4 drives
> (160GB and 320GB) and a mirror on the remaining space. In theory this
> should gain me write performance.
>
> Thanks,
> Ryan
>
> On Fri, Mar 13, 2009 at 11:22 AM, Bill Davidsen <davidsen@tmr.com> wrote:
>> Ryan Wagoner wrote:
>>>
>>> I'm glad I'm not the only one experiencing the issue. Luckily the
>>> issues on both my systems aren't as bad. I don't have any errors
>>> showing in /var/log/messages on either system. I've been trying to
>>> track down this issue for about a year now. I just recently my the
>>> connection with RAID 1 and mdadm when copying data on the second
>>> system.
>>>
>>> Unfortunately it looks like the fix is to avoid software RAID 1. I
>>> prefer software RAID over hardware RAID on my home systems for the
>>> flexibility it offers, especially since I can easily move the disks
>>> between systems in the case of hardware failure.
>>>
>>> If I can find time to migrate the VMs, which run my web sites and
>>> email to another machine, I'll reinstall the one system utilizing RAID
>>> 1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
>>> just pass the remaining disks through.
>>>
>
> FYi - you can potentially get a big performance penalty when running a LSI raid card in jbod mode. The impact varies depending on a lot of things .. Try loading the jbod fimware on the card if it supports this and re run benchmarks
>
> david
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: High IO Wait with RAID 1
@ 2009-03-13 18:02 David Lethe
2009-03-13 18:29 ` Ryan Wagoner
0 siblings, 1 reply; 14+ messages in thread
From: David Lethe @ 2009-03-13 18:02 UTC (permalink / raw)
To: Ryan Wagoner, Bill Davidsen; +Cc: Alain Williams, linux-raid
-----Original Message-----
From: "Ryan Wagoner" <rswagoner@gmail.com>
Subj: Re: High IO Wait with RAID 1
Date: Fri Mar 13, 2009 12:45 pm
Size: 2K
To: "Bill Davidsen" <davidsen@tmr.com>
cc: "Alain Williams" <addw@phcomp.co.uk>; "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
Yeah I understand the basics to RAID and the effect cache has on
performance. It just seems that RAID 1 should offer better write
performance than a 3 drive RAID 5 array. However I haven't run the
numbers so I could be wrong.
It could be just that I expect too much from RAID 1. I'm debating
about reloading the box with RAID 10 across 160GB of the 4 drives
(160GB and 320GB) and a mirror on the remaining space. In theory this
should gain me write performance.
Thanks,
Ryan
On Fri, Mar 13, 2009 at 11:22 AM, Bill Davidsen <davidsen@tmr.com> wrote:
> Ryan Wagoner wrote:
>>
>> I'm glad I'm not the only one experiencing the issue. Luckily the
>> issues on both my systems aren't as bad. I don't have any errors
>> showing in /var/log/messages on either system. I've been trying to
>> track down this issue for about a year now. I just recently my the
>> connection with RAID 1 and mdadm when copying data on the second
>> system.
>>
>> Unfortunately it looks like the fix is to avoid software RAID 1. I
>> prefer software RAID over hardware RAID on my home systems for the
>> flexibility it offers, especially since I can easily move the disks
>> between systems in the case of hardware failure.
>>
>> If I can find time to migrate the VMs, which run my web sites and
>> email to another machine, I'll reinstall the one system utilizing RAID
>> 1 on the LSI controller. It doesn't support RAID 5 so I'm hoping I can
>> just pass the remaining disks through.
>>
FYi - you can potentially get a big performance penalty when running a LSI raid card in jbod mode. The impact varies depending on a lot of things .. Try loading the jbod fimware on the card if it supports this and re run benchmarks
david
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2009-03-13 22:10 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-03-12 23:46 High IO Wait with RAID 1 Ryan Wagoner
2009-03-13 0:48 ` Alain Williams
2009-03-13 3:21 ` Ryan Wagoner
2009-03-13 9:39 ` Robin Hill
2009-03-13 10:17 ` Alain Williams
[not found] ` <7d86ddb90903130519p4268dc33vc8ad42b53aefa2e2@mail.gmail.com>
2009-03-13 12:21 ` Fwd: " Ryan Wagoner
2009-03-13 16:22 ` Bill Davidsen
2009-03-13 17:42 ` Ryan Wagoner
2009-03-13 18:37 ` David Rees
2009-03-13 18:42 ` David Rees
2009-03-13 14:48 ` John Robinson
2009-03-13 18:02 David Lethe
2009-03-13 18:29 ` Ryan Wagoner
2009-03-13 22:10 ` David Rees
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.