disk performance strange/low

disk performance strange/low

[Michael Monnerie]

I've got an Areca RAID Controller with 5x 1TB disks, and in the last 
time I often found it to be strange. Now I investigated, and found this. 
>From another server I NFS mount a share, and make
dd if=/dev/sda of=/server/share/test.dd bs=1024k
so that should do big sequential writes. But the performance goes up and 
down, like you see in this output from "iostat -kx 5 555". You see write 
KB/s is 5824, 23207, 87283, 704, 40187. Up and down all the time. Could 
that be solved by the patch you discussed last time, or is there 
something else?

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-
sz avgqu-sz   await  svctm  %util
sdb               0,00   401,20    1,20   77,09     4,78  5824,00   
148,91   152,25 2595,77  12,76  99,92

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00   34,77   67,03    2,20    7,21
sdb               0,00  1094,40    2,20  245,80     8,80 23207,20   
187,23   153,64   43,24   4,03 100,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00   41,08   56,83    0,30   15,65
sdb               0,00   295,20    6,40  791,20    25,60 87283,40   
218,93   119,19  259,89   1,21  96,56

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00   61,45   55,92    0,50    4,62
sdb               0,00     3,60    0,00    4,60     0,00   704,80   
306,43   140,01 2979,30 217,39 100,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00   21,12   75,50    0,10   10,06
sdb               0,00  1252,80    0,20  270,00     0,80 40187,20   
297,47   151,38 1160,49   3,70  99,84


mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: disk performance strange/low

[Christoph Hellwig]

On Mon, Aug 17, 2009 at 11:11:07PM +0200, Michael Monnerie wrote:
> I've got an Areca RAID Controller with 5x 1TB disks, and in the last 
> time I often found it to be strange. Now I investigated, and found this. 
> >From another server I NFS mount a share, and make
> dd if=/dev/sda of=/server/share/test.dd bs=1024k
> so that should do big sequential writes. But the performance goes up and 
> down, like you see in this output from "iostat -kx 5 555". You see write 
> KB/s is 5824, 23207, 87283, 704, 40187. Up and down all the time. Could 
> that be solved by the patch you discussed last time, or is there 
> something else?

Yes, that sounds like the problem of the VM not feeding us enough
pages to write.  What kernel is this on?  Latest 2.6.31-rc has a
workaround for it.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: disk performance strange/low

[Michael Monnerie]

On Montag 17 August 2009 Christoph Hellwig wrote:
> Yes, that sounds like the problem of the VM not feeding us enough
> pages to write.  What kernel is this on?  Latest 2.6.31-rc has a
> workaround for it.

2.6.27.23-0.1-xen from openSUSE 11.1
I can't switch as I use it with the free Xen. Thanks for the info.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: disk performance strange/low

[Christoph Hellwig]

On Tue, Aug 18, 2009 at 04:08:05AM +0200, Michael Monnerie wrote:
> On Montag 17 August 2009 Christoph Hellwig wrote:
> > Yes, that sounds like the problem of the VM not feeding us enough
> > pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a
> > workaround for it.
> 
> 2.6.27.23-0.1-xen from openSUSE 11.1
> I can't switch as I use it with the free Xen. Thanks for the info.

Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from
mainline - it should apply just fine to a 2.6.27-ish kernel.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: disk performance strange/low

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 1031 bytes --]

On Dienstag 18 August 2009 Christoph Hellwig wrote:
> On Tue, Aug 18, 2009 at 04:08:05AM +0200, Michael Monnerie wrote:
> > On Montag 17 August 2009 Christoph Hellwig wrote:
> > > Yes, that sounds like the problem of the VM not feeding us enough
> > > pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a
> > > workaround for it.
> >
> > 2.6.27.23-0.1-xen from openSUSE 11.1
> > I can't switch as I use it with the free Xen. Thanks for the info.
>
> Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from
> mainline - it should apply just fine to a 2.6.27-ish kernel.

Thank you. I'm not a patcher, could you tell me how to do that?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: disk performance strange/low

[Christoph Hellwig]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Wed, Aug 26, 2009 at 07:20:36PM +0200, Michael Monnerie wrote:
> On Dienstag 18 August 2009 Christoph Hellwig wrote:
> > On Tue, Aug 18, 2009 at 04:08:05AM +0200, Michael Monnerie wrote:
> > > On Montag 17 August 2009 Christoph Hellwig wrote:
> > > > Yes, that sounds like the problem of the VM not feeding us enough
> > > > pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a
> > > > workaround for it.
> > >
> > > 2.6.27.23-0.1-xen from openSUSE 11.1
> > > I can't switch as I use it with the free Xen. Thanks for the info.
> >
> > Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from
> > mainline - it should apply just fine to a 2.6.27-ish kernel.
> 
> Thank you. I'm not a patcher, could you tell me how to do that?

Just take the patch I attached and apply it using

	patch -p1 < xfs-nr-to-write.diff

if you want to make sure it applies cleanly do a

	patch -p1 --dry-run < xfs-nr-to-write.diff

beforee to check if it doesn't cause any reject.

After that rebuild your kernel.

[-- Attachment #2: xfs-nr-to-write.diff --]
[-- Type: text/plain, Size: 1239 bytes --]

commit c8a4051c3731b6db224482218cfd535ab9393ff8
Author: Eric Sandeen <sandeen@sandeen.net>
Date:   Fri Jul 31 00:02:17 2009 -0500

    xfs: bump up nr_to_write in xfs_vm_writepage
    
    VM calculation for nr_to_write seems off.  Bump it way
    up, this gets simple streaming writes zippy again.
    To be reviewed again after Jens' writeback changes.
    
    Signed-off-by: Christoph Hellwig <hch@infradead.org>
    Signed-off-by: Eric Sandeen <sandeen@sandeen.net>
    Cc: Chris Mason <chris.mason@oracle.com>
    Reviewed-by: Felix Blyakher <felixb@sgi.com>
    Signed-off-by: Felix Blyakher <felixb@sgi.com>

diff --git a/fs/xfs/linux-2.6/xfs_aops.c b/fs/xfs/linux-2.6/xfs_aops.c
index 7ec89fc..aecf251 100644
--- a/fs/xfs/linux-2.6/xfs_aops.c
+++ b/fs/xfs/linux-2.6/xfs_aops.c
@@ -1268,6 +1268,14 @@ xfs_vm_writepage(
 	if (!page_has_buffers(page))
 		create_empty_buffers(page, 1 << inode->i_blkbits, 0);
 
+
+	/*
+	 *  VM calculation for nr_to_write seems off.  Bump it way
+	 *  up, this gets simple streaming writes zippy again.
+	 *  To be reviewed again after Jens' writeback changes.
+	 */
+	wbc->nr_to_write *= 4;
+
 	/*
 	 * Convert delayed allocate, unwritten or unmapped space
 	 * to real space and flush out to disk.

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs