RAID-0 read perf. decrease after 2.4.20

RAID-0 read perf. decrease after 2.4.20

[Mikael Johansson]

Hello All!

Has anyone else experienced a drastic drop in read performance on software
RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
lab with double IDE disks on different channels set up as RAID-0. Some
bonnie++ benchmark results with various kernels, on the same machine
(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
6Y060L0):
		write	read
2.4.20-ac1:	88,000 	135,000 K/sec
2.4.21-pre7:	93,000   75,000
2.4.22-ac4:	94,000	 82,000

So the write speed has gone up a bit, but the read speed performance has
plunged. Any ideas on what happened between 2.4.20 and 2.4.21 and what to
do about it? I'm eagerly awaiting suggestions, good read speed is quite
critical for many of our calculations :-) I will of course provide more
details if necessary.

Have a nice day,
    Mikael J.
    http://www.helsinki.fi/~mpjohans/

Re: RAID-0 read perf. decrease after 2.4.20

[Marcelo Tosatti]

On Thu, 20 Nov 2003, Mikael Johansson wrote:

> 
> Hello All!
> 
> Has anyone else experienced a drastic drop in read performance on software
> RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
> lab with double IDE disks on different channels set up as RAID-0. Some
> bonnie++ benchmark results with various kernels, on the same machine
> (Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
> 6Y060L0):
> 		write	read
> 2.4.20-ac1:	88,000 	135,000 K/sec
> 2.4.21-pre7:	93,000   75,000
> 2.4.22-ac4:	94,000	 82,000
> 
> So the write speed has gone up a bit, but the read speed performance has
> plunged. Any ideas on what happened between 2.4.20 and 2.4.21 and what to
> do about it? I'm eagerly awaiting suggestions, good read speed is quite
> critical for many of our calculations :-) I will of course provide more
> details if necessary.

There have been no significant changes in the RAID driver in 2.4.21, so I
suspect the cause for the slowdowns might the changes to the IO scheduler
(ll_rw_blk.c) or VM changes.

Isolating the -pre which the slowdown starts helps a lot.

Re: RAID-0 read perf. decrease after 2.4.20

[Mikael Johansson]

Hello Marcelo and All!

On Fri, 21 Nov 2003, Marcelo Tosatti wrote:

> There have been no significant changes in the RAID driver in 2.4.21, so I
> suspect the cause for the slowdowns might the changes to the IO scheduler
> (ll_rw_blk.c) or VM changes.
>
> Isolating the -pre which the slowdown starts helps a lot.

I tested a few kernels to pin point the where tha changes occured. It
turned out that both 2.4.20 and 2.4.21-pre1 have bad read performance. The
2.4.20-ac's show good read speed. I tested it on two machines (different
from yesterday). I also checked the VIA chipset drivers version; that's
not the reason for the differences.

Athlon XP 2400+, 2.09 GHz, 1.5 GB RAM, 2*160 GB Maxtor Maxtor 6Y080L0
		VIA	write	read
2.4.19		none	10,000	  9,000 K/sec (chipset not supported)
2.4.20		3.35	73,000	 88,000
2.4.20-ac1	3.35-ac	70,000	135,000
2.4.20-ac2	3.35-ac	71,000	140,000
2.4.21-pre1	3.35-ac 71,000	 79,000
2.4.21-pre3	3.35-ac 71,000   79,000

Athlon 1.4 GHz, 1.5 GB RAM, 2*80 GB IC35L040AVER07-0 (IBM, I think)
2.4.13-ac8	?	49,000	 44,000 K/sec
2.4.19		3.34	53,000	 41,000
2.4.20-ac2	3.35-ac	50,000	 69,000
2.4.21-pre1	3.35-ac	53,000	 46,000

So there was apparently something very right with the 2.4.20-ac's. It
would be nice to get it back :-)

Have a nice weekend,
    Mikael J.
    http://www.helsinki.fi/~mpjohans/

Re: RAID-0 read perf. decrease after 2.4.20

[Marcelo Tosatti]

On Fri, 21 Nov 2003, Mikael Johansson wrote:

> 
> Hello Marcelo and All!

Hi Mikael,

> 
> On Fri, 21 Nov 2003, Marcelo Tosatti wrote:
> 
> > There have been no significant changes in the RAID driver in 2.4.21, so I
> > suspect the cause for the slowdowns might the changes to the IO scheduler
> > (ll_rw_blk.c) or VM changes.
> >
> > Isolating the -pre which the slowdown starts helps a lot.
> 
> I tested a few kernels to pin point the where tha changes occured. It
> turned out that both 2.4.20 and 2.4.21-pre1 have bad read performance. The
> 2.4.20-ac's show good read speed. I tested it on two machines (different
> from yesterday). I also checked the VIA chipset drivers version; that's
> not the reason for the differences.
> 
> Athlon XP 2400+, 2.09 GHz, 1.5 GB RAM, 2*160 GB Maxtor Maxtor 6Y080L0
> 		VIA	write	read
> 2.4.19		none	10,000	  9,000 K/sec (chipset not supported)
> 2.4.20		3.35	73,000	 88,000
> 2.4.20-ac1	3.35-ac	70,000	135,000
> 2.4.20-ac2	3.35-ac	71,000	140,000
> 2.4.21-pre1	3.35-ac 71,000	 79,000
> 2.4.21-pre3	3.35-ac 71,000   79,000
> 
> Athlon 1.4 GHz, 1.5 GB RAM, 2*80 GB IC35L040AVER07-0 (IBM, I think)
> 2.4.13-ac8	?	49,000	 44,000 K/sec
> 2.4.19		3.34	53,000	 41,000
> 2.4.20-ac2	3.35-ac	50,000	 69,000
> 2.4.21-pre1	3.35-ac	53,000	 46,000
> 
> So there was apparently something very right with the 2.4.20-ac's. It
> would be nice to get it back :-)

2.4.20-aa included rmap and some VM modifications most notably
"drop_behind()" logic which I believe should be the reason for the huge
read speedups. Can you please try it? Against 2.4.23.

--- mm/filemap.c	2003-12-08 10:26:58.000000000 -0200
+++ mm/filemap.c.orig	2003-12-08 10:24:57.000000000 -0200
@@ -1055,56 +1055,6 @@
 	return page;	
 }
 
-
- /* We combine this with read-ahead to deactivate pages when we
- * think there's sequential IO going on. Note that this is
- * harmless since we don't actually evict the pages from memory
- * but just move them to the inactive list.
- *
- * TODO:
- * - make the readahead code smarter
- * - move readahead to the VMA level so we can do the same
- *   trick with mmap()
- *
- * Rik van Riel, 2000
- */
-static void drop_behind(struct file * file, unsigned long index)
-{
-       struct inode *inode = file->f_dentry->d_inode;
-       struct address_space *mapping = inode->i_mapping;
-       struct page *page;
-       unsigned long start;
-
-       /* Nothing to drop-behind if we're on the first page. */
-       if (!index)
-               return;
-
-       if (index > file->f_rawin)
-               start = index - file->f_rawin;
-       else
-               start = 0;
-
-       /*
-        * Go backwards from index-1 and drop all pages in the
-        * readahead window. Since the readahead window may have
-        * been increased since the last time we were called, we
-        * stop when the page isn't there.
-        */
-       spin_lock(&pagemap_lru_lock);
-       while (--index >= start) {
-               struct page **hash = page_hash(mapping, index);
-               spin_lock(&pagecache_lock);
-               page = __find_page_nolock(mapping, index, *hash);
-               spin_unlock(&pagecache_lock);
-               if (!page || !PageActive(page))
-                       break;
-               drop_page(page);
-       }
-       spin_unlock(&pagemap_lru_lock);
-}
-
-
-
 /*
  * Same as grab_cache_page, but do not wait if the page is unavailable.
  * This is intended for speculative data generators, where the data can
@@ -1376,12 +1326,6 @@
 		if (filp->f_ramax > max_readahead)
 			filp->f_ramax = max_readahead;
 
-               /*
-                * Move the pages that have already been passed
-                * to the inactive list.
-                */
-               drop_behind(filp, index);
-
 #ifdef PROFILE_READAHEAD
 		profile_readahead((reada_ok == 2), filp);
 #endif

Re: RAID-0 read perf. decrease after 2.4.20

[Marc-Christian Petersen]

On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:

Hi Marcelo,

> 2.4.20-aa included rmap and some VM modifications most notably
> "drop_behind()" logic which I believe should be the reason for the huge
> read speedups. Can you please try it? Against 2.4.23.

-aa tree never had -rmap :)

ciao, Marc

Re: RAID-0 read perf. decrease after 2.4.20

[Marcelo Tosatti]

On Mon, 8 Dec 2003, Marc-Christian Petersen wrote:

> On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
> 
> Hi Marcelo,
> 
> > 2.4.20-aa included rmap and some VM modifications most notably
> > "drop_behind()" logic which I believe should be the reason for the huge
> > read speedups. Can you please try it? Against 2.4.23.
> 
> -aa tree never had -rmap :)

-ac I mean, sorry. 

Re: RAID-0 read perf. decrease after 2.4.20

[Mikael Johansson]

Hello again!

Finally had time to test this on IDE, "better late than even later"...

On Thu, 18 Dec 2003, Martin Knoblauch wrote:

> > >  Just some feedback:
> > >
> > > echo 511 > /proc/sys/vm/max-readahead
> > >
> > >  brings back the read performance of my 30 disks on 4 controller
> > > LVM/RAID0.
> >
> > Great.
> >
>
>  Indeed :-) Just to clarify - the modification of max-readahead was
> sufficient to "fix" the observed read performance degradation. I did
> not apply (or reverse) anything on top of 2.4.22.
>
>  Actually 255 would have been sufficient, 511 proved to be a small bit
> better :-))

The "max-readahead" does not seem to affect IDE-RAID, at least not with
the 2.4.27-pre6 I just compiled on two machines. So the problem still
exists, read speed on 2.4.20-ac2 is almost twice as fast as on later
kernels :-/

Have a nice summer,
    Mikael J.
    http://www.helsinki.fi/~mpjohans/

Re: RAID-0 read perf. decrease after 2.4.20

[Marcelo Tosatti]

On Thu, 18 Dec 2003, Mike Fedyk wrote:

> On Thu, Dec 18, 2003 at 11:41:11AM -0200, Marcelo Tosatti wrote:
> > On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> > >  Just some feedback:
> > >
> > > echo 511 > /proc/sys/vm/max-readahead
> > >
> > >  brings back the read performance of my 30 disks on 4 controller
> > > LVM/RAID0.
> >
> > Great.
>
> Maybe a new default is in order?

I believe the defaults have been changed because they harm smaller
workloads.

Re: RAID-0 read perf. decrease after 2.4.20

[Mike Fedyk]

On Thu, Dec 18, 2003 at 11:41:11AM -0200, Marcelo Tosatti wrote:
> On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> >  Just some feedback:
> > 
> > echo 511 > /proc/sys/vm/max-readahead
> > 
> >  brings back the read performance of my 30 disks on 4 controller
> > LVM/RAID0.
> 
> Great.

Maybe a new default is in order?

Re: RAID-0 read perf. decrease after 2.4.20

[Martin Knoblauch]

--- Marcelo Tosatti <marcelo.tosatti@cyclades.com> wrote:
> 
> 
> On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> 
> > On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
> > 
> > Hi Marcelo,
> > 
> > > 2.4.20-aa included rmap and some VM modifications most notably
> > > "drop_behind()" logic which I believe should be the reason for
> the
> > huge
> > > read speedups. Can you please try it? Against 2.4.23.
> > 
> >  Just some feedback:
> > 
> > echo 511 > /proc/sys/vm/max-readahead
> > 
> >  brings back the read performance of my 30 disks on 4 controller
> > LVM/RAID0.
> 
> Great.
> 

 Indeed :-) Just to clarify - the modification of max-readahead was
sufficient to "fix" the observed read performance degradation. I did
not apply (or reverse) anything on top of 2.4.22.

 Actually 255 would have been sufficient, 511 proved to be a small bit
better :-))

Martin

=====
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

Re: RAID-0 read perf. decrease after 2.4.20

[Marcelo Tosatti]

On Tue, 16 Dec 2003, Martin Knoblauch wrote:

> On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
> 
> Hi Marcelo,
> 
> > 2.4.20-aa included rmap and some VM modifications most notably
> > "drop_behind()" logic which I believe should be the reason for the
> huge
> > read speedups. Can you please try it? Against 2.4.23.
> 
>  Just some feedback:
> 
> echo 511 > /proc/sys/vm/max-readahead
> 
>  brings back the read performance of my 30 disks on 4 controller
> LVM/RAID0.

Great.

Re: RAID-0 read perf. decrease after 2.4.20

[Martin Knoblauch]

On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:

Hi Marcelo,

> 2.4.20-aa included rmap and some VM modifications most notably
> "drop_behind()" logic which I believe should be the reason for the
huge
> read speedups. Can you please try it? Against 2.4.23.

 Just some feedback:

echo 511 > /proc/sys/vm/max-readahead

 brings back the read performance of my 30 disks on 4 controller
LVM/RAID0.

Cheers
Martin

=====
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

Re: RAID-0 read perf. decrease after 2.4.20

[Martin Knoblauch]

>>
>
>And this isn't the read-ahead size change thing?
>

 Hmm. Sounds interesting/promising. Do you have any pointers to the
change? Which kind of disks does it affect? All? SCSI? IDE?

Thanks
Martin

=====
------------------------------------------------------
Martin Knoblauch
email: knobi@knobisoft.de or knobi@rocketmail.com
www:   http://www.knobisoft.de

Re: RAID-0 read perf. decrease after 2.4.20

[Joshua Schmidlkofer]

On Mon, 2003-11-24 at 02:05, Martin Knoblauch wrote:
> >Hello All!
> >
> >Has anyone else experienced a drastic drop in read performance on
> >software
> >RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
> >lab with double IDE disks on different channels set up as RAID-0. Some
> >bonnie++ benchmark results with various kernels, on the same machine
> >(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
> >6Y060L0):
> >write read
> >2.4.20-ac1: 88,000 135,000 K/sec
> >2.4.21-pre7: 93,000 75,000
> >2.4.22-ac4: 94,000 82,000
> 
> Hi,
> 
>  I can attest a similar drop in read performance on a IA64 box going
> from a 2.4.19ish kernel to 2.4.22. In our setup the RAID0 is LVM, not
> MD.The RAID is used a a scratch device for a out-of-core finite element
> program (NASTRAN).
> 
>  The setup is some 20 disks on 4 controllers. "iozone" read/reread
> Performance went from about 400MB/sec to 260 MB/sec, while write went
> up a notch. Unfortunatelly the read performance is more important for
> the application in question.
> 
>  Due to the fact that I have no controll over the use of the system I
> cannot make any experiments to find out what killed performance. Sorry
> :-(
> 
> Martin
> 
> =====
> ------------------------------------------------------
> Martin Knoblauch
> email: knobi@knobisoft.de or knobi@rocketmail.com
> www:   http://www.knobisoft.de
> 
> 

And this isn't the read-ahead size change thing?

Re: RAID-0 read perf. decrease after 2.4.20

[Martin Knoblauch]

>Hello All!
>
>Has anyone else experienced a drastic drop in read performance on
>software
>RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
>lab with double IDE disks on different channels set up as RAID-0. Some
>bonnie++ benchmark results with various kernels, on the same machine
>(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
>6Y060L0):
>write read
>2.4.20-ac1: 88,000 135,000 K/sec
>2.4.21-pre7: 93,000 75,000
>2.4.22-ac4: 94,000 82,000

Hi,

 I can attest a similar drop in read performance on a IA64 box going
from a 2.4.19ish kernel to 2.4.22. In our setup the RAID0 is LVM, not
MD.The RAID is used a a scratch device for a out-of-core finite element
program (NASTRAN).

 The setup is some 20 disks on 4 controllers. "iozone" read/reread
Performance went from about 400MB/sec to 260 MB/sec, while write went
up a notch. Unfortunatelly the read performance is more important for
the application in question.

 Due to the fact that I have no controll over the use of the system I
cannot make any experiments to find out what killed performance. Sorry
:-(

Martin

=====
------------------------------------------------------
Martin Knoblauch
email: knobi@knobisoft.de or knobi@rocketmail.com
www:   http://www.knobisoft.de