* RAID-0 read perf. decrease after 2.4.20
@ 2003-11-20 16:45 Mikael Johansson
2003-11-21 13:47 ` Marcelo Tosatti
0 siblings, 1 reply; 15+ messages in thread
From: Mikael Johansson @ 2003-11-20 16:45 UTC (permalink / raw)
To: linux-raid, linux-kernel
Hello All!
Has anyone else experienced a drastic drop in read performance on software
RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
lab with double IDE disks on different channels set up as RAID-0. Some
bonnie++ benchmark results with various kernels, on the same machine
(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
6Y060L0):
write read
2.4.20-ac1: 88,000 135,000 K/sec
2.4.21-pre7: 93,000 75,000
2.4.22-ac4: 94,000 82,000
So the write speed has gone up a bit, but the read speed performance has
plunged. Any ideas on what happened between 2.4.20 and 2.4.21 and what to
do about it? I'm eagerly awaiting suggestions, good read speed is quite
critical for many of our calculations :-) I will of course provide more
details if necessary.
Have a nice day,
Mikael J.
http://www.helsinki.fi/~mpjohans/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-11-20 16:45 RAID-0 read perf. decrease after 2.4.20 Mikael Johansson
@ 2003-11-21 13:47 ` Marcelo Tosatti
2003-11-21 19:56 ` Mikael Johansson
0 siblings, 1 reply; 15+ messages in thread
From: Marcelo Tosatti @ 2003-11-21 13:47 UTC (permalink / raw)
To: Mikael Johansson; +Cc: linux-raid, linux-kernel
On Thu, 20 Nov 2003, Mikael Johansson wrote:
>
> Hello All!
>
> Has anyone else experienced a drastic drop in read performance on software
> RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
> lab with double IDE disks on different channels set up as RAID-0. Some
> bonnie++ benchmark results with various kernels, on the same machine
> (Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
> 6Y060L0):
> write read
> 2.4.20-ac1: 88,000 135,000 K/sec
> 2.4.21-pre7: 93,000 75,000
> 2.4.22-ac4: 94,000 82,000
>
> So the write speed has gone up a bit, but the read speed performance has
> plunged. Any ideas on what happened between 2.4.20 and 2.4.21 and what to
> do about it? I'm eagerly awaiting suggestions, good read speed is quite
> critical for many of our calculations :-) I will of course provide more
> details if necessary.
There have been no significant changes in the RAID driver in 2.4.21, so I
suspect the cause for the slowdowns might the changes to the IO scheduler
(ll_rw_blk.c) or VM changes.
Isolating the -pre which the slowdown starts helps a lot.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-11-21 13:47 ` Marcelo Tosatti
@ 2003-11-21 19:56 ` Mikael Johansson
2003-12-08 12:47 ` Marcelo Tosatti
0 siblings, 1 reply; 15+ messages in thread
From: Mikael Johansson @ 2003-11-21 19:56 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-raid, linux-kernel
Hello Marcelo and All!
On Fri, 21 Nov 2003, Marcelo Tosatti wrote:
> There have been no significant changes in the RAID driver in 2.4.21, so I
> suspect the cause for the slowdowns might the changes to the IO scheduler
> (ll_rw_blk.c) or VM changes.
>
> Isolating the -pre which the slowdown starts helps a lot.
I tested a few kernels to pin point the where tha changes occured. It
turned out that both 2.4.20 and 2.4.21-pre1 have bad read performance. The
2.4.20-ac's show good read speed. I tested it on two machines (different
from yesterday). I also checked the VIA chipset drivers version; that's
not the reason for the differences.
Athlon XP 2400+, 2.09 GHz, 1.5 GB RAM, 2*160 GB Maxtor Maxtor 6Y080L0
VIA write read
2.4.19 none 10,000 9,000 K/sec (chipset not supported)
2.4.20 3.35 73,000 88,000
2.4.20-ac1 3.35-ac 70,000 135,000
2.4.20-ac2 3.35-ac 71,000 140,000
2.4.21-pre1 3.35-ac 71,000 79,000
2.4.21-pre3 3.35-ac 71,000 79,000
Athlon 1.4 GHz, 1.5 GB RAM, 2*80 GB IC35L040AVER07-0 (IBM, I think)
2.4.13-ac8 ? 49,000 44,000 K/sec
2.4.19 3.34 53,000 41,000
2.4.20-ac2 3.35-ac 50,000 69,000
2.4.21-pre1 3.35-ac 53,000 46,000
So there was apparently something very right with the 2.4.20-ac's. It
would be nice to get it back :-)
Have a nice weekend,
Mikael J.
http://www.helsinki.fi/~mpjohans/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-11-21 19:56 ` Mikael Johansson
@ 2003-12-08 12:47 ` Marcelo Tosatti
2003-12-08 13:11 ` Marc-Christian Petersen
0 siblings, 1 reply; 15+ messages in thread
From: Marcelo Tosatti @ 2003-12-08 12:47 UTC (permalink / raw)
To: Mikael Johansson; +Cc: linux-raid, linux-kernel, riel, knobi, Jens Axboe, mason
On Fri, 21 Nov 2003, Mikael Johansson wrote:
>
> Hello Marcelo and All!
Hi Mikael,
>
> On Fri, 21 Nov 2003, Marcelo Tosatti wrote:
>
> > There have been no significant changes in the RAID driver in 2.4.21, so I
> > suspect the cause for the slowdowns might the changes to the IO scheduler
> > (ll_rw_blk.c) or VM changes.
> >
> > Isolating the -pre which the slowdown starts helps a lot.
>
> I tested a few kernels to pin point the where tha changes occured. It
> turned out that both 2.4.20 and 2.4.21-pre1 have bad read performance. The
> 2.4.20-ac's show good read speed. I tested it on two machines (different
> from yesterday). I also checked the VIA chipset drivers version; that's
> not the reason for the differences.
>
> Athlon XP 2400+, 2.09 GHz, 1.5 GB RAM, 2*160 GB Maxtor Maxtor 6Y080L0
> VIA write read
> 2.4.19 none 10,000 9,000 K/sec (chipset not supported)
> 2.4.20 3.35 73,000 88,000
> 2.4.20-ac1 3.35-ac 70,000 135,000
> 2.4.20-ac2 3.35-ac 71,000 140,000
> 2.4.21-pre1 3.35-ac 71,000 79,000
> 2.4.21-pre3 3.35-ac 71,000 79,000
>
> Athlon 1.4 GHz, 1.5 GB RAM, 2*80 GB IC35L040AVER07-0 (IBM, I think)
> 2.4.13-ac8 ? 49,000 44,000 K/sec
> 2.4.19 3.34 53,000 41,000
> 2.4.20-ac2 3.35-ac 50,000 69,000
> 2.4.21-pre1 3.35-ac 53,000 46,000
>
> So there was apparently something very right with the 2.4.20-ac's. It
> would be nice to get it back :-)
2.4.20-aa included rmap and some VM modifications most notably
"drop_behind()" logic which I believe should be the reason for the huge
read speedups. Can you please try it? Against 2.4.23.
--- mm/filemap.c 2003-12-08 10:26:58.000000000 -0200
+++ mm/filemap.c.orig 2003-12-08 10:24:57.000000000 -0200
@@ -1055,56 +1055,6 @@
return page;
}
-
- /* We combine this with read-ahead to deactivate pages when we
- * think there's sequential IO going on. Note that this is
- * harmless since we don't actually evict the pages from memory
- * but just move them to the inactive list.
- *
- * TODO:
- * - make the readahead code smarter
- * - move readahead to the VMA level so we can do the same
- * trick with mmap()
- *
- * Rik van Riel, 2000
- */
-static void drop_behind(struct file * file, unsigned long index)
-{
- struct inode *inode = file->f_dentry->d_inode;
- struct address_space *mapping = inode->i_mapping;
- struct page *page;
- unsigned long start;
-
- /* Nothing to drop-behind if we're on the first page. */
- if (!index)
- return;
-
- if (index > file->f_rawin)
- start = index - file->f_rawin;
- else
- start = 0;
-
- /*
- * Go backwards from index-1 and drop all pages in the
- * readahead window. Since the readahead window may have
- * been increased since the last time we were called, we
- * stop when the page isn't there.
- */
- spin_lock(&pagemap_lru_lock);
- while (--index >= start) {
- struct page **hash = page_hash(mapping, index);
- spin_lock(&pagecache_lock);
- page = __find_page_nolock(mapping, index, *hash);
- spin_unlock(&pagecache_lock);
- if (!page || !PageActive(page))
- break;
- drop_page(page);
- }
- spin_unlock(&pagemap_lru_lock);
-}
-
-
-
/*
* Same as grab_cache_page, but do not wait if the page is unavailable.
* This is intended for speculative data generators, where the data can
@@ -1376,12 +1326,6 @@
if (filp->f_ramax > max_readahead)
filp->f_ramax = max_readahead;
- /*
- * Move the pages that have already been passed
- * to the inactive list.
- */
- drop_behind(filp, index);
-
#ifdef PROFILE_READAHEAD
profile_readahead((reada_ok == 2), filp);
#endif
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-08 12:47 ` Marcelo Tosatti
@ 2003-12-08 13:11 ` Marc-Christian Petersen
2003-12-08 13:21 ` Marcelo Tosatti
0 siblings, 1 reply; 15+ messages in thread
From: Marc-Christian Petersen @ 2003-12-08 13:11 UTC (permalink / raw)
To: Marcelo Tosatti, Mikael Johansson
Cc: linux-raid, linux-kernel, riel, knobi, Jens Axboe, mason
On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
Hi Marcelo,
> 2.4.20-aa included rmap and some VM modifications most notably
> "drop_behind()" logic which I believe should be the reason for the huge
> read speedups. Can you please try it? Against 2.4.23.
-aa tree never had -rmap :)
ciao, Marc
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-08 13:11 ` Marc-Christian Petersen
@ 2003-12-08 13:21 ` Marcelo Tosatti
0 siblings, 0 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2003-12-08 13:21 UTC (permalink / raw)
To: Marc-Christian Petersen
Cc: Marcelo Tosatti, Mikael Johansson, linux-raid, linux-kernel,
riel, knobi, Jens Axboe, mason
On Mon, 8 Dec 2003, Marc-Christian Petersen wrote:
> On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
>
> Hi Marcelo,
>
> > 2.4.20-aa included rmap and some VM modifications most notably
> > "drop_behind()" logic which I believe should be the reason for the huge
> > read speedups. Can you please try it? Against 2.4.23.
>
> -aa tree never had -rmap :)
-ac I mean, sorry.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-18 14:37 ` Martin Knoblauch
@ 2004-07-04 23:28 ` Mikael Johansson
0 siblings, 0 replies; 15+ messages in thread
From: Mikael Johansson @ 2004-07-04 23:28 UTC (permalink / raw)
To: Martin Knoblauch; +Cc: Marcelo Tosatti, linux-kernel
Hello again!
Finally had time to test this on IDE, "better late than even later"...
On Thu, 18 Dec 2003, Martin Knoblauch wrote:
> > > Just some feedback:
> > >
> > > echo 511 > /proc/sys/vm/max-readahead
> > >
> > > brings back the read performance of my 30 disks on 4 controller
> > > LVM/RAID0.
> >
> > Great.
> >
>
> Indeed :-) Just to clarify - the modification of max-readahead was
> sufficient to "fix" the observed read performance degradation. I did
> not apply (or reverse) anything on top of 2.4.22.
>
> Actually 255 would have been sufficient, 511 proved to be a small bit
> better :-))
The "max-readahead" does not seem to affect IDE-RAID, at least not with
the 2.4.27-pre6 I just compiled on two machines. So the problem still
exists, read speed on 2.4.20-ac2 is almost twice as fast as on later
kernels :-/
Have a nice summer,
Mikael J.
http://www.helsinki.fi/~mpjohans/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-18 19:13 ` Mike Fedyk
@ 2003-12-18 19:17 ` Marcelo Tosatti
0 siblings, 0 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2003-12-18 19:17 UTC (permalink / raw)
To: Mike Fedyk; +Cc: Marcelo Tosatti, Martin Knoblauch, linux-kernel
On Thu, 18 Dec 2003, Mike Fedyk wrote:
> On Thu, Dec 18, 2003 at 11:41:11AM -0200, Marcelo Tosatti wrote:
> > On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> > > Just some feedback:
> > >
> > > echo 511 > /proc/sys/vm/max-readahead
> > >
> > > brings back the read performance of my 30 disks on 4 controller
> > > LVM/RAID0.
> >
> > Great.
>
> Maybe a new default is in order?
I believe the defaults have been changed because they harm smaller
workloads.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-18 13:41 ` Marcelo Tosatti
2003-12-18 14:37 ` Martin Knoblauch
@ 2003-12-18 19:13 ` Mike Fedyk
2003-12-18 19:17 ` Marcelo Tosatti
1 sibling, 1 reply; 15+ messages in thread
From: Mike Fedyk @ 2003-12-18 19:13 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Martin Knoblauch, linux-kernel
On Thu, Dec 18, 2003 at 11:41:11AM -0200, Marcelo Tosatti wrote:
> On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> > Just some feedback:
> >
> > echo 511 > /proc/sys/vm/max-readahead
> >
> > brings back the read performance of my 30 disks on 4 controller
> > LVM/RAID0.
>
> Great.
Maybe a new default is in order?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-18 13:41 ` Marcelo Tosatti
@ 2003-12-18 14:37 ` Martin Knoblauch
2004-07-04 23:28 ` Mikael Johansson
2003-12-18 19:13 ` Mike Fedyk
1 sibling, 1 reply; 15+ messages in thread
From: Martin Knoblauch @ 2003-12-18 14:37 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-kernel
--- Marcelo Tosatti <marcelo.tosatti@cyclades.com> wrote:
>
>
> On Tue, 16 Dec 2003, Martin Knoblauch wrote:
>
> > On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
> >
> > Hi Marcelo,
> >
> > > 2.4.20-aa included rmap and some VM modifications most notably
> > > "drop_behind()" logic which I believe should be the reason for
> the
> > huge
> > > read speedups. Can you please try it? Against 2.4.23.
> >
> > Just some feedback:
> >
> > echo 511 > /proc/sys/vm/max-readahead
> >
> > brings back the read performance of my 30 disks on 4 controller
> > LVM/RAID0.
>
> Great.
>
Indeed :-) Just to clarify - the modification of max-readahead was
sufficient to "fix" the observed read performance degradation. I did
not apply (or reverse) anything on top of 2.4.22.
Actually 255 would have been sufficient, 511 proved to be a small bit
better :-))
Martin
=====
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-12-16 12:51 Martin Knoblauch
@ 2003-12-18 13:41 ` Marcelo Tosatti
2003-12-18 14:37 ` Martin Knoblauch
2003-12-18 19:13 ` Mike Fedyk
0 siblings, 2 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2003-12-18 13:41 UTC (permalink / raw)
To: Martin Knoblauch; +Cc: linux-kernel
On Tue, 16 Dec 2003, Martin Knoblauch wrote:
> On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
>
> Hi Marcelo,
>
> > 2.4.20-aa included rmap and some VM modifications most notably
> > "drop_behind()" logic which I believe should be the reason for the
> huge
> > read speedups. Can you please try it? Against 2.4.23.
>
> Just some feedback:
>
> echo 511 > /proc/sys/vm/max-readahead
>
> brings back the read performance of my 30 disks on 4 controller
> LVM/RAID0.
Great.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
@ 2003-12-16 12:51 Martin Knoblauch
2003-12-18 13:41 ` Marcelo Tosatti
0 siblings, 1 reply; 15+ messages in thread
From: Martin Knoblauch @ 2003-12-16 12:51 UTC (permalink / raw)
To: linux-kernel
On Monday 08 December 2003 13:47, Marcelo Tosatti wrote:
Hi Marcelo,
> 2.4.20-aa included rmap and some VM modifications most notably
> "drop_behind()" logic which I believe should be the reason for the
huge
> read speedups. Can you please try it? Against 2.4.23.
Just some feedback:
echo 511 > /proc/sys/vm/max-readahead
brings back the read performance of my 30 disks on 4 controller
LVM/RAID0.
Cheers
Martin
=====
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
@ 2003-11-25 9:41 Martin Knoblauch
0 siblings, 0 replies; 15+ messages in thread
From: Martin Knoblauch @ 2003-11-25 9:41 UTC (permalink / raw)
To: linux-kernel; +Cc: kernel
>>
>
>And this isn't the read-ahead size change thing?
>
Hmm. Sounds interesting/promising. Do you have any pointers to the
change? Which kind of disks does it affect? All? SCSI? IDE?
Thanks
Martin
=====
------------------------------------------------------
Martin Knoblauch
email: knobi@knobisoft.de or knobi@rocketmail.com
www: http://www.knobisoft.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
2003-11-24 10:05 Martin Knoblauch
@ 2003-11-25 5:01 ` Joshua Schmidlkofer
0 siblings, 0 replies; 15+ messages in thread
From: Joshua Schmidlkofer @ 2003-11-25 5:01 UTC (permalink / raw)
To: Linux Kernel Mailing List
On Mon, 2003-11-24 at 02:05, Martin Knoblauch wrote:
> >Hello All!
> >
> >Has anyone else experienced a drastic drop in read performance on
> >software
> >RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
> >lab with double IDE disks on different channels set up as RAID-0. Some
> >bonnie++ benchmark results with various kernels, on the same machine
> >(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
> >6Y060L0):
> >write read
> >2.4.20-ac1: 88,000 135,000 K/sec
> >2.4.21-pre7: 93,000 75,000
> >2.4.22-ac4: 94,000 82,000
>
> Hi,
>
> I can attest a similar drop in read performance on a IA64 box going
> from a 2.4.19ish kernel to 2.4.22. In our setup the RAID0 is LVM, not
> MD.The RAID is used a a scratch device for a out-of-core finite element
> program (NASTRAN).
>
> The setup is some 20 disks on 4 controllers. "iozone" read/reread
> Performance went from about 400MB/sec to 260 MB/sec, while write went
> up a notch. Unfortunatelly the read performance is more important for
> the application in question.
>
> Due to the fact that I have no controll over the use of the system I
> cannot make any experiments to find out what killed performance. Sorry
> :-(
>
> Martin
>
> =====
> ------------------------------------------------------
> Martin Knoblauch
> email: knobi@knobisoft.de or knobi@rocketmail.com
> www: http://www.knobisoft.de
>
>
And this isn't the read-ahead size change thing?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: RAID-0 read perf. decrease after 2.4.20
@ 2003-11-24 10:05 Martin Knoblauch
2003-11-25 5:01 ` Joshua Schmidlkofer
0 siblings, 1 reply; 15+ messages in thread
From: Martin Knoblauch @ 2003-11-24 10:05 UTC (permalink / raw)
To: linux-kernel
>Hello All!
>
>Has anyone else experienced a drastic drop in read performance on
>software
>RAID-0 with post 2.4.20 kernels? We have a few Athlon XP's here at our
>lab with double IDE disks on different channels set up as RAID-0. Some
>bonnie++ benchmark results with various kernels, on the same machine
>(Athlon XP 2400+, 2 GHz, 1.5 GB RAM, VIA chipset, 2*Maxtor 120 GB
>6Y060L0):
>write read
>2.4.20-ac1: 88,000 135,000 K/sec
>2.4.21-pre7: 93,000 75,000
>2.4.22-ac4: 94,000 82,000
Hi,
I can attest a similar drop in read performance on a IA64 box going
from a 2.4.19ish kernel to 2.4.22. In our setup the RAID0 is LVM, not
MD.The RAID is used a a scratch device for a out-of-core finite element
program (NASTRAN).
The setup is some 20 disks on 4 controllers. "iozone" read/reread
Performance went from about 400MB/sec to 260 MB/sec, while write went
up a notch. Unfortunatelly the read performance is more important for
the application in question.
Due to the fact that I have no controll over the use of the system I
cannot make any experiments to find out what killed performance. Sorry
:-(
Martin
=====
------------------------------------------------------
Martin Knoblauch
email: knobi@knobisoft.de or knobi@rocketmail.com
www: http://www.knobisoft.de
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2004-07-04 23:28 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-11-20 16:45 RAID-0 read perf. decrease after 2.4.20 Mikael Johansson
2003-11-21 13:47 ` Marcelo Tosatti
2003-11-21 19:56 ` Mikael Johansson
2003-12-08 12:47 ` Marcelo Tosatti
2003-12-08 13:11 ` Marc-Christian Petersen
2003-12-08 13:21 ` Marcelo Tosatti
2003-11-24 10:05 Martin Knoblauch
2003-11-25 5:01 ` Joshua Schmidlkofer
2003-11-25 9:41 Martin Knoblauch
2003-12-16 12:51 Martin Knoblauch
2003-12-18 13:41 ` Marcelo Tosatti
2003-12-18 14:37 ` Martin Knoblauch
2004-07-04 23:28 ` Mikael Johansson
2003-12-18 19:13 ` Mike Fedyk
2003-12-18 19:17 ` Marcelo Tosatti
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).